Dataset not updating.

SaranyaT

Can someone help me with this issue?

I'm trying to pull the data from databricks (using python script) into domo dataset.

First I'm trying to create the dataset for a single date using

#domo.ds_create(df2,'datasetname')

then I'm trying to update the same dataset daily. the below code worked till yesterday and its only replacing it from today onwards. not sure what went wrong.

import pandas as pd
import numpy as np

df = spark.sql("Select a, b,c from table a where date > 2023-08-01") — giving a example query

df2 = df.toPandas()

domo.ds_get('datasetid')

Update the Domo dataset

domo.ds_update('datasetid', df_up = df2)

Retrieve dataset from Domo

dataset_id = 'datasetid'
try:
original_data = domo.ds_get(dataset_id)
except Exception as error:
print(f'Error retrieving dataset: {error}')
# Handle error as needed
...

Check if data is empty

if original_data.empty:
print('No data retrieved from Domo')
# Handle empty data as needed
...
else:
# Append data to itself
updated_data = original_data.append(original_data).reset_index(drop=True)

# Update dataset in Domo

try:

    domo.ds_update(dataset_id, df_up=updated_data)

except Exception as error:

    print(f'Error updating dataset: {error}')

    # Handle error as needed

GrantSmith

Have you validated the data you're getting back from your SQL query? Is it returning all of the data you're expecting?