Domo API to import files to domo

sobo · May 14

domo = Domo(client_id, client_secret,api_host=api_host)

I want to add an additional argument here that is updatemethod=Append. I am not sure what the correct syntax for this would be.

For extra context,

I am currently reading files into python and essentially once I read these files in, I want to combine them and then upload them to Domo. The issue is that in total these files have 8,000,000 rows and I am unable to upload them all at once. I was wondering if I could upload one file at a time to the same dataset ID and append them to the existing data.

Can someone please help me with this?

ArborRose · May 14

To add the updatemethod=Append argument to your Domo object instantiation in Python, you can include it as a keyword argument like this

domo = Domo(client_id, client_secret, api_host=api_host, updatemethod="Append")

MattTheGuru · May 14

Yeah, let's see if we can get this sorted out.

Are you having trouble with:
1.) The aggregating of data in python?
2.) The uploading of files?
3.) The appending of data?
…or all three of the above items?

Could you send a screenshot of what you have so far? This might be a larger task than a simple answer, but we'll see what we can get done for you!

ArborRose · May 14

To add the updatemethod=Append argument to your Domo object instantiation in Python, you can include it as a keyword argument like this

domo = Domo(client_id, client_secret, api_host=api_host, updatemethod="Append")

sobo · May 14

I am able to bring in the files from my server and append them into a dataframe with 8,000,000 rows. But I can't upload this to domo because I get an error. I even divided them into many different dataframes and tried to upload them as different domo datasets but its didn't work

I get this error: too many non-consecutive part ids. Make consecutive or upload multiple data versions python error

def func1():
filePath = 'filepath

numOfFiles = len(sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True))
print(numOfFiles)
df_Combined = pd.DataFrame()

for i in range(0,numOfFiles):
    mostRecentFilePath = sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True)[i]
    print(str(i), ': ', mostRecentFilePath)
    
    

    df = pd.read_excel(mostRecentFilePath)

    df['File Name'] = mostRecentFilePath
    df_Combined = df_Combined.append(df,ignore_index=True)

df_Combined1 = df_Combined[:800000]
df_Combined2 = df_Combined[800000:1600000]
df_Combined3 = df_Combined[1600000:2400000]
df_Combined4 = df_Combined[2400000:3200000]
df_Combined5 = df_Combined[3200000:4000000]
df_Combined6 = df_Combined[4000000:]

ArborRose · May 14

One possible solution is to ensure that each chunk you upload is consecutive in terms of the data. It seems like you're attempting to split the data into chunks of 800,000 rows each, but it's possible that the rows aren't consecutive across these chunks.

Perhaps something like this:

Import necessary libraries
import pandas as pd

import glob

import os
Function to read and append files
def func1():

filePath = 'filepath'

numOfFiles = len(sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True))

print(numOfFiles)

df_chunks = []
# Read and append files into chunks

for i in range(0, numOfFiles):

    mostRecentFilePath = sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True)[i]

    print(str(i), ': ', mostRecentFilePath)


    df = pd.read_excel(mostRecentFilePath)

    df['File Name'] = mostRecentFilePath

    df_chunks.append(df)


# Upload each chunk separately

for idx, df_chunk in enumerate(df_chunks):

    # Split the chunk into smaller parts if necessary

    for i in range(0, len(df_chunk), 800000):

        df_part = df_chunk[i:i+800000]

        # Upload df_part to Domo here

        # For example:

        # domo.upload_dataframe(df_part, dataset_name=f'dataset_part_{idx}_{i}')

Domo API to import files to domo

Best Answer

Answers

Categories