Domo API to import files to domo

sobo
sobo Member
in APIs & Domo Developer

domo = Domo(client_id, client_secret,api_host=api_host)

I want to add an additional argument here that is updatemethod=Append. I am not sure what the correct syntax for this would be.

For extra context,

I am currently reading files into python and essentially once I read these files in, I want to combine them and then upload them to Domo. The issue is that in total these files have 8,000,000 rows and I am unable to upload them all at once. I was wondering if I could upload one file at a time to the same dataset ID and append them to the existing data.

Can someone please help me with this?

Tagged:

Best Answer

  • ArborRose
    ArborRose Coach
    Answer βœ“

    To add the updatemethod=Append argument to your Domo object instantiation in Python, you can include it as a keyword argument like this

    domo = Domo(client_id, client_secret, api_host=api_host, updatemethod="Append")

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

Answers

  • MattTheGuru
    MattTheGuru Contributor

    Yeah, let's see if we can get this sorted out.

    Are you having trouble with:
    1.) The aggregating of data in python?
    2.) The uploading of files?
    3.) The appending of data?
    …or all three of the above items?

    Could you send a screenshot of what you have so far? This might be a larger task than a simple answer, but we'll see what we can get done for you!

    ** Was this post helpful? Click πŸ’‘/πŸ’–/πŸ‘/😊 below. **
    ** If it solved your problem. Accept it as a solution! βœ”οΈ **

    Or do you need more help? https://calendly.com/matthew-kastner/15-minute-chat
    Did I help you out? Feedback is priceless and will help me more than you know.Write a review!

  • ArborRose
    ArborRose Coach
    Answer βœ“

    To add the updatemethod=Append argument to your Domo object instantiation in Python, you can include it as a keyword argument like this

    domo = Domo(client_id, client_secret, api_host=api_host, updatemethod="Append")

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

  • sobo
    sobo Member

    I am able to bring in the files from my server and append them into a dataframe with 8,000,000 rows. But I can't upload this to domo because I get an error. I even divided them into many different dataframes and tried to upload them as different domo datasets but its didn't work

    I get this error: too many non-consecutive part ids. Make consecutive or upload multiple data versions python error

    def func1():
    filePath = 'filepath

    numOfFiles = len(sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True))
print(numOfFiles)
df_Combined = pd.DataFrame()

for i in range(0,numOfFiles):
    mostRecentFilePath = sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True)[i]
    print(str(i), ': ', mostRecentFilePath)
    
    

    df = pd.read_excel(mostRecentFilePath)

    df['File Name'] = mostRecentFilePath
    df_Combined = df_Combined.append(df,ignore_index=True)

df_Combined1 = df_Combined[:800000]
df_Combined2 = df_Combined[800000:1600000]
df_Combined3 = df_Combined[1600000:2400000]
df_Combined4 = df_Combined[2400000:3200000]
df_Combined5 = df_Combined[3200000:4000000]
df_Combined6 = df_Combined[4000000:]
  • ArborRose
    ArborRose Coach

    One possible solution is to ensure that each chunk you upload is consecutive in terms of the data. It seems like you're attempting to split the data into chunks of 800,000 rows each, but it's possible that the rows aren't consecutive across these chunks.

    Perhaps something like this:

    Import necessary libraries
import pandas as pd
    
import glob
    
import os
Function to read and append files
def func1():
    
filePath = 'filepath'
    
numOfFiles = len(sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True))
    
print(numOfFiles)
    
df_chunks = []
# Read and append files into chunks
    
for i in range(0, numOfFiles):
    
    mostRecentFilePath = sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True)[i]
    
    print(str(i), ': ', mostRecentFilePath)

    
    df = pd.read_excel(mostRecentFilePath)
    
    df['File Name'] = mostRecentFilePath
    
    df_chunks.append(df)

    
# Upload each chunk separately
    
for idx, df_chunk in enumerate(df_chunks):
    
    # Split the chunk into smaller parts if necessary
    
    for i in range(0, len(df_chunk), 800000):
    
        df_part = df_chunk[i:i+800000]
    
        # Upload df_part to Domo here
    
        # For example:
    
        # domo.upload_dataframe(df_part, dataset_name=f'dataset_part_{idx}_{i}')

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

Categories