Domo API to import files to domo

Options

domo = Domo(client_id, client_secret,api_host=api_host)

I want to add an additional argument here that is updatemethod=Append. I am not sure what the correct syntax for this would be.

For extra context,

I am currently reading files into python and essentially once I read these files in, I want to combine them and then upload them to Domo. The issue is that in total these files have 8,000,000 rows and I am unable to upload them all at once. I was wondering if I could upload one file at a time to the same dataset ID and append them to the existing data.

Can someone please help me with this?

Tagged:

Best Answer

  • ArborRose
    ArborRose Coach
    Answer ✓
    Options

    To add the updatemethod=Append argument to your Domo object instantiation in Python, you can include it as a keyword argument like this

    domo = Domo(client_id, client_secret, api_host=api_host, updatemethod="Append")

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

Answers

  • MattTheGuru
    MattTheGuru Contributor
    Options

    Yeah, let's see if we can get this sorted out.

    Are you having trouble with:
    1.) The aggregating of data in python?
    2.) The uploading of files?
    3.) The appending of data?
    …or all three of the above items?

    Could you send a screenshot of what you have so far? This might be a larger task than a simple answer, but we'll see what we can get done for you!

    Please 💡/💖/👍/😊 this post if you read it and found it helpful.
    Please accept the answer if it solved your problem.

  • ArborRose
    ArborRose Coach
    Answer ✓
    Options

    To add the updatemethod=Append argument to your Domo object instantiation in Python, you can include it as a keyword argument like this

    domo = Domo(client_id, client_secret, api_host=api_host, updatemethod="Append")

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

  • sobo
    sobo Member
    Options

    I am able to bring in the files from my server and append them into a dataframe with 8,000,000 rows. But I can't upload this to domo because I get an error. I even divided them into many different dataframes and tried to upload them as different domo datasets but its didn't work

    I get this error: too many non-consecutive part ids. Make consecutive or upload multiple data versions python error

    def func1():
    filePath = 'filepath

    numOfFiles = len(sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True))
    print(numOfFiles)
    df_Combined = pd.DataFrame()
    
    for i in range(0,numOfFiles):
        mostRecentFilePath = sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True)[i]
        print(str(i), ': ', mostRecentFilePath)
        
        
    
        df = pd.read_excel(mostRecentFilePath)
    
        df['File Name'] = mostRecentFilePath
        df_Combined = df_Combined.append(df,ignore_index=True)
    
    df_Combined1 = df_Combined[:800000]
    df_Combined2 = df_Combined[800000:1600000]
    df_Combined3 = df_Combined[1600000:2400000]
    df_Combined4 = df_Combined[2400000:3200000]
    df_Combined5 = df_Combined[3200000:4000000]
    df_Combined6 = df_Combined[4000000:]
    

  • ArborRose
    Options

    One possible solution is to ensure that each chunk you upload is consecutive in terms of the data. It seems like you're attempting to split the data into chunks of 800,000 rows each, but it's possible that the rows aren't consecutive across these chunks.

    Perhaps something like this:

    Import necessary libraries
    import pandas as pd
    import glob
    import os Function to read and append files def func1():
    filePath = 'filepath'
    numOfFiles = len(sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True))
    print(numOfFiles)
    df_chunks = [] # Read and append files into chunks
    for i in range(0, numOfFiles):
    mostRecentFilePath = sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True)[i]
    print(str(i), ': ', mostRecentFilePath)

    df = pd.read_excel(mostRecentFilePath)
    df['File Name'] = mostRecentFilePath
    df_chunks.append(df)

    # Upload each chunk separately
    for idx, df_chunk in enumerate(df_chunks):
    # Split the chunk into smaller parts if necessary
    for i in range(0, len(df_chunk), 800000):
    df_part = df_chunk[i:i+800000]
    # Upload df_part to Domo here
    # For example:
    # domo.upload_dataframe(df_part, dataset_name=f'dataset_part_{idx}_{i}')

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **