APIs & Domo Developer

APIs & Domo Developer

Domo API to import files to domo

domo = Domo(client_id, client_secret,api_host=api_host)

I want to add an additional argument here that is updatemethod=Append. I am not sure what the correct syntax for this would be.

For extra context,

I am currently reading files into python and essentially once I read these files in, I want to combine them and then upload them to Domo. The issue is that in total these files have 8,000,000 rows and I am unable to upload them all at once. I was wondering if I could upload one file at a time to the same dataset ID and append them to the existing data.

Can someone please help me with this?

Tagged:

Welcome!

It looks like you're new here. Members get access to exclusive content, events, rewards, and more. Sign in or register to get started.
Sign In

Best Answer

  • Coach
    Answer ✓

    To add the updatemethod=Append argument to your Domo object instantiation in Python, you can include it as a keyword argument like this

    domo = Domo(client_id, client_secret, api_host=api_host, updatemethod="Append")

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

Answers

  • Contributor

    Yeah, let's see if we can get this sorted out.

    Are you having trouble with:
    1.) The aggregating of data in python?
    2.) The uploading of files?
    3.) The appending of data?
    …or all three of the above items?

    Could you send a screenshot of what you have so far? This might be a larger task than a simple answer, but we'll see what we can get done for you!

    ** Was this post helpful? Click 💡/💖/👍/😊 below. **
    ** If it solved your problem. Accept it as a solution! ✔️ **

    Or do you need more help? https://calendly.com/matthew-kastner/15-minute-chat
    Did I help you out? Feedback is priceless and will help me more than you know.Write a review!

  • Coach
    Answer ✓

    To add the updatemethod=Append argument to your Domo object instantiation in Python, you can include it as a keyword argument like this

    domo = Domo(client_id, client_secret, api_host=api_host, updatemethod="Append")

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

  • Member

    I am able to bring in the files from my server and append them into a dataframe with 8,000,000 rows. But I can't upload this to domo because I get an error. I even divided them into many different dataframes and tried to upload them as different domo datasets but its didn't work

    I get this error: too many non-consecutive part ids. Make consecutive or upload multiple data versions python error

    def func1():
    filePath = 'filepath

    1. numOfFiles = len(sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True))
    2. print(numOfFiles)
    3. df_Combined = pd.DataFrame()
    4.  
    5. for i in range(0,numOfFiles):
    6. mostRecentFilePath = sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True)[i]
    7. print(str(i), ': ', mostRecentFilePath)
    8.  
    9. df = pd.read_excel(mostRecentFilePath)
    10.  
    11. df['File Name'] = mostRecentFilePath
    12. df_Combined = df_Combined.append(df,ignore_index=True)
    13.  
    14. df_Combined1 = df_Combined[:800000]
    15. df_Combined2 = df_Combined[800000:1600000]
    16. df_Combined3 = df_Combined[1600000:2400000]
    17. df_Combined4 = df_Combined[2400000:3200000]
    18. df_Combined5 = df_Combined[3200000:4000000]
    19. df_Combined6 = df_Combined[4000000:]
  • One possible solution is to ensure that each chunk you upload is consecutive in terms of the data. It seems like you're attempting to split the data into chunks of 800,000 rows each, but it's possible that the rows aren't consecutive across these chunks.

    Perhaps something like this:

    1. Import necessary libraries
    2. import pandas as pd
    3. import glob
    4. import os
    5. Function to read and append files
    6. def func1():
    7. filePath = 'filepath'
    8. numOfFiles = len(sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True))
    9. print(numOfFiles)
    10. df_chunks = []
    11. # Read and append files into chunks
    12. for i in range(0, numOfFiles):
    13. mostRecentFilePath = sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True)[i]
    14. print(str(i), ': ', mostRecentFilePath)

    15. df = pd.read_excel(mostRecentFilePath)
    16. df['File Name'] = mostRecentFilePath
    17. df_chunks.append(df)

    18. # Upload each chunk separately
    19. for idx, df_chunk in enumerate(df_chunks):
    20. # Split the chunk into smaller parts if necessary
    21. for i in range(0, len(df_chunk), 800000):
    22. df_part = df_chunk[i:i+800000]
    23. # Upload df_part to Domo here
    24. # For example:
    25. # domo.upload_dataframe(df_part, dataset_name=f'dataset_part_{idx}_{i}')

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

Welcome!

It looks like you're new here. Members get access to exclusive content, events, rewards, and more. Sign in or register to get started.
Sign In