Domo API to import files to domo
domo = Domo(client_id, client_secret,api_host=api_host)
I want to add an additional argument here that is updatemethod=Append. I am not sure what the correct syntax for this would be.
For extra context,
I am currently reading files into python and essentially once I read these files in, I want to combine them and then upload them to Domo. The issue is that in total these files have 8,000,000 rows and I am unable to upload them all at once. I was wondering if I could upload one file at a time to the same dataset ID and append them to the existing data.
Can someone please help me with this?
Best Answer
-
To add the updatemethod=Append argument to your Domo object instantiation in Python, you can include it as a keyword argument like this
domo = Domo(client_id, client_secret, api_host=api_host, updatemethod="Append")** Was this post helpful? Click Agree or Like below. **
** Did this solve your problem? Accept it as a solution! **1
Answers
-
Yeah, let's see if we can get this sorted out.
Are you having trouble with:
1.) The aggregating of data in python?
2.) The uploading of files?
3.) The appending of data?
…or all three of the above items?
Could you send a screenshot of what you have so far? This might be a larger task than a simple answer, but we'll see what we can get done for you!** Was this post helpful? Click 💡/💖/👍/😊 below. **
** If it solved your problem. Accept it as a solution! ✔️ **
Or do you need more help? https://calendly.com/matthew-kastner/15-minute-chat
Did I help you out? Feedback is priceless and will help me more than you know.Write a review!1 -
To add the updatemethod=Append argument to your Domo object instantiation in Python, you can include it as a keyword argument like this
domo = Domo(client_id, client_secret, api_host=api_host, updatemethod="Append")** Was this post helpful? Click Agree or Like below. **
** Did this solve your problem? Accept it as a solution! **1 -
I am able to bring in the files from my server and append them into a dataframe with 8,000,000 rows. But I can't upload this to domo because I get an error. I even divided them into many different dataframes and tried to upload them as different domo datasets but its didn't work
I get this error: too many non-consecutive part ids. Make consecutive or upload multiple data versions python error
def func1():
filePath = 'filepathnumOfFiles = len(sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True)) print(numOfFiles) df_Combined = pd.DataFrame() for i in range(0,numOfFiles): mostRecentFilePath = sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True)[i] print(str(i), ': ', mostRecentFilePath) df = pd.read_excel(mostRecentFilePath) df['File Name'] = mostRecentFilePath df_Combined = df_Combined.append(df,ignore_index=True) df_Combined1 = df_Combined[:800000] df_Combined2 = df_Combined[800000:1600000] df_Combined3 = df_Combined[1600000:2400000] df_Combined4 = df_Combined[2400000:3200000] df_Combined5 = df_Combined[3200000:4000000] df_Combined6 = df_Combined[4000000:]
0 -
One possible solution is to ensure that each chunk you upload is consecutive in terms of the data. It seems like you're attempting to split the data into chunks of 800,000 rows each, but it's possible that the rows aren't consecutive across these chunks.
Perhaps something like this:Import necessary libraries import pandas as pd
import glob
import os Function to read and append files def func1():
filePath = 'filepath'
numOfFiles = len(sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True))
print(numOfFiles)
df_chunks = [] # Read and append files into chunks
for i in range(0, numOfFiles):
mostRecentFilePath = sorted(glob.iglob(filePath), key=os.path.getctime, reverse=True)[i]
print(str(i), ': ', mostRecentFilePath)
df = pd.read_excel(mostRecentFilePath)
df['File Name'] = mostRecentFilePath
df_chunks.append(df)
# Upload each chunk separately
for idx, df_chunk in enumerate(df_chunks):
# Split the chunk into smaller parts if necessary
for i in range(0, len(df_chunk), 800000):
df_part = df_chunk[i:i+800000]
# Upload df_part to Domo here
# For example:
# domo.upload_dataframe(df_part, dataset_name=f'dataset_part_{idx}_{i}')** Was this post helpful? Click Agree or Like below. **
** Did this solve your problem? Accept it as a solution! **0
Categories
- All Categories
- 1.7K Product Ideas
- 1.7K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 295 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.8K Transform
- 97 SQL DataFlows
- 608 Datasets
- 2.1K Magic ETL
- 3.8K Visualize
- 2.4K Charting
- 709 Beast Mode
- 49 App Studio
- 39 Variables
- 667 Automate
- 170 Apps
- 446 APIs & Domo Developer
- 44 Workflows
- 7 DomoAI
- 33 Predict
- 13 Jupyter Workspaces
- 20 R & Python Tiles
- 391 Distribute
- 111 Domo Everywhere
- 274 Scheduled Reports
- 6 Software Integrations
- 115 Manage
- 112 Governance & Security
- Domo Community Gallery
- 31 Product Releases
- 9 Domo University
- 5.3K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 103 Community Announcements
- 4.8K Archive