Is PyDomo appropriate for my data update work?
Hello. I've been reviewing different methods of updating data in Domo. I'm currently using Workbench, but am considering creating a completely hands-off setup using PyDomo. So, here's my scenario:
We have a number of datasets that originate from ServiceNow. I put those through some local processing using Python scripts, which generate CSV files. Those files are used by Workbench to update the Domo datasets. Good so far, although Workbench seems to have some minor quirks. But this raises a reasonable question: As long as all of this work is already being done in Python, why not just push the changes directly from there?
The only thing holding me back is this statement on the README for the SDK:
Use DataSets for fairly static data sources that only require occasional updates via data replacement
We have 7 datasets that may have, depending on the set, anywhere from 20 to 2,500 updates or additions out to datasets containing up to 175,000 rows. I wouldn't call these either or "occasional" or only "replacement", but perhaps I'm not thinking at the same scale as the Domo engineers. So how about it... Is PyDomo a viable way to make changes like this?
Best Answer
-
Yes, I use pydomo for a lot of scripts and it works well. There are two update methods, data sets and streams. You can use streams to upload data in segments and finally commit the stream when it's done which are better for larger datasets.
Overall yes, I'd recommend pydomo in your use case as it'll help streamline your data pipeline
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**1
Answers
-
Yes, I use pydomo for a lot of scripts and it works well. There are two update methods, data sets and streams. You can use streams to upload data in segments and finally commit the stream when it's done which are better for larger datasets.
Overall yes, I'd recommend pydomo in your use case as it'll help streamline your data pipeline
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**1 -
Thank you. I appreciate the response, particularly from direct experience. I'll explore this further… I look forward to this being a completely hands-off process.
Out of curiosity, have you done anything like syncing data from within a Databrick or similar context?
0
Categories
- All Categories
- 1.7K Product Ideas
- 1.7K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 295 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.8K Transform
- 98 SQL DataFlows
- 608 Datasets
- 2.1K Magic ETL
- 3.8K Visualize
- 2.5K Charting
- 713 Beast Mode
- 50 App Studio
- 39 Variables
- 669 Automate
- 170 Apps
- 447 APIs & Domo Developer
- 45 Workflows
- 7 DomoAI
- 34 Predict
- 14 Jupyter Workspaces
- 20 R & Python Tiles
- 391 Distribute
- 111 Domo Everywhere
- 274 Scheduled Reports
- 6 Software Integrations
- 116 Manage
- 113 Governance & Security
- Domo Community Gallery
- 31 Product Releases
- 9 Domo University
- 5.3K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 104 Community Announcements
- 4.8K Archive