Large dataset import in Domo Workbench
Hi everyone, I have a very large dataset (18M rows) that I want to import and it takes too long. The query is already a grouped query so there is no unique ID. I am just trying to figure out if there is a way to import possibly a one time chunk (the data is for 2 years) and then maybe append the data after that. So maybe run a 2 year chunk, and then run a daily job that just gets the data that was there for a few days (maybe with an append). How can this be done without a unique ID?
Answers
-
Yes, there is a way to split up timed sections and append. You can import via workbench, or connect to your data with an api. Create smaller datasets such as data_yr_2021, data_yr_2022, data_qtr1, data_qtr2, etc. Then create a workflow and use the append tile to append the datasets. That way, you can pull previous years once or once per month, and more recent data daily. The workflow will give a full dataset result with all records for all times. I do this at my company to pull several years worth of data for analysis, but its only pulling the quarter to date on a daily basis.
** Was this post helpful? Click Agree or Like below. **
** Did this solve your problem? Accept it as a solution! **0 -
I’d recommend using the CLI tool to do a large dataset import as it will use the streams API and split your data into chunks and load your data faster
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**0
Categories
- All Categories
- 1.7K Product Ideas
- 1.7K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 295 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.8K Transform
- 97 SQL DataFlows
- 608 Datasets
- 2.1K Magic ETL
- 3.8K Visualize
- 2.4K Charting
- 710 Beast Mode
- 49 App Studio
- 39 Variables
- 667 Automate
- 170 Apps
- 446 APIs & Domo Developer
- 44 Workflows
- 7 DomoAI
- 33 Predict
- 13 Jupyter Workspaces
- 20 R & Python Tiles
- 391 Distribute
- 111 Domo Everywhere
- 274 Scheduled Reports
- 6 Software Integrations
- 115 Manage
- 112 Governance & Security
- Domo Community Gallery
- 31 Product Releases
- 9 Domo University
- 5.3K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 103 Community Announcements
- 4.8K Archive