Many rows
I need to find a way to pull in many rows of data. I have two months worth of data which is essentially 110M rows. I need to pull in at least a year. Is there a way to do this? I tried the ETL partitioning method but it said 'too many rows of data'. Would workbench work with partitioning here or would I get the same error? Any suggestions?
Best Answer
Answers
-
@mroker Where is your data currently stored? Depending on whether it's a spreadsheet, ODBC, cloud, etc. there are different connection options available. If importing all the rows at once is an issue, then you may need to upload in batches and appending the batches onto each other.
0 -
ODBC. So, I can append, was hoping to partition. I just wanted to see what the limitations are like on the ETL/Dataset side of things. At the end of the day, having a year's worth of data would be like 700M rows which to me is a massive dataset. Also, wanted to make sure partition would even work.
0
Categories
- All Categories
- 2K Product Ideas
- 2K Ideas Exchange
- 1.6K Connect
- 1.3K Connectors
- 311 Workbench
- 6 Cloud Amplifier
- 9 Federated
- 3.8K Transform
- 656 Datasets
- 115 SQL DataFlows
- 2.2K Magic ETL
- 813 Beast Mode
- 3.3K Visualize
- 2.5K Charting
- 81 App Studio
- 45 Variables
- 771 Automate
- 190 Apps
- 481 APIs & Domo Developer
- 77 Workflows
- 23 Code Engine
- 36 AI and Machine Learning
- 19 AI Chat
- AI Playground
- AI Projects and Models
- 17 Jupyter Workspaces
- 410 Distribute
- 120 Domo Everywhere
- 280 Scheduled Reports
- 10 Software Integrations
- 142 Manage
- 138 Governance & Security
- 8 Domo Community Gallery
- 48 Product Releases
- 12 Domo University
- 5.4K Community Forums
- 41 Getting Started
- 31 Community Member Introductions
- 114 Community Announcements
- 4.8K Archive