Does anyone know the limitations of dataset size and performance impact?
I have been tasked with creating a daily snapshot of our quickstart datasets. I have reached out to our consultants and domo support, they basically suggest that we consider making smaller snapshots. Coming from a MsSQL environment, there have been situations in which day to day transactions are need to track year to year changes as of specific dates. On top of that i can build a weekly/monthly snapshot if i have the daily. Which brings me to my question, does or has anyone ran into a size limitation? If so what was the size of the dataset in which perfomance or reports are impacted?
Comments
-
Standard dataflows are based off of MySql. That performs reasonably well up to about 5 million rows. After that size, we can switch to Redshift SQL (internal to Domo consultants) which handles larger datasets better. I've seen redshift dataflows handle up to 50 million rows reasonably well. MagicETL is built on a separate engine that is also optimized to handle very large datasets. If performance starts to be impacted, we have an ETL team here that could give some advice.
How many rows of data a day in the snapshot are you anticipating?
I work for Domo.
**Say "Thanks" by clicking the thumbs up in the post that helped you.
**Please mark the post that solves your problem as "Accepted Solution"3
Categories
- All Categories
- 1.7K Product Ideas
- 1.7K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 295 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.8K Transform
- 97 SQL DataFlows
- 608 Datasets
- 2.1K Magic ETL
- 3.8K Visualize
- 2.4K Charting
- 710 Beast Mode
- 50 App Studio
- 39 Variables
- 668 Automate
- 170 Apps
- 446 APIs & Domo Developer
- 45 Workflows
- 7 DomoAI
- 33 Predict
- 13 Jupyter Workspaces
- 20 R & Python Tiles
- 391 Distribute
- 111 Domo Everywhere
- 274 Scheduled Reports
- 6 Software Integrations
- 116 Manage
- 113 Governance & Security
- Domo Community Gallery
- 31 Product Releases
- 9 Domo University
- 5.3K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 103 Community Announcements
- 4.8K Archive