Does anyone know the limitations of dataset size and performance impact?
I have been tasked with creating a daily snapshot of our quickstart datasets. I have reached out to our consultants and domo support, they basically suggest that we consider making smaller snapshots. Coming from a MsSQL environment, there have been situations in which day to day transactions are need to track year to year changes as of specific dates. On top of that i can build a weekly/monthly snapshot if i have the daily. Which brings me to my question, does or has anyone ran into a size limitation? If so what was the size of the dataset in which perfomance or reports are impacted?
Comments
-
Standard dataflows are based off of MySql. That performs reasonably well up to about 5 million rows. After that size, we can switch to Redshift SQL (internal to Domo consultants) which handles larger datasets better. I've seen redshift dataflows handle up to 50 million rows reasonably well. MagicETL is built on a separate engine that is also optimized to handle very large datasets. If performance starts to be impacted, we have an ETL team here that could give some advice.
How many rows of data a day in the snapshot are you anticipating?
I work for Domo.
**Say "Thanks" by clicking the thumbs up in the post that helped you.
**Please mark the post that solves your problem as "Accepted Solution"3
Categories
- All Categories
- 1.4K Product Ideas
- 1.4K Ideas Exchange
- 1.4K Connect
- 1.1K Connectors
- 278 Workbench
- 4 Cloud Amplifier
- 4 Federated
- 2.7K Transform
- 89 SQL DataFlows
- 557 Datasets
- 2K Magic ETL
- 3.3K Visualize
- 2.3K Charting
- 571 Beast Mode
- 11 App Studio
- 27 Variables
- 579 Automate
- 141 Apps
- 414 APIs & Domo Developer
- 23 Workflows
- 1 DomoAI
- 28 Predict
- 12 Jupyter Workspaces
- 16 R & Python Tiles
- 352 Distribute
- 92 Domo Everywhere
- 258 Scheduled Reports
- 2 Software Integrations
- 92 Manage
- 89 Governance & Security
- 9 Product Release Questions
- Community Forums
- 42 Getting Started
- 28 Community Member Introductions
- 86 Community Announcements
- 4.8K Archive