Does anyone know the limitations of dataset size and performance impact?

CoreVest

I have been tasked with creating a daily snapshot of our quickstart datasets. I have reached out to our consultants and domo support, they basically suggest that we consider making smaller snapshots. Coming from a MsSQL environment, there have been situations in which day to day transactions are need to track year to year changes as of specific dates. On top of that i can build a weekly/monthly snapshot if i have the daily. Which brings me to my question, does or has anyone ran into a size limitation? If so what was the size of the dataset in which perfomance or reports are impacted?

jaredpk

Standard dataflows are based off of MySql.  That performs reasonably well up to about 5 million rows.  After that size, we can switch to Redshift SQL (internal to Domo consultants) which handles larger datasets better.  I've seen redshift dataflows handle up to 50 million rows reasonably well.   MagicETL is built on a separate engine that is also optimized to handle very large datasets.  If performance starts to be impacted, we have an ETL team here that could give some advice.

 

How many rows of data a day in the snapshot are you anticipating?

kshah008

@CoreVest, can you answer jaredpk's follow-up question to better assist you?