Redshift -- Significant Slow Downs Recently? (November 2020)

RACERX

Good day Domo folks,

Is anyone else noticing a significant slow-down with their Redshift transformations? My average run-time has doubled and in some cases, tripled or more. This started on-or-around 11/16/2020. I reached out to support but was advised "Domo has not control as it is AWS/Amazon".

 

The MagicETL feature is nice but really not a practical solution for our needs as it is too clunky for our flows. MySQL would be fine but the row limits tend to be prohibitive for most of our uses, so I'd rather stick with one flavor of SQL rather than jump back and forth between them for future maintenance purposes.

 

I'm hoping this is just an intermittent issue but I'm afraid this slowness might be the new normal. 

jaeW_at_Onyx

Redshift dataflows can take anywhere from 0 to 15 minutes before it actually starts running.  if you look at the details of the execution, you'll get a more accurate sense of the performance of your ETL once the data has been loaded into redshift.

 

Similarly, after you run the etl the data has to be transferred back to Domo and indexed into Vault (flat file storage) which is generally consistent, but grows linearly with row growth and scales with the cardinality of your columns (b/c the data does have to be indexed in Adrenaline -- database engine).

 

Magic 2.0 will have the most consistent performance, b/c as your support person indicated, Domo has full control over the Magic ETL engine.  You still have the same ingest / egress (time) cost when executing the ETL, but you don't have the up to 15 minute wait for redshift / mysql.  also, b/c Magic isn't built on a database engine, the transformation process can begin before the entire dataset is loaded into Magic (as opposed to redshift and mysql, where you have to wait until the entire dataset has been loaded into a table (and index in redshift's case) before querying can commence.

 

TLDR.  Learn to use Magic for the most consistent execution times.

RACERX

Thanks @jaeW_at_Onyx I appreciate the insight. My concern is more practical. Something worked well, was endorsed, was used to build hundreds of transformations, and ran flawlessly for quite some time. Now it does not for seemingly no real reason -- data size and schedules had no material change.

 

Using a traditional flavor of SQL is more accessible to anyone I hire, it is easier to document, I can store it externally, and it is much easier to troubleshoot and adjust. In my experience it is easy to look at 100 explicit well-formatted lines than it is to scour and juggle tons of equivalent UI buttons.

 

To me, switching to MagicETL is like asking a musician to give up their traditional instrument because a computer synthesizer can replace and do it for them.