Redshift Performance Problem
I have call center data that I pull in through inContact and I need to add a date field from a date/time field in order to not have problems with time. To do this, I've made a redshift flow that selects all the fields and truncates the date/time field to a date field:
Select field1, field2, ...,
TRUNC(CONVERT_TIMEZONE('UTC','America/Chicago',"contactStart")) AS contactStartdate
from table
Since the data comes in as UTC, I first convert it to central and the truncate it. That way I get the correct date out of the stamp. The table has 52 columns and usually about 3000-4000 rows being replaced every 15min.
My problem is I need to update this dataset every 15min, but the redshift query sometime takes 30 seconds or over 10-15min. Any ideas on what is going on/how I can get it to consistently run for only 30 seconds?
Best Answer
-
I was curious about Redshift run times being inconsistent and posed a question on DOJO day. Here is the question and response:
https://dojo.domo.com/t5/Beast-Mode-ETL-Dataflow/Redshift-vs-MySQL-vs-ETL/m-p/38009
I tested one of my data flows that had the same requirements you have (select all columns, but convert the timestamps) using both Redshift and MySQL and I did see both more consistent run times and a lower 30 day average duration when using MySQL.
So you might want to make a copy of your data flow & convert it to MySQL to see if you can achieve the results you are looking for.
0
Answers
-
I was curious about Redshift run times being inconsistent and posed a question on DOJO day. Here is the question and response:
https://dojo.domo.com/t5/Beast-Mode-ETL-Dataflow/Redshift-vs-MySQL-vs-ETL/m-p/38009
I tested one of my data flows that had the same requirements you have (select all columns, but convert the timestamps) using both Redshift and MySQL and I did see both more consistent run times and a lower 30 day average duration when using MySQL.
So you might want to make a copy of your data flow & convert it to MySQL to see if you can achieve the results you are looking for.
0
Categories
- 10.6K All Categories
- 8 Connect
- 918 Connectors
- 250 Workbench
- 477 Transform
- 1.8K Magic ETL
- 69 SQL DataFlows
- 478 Datasets
- 218 Visualize
- 260 Beast Mode
- 2.1K Charting
- 12 Variables
- 19 Automate
- 356 APIs & Domo Developer
- 89 Apps
- 3 Workflows
- 20 Predict
- 5 Jupyter Workspaces
- 15 R & Python Tiles
- 249 Distribute
- 65 Domo Everywhere
- 243 Scheduled Reports
- 21 Manage
- 42 Governance & Security
- 191 Product Ideas
- 1.2K Ideas Exchange
- 11 Community Forums
- 27 Getting Started
- 14 Community Member Introductions
- 55 Community News
- 4.5K Archive