Magic ETL - Using Group by - run time impact question
Wondering what is the best practice to reduce overall run time of a ETL dataflow...
I've got 5 input datasets totalling about 14M rows of data that are ultimately appended together in the final result.
Is it better to use group by in the magic ETL for each dataset (5 steps) or one consolidated step after the appends?
Also, are there certain items in Magic ETL that are run time bandits?
Comments
-
Is anyone able to help out with this request?
0 -
Hello @swagner,
It is hard to determine what might be causing long run times in an ETL without looking at the ETL in specific.
Generally, it is most efficient to use a select columns tile and select only the columns you need.
Next would be to filter your data down to only the rows you need.
After filtering your data to only the columns and rows that you need grouping your data will help reduce the size.
In regards to your group by question. Normally it will not make a difference between doing the group by's before the append vs after. If you can provide a screenshot of your ETL I can look to see if there are any other steps that we might be able to optimize.**Say “Thanks" by clicking the thumbs up in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"1
Categories
- All Categories
- 1.5K Product Ideas
- 1.5K Ideas Exchange
- 1.4K Connect
- 1.1K Connectors
- 278 Workbench
- 4 Cloud Amplifier
- 4 Federated
- 2.7K Transform
- 89 SQL DataFlows
- 560 Datasets
- 2K Magic ETL
- 3.3K Visualize
- 2.3K Charting
- 575 Beast Mode
- 13 App Studio
- 28 Variables
- 582 Automate
- 141 Apps
- 414 APIs & Domo Developer
- 26 Workflows
- 1 DomoAI
- 28 Predict
- 12 Jupyter Workspaces
- 16 R & Python Tiles
- 356 Distribute
- 95 Domo Everywhere
- 259 Scheduled Reports
- 2 Software Integrations
- 92 Manage
- 89 Governance & Security
- 9 Product Release Questions
- Community Forums
- 42 Getting Started
- 28 Community Member Introductions
- 89 Community Announcements
- 4.8K Archive