Magic ETL - Using Group by - run time impact question
Wondering what is the best practice to reduce overall run time of a ETL dataflow...
I've got 5 input datasets totalling about 14M rows of data that are ultimately appended together in the final result.
Is it better to use group by in the magic ETL for each dataset (5 steps) or one consolidated step after the appends?
Also, are there certain items in Magic ETL that are run time bandits?
Comments
-
Is anyone able to help out with this request?
0 -
Hello @swagner,
It is hard to determine what might be causing long run times in an ETL without looking at the ETL in specific.
Generally, it is most efficient to use a select columns tile and select only the columns you need.
Next would be to filter your data down to only the rows you need.
After filtering your data to only the columns and rows that you need grouping your data will help reduce the size.
In regards to your group by question. Normally it will not make a difference between doing the group by's before the append vs after. If you can provide a screenshot of your ETL I can look to see if there are any other steps that we might be able to optimize.**Say “Thanks" by clicking the thumbs up in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"1
Categories
- All Categories
- 1.5K Product Ideas
- 1.5K Ideas Exchange
- 1.4K Connect
- 1.1K Connectors
- 283 Workbench
- 4 Cloud Amplifier
- 4 Federated
- 2.7K Transform
- 90 SQL DataFlows
- 568 Datasets
- 2K Magic ETL
- 3.4K Visualize
- 2.4K Charting
- 600 Beast Mode
- 14 App Studio
- 29 Variables
- 599 Automate
- 151 Apps
- 417 APIs & Domo Developer
- 29 Workflows
- 2 DomoAI
- 28 Predict
- 12 Jupyter Workspaces
- 16 R & Python Tiles
- 362 Distribute
- 99 Domo Everywhere
- 261 Scheduled Reports
- 2 Software Integrations
- 98 Manage
- 95 Governance & Security
- 15 Product Releases
- Community Forums
- 37 Getting Started
- 28 Community Member Introductions
- 90 Community Announcements
- 4.8K Archive