Reduce loading time of input dataset in Magic ETL
I have a Magic ETL dataflow using an input dataset with 10M rows. It takes around 20 mins just to prepare this input dataset, and consumes most of the ETL processing time.
This ETL is used by a custom app, so ideally we would like it to finish as fast as possible. Currently it is running way too long and the bottleneck seems to be the loading of the input dataset.
Wonder if there is a better way of doing ETL with such huge input data? Or is there any way to improve its performance?
Comments
-
If you aren't doing much in your Magic ETL, you might consider DataFusion, also known as Blend. It is much better equipped to handle datasets that are in the millions of rows.
https://knowledge.domo.com/Prepare/Magic_Transforms/DataFusion
**Check out my Domo Tips & Tricks Videos
**Make sure to any users posts that helped you.
**Please mark as accepted the ones who solved your issue.2 -
Thanks @MarkSnodgrass for the reply. However, we need to do some complex calculations and the ETL needs to be triggered by a custom app, so Blend is not a feasible solution.
0 -
Hiya - I just ran a small test in magic ETL using an input of 10.5M records and simply output the result to another dataset. The whole process took < 4 mins. You might want to try to isolate your load process from the actual transformation process to see if the load process is indeed the bottleneck. if it is, then it is worth contacting DOMO helpdesk to check the configuration of your workspace.
0
Categories
- All Categories
- 1.7K Product Ideas
- 1.7K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 295 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.8K Transform
- 97 SQL DataFlows
- 608 Datasets
- 2.1K Magic ETL
- 3.8K Visualize
- 2.4K Charting
- 709 Beast Mode
- 49 App Studio
- 39 Variables
- 667 Automate
- 170 Apps
- 446 APIs & Domo Developer
- 44 Workflows
- 7 DomoAI
- 33 Predict
- 13 Jupyter Workspaces
- 20 R & Python Tiles
- 391 Distribute
- 111 Domo Everywhere
- 274 Scheduled Reports
- 6 Software Integrations
- 115 Manage
- 112 Governance & Security
- Domo Community Gallery
- 31 Product Releases
- 9 Domo University
- 5.3K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 103 Community Announcements
- 4.8K Archive