MySQL Recursion Performance/Speed Issues
Hello! I am looking for suggestions on how to improve the performance of my mysql dataflow.
Currently I have a recursive dataflow built with a Google Analytics connector, and I'm using a concatenated key created by joining multiple columns together to delete and upsert new data from the historical dataset.
I'm using the following syntax in a SQL transform function:
DELETE from historical_data WHERE concat_key IN (SELECT concat_key FROM new_data);
The recursion works but it takes really long to run (50+ minutes for 82K rows of total input). I'm currently manually running it and the historical dataset has about 44K and new data has about 38K. Has anyone had experience working with mySQL recursions and can suggest a different way to create a recursive dataflow that increases the speed?
Best Answer
-
You may want to review this KB article regarding a recursive MySQL dataflow. You can avoid having to execute a delete statement, which is likely causing your timing issues.
https://domohelp.domo.com/hc/en-us/articles/360043428133-Creating-a-Recursive-Snapshot-SQL-DataFlow
**Check out my Domo Tips & Tricks Videos
**Make sure to any users posts that helped you.
**Please mark as accepted the ones who solved your issue.0
Answers
-
You may want to review this KB article regarding a recursive MySQL dataflow. You can avoid having to execute a delete statement, which is likely causing your timing issues.
https://domohelp.domo.com/hc/en-us/articles/360043428133-Creating-a-Recursive-Snapshot-SQL-DataFlow
**Check out my Domo Tips & Tricks Videos
**Make sure to any users posts that helped you.
**Please mark as accepted the ones who solved your issue.0 -
Thank you @MarkSnodgrass !
I thought you would find it amusing to know that this helped reduce the processing time for my dataflow from 1h 17m to 53s.
1 -
Awesome @etszyingho !!!
**Check out my Domo Tips & Tricks Videos
**Make sure to any users posts that helped you.
**Please mark as accepted the ones who solved your issue.0 -
@etszyingho if you can mark the answer as accepted, that will help others in the community. Great to see the massive improvement!
**Check out my Domo Tips & Tricks Videos
**Make sure to any users posts that helped you.
**Please mark as accepted the ones who solved your issue.0 -
Hi @etszyingho
I'd recommend utilizing ETL 2.0 if at all possible as it will be more performant than MySQL dataflows but it looks like you're at a good spot with your current performance with @MarkSnodgrass 's solution.
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**1
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.6K Connect
- 1.2K Connectors
- 300 Workbench
- 6 Cloud Amplifier
- 9 Federated
- 2.9K Transform
- 102 SQL DataFlows
- 626 Datasets
- 2.2K Magic ETL
- 3.9K Visualize
- 2.5K Charting
- 755 Beast Mode
- 61 App Studio
- 41 Variables
- 693 Automate
- 178 Apps
- 456 APIs & Domo Developer
- 49 Workflows
- 10 DomoAI
- 38 Predict
- 16 Jupyter Workspaces
- 22 R & Python Tiles
- 398 Distribute
- 115 Domo Everywhere
- 276 Scheduled Reports
- 7 Software Integrations
- 130 Manage
- 127 Governance & Security
- 8 Domo Community Gallery
- 38 Product Releases
- 11 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 110 Community Announcements
- 4.8K Archive