MySQL Recursion Performance/Speed Issues

etszyingho Member
edited March 2023 in SQL DataFlows

Hello! I am looking for suggestions on how to improve the performance of my mysql dataflow.

Currently I have a recursive dataflow built with a Google Analytics connector, and I'm using a concatenated key created by joining multiple columns together to delete and upsert new data from the historical dataset.

I'm using the following syntax in a SQL transform function:

DELETE from historical_data WHERE concat_key IN (SELECT concat_key FROM new_data);

The recursion works but it takes really long to run (50+ minutes for 82K rows of total input). I'm currently manually running it and the historical dataset has about 44K and new data has about 38K. Has anyone had experience working with mySQL recursions and can suggest a different way to create a recursive dataflow that increases the speed?


Best Answer