I have a Magic ETL dataflow using an input dataset with 10M rows. It takes around 20 mins just to prepare this input dataset, and consumes most of the ETL processing time.
This ETL is used by a custom app, so ideally we would like it to finish as fast as possible. Currently it is running way too long and the bottleneck seems to be the loading of the input dataset.
Wonder if there is a better way of doing ETL with such huge input data? Or is there any way to improve its performance?