Option to manually run Magic ETL v2 without optimization

MichelleH

I love that ETL v2 allows automatic query optimization because it significantly cuts down the run time of our ETL pipeline. However I have run into some issues with some rows not being processed when rows get appended to the input dataset while the dataflow is running. The next time I run the dataflow, the ETL optimization thinks that those rows have already been processed because they were added to the input before the dataflow was run last (based on dataflow end time). As a result, I have to temporarily edit the dataflow and opt out of optimization so that I can manually rerun the dataflow with all rows, not just the ones recently added.

It would be helpful to have an option to manually run the dataflow without the optimization so that it's forced to process every row, but continue to optimize the regularly scheduled runs.