Option to manually run Magic ETL v2 without optimization


I love that ETL v2 allows automatic query optimization because it significantly cuts down the run time of our ETL pipeline. However I have run into some issues with some rows not being processed when rows get appended to the input dataset while the dataflow is running. The next time I run the dataflow, the ETL optimization thinks that those rows have already been processed because they were added to the input before the dataflow was run last (based on dataflow end time). As a result, I have to temporarily edit the dataflow and opt out of optimization so that I can manually rerun the dataflow with all rows, not just the ones recently added.
It would be helpful to have an option to manually run the dataflow without the optimization so that it's forced to process every row, but continue to optimize the regularly scheduled runs.
Categories
- 10.6K All Categories
- 8 Connect
- 918 Connectors
- 250 Workbench
- 477 Transform
- 1.8K Magic ETL
- 69 SQL DataFlows
- 478 Datasets
- 218 Visualize
- 260 Beast Mode
- 2.1K Charting
- 12 Variables
- 19 Automate
- 356 APIs & Domo Developer
- 89 Apps
- 3 Workflows
- 20 Predict
- 5 Jupyter Workspaces
- 15 R & Python Tiles
- 249 Distribute
- 65 Domo Everywhere
- 243 Scheduled Reports
- 21 Manage
- 42 Governance & Security
- 191 Product Ideas
- 1.2K Ideas Exchange
- 11 Community Forums
- 27 Getting Started
- 14 Community Member Introductions
- 55 Community News
- 4.5K Archive