Option to manually run Magic ETL v2 without optimization


I love that ETL v2 allows automatic query optimization because it significantly cuts down the run time of our ETL pipeline. However I have run into some issues with some rows not being processed when rows get appended to the input dataset while the dataflow is running. The next time I run the dataflow, the ETL optimization thinks that those rows have already been processed because they were added to the input before the dataflow was run last (based on dataflow end time). As a result, I have to temporarily edit the dataflow and opt out of optimization so that I can manually rerun the dataflow with all rows, not just the ones recently added.
It would be helpful to have an option to manually run the dataflow without the optimization so that it's forced to process every row, but continue to optimize the regularly scheduled runs.
Categories
- 7.7K All Categories
- 12 Connect
- 925 Connectors
- 247 Workbench
- 441 Transform
- 1.7K Magic ETL
- 61 SQL DataFlows
- 456 Datasets
- 88 Visualize
- 222 Beast Mode
- 2.1K Charting
- 8 Variables
- 5 Automate
- 349 APIs & Domo Developer
- 85 Apps
- Workflows
- 17 Predict
- 3 Jupyter Workspaces
- 14 R & Python Tiles
- 242 Distribute
- 60 Domo Everywhere
- 241 Scheduled Reports
- 18 Manage
- 39 Governance & Security
- 46 Product Ideas
- 1.1K Ideas Exchange
- 6 Community Forums
- 19 Getting Started
- 6 Community Member Introductions
- 50 Community News
- 18 Event Recordings
- 577 日本支部