Option to manually run Magic ETL v2 without optimization


I love that ETL v2 allows automatic query optimization because it significantly cuts down the run time of our ETL pipeline. However I have run into some issues with some rows not being processed when rows get appended to the input dataset while the dataflow is running. The next time I run the dataflow, the ETL optimization thinks that those rows have already been processed because they were added to the input before the dataflow was run last (based on dataflow end time). As a result, I have to temporarily edit the dataflow and opt out of optimization so that I can manually rerun the dataflow with all rows, not just the ones recently added.
It would be helpful to have an option to manually run the dataflow without the optimization so that it's forced to process every row, but continue to optimize the regularly scheduled runs.
Categories
- All Categories
- 1.9K Product Ideas
- 1.9K Ideas Exchange
- 1.6K Connect
- 1.3K Connectors
- 305 Workbench
- 6 Cloud Amplifier
- 9 Federated
- 3K Transform
- 106 SQL DataFlows
- 646 Datasets
- 2.2K Magic ETL
- 4K Visualize
- 2.5K Charting
- 774 Beast Mode
- 75 App Studio
- 43 Variables
- 728 Automate
- 186 Apps
- 467 APIs & Domo Developer
- 61 Workflows
- 14 DomoAI
- 40 Predict
- 17 Jupyter Workspaces
- 23 R & Python Tiles
- 403 Distribute
- 117 Domo Everywhere
- 277 Scheduled Reports
- 9 Software Integrations
- 136 Manage
- 133 Governance & Security
- 8 Domo Community Gallery
- 44 Product Releases
- 12 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 112 Community Announcements
- 4.8K Archive