Magic ETL Dataflow Schedule: add trigger for Dataset has not updated

The Dataflow schedule has a setting to trigger when input datasets have updated. It would be helpful if there was a way to schedule it run at a certain time if an input has not updated. There are occasionally issues with Workbench that cause input datasets to not run, but in some cases we have recursive dataflows downstream that still need to run to capture a daily snapshot. With this option, it would at least capture the most recent data again.

I've addressed this by setting critical dataflows to run both on input dataset update and at a scheduled time shortly after, but this means it often runs twice unnecessarily

Tagged:
1
1 votes