Run Recursive Dataflow only when new data comes

AshCD · 2024-02-23T05:57:03+00:00

There was an error rendering this rich post.

I am trying to create a recursive dataflow which should run only when new data comes. Here are the conditions for that,

We are suppose to get weekly data in our CSV connector every Monday. We have set out recursive dataflow to run on Wednesday.
We use week number as our constraint column.
But sometimes, we get data late on Thursday or not even get data at all in that particular week.
Sometimes they send wrong data on Monday. Hence we run our recursive flow on Wednesday to provide them one more day extra (i.e. Tuesday) to send correct data. (Hence we have not used the trigger - Run ETL only when dataset updates)

So, is there a way to trigger recursive flow while maintaining all above conditions? For now we are running ETL manually but we want to automate the process.

Find more posts tagged with

Magic ETL

Accepted answers

ArborRose

On the recursive ETL, you can set the schedule that triggers. The timestamp formula you use will dictate what gets saved. I use a formula that runs on Monday on one my recursions, that gathers from the previous Sunday through the previous Saturday.

DataMaven

If your recursive is set up properly, #4 should not be a factor. Then, you can simply have it run when the dataset updates. If the wrong data is sent, it should self-correct when the correct data comes through. If there aspect of the data or setup that make this not the case, please provide details.

If there is an issue with end-users seeing the unvalidated data, you can set up a webform with the week information and mark the weeks as validated and join to your recursive (and set as a trigger). This way, you can filter the final output dataset or dashboards to omit unvalidated results and buy the time between original submission and validation. This will also enable you to have dashboards set up specifically for validation without confusing your end-users!

All comments

ArborRose

DataMaven