Run Recursive Dataflow only when new data comes

Options
AshCD
AshCD Member
in Magic ETL

I am trying to create a recursive dataflow which should run only when new data comes. Here are the conditions for that,

  1. We are suppose to get weekly data in our CSV connector every Monday. We have set out recursive dataflow to run on Wednesday.
  2. We use week number as our constraint column.
  3. But sometimes, we get data late on Thursday or not even get data at all in that particular week.
  4. Sometimes they send wrong data on Monday. Hence we run our recursive flow on Wednesday to provide them one more day extra (i.e. Tuesday) to send correct data. (Hence we have not used the trigger - Run ETL only when dataset updates)

So, is there a way to trigger recursive flow while maintaining all above conditions? For now we are running ETL manually but we want to automate the process.

Tagged:

Best Answers

  • ArborRose
    ArborRose Coach
    Answer ✓
    Options

    On the recursive ETL, you can set the schedule that triggers. The timestamp formula you use will dictate what gets saved. I use a formula that runs on Monday on one my recursions, that gathers from the previous Sunday through the previous Saturday.

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

  • DataMaven
    DataMaven Coach
    Answer ✓
    Options

    If your recursive is set up properly, #4 should not be a factor. Then, you can simply have it run when the dataset updates. If the wrong data is sent, it should self-correct when the correct data comes through. If there aspect of the data or setup that make this not the case, please provide details.

    If there is an issue with end-users seeing the unvalidated data, you can set up a webform with the week information and mark the weeks as validated and join to your recursive (and set as a trigger). This way, you can filter the final output dataset or dashboards to omit unvalidated results and buy the time between original submission and validation. This will also enable you to have dashboards set up specifically for validation without confusing your end-users!

    DataMaven
    Breaking Down Silos - Building Bridges
    **Say "Thanks" by clicking a reaction in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"

Answers

  • ArborRose
    ArborRose Coach
    Answer ✓
    Options

    On the recursive ETL, you can set the schedule that triggers. The timestamp formula you use will dictate what gets saved. I use a formula that runs on Monday on one my recursions, that gathers from the previous Sunday through the previous Saturday.

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

  • DataMaven
    DataMaven Coach
    Answer ✓
    Options

    If your recursive is set up properly, #4 should not be a factor. Then, you can simply have it run when the dataset updates. If the wrong data is sent, it should self-correct when the correct data comes through. If there aspect of the data or setup that make this not the case, please provide details.

    If there is an issue with end-users seeing the unvalidated data, you can set up a webform with the week information and mark the weeks as validated and join to your recursive (and set as a trigger). This way, you can filter the final output dataset or dashboards to omit unvalidated results and buy the time between original submission and validation. This will also enable you to have dashboards set up specifically for validation without confusing your end-users!

    DataMaven
    Breaking Down Silos - Building Bridges
    **Say "Thanks" by clicking a reaction in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"

Categories