Adding new Historical Data to an Already Existing Recursive Dataflow

TC1199A
TC1199A Member
in Magic ETL

I am looking for a way to add historical data into my already existing recursive dataflow. My two input datasets are 1.) an email connector that uploads once a day, 2.) a copy of my output dataset as an input dataset acting as my historical records. My output dataset has data from 2021-2023 but I would like to add data from 2020. Is there a way for me to manually upload an excel file with the 2020 data into my recursive dataflow so that it will be stored in my historical dataset. I have tried to append the 2020 data to the recursive dataflow and it seems to work but when i go back in to delete the 2020 input it is not being stored in my historical records.

Tagged:

Best Answer

  • ColemenWilson
    ColemenWilson Coach
    Answer ✓

    @TC1199A and I met and determined that the data was likely being removed from a filter tile in the ETL. He will rebuild and if that wasn't the case report back here. While walking through the ETL I shared some things that Taylor and Davis found helpful and thought I would share them here for those that don't know:

    1. When using snapshots/recursive dataflows there is a chance for data loss. It is always a good idea to export data before making changes that could potentially be destructive so that you have a backup.
    2. You can revert to previous versions of a dataflow in the history tab as shown below:
    Screen Shot 2023-11-10 at 1.33.43 PM.png

    If I solved your problem, please select "yes" above

Answers

  • ColemenWilson
    ColemenWilson Coach
    edited November 2023

    Once it is a part of the output dataset input (recursive) it shouldn't disappear unless you have some filtering in the ETL that would remove it. Could you provide a bit more details and maybe some screenshots or a screen recording showing the ETL?

    If I solved your problem, please select "yes" above

  • pauljames
    pauljames Contributor

    Hi @TC1199A! Any specific reason why you want to delete the 2020 input dataset after inclusion?

    IF I SOLVED YOUR PROBLEM, PLEASE "ACCEPT" MY ANSWER AS A SOLUTION. THANK YOU!

  • TC1199A
    TC1199A Member

    Hey @pauljames, Mainly to clean up the dataflow.

  • TC1199A
    TC1199A Member

    @ColemenWilson is there a specific place within my recursive Dataflow I should be appending the 2020 data? currently I am adding the excel file with the 2020 data to the append tile in my recursive dataflow just before the final output.

  • ColemenWilson
    ColemenWilson Coach

    @TC1199A Happy to take a look at it with you if you'd like to schedule some time to walk through it: https://calendly.com/cwilsondomo/15min

    If I solved your problem, please select "yes" above

  • pauljames
    pauljames Contributor

    @TC1199A , are you applying the recursive key/identifier to this 2020 data? You might need to append before the key so it can be saved.

    IF I SOLVED YOUR PROBLEM, PLEASE "ACCEPT" MY ANSWER AS A SOLUTION. THANK YOU!

  • ColemenWilson
    ColemenWilson Coach
    Answer ✓

    @TC1199A and I met and determined that the data was likely being removed from a filter tile in the ETL. He will rebuild and if that wasn't the case report back here. While walking through the ETL I shared some things that Taylor and Davis found helpful and thought I would share them here for those that don't know:

    1. When using snapshots/recursive dataflows there is a chance for data loss. It is always a good idea to export data before making changes that could potentially be destructive so that you have a backup.
    2. You can revert to previous versions of a dataflow in the history tab as shown below:
    Screen Shot 2023-11-10 at 1.33.43 PM.png

    If I solved your problem, please select "yes" above

Categories