Adding new Historical Data to an Already Existing Recursive Dataflow

TC1199A

I am looking for a way to add historical data into my already existing recursive dataflow. My two input datasets are 1.) an email connector that uploads once a day, 2.) a copy of my output dataset as an input dataset acting as my historical records. My output dataset has data from 2021-2023 but I would like to add data from 2020. Is there a way for me to manually upload an excel file with the 2020 data into my recursive dataflow so that it will be stored in my historical dataset. I have tried to append the 2020 data to the recursive dataflow and it seems to work but when i go back in to delete the 2020 input it is not being stored in my historical records.

ColemenWilson

@TC1199A and I met and determined that the data was likely being removed from a filter tile in the ETL. He will rebuild and if that wasn't the case report back here. While walking through the ETL I shared some things that Taylor and Davis found helpful and thought I would share them here for those that don't know:

1. When using snapshots/recursive dataflows there is a chance for data loss. It is always a good idea to export data before making changes that could potentially be destructive so that you have a backup.
2. You can revert to previous versions of a dataflow in the history tab as shown below:

ColemenWilson

Once it is a part of the output dataset input (recursive) it shouldn't disappear unless you have some filtering in the ETL that would remove it. Could you provide a bit more details and maybe some screenshots or a screen recording showing the ETL?

pauljames

Hi @TC1199A! Any specific reason why you want to delete the 2020 input dataset after inclusion?

TC1199A

Hey @pauljames, Mainly to clean up the dataflow.

TC1199A

@ColemenWilson is there a specific place within my recursive Dataflow I should be appending the 2020 data? currently I am adding the excel file with the 2020 data to the append tile in my recursive dataflow just before the final output.

ColemenWilson

@TC1199A Happy to take a look at it with you if you'd like to schedule some time to walk through it: 15 Minute Meeting - Colemen Wilson

pauljames

@TC1199A , are you applying the recursive key/identifier to this 2020 data? You might need to append before the key so it can be saved.

ColemenWilson

@TC1199A and I met and determined that the data was likely being removed from a filter tile in the ETL. He will rebuild and if that wasn't the case report back here. While walking through the ETL I shared some things that Taylor and Davis found helpful and thought I would share them here for those that don't know:

1. When using snapshots/recursive dataflows there is a chance for data loss. It is always a good idea to export data before making changes that could potentially be destructive so that you have a backup.
2. You can revert to previous versions of a dataflow in the history tab as shown below: