Adding new Historical Data to an Already Existing Recursive Dataflow
I am looking for a way to add historical data into my already existing recursive dataflow. My two input datasets are 1.) an email connector that uploads once a day, 2.) a copy of my output dataset as an input dataset acting as my historical records. My output dataset has data from 2021-2023 but I would like to add data from 2020. Is there a way for me to manually upload an excel file with the 2020 data into my recursive dataflow so that it will be stored in my historical dataset. I have tried to append the 2020 data to the recursive dataflow and it seems to work but when i go back in to delete the 2020 input it is not being stored in my historical records.
Best Answer
-
@TC1199A and I met and determined that the data was likely being removed from a filter tile in the ETL. He will rebuild and if that wasn't the case report back here. While walking through the ETL I shared some things that Taylor and Davis found helpful and thought I would share them here for those that don't know:
- When using snapshots/recursive dataflows there is a chance for data loss. It is always a good idea to export data before making changes that could potentially be destructive so that you have a backup.
- You can revert to previous versions of a dataflow in the history tab as shown below:
If I solved your problem, please select "yes" above
1
Answers
-
Once it is a part of the output dataset input (recursive) it shouldn't disappear unless you have some filtering in the ETL that would remove it. Could you provide a bit more details and maybe some screenshots or a screen recording showing the ETL?
If I solved your problem, please select "yes" above
0 -
Hey @pauljames, Mainly to clean up the dataflow.
0 -
@ColemenWilson is there a specific place within my recursive Dataflow I should be appending the 2020 data? currently I am adding the excel file with the 2020 data to the append tile in my recursive dataflow just before the final output.
0 -
@TC1199A Happy to take a look at it with you if you'd like to schedule some time to walk through it:
If I solved your problem, please select "yes" above
0 -
@TC1199A and I met and determined that the data was likely being removed from a filter tile in the ETL. He will rebuild and if that wasn't the case report back here. While walking through the ETL I shared some things that Taylor and Davis found helpful and thought I would share them here for those that don't know:
- When using snapshots/recursive dataflows there is a chance for data loss. It is always a good idea to export data before making changes that could potentially be destructive so that you have a backup.
- You can revert to previous versions of a dataflow in the history tab as shown below:
If I solved your problem, please select "yes" above
1
Categories
- All Categories
- 1.5K Product Ideas
- 1.5K Ideas Exchange
- 1.4K Connect
- 1.1K Connectors
- 280 Workbench
- 4 Cloud Amplifier
- 4 Federated
- 2.7K Transform
- 90 SQL DataFlows
- 563 Datasets
- 2K Magic ETL
- 3.4K Visualize
- 2.3K Charting
- 592 Beast Mode
- 13 App Studio
- 28 Variables
- 588 Automate
- 143 Apps
- 417 APIs & Domo Developer
- 27 Workflows
- 1 DomoAI
- 28 Predict
- 12 Jupyter Workspaces
- 16 R & Python Tiles
- 361 Distribute
- 99 Domo Everywhere
- 260 Scheduled Reports
- 2 Software Integrations
- 94 Manage
- 91 Governance & Security
- 9 Product Releases
- Community Forums
- 37 Getting Started
- 28 Community Member Introductions
- 90 Community Announcements
- 4.8K Archive