Is it possible with Magic ETL to append a dataflow
Best Answer
-
Yes you can append with Magic ETL. What I am not sure of from your post is if you want to append or if you want to update? If you are appending, sounds like you are creating a stacked Data source with date stamped versions of the same datasource? If so, bring the data source in, add a date field to identify it. for the out flow - select the Append function in the Edit columns section. I have a simple version that looks like the following:
The first two use the same data source and I have added a date to them so when appended I ge the stacked dataset one with the updated data.
Dojo Community Member
** Please like responses by clicking on the thumbs up
** Please Accept / check the answer that solved your problem / answered your question.1
Answers
-
Yes you can append with Magic ETL. What I am not sure of from your post is if you want to append or if you want to update? If you are appending, sounds like you are creating a stacked Data source with date stamped versions of the same datasource? If so, bring the data source in, add a date field to identify it. for the out flow - select the Append function in the Edit columns section. I have a simple version that looks like the following:
The first two use the same data source and I have added a date to them so when appended I ge the stacked dataset one with the updated data.
Dojo Community Member
** Please like responses by clicking on the thumbs up
** Please Accept / check the answer that solved your problem / answered your question.1 -
That's exactly what I was trying to do, thank you!
1 -
Hi,
I want to append a static dataset A and an live one B (that is updated every week). If i used this method, will i lose my first week data once i approach the second week? Thank you
Let's say we are on 1st week June 2017
A contains data Feb 2017 - May 2017
B contains data 1st week Jun 2017
Append will result Feb 2017 - 1st week Jun 2017
2nd week June 2017
A still contains data Feb 2017 - May 2017
B contains data 2nd week Jun 2017
Append --> will i lose my 1st week Jun 2017?
Thank you
1 -
@user05527 wrote:Hi,
I want to append a static dataset A and an live one B (that is updated every week). If i used this method, will i lose my first week data once i approach the second week? Thank you
Let's say we are on 1st week June 2017
A contains data Feb 2017 - May 2017
B contains data 1st week Jun 2017
Append will result Feb 2017 - 1st week Jun 2017
2nd week June 2017
A still contains data Feb 2017 - May 2017
B contains data 2nd week Jun 2017
Append --> will i lose my 1st week Jun 2017?
Thank you
Did you ever get a response or figure out a way to do this and keep all data intact? This is my exact scenario.
Thanks!
--Nick
1 -
It's possible with a Recursive DataFlow: https://knowledge.domo.com/Prepare/DataFlow_Tips_and_Tricks/Creating_a_Recursive%2F%2FSnapshot_ETL_DataFlow
There's an append ETL option for Redshift that's in beta per this domopalooza2020 webinar: https://console.on24.com/view/presentation/flash/ended.html?eventid=2218249&key=BC29488B0C99F2B34D25E651FCF857FE&text_language_id=en&powered-by-on24-visibility=Yes.
1 -
@jimmy Dataset Copy is just a type of Domo connector. It will copy a dataset. Set it to Append instead of Replace.
Use Filters to limit the source dataset to just the subset of data you want to Append.
Jae Wilson
Check out my 🎥 Domo Training YouTube Channel 👨💻
**Say "Thanks" by clicking the ❤️ in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"1
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 296 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.9K Transform
- 100 SQL DataFlows
- 614 Datasets
- 2.2K Magic ETL
- 3.8K Visualize
- 2.5K Charting
- 729 Beast Mode
- 53 App Studio
- 40 Variables
- 677 Automate
- 173 Apps
- 451 APIs & Domo Developer
- 45 Workflows
- 8 DomoAI
- 34 Predict
- 14 Jupyter Workspaces
- 20 R & Python Tiles
- 394 Distribute
- 113 Domo Everywhere
- 275 Scheduled Reports
- 6 Software Integrations
- 121 Manage
- 118 Governance & Security
- Domo Community Gallery
- 32 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 108 Community Announcements
- 4.8K Archive