Append Output dataset help

A2024

Hello all,

I have an Magic ETL (A) that with a output dataset - say OUTPUT_1.

What I am looking for, is when every time I run the ETL, I need the output results to be appended. At the same time it should not create duplicate records.

I created a new Magic ETL (B) with this OUTPUT_1 as input dataset and outputted to different dataset. And set this to update when Magic ETL (A) is updated..

The issue with this - I see the records are updated and I don't see an option to add the records.

To summarize - 2 issues - How to append the output data? Also, I want to make sure the same data is not duplicated. Unfortunately this does not have a date field. The only identifier I have is file_name.

Not sure what I am missing.. Any inputs would be very helpful to proceed.

GrantSmith

Hi @Aag2023

It sounds like you need a recursive data flow. There is a great KB article on it here: https://domo-support.domo.com/s/article/360057087393?language=en_US

Ashleigh

@Aag2023 there are also some great videos on YouTube about recursive flows. @jaeW_at_Onyx has a few that I know are great.

https://www.youtube.com/watch?v=JNQFZCj8JcQ

GrantSmith

Hi @Aag2023

It sounds like you need a recursive data flow. There is a great KB article on it here: https://domo-support.domo.com/s/article/360057087393?language=en_US

A2024

Thank you @GrantSmith This is very helpful. I will look into this link and will implement recursive data flow for my case.

Ashleigh

@Aag2023 there are also some great videos on YouTube about recursive flows. @jaeW_at_Onyx has a few that I know are great.

https://www.youtube.com/watch?v=JNQFZCj8JcQ