Append Output dataset help


Hello all,

I have an Magic ETL (A) that with a output dataset - say OUTPUT_1.

What I am looking for, is when every time I run the ETL, I need the output results to be appended. At the same time it should not create duplicate records.

I created a new Magic ETL (B) with this OUTPUT_1 as input dataset and outputted to different dataset. And set this to update when Magic ETL (A) is updated..

The issue with this - I see the records are updated and I don't see an option to add the records.

To summarize - 2 issues - How to append the output data? Also, I want to make sure the same data is not duplicated. Unfortunately this does not have a date field. The only identifier I have is file_name.

Not sure what I am missing.. Any inputs would be very helpful to proceed.

Best Answers