Cloud Amplifier - Incremental Load

ChintanParekh

I'm Running MAGIC ETL to Transfer Data from Cloud Amplifier dataset (connected to Snowflake view with about 200 million rows) to a dataset without any transformation.

Every time it takes about 1 hour 20 minutes to fetch data from Cloud Amplifier dataset (S3LoaderAction) as it seems it is doing full Data fetch all the time.

So, I'm wondering if this action can be configured to just load incremental as underlying view is having incremental data …it adds yesterday sales data which is not going to be change once landed.

ArborRose

If your dataset has something you can use to identify the incremental changes, yes. A recursive data set can pull records and append the records to an existing dataset. I would look at doing something similar. You just need to make sure your ETL accounts for changes in records.

The following link should give an idea…

domo-support.domo.com/s/article/360057087393?language=en_US

ArborRose

If your dataset has something you can use to identify the incremental changes, yes. A recursive data set can pull records and append the records to an existing dataset. I would look at doing something similar. You just need to make sure your ETL accounts for changes in records.

The following link should give an idea…

domo-support.domo.com/s/article/360057087393?language=en_US