Cloud Amplifier - Incremental Load

I'm Running MAGIC ETL to Transfer Data from Cloud Amplifier dataset (connected to Snowflake view with about 200 million rows) to a dataset without any transformation.

Every time it takes about 1 hour 20 minutes to fetch data from Cloud Amplifier dataset (S3LoaderAction) as it seems it is doing full Data fetch all the time.

So, I'm wondering if this action can be configured to just load incremental as underlying view is having incremental data …it adds yesterday sales data which is not going to be change once landed.

Best Answer

  • ArborRose
    ArborRose Contributor
    edited January 17 Answer ✓

    If your dataset has something you can use to identify the incremental changes, yes. A recursive data set can pull records and append the records to an existing dataset. I would look at doing something similar. You just need to make sure your ETL accounts for changes in records.

    The following link should give an idea…

    domo-support.domo.com/s/article/360057087393?language=en_US

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

Answers

  • ArborRose
    ArborRose Contributor
    edited January 17 Answer ✓

    If your dataset has something you can use to identify the incremental changes, yes. A recursive data set can pull records and append the records to an existing dataset. I would look at doing something similar. You just need to make sure your ETL accounts for changes in records.

    The following link should give an idea…

    domo-support.domo.com/s/article/360057087393?language=en_US

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **