Configuration Options for Dataset Copy Dataset Connector

walker_page

I need to copy a dataset from one instance to another. On the first run, I need to copy the entire dataset. Subsequently, only the new/delta data in the source needs to get appended to the dataset in the other instance. I would like to avoid loading a complete refresh to the destination dataset every time new data is added to the source.

I am using the Dataset Copy Dataset Connector, documented here: https://domohelp.domo.com/hc/en-us/articles/360043436533-DataSet-Copy-DataSet-Connector

I'm unsure about the correct configurations to choose to get the above desired results. Specifically, I'm unsure which of these options to select in the Details section:

Initially, I selected 'Fetch incremental data', but when I did that the datatypes for some columns in the destination dataset didn't match the source. And the application doesn't allow selecting 'Fetch incremental data' and choosing 'Yes' in the 'USE ORIGIN DATASET'S SCHEMA' option.

So I'm thinking maybe I need to choose 'Yes' for both 'ONLY UPDATED DATA' and 'USE ORIGIN DATASET'S SCHEMA', but I don't understand what the 'ONLY UPDATED DATA' option is supposed to do:

Does it retrieve the entire source dataset whenever it is updated? Or does it only retrieve the updated/new data in the source whenever new data is added?

GrantSmith

I believe the only updated data flag will only kick off the dataset copy of the origin dataset has been run after the last time the copy has run. This way it won’t keep importing the same data multiple times.