Appending new data in DOMO

HLUNKAD

updating the domo dataset every month with the latest record without doing the manual import. suggestion needed.

ColemenWilson

Many connectors have the option to either replace or append data each time the data updates. For the ones that dont, you can create a recursive dataflow to snapshot the data overtime. Here is a great recording that walks through how to build a recursive dataflow:

https://www.youtube.com/watch?v=JNQFZCj8JcQ

trafalger

Another great option is the dataset copy - you can create a copy with whatever interval you want and have it append.

GrantSmith

A recursive dataflow will allow you to UPSERT your date (update if it already exist or insert a new record if it doesn't). So if you need to re-run your dataset it'll protect against duplicated data. The caveat to recursive dataflows is that by nature as the dataset grows the ETL will take longer to run as there's more data to import. In your case since it's a montly snapshot that likely won't be an issue.

Using dataset copy and setting it to append will be quicker to process but won't protect against duplicated data if it ever runs more than once in a month.

A simpler option is a new feature to MagicETL where you can specify the output method on an Output Dataset tile:

Just input your original dataset and then set it to partition (this will update, insert, or delete records), and make sure you set a partition key for the month and year. You can use a formula tile and add in a Month field using the LAST_DAY function:

LAST_DAY(`date`)

Just make sure you pull the entire month in otherwise it'll overwrite your old whole month with a partial month if it's not entirely pulled in.

ColemenWilson

Many connectors have the option to either replace or append data each time the data updates. For the ones that dont, you can create a recursive dataflow to snapshot the data overtime. Here is a great recording that walks through how to build a recursive dataflow:

https://www.youtube.com/watch?v=JNQFZCj8JcQ

trafalger

Another great option is the dataset copy - you can create a copy with whatever interval you want and have it append.

GrantSmith

A recursive dataflow will allow you to UPSERT your date (update if it already exist or insert a new record if it doesn't). So if you need to re-run your dataset it'll protect against duplicated data. The caveat to recursive dataflows is that by nature as the dataset grows the ETL will take longer to run as there's more data to import. In your case since it's a montly snapshot that likely won't be an issue.

Using dataset copy and setting it to append will be quicker to process but won't protect against duplicated data if it ever runs more than once in a month.

A simpler option is a new feature to MagicETL where you can specify the output method on an Output Dataset tile:

Just input your original dataset and then set it to partition (this will update, insert, or delete records), and make sure you set a partition key for the month and year. You can use a formula tile and add in a Month field using the LAST_DAY function:

LAST_DAY(`date`)

Just make sure you pull the entire month in otherwise it'll overwrite your old whole month with a partial month if it's not entirely pulled in.