Appending new data in DOMO
Best Answers
-
Many connectors have the option to either replace or append data each time the data updates. For the ones that dont, you can create a recursive dataflow to snapshot the data overtime. Here is a great recording that walks through how to build a recursive dataflow:
If I solved your problem, please select "yes" above
0 -
Another great option is the dataset copy - you can create a copy with whatever interval you want and have it append.
1 -
A recursive dataflow will allow you to UPSERT your date (update if it already exist or insert a new record if it doesn't). So if you need to re-run your dataset it'll protect against duplicated data. The caveat to recursive dataflows is that by nature as the dataset grows the ETL will take longer to run as there's more data to import. In your case since it's a montly snapshot that likely won't be an issue.
Using dataset copy and setting it to append will be quicker to process but won't protect against duplicated data if it ever runs more than once in a month.
A simpler option is a new feature to MagicETL where you can specify the output method on an Output Dataset tile:
Just input your original dataset and then set it to partition (this will update, insert, or delete records), and make sure you set a partition key for the month and year. You can use a formula tile and add in a Month field using the LAST_DAY function:
LAST_DAY(`date`)
Just make sure you pull the entire month in otherwise it'll overwrite your old whole month with a partial month if it's not entirely pulled in.
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**0
Answers
-
Many connectors have the option to either replace or append data each time the data updates. For the ones that dont, you can create a recursive dataflow to snapshot the data overtime. Here is a great recording that walks through how to build a recursive dataflow:
If I solved your problem, please select "yes" above
0 -
Another great option is the dataset copy - you can create a copy with whatever interval you want and have it append.
1 -
A recursive dataflow will allow you to UPSERT your date (update if it already exist or insert a new record if it doesn't). So if you need to re-run your dataset it'll protect against duplicated data. The caveat to recursive dataflows is that by nature as the dataset grows the ETL will take longer to run as there's more data to import. In your case since it's a montly snapshot that likely won't be an issue.
Using dataset copy and setting it to append will be quicker to process but won't protect against duplicated data if it ever runs more than once in a month.
A simpler option is a new feature to MagicETL where you can specify the output method on an Output Dataset tile:
Just input your original dataset and then set it to partition (this will update, insert, or delete records), and make sure you set a partition key for the month and year. You can use a formula tile and add in a Month field using the LAST_DAY function:
LAST_DAY(`date`)
Just make sure you pull the entire month in otherwise it'll overwrite your old whole month with a partial month if it's not entirely pulled in.
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**0
Categories
- All Categories
- 1.2K Product Ideas
- 1.2K Ideas Exchange
- 1.4K Connect
- 1.1K Connectors
- 273 Workbench
- 2 Cloud Amplifier
- 3 Federated
- 2.7K Transform
- 78 SQL DataFlows
- 525 Datasets
- 2.1K Magic ETL
- 2.9K Visualize
- 2.2K Charting
- 437 Beast Mode
- 23 Variables
- 513 Automate
- 115 Apps
- 390 APIs & Domo Developer
- 8 Workflows
- 26 Predict
- 10 Jupyter Workspaces
- 16 R & Python Tiles
- 332 Distribute
- 77 Domo Everywhere
- 255 Scheduled Reports
- 66 Manage
- 66 Governance & Security
- 1 Product Release Questions
- Community Forums
- 40 Getting Started
- 26 Community Member Introductions
- 68 Community Announcements
- 4.8K Archive