Thanks @GrantSmith, does this method can be used to alse delete the records that were removed from the source?
Workbench
Workbench
Incremental refreshes using Domo Workbench
Christianjmzc
Member
in Workbench
Hi everyone, is it possible to do incremental refreshes using Domo Workbench?
I have several datasets with more than 15 or 20 million rows and I'm trying to find a way to only load new data using the Domo Workbench instead of full replace, which is time and resource consuming.
I'm using ODBC Connection provider.
0
Best Answer
-
Do you have a created or updated timestamp in your dataset? You can use variables to only get the updated or created records based on when the workbench job last ran.
You can read about lastvalue here:
Customer Support Community**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**0
Answers
-
Do you have a created or updated timestamp in your dataset? You can use variables to only get the updated or created records based on when the workbench job last ran.
You can read about lastvalue here:
Customer Support Community**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**0 -
You should be able to do this utilizing the partition support in Workbench. Here is the KB article on it.
https://domo-support.domo.com/s/article/360062446514?language=en_US**Check out my Domo Tips & Tricks Videos
**Make sure to
any users posts that helped you.
**Please mark as accepted the ones who solved your issue.1 -
Thanks @GrantSmith, does this method can be used to alse delete the records that were removed from the source?
0 -
Hi @MarkSnodgrass, I'll try this and provide an update this week. Thank you!
1 -
No, it will only pull in the data that was updated based on the update timestamp from your database thus reducing the amount of data pulled in and processed. If you're wanting to remove old data then you can look into partitioning as @MarkSnodgrass mentioned.
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**1
Categories
- All Categories
- 2K Product Ideas
- 2K Ideas Exchange
- 1.6K Connect
- 1.3K Connectors
- 311 Workbench
- 7 Cloud Amplifier
- 9 Federated
- 3K Transform
- 114 SQL DataFlows
- 654 Datasets
- 2.2K Magic ETL
- 4.1K Visualize
- 2.5K Charting
- 803 Beast Mode
- 79 App Studio
- 44 Variables
- 758 Automate
- 188 Apps
- 480 APIs & Domo Developer
- 73 Workflows
- 17 DomoAI
- 40 Predict
- 17 Jupyter Workspaces
- 23 R & Python Tiles
- 408 Distribute
- 119 Domo Everywhere
- 279 Scheduled Reports
- 10 Software Integrations
- 141 Manage
- 137 Governance & Security
- 8 Domo Community Gallery
- 47 Product Releases
- 12 Domo University
- 5.4K Community Forums
- 41 Getting Started
- 31 Community Member Introductions
- 114 Community Announcements
- 4.8K Archive