Incremental refreshes using Domo Workbench

Christianjmzc · November 2023

Hi everyone, is it possible to do incremental refreshes using Domo Workbench?

I have several datasets with more than 15 or 20 million rows and I'm trying to find a way to only load new data using the Domo Workbench instead of full replace, which is time and resource consuming.

I'm using ODBC Connection provider.

GrantSmith · November 2023

Do you have a created or updated timestamp in your dataset? You can use variables to only get the updated or created records based on when the workbench job last ran.

You can read about lastvalue here: https://domo-support.domo.com/s/article/4407020836887?language=en_US

GrantSmith · November 2023

Do you have a created or updated timestamp in your dataset? You can use variables to only get the updated or created records based on when the workbench job last ran.

You can read about lastvalue here: https://domo-support.domo.com/s/article/4407020836887?language=en_US

MarkSnodgrass · November 2023

You should be able to do this utilizing the partition support in Workbench. Here is the KB article on it.
https://domo-support.domo.com/s/article/360062446514?language=en_US

Christianjmzc · November 2023

Thanks @GrantSmith, does this method can be used to alse delete the records that were removed from the source?

Christianjmzc · November 2023

Hi @MarkSnodgrass, I'll try this and provide an update this week. Thank you!

GrantSmith · November 2023

https://community-forums.domo.com/main/discussion/comment/83254#Comment_83254

No, it will only pull in the data that was updated based on the update timestamp from your database thus reducing the amount of data pulled in and processed. If you're wanting to remove old data then you can look into partitioning as @MarkSnodgrass mentioned.

Incremental refreshes using Domo Workbench

Best Answer

Answers

Categories