Importing Data into Domo from our Connectors

Mike_Shrift · 2021-12-07T16:35:44+00:00

There was an error rendering this rich post.

A lot of our data sources are becoming fairly large over a few million rows, and we have noticed refresh speeds getting much slower.

I think the solution is to have data appended as of last import rather than replace the whole thing? But I'm curious to know if anyone else has a better solution?

Find more posts tagged with

Dataset

Data Flows

Comments

MarkSnodgrass

If you are using Domo Workbench 5.1, they have the option of partition support, which can help when dealing with large datasets. Here's the KB article about it.

https://domohelp.domo.com/hc/en-us/articles/360062446514-Workbench-5-1-Partition-Support

If you are using another type of connector, appending is the most common way to minimize refresh speeds.

GrantSmith

You can also utilize UPSERT within workbench as well to update / create records so you only need to pull in the changes assuming your data doesn't get deleted out of your tables. If they do you'd need to use partitioning.