Large dataset import in Domo Workbench

mroker · 2023-03-05T00:30:30+00:00

There was an error rendering this rich post.

Hi everyone, I have a very large dataset (18M rows) that I want to import and it takes too long. The query is already a grouped query so there is no unique ID. I am just trying to figure out if there is a way to import possibly a one time chunk (the data is for 2 years) and then maybe append the data after that. So maybe run a 2 year chunk, and then run a daily job that just gets the data that was there for a few days (maybe with an append). How can this be done without a unique ID?

Find more posts tagged with

Accepted answers

All comments

ArborRose

Yes, there is a way to split up timed sections and append. You can import via workbench, or connect to your data with an api. Create smaller datasets such as data_yr_2021, data_yr_2022, data_qtr1, data_qtr2, etc. Then create a workflow and use the append tile to append the datasets. That way, you can pull previous years once or once per month, and more recent data daily. The workflow will give a full dataset result with all records for all times. I do this at my company to pull several years worth of data for analysis, but its only pulling the quarter to date on a daily basis.

GrantSmith

I’d recommend using the CLI tool to do a large dataset import as it will use the streams API and split your data into chunks and load your data faster