What is the best way to have updating data from mongo collection?

banderson · March 2021

Currently have some time series data that I would need the full historical data, but only needed it to update every hour or so. The update needs to be a merge, so any records that are changed are updated correctly. Right now I don't see a good way to do this within domo ui, and only see custom connectors or api feeds to update within domo.

Is there a good way to accomplish this with the connectors?

GrantSmith · March 2021

Hi @banderson

it sounds like you’re talking about an upsert type of connection. There are some connectors which support upsert however it appears that the MongoDB connector isn’t one of them. You could try and do something like a recursive dataflow to handle the upsert logic yourself however note that it doesn’t scale as well as the dataset gets larger.

There are some good KB articles about this

Magic ETL v1

https://knowledge.domo.com/Prepare/DataFlow_Tips_and_Tricks/Creating_a_Recursive%2F%2FSnapshot_ETL_DataFlow

Magic ETL v2

https://knowledge.domo.com/Prepare/DataFlow_Tips_and_Tricks/Creating_a_Recursive%2F%2FSnapshot_DataFlow_in_Magic_ETL_v2_%28Beta%29

banderson · March 2021

Thanks Grant. This will be for a data set that is large that we expect to get exponentially larger. I'll take a look at those and see if they can still be used.

GrantSmith · March 2021

you may also want to look into data assembler if you have a large dataset and could possibly set it up using some partitions but it’s premium feature. There are a few guidelines for using it which are documented in the knowledge base

https://knowledge.domo.com/The_Appstore/Available_Apps/Data_Assembler_Usage_Guidelines

jaeW_at_Onyx · March 2021

@banderson use UPSERT + Domo CLI

https://knowledge.domo.com/Administer/Other_Administrative_Tools/Command_Line_Interface_(CLI)_Tool#section_34

https://www.youtube.com/watch?v=kHSeSbSFS1Q (sorry less polished video, but gets the job done).

What is the best way to have updating data from mongo collection?

Answers

Categories