What is the best way to have updating data from mongo collection?

banderson

Currently have some time series data that I would need the full historical data, but only needed it to update every hour or so. The update needs to be a merge, so any records that are changed are updated correctly. Right now I don't see a good way to do this within domo ui, and only see custom connectors or api feeds to update within domo.

Is there a good way to accomplish this with the connectors?

GrantSmith

Hi @banderson

it sounds like you’re talking about an upsert type of connection. There are some connectors which support upsert however it appears that the MongoDB connector isn’t one of them. You could try and do something like a recursive dataflow to handle the upsert logic yourself however note that it doesn’t scale as well as the dataset gets larger.

There are some good KB articles about this

Magic ETL v1

https://knowledge.domo.com/Prepare/DataFlow_Tips_and_Tricks/Creating_a_Recursive%2F%2FSnapshot_ETL_DataFlow

Magic ETL v2

https://knowledge.domo.com/Prepare/DataFlow_Tips_and_Tricks/Creating_a_Recursive%2F%2FSnapshot_DataFlow_in_Magic_ETL_v2_%28Beta%29

banderson

Thanks Grant. This will be for a data set that is large that we expect to get exponentially larger. I'll take a look at those and see if they can still be used.

GrantSmith

you may also want to look into data assembler if you have a large dataset and could possibly set it up using some partitions but it’s premium feature. There are a few guidelines for using it which are documented in the knowledge base

https://knowledge.domo.com/The_Appstore/Available_Apps/Data_Assembler_Usage_Guidelines

jaeW_at_Onyx

@banderson use UPSERT + Domo CLI

https://knowledge.domo.com/Administer/Other_Administrative_Tools/Command_Line_Interface_(CLI)_Tool#section_34

https://www.youtube.com/watch?v=kHSeSbSFS1Q (sorry less polished video, but gets the job done).