What is the best way to have updating data from mongo collection?
Currently have some time series data that I would need the full historical data, but only needed it to update every hour or so. The update needs to be a merge, so any records that are changed are updated correctly. Right now I don't see a good way to do this within domo ui, and only see custom connectors or api feeds to update within domo.
Is there a good way to accomplish this with the connectors?
Answers
-
Hi @banderson
it sounds like you’re talking about an upsert type of connection. There are some connectors which support upsert however it appears that the MongoDB connector isn’t one of them. You could try and do something like a recursive dataflow to handle the upsert logic yourself however note that it doesn’t scale as well as the dataset gets larger.
There are some good KB articles about this
Magic ETL v1
Magic ETL v2
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**1 -
Thanks Grant. This will be for a data set that is large that we expect to get exponentially larger. I'll take a look at those and see if they can still be used.
0 -
you may also want to look into data assembler if you have a large dataset and could possibly set it up using some partitions but it’s premium feature. There are a few guidelines for using it which are documented in the knowledge base
https://knowledge.domo.com/The_Appstore/Available_Apps/Data_Assembler_Usage_Guidelines
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**1 -
@banderson use UPSERT + Domo CLI
https://www.youtube.com/watch?v=kHSeSbSFS1Q (sorry less polished video, but gets the job done).
Jae Wilson
Check out my 🎥 Domo Training YouTube Channel 👨💻
**Say "Thanks" by clicking the ❤️ in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"1
Categories
- All Categories
- 1.9K Product Ideas
- 1.9K Ideas Exchange
- 1.6K Connect
- 1.3K Connectors
- 303 Workbench
- 6 Cloud Amplifier
- 9 Federated
- 2.9K Transform
- 104 SQL DataFlows
- 637 Datasets
- 2.2K Magic ETL
- 3.9K Visualize
- 2.5K Charting
- 763 Beast Mode
- 65 App Studio
- 42 Variables
- 709 Automate
- 184 Apps
- 460 APIs & Domo Developer
- 54 Workflows
- 11 DomoAI
- 39 Predict
- 16 Jupyter Workspaces
- 23 R & Python Tiles
- 401 Distribute
- 116 Domo Everywhere
- 277 Scheduled Reports
- 8 Software Integrations
- 134 Manage
- 131 Governance & Security
- 8 Domo Community Gallery
- 44 Product Releases
- 12 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 111 Community Announcements
- 4.8K Archive