What is the best way to have updating data from mongo collection?

Currently have some time series data that I would need the full historical data, but only needed it to update every hour or so. The update needs to be a merge, so any records that are changed are updated correctly. Right now I don't see a good way to do this within domo ui, and only see custom connectors or api feeds to update within domo.
Is there a good way to accomplish this with the connectors?
Answers
-
Hi @banderson
it sounds like you’re talking about an upsert type of connection. There are some connectors which support upsert however it appears that the MongoDB connector isn’t one of them. You could try and do something like a recursive dataflow to handle the upsert logic yourself however note that it doesn’t scale as well as the dataset gets larger.
There are some good KB articles about this
Magic ETL v1
Magic ETL v2
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**1 -
Thanks Grant. This will be for a data set that is large that we expect to get exponentially larger. I'll take a look at those and see if they can still be used.
0 -
you may also want to look into data assembler if you have a large dataset and could possibly set it up using some partitions but it’s premium feature. There are a few guidelines for using it which are documented in the knowledge base
https://knowledge.domo.com/The_Appstore/Available_Apps/Data_Assembler_Usage_Guidelines
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**1 -
@banderson use UPSERT + Domo CLI
https://www.youtube.com/watch?v=kHSeSbSFS1Q (sorry less polished video, but gets the job done).
Jae Wilson
Check out my 🎥 Domo Training YouTube Channel 👨💻
**Say "Thanks" by clicking the ❤️ in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"1
Categories
- All Categories
- 2K Product Ideas
- 2K Ideas Exchange
- 1.6K Connect
- 1.3K Connectors
- 311 Workbench
- 6 Cloud Amplifier
- 9 Federated
- 3.8K Transform
- 656 Datasets
- 115 SQL DataFlows
- 2.2K Magic ETL
- 811 Beast Mode
- 3.3K Visualize
- 2.5K Charting
- 80 App Studio
- 45 Variables
- 771 Automate
- 190 Apps
- 481 APIs & Domo Developer
- 77 Workflows
- 23 Code Engine
- 36 AI and Machine Learning
- 19 AI Chat
- AI Playground
- AI Projects and Models
- 17 Jupyter Workspaces
- 410 Distribute
- 120 Domo Everywhere
- 280 Scheduled Reports
- 10 Software Integrations
- 142 Manage
- 138 Governance & Security
- 8 Domo Community Gallery
- 48 Product Releases
- 12 Domo University
- 5.4K Community Forums
- 41 Getting Started
- 31 Community Member Introductions
- 114 Community Announcements
- 4.8K Archive