Remove Duplicates for Append Before it Gets to Domo


I have some data i want to bring in but unfortunately the source data is a "Keep most recent X records" not a "Keep the last X days records".
One option I know would be to just append it daily and then have a dataflow that removes the duplicates and it would give me what I'm looking for, however this would mean my original Domo dataset would become bloated (~365000 rows/yr). How might I remove duplicates before it gets to Domo?
**Make sure to like any users posts that helped you and accept the ones who solved your issue.**
Best Answer
-
Sounds to me like you'd have to do this programatically using Domo's developer APIs. You could pull the data from Domo, add and deduplicate source system data, and reupload the cleaned data back to Domo. Repeated on whatever cadence you need.
Aside from that, using a dataflow is actually a good idea, and 365k annual rows isn't a huge deal in my book. We have datasets in the tens of millions and there are customers with hundreds of millions or billions in their Domo instances. That seems like a secondary concern to me.
Aaron
MajorDomo @ Merit Medical
**Say "Thanks" by clicking the heart in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"1
Answers
-
Sounds to me like you'd have to do this programatically using Domo's developer APIs. You could pull the data from Domo, add and deduplicate source system data, and reupload the cleaned data back to Domo. Repeated on whatever cadence you need.
Aside from that, using a dataflow is actually a good idea, and 365k annual rows isn't a huge deal in my book. We have datasets in the tens of millions and there are customers with hundreds of millions or billions in their Domo instances. That seems like a secondary concern to me.
Aaron
MajorDomo @ Merit Medical
**Say "Thanks" by clicking the heart in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"1
Categories
- 10.5K All Categories
- 7 Connect
- 917 Connectors
- 250 Workbench
- 465 Transform
- 1.7K Magic ETL
- 69 SQL DataFlows
- 477 Datasets
- 193 Visualize
- 252 Beast Mode
- 2.1K Charting
- 11 Variables
- 17 Automate
- 354 APIs & Domo Developer
- 89 Apps
- 3 Workflows
- 20 Predict
- 5 Jupyter Workspaces
- 15 R & Python Tiles
- 246 Distribute
- 62 Domo Everywhere
- 243 Scheduled Reports
- 21 Manage
- 42 Governance & Security
- 173 Product Ideas
- 1.2K Ideas Exchange
- 12 Community Forums
- 27 Getting Started
- 14 Community Member Introductions
- 55 Community News
- 4.5K Archive