Append with removing rows of data
I am wondering if there is way to setup the following scenario. I have an append that runs in the morning and all the new data gets added properly, however, there are situations where some of the source data is no longer needed for example if I started with 110 rows of data yesterday and throughout the day 10 were removed, then I only need 100 rows of data now. But what would happen if I also had some new rows in the source as well... for example 5 new rows, then my total show be 105
I attached a little chart for better visibility
I know a full replace will take care of this, but we are trying to minimize our row processing budget. Any solution for this or just simply the replace IS the only solution.?
Thank you
Answers
-
Have you looked into using a recursive dataflow to "partition" your data and filter out any old records and replace them with new records?
If you're using workbench have you looked into partitions?
https://domohelp.domo.com/hc/en-us/articles/360062446514-Workbench-5-1-Partition-Support
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**0 -
For the first one wouldn't that still require me to bring in all the data into DOMO then apply the recursion in the ETL? Which at that point I might as well just do a replace of all the data at once.
For the second one, the thing is I am not keeping history in these sources. Therefore DOMO will have more data than what I want to bring in, so when doing the append with the upsert, I was thinking if there was a way that DOMO could detect "Oh, this unique id no longer exists in the data being brought in, delete record from dataset"
0 -
You'd bring in the dataset that's already in Domo, compare to the records you wish to update and filter out the ones you don't want. You'd only be importing the updated partitions into Domo and Domo would have the existing dataset already on hand
The only way Domo can handle removed records is to do a full replace or a partition where it'll fully replace part of your dataset.
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**0
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.6K Connect
- 1.2K Connectors
- 300 Workbench
- 6 Cloud Amplifier
- 9 Federated
- 2.9K Transform
- 101 SQL DataFlows
- 622 Datasets
- 2.2K Magic ETL
- 3.9K Visualize
- 2.5K Charting
- 747 Beast Mode
- 59 App Studio
- 41 Variables
- 686 Automate
- 176 Apps
- 453 APIs & Domo Developer
- 47 Workflows
- 10 DomoAI
- 36 Predict
- 15 Jupyter Workspaces
- 21 R & Python Tiles
- 396 Distribute
- 113 Domo Everywhere
- 276 Scheduled Reports
- 7 Software Integrations
- 126 Manage
- 123 Governance & Security
- 8 Domo Community Gallery
- 38 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 109 Community Announcements
- 4.8K Archive