Delete rows of data (based on criteria) from data set stored in Domo cloud
I have a data set that appends about 1M rows daily up to a data set stored in Domo - has been in use for about 3 months. The data set is approach 100M rows, and we no longer need the majority of the data. This table contains daily inventory levels for all items, our management determined they only really need items with a few specific velocity codes (fastest selling items only).
Example: we want to keep items with a velocity code of A, B, C, D, or E only, all others can be deleted.
My thought is that if I can remove the unwanted rows from the data set stored in Domo cloud... problem sloved. Otherwise I need to essentially start over with a new data set with the revised criteria. I realize I can filter out the unwanted rows, but the ETL run time is impacted as it has to evaluate all of those millions of rows every time it runs.
Any ideas?
Comments
-
Where is the data coming from? Under the job settings, if you can set it to 'replace', run it once again for a full replace, and then set it back to append for future processing. That way you effectively delete everything and include only what you want going forward.
That's all dependent on the replace/append options available to you.
Aaron
MajorDomo @ Merit Medical
**Say "Thanks" by clicking the heart in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"0 -
Yeah particularly if this is feeding into a dataflow/ETL it would be easiest to configure it to the new criteria and do a replace run and then start the append back up from there.
0
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.6K Connect
- 1.2K Connectors
- 300 Workbench
- 6 Cloud Amplifier
- 9 Federated
- 2.9K Transform
- 101 SQL DataFlows
- 622 Datasets
- 2.2K Magic ETL
- 3.9K Visualize
- 2.5K Charting
- 748 Beast Mode
- 59 App Studio
- 41 Variables
- 686 Automate
- 176 Apps
- 453 APIs & Domo Developer
- 47 Workflows
- 10 DomoAI
- 36 Predict
- 15 Jupyter Workspaces
- 21 R & Python Tiles
- 396 Distribute
- 113 Domo Everywhere
- 276 Scheduled Reports
- 7 Software Integrations
- 125 Manage
- 122 Governance & Security
- 8 Domo Community Gallery
- 38 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 109 Community Announcements
- 4.8K Archive