How to delete rows from an input dataset and replace the dataset
I was trying to use SQL transformations, but I cannot make the output dataset the same as the input dataset. Could someone help?
Comments
-
The solution is a partial recurive flow but in your case, you aren't maintaining history rather, delete/replacing the input with the output.
I've created a flow to demonstrate how you can do this using simple SQL flows adding a constant that will distinguish which rows are obsolete vs. not.
Let me know if this helps.
Test Data
Include a flag/constant that will help you distinguish which rows are obsoleteChoose the Custom SQLDelete rows which are Obsoleteuse this output as your new input
0 -
Thank you for the detailed explanation. I did not know about the DELETE command, and that was very helpful.
While this is a significant improvement from what I thought I could do, there's still this issue that remains:
The new data that comes in is never obsolete. Its only after a certain time that it becomes obsolete. That means that I cannot instantly do operations on incoming data, rather I have to store it in a dataset, and keep running delete statements on that to remove unwanted data.
So while my end output in purged, there is this huge dataset in the middle that I still have to maintain.
Any idea on how I can reduce the size of the middle dataset as well?
1
Categories
- All Categories
- 1.7K Product Ideas
- 1.7K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 292 Workbench
- 4 Cloud Amplifier
- 8 Federated
- 2.8K Transform
- 95 SQL DataFlows
- 602 Datasets
- 2.1K Magic ETL
- 3.7K Visualize
- 2.4K Charting
- 697 Beast Mode
- 43 App Studio
- 39 Variables
- 658 Automate
- 170 Apps
- 441 APIs & Domo Developer
- 42 Workflows
- 5 DomoAI
- 32 Predict
- 12 Jupyter Workspaces
- 20 R & Python Tiles
- 386 Distribute
- 111 Domo Everywhere
- 269 Scheduled Reports
- 6 Software Integrations
- 113 Manage
- 110 Governance & Security
- 8 Domo University
- 30 Product Releases
- Community Forums
- 39 Getting Started
- 29 Community Member Introductions
- 98 Community Announcements
- Domo Community Gallery
- 4.8K Archive