Delete rows of data (based on criteria) from data set stored in Domo cloud

swagner

I have a data set that appends about 1M rows daily up to a data set stored in Domo - has been in use for about 3 months.  The data set is approach 100M rows, and we no longer need the majority of the data.  This table contains daily inventory levels for all items, our management determined they only really need items with a few specific velocity codes (fastest selling items only).

 

Example:  we want to keep items with a velocity code of A, B, C, D, or E only, all others can be deleted.

 

My thought is that if I can remove the unwanted rows from the data set stored in Domo cloud... problem sloved.  Otherwise I need to essentially start over with a new data set with the revised criteria.  I realize I can filter out the unwanted rows, but the ETL run time is impacted as it has to evaluate all of those millions of rows every time it runs.

 

Any ideas?

AS

Where is the data coming from? Under the job settings, if you can set it to 'replace', run it once again for a full replace, and then set it back to append for future processing.  That way you effectively delete everything and include only what you want going forward.  

That's all dependent on the replace/append options available to you.

Unknown

Yeah particularly if this is feeding into a dataflow/ETL it would be easiest to configure it to the new criteria and do a replace run and then start the append back up from there.