best way to replace a subset of data

hanmari

I have a large historical dataset to which I append records at the end of each day. I have been asked to correct the distribution of values for a small subset of the data over the past nine months. I can use the data repair tool feature to remove all of the data imports for that time, but this would force me to destroy and rebuild much more data than is necessary. Is there a scalpel I can use to excise and replace the exact records I need to adjust? I do not wish to take a sledge hammer to large portions of the dataset.

Could I perhaps temporarily alter the import SQL and change its type to use a REPLACE strategy and only update for the intended subset of the data, then change the import back into an APPEND import strategy? Or would this drop all of the non-targeted records from the historical table?

If I need to wholesale replace all the imports over the past nine months, then it might be better for me to use a workflow to move all of the earlier data to a separate table, then erase the original table and import the last nine months of data into another table. I could then use another transformation to combine the two tables into a replacement for the original table. This might be a better long-term strategy for maintaining a historical data set. It would allow me to maintain historical data in separate tables that I could then combine for my long-term reports.

MarkSnodgrass

I would suggest you Magic ETL to clean this up. You can have two filter tiles (one that has the data you need to clean, and the other data that is does not need clean). You can then do your cleaning using a formula tile and then append the two back together and output to a single dataset. You would then build your cards off of this dataset. A basic example would look like this:

jaeW_at_Onyx

use @MarkSnodgrass recommendation plus partitioning in Magic

https://domo-support.domo.com/s/article/000005150?language=en_US

MarkSnodgrass

I would suggest you Magic ETL to clean this up. You can have two filter tiles (one that has the data you need to clean, and the other data that is does not need clean). You can then do your cleaning using a formula tile and then append the two back together and output to a single dataset. You would then build your cards off of this dataset. A basic example would look like this:

jaeW_at_Onyx

use @MarkSnodgrass recommendation plus partitioning in Magic

https://domo-support.domo.com/s/article/000005150?language=en_US