Managing Data Retention Policy

amehdad · 2021-10-11T04:42:40+00:00

There was an error rendering this rich post.

Currently, there's no feature to create and manage custom rules for handling historical data within datasets. Companies have specific rules on how long they are allowed to keep different types of data and it looks like it is not easy to enforce this in Domo.

There should be settings in DataSets where we can define conditions and scheduling on the checks and actions to execute (e.g. Which data to archive or remove entirely from storage).

Find more posts tagged with

Admin

Governance

Dataset

Submitted

Status: Submitted and awaiting review from product team

Comments

GrantSmith

You could utilize Magic ETL dataflows to check the date on the data and either split it out and archive it or remove it from the dataset depending on your company's data retention policy.

jaeW_at_Onyx

this can be automated via APIs. use the DatasetAPI or QueryAPI to find out it's age.

https://developer.domo.com/docs/dataset-api-reference/dataset#Delete%20a%20DataSet

the tough part will be if you want to retire rows of data (as opposed to entire datasets). you'll need a recursive query with a partitioning model to drop old data.

also in the java cli there are functions to assist with setting the retention policy.

amehdad

Some interesting options gents but surely there could a more convenient solution towards lean data instances.

srajput

Hello, is there more information available on this. We have the same need. From what I understand after talking to our Technical Account Manager, DataSets that are unused (how is this determined, is there a time limit?) will transition to a non-queryable state (automated Domo process). Somesort of an alerting process can be built to show this, but then it comes down to the end user to determine whether to make it queryable again or not. I'm guessing that that end user can also decide to delete this DataSet if it falls beyond a company's retention period, but that will need to be done manually as no automated solution exists. Is that true?