Managing Data Retention Policy

amehdad
amehdad Coach
edited December 2023 in Datasets Ideas

Currently, there's no feature to create and manage custom rules for handling historical data within datasets. Companies have specific rules on how long they are allowed to keep different types of data and it looks like it is not easy to enforce this in Domo. 

There should be settings in DataSets where we can define conditions and scheduling on the checks and actions to execute (e.g. Which data to archive or remove entirely from storage). 

Tagged:
22
Up
22 votes

Submitted · Last Updated

Comments

  • GrantSmith
    GrantSmith Coach

    You could utilize Magic ETL dataflows to check the date on the data and either split it out and archive it or remove it from the dataset depending on your company's data retention policy.

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • jaeW_at_Onyx
    jaeW_at_Onyx Coach

    this can be automated via APIs. use the DatasetAPI or QueryAPI to find out it's age.

    https://developer.domo.com/docs/dataset-api-reference/dataset#Delete%20a%20DataSet


    the tough part will be if you want to retire rows of data (as opposed to entire datasets). you'll need a recursive query with a partitioning model to drop old data.


    also in the java cli there are functions to assist with setting the retention policy.

    Jae Wilson
    Check out my 🎥 Domo Training YouTube Channel 👨‍💻

    **Say "Thanks" by clicking the ❤️ in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"
  • srajput
    srajput Member

    Hello, is there more information available on this. We have the same need. From what I understand after talking to our Technical Account Manager, DataSets that are unused (how is this determined, is there a time limit?) will transition to a non-queryable state (automated Domo process). Somesort of an alerting process can be built to show this, but then it comes down to the end user to determine whether to make it queryable again or not. I'm guessing that that end user can also decide to delete this DataSet if it falls beyond a company's retention period, but that will need to be done manually as no automated solution exists. Is that true?

Categories