How to purge data from an existing dataset via API?

mattAnonomatic
mattAnonomatic Member
edited March 2023 in Datasets

Hello Domo experts,

We are trying to find a way to use the API, or other non-UI approach, to purge all the data from any existing Domo dataset.

A little background may help...

Our company (Anonomatic) provides an add-on to Domo called PII Vault. What it enables is for Domo customers to get full Domo functionality but never have to load customer/patient/etc. personal identifiers (PII) into Domo.

That process works just find and we have healthcare and financial services customers happily using it. However, we recently had some requests from our Domo customers to enable them to take data which is already in a Domo dataset (Dataset A) which has PII in it and run it through our process to create Dataset B which will not have PII. Because of what we do, Dataset B can be made available to basically anyone in their organization without worrying about data privacy issues and they can still do everything else they would do with Dataset A.

The problem is that once we create Dataset B, the data in Dataset A is still there and so the data privacy issue is not fully resolved.

Deleting Dataset A is not an option. They have existing jobs that populate it and they do not want to have to change them. Instead, they want our process to run after it is populated and then clean it out once we are done.

So back to my question. Is this possible? We have scoured the Domo API documentation but there does not seem to be any way to delete records from an existing dataset.

We even tried importing a single blank row into the source dataset but because the dataset was not created via API, it cannot be modified via API.


Thank you all,

Answers

  • ArborRose
    ArborRose Coach

    I had a situation where I needed to purge/truncate and I couldn't figure a way in Domo. I ended up creating a dataset using mysql dataflow where the logic creates a B dataset based on A. The source dataset is never accessed, the output from the mysql becomes the source.

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

  • mattAnonomatic
    mattAnonomatic Member

    Yeah, that is what I think we are left with as well.

    We even have an Augmented Writeback Connector which allows them to automatically anonymize, mask or encrypt their data whenever new data shows up.

    The issue is that often our customers do not want this data to reside in Domo any longer than it takes to process it.

    Thanks for the response.

  • McSQL
    McSQL Domo Employee

    @mattAnonomatic using the domo cli tool or directly via the api, you can upload an empty file to the data source and it will effectively truncate the data set. This statement isn't fully accurate: "We even tried importing a single blank row into the source dataset but because the dataset was not created via API, it cannot be modified via API." Domo data sets can be modified via api regardless of how they were created. It takes a couple of api calls if hitting the undocumented APIs directly. The CLI handles the necessary API calls in one command which is nice.

    **Say “Thanks" by clicking the thumbs up in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"
  • mattAnonomatic
    mattAnonomatic Member

    I'm sorry but I have to disagree with your statement about the API working on any dataset.

    This is the exact error I am receiving:

    {"status":400,"statusReason":"Bad Request","message":"Only DataSets created with the API can be updated via APIs.","toe":"21HOCP2MCI-6U15A-R5OUV"}

  • Adolfo
    Adolfo Member
    edited April 2023

    Having the same issue. Would be great to be able to use TRUNCATE TABLE within DOMO to purge the dataset.

    What I am currently doing is manually switching the Update Mode to Replace, then uploading a file to purge the data, then switch it back to Append to accept new data.

Categories