Copying dataset pdp settings

Is there a way to copy pdp settings between datasets?

Best Answer

  • WHM
    WHM Contributor
    Answer ✓

    I scripted it using PowerShell and the API... All you need to add is the access token and the dataset ids. there are comments in the code from past projects and it is not perfect but it worked for our purpose. We have thousands of PDP rules and the excel pugin bogs down. There is also a PDP utility but you have to create a dataset of PDP rules... This was easier.

     


    function Get-Dataset($datasetId, $Authentication) {
    #Write-Host $Authentication
    $headers = @{
    Accept = 'text/csv'
    Authorization = $Authentication.access_token
    }
    #Write-Host "https://api.domo.com/v1/datasets/$datasetId/data?includeHeader=true&fileName=dump.csv" -Headers $headers
    try {
    $csv = Invoke-RestMethod -Method Get -Uri "https://api.domo.com/v1/datasets/$datasetId/data?includeHeader=true&fileName=dump.csv" -Headers $headers
    $csv = $csv -replace '\\N', $null
    $data = ConvertFrom-Csv $csv
    #Write-Host "must have Successfully Read CSV File"
    }
    catch {
    Write-Host "Failed to Read Dataset CSV File"
    Write-Host $_.ErrorDetails
    Write-Host $_.Exception
    Write-Host $_.FullyQualifiedErrorId
    Start-Sleep -Seconds 10
    }
    return $data
    }

    function Get-Dataset-Object($datasetId, $Authentication) {
    #Write-Host $Authentication
    $NeedData = $true
    While( $NeedData -eq $true -and $Cnct_Cntr -lt 100){
    try {
    $headers = @{
    Accept = 'application/json'
    Authorization = $Authentication.access_token
    }
    #Write-Host "https://api.domo.com/v1/datasets/$datasetId" -Headers $headers
    $data = Invoke-RestMethod -Method Get -Uri "https://api.domo.com/v1/datasets/$datasetId" -Headers $headers
    #Write-Host "must have Successfully Read CSV File"
    $Cnct_Cntr++
    $NeedData = $false
    }
    catch {
    Write-Host "Failed to get dataset object"
    Write-Host $_.ErrorDetails
    Write-Host $_.Exception
    Write-Host $_.FullyQualifiedErrorId
    $Cnct_Cntr++
    Write-Host $Cnct_Cntr
    Start-Sleep -Seconds 10
    }
    }
    return $data
    }

    function Get-Dataset-PDP($datasetId, $Authentication) {
    #Write-Host $Authentication
    $NeedData = $true
    While( $NeedData -eq $true -and $Cnct_Cntr -lt 100){
    try {
    $headers = @{
    Accept = 'application/json'
    Authorization = $Authentication.access_token
    }
    #Write-Host "https://api.domo.com/v1/datasets/$datasetId" -Headers $headers
    $data = Invoke-RestMethod -Method Get -Uri "https://api.domo.com/v1/datasets/$datasetId/policies" -Headers $headers
    #Write-Host "must have Successfully Read CSV File"
    $Cnct_Cntr++
    $NeedData = $false
    }
    catch {
    Write-Host "Failed to get dataset object"
    Write-Host $_.ErrorDetails
    Write-Host $_.Exception
    Write-Host $_.FullyQualifiedErrorId
    $Cnct_Cntr++
    Write-Host $Cnct_Cntr
    Start-Sleep -Seconds 10
    }
    }
    return $data
    }

    function Get-Authorization($clientID, $clientSecret) {
    $basicAuth = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $clientID, $clientSecret)));
    $authenticationHeader = @{
    Authorization = ("Basic {0}" -f $basicAuth)
    }
    $NeedToken = $true
    $Cnct_Cntr = 0

    #keep trying to get a token
    While( $NeedToken -eq $true -and $Cnct_Cntr -lt 100){
    try {
    $data = Invoke-RestMethod -Method Post -Uri "https://api.domo.com/oauth/token?grant_type=client_credentials&scope=data%20user" -Headers $authenticationHeader
    # $data = Invoke-RestMethod -Method Post -Uri "https://api.domo.com/oauth/token?grant_type=client_credentials&scope=full" -Headers $authenticationHeader
    $data.access_token = "bearer $($data.access_token)"
    # Write-Host "Got Token"
    $Cnct_Cntr++
    $NeedToken = $false
    }
    catch {
    #$ErrorActionPreference = Stop #this does not recognize "Stop"
    Write-Host "Failed to get Token"
    Write-Host $_.ErrorDetails
    Write-Host $_.Exception
    Write-Host $_.FullyQualifiedErrorId
    $Cnct_Cntr++
    Write-Host $Cnct_Cntr
    Start-Sleep -Seconds 10
    }
    }
    return $data
    }

    function Send-ToEmail([string]$email, [string]$body){

    $message = new-object Net.Mail.MailMessage;
    $message.From = "Domo@daltile.com";
    $message.To.Add($email);
    $message.Subject = "Error Running Vistex jobs Script";
    $message.Body = $body;
    $smtp = new-object Net.Mail.SmtpClient("smtpmail.mohawkind.com");
    $smtp.send($message);
    write-host "Mail Sent" ;
    }


    Write-Host "go get auth token"
    $Authentication = Get-Authorization -clientID $apiKey -clientSecret $apiToken
    Write-Host "have token"
    #Write-Host $Authentication
    #$MetData = Get-Dataset -datasetId $datasetmetadata -Authentication $Authentication

    $output = ''
    $haserr = $true
    $Have_PDP = $false
    #get the list of WB jobs
    while(!$Have_PDP){
    try{
    $Authentication = Get-Authorization -clientID $apiKey -clientSecret $apiToken
    $DS_PDP = Get-Dataset-PDP -datasetId $PDPSourceDatasetID -Authentication $Authentication
    $Have_PDP = $true
    }
    catch{
    Write-Host "Failed to get dataset list of PDP rules"
    Write-Host $_.ErrorDetails
    Write-Host $_.Exception
    Write-Host $_.FullyQualifiedErrorId
    Start-Sleep -Seconds 10
    }
    }
    # Cycle through the list of jobs ensuring they run successfully.

     


    $PDPs = $DS_PDP.Where{$_."name" -like "z_rep_*" -or $_."name" -like "z_ssc_*" } #| Sort-Object -Property $_."Execution Order"| Sort-Object -Property $_."Execution Order"
    foreach($PDP in $PDPs){
    $haserr = $true
    $output = ''
    $msg = ''
    $Count = 0

    Try{
    $hdrs = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
    $hdrs.Add("Content-Type", "application/json")
    $hdrs.Add("Accept", 'application/json')
    $hdrs.Add("Authorization", $Authentication.access_token)
    $filter = $PDP.filters
    $name = $PDP.name
    $type = $PDP.type
    $users = $PDP.users
    $groups = $PDP.groups


    $i = 0
    $pdp_rule = '{ "name": "'+$name+'",'
    $fString = '"filters": ['
    foreach($f in $PDP.filters){
    $fString += '{ "column": "'+$f.column+'", "values": [ "'+$f.values+'" ], "operator": "'+$f.operator+'" }'
    $i++
    if($i -lt $PDP.filters.count){$fString += ','}
    }

    $pdp_rule += $fString +' ], "groups": [ "'+$groups+'" ] }'
    # {
    # "name": "z_ssc_4000",
    # "filters": [ {
    # "column": "business_area_number",
    # "values": [ "4000" ],
    # "operator": "EQUALS"
    # } ],
    # "groups": [ 612003576 ]
    #}


    $post = Invoke-WebRequest -Method POST -Uri "https://api.domo.com/v1/datasets/$DestinationDatasetId/policies" -UseBasicParsing -Body $pdp_rule -Headers $hdrs -ContentType 'application/json'

    Write-Host "updated $name in Proex dataset PDP "
    }

    catch {
    Write-Host "Failed to add $email to Domo"
    Write-Host $_.ErrorDetails
    Write-Host $_.Exception
    Write-Host $_.FullyQualifiedErrorId

    }

    }#end foreach job


    $now = Get-Date -Format g
    Write-Host "$now Done."

Answers

  • There isn't a process currently place through the interface that lets you do this.  I think there's a post on the Ideas Exchange for it, though.  Ideally, I think, there would be a list of PDPs stored independent of datasets, and you could apply a group of policies to datasets at the click of a button.

     

    That said, we're using a not-so-official software application that Domo skunkworks wrote that lets us apply policies across different datasets using a config file hosted in Domo via a webform dataset.  It's saved us a ton of time and serves the same purpose.  You should inquire with your success manager or rep about that.

    Aaron
    MajorDomo @ Merit Medical

    **Say "Thanks" by clicking the heart in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"
  • I was trying to search for this ap but couldn't. Is there anyway that I can access and use it to copy over PDP to differnt data set?

  • I'm sure this application isn't available to download.  You'll have to ask Domo directly.

    I'd also watch out in the coming months to see if something comes into the product.  I wouldn't be surprised to see something at Domopalooza, for example.

    Aaron
    MajorDomo @ Merit Medical

    **Say "Thanks" by clicking the heart in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"
  • I'm revisiting this because I've continued to builed datasets that have the exact same PDP settings as previous datasets, and I still don't see a way to copy the settings from one dataset to another.  

     

    Does this exist or do I have to continue recreating the wheel on every dataset?

  • Have you tried the Excel plugin for DOMO, this allows you to apply PDP security to any dataset you want and is a fast and easy way to apply PDP policies that are exact copies to other datasets.

    Just change the DatasetID and apply and your good.

    This is also very easy to make slight modifications to PDP's that are close but not the same.

    It is also an easy way to manage other components such as user accounts etc.

    Randy

  • I will request the admin version of that.

     

    Thanks.

  • You should be able to download it from the Admin/Tools Download section.

    Randy

  • The admin one requires a special request, but I did get.  It's not quite the elegant solution I was hoping for within DOMO, but it will work for when I have to add bulk permissions.  

     

  • WHM
    WHM Contributor
    Answer ✓

    I scripted it using PowerShell and the API... All you need to add is the access token and the dataset ids. there are comments in the code from past projects and it is not perfect but it worked for our purpose. We have thousands of PDP rules and the excel pugin bogs down. There is also a PDP utility but you have to create a dataset of PDP rules... This was easier.

     


    function Get-Dataset($datasetId, $Authentication) {
    #Write-Host $Authentication
    $headers = @{
    Accept = 'text/csv'
    Authorization = $Authentication.access_token
    }
    #Write-Host "https://api.domo.com/v1/datasets/$datasetId/data?includeHeader=true&fileName=dump.csv" -Headers $headers
    try {
    $csv = Invoke-RestMethod -Method Get -Uri "https://api.domo.com/v1/datasets/$datasetId/data?includeHeader=true&fileName=dump.csv" -Headers $headers
    $csv = $csv -replace '\\N', $null
    $data = ConvertFrom-Csv $csv
    #Write-Host "must have Successfully Read CSV File"
    }
    catch {
    Write-Host "Failed to Read Dataset CSV File"
    Write-Host $_.ErrorDetails
    Write-Host $_.Exception
    Write-Host $_.FullyQualifiedErrorId
    Start-Sleep -Seconds 10
    }
    return $data
    }

    function Get-Dataset-Object($datasetId, $Authentication) {
    #Write-Host $Authentication
    $NeedData = $true
    While( $NeedData -eq $true -and $Cnct_Cntr -lt 100){
    try {
    $headers = @{
    Accept = 'application/json'
    Authorization = $Authentication.access_token
    }
    #Write-Host "https://api.domo.com/v1/datasets/$datasetId" -Headers $headers
    $data = Invoke-RestMethod -Method Get -Uri "https://api.domo.com/v1/datasets/$datasetId" -Headers $headers
    #Write-Host "must have Successfully Read CSV File"
    $Cnct_Cntr++
    $NeedData = $false
    }
    catch {
    Write-Host "Failed to get dataset object"
    Write-Host $_.ErrorDetails
    Write-Host $_.Exception
    Write-Host $_.FullyQualifiedErrorId
    $Cnct_Cntr++
    Write-Host $Cnct_Cntr
    Start-Sleep -Seconds 10
    }
    }
    return $data
    }

    function Get-Dataset-PDP($datasetId, $Authentication) {
    #Write-Host $Authentication
    $NeedData = $true
    While( $NeedData -eq $true -and $Cnct_Cntr -lt 100){
    try {
    $headers = @{
    Accept = 'application/json'
    Authorization = $Authentication.access_token
    }
    #Write-Host "https://api.domo.com/v1/datasets/$datasetId" -Headers $headers
    $data = Invoke-RestMethod -Method Get -Uri "https://api.domo.com/v1/datasets/$datasetId/policies" -Headers $headers
    #Write-Host "must have Successfully Read CSV File"
    $Cnct_Cntr++
    $NeedData = $false
    }
    catch {
    Write-Host "Failed to get dataset object"
    Write-Host $_.ErrorDetails
    Write-Host $_.Exception
    Write-Host $_.FullyQualifiedErrorId
    $Cnct_Cntr++
    Write-Host $Cnct_Cntr
    Start-Sleep -Seconds 10
    }
    }
    return $data
    }

    function Get-Authorization($clientID, $clientSecret) {
    $basicAuth = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $clientID, $clientSecret)));
    $authenticationHeader = @{
    Authorization = ("Basic {0}" -f $basicAuth)
    }
    $NeedToken = $true
    $Cnct_Cntr = 0

    #keep trying to get a token
    While( $NeedToken -eq $true -and $Cnct_Cntr -lt 100){
    try {
    $data = Invoke-RestMethod -Method Post -Uri "https://api.domo.com/oauth/token?grant_type=client_credentials&scope=data%20user" -Headers $authenticationHeader
    # $data = Invoke-RestMethod -Method Post -Uri "https://api.domo.com/oauth/token?grant_type=client_credentials&scope=full" -Headers $authenticationHeader
    $data.access_token = "bearer $($data.access_token)"
    # Write-Host "Got Token"
    $Cnct_Cntr++
    $NeedToken = $false
    }
    catch {
    #$ErrorActionPreference = Stop #this does not recognize "Stop"
    Write-Host "Failed to get Token"
    Write-Host $_.ErrorDetails
    Write-Host $_.Exception
    Write-Host $_.FullyQualifiedErrorId
    $Cnct_Cntr++
    Write-Host $Cnct_Cntr
    Start-Sleep -Seconds 10
    }
    }
    return $data
    }

    function Send-ToEmail([string]$email, [string]$body){

    $message = new-object Net.Mail.MailMessage;
    $message.From = "Domo@daltile.com";
    $message.To.Add($email);
    $message.Subject = "Error Running Vistex jobs Script";
    $message.Body = $body;
    $smtp = new-object Net.Mail.SmtpClient("smtpmail.mohawkind.com");
    $smtp.send($message);
    write-host "Mail Sent" ;
    }


    Write-Host "go get auth token"
    $Authentication = Get-Authorization -clientID $apiKey -clientSecret $apiToken
    Write-Host "have token"
    #Write-Host $Authentication
    #$MetData = Get-Dataset -datasetId $datasetmetadata -Authentication $Authentication

    $output = ''
    $haserr = $true
    $Have_PDP = $false
    #get the list of WB jobs
    while(!$Have_PDP){
    try{
    $Authentication = Get-Authorization -clientID $apiKey -clientSecret $apiToken
    $DS_PDP = Get-Dataset-PDP -datasetId $PDPSourceDatasetID -Authentication $Authentication
    $Have_PDP = $true
    }
    catch{
    Write-Host "Failed to get dataset list of PDP rules"
    Write-Host $_.ErrorDetails
    Write-Host $_.Exception
    Write-Host $_.FullyQualifiedErrorId
    Start-Sleep -Seconds 10
    }
    }
    # Cycle through the list of jobs ensuring they run successfully.

     


    $PDPs = $DS_PDP.Where{$_."name" -like "z_rep_*" -or $_."name" -like "z_ssc_*" } #| Sort-Object -Property $_."Execution Order"| Sort-Object -Property $_."Execution Order"
    foreach($PDP in $PDPs){
    $haserr = $true
    $output = ''
    $msg = ''
    $Count = 0

    Try{
    $hdrs = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
    $hdrs.Add("Content-Type", "application/json")
    $hdrs.Add("Accept", 'application/json')
    $hdrs.Add("Authorization", $Authentication.access_token)
    $filter = $PDP.filters
    $name = $PDP.name
    $type = $PDP.type
    $users = $PDP.users
    $groups = $PDP.groups


    $i = 0
    $pdp_rule = '{ "name": "'+$name+'",'
    $fString = '"filters": ['
    foreach($f in $PDP.filters){
    $fString += '{ "column": "'+$f.column+'", "values": [ "'+$f.values+'" ], "operator": "'+$f.operator+'" }'
    $i++
    if($i -lt $PDP.filters.count){$fString += ','}
    }

    $pdp_rule += $fString +' ], "groups": [ "'+$groups+'" ] }'
    # {
    # "name": "z_ssc_4000",
    # "filters": [ {
    # "column": "business_area_number",
    # "values": [ "4000" ],
    # "operator": "EQUALS"
    # } ],
    # "groups": [ 612003576 ]
    #}


    $post = Invoke-WebRequest -Method POST -Uri "https://api.domo.com/v1/datasets/$DestinationDatasetId/policies" -UseBasicParsing -Body $pdp_rule -Headers $hdrs -ContentType 'application/json'

    Write-Host "updated $name in Proex dataset PDP "
    }

    catch {
    Write-Host "Failed to add $email to Domo"
    Write-Host $_.ErrorDetails
    Write-Host $_.Exception
    Write-Host $_.FullyQualifiedErrorId

    }

    }#end foreach job


    $now = Get-Date -Format g
    Write-Host "$now Done."

  • Hi @Randyb , can you provide some more information on how you accomplished this using the Excel plugin?  I've got a massive PDP update project I need to undertake, and this sounds like a great option for us.  Are you exporting a specific dataset into Excel then re-uploading with changes, or something else?

     

    Thanks!

     

    -Russell 

  • @rhollander There are a couple of ways you can do this, you can download the excel plugin which then allows you to pull the data down along with the different PDP profiles. You can then build essentially a user list for a specific PDPPolicy and upload it.

    Its been awhile since I've had to use it but it does make large scale PDP policy setups fairly easy.
    If you need to make changes that don't involve creating PDP policies you can also pull down the entire user list from DOMO in excel format and update access etc in the sheet and then re upload that in bulk. Another quick way to reassign access to entire pages etc.
     
    Randy
  • https://knowledge.domo.com/Connect/Personalized_Data_Permissions_(PDP)/PDP_Policy_Autocreation

     

    @rhollander , take a look at this utility, may help you automate the process

    Jae Wilson
    Check out my 🎥 Domo Training YouTube Channel 👨‍💻

    **Say "Thanks" by clicking the ❤️ in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"