Copying dataset pdp settings

Josh-REO

Is there a way to copy pdp settings between datasets?

WHM

I scripted it using PowerShell and the API... All you need to add is the access token and the dataset ids. there are comments in the code from past projects and it is not perfect but it worked for our purpose. We have thousands of PDP rules and the excel pugin bogs down. There is also a PDP utility but you have to create a dataset of PDP rules... This was easier.

 

function Get-Dataset($datasetId, $Authentication) {
#Write-Host $Authentication
$headers = @{
Accept = 'text/csv'
Authorization = $Authentication.access_token
}
#Write-Host "https://api.domo.com/v1/datasets/$datasetId/data?includeHeader=true&fileName=dump.csv" -Headers $headers
try {
$csv = Invoke-RestMethod -Method Get -Uri "https://api.domo.com/v1/datasets/$datasetId/data?includeHeader=true&fileName=dump.csv" -Headers $headers
$csv = $csv -replace '\\N', $null
$data = ConvertFrom-Csv $csv
#Write-Host "must have Successfully Read CSV File"
}
catch {
Write-Host "Failed to Read Dataset CSV File"
Write-Host $_.ErrorDetails
Write-Host $_.Exception
Write-Host $_.FullyQualifiedErrorId
Start-Sleep -Seconds 10
}
return $data
}

function Get-Dataset-Object($datasetId, $Authentication) {
#Write-Host $Authentication
$NeedData = $true
While( $NeedData -eq $true -and $Cnct_Cntr -lt 100){
try {
$headers = @{
Accept = 'application/json'
Authorization = $Authentication.access_token
}
#Write-Host "https://api.domo.com/v1/datasets/$datasetId" -Headers $headers
$data = Invoke-RestMethod -Method Get -Uri "https://api.domo.com/v1/datasets/$datasetId" -Headers $headers
#Write-Host "must have Successfully Read CSV File"
$Cnct_Cntr++
$NeedData = $false
}
catch {
Write-Host "Failed to get dataset object"
Write-Host $_.ErrorDetails
Write-Host $_.Exception
Write-Host $_.FullyQualifiedErrorId
$Cnct_Cntr++
Write-Host $Cnct_Cntr
Start-Sleep -Seconds 10
}
}
return $data
}

function Get-Dataset-PDP($datasetId, $Authentication) {
#Write-Host $Authentication
$NeedData = $true
While( $NeedData -eq $true -and $Cnct_Cntr -lt 100){
try {
$headers = @{
Accept = 'application/json'
Authorization = $Authentication.access_token
}
#Write-Host "https://api.domo.com/v1/datasets/$datasetId" -Headers $headers
$data = Invoke-RestMethod -Method Get -Uri "https://api.domo.com/v1/datasets/$datasetId/policies" -Headers $headers
#Write-Host "must have Successfully Read CSV File"
$Cnct_Cntr++
$NeedData = $false
}
catch {
Write-Host "Failed to get dataset object"
Write-Host $_.ErrorDetails
Write-Host $_.Exception
Write-Host $_.FullyQualifiedErrorId
$Cnct_Cntr++
Write-Host $Cnct_Cntr
Start-Sleep -Seconds 10
}
}
return $data
}

function Get-Authorization($clientID, $clientSecret) {
$basicAuth = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $clientID, $clientSecret)));
$authenticationHeader = @{
Authorization = ("Basic {0}" -f $basicAuth)
}
$NeedToken = $true
$Cnct_Cntr = 0

#keep trying to get a token
While( $NeedToken -eq $true -and $Cnct_Cntr -lt 100){
try {
$data = Invoke-RestMethod -Method Post -Uri "https://api.domo.com/oauth/token?grant_type=client_credentials&scope=data%20user" -Headers $authenticationHeader
# $data = Invoke-RestMethod -Method Post -Uri "https://api.domo.com/oauth/token?grant_type=client_credentials&scope=full" -Headers $authenticationHeader
$data.access_token = "bearer $($data.access_token)"
# Write-Host "Got Token"
$Cnct_Cntr++
$NeedToken = $false
}
catch {
#$ErrorActionPreference = Stop #this does not recognize "Stop"
Write-Host "Failed to get Token"
Write-Host $_.ErrorDetails
Write-Host $_.Exception
Write-Host $_.FullyQualifiedErrorId
$Cnct_Cntr++
Write-Host $Cnct_Cntr
Start-Sleep -Seconds 10
}
}
return $data
}

function Send-ToEmail([string]$email, [string]$body){

$message = new-object Net.Mail.MailMessage;
$message.From = "Domo@daltile.com";
$message.To.Add($email);
$message.Subject = "Error Running Vistex jobs Script";
$message.Body = $body;
$smtp = new-object Net.Mail.SmtpClient("smtpmail.mohawkind.com");
$smtp.send($message);
write-host "Mail Sent" ;
}

Write-Host "go get auth token"
$Authentication = Get-Authorization -clientID $apiKey -clientSecret $apiToken
Write-Host "have token"
#Write-Host $Authentication
#$MetData = Get-Dataset -datasetId $datasetmetadata -Authentication $Authentication

$output = ''
$haserr = $true
$Have_PDP = $false
#get the list of WB jobs
while(!$Have_PDP){
try{
$Authentication = Get-Authorization -clientID $apiKey -clientSecret $apiToken
$DS_PDP = Get-Dataset-PDP -datasetId $PDPSourceDatasetID -Authentication $Authentication
$Have_PDP = $true
}
catch{
Write-Host "Failed to get dataset list of PDP rules"
Write-Host $_.ErrorDetails
Write-Host $_.Exception
Write-Host $_.FullyQualifiedErrorId
Start-Sleep -Seconds 10
}
}
# Cycle through the list of jobs ensuring they run successfully.

 

$PDPs = $DS_PDP.Where{$_."name" -like "z_rep_*" -or $_."name" -like "z_ssc_*" } #| Sort-Object -Property $_."Execution Order"| Sort-Object -Property $_."Execution Order"
foreach($PDP in $PDPs){
$haserr = $true
$output = ''
$msg = ''
$Count = 0

Try{
$hdrs = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$hdrs.Add("Content-Type", "application/json")
$hdrs.Add("Accept", 'application/json')
$hdrs.Add("Authorization", $Authentication.access_token)
$filter = $PDP.filters
$name = $PDP.name
$type = $PDP.type
$users = $PDP.users
$groups = $PDP.groups

$i = 0
$pdp_rule = '{ "name": "'+$name+'",'
$fString = '"filters": ['
foreach($f in $PDP.filters){
$fString += '{ "column": "'+$f.column+'", "values": [ "'+$f.values+'" ], "operator": "'+$f.operator+'" }'
$i++
if($i -lt $PDP.filters.count){$fString += ','}
}

$pdp_rule += $fString +' ], "groups": [ "'+$groups+'" ] }'
# {
# "name": "z_ssc_4000",
# "filters": [ {
# "column": "business_area_number",
# "values": [ "4000" ],
# "operator": "EQUALS"
# } ],
# "groups": [ 612003576 ]
#}

$post = Invoke-WebRequest -Method POST -Uri "https://api.domo.com/v1/datasets/$DestinationDatasetId/policies" -UseBasicParsing -Body $pdp_rule -Headers $hdrs -ContentType 'application/json'

Write-Host "updated $name in Proex dataset PDP "
}

catch {
Write-Host "Failed to add $email to Domo"
Write-Host $_.ErrorDetails
Write-Host $_.Exception
Write-Host $_.FullyQualifiedErrorId

}

}#end foreach job

$now = Get-Date -Format g
Write-Host "$now Done."

AS

There isn't a process currently place through the interface that lets you do this.  I think there's a post on the Ideas Exchange for it, though.  Ideally, I think, there would be a list of PDPs stored independent of datasets, and you could apply a group of policies to datasets at the click of a button.

 

That said, we're using a not-so-official software application that Domo skunkworks wrote that lets us apply policies across different datasets using a config file hosted in Domo via a webform dataset.  It's saved us a ton of time and serves the same purpose.  You should inquire with your success manager or rep about that.

Unknown

I was trying to search for this ap but couldn't. Is there anyway that I can access and use it to copy over PDP to differnt data set?

AS

I'm sure this application isn't available to download.  You'll have to ask Domo directly.

I'd also watch out in the coming months to see if something comes into the product.  I wouldn't be surprised to see something at Domopalooza, for example.

Josh-REO

I'm revisiting this because I've continued to builed datasets that have the exact same PDP settings as previous datasets, and I still don't see a way to copy the settings from one dataset to another.  

 

Does this exist or do I have to continue recreating the wheel on every dataset?

Randyb

Have you tried the Excel plugin for DOMO, this allows you to apply PDP security to any dataset you want and is a fast and easy way to apply PDP policies that are exact copies to other datasets.

Just change the DatasetID and apply and your good.

This is also very easy to make slight modifications to PDP's that are close but not the same.

It is also an easy way to manage other components such as user accounts etc.

Randy

Josh-REO

I will request the admin version of that.

 

Thanks.

Randyb

You should be able to download it from the Admin/Tools Download section.

Randy

Josh-REO

The admin one requires a special request, but I did get.  It's not quite the elegant solution I was hoping for within DOMO, but it will work for when I have to add bulk permissions.  

 

WHM

I scripted it using PowerShell and the API... All you need to add is the access token and the dataset ids. there are comments in the code from past projects and it is not perfect but it worked for our purpose. We have thousands of PDP rules and the excel pugin bogs down. There is also a PDP utility but you have to create a dataset of PDP rules... This was easier.

 

function Get-Dataset($datasetId, $Authentication) {
#Write-Host $Authentication
$headers = @{
Accept = 'text/csv'
Authorization = $Authentication.access_token
}
#Write-Host "https://api.domo.com/v1/datasets/$datasetId/data?includeHeader=true&fileName=dump.csv" -Headers $headers
try {
$csv = Invoke-RestMethod -Method Get -Uri "https://api.domo.com/v1/datasets/$datasetId/data?includeHeader=true&fileName=dump.csv" -Headers $headers
$csv = $csv -replace '\\N', $null
$data = ConvertFrom-Csv $csv
#Write-Host "must have Successfully Read CSV File"
}
catch {
Write-Host "Failed to Read Dataset CSV File"
Write-Host $_.ErrorDetails
Write-Host $_.Exception
Write-Host $_.FullyQualifiedErrorId
Start-Sleep -Seconds 10
}
return $data
}

function Get-Dataset-Object($datasetId, $Authentication) {
#Write-Host $Authentication
$NeedData = $true
While( $NeedData -eq $true -and $Cnct_Cntr -lt 100){
try {
$headers = @{
Accept = 'application/json'
Authorization = $Authentication.access_token
}
#Write-Host "https://api.domo.com/v1/datasets/$datasetId" -Headers $headers
$data = Invoke-RestMethod -Method Get -Uri "https://api.domo.com/v1/datasets/$datasetId" -Headers $headers
#Write-Host "must have Successfully Read CSV File"
$Cnct_Cntr++
$NeedData = $false
}
catch {
Write-Host "Failed to get dataset object"
Write-Host $_.ErrorDetails
Write-Host $_.Exception
Write-Host $_.FullyQualifiedErrorId
$Cnct_Cntr++
Write-Host $Cnct_Cntr
Start-Sleep -Seconds 10
}
}
return $data
}

function Get-Dataset-PDP($datasetId, $Authentication) {
#Write-Host $Authentication
$NeedData = $true
While( $NeedData -eq $true -and $Cnct_Cntr -lt 100){
try {
$headers = @{
Accept = 'application/json'
Authorization = $Authentication.access_token
}
#Write-Host "https://api.domo.com/v1/datasets/$datasetId" -Headers $headers
$data = Invoke-RestMethod -Method Get -Uri "https://api.domo.com/v1/datasets/$datasetId/policies" -Headers $headers
#Write-Host "must have Successfully Read CSV File"
$Cnct_Cntr++
$NeedData = $false
}
catch {
Write-Host "Failed to get dataset object"
Write-Host $_.ErrorDetails
Write-Host $_.Exception
Write-Host $_.FullyQualifiedErrorId
$Cnct_Cntr++
Write-Host $Cnct_Cntr
Start-Sleep -Seconds 10
}
}
return $data
}

function Get-Authorization($clientID, $clientSecret) {
$basicAuth = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $clientID, $clientSecret)));
$authenticationHeader = @{
Authorization = ("Basic {0}" -f $basicAuth)
}
$NeedToken = $true
$Cnct_Cntr = 0

#keep trying to get a token
While( $NeedToken -eq $true -and $Cnct_Cntr -lt 100){
try {
$data = Invoke-RestMethod -Method Post -Uri "https://api.domo.com/oauth/token?grant_type=client_credentials&scope=data%20user" -Headers $authenticationHeader
# $data = Invoke-RestMethod -Method Post -Uri "https://api.domo.com/oauth/token?grant_type=client_credentials&scope=full" -Headers $authenticationHeader
$data.access_token = "bearer $($data.access_token)"
# Write-Host "Got Token"
$Cnct_Cntr++
$NeedToken = $false
}
catch {
#$ErrorActionPreference = Stop #this does not recognize "Stop"
Write-Host "Failed to get Token"
Write-Host $_.ErrorDetails
Write-Host $_.Exception
Write-Host $_.FullyQualifiedErrorId
$Cnct_Cntr++
Write-Host $Cnct_Cntr
Start-Sleep -Seconds 10
}
}
return $data
}

function Send-ToEmail([string]$email, [string]$body){

$message = new-object Net.Mail.MailMessage;
$message.From = "Domo@daltile.com";
$message.To.Add($email);
$message.Subject = "Error Running Vistex jobs Script";
$message.Body = $body;
$smtp = new-object Net.Mail.SmtpClient("smtpmail.mohawkind.com");
$smtp.send($message);
write-host "Mail Sent" ;
}

Write-Host "go get auth token"
$Authentication = Get-Authorization -clientID $apiKey -clientSecret $apiToken
Write-Host "have token"
#Write-Host $Authentication
#$MetData = Get-Dataset -datasetId $datasetmetadata -Authentication $Authentication

$output = ''
$haserr = $true
$Have_PDP = $false
#get the list of WB jobs
while(!$Have_PDP){
try{
$Authentication = Get-Authorization -clientID $apiKey -clientSecret $apiToken
$DS_PDP = Get-Dataset-PDP -datasetId $PDPSourceDatasetID -Authentication $Authentication
$Have_PDP = $true
}
catch{
Write-Host "Failed to get dataset list of PDP rules"
Write-Host $_.ErrorDetails
Write-Host $_.Exception
Write-Host $_.FullyQualifiedErrorId
Start-Sleep -Seconds 10
}
}
# Cycle through the list of jobs ensuring they run successfully.

 

$PDPs = $DS_PDP.Where{$_."name" -like "z_rep_*" -or $_."name" -like "z_ssc_*" } #| Sort-Object -Property $_."Execution Order"| Sort-Object -Property $_."Execution Order"
foreach($PDP in $PDPs){
$haserr = $true
$output = ''
$msg = ''
$Count = 0

Try{
$hdrs = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$hdrs.Add("Content-Type", "application/json")
$hdrs.Add("Accept", 'application/json')
$hdrs.Add("Authorization", $Authentication.access_token)
$filter = $PDP.filters
$name = $PDP.name
$type = $PDP.type
$users = $PDP.users
$groups = $PDP.groups

$i = 0
$pdp_rule = '{ "name": "'+$name+'",'
$fString = '"filters": ['
foreach($f in $PDP.filters){
$fString += '{ "column": "'+$f.column+'", "values": [ "'+$f.values+'" ], "operator": "'+$f.operator+'" }'
$i++
if($i -lt $PDP.filters.count){$fString += ','}
}

$pdp_rule += $fString +' ], "groups": [ "'+$groups+'" ] }'
# {
# "name": "z_ssc_4000",
# "filters": [ {
# "column": "business_area_number",
# "values": [ "4000" ],
# "operator": "EQUALS"
# } ],
# "groups": [ 612003576 ]
#}

$post = Invoke-WebRequest -Method POST -Uri "https://api.domo.com/v1/datasets/$DestinationDatasetId/policies" -UseBasicParsing -Body $pdp_rule -Headers $hdrs -ContentType 'application/json'

Write-Host "updated $name in Proex dataset PDP "
}

catch {
Write-Host "Failed to add $email to Domo"
Write-Host $_.ErrorDetails
Write-Host $_.Exception
Write-Host $_.FullyQualifiedErrorId

}

}#end foreach job

$now = Get-Date -Format g
Write-Host "$now Done."

rhollander

Hi @Randyb , can you provide some more information on how you accomplished this using the Excel plugin?  I've got a massive PDP update project I need to undertake, and this sounds like a great option for us.  Are you exporting a specific dataset into Excel then re-uploading with changes, or something else?

 

Thanks!

 

-Russell 

Randyb

@rhollander There are a couple of ways you can do this, you can download the excel plugin which then allows you to pull the data down along with the different PDP profiles. You can then build essentially a user list for a specific PDPPolicy and upload it.

Its been awhile since I've had to use it but it does make large scale PDP policy setups fairly easy.

If you need to make changes that don't involve creating PDP policies you can also pull down the entire user list from DOMO in excel format and update access etc in the sheet and then re upload that in bulk. Another quick way to reassign access to entire pages etc.

 

Randy

jaeW_at_Onyx

https://knowledge.domo.com/Connect/Personalized_Data_Permissions_(PDP)/PDP_Policy_Autocreation

 

@rhollander , take a look at this utility, may help you automate the process