Understanding PDP, DataSets, and DataFlows

DataSquirrel
DataSquirrel Contributor
in Archive

I've finally had a look at PDP as I need to retrofit a page with dozens of cards to filter the data on display for specific users and user groups. When I'm learning something new, I usually read the docs, try it out, and make junky little sketches. If I can't make a sketch, I definitely don't understand what I'm looking at. On the other, just because I can make a sketch it doesn't mean that the sketch is right. I figured this would be a good time to ask a few conceptual questions about PDP, DataSets and DataFlows and, hopefully, deepen my understanding of how Domo functions internally. 

 

First up a Terrible Little Diagram for a DataSet with one filtering PDP policy:

pdp_dataset.png

As I understand it, PDP policies apply to a specific DataSet, so they can really be thought of as a property of the DataSet. I illustrated it as above as 1) it makes more sense for a policy to stand on its own and 2) we'll end up making redundant policies as a matter of course as Domo doesn't support policy sharing. 

 

What I'm not sure I've gotten at all right is the box labelled "Virtual Filtered DataSet." Is that about right? I have the impression that a DataSet is rendered/indexed/stored within Domo and that a PDP filter is a runtime/dynamic operation. I got this idea after trying to map out the DataFlow behavior:

pdp_dataflow.png

 

According to the docs, a DataSet with a PDP policy cannot be used as an input to a DataFlow, but an output DataSet can have PDP policies attached. If DataFlow outputs are fully rendered DataSets that are saved and indexed, and PDP policies are only applied dynamically at runtime based on a specific user context, then this all kind of makes sense. Hence the "Virtual Filtered DataSet" in the diagram.

 

But I could have it all wrong, I have no idea. Does anyone know about these points:

 

  • Are DataSets parsed, stored, and indexed during import? 
  • Are DataFlow output DataSets parsed, stored, and indexed during DataFlow run/update?
  • Are PDP filters only applied at runtime? Meaning there's no serialized version stored in advanced?

If anyone's got some info or insight, I'd love to hear about it. I'm not sure that I can claim that the information will do me any specific good right now, but getting a more fully developed mental model of how Domo works will definitely help me going forward.

 

Thanks

 

 

Comments

  • ST_Superman
    ST_Superman Domo Employee

    Your idea of a virtual filtered dataset is fairly accurate.  I think of PDP as more of a redacted file.  You get your output dataset, you build your cards with it, and then PDP goes through and "hides" any data that a user does not have access to.  

     

    You can use a PDP enabled data set as an input dataset for a data flow.  However, the PDP rules will not be applied.  So, the dataset that gets included in the data flow would not have any data "hidden" unless you applied a new set of PDP rules to the output of the new data flow.  

     

    A data set with PDP policies activated on it is still a complete data set.  For example, if you have 100 PDP policies set up for a data set, Domo does not create 100 different versions of that data set.  Rather, it uses those policies when the User accesses the data to filter out any data they do not have permission to see.  And yes, very complex PDP policies can slow down an end users experience when using Domo.  It is best to try to structure your data in such a way to allow clean, simple PDP policies to be written.

  • DataSquirrel
    DataSquirrel Contributor

    Thanks for the thoughtful replyl. You image of a redacted file is great. This makes the process a whole lot simpler to explain to people than "filter" or "policy", etc.

     

    Since "complexity is the enemy of security", I'm taking your suggestion about structuring the data in advance to support simple rules. I'm going back now to retrofit a bunch of DataSets to include filter grouping columns. This way we'll have a lot of one-statement rules that don't need modification as we add new groups. For example, where we have Facility now we can easily add in Region/Division/etc. for the filters we need. In cases where we don't already have some kind of natural filter column, we can synthesize one in advance to simplify PDP.

     

    Thanks!

Categories