Change Certification from Datasets to Dataflows

Craig_Lynch

Certification is currently requested and managed via a dataset, however we've found that in some cases changes to the dataflow don't cause the certification to expire. This introduces potential risk in cases where changes could be made that affect the accuracy of the data, but the dataset could still be flagged as Certified.

Also, although it's a minor issue, there's no indication on a dataflow that an output dataset is certified or not. So a developer could mistakenly update a dataflow breaking certification.

Additional rules could be put in place from a DOMO technical perspective, however I believe a better option would be to move the certification to the dataflow.

By certifying the dataflow:

• The actual object that governs the data accuracy/quality in the dataset/s is certified
• This could enable a visual reminder within the ETL to a developer they are editing a certified object
• The output dataset/s could inherit the certification from the dataflow
• If certification is broken/expired, this could then result in the certification of the output dataset/s expiring
• Better management of the certification would be enabled through the versions of the dataflow, as soon as a new version is created, certification should break/expire
• It could also enable quick rectification of accidental certification expiration by reverting to the prior certified version of the dataflow

This next part is possibly a stretch, and would undoubtedly require some expansion to the metadata of datasets/dataflows, but this could also enable the connection of certification between multiple datasets within a lineage.

For example:

• dataflow B is certified using certified dataset A as a source
• someone breaks the certification on dataflow A
• dataflow B certification then breaks, triggering an alert to that dataflow owner/s and subsequently removing certification from dataset B

This would enable proactive management of full dataset lineages, encourage certification to be applied/broken taking downstream effects into consideration, and encourage the use of certified objects more due to the added governance this brings.