Data Flows - Testing Use Case Filters

StevenJRowe

Use Case:

When building data flows with data sets in millions of rows, it's hard to test if the logic works across those rows unless I add filter tiles that filter out data at specific points during the Data Flow build process. This can add complexity when adding many test cases at specific points.

The Idea:

It would be ideal to have a drop-down next to the preview button to add a predefined series of filters based on the input data sets. So you would select a dropdown that contains the list of the data inputs you have brought into the Data Flow, and then you can build your filters of those inputs and save them as "Test Case - Northern America Only" for example. Then, when updates are made to that data flow in the future, you can re-use those testing filters.

EdEngalan

if I’m reading this correctly, it could be a test case management feature added to magic and each test case could output a dataset that you can then leverage for a larger monitoring and compliance effort? Pretty cool if I understand you correctly

akeating

Hi Steven,

I'm not sure if it would be helpful in your exact use case, but I often create dataset views of my input datasets filtered down to my test rows. I use those dataset views as my inputs until my dataflow produces the results I want, and then I change the inputs to use the real datasets.

That way, I don't have to complicate the view in Magic, but I can still test with the specific rows I need.