Best practice for removing historical values

renee12345

I have a snapshot dataflow that has captured bad values. I only need to filter out those values that don't match the current format. What is the best practice for this?

DavidChurchman

You can add multiple conditions to a filter tile, then there is the option of "all" or "any" of the rules to be true. Basically, logical "AND" or logical "OR". If it's more complex than that, you can create a "formula rule" within the filter tile. KB article about filter formula

ColemenWilson

Yep, you would do what @DavidChurchman said and set each of those 3 conditions as conditions in the filter with "Include rows that meet ALL of the following rules" set.

ColemenWilson

Can you search for the rows that don't match the current format in a dynamic way? If so you could just apply a filter tile to the ETL to remove the rows that don't match. How do you identify the bad rows?

renee12345

@colemenwilson do you know the best way to utilize the filter formula tile to exclude multiple values? ex:

column1 != 'one'

column1 != 'two'

column1 != 'three'

Is there a way to do this in one formula?

DavidChurchman

You can add multiple conditions to a filter tile, then there is the option of "all" or "any" of the rules to be true. Basically, logical "AND" or logical "OR". If it's more complex than that, you can create a "formula rule" within the filter tile. KB article about filter formula

ColemenWilson

Yep, you would do what @DavidChurchman said and set each of those 3 conditions as conditions in the filter with "Include rows that meet ALL of the following rules" set.