Best practice for removing historical values

I have a snapshot dataflow that has captured bad values. I only need to filter out those values that don't match the current format. What is the best practice for this?

Best Answers

  • DavidChurchman
    DavidChurchman Contributor
    Answer ✓

    You can add multiple conditions to a filter tile, then there is the option of "all" or "any" of the rules to be true. Basically, logical "AND" or logical "OR". If it's more complex than that, you can create a "formula rule" within the filter tile. KB article about filter formula

  • ColemenWilson
    Answer ✓

    Yep, you would do what @DavidChurchman said and set each of those 3 conditions as conditions in the filter with "Include rows that meet ALL of the following rules" set.

    If I solved your problem, please "accept" my answer as the solution

Answers

  • Can you search for the rows that don't match the current format in a dynamic way? If so you could just apply a filter tile to the ETL to remove the rows that don't match. How do you identify the bad rows?

    If I solved your problem, please "accept" my answer as the solution

  • @colemenwilson do you know the best way to utilize the filter formula tile to exclude multiple values? ex:

    column1 != 'one'

    column1 != 'two'

    column1 != 'three'

    Is there a way to do this in one formula?

  • DavidChurchman
    DavidChurchman Contributor
    Answer ✓

    You can add multiple conditions to a filter tile, then there is the option of "all" or "any" of the rules to be true. Basically, logical "AND" or logical "OR". If it's more complex than that, you can create a "formula rule" within the filter tile. KB article about filter formula

  • ColemenWilson
    Answer ✓

    Yep, you would do what @DavidChurchman said and set each of those 3 conditions as conditions in the filter with "Include rows that meet ALL of the following rules" set.

    If I solved your problem, please "accept" my answer as the solution