Filter within dataflow

TanyaB
TanyaB Member

Hi there - I'm working on a dataflow (ETL) and I'm having to filter on multiple values (1.5k). What is the best way to do this?

I've looked at the filter option, but it seems to be for a specific value.

And I've looked at importing the values as a dataset, joining the data and then adding a filter that way.

Thank you!

Answers

  • Could you share more information on what you are trying to filter?

    In Magic ETL, the filter tile gives two options:
    1. Add Filter Rule: user friendly option that allows you to click and select how you want to filter the data
    2. Add Formula Rule: This is my preferred option, as it allows for using SQL to filter your data. This is great if you have more complicated filtering needs.

    If you share a bit more on what your data looks like and how you are trying to filter it I could make a recommendation.

    If I solved your problem, please select "yes" above

  • TanyaB
    TanyaB Member

    Thank you - I see the Add Formula Rule now.

    For a list of values, what function should I use? i.e. Member NOT IN (5a6,12c,64o,10bc,132v……)

  • AndreaHenderson
    AndreaHenderson Domo Product Manager
    edited June 4

    If you're filtering on that number of values, it might be easiest to create a DataSet containing those values and use it as an input to the Magic ETL DataFlow, at which point you could use a join to filter on those values (inner join to filter to the list of values, or outer join and then subsequent filter to filter to the list of everything except the values). Alternatively, if the filter values follow some kind of definable pattern (like all starting with the same string, or all being within a certain range of ordinal values) then you could write a formula rule in the Filter tile, like Colemen is suggesting. Essentially, if you can express your filter rule using SQL, you can write a formula rule in the Filter tile to represent it.

    I wouldn't recommend trying to create a massive IN/NOT IN list of distinct values to filter on within the tile, that seems like it would be tedious and potentially error-prone.

    Domo Product Manager for Data Transformation (MagicETL)