Filter within dataflow
Hi there - I'm working on a dataflow (ETL) and I'm having to filter on multiple values (1.5k). What is the best way to do this?
I've looked at the filter option, but it seems to be for a specific value.
And I've looked at importing the values as a dataset, joining the data and then adding a filter that way.
Thank you!
Answers
-
Could you share more information on what you are trying to filter?
In Magic ETL, the filter tile gives two options:
1. Add Filter Rule: user friendly option that allows you to click and select how you want to filter the data
2. Add Formula Rule: This is my preferred option, as it allows for using SQL to filter your data. This is great if you have more complicated filtering needs.
If you share a bit more on what your data looks like and how you are trying to filter it I could make a recommendation.If I solved your problem, please select "yes" above
0 -
Thank you - I see the Add Formula Rule now.
For a list of values, what function should I use? i.e. Member NOT IN (5a6,12c,64o,10bc,132v……)
0 -
If you're filtering on that number of values, it might be easiest to create a DataSet containing those values and use it as an input to the Magic ETL DataFlow, at which point you could use a join to filter on those values (inner join to filter to the list of values, or outer join and then subsequent filter to filter to the list of everything except the values). Alternatively, if the filter values follow some kind of definable pattern (like all starting with the same string, or all being within a certain range of ordinal values) then you could write a formula rule in the Filter tile, like Colemen is suggesting. Essentially, if you can express your filter rule using SQL, you can write a formula rule in the Filter tile to represent it.
I wouldn't recommend trying to create a massive IN/NOT IN list of distinct values to filter on within the tile, that seems like it would be tedious and potentially error-prone.
Domo Product Manager for Data Transformation (MagicETL)
2
Categories
- All Categories
- Product Ideas
- 2.1K Ideas Exchange
- Connect
- 1.3K Connectors
- 309 Workbench
- 7 Cloud Amplifier
- 10 Federated
- Transform
- 664 Datasets
- 120 SQL DataFlows
- 2.3K Magic ETL
- 825 Beast Mode
- Visualize
- 2.6K Charting
- 88 App Studio
- 46 Variables
- Automate
- 195 Apps
- 486 APIs & Domo Developer
- 94 Workflows
- 24 Code Engine
- AI and Machine Learning
- 23 AI Chat
- 4 AI Projects and Models
- 18 Jupyter Workspaces
- Distribute
- 119 Domo Everywhere
- 283 Scheduled Reports
- 11 Software Integrations
- Manage
- 143 Governance & Security
- 11 Domo Community Gallery
- 49 Product Releases
- 13 Domo University
- Community Forums
- 41 Getting Started
- 31 Community Member Introductions
- 116 Community Announcements
- 5K Archive