Trying to filter out rows, where a certain column has NULL values
Hi,
I have a dataset of almost 2 billion rows. I need to filter out rows, where a certain column has NULL values.
What is the best and fastest way to do it?
I tried creating a DataFlow using the mySQL option, but the flow has been running for more than 24 hours now.
Could any of you please help in understanding how this process can be made to run faster?
Comments
-
With large datasets, I normally have more luck (where speed is concerned) using the ETL tool.
You could accomplish what you're looking for using 3 widgets, Input Dataset into Filter Rows into Output Dataset.
On the filter rows, just select the row you want to filter and there is an option for IS NOT NULL.
That should hopefully be all you need. If not, just let me know.
Sincerely,
Valiant
**Please mark "Accept as Solution" if this post solves your problem
**Say "Thanks" by clicking the "heart" in the post that helped you.2 -
Are you needing to filter those rows out completely, or do you want to improve your filtering ability in your visualizations? Is this a text field or numeric?
If you want to keep the rows, but need the null values filled in, you can use the mapper. This works for both text and numeric fields.
If you are simply needing to remove them efficiently, @Valiant 's solution is perfect.
DataMaven
Breaking Down Silos - Building Bridges
**Say "Thanks" by clicking a reaction in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"1
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.6K Connect
- 1.2K Connectors
- 300 Workbench
- 6 Cloud Amplifier
- 9 Federated
- 2.9K Transform
- 101 SQL DataFlows
- 622 Datasets
- 2.2K Magic ETL
- 3.9K Visualize
- 2.5K Charting
- 748 Beast Mode
- 59 App Studio
- 41 Variables
- 686 Automate
- 176 Apps
- 453 APIs & Domo Developer
- 47 Workflows
- 10 DomoAI
- 36 Predict
- 15 Jupyter Workspaces
- 21 R & Python Tiles
- 396 Distribute
- 113 Domo Everywhere
- 276 Scheduled Reports
- 7 Software Integrations
- 125 Manage
- 122 Governance & Security
- 8 Domo Community Gallery
- 38 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 109 Community Announcements
- 4.8K Archive