Trying to filter out rows, where a certain column has NULL values
Hi,
I have a dataset of almost 2 billion rows. I need to filter out rows, where a certain column has NULL values.
What is the best and fastest way to do it?
I tried creating a DataFlow using the mySQL option, but the flow has been running for more than 24 hours now.
Could any of you please help in understanding how this process can be made to run faster?
Comments
-
With large datasets, I normally have more luck (where speed is concerned) using the ETL tool.
You could accomplish what you're looking for using 3 widgets, Input Dataset into Filter Rows into Output Dataset.
On the filter rows, just select the row you want to filter and there is an option for IS NOT NULL.
That should hopefully be all you need. If not, just let me know.
Sincerely,
Valiant
**Please mark "Accept as Solution" if this post solves your problem
**Say "Thanks" by clicking the "heart" in the post that helped you.2 -
Are you needing to filter those rows out completely, or do you want to improve your filtering ability in your visualizations? Is this a text field or numeric?
If you want to keep the rows, but need the null values filled in, you can use the mapper. This works for both text and numeric fields.
If you are simply needing to remove them efficiently, @Valiant 's solution is perfect.
DataMaven
Breaking Down Silos - Building Bridges
**Say "Thanks" by clicking a reaction in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"1
Categories
- 10.5K All Categories
- 3 Connect
- 913 Connectors
- 250 Workbench
- 458 Transform
- 1.7K Magic ETL
- 69 SQL DataFlows
- 476 Datasets
- 183 Visualize
- 249 Beast Mode
- 2.1K Charting
- 11 Variables
- 16 Automate
- 354 APIs & Domo Developer
- 88 Apps
- 3 Workflows
- 20 Predict
- 5 Jupyter Workspaces
- 15 R & Python Tiles
- 245 Distribute
- 62 Domo Everywhere
- 242 Scheduled Reports
- 20 Manage
- 41 Governance & Security
- 168 Product Ideas
- 1.2K Ideas Exchange
- 9 Community Forums
- 27 Getting Started
- 14 Community Member Introductions
- 55 Community News
- 4.5K Archive