Can you filter data from input dataset in a MySQL Flow
Hello,
I have a dataset that has data for the past 20 fiscal years. I use it in a flow that only looks at the last two fiscal years and when the flow runs it takes a good 10-20 minutes just to read the data in the dataset and then start transforming it.
I was thinking of expediting the loading process by pre-filtering the last 2 years only. I would have tried with dataset views but these are no accepted yet as input datasets in a flow.
I do not want to create another flow that will populate and additional dataset just with data for the past two years. I am looking for a solution that would leverage the existing dataset and put a filter on it.
Any suggestion is greatly appreciated.
Answers
-
Are you using a MySQL dataflow or a Magic ETL data flow?
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**0 -
Hi Grant,
it is a MySQL dataflow
Thank you
0 -
I have a similar question. I currently use a transform to filter my input dataset to the relevant rows. However this method is very slow (~5 minutes to run my dataflow).
0 -
if you’re use a dataset view to profile your dataset and then feed it into a magic Etl dataflow that will decrease processing time. This won’t create a new dataset but have a virtual one as the input. This will allow for less data to be transferred to the Etl platform decreasing the amount of time needed to process the data
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**0
Categories
- All Categories
- 1.7K Product Ideas
- 1.7K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 292 Workbench
- 4 Cloud Amplifier
- 8 Federated
- 2.8K Transform
- 95 SQL DataFlows
- 603 Datasets
- 2.1K Magic ETL
- 3.7K Visualize
- 2.4K Charting
- 697 Beast Mode
- 43 App Studio
- 39 Variables
- 658 Automate
- 170 Apps
- 441 APIs & Domo Developer
- 42 Workflows
- 5 DomoAI
- 32 Predict
- 12 Jupyter Workspaces
- 20 R & Python Tiles
- 386 Distribute
- 111 Domo Everywhere
- 269 Scheduled Reports
- 6 Software Integrations
- 113 Manage
- 110 Governance & Security
- 8 Domo University
- 30 Product Releases
- Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 98 Community Announcements
- Domo Community Gallery
- 4.8K Archive