Can you filter data from input dataset in a MySQL Flow

FXM32
FXM32 Member
in Magic ETL

Hello,

I have a dataset that has data for the past 20 fiscal years. I use it in a flow that only looks at the last two fiscal years and when the flow runs it takes a good 10-20 minutes just to read the data in the dataset and then start transforming it.

I was thinking of expediting the loading process by pre-filtering the last 2 years only. I would have tried with dataset views but these are no accepted yet as input datasets in a flow.

I do not want to create another flow that will populate and additional dataset just with data for the past two years. I am looking for a solution that would leverage the existing dataset and put a filter on it.

Any suggestion is greatly appreciated.

Answers

  • GrantSmith
    GrantSmith Coach

    Are you using a MySQL dataflow or a Magic ETL data flow?

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • FXM32
    FXM32 Member

    Hi Grant,

    it is a MySQL dataflow

    Thank you

  • galfinito
    galfinito Member

    I have a similar question. I currently use a transform to filter my input dataset to the relevant rows. However this method is very slow (~5 minutes to run my dataflow).

  • GrantSmith
    GrantSmith Coach

    if you’re use a dataset view to profile your dataset and then feed it into a magic Etl dataflow that will decrease processing time. This won’t create a new dataset but have a virtual one as the input. This will allow for less data to be transferred to the Etl platform decreasing the amount of time needed to process the data

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**

Categories