Magic ETL - Using Group by - run time impact question

swagner
swagner Contributor
in Archive

Wondering what is the best practice to reduce overall run time of a ETL dataflow...

 

I've got 5 input datasets totalling about 14M rows of data that are ultimately appended together in the final result. 

 

Is it better to use group by in the magic ETL for each dataset (5 steps) or one consolidated step after the appends?

 

Also, are there certain items in Magic ETL that are run time bandits?

Comments

  • Chronos
    Chronos Member

    Is anyone able to help out with this request?

  • Gimli
    Gimli Contributor

    Hello @swagner, 

    It is hard to determine what might be causing long run times in an ETL without looking at the ETL in specific. 
    Generally, it is most efficient to use a select columns tile and select only the columns you need. 
    Next would be to filter your data down to only the rows you need. 
    After filtering your data to only the columns and rows that you need grouping your data will help reduce the size. 

    In regards to your group by question. Normally it will not make a difference between doing the group by's before the append vs after. If you can provide a screenshot of your ETL I can look to see if there are any other steps that we might be able to optimize. 

    **Say “Thanks" by clicking the thumbs up in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"

Categories