Subquery versus Output Table

cwolman
cwolman
in Magic ETL

I am always looking to optimize my dataflows and was curious if it was better to have multiple output tables in a dataflow or less output tables by utilizing subqueries?  For debugging and readability it is nice to have multiple output tables but I do not want to do this if it will have a negative impact on performance.

 

Thanks in advance for your responses.


-----------------
Chris

Best Answer

  • PodiumMason
    PodiumMason
    Answer ✓

    hey @cwolman!

     

    For larger datasets at least, having multiple transforms is a good way to go, in my experience subqueries seem to bog down a bit for the larger sizes. 

     

    For smaller datasets, do it either way, whatever works best for you.

     

    Seems like you're on the right track!

    **Say 'Thanks' by clicking the thumbs up in the post that helped you.
    **Please mark the post that solves your problem as 'Accepted Solution'

Answers

  • PodiumMason
    PodiumMason
    Answer ✓

    hey @cwolman!

     

    For larger datasets at least, having multiple transforms is a good way to go, in my experience subqueries seem to bog down a bit for the larger sizes. 

     

    For smaller datasets, do it either way, whatever works best for you.

     

    Seems like you're on the right track!

    **Say 'Thanks' by clicking the thumbs up in the post that helped you.
    **Please mark the post that solves your problem as 'Accepted Solution'

Categories