when should we use datafusion over etl data flows

karthik
karthik Member
edited March 2023 in Datasets

hi,

when should we use data fusion over ETL data flows and what are the differences between them.

thank you

Best Answer

  • AS
    AS Coach
    Answer ✓

    Datafusion is primarily used for the simplest of use cases when you want to just join or union datasets together.  It also runs VERY fast.

    ETL has much more functionality and is better for most use cases involving conditional logic, column filtering, custom calculations, grouping, filtering, formatting, etc. 

    Aaron
    MajorDomo @ Merit Medical

    **Say "Thanks" by clicking the heart in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"

Answers

  • AS
    AS Coach
    Answer ✓

    Datafusion is primarily used for the simplest of use cases when you want to just join or union datasets together.  It also runs VERY fast.

    ETL has much more functionality and is better for most use cases involving conditional logic, column filtering, custom calculations, grouping, filtering, formatting, etc. 

    Aaron
    MajorDomo @ Merit Medical

    **Say "Thanks" by clicking the heart in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"

Categories