Upsert

Stu5677
Stu5677 Member
in Magic ETL

Has anyone started to switch their recursive dataflows to use the new Upsert function? Are you still planning to build recursive dataflows? I have a few but going back and fourth on if switching is worth it/is it best practice?

Answers

  • GrantSmith
    GrantSmith Coach

    Converting to the new upsert will allow your dataflow to run quicker, as you'll only pull in the data that you're updating instead of the entire dataflow. With a recursive dataflow, it starts to run slower over time as it needs to pull in the ever-increasing dataset.

    It also simplifies the logic of your data flow.

    I'd recommend switching over.

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • ggenovese
    ggenovese Contributor

    I feel that there is still a use case for recursive dataflows when your data does not have a unique identifier. In order to perform and UPSERT operation you need to have a unique key, but a recursive doesn't have this limitation.

Categories