Upsert

Stu5677
Stu5677 Member
in Magic ETL

Has anyone started to switch their recursive dataflows to use the new Upsert function? Are you still planning to build recursive dataflows? I have a few but going back and fourth on if switching is worth it/is it best practice?

Best Answers

  • GrantSmith
    GrantSmith Coach
    Answer ✓

    Converting to the new upsert will allow your dataflow to run quicker, as you'll only pull in the data that you're updating instead of the entire dataflow. With a recursive dataflow, it starts to run slower over time as it needs to pull in the ever-increasing dataset.

    It also simplifies the logic of your data flow.

    I'd recommend switching over.

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • ggenovese
    ggenovese Contributor
    Answer ✓

    I feel that there is still a use case for recursive dataflows when your data does not have a unique identifier. In order to perform and UPSERT operation you need to have a unique key, but a recursive doesn't have this limitation.

Answers

  • GrantSmith
    GrantSmith Coach
    Answer ✓

    Converting to the new upsert will allow your dataflow to run quicker, as you'll only pull in the data that you're updating instead of the entire dataflow. With a recursive dataflow, it starts to run slower over time as it needs to pull in the ever-increasing dataset.

    It also simplifies the logic of your data flow.

    I'd recommend switching over.

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • ggenovese
    ggenovese Contributor
    Answer ✓

    I feel that there is still a use case for recursive dataflows when your data does not have a unique identifier. In order to perform and UPSERT operation you need to have a unique key, but a recursive doesn't have this limitation.

Categories