Upsert

Stu5677

Has anyone started to switch their recursive dataflows to use the new Upsert function? Are you still planning to build recursive dataflows? I have a few but going back and fourth on if switching is worth it/is it best practice?

GrantSmith

Converting to the new upsert will allow your dataflow to run quicker, as you'll only pull in the data that you're updating instead of the entire dataflow. With a recursive dataflow, it starts to run slower over time as it needs to pull in the ever-increasing dataset.

It also simplifies the logic of your data flow.

I'd recommend switching over.

ggenovese

I feel that there is still a use case for recursive dataflows when your data does not have a unique identifier. In order to perform and UPSERT operation you need to have a unique key, but a recursive doesn't have this limitation.

GrantSmith

Converting to the new upsert will allow your dataflow to run quicker, as you'll only pull in the data that you're updating instead of the entire dataflow. With a recursive dataflow, it starts to run slower over time as it needs to pull in the ever-increasing dataset.

It also simplifies the logic of your data flow.

I'd recommend switching over.

ggenovese

I feel that there is still a use case for recursive dataflows when your data does not have a unique identifier. In order to perform and UPSERT operation you need to have a unique key, but a recursive doesn't have this limitation.