Use recursive dataflows to update datasets

user02464

We have 20 to 30 massive tables (+40 million rows) that need to be updated. I am wondering whether a recursive dataflow could be an alternative to update all these datasets. I have researched the option of recursive dataflows, but it seems to me that it is a two step process (first, add the input datasets  to create an output dataset, then add it as an input and remove the original dataset), so my question is whether this can run automatically as a single dataflow, so that we do not need to manually update the dataset. What I believe it requires is running the ETL process first to create the output dataset, and then again to append the historical with the updated data. Is this correct?

 

Does this mean that we have to manually run two ETL processes to update the datasets or can we make this process automatic? Thanks in advance!

Valiant

Hi,

 

This was topic that was brought up a while back and had some really thorough explanations from Domo reps. Check out this link:

https://dojo.domo.com/t5/Beast-Mode-ETL-Dataflow/How-can-I-add-only-unique-values-from-one-dataset-to-another/td-p/16549

 

They do a better job detailing it than I would.

 

Hope this helps,

ValiantSpur

 

**Please mark "Accept as Solution" if this post solves your problem
**Say "Thanks" by clicking the "heart" in the post that helped you.