Append a dataset without using recursive design

Jessica

Is there any other way to append to a dataset without using a recursive design in Magic ETL?

Jessica

Thanks I found the append in DOMO Workbench. The basic csv connector from Workbench does not have a setting on the DOMO application side.

It's on the Workbench side.

Thanks again

MarkSnodgrass

@Jessica

You will probably need to provide more specifics as to what fields are in your dataset to work with, but generally I would say yes you can do this without a recursive.

In the ETL, you can split off the IDs and dates in your dataset and then group by max date. Then join it back to your main dataset and join on the date and id. This would give you the latest of each your unique rows.

Hope this helps.

GrantSmith

@Jessica

It depends if you're wanting to ignore duplicate records or not and how you're pulling in your data. If you're using Workbench you could utilize and UPSERT to overwrite existing records or an APPEND to just tack on the result set to the existing domo dataset.

Another option would be to utilize a dataset copy connector to take snapshots and set it to append but it could possibly have duplicate rows.

Most connectors also support APPEND as the update method.

jaeW_at_Onyx

Without hacking Domo, you can only APPEND or UPSERT data during ingest (not after ETLs have been constructed). That said, if you get crafty you can alter a dataflow to APPEND instead of REPLACE (but not UPSERT)

https://www.youtube.com/watch?v=YgJLijj7vmw&list=PLUy_qbtzH0S6-5oDbx3BsIv2Xk-JxJxWi&index=15

In this video i got the dataflow to APPEND but was unable to get it to UPSERT as desired.

Jessica

Thanks I found the append in DOMO Workbench. The basic csv connector from Workbench does not have a setting on the DOMO application side.

It's on the Workbench side.

Thanks again