Connect and store dataset

Options
Winnie_1234
Winnie_1234 Member
in Workbench

hi, i need to connect to sap via odbc using workbench to get 530 millions data. these data must be stored in domo for data viz. they are range between 2023 until 2025 (still growing daily).

i came across multiple methods online but not sure if they do work/or if it's the best way: append? partition? upsert? workflow for automation? recursive dataflow? also to note that data in 2023 and 2024 are static, but data 2025 might still have updates, i just need the latest. if possible, could we also subtly detect if there're changes in 2023 and 2024 data, and replace accordingly? pls advise

Answers

  • GrantSmith
    GrantSmith Coach

    I'd recommend trying using partitions so that if you have updated data or data being removed it'll take into account. You'll just need to make sure your entire data for each partition is being ingested otherwise you may lose data. If you have a specific key you can identify unique rows with then using an upsert may be an option as well.

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**

Categories