Concatenating records from a scheduled run at EOD

user054938

Hi everyone, I have a job that loads nearly 5k records (and ~100 columns) worth of data every day towards EOD. I want to be able to create a data set that concatenates the records the data from each day (say either for 2 weeks or 1 month). Ideally for the data set from each day, I will add another column (something to the effect of today() as BUSINESS_DATE)  along with the columns from the data set. The next day, I'd get the new data set which would have that day's date populated and the records appended to the "union/concatenated" records.

 

I'd really appreciate any advice on a way to do it using DOMO's existing ETL tools (or any other tool) I've looked at "Append Rows" in "Magic ET" but I'm not able to set up.

 

Thanks in advance!

GrantSmith

Hi @user054938 

 

You'll want to look into a recursive/snapshot dataflow where you can add the current timestamp when the dataflow runs and then append it to your output dataset. The benefit of this is that you can make sure no duplicate records are included in your output dataset and you can limit your dataset to a specific timeframe (for example only include the last month of data). Below I've linked to the documentation for doing a recurring / snapshot dataflow

 

ETL 2.0:

https://knowledge.domo.com/Prepare/DataFlow_Tips_and_Tricks/Creating_a_Recursive%2F%2FSnapshot_DataFlow_in_Magic_ETL_v2_(Beta)

 

ETL 1.0:

https://knowledge.domo.com/Prepare/DataFlow_Tips_and_Tricks/Creating_a_Recursive%2F%2FSnapshot_ETL_DataFlow 

jaeW_at_Onyx

can you just APPEND the dataset as you load it into Domo?  Maybe add the current_date() as part of the ingestion process?

 

Looooooong term it'll be faster than recursive dataflows but if i'm honest for a 5k row dataset, recursive will work fine.  It just comes with risks if you ever accidentally blow up your data, because you're not retaining a dataset in Domo with full history.