Concatenating records from a scheduled run at EOD
Hi everyone, I have a job that loads nearly 5k records (and ~100 columns) worth of data every day towards EOD. I want to be able to create a data set that concatenates the records the data from each day (say either for 2 weeks or 1 month). Ideally for the data set from each day, I will add another column (something to the effect of today() as BUSINESS_DATE) along with the columns from the data set. The next day, I'd get the new data set which would have that day's date populated and the records appended to the "union/concatenated" records.
I'd really appreciate any advice on a way to do it using DOMO's existing ETL tools (or any other tool) I've looked at "Append Rows" in "Magic ET" but I'm not able to set up.
Thanks in advance!
Comments
-
Hi @user054938
You'll want to look into a recursive/snapshot dataflow where you can add the current timestamp when the dataflow runs and then append it to your output dataset. The benefit of this is that you can make sure no duplicate records are included in your output dataset and you can limit your dataset to a specific timeframe (for example only include the last month of data). Below I've linked to the documentation for doing a recurring / snapshot dataflow
ETL 2.0:
ETL 1.0:
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**0 -
can you just APPEND the dataset as you load it into Domo? Maybe add the current_date() as part of the ingestion process?
Looooooong term it'll be faster than recursive dataflows but if i'm honest for a 5k row dataset, recursive will work fine. It just comes with risks if you ever accidentally blow up your data, because you're not retaining a dataset in Domo with full history.
Jae Wilson
Check out my 🎥 Domo Training YouTube Channel 👨💻
**Say "Thanks" by clicking the ❤️ in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"0
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 300 Workbench
- 6 Cloud Amplifier
- 9 Federated
- 2.9K Transform
- 100 SQL DataFlows
- 622 Datasets
- 2.2K Magic ETL
- 3.9K Visualize
- 2.5K Charting
- 744 Beast Mode
- 58 App Studio
- 41 Variables
- 686 Automate
- 176 Apps
- 453 APIs & Domo Developer
- 47 Workflows
- 10 DomoAI
- 36 Predict
- 15 Jupyter Workspaces
- 21 R & Python Tiles
- 395 Distribute
- 113 Domo Everywhere
- 276 Scheduled Reports
- 6 Software Integrations
- 125 Manage
- 122 Governance & Security
- 8 Domo Community Gallery
- 38 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 108 Community Announcements
- 4.8K Archive