SNAPSHOT ETL DATAFLOW
Hi Beatiful People -
I was able to learn how to create a snapshot etl dataflow,
just a quick question,
where in the dataflow can I add a column that shows the date for the historical data and the date column for the appended data?
Best Answers
-
yes. or can I use the batch date for that?
0 -
Ask your CSM if it's possible to pair Salesforce with UPSERT or APPEND.
https://www.domo.com/appstore/connector/salesforce-with-upsert-connector/overview
While you can use ETL to build out this pipeline, Salesforce + UPSERT would be the Domo-preferred method (as opposed to building a recursive dataflow) for building history assuming the dataset is small enough (low millions of rows) and with some updates to MagicETL, this will scale pretty nicely.
Jae Wilson
Check out my 🎥 Domo Training YouTube Channel 👨💻
**Say "Thanks" by clicking the ❤️ in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"0
Answers
-
It appears that the screenshot you posted is from the Knowledge base article (https://knowledge.domo.com/Prepare/DataFlow_Tips_and_Tricks/Creating_a_Recursive%2F%2FSnapshot_ETL_DataFlow) which is what I'm going off of. The way that's configured is it joins the historical with the live data and ignores / filters any historical data which exists in the live file thus you lose and information from the historical dataset.
Are you wanting to keep a date of when it was first added to the dataset and another when it was last updated on the dataset?
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**0 -
yes. or can I use the batch date for that?
0 -
you mean to say, it will lose any data when the new data is appended?
I need a data to be appended and create a snapshot of historical data...
my data comes from salesforce which doesnt store historical data...
0 -
Ask your CSM if it's possible to pair Salesforce with UPSERT or APPEND.
https://www.domo.com/appstore/connector/salesforce-with-upsert-connector/overview
While you can use ETL to build out this pipeline, Salesforce + UPSERT would be the Domo-preferred method (as opposed to building a recursive dataflow) for building history assuming the dataset is small enough (low millions of rows) and with some updates to MagicETL, this will scale pretty nicely.
Jae Wilson
Check out my 🎥 Domo Training YouTube Channel 👨💻
**Say "Thanks" by clicking the ❤️ in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"0 -
subscribed to your youtube channel Onyx Reporting
1 -
I'm appending daily data for historical use. Because of the size I don't want store all of the dates. I just want the last 8 days only and remove the other dates. How can I do this? I can filter for the last 8 days, but it still stores other dates data.
0
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 296 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.9K Transform
- 100 SQL DataFlows
- 614 Datasets
- 2.2K Magic ETL
- 3.8K Visualize
- 2.5K Charting
- 729 Beast Mode
- 53 App Studio
- 40 Variables
- 677 Automate
- 173 Apps
- 451 APIs & Domo Developer
- 45 Workflows
- 8 DomoAI
- 34 Predict
- 14 Jupyter Workspaces
- 20 R & Python Tiles
- 394 Distribute
- 113 Domo Everywhere
- 275 Scheduled Reports
- 6 Software Integrations
- 121 Manage
- 118 Governance & Security
- Domo Community Gallery
- 32 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 108 Community Announcements
- 4.8K Archive