Using merge in snowflake connector

Hi,
I am trying to optimize efficiency with the snowflake connector by changing the update method to merge. My data set has a column that shows when each row was last updated. I want to use this to pull only data that has been updated since the last time the dataset ran in domo. However, I can't seem to find any way to update my query to know the last time the dataset was run in domo. Is it possible to pull this timestamp into my query? Has anyone dealt with anything like this in the past?
Best Answer
-
@ryangreiner9 Which Snowflake connector are you using?
The Snowflake Partition Connector allows you to only fetch data from a trailing number of days to merge with the existing data. Rather than referencing the last refresh timestamp from the Domo connector, I'd suggest specifying the trailing number of days based on the connector's schedule settings with a few days buffer. For example if you want to refresh you dataset once a day, then I would configure the connector to pull rows where the last updated date is within the last 7 days. This will only ingest the limited amount of data, but will also provide a safety net if there is an outage for a few days.
Here is the documentation on the connector:
0
Answers
-
@ryangreiner9 Which Snowflake connector are you using?
The Snowflake Partition Connector allows you to only fetch data from a trailing number of days to merge with the existing data. Rather than referencing the last refresh timestamp from the Domo connector, I'd suggest specifying the trailing number of days based on the connector's schedule settings with a few days buffer. For example if you want to refresh you dataset once a day, then I would configure the connector to pull rows where the last updated date is within the last 7 days. This will only ingest the limited amount of data, but will also provide a safety net if there is an outage for a few days.
Here is the documentation on the connector:
0
Categories
- All Categories
- 2K Product Ideas
- 2K Ideas Exchange
- 1.6K Connect
- 1.3K Connectors
- 311 Workbench
- 6 Cloud Amplifier
- 9 Federated
- 3.8K Transform
- 657 Datasets
- 115 SQL DataFlows
- 2.2K Magic ETL
- 815 Beast Mode
- 3.3K Visualize
- 2.5K Charting
- 81 App Studio
- 45 Variables
- 775 Automate
- 190 Apps
- 481 APIs & Domo Developer
- 81 Workflows
- 23 Code Engine
- 40 AI and Machine Learning
- 20 AI Chat
- 1 AI Playground
- 1 AI Projects and Models
- 18 Jupyter Workspaces
- 410 Distribute
- 120 Domo Everywhere
- 280 Scheduled Reports
- 10 Software Integrations
- 144 Manage
- 140 Governance & Security
- 8 Domo Community Gallery
- 48 Product Releases
- 12 Domo University
- 5.4K Community Forums
- 41 Getting Started
- 31 Community Member Introductions
- 114 Community Announcements
- 4.8K Archive