Using merge in snowflake connector

Hi,
I am trying to optimize efficiency with the snowflake connector by changing the update method to merge. My data set has a column that shows when each row was last updated. I want to use this to pull only data that has been updated since the last time the dataset ran in domo. However, I can't seem to find any way to update my query to know the last time the dataset was run in domo. Is it possible to pull this timestamp into my query? Has anyone dealt with anything like this in the past?
Best Answer
-
@ryangreiner9 Which Snowflake connector are you using?
The Snowflake Partition Connector allows you to only fetch data from a trailing number of days to merge with the existing data. Rather than referencing the last refresh timestamp from the Domo connector, I'd suggest specifying the trailing number of days based on the connector's schedule settings with a few days buffer. For example if you want to refresh you dataset once a day, then I would configure the connector to pull rows where the last updated date is within the last 7 days. This will only ingest the limited amount of data, but will also provide a safety net if there is an outage for a few days.
Here is the documentation on the connector:
0
Answers
-
@ryangreiner9 Which Snowflake connector are you using?
The Snowflake Partition Connector allows you to only fetch data from a trailing number of days to merge with the existing data. Rather than referencing the last refresh timestamp from the Domo connector, I'd suggest specifying the trailing number of days based on the connector's schedule settings with a few days buffer. For example if you want to refresh you dataset once a day, then I would configure the connector to pull rows where the last updated date is within the last 7 days. This will only ingest the limited amount of data, but will also provide a safety net if there is an outage for a few days.
Here is the documentation on the connector:
0
Categories
- All Categories
- 2K Product Ideas
- 2K Ideas Exchange
- 1.6K Connect
- 1.3K Connectors
- 311 Workbench
- 7 Cloud Amplifier
- 9 Federated
- 3K Transform
- 114 SQL DataFlows
- 654 Datasets
- 2.2K Magic ETL
- 4.1K Visualize
- 2.5K Charting
- 803 Beast Mode
- 79 App Studio
- 44 Variables
- 758 Automate
- 188 Apps
- 480 APIs & Domo Developer
- 73 Workflows
- 17 DomoAI
- 40 Predict
- 17 Jupyter Workspaces
- 23 R & Python Tiles
- 408 Distribute
- 119 Domo Everywhere
- 279 Scheduled Reports
- 10 Software Integrations
- 141 Manage
- 137 Governance & Security
- 8 Domo Community Gallery
- 47 Product Releases
- 12 Domo University
- 5.4K Community Forums
- 41 Getting Started
- 31 Community Member Introductions
- 114 Community Announcements
- 4.8K Archive