Using merge in snowflake connector

Options
ryangreiner9
ryangreiner9 Member
edited July 2024 in Connectors

Hi,

I am trying to optimize efficiency with the snowflake connector by changing the update method to merge. My data set has a column that shows when each row was last updated. I want to use this to pull only data that has been updated since the last time the dataset ran in domo. However, I can't seem to find any way to update my query to know the last time the dataset was run in domo. Is it possible to pull this timestamp into my query? Has anyone dealt with anything like this in the past?

Tagged:

Best Answer

  • MichelleH
    MichelleH Coach
    Answer ✓

    @ryangreiner9 Which Snowflake connector are you using?

    The Snowflake Partition Connector allows you to only fetch data from a trailing number of days to merge with the existing data. Rather than referencing the last refresh timestamp from the Domo connector, I'd suggest specifying the trailing number of days based on the connector's schedule settings with a few days buffer. For example if you want to refresh you dataset once a day, then I would configure the connector to pull rows where the last updated date is within the last 7 days. This will only ingest the limited amount of data, but will also provide a safety net if there is an outage for a few days.

    Here is the documentation on the connector: https://domo-support.domo.com/s/article/360057013754?language=en_US

Answers

  • MichelleH
    MichelleH Coach
    Answer ✓

    @ryangreiner9 Which Snowflake connector are you using?

    The Snowflake Partition Connector allows you to only fetch data from a trailing number of days to merge with the existing data. Rather than referencing the last refresh timestamp from the Domo connector, I'd suggest specifying the trailing number of days based on the connector's schedule settings with a few days buffer. For example if you want to refresh you dataset once a day, then I would configure the connector to pull rows where the last updated date is within the last 7 days. This will only ingest the limited amount of data, but will also provide a safety net if there is an outage for a few days.

    Here is the documentation on the connector: https://domo-support.domo.com/s/article/360057013754?language=en_US

Categories