SFDC Connector - Delta/Incremental

Nick_Bertz
Nick_Bertz Member
in Connectors

Hello,

 

We use SFDC Datasets, and we've always pulled the entire dataset.  I've noticed their are options to do incremental/delta loads.  I am curious how to get started with these options.  Ideally i'd like to pull only new or changed records on a nightly basis as opposed to full data loads.

 

What i'm looking for is some guidance on how to properly configure this, i tried reviewing the documentation but i couldn't determine the proper approach to configure the selections on the SFDC Dataset Connector.

 

1. Load history

2. Begin loading only the changes nightly (which covers the past and the future)

 

Thank you!

Best Answer

  • jaredpk
    jaredpk Domo Employee
    Answer ✓

    Hi Nick -- I see where you are going with this.  One possible solution could be as follows:

     

    • Import only the last day's data based on system Mod Stamp
    • Compare the new data to the existing dataset in Domo.
    • Based on the lastest day's data and the dataset Unique identifier (opportunityID, AccountID etc.) recursively pull in the latest updates.

     

    I believe this could work with a little testing.  I'm leaving the dojo day session, but you can reach out to me with more specifics of the dataset if you would like.

    I work for Domo.
    **Say "Thanks" by clicking the thumbs up in the post that helped you.
    **Please mark the post that solves your problem as "Accepted Solution"

Answers

  • jaredpk
    jaredpk Domo Employee

    Hi -- the SFDC connector is pretty robust.  So on datasets less than a few hundred thousand rows, replace should work well.

     

    On larger datasets, you would need to run the data through a recursive dataflow and then deduplicate the results to find the latest updates.  Instructions for SQL and MagicETL recursive dataflows are below:

     

    https://knowledge.domo.com/Prepare/DataFlow_Tips_and_Tricks/Creating_a_Recursive%2F%2FSnapshot_ETL_DataFlow 

     

    https://knowledge.domo.com/Prepare/DataFlow_Tips_and_Tricks/Creating_a_Recursive%2F%2FSnapshot_ETL_DataFlow 

    I work for Domo.
    **Say "Thanks" by clicking the thumbs up in the post that helped you.
    **Please mark the post that solves your problem as "Accepted Solution"
  • jaredpk
    jaredpk Domo Employee

    PS ... Hi Nick.  Just saw it was you.

    I work for Domo.
    **Say "Thanks" by clicking the thumbs up in the post that helped you.
    **Please mark the post that solves your problem as "Accepted Solution"
  • Nick_Bertz
    Nick_Bertz Member

    Hi Jared,

     

    The image below of our SFDC connector is what i was specifically referring too...unless this is used mostly as strictly targeted data, or for one-offs.  I was hoping i could leverage this functionality to load history, then only load changes, i just wasn't able to determine the best usage from the existing documentation.

     

    Happy to connect offline as well if that makes more sense.

     

    I noticed that when i use 'Browse Objects and Fields' for the 'Opportunity' object, i get the following options:SFDC Connector.png

     

     

     

     

     

  • jaredpk
    jaredpk Domo Employee
    Answer ✓

    Hi Nick -- I see where you are going with this.  One possible solution could be as follows:

     

    • Import only the last day's data based on system Mod Stamp
    • Compare the new data to the existing dataset in Domo.
    • Based on the lastest day's data and the dataset Unique identifier (opportunityID, AccountID etc.) recursively pull in the latest updates.

     

    I believe this could work with a little testing.  I'm leaving the dojo day session, but you can reach out to me with more specifics of the dataset if you would like.

    I work for Domo.
    **Say "Thanks" by clicking the thumbs up in the post that helped you.
    **Please mark the post that solves your problem as "Accepted Solution"

Categories