How to shift from Replace to Append Mode for RedShift tables without loosing data ?

pierreeric
pierreeric
in Archive

Hi, 

We have connected a bunch of massive volume of data to Domo by connecting to our RedShift database. Some of those tables are 50M+ rows and the nightly sync of those becomes longer and longer as we position them as "replace".

Can you share best tricks to move Domo to an "append" mode while ensuring that there's no data gap or data duplication by doing so?

Currently we had to stop a few of the bigger tables to be ingested (300M+ rows / 12Gb + datasets) and some of our tables takes more than 8 hours to sync.

Thanks in advance

Comments

  • [Deleted User]
    [Deleted User] Member

    @pierreeric I understand @zcameron has been working with you on this. Once its sorted out you can share back the solution.

     

    Thanks!

  • pierreeric
    pierreeric

    The only acceptable and efficient way we found to safely move RedShift tables to "Append" mode in domo was to do the following:

    1 - pre-generate a RedShift view on the RedShift side that only captures the updated data from the day before (cleared of duplicates or missing entries)
    2 - then and only then connect this "day minus one" view as the datasource in Domo and toggle it into append mode mechanism.

    Step 1 being very important as you can control on RedShift what the data should be (free of duplicates or data gaps) - something Domo cannot ensure.

Categories