Why no Batch_IDs when using Google Sheets connector?

Sell_the_Wings
Sell_the_Wings Member
edited September 2021 in Connectors

I've been using Domo for well over a year, and most of my source data is gathered using the PostgreSQL connector. I also work with a lot of Excel files that I grab via the Dropbox connector.

I have several data sets that I have built dataflows that allow me to build "full history" datasets. I rely on the existence of Batch_IDs and Batch_Last_Run fields to assist with the Rank&Window functionality needed to build these historic files.

Recently, I have started working with the GoogleSheet connector, and found it quite convenient and easy. However, when I started to build a dataflow today that would support building a full-history table I realized that there are no Batch_ID or Batch_LastRun values created and added to my records when I use the Google Sheet connector.

I've tried using the GoogleSheet connector "Discovery" method of finding the file as well as the GoogleSheet ID# method as well. I've iterated through several manual updates as well as 'scheduled' runs... but I see no hint of Batch info using either of the two methods mentioned above.

Is there something fundamental to the GoogleSheet connector that rules out Batch data being produced?

Tagged:

Best Answer

  • GrantSmith
    GrantSmith Coach
    edited September 2021 Answer ✓

    The GoogleSheet connector wasn't written to include a batch id or batch last run. You can simulate it using a dataflow to add the current timestamp as a constant and then selecting the max batch ID +1 for your new batch ID and add it to the data you're importing. You'd need to use a recursive dataflow to append the new records to the existing dataset and not have the google sheet connector set to append but rather replace.

    https://domohelp.domo.com/hc/en-us/articles/360057087393-Creating-a-Recursive-Snapshot-DataFlow-in-Magic-ETL-v2-Beta-

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**

Answers

  • GrantSmith
    GrantSmith Coach
    edited September 2021 Answer ✓

    The GoogleSheet connector wasn't written to include a batch id or batch last run. You can simulate it using a dataflow to add the current timestamp as a constant and then selecting the max batch ID +1 for your new batch ID and add it to the data you're importing. You'd need to use a recursive dataflow to append the new records to the existing dataset and not have the google sheet connector set to append but rather replace.

    https://domohelp.domo.com/hc/en-us/articles/360057087393-Creating-a-Recursive-Snapshot-DataFlow-in-Magic-ETL-v2-Beta-

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • Sell_the_Wings
    Sell_the_Wings Member

    Thanks Grant.

    Knowing that this is a feature and not a bug in Google Connector will allow me to implement a work around fairly easily. Thanks for the suggestion.

    Really appreciate the quick response!

Categories