Why no Batch_IDs when using Google Sheets connector?

Sell_the_Wings · 2021-09-23T18:54:26+00:00

There was an error rendering this rich post.

I've been using Domo for well over a year, and most of my source data is gathered using the PostgreSQL connector. I also work with a lot of Excel files that I grab via the Dropbox connector.

I have several data sets that I have built dataflows that allow me to build "full history" datasets. I rely on the existence of Batch_IDs and Batch_Last_Run fields to assist with the Rank&Window functionality needed to build these historic files.

Recently, I have started working with the GoogleSheet connector, and found it quite convenient and easy. However, when I started to build a dataflow today that would support building a full-history table I realized that there are no Batch_ID or Batch_LastRun values created and added to my records when I use the Google Sheet connector.

I've tried using the GoogleSheet connector "Discovery" method of finding the file as well as the GoogleSheet ID# method as well. I've iterated through several manual updates as well as 'scheduled' runs... but I see no hint of Batch info using either of the two methods mentioned above.

Is there something fundamental to the GoogleSheet connector that rules out Batch data being produced?

Find more posts tagged with

Connectors

Accepted answers

GrantSmith

The GoogleSheet connector wasn't written to include a batch id or batch last run. You can simulate it using a dataflow to add the current timestamp as a constant and then selecting the max batch ID +1 for your new batch ID and add it to the data you're importing. You'd need to use a recursive dataflow to append the new records to the existing dataset and not have the google sheet connector set to append but rather replace.

https://domohelp.domo.com/hc/en-us/articles/360057087393-Creating-a-Recursive-Snapshot-DataFlow-in-Magic-ETL-v2-Beta-

All comments

GrantSmith

https://domohelp.domo.com/hc/en-us/articles/360057087393-Creating-a-Recursive-Snapshot-DataFlow-in-Magic-ETL-v2-Beta-

Sell_the_Wings

Thanks Grant.

Knowing that this is a feature and not a bug in Google Connector will allow me to implement a work around fairly easily. Thanks for the suggestion.

Really appreciate the quick response!