Updating a DataSet Triggered by the Finish of a Glue Job

Tow

We are using a Connector to retrieve data from Amazon Redshift and display it in Domo.

We use AWS Glue to retrieve data from Oracle and store it in Redshift, but I don't think it's possible to trigger the end of the Glue job to update the DataSet in Domo.

Currently we schedule the DataSet update around the time the Glue job finishes, but in some cases the timing doesn't work due to delays in the Glue job.

If anyone has successfully handled such cases, could you please share them?

jaeW_at_Onyx

if you don't want to use python you can use the Java CLI, https://domohelp.domo.com/hc/en-us/articles/360043437733-Command-Line-Interface-CLI-Tool#scripting

or build an app by sniffing API traffic.

GrantSmith

Hi @Tow

Typically when I have a more complex scheduling pipeline like this with data outside of Domo I’ll use a Python script to kick off each of the jobs serially in a single script so I know it’ll run each step in order.

jaeW_at_Onyx

if you don't want to use python you can use the Java CLI, https://domohelp.domo.com/hc/en-us/articles/360043437733-Command-Line-Interface-CLI-Tool#scripting

or build an app by sniffing API traffic.

GrantSmith

@jaeW_at_Onyx I believe the issue is that the Glue process happens on AWS's end so the CLI wouldn't be able to kick it off and know when that job has finished.

jaeW_at_Onyx

@GrantSmith , you'd use the CLI to start the execution of the dataset in Domo. presumably, the trigger would be the completion of the Glue job.

Tow

Thank you, @GrantSmith and @jaeW_at_Onyx, for being so helpful. I've never used the CLI before, so I guess I'll need to fully test it. I will make it a future issue.