Updating a DataSet Triggered by the Finish of a Glue Job
We are using a Connector to retrieve data from Amazon Redshift and display it in Domo.
We use AWS Glue to retrieve data from Oracle and store it in Redshift, but I don't think it's possible to trigger the end of the Glue job to update the DataSet in Domo.
Currently we schedule the DataSet update around the time the Glue job finishes, but in some cases the timing doesn't work due to delays in the Glue job.
If anyone has successfully handled such cases, could you please share them?
Best Answer
-
if you don't want to use python you can use the Java CLI, https://domohelp.domo.com/hc/en-us/articles/360043437733-Command-Line-Interface-CLI-Tool#scripting
or build an app by sniffing API traffic.
Jae Wilson
Check out my 🎥 Domo Training YouTube Channel 👨💻
**Say "Thanks" by clicking the ❤️ in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"0
Answers
-
Hi @Tow
Typically when I have a more complex scheduling pipeline like this with data outside of Domo I’ll use a Python script to kick off each of the jobs serially in a single script so I know it’ll run each step in order.
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**2 -
if you don't want to use python you can use the Java CLI, https://domohelp.domo.com/hc/en-us/articles/360043437733-Command-Line-Interface-CLI-Tool#scripting
or build an app by sniffing API traffic.
Jae Wilson
Check out my 🎥 Domo Training YouTube Channel 👨💻
**Say "Thanks" by clicking the ❤️ in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"0 -
@jaeW_at_Onyx I believe the issue is that the Glue process happens on AWS's end so the CLI wouldn't be able to kick it off and know when that job has finished.
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**0 -
@GrantSmith , you'd use the CLI to start the execution of the dataset in Domo. presumably, the trigger would be the completion of the Glue job.
Jae Wilson
Check out my 🎥 Domo Training YouTube Channel 👨💻
**Say "Thanks" by clicking the ❤️ in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"1 -
Thank you, @GrantSmith and @jaeW_at_Onyx, for being so helpful. I've never used the CLI before, so I guess I'll need to fully test it. I will make it a future issue.
0
Categories
- All Categories
- 1.7K Product Ideas
- 1.7K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 295 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.8K Transform
- 98 SQL DataFlows
- 608 Datasets
- 2.1K Magic ETL
- 3.8K Visualize
- 2.5K Charting
- 712 Beast Mode
- 50 App Studio
- 39 Variables
- 668 Automate
- 170 Apps
- 446 APIs & Domo Developer
- 45 Workflows
- 7 DomoAI
- 34 Predict
- 14 Jupyter Workspaces
- 20 R & Python Tiles
- 391 Distribute
- 111 Domo Everywhere
- 274 Scheduled Reports
- 6 Software Integrations
- 116 Manage
- 113 Governance & Security
- Domo Community Gallery
- 31 Product Releases
- 9 Domo University
- 5.3K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 104 Community Announcements
- 4.8K Archive