Long running ETL flows alerting strategies

pimogo · January 2021

Any strategies i can employ to alert myself whenever a query that generally takes seconds to run runs for longer than it should?

I have a query that has on rare occasions been found running for hours when it should run in seconds. I'd like a way to be alerted to this situation without having to come across it by chance.

Thanks!

jaeW_at_Onyx · January 2021

you could use the domo_governance dataset or the DomoStats_DataflowHistory (i believe) dataset to see how long it's taking dataflows to execute, but they would have to be updated periodically ... usually Domo prefers the governance datasets to only update daily.

are you running this ETL multiple times a day?

you could build a series of scripts against the dataflow oriented APIs by leveraging the domo command line utility. https://domohelp.domo.com/hc/en-us/articles/360043437733-Command-Line-Interface-CLI-Tool#:~:text=Domo's%20Command%20Line%20Interface%20(CLI,here%20to%20view%20the%20changelog.

alternatively, instead of testing the dataflow, test the artificat, the datasets. If you're updating the every 15 minutes, then just report any dataset that hasn't updated in the last half hour.

Long running ETL flows alerting strategies

Comments

Categories