Remotely trigger a DataSet refresh?

DataSquirrel · 2021-09-16T08:34:56+00:00

There was an error rendering this rich post.

I'm checking out some of our options for speeding up data ingestion in Domo. A few of our Postgres pulls are taking longer than we would like, and I'm exploring creating a materialized view to cut the PG processing time down to 0, from Domo's perspective. That still leaves the data transfer time, and the Domo processing time, but it should help a bit.

I'd prefer to make sure that the view has finished refreshing before Domo does its push. To this end, I'm thinking "do the `REFRESH MATERIALIZED VIEW` in Postgres, then kick off the Domo pull of that view using the Domo DataSet API." But, I don't see any such feature in the API.

https://developer.domo.com/docs/dataset-api-reference/dataset

I also checked the Java CLI tools, but don't see a "run DataSet refresh" feature there either.

https://domohelp.domo.com/hc/en-us/articles/360043437733-Command-Line-Interface-CLI-Tool

(The CLI would be less convenient, in this. case.)

I thought I'd seen such a feature, was I dreaming?

Thanks in advance.

Find more posts tagged with

SQL

Connectors

Dataset

Comments

GrantSmith

Are you using workbench or the connector to pull your Postgres data? Typically when I needed more control locally over my pipeline I’ll write a Python script to process my data locally within the script and then upload the resulting dataset via the API/pydomo SDK

jaeW_at_Onyx

@DataSquirrel ,

Wheras Grant is assuming you're ingesting the data via Workbench, I'm assuming you're using the web connector library.

I don't know off top, but if i had to investigate this problem, these are the things I'd do.

1) check PyDomo to see if you can trigger via Python

2) check java cli to see if you can execute a dataset

3) monitor network traffic when you click the "run" button in the Domo API.

DataSquirrel

Thanks @GrantSmith, I enjoyed the presentation that you were a part of too. No, we're not using Workbench. We've got a handful of jobs in Workbench where we generate an Excel file (out of 4D), and dump it to disk. These go back years, we're not adding any more of those. Most of our data comes via Postgres, where we can aggregate and, if we want, analyze a lot before ingestion by Domo.

Thanks @jaeW_at_Onyx, correct, we're using the Postgres SSH connector. I'd like to configure jobs in there, and then remotely kick them off.

Just checked the docs, and it looks like the Java CLI and PyDomo are wrappers for the public APIs. I'm not finding any 'refresh DataSet' in any of the APIs...thought I had. I went through all of the public (1.0) APIs a few years back, and set up code to run all of the catalog reading code. So, I'd be comfortable using the APIs directly, but I don't see a call to invoke :(

Seems like a funny feature not to support, so I'm thinking I'm missing it.

And, @jaeW_at_Onyx , yeah, I could sniff out what undocumented APIs the Domo Web UI uses to go what it does. I thought about that for some detailed I needed that the read APIs do not provide. Personal choice. I figure it's impolite to hack at unpublished APIs and, in any case, it's pretty tedious to do. Plus, then you're on your own if the vendor decides to change how they're unpublished APIs function. I mean, they've not made promise in any form not to change them.

jaeW_at_Onyx

@DataSquirrel

https://www.youtube.com/watch?v=hRwrZABP8RE&t=188s

from the CTO and one of the leads on the engineering team.

DataSquirrel

Mind. Blown. Jae, that was excellent. I'm not exaggerating when I say that you've just radically shifted my feelings about Domo. Thanks for organizing that session, and thanks for sharing the link.

Domo's attitude about discoverable-but-undocumented APIs is outstanding. I had no idea...I'm so used to corporate attitudes being hostile to that sort of thing. Kudos Domo!

That tip about the different authentication methods for "Public" and "public" APIs is pure gold.

I just updated my JavaCLI, and spotted the call I was thinking of, dataset-run-now...it was in the version I had, I just didn't see it. The docs are a little thin, but there's just a ton in that tool:

https://domohelp.domo.com/hc/en-us/articles/360043437733-Command-Line-Interface-CLI-Tool

I tried it out about 9 months ago, following along with some of your other videos. We're hitting a point where UPSERT would make sense for us, if we can get it working, or PARTITION. We're using REPLACE now...but, big...and the APPEND options aren't a great match for us.

jaeW_at_Onyx

WHOOP!!!

I'll just leave this here.

https://datacrew.circle.so/c/developerdomocom-community-edition/full-authentication-with-domo-apis

and maybe this

https://datacrew.circle.so/c/developerdomocom-community-edition/a-comprehensive-list-of-domo-apis

DataSquirrel

Nice one!

DataSquirrel

Thanks @jaeW_at_Onyx, correct, we're using the Postgres SSH connector. I'd like to configure jobs in there, and then remotely kick them off.

Seems like a funny feature not to support, so I'm thinking I'm missing it.