Remotely trigger a DataSet refresh?

DataSquirrel
DataSquirrel Contributor
in APIs & Domo Developer

I'm checking out some of our options for speeding up data ingestion in Domo. A few of our Postgres pulls are taking longer than we would like, and I'm exploring creating a materialized view to cut the PG processing time down to 0, from Domo's perspective. That still leaves the data transfer time, and the Domo processing time, but it should help a bit.

I'd prefer to make sure that the view has finished refreshing before Domo does its push. To this end, I'm thinking "do the `REFRESH MATERIALIZED VIEW` in Postgres, then kick off the Domo pull of that view using the Domo DataSet API." But, I don't see any such feature in the API.

https://developer.domo.com/docs/dataset-api-reference/dataset

I also checked the Java CLI tools, but don't see a "run DataSet refresh" feature there either.

https://domohelp.domo.com/hc/en-us/articles/360043437733-Command-Line-Interface-CLI-Tool

(The CLI would be less convenient, in this. case.)

I thought I'd seen such a feature, was I dreaming?


Thanks in advance.

Tagged:

Comments

  • GrantSmith
    GrantSmith Coach

    Are you using workbench or the connector to pull your Postgres data? Typically when I needed more control locally over my pipeline I’ll write a Python script to process my data locally within the script and then upload the resulting dataset via the API/pydomo SDK

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • jaeW_at_Onyx
    jaeW_at_Onyx Coach

    @DataSquirrel ,

    Wheras Grant is assuming you're ingesting the data via Workbench, I'm assuming you're using the web connector library.


    I don't know off top, but if i had to investigate this problem, these are the things I'd do.


    1) check PyDomo to see if you can trigger via Python

    2) check java cli to see if you can execute a dataset

    3) monitor network traffic when you click the "run" button in the Domo API.

    Jae Wilson
    Check out my 🎥 Domo Training YouTube Channel 👨‍💻

    **Say "Thanks" by clicking the ❤️ in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"
  • DataSquirrel
    DataSquirrel Contributor

    Thanks @GrantSmith, I enjoyed the presentation that you were a part of too. No, we're not using Workbench. We've got a handful of jobs in Workbench where we generate an Excel file (out of 4D), and dump it to disk. These go back years, we're not adding any more of those. Most of our data comes via Postgres, where we can aggregate and, if we want, analyze a lot before ingestion by Domo.

    Thanks @jaeW_at_Onyx, correct, we're using the Postgres SSH connector. I'd like to configure jobs in there, and then remotely kick them off.

    Just checked the docs, and it looks like the Java CLI and PyDomo are wrappers for the public APIs. I'm not finding any 'refresh DataSet' in any of the APIs...thought I had. I went through all of the public (1.0) APIs a few years back, and set up code to run all of the catalog reading code. So, I'd be comfortable using the APIs directly, but I don't see a call to invoke :(

    Seems like a funny feature not to support, so I'm thinking I'm missing it.

    And, @jaeW_at_Onyx , yeah, I could sniff out what undocumented APIs the Domo Web UI uses to go what it does. I thought about that for some detailed I needed that the read APIs do not provide. Personal choice. I figure it's impolite to hack at unpublished APIs and, in any case, it's pretty tedious to do. Plus, then you're on your own if the vendor decides to change how they're unpublished APIs function. I mean, they've not made promise in any form not to change them.

  • jaeW_at_Onyx
    jaeW_at_Onyx Coach

    @DataSquirrel


    https://www.youtube.com/watch?v=hRwrZABP8RE&t=188s

    from the CTO and one of the leads on the engineering team.

    Jae Wilson
    Check out my 🎥 Domo Training YouTube Channel 👨‍💻

    **Say "Thanks" by clicking the ❤️ in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"
  • DataSquirrel
    DataSquirrel Contributor
    edited September 2021

    Mind. Blown. Jae, that was excellent. I'm not exaggerating when I say that you've just radically shifted my feelings about Domo. Thanks for organizing that session, and thanks for sharing the link.

    Domo's attitude about discoverable-but-undocumented APIs is outstanding. I had no idea...I'm so used to corporate attitudes being hostile to that sort of thing. Kudos Domo!

    That tip about the different authentication methods for "Public" and "public" APIs is pure gold.

    I just updated my JavaCLI, and spotted the call I was thinking of, dataset-run-now...it was in the version I had, I just didn't see it. The docs are a little thin, but there's just a ton in that tool:

    https://domohelp.domo.com/hc/en-us/articles/360043437733-Command-Line-Interface-CLI-Tool

    I tried it out about 9 months ago, following along with some of your other videos. We're hitting a point where UPSERT would make sense for us, if we can get it working, or PARTITION. We're using REPLACE now...but, big...and the APPEND options aren't a great match for us.

  • jaeW_at_Onyx
    jaeW_at_Onyx Coach

    WHOOP!!!

    I'll just leave this here.

    https://datacrew.circle.so/c/developerdomocom-community-edition/full-authentication-with-domo-apis


    and maybe this


    https://datacrew.circle.so/c/developerdomocom-community-edition/a-comprehensive-list-of-domo-apis

    Jae Wilson
    Check out my 🎥 Domo Training YouTube Channel 👨‍💻

    **Say "Thanks" by clicking the ❤️ in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"
  • DataSquirrel
    DataSquirrel Contributor

    Nice one!

  • DataSquirrel
    DataSquirrel Contributor

    Thanks @GrantSmith, I enjoyed the presentation that you were a part of too. No, we're not using Workbench. We've got a handful of jobs in Workbench where we generate an Excel file (out of 4D), and dump it to disk. These go back years, we're not adding any more of those. Most of our data comes via Postgres, where we can aggregate and, if we want, analyze a lot before ingestion by Domo.

    Thanks @jaeW_at_Onyx, correct, we're using the Postgres SSH connector. I'd like to configure jobs in there, and then remotely kick them off.

    Just checked the docs, and it looks like the Java CLI and PyDomo are wrappers for the public APIs. I'm not finding any 'refresh DataSet' in any of the APIs...thought I had. I went through all of the public (1.0) APIs a few years back, and set up code to run all of the catalog reading code. So, I'd be comfortable using the APIs directly, but I don't see a call to invoke :(

    Seems like a funny feature not to support, so I'm thinking I'm missing it.

    And, @jaeW_at_Onyx , yeah, I could sniff out what undocumented APIs the Domo Web UI uses to go what it does. I thought about that for some detailed I needed that the read APIs do not provide. Personal choice. I figure it's impolite to hack at unpublished APIs and, in any case, it's pretty tedious to do. Plus, then you're on your own if the vendor decides to change how they're unpublished APIs function. I mean, they've not made promise in any form not to change them.

Categories