API to perform 'Run Now' for raw dataset (Orchestrating pushing data into Domo)

efim_poberezkin
efim_poberezkin Member
in Archive

Hi,

 

I’d like to ask for advice on Domo’s API and what would be the best possible way to solve a specific use case.

 

My team is currently faced with a task of pushing data into Domo from S3. The data is first being processed in Databricks in a separate ETL and we orchestrate all steps in Airflow.

 

After trying out different methods it seems like Domo S3 connector is the most stable and performant approach. However, we’re not able to find API to run a raw dataset update (basically same thing as ‘Run Now’ option but via API).

 

It seems that Domo raw datasets based on connectors can only be scheduled inside Domo but that would split our orchestration into two independent parts which would implicitly depend on each other’s timing.

 

So far we’ve been able to orchestrate data upload with Domo Dataset API but that seems far less effective and reliable, besides we had to implement several hacks and workarounds.

 

Also we’ve checked out documentation on Domo Streams API which is said to be built for pushing large amounts of data and could be a way to go, but as we understood a new dataset is created every time an upload is performed, which isn’t an option, because then the following Domo ETL would have to be rebuilt every time (i.e. start with a different input dataset).

 

So from what we’ve gathered the best way would be to go with Domo S3 connector if it could be externally launched via some kind of Http request. Could anybody please advise if there’s API for that or maybe a beta feature that could be enabled?

 

TLDR. Is there a way to perform a 'Run Now' operation for a raw dataset based on an S3 connector via API / by making an Http request?

 

Thank you in advance, any help is appreciated.

Comments

  • camcusick_07
    camcusick_07 Domo Employee

    Hello,

     

    That is actually not how the steam API works. It will not create a whole new dataset every time. So if you are looking to push large amounts of data, then this would be a good option for you.

     

    Thanks,

     

     

  • user04751
    user04751 Member

    Hi, we've already tried out Stream API and indeed it turned out well. Thank you for advice anyway!

     

    E: Wrong user, forgot to log in.

  • camcusick_07
    camcusick_07 Domo Employee

    No problem! Have a great week.

  • user07399
    user07399 Member

    Does the stream API actually "Run" the dataset after you have inserted data?

  • user055174
    user055174 Member

    Hi, I have an existing dataset (I am using the MariaDB connector for it) which I would like to run remotely using an API. Is it possible to achieve this?

Categories