Programmatically run a dataflow/dataset

@jaeW_at_Onyx  thanks for your fast answer! I think either scripting the CLI commands or using the corresponding APIs will let me to the job.

I have actually two questions for you, one for datasets and one for dataflows.

1) Dataflows

What is the difference between using the dataflow-run-now -i <ID> command and the corresponding API called by my script? Even if not documented, isn't it basically the same approach?

2) Datasets

I have not been able to use the command dataset-run-now -i <ID> -p, it gives me a java.lang.NumberFormatException: are you able to tell me why?. The <ID> I am using is the one I can catch from the URL of the dataset.

However, i have been able to find the stream_id of the dataset using the command list-dataset -i <ID>.

Then I tried dataset-run-now -i <STREAM_ID> -s and it worked. What is the difference?

Also, i see this is basically using the (documented) stream API, so maybe it is better to use this second approach for datasets coming from connectors?

First of all, thank you so much for your answers. It is really helping me.

My goal here is to have an external scheduler that let me manage different executions in different systems, not only in Domo. That's why I'm trying to access all the systems through their API, so I can actually write my manager as I wish with my freedom. I don't think this is something Workbench can help me with, or am I missing something?

By the way, for what concern datasets and their ids / stream_ids I bet this is something we would not really know, but at least I have been able to use the documented stream api to launch programmatically the dataset connectors, so I'm very happy to use the documented APIs.

Instead I'm still having some troubles with dataflows. I've run successfully this command:

dataflow-run-now -i 350

And what I see from the CLI is that this API is called:

Executing POST: https://[my_domain].domo.com/api/dataprocessing/v1/dataflows/350/executions

However, when trying to execute a POST request (in a Python script) using this URL and the authentication that works perfectly with the other APIs I get this result:

{"status":401,"statusReason":"Unauthorized","path":"/api/dataprocessing/v1/dataflows/350/executions","message":"Full authentication is required to access this resource","toe":"XXXXXXXXXXXXX"}

To generate the token I'm using the same credentials (I'm an admin) I use to connect to the Java-CLI, so this really makes no sense to me. Do you or anyone have any suggestion on what could be the problem?

I may try the approach with the Java CLI scripting, but I'm stubborn and I would definitely like to understand what is the difference in the two approaches ?

Hi Jae, of course I'm going to do a writeup for the whole community!

Let's start (maybe at the end I will host this code somewhere improving it and commenting properly) with dataset powered into Domo through a connector: this is the solution you can use to run it with Python and Pydomo using supported and documented API.

from pydomo import Domo

client_id = <CLIENT_ID>
client_secret = <CLIENT_SECRET>
api_host = 'api.domo.com'

stream_id = <STREAM_ID>

domo = Domo(client_id, client_secret, api_host=api_host)
streams = domo.streams
execution = streams.create_execution(stream_id)

 Just a not on how to get the <STREAM_ID>.

From the Domo Web Interface you are able to get only the <DATASET_ID>, but as we stated before behind the scenes there is a <STREAM_ID> associated to the dataset.

Right now I have two approaches:

1. Use the java-cli and the command list-dataset -i <DATASET_ID> and extract the <STREAM_ID> from the output given by the java-cli
2. Loop over the streams and find the one associated to your <DATASET_ID>. I would not recommend using this solution though:

def get_stream_id_by_dataset_id(domo, list_limit, dataset_id) :
    streams = domo.streams
    stream_list = streams.list(list_limit, 0)
    dataset_streams = [stream for stream in stream_list if stream['dataSet']['id'] == dataset_id]
    if(len(dataset_streams)==1):
        return dataset_streams[0]['id']
    else:
        raise Exception('No stream is associated to the given dataset')​

Let's go back to our dataflows. I think I can state there is no way to use documented API to run programmatically a dataflow.

However we can use the dataprocessing API with full authentication using the token, as you suggested.
This is a working script you can use once you have such a token:

import requests

domo_token = <ACCESS_TOKEN>
domo_instance = <YOUR_DOMO_INSTANCE>
dataflow_id = <DATAFLOW_ID>
dataflow_api = 'https://{}.domo.com/api/dataprocessing/v1/dataflows/{}/executions'.format(domo_instance, dataflow_id)

print(dataflow_api) 
headers = {'Content-Type'   : 'application/json',
           'Accept'         : 'application/json',
           'X-DOMO-Developer-Token'  : domo_token}

execute_dataflow_response = requests.post(url = dataflow_api, headers = headers)

if execute_dataflow_response.status_code == 200 :
    print(execute_dataflow_response.json())
else :
    print(execute_dataflow_response.json())
    raise Exception('Dataflow execution API response status code {}'.format(execute_dataflow_response.status_code))

The very last problem I encountered was about how to get the token. Right now I get the token by just following this guide (official Domo documentation), but I couldn't get it through an API request.

What I tried is to make a POST request to 'https://<my-domo-instance>.domo.com/api/content/v2/authentication' with this data:

body_att = {
        "method" : "password",
        "emailAddress" : <MY_EMAIL>,
        "password" : <MY_PASSWORD>}

However, I got this response:

{'status': 400, 'statusReason': 'Bad Request', 'message': 'Bad Request', 'toe': 'XXXXXXXXXXXXXXXXXXXX'}

I tried the same switching to v1 ( 'https://<my-domo-instance>.domo.com/api/content/v1/authentication') and I received the 'unauthorized' error:

{'status': 401, 'statusReason': 'Unauthorized', 'path': '/api/content/v1/authentication', 'message': 'Full authentication is required to access this resource', 'toe': 'XXXXXXXXXXXXXXX'}

Do you have any clue on what may be the reason?
However I already want to thank you because even if I have to perform the manual step for managing the token, I achieved the goal to run the dataflow from my script!!! Thanks!!