Make ETL generated dataset to be updated by API?

I have a dataset created by ETL. Then I want to append more data to it by API. It appears that I cannot create a stream from the dataset. According to my reading on Internet, I cannot update an ETL generated dataset by API. Is it true? If yes, what will be my solutions? I thought a few:

1. Create another new dataset by API, and then create ETL to copy the data from the first one to this one. But it looks like in ETL you can only create a dataflow from existing datasets to new datasets.

2. Export from ETL dataset to CSV and then import them into API created dataset. It sounds feasible. But the dataset is huge. It may take hours or even days to do it. Will DOMO can do it internally so that it will be faster?

Appreciate it if someone can share the answer if knows

Find more posts tagged with

Comments

ST_Superman

I would try to get your first option working. You would need to first create your new dataset via the API. Once you have run the API and created the dataset, you should then be able to use that dataset in your ETL.

When you say updated by API, are you wanting to append rows? Join new columns? Replace rows?

user15634

I cannot find how to make the first option work in my DOMO account. I'm doing the option 2 right now. Luckily I was able to export all data into a file. Even though it is a large file, the download was successful. Now I'm splitting the file into smaller pieces and uploading them.

To your another question, my update operation only append rows. not change columns or something.

ST_Superman

If you are just appending rows, can't you just use a POST call?

function uploadFile(name, description="", isPublic=true, file){
  const formData  = new FormData();
  formData.append('file', file);
  const url = `/domo/data-files/v1?name=
             ${name}&description=${description}&public=${isPublic}`;
  const options = { contentType: 'multipart' };

  return domo.post(url, formData, options);
}

https://developer.domo.com/docs/dev-studio-references/files-api

user15634

What is "domo" in "domo.post" in your code?

I'm using Java Stream API to append rows. I cannot create a stream on a dataset created by ETL. What I read on Internet is that I can only create a stream on datasets create by Dataset API.