Commiting data before update in dataflow

user09600

Hi, i would like to know if there is a way to commit data before getting update in

dataflow.

We have use kafka to push records to dataset from file.  with kafka we can set up how many records we expect so before pushing to dataset we can confirm  the number of records in the file and commit it .

Now we would like to avoid having file and push in kind of constant stream the data to our dataset/dataflow.

Let's say our stream data consist of 100 records.

If only 50 records get consume and 50 records are delayed how we can avoid data to be pushed in dataset until all 100 records are ready?

jaeW_at_Onyx

IMHO, easiest way to accomplish that would be to build out that logic in scripting and then at the end of the script trigger data transfer into Domo.

 

The JavaCLI would be a handy tool for pushing data to Domo

 

This is the official documentation here

https://knowledge.domo.com/Administer/Other_Administrative_Tools/Command_Line_Interface_(CLI)_Tool

In this tutorial video, I show off one of the many functionalities built into the CLI, renaming datasets and altering schemas.

https://www.youtube.com/watch?v=9HYx1vn3HRM

 

The CLI command could then be integrated into your PowerShell, Python, java script.

 

Alternatively, you can trigger WorkBench jobs via the command line:

https://knowledge.domo.com/Connect/Connecting_to_Data_Using_Workbench_5/03Running_Workbench_5#Running_Workbench_from_the_command_line ... so .. same idea just with WB.

 

Hope that helps!