Commiting data before update in dataflow
Hi, i would like to know if there is a way to commit data before getting update in
We have use kafka to push records to dataset from file. with kafka we can set up how many records we expect so before pushing to dataset we can confirm the number of records in the file and commit it .
Now we would like to avoid having file and push in kind of constant stream the data to our dataset/dataflow.
Let's say our stream data consist of 100 records.
If only 50 records get consume and 50 records are delayed how we can avoid data to be pushed in dataset until all 100 records are ready?
IMHO, easiest way to accomplish that would be to build out that logic in scripting and then at the end of the script trigger data transfer into Domo.
The JavaCLI would be a handy tool for pushing data to Domo
This is the official documentation here
In this tutorial video, I show off one of the many functionalities built into the CLI, renaming datasets and altering schemas.
The CLI command could then be integrated into your PowerShell, Python, java script.
Alternatively, you can trigger WorkBench jobs via the command line:
https://knowledge.domo.com/Connect/Connecting_to_Data_Using_Workbench_5/03Running_Workbench_5#Running_Workbench_from_the_command_line ... so .. same idea just with WB.
Hope that helps!Jae Wilson
Check out my 🎥 Domo Training YouTube Channel 👨💻
**Say "Thanks" by clicking the ❤️ in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"0
- 10.5K All Categories
- 8 Connect
- 918 Connectors
- 250 Workbench
- 470 Transform
- 1.7K Magic ETL
- 69 SQL DataFlows
- 477 Datasets
- 193 Visualize
- 252 Beast Mode
- 2.1K Charting
- 11 Variables
- 80 Cards, Dashboards, Stories
- 17 Automate
- 354 APIs & Domo Developer
- 89 Apps
- 3 Workflows
- 20 Predict
- 5 Jupyter Workspaces
- 15 R & Python Tiles
- 247 Distribute
- 63 Domo Everywhere
- 243 Scheduled Reports
- 21 Manage
- 42 Governance & Security
- 174 Product Ideas
- 1.2K Ideas Exchange
- 12 Community Forums
- 27 Getting Started
- 14 Community Member Introductions
- 55 Community News
- 4.5K Archive