Commiting data before update in dataflow
Hi, i would like to know if there is a way to commit data before getting update in
dataflow.
We have use kafka to push records to dataset from file. with kafka we can set up how many records we expect so before pushing to dataset we can confirm the number of records in the file and commit it .
Now we would like to avoid having file and push in kind of constant stream the data to our dataset/dataflow.
Let's say our stream data consist of 100 records.
If only 50 records get consume and 50 records are delayed how we can avoid data to be pushed in dataset until all 100 records are ready?
Comments
-
IMHO, easiest way to accomplish that would be to build out that logic in scripting and then at the end of the script trigger data transfer into Domo.
The JavaCLI would be a handy tool for pushing data to Domo
This is the official documentation here
https://knowledge.domo.com/Administer/Other_Administrative_Tools/Command_Line_Interface_(CLI)_Tool
In this tutorial video, I show off one of the many functionalities built into the CLI, renaming datasets and altering schemas.
https://www.youtube.com/watch?v=9HYx1vn3HRM
The CLI command could then be integrated into your PowerShell, Python, java script.
Alternatively, you can trigger WorkBench jobs via the command line:
https://knowledge.domo.com/Connect/Connecting_to_Data_Using_Workbench_5/03Running_Workbench_5#Running_Workbench_from_the_command_line ... so .. same idea just with WB.
Hope that helps!
Jae Wilson
Check out my 🎥 Domo Training YouTube Channel 👨💻
**Say "Thanks" by clicking the ❤️ in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"0
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.6K Connect
- 1.2K Connectors
- 300 Workbench
- 6 Cloud Amplifier
- 9 Federated
- 2.9K Transform
- 101 SQL DataFlows
- 622 Datasets
- 2.2K Magic ETL
- 3.9K Visualize
- 2.5K Charting
- 748 Beast Mode
- 59 App Studio
- 41 Variables
- 686 Automate
- 176 Apps
- 453 APIs & Domo Developer
- 47 Workflows
- 10 DomoAI
- 36 Predict
- 15 Jupyter Workspaces
- 21 R & Python Tiles
- 396 Distribute
- 113 Domo Everywhere
- 276 Scheduled Reports
- 7 Software Integrations
- 125 Manage
- 122 Governance & Security
- 8 Domo Community Gallery
- 38 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 109 Community Announcements
- 4.8K Archive