Monitor Dataflow in Card
Is it possible to monitor number of dataflow, failed data flow, sucessfully ran dataflow per day in a card? In short, is there any monitoring tool available for dataflow?
Best Answer
-
I've suggested we get a Domo connector for Domo so I can get data on my data. If they call it the Yo Dawg connector it will just make my day.
For monitoring I use a combination of things, most importantly I've setup alerts on all my dataflows. That is critical for monitoring for me. (I guess I could also setup a rule to basically log when I get those failure messages in and then import that data into Domo).
I like the new Data Warehouse for big picture monitoring of my dataflows. I just used it this morning to find out I left workbench open like a maroon and didn't have some data upload last night.
Between those two things I feel like i can monitor things pretty well but I would love to be able to get data on the data like I said. I would love to know how much data I'm pumping into Domo and how much that is growing. 2016 will be our first full year of Domo and by this time next year I'll really want to know more. Since I use Workbench and it generates log files I could scrape that and generate the data I need but it would be nice to just have that done for me.2
Answers
-
I've suggested we get a Domo connector for Domo so I can get data on my data. If they call it the Yo Dawg connector it will just make my day.
For monitoring I use a combination of things, most importantly I've setup alerts on all my dataflows. That is critical for monitoring for me. (I guess I could also setup a rule to basically log when I get those failure messages in and then import that data into Domo).
I like the new Data Warehouse for big picture monitoring of my dataflows. I just used it this morning to find out I left workbench open like a maroon and didn't have some data upload last night.
Between those two things I feel like i can monitor things pretty well but I would love to be able to get data on the data like I said. I would love to know how much data I'm pumping into Domo and how much that is growing. 2016 will be our first full year of Domo and by this time next year I'll really want to know more. Since I use Workbench and it generates log files I could scrape that and generate the data I need but it would be nice to just have that done for me.2 -
@magicdust, did SeanPT's response help answer your question?
0 -
@magicdust - so far I'm doing it a couple of different ways.
1) Most of my data is coming in from MS SQL so when I run a workbench job that appends data I also write back to a log table I created to let me know which records were uploaded and when.
2) I parse the log files in C:\ProgramData\DomoWorkbench for the jobs really care about and look for errors in the data (like sometimes it will find an invalid character). I'm using powershell in windows to do all of that heavy lifting.
Scripting is probably the real answer to your question if you are using Workbench. If you know how many rows of data were sent AND how many MB you have two different measurements you can use. I care more bout the rows of data sent and since on my biggest datasets I have to replace each day I like being able to see how much I'm uploading each day and how much that is growing.
Again, I really feel like that data should just be available in domo though.
1
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 297 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.9K Transform
- 100 SQL DataFlows
- 616 Datasets
- 2.2K Magic ETL
- 3.8K Visualize
- 2.5K Charting
- 729 Beast Mode
- 54 App Studio
- 40 Variables
- 678 Automate
- 173 Apps
- 451 APIs & Domo Developer
- 46 Workflows
- 8 DomoAI
- 34 Predict
- 14 Jupyter Workspaces
- 20 R & Python Tiles
- 394 Distribute
- 113 Domo Everywhere
- 275 Scheduled Reports
- 6 Software Integrations
- 121 Manage
- 118 Governance & Security
- Domo Community Gallery
- 32 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 108 Community Announcements
- 4.8K Archive