Monitor Dataflow in Card

magicdust
magicdust Member
in Connectors

Is it possible to monitor number of dataflow, failed data flow, sucessfully ran dataflow per day in a card? In short, is there any monitoring tool available for dataflow?

 

Best Answer

  • SeanPT
    SeanPT Contributor
    Answer ✓

    I've suggested we get a Domo connector for Domo so I can get data on my data. If they call it the Yo Dawg connector it will just make my day.

     

    For monitoring I use a combination of things, most importantly I've setup alerts on all my dataflows. That is critical for monitoring for me. (I guess I could also setup a rule to basically log when I get those failure messages in and then import that data into Domo).

     

    I like the new Data Warehouse for big picture monitoring of my dataflows. I just used it this morning to find out I left workbench open like a maroon and didn't have some data upload last night.


    Between those two things I feel like i can monitor things pretty well but I would love to be able to get data on the data like I said. I would love to know how much data I'm pumping into Domo and how much that is growing. 2016 will be our first full year of Domo and by this time next year I'll really want to know more. Since I use Workbench and it generates log files I could scrape that and generate the data I need but it would be nice to just have that done for me.

Answers

  • SeanPT
    SeanPT Contributor
    Answer ✓

    I've suggested we get a Domo connector for Domo so I can get data on my data. If they call it the Yo Dawg connector it will just make my day.

     

    For monitoring I use a combination of things, most importantly I've setup alerts on all my dataflows. That is critical for monitoring for me. (I guess I could also setup a rule to basically log when I get those failure messages in and then import that data into Domo).

     

    I like the new Data Warehouse for big picture monitoring of my dataflows. I just used it this morning to find out I left workbench open like a maroon and didn't have some data upload last night.


    Between those two things I feel like i can monitor things pretty well but I would love to be able to get data on the data like I said. I would love to know how much data I'm pumping into Domo and how much that is growing. 2016 will be our first full year of Domo and by this time next year I'll really want to know more. Since I use Workbench and it generates log files I could scrape that and generate the data I need but it would be nice to just have that done for me.

  • kshah008
    kshah008 Contributor

    @magicdust, did SeanPT's response help answer your question?

  • magicdust
    magicdust Member

    @kshah008 Yup

  • magicdust
    magicdust Member

    @SeanPT how do you process log files to get how much data was sent to domo? Because I had a quick look on the logs and the info I found was regarding nuber of data rows sent, etc and how much MB/GB  data was sent.

  • SeanPT
    SeanPT Contributor

    @magicdust - so far I'm doing it a couple of different ways.

     

    1) Most of my data is coming in from MS SQL so when I run a workbench job that appends data I also write back to a log table I created to let me know which records were uploaded and when.

    2) I parse the log files in C:\ProgramData\DomoWorkbench for the jobs really care about and look for errors in the data (like sometimes it will find an invalid character). I'm using powershell in windows to do all of that heavy lifting.

     

    Scripting is probably the real answer to your question if you are using Workbench. If you know how many rows of data were sent AND how many MB you have two different measurements you can use. I care more bout the rows of data sent and since on my biggest datasets I have to replace each day I like being able to see how much I'm uploading each day and how much that is growing.

     

    Again, I really feel like that data should just be available in domo though.

Categories