Best Of

Howdy all. Love the convo!

At Turo I did two things that dramatically reduced ad-hoc requests:
1. Determined which tables in DB were used to answer the mass of the requests (it was like 5 core tables) & then just add those as a table on an App Studio page with the top 5-6 filters at the top of the page. People just use that to aggregate, filter and self-enable. Access only to those we were ok downloading CSVs. The majority of ad-hoc led to CSV then Pivot Tables.
2. I created a COE of Data Champions in each department & taught them how to build in Domo + best practices on validation & hygiene.

There's nuance around the size of tables & how many columns… but these two things dropped adhoc from around 20-40 requests per month down to single digits.

One of my favorite things about being a solutions engineer and former Domo customer is getting to see how different organizations use Domo on a daily basis. I can confidently say AI features such as AI chat, AI service layer endpoints, and the agent tile in workflows can help with ad-hoc analysis and data discovery. I also think some core components around this conversation are (1) opportunity cost and (2) the idea that the data pie can always get bigger, so you don't need to worry too much about how it gets sliced.

(1) Opportunity Cost and Scarcity

Data teams (and really anyone in business) should be obsessed with opportunity cost, or the idea that when you spend time on one thing, you give up spending time on the next-best alternative. I'm of the belief that data teams are usually pretty lean for the amount of work they are expected to do. A lot of this work will be ad-hoc analysis, but not all ad-hoc analysis is created equal. If my entire job were just answering questions like "How many x did we do last year?" or "Who did the most y?", I would not still be in data 😄.

In my opinion, this is an excellent use case for AI to become involved. AI chat can do a great job answering these kinds of deterministic questions, and it will never get frustrated with the amount of follow-up questions that are asked, as it has an unlimited amount of time to spend with the user and their questions. It's also nice because AI can help answer questions at any hour of the day, which can be particularly useful for global teams. I've seen examples of AI chat being used at 2 AM to help answer valuable questions that couldn't be answered by a dashboard. I don't know about everyone else, but I'm a big believer in work-life balance and firmly believe that there will always be scarcity when it comes to data needs. The question is whether you believe AI can help minimize that scarcity.

On a practical note, using AI chat + the AI Chat Sessions DomoStats report can actually help you create a sort of "data flywheel", helping you get insights into what your end-users are asking when they're not submitting tickets for ad-hoc requests. I've also found using the AI service layer endpoints + AppDB to store user sessions can provide great insights into the needs of your end-users before they even ask you anything.

(2) The Data Pie Can Always Get Bigger

When it comes to AI with data, I don't believe it needs to be an "either/or" discussion. Why do you believe if you incorporate AI, that suddenly means you won't have as strong relationships with your stakeholders or will miss out on opportunities to build connections? In practice, I have not seen this to be the case.

I'm a firm believer in focusing on value creation above all else, and that working in data actually involves much more soft skills and a strategic mindset than we sometimes like to think. So many data teams are bogged down in resolving tickets, getting the sprint done, etc. when they should be focusing on getting in the room with their partners and having strategic conversations about the business.

Can the data pie get bigger if you incorporate AI? Right now I'm leaning towards think so, but I'd love to hear others thoughts on this as well. Like I mentioned before, I've worked for several Domo customers and have tried to conduct these sort of experiments with varying degrees of success. I'm not sure exactly how to define success or how to measure it. Again, it all comes back to driving real business outcomes, and I do think AI can help you find more time to focus on higher order problems.

@DavidChurchman appreciate the thoughtful response! I'm thinking we might have a fundamental disagreement on some of these issues. I'm not here to convince you of anything, but I'm happy to share my personal experiences and pragmatic advice when it comes to rolling out AI tools (such as chat) across an organization:

• Start with a small beta group who are heavy Domo users and do heavy testing yourself
  • I love how DomoStats can give you a perfect target list of who would be good candidates for certain initiatives. In one rollout I participated in, we chose a group of our customer success managers who were hitting the same dashboard multiple times a day to ask easy questions.
  • Find out what the top 25 questions are from the beta group and make sure the chat can answer them correctly. You can leverage the new Documents feature to give your agent a knowledge base of most commonly asked questions and how they should be answered (and then update this knowledge base as more questions pop-up, making it an iterative process)
    
• Monitor questions being asked and responses given by the assistant
  • AI Chat Sessions DomoStats gives you a detailed log showing who asked the question, what sources were used, and how the assistant responded
  • When we rolled out AI chat to a global team, I was watching the chat sessions like a hawk to make sure the assistant was answering them correctly. Anytime it got it wrong, I would reach out to the user personally, and this would usually spark a great conversation. (check out the new Webhook activity log beta for close to real-time monitoring)
    
• Spend time refining your data models and AI dictionaries
  • Rolling out something like AI chat can actually be a great indicator for how robust your data models truly are. If you can't succinctly explain the definition of a column or granularity of a dataset, your data models need refinement
  • I recently read this article explaining how much better LLMs have gotten at natural language processing, especially when given a semantic layer. As a case study, I remember first trying out AI chat back in 2023 and see it struggle with the most basic questions. Fast forward to today, and it's night and day different. If you haven't looked at the updated chat that was just released, I would highly recommend it.

This is the part of your answer I'm most curious about:

I've worked for several Domo customers and have tried to conduct these sort of experiments with varying degrees of success. 

That's the sort of thing I look for when I try to discern AI-hype from AI-reality. Has the person advocating this AI tool used it in the real world, and if so, what were the pitfalls? When someone only talks about the upside, I start to get skeptical fast, because the downside can be pretty tremendous in action. I'd love to hear more about your experiments: what worked about them, and what are the pitfalls we should avoid?

I think you asked me two questions in your post, and their answers are interrelated for me. 1) Why do I think incorporating AI will hurt my relationships with stakeholders. 2) Can the data pie get bigger if I incorporate AI? (Maybe these were asked rhetorically to the broader audience, but I'll try to answer them anyways).

To start with, I'll ask my own two (rhetorical) questions: What is the number one way to lose trust with stakeholders? Bad data. Is it easier to build trust from scratch or rebuild trust after it has been broken? Build it from scratch.

So, why do I think incorporating AI will hurt my relationships with stakeholder? Because it is an unstoppable spigot of bad data. I generally know how to validate and present clean, reliable data, but I do not know how to validate every conceivable response to an AI chatbot based on my data. And I don't believe anyone can, but I am trying to keep an open mind to it eventually being possible. The risks of someone asking an AI chat bot a question and getting a bad answer are not zero. Trust is my most important asset, and I don't risk it lightly.

Do I believe that the data pie can get bigger if I incorporate AI? Surely, but I'll ask yet another rhetorical question: is a bigger pie always better? We served individual-serving pies at my wedding (neither my wife nor I like cake). I'd much rather my data looks like one of those carefully constructed pastries presented artfully by hand than a series of ample warm cowpies, dropped indiscriminately across my field. Are there opportunities that I'm missing by failing to wade through fields of muck? (Or in this scenario, serving up the field of muck to stakeholders). Possibly. Probably. But, for now, I feel more inclined to laboriously scale up my edible pies by hand than to scale up everything first, assuming I'll figure out how to make my giant pies edible later.

@damen Made you a little video I hope this is what you need:

The issue is you might want to use a variable instead of a dropdown filter because if you use a filter you are taking data out of the chart which is ruining the average.

https://www.dashboarddudes.com/newsletter-sign-up