Walking the Talk: Random Exploration of a Chatbot API

Three related coverage risks stood out when I joined a new project to build a chatbot API for Ada Health’s medical symptom checker. With an infinite space of possible chats, how could we:  

• look for unintended consequences of changes as we built the API from the ground up
• discover some of the edge and corner cases bugs that would surely exist 
• exercise the API to any significant extent before the very tight first deadline for delivery 

To help mitigate these risks I built a client which would randomly walk through dialogs, unattended, and report on what it had found. 

In this talk, I’ll describe how I implemented that client by iteratively adding functionality that I hoped would facilitate my exploration of changes and fixes to the emerging API. I’ll give examples of features that worked well (such as configuration of probabilities for different types of answers) and those that did not (such as checking for specific classes of medical outcome), explain how I built on top of the client to make a load testing tool, and think about what I’d do differently next time. 

Key takeaways: 

• automation is a tool for exploration 
• starting small, and early, and building incrementally, can be powerful 
• you can use tools to review the results of your bulk machine-generated results