As the Conversational Designer on an Immersive Experience team within IBM Watson, I designed and implemented a chatbot to demonstrate how NYC.gov's 311 reporting functionality could be achieved through chat.
311 Report Bot
This chatbot was part of a multi-modal demo experience shown to clients interested in learning about the capabilities of IBM Watson's Conversation Service to decide whether to adopt the service for their own company's needs.
Note: Since the completion of this project, IBM Waston's Conversation Service has been renamed to Watson Assistant.
Touch-table Chat UI, designed and generously provided by IBM Immersive Design Lead, Jenny Woo
Discover & Define
I started the project's discovery process by understanding the users who would be interacting with the 311 chatbot, and researching the problem space to identify a design approach that best matched those users' needs.
Proto-Personas
I created two proto-personas based off of the structure of immersive demo presentations:
-
An Engagement Leader who would present the demo (the primary user of 311Bot)
-
A Client, who would be receiving the presentation (observer and/or secondary user of 311Bot)
Domain Research
I researched how NYC 311 reporting works on NYC.gov's online portal and discovered a lot of complexity in the 1) report type architecture and 2) report filing user flow.
1. Existing Report Type Architecture
-
18 main categories
-
Average of 5 "level one" sub-categories for each main category​
-
That's an averaged total of 90 categories.
Screenshot of NYC.gov's 311 report type structure.
2. Existing Report Filing User Flow
I discovered that, once the right issue type was located within the categorization system, the process for filing the actual report was also somewhat complex.
Screenshot of the steps required for reporting a 311 issue on NYC.gov.
1 WHAT
3-5 pieces of information describing the reported issue ​
2 WHERE
The full address of where the issue occurred/is occurring
3 WHO
Form to provide Name, Contact Info, & Address for the reportee
4 REVIEW
Summary of the details provided before submitting the report
After submit:
Once the report is submitted, it gets routed to the appropriate agency:
-
the NYPD,
-
the DEP (Department of Environmental Protection),
-
the DOT (Department of Transportation),
-
etc.
Design Approach
The 311 Immersive Demo was meant to be informative, but brief.
Gwen and Robert would need to observe crucial aspects about the Chatbot service through just a couple interactions with the chatbot.
Dialog IA
Simplify NYC's structure:
Use just 4-8 categories
​
​
Dialog Flow
Simplify the flow to:
1 Issue Type
2 Location
3 Route to Agency
Dialog Design
The exercises required to accomplish 311Bot's dialog design, capturing the two design approaches listed above, focused on Data Mapping and Dialog Flow design.
Classification
Since chatbots, at their core, are essentially just classifiers, highlighting the classification of report types (and keywords for some sub-categories, too) was key in demonstrating the bot's capabilities.
This was a key area for creating moments of delight for 311Bot's users.
A real user interaction with 311Bot showing a moment of delight.
Let's break it down into 3 parts:
-
Report Types,
-
Sub-Categories, and
-
Data Mapping.
1. Report Types
Since my design approach allowed me to limit the scope of how many report type categories 311Bot would need to be able to handle, I was able to focus in on defining those categories well.
Graphic showing the definition of the "Noise" category and classification of a new issue report.
Goal: Ensure 311Bot's classification of issue reports was spot-on
Design Strategy: Carefully gather and groom data for each category to ensure report type variance representation
Reasoning: The better the category's data representation, the better the likelihood that 311Bot would correctly classify reports it had never seen/heard before
2. Sub-Categories
Identification of sub-category keywords was a second, but very important, level of classification I designed into 311Bot's dialog structure.
Graphic showing the data representation of the Noise sub-category 'neighbor'.
Identifying sub-category keywords helps 311Bot to achieve 3 things:
-
​increase opportunities to delight users
-
further communicate the overall report type classification
-
successfully route report to an Agency
3. Data Mapping
In 311Bot's dialog structure, I mapped report types and sub-categories to agency routing assignments, according to NYC311's existing structure.
Graphic showing the data mapping of Issue Reports & Sub-categories to Agency.
Dialog Flow
Next, I needed to design how 311Bot's classification schema would facilitate the flow of conversation.
1. Natural Conversation Flow
As was decided for 311Bot's design approach based on domain research, the number of steps required to file a report was simplified from NYC 311's portal.
Graphic showing 311Bot's conversation flow.
2. Flow & Tree Diagrams
I created multiple types of diagrams to illustrate the overall dialog flow, and the details of each possible dialog branch.
Diagram of 311Bot's high-level dialog flow.
Tree diagram detailing the dialog steps in each dialog branch
Diagram of 311Bot's high-level dialog flow.
Diagrams depicting 311Bot's dialog flow and dialog branch details.
These diagrams were kept up to date throughout the 311 Immersive Demo Project evolved to help communicate the capabilities of 311Bot to the rest of the Immersive Experience team.
Test & Iterate
I performed several different types of testing and validation on 311Bot's data coverage, classification accuracy, and the naturalness of the conversation interactions.
User Testing
I crowdsourced validation of 311Bot's classification schema and data representation, and observed many users interacting with 311Bot to find gaps, like the one you see below.
Iteration
The best way to improve a chatbot's ability to correctly classify is to improve its data, so I added the report types that my test participants discovered as gaps to 311Bot's data.
Graphic depicting adding an issue report data point to the "Graffiti" category's data set.