Grocery App

Multi-Modal Voice App Design, Voiceflow, LLM Prompt Engineering

This is an in-progress personal project featuring early stage design concepts for a voice/multi-modal application.

So far, I have mostly focused on the "Recipe Finder" portion of this app.

More to come soon!

Intro

The concept for the larger application experience for this project is an application to assist with creating and managing Grocery Lists.

Some example voice paths I envisioned for this app:

create a new list
hear existing lists
modify an existing list
- add item(s)
- remove item(s)
- change existing item(s)
rename existing list
delete existing list
combine 2 or more existing lists

Survey Research

I conducted a brief survey to gather some initial information about people's grocery shopping and cooking habits, and how they went about finding recipes and deciding what to cook.

I noticed some patterns within the survey response data. A number of people mentioned they consider what ingredients they already have when deciding what meals to cook.

Additionally, I asked survey participants to select criteria that was the most important to them when looking for and selecting recipes to cook for their household.

This question had 6 criteria to select from:

a specific ingredient (Ex: "chicken recipes")
healthfulness (Ex: "healthy dinners")
type of dish (Ex: "eggplant parmesan")
type of food (Ex: "Mexican")
ingredient(s) that I know I already have
Other - in your own words

Survey participants were able to select multiple criteria, or select "other" and enter their own.

60% of folks selected "a specific ingredient", which was the most-selected criteria choice. The next most-selected was "ingredient(s) that I know I already have", which was chosen by 40% of participants.

User Persona

I created a preliminary user persona based on these survey findings.

JOB TO BE DONE

Cook a tasty meal using groceries in the house

JOB TASKS

make grocery list
- inventory pantry and fridge
buy groceries
cook meals

ENVIRONMENT

Imagine Pam standing in front of her fridge or pantry, trying to think of a recipe ...

Recipe Finder

So, that leads me to the concept of a "Recipe Finder" tool that could live inside this Grocery List Manager App.

This tool allows a user (like Pam) to get some recipe ideas based on an ingredient they'd like to use.

Conversation Flow (Simple)

1-INGREDIENT

General Flow

Below is a diagram illustrating the general one-ingredient, "happy path" flow of the recipe finder concept.

Voiceflow Prototype

Below is a Voiceflow diagram/prototype that demonstrates a happy path for a simple (phase 1) version of this recipe finder tool. In this flow, the user prompts the assistant for recipes by specifying just one ingredient.

Click to view larger.

Feel free to interact with this voice prototype -- click below to open it in separate window.

View VoiceFlow Prototype

GPT Prompt Engineering

Two of the steps in this prototype flow involve passing along information from the user's request to GPT prompt and getting that generated response back.

Generate 3 Recipes

Temperature:

0.20

GPT Engine:

GPT-3.5 Turbo

This step prompts GPT to generate three recipe ideas using the user's desired ingredient. I wrote specific system instructions and defined temperature.

System Instructions:

You are a recipe-finder tool that can give recipe recommendations based on a desired ingredient.
Users will tell you at least one ingredient they want to use in a recipe.
Generate three titles of recipes that use their specific ingredient, reply to the user with the list of recipe option titles only.

When listing out the recipes you generated, do not number the list. Instead, state them in a list like the following: “One recipe idea is broccoli and cheese soup. A second recipe is broccoli stir-fry. And a third recipe idea is broccoli stuffed pasta.”

Don’t include anything else in your response. Be as conscise as possible.

Generate Recipe Summary

Temperature:

0.20

GPT Engine:

GPT-3.5 Turbo

This step prompts GPT to generate a summary of the recipe that the user selected.

GPT is very good at understanding what a user means, even when they don't repeat word-for-word the recipe title. (e.g. "the soup", instead of "Broccoli and Cheddar soup")

In this step, I experimented with including instructions in the prompt rather than in the system content.

Conversation Flow (Complex)

>1 INGREDIENT

I think that a more robust solution, and better user experience, for this recipe finder tool would allow the user to be able to provide more than one ingredient to be included in each of the generated recipe ideas.

The ingredients the user gives can be understood as a list object, its length varying depending on how many ingredients are identified in the user's request.

Icons created by Vectors Market, Muhammad Atif, Nsit, Freepik, Luvdat - Flaticon

GPT Assistant

For this more complex flow, I created a GPT assistant using open AI.

System Instructions

The prompt and system instructions I wrote for this assistant differ slightly from the VoiceFlow prototype.

Here are a few key authoring choices:

"... give recipe recommendations based on a list of desired ingredients"
"Users will tell you at least one (but could be many) ingredients they want to use in a recipe. "
"Generate three titles of recipes that use their specific ingredient(s) ..."

System Instructions:

You are a recipe-finder tool that can give recipe recommendations based on a list of desired ingredients.
Users will tell you at least one (but could be many) ingredients they want to use in a recipe.

Generate three titles of recipes that use their specific ingredient(s), reply to the user with the list of recipe option titles only. Don't include anything else in your response. Wait for the user to reply with which recipe they want.

When the user selects one of the recipes you've provided, generate a summary of the user's desired recipe including the Title, approximate cook time, and list of required ingredients. Reply with only that summary; don't include any other content in the response.

Assistant - OpenAI Playground

I used the "Playground" in "Assistants" mode to test out my instructions and arrive at a version that achieved a pretty close user experience to what I was able to achieve in VoiceFlow.

See screenshot below for a 'happy path' conversation example.

Temperature:

default

GPT Engine:

GPT-3.5 Turbo

Click to view larger.

Assistant - Python OpenAI API

I also created an assistant bot with OpenAI's completions API using a python script.

Here's a video of a chat with that bot below via my terminal. (Hopefully one day I'll integrate a chat box on this page!)

Temperature:

0.20

GPT Engine:

GPT-3.5 Turbo

Next Steps

As with most personal projects, this project is on-going.

The next steps I intend to pursue for this project:

Do some W.O.Z. testing to validate content
- GPT's recipe summary would certainly need to be adjusted for a voice response
- Opportunity here to provide a "style guide" to GPT to inform word choice, etc.
Design "Repeat" Flows
- I think one of the most important non-happy-path flows to start with is if the user needs to have either the 3 recipe titles or the selected recipe's summary repeated.
- Though the intention for the larger app concept has a multi-modal experience in mind, one of the intended use cases for the recipe finder tool is when user is standing at their fridge/pantry, AKA not in front of a device.
  - Voice experiences should always provide avenues for having information repeated.
Additional User Research
- Execute another round of survey research to gather insight on what components of a recipe are the most important to folks when choosing what to cook
  - The insights of this research will better inform what the GPT bot should include in its recipe summary.

BACK TO HOME

Recipe Finder

Next Steps