My Role

Lead Product Designer

Interviews, Prototyping, Testing

Company

Amelia R&D

Context (What is Amelia?)

Amelia is a conversational agent focused on creating the most human chat agent in the market. The technology behind it is indeed impressive. When correctly trained, it can achieve an impressive 80-90 percent accuracy.

Amelia uses different machine learning models to understand user intent and return a correct answer. Each one of these models needs to be trained with different types of data. For example, some of the models require Q&A pairs, while others can digest big blocks of text. Equally, some of these models are fine-tuned to respond to human small talk (e.g., How are you doing?), while others are created to answer hard facts (distance to the moon).

When Amelia receives an utterance from the user, all the trained models fire up and, depending on the question, some will return an answer with a confidence score between 0.0 and 1.0 (0.1 = not sure at all, 9.0 = very confident). This score is then processed by an arbitration algorithm that decides which answer to pass on to the user. All this happens in milliseconds, making it feel like the response was provided by a regular person.

Lack of Datasets

Like many ML-based systems, Amelia’s achilles heel has always been the training data required to train the models correctly. The tools used to achieve this are a constellation of smaller apps, each one of them focused on compiling different types of data and normally maintained by different teams. This creates a nightmare situation where it is impossible to have visibility over all of the training data. Another problem with having a lot of disperse tooling is that it blocks the team from selling Amelia to other companies interested in using it.

This realization caused Amelia leadership to start multiple, ambitious initiatives, including aligning all the micro-services, moving to the cloud, generating multiple SDKs and designing a consolidated training tool. For this last one, I was fortunate to participate as the lead UX.

Challenges

• Consolidate multiple apps in one easy-to-use training tool.

• Design an easy-to-use search that can be used across all systems.

• Work with multiple teams, especially with ML engineers, to create the best possible solution.

Research

Qualitative Interviews

The first round of interviews I did was with stakeholders, data scientists, conversational designers and engineers. I wanted to be extremely familiar with the problem before starting to sketch any solution.

My research objectives were:

• To understand the subsystems that encompass Amelia.

• To define a hierarchy for all the tasks to be done.

• To understand the users’ backgrounds and what tools they are familiar with.

• To define the success metrics for each task.

Insight

The early insight showed that:

• ❌🔍 Locating training data is extremely difficult to achieve. Sometimes engineers need to rely on querying database directly (not scalable).
• ❌ ⚡ The learning curve is very steep for newcomers, especially because all the apps are very different. This leads to users only using modules they are familiar with, but not learning new ones.
• ❌📔 The documentation is practically nonexistent—only some loose docs that engineers pass around.
• 👀 External collaborators (e.g., Mechanical Turk) are very common, making it important to limit access with roles.

User Profiles

• Computer Science, Mathematics
• Uses: VSC, Kaggle, Tensor Flow, Jupyter.
• Tasks:
  • Fine-tuning ML with existing datasets, designing challenges for data validators, running experiments.
• Pain points:

• Computer Science, Engineer
• Uses: Github, Visual Studio Code, Jira, Terminal.
• Tasks:
  • Maintaining the apps running Amelia, creating backups and deploying new trained models.
  • Troubleshoot users and help data scientists with tooling.
• Pain points:

• Any Bachelor’s Degree
• Uses: Chrome, Office, Social Media.
• Tasks:
  • Participating in experiments as the ‘human in the loop,’ going through big datasets and helping with the disambiguation of data.
• Pain points:

Iterations

I first printed all the screens on good, old-fashioned paper and pinned them to the wall. The only thing that unified them was the UI style, but they all looked like completely different applications. It was clear that every app was conceptualized separately by different teams.

It was also evident that they all shared things in common. For example, they all had libraries, a section for properties, metrics and a workspace to do most of the work.

I recalled an article about the 30-year IDE evolution and how Visual Studio Code was the pinnacle of dozens of experiments about the best way to organize code.

Training models for machine learning are certainly not the same as writing code for a programming language, but these two paradigms have a lot of things in common. They both share assets and also need to handle code and parameters. Modern IDEs can provide us with valuable hints on how to deal with this complexity from a design point of view.

The Solution

After getting the project manager and stakeholder to greenlight the IDE idea, my mission was to design something that was easy to use, but overall make all of our users productive under the same interface. To be sure about this, we conducted several usability tests before landing on a version that had a good balance between usability and complexity. This final solution included four main blocks: Library, Tools, Workspace and Properties.

An important thing to mention about this solution is that it behaves as an app and not like a website. Some of the previous apps in Amelia were designed using a ‘page’ paradigm where you need to navigate to different sections to get things done. The problem with this approach is that you need to save your progress before moving to a different page. In this new app, everything happens in the same place. Creating, editing and deleting gets updated in real time. And because now everything is based on the HTML canvas, complex behaviors like drag-and-drop and real time collaboration (and a lot more) are enabled.

The scope of this case study only scratches the tip of the iceberg of what this redesign represented for the team in terms of work, discussions and continued testing with final users. But it was this the catalyst of important conversations around every important aspect of the app. The final solution included things like in-context documentation, debugging, search and roles.

We chose a submodule called BPN to be the first one to be migrated to the new layout, particularly because it was one of the most complicated ones. We then continued with an NLP Debugger, a Conversation Tracker and an Asset Explorer.

Finally, the layout I’m most proud about was the Search Feature that was capable of searching across all the training data. I’m proud about this because it required minimum changes on the backend supporting it. The design decision to put all the libraries in the same place was the key to making this happen.

Results

• Amelia went from 5 to more than 200 enterprise customers in less than 2 years.

• More than 3,000 new users.

• 400% more datasets trained.

• Established design as a pillar of innovation.

Project Learning

• It is crucial to get stakeholders to buy into the project early, especially when it will impact a large part of the business.

• Continuous testing and reliance on data are the best ways to settle two different points of view.

• Any company works in isolation. Company environments are unique and can be insulated. It's a good idea to spend some time exploring their ecosystem and the tools your user is familiar with.