background.png

PwC / Larson Co.

Performance Management App

Sept 2016 to Feb 2017

 

THE PRODUCT

My team and I launched a pilot with dozens of field technicians. We built out a mobile performance insight app that connected with a desktop app for the manager.

The result was a 19% decrease in return rates (their #1 most impactful metric).

For the field technicians, we were trying to solve problems revolving around lack of performance feedback, insufficient information to improve, and general sense of isolation. 

For the managers of the field technicians, we were trying to solve a problem of managing a large staff of traveling field technicians all across the North-Eastern part of the United States. From communicating expectations to seeing trends and patterns, we built a tool to give them deeper insight into their workforce.

 
 

INTRODUCTION

This was a 6-month research and product exploration project with my client PwC and their partner-client Oscar W. Larson Co. (Larson).

Larson is a leader in full-service petroleum and fluid handling equipment contracting with a yearly revenue of almost $100 million. They have over 300 field-technicians who work across the Midwest.

While initially tasked to explore a gamification angle to improve motivation and performance of field technicians, understanding and uncovering problems that these technicians had were the focus of this project.

My role was the lead Product Designer and led all research, UX design, UI design, and some front-end styling work.

 
 

THE CHALLENGE

The project's challenge can be summed up with these two 'how might we' statements:

  • How might we increase the efficiency of the technicians and managers in order to provide the best customer service and solutions?
  • How might we extrapolate the learnings and product to apply it to different verticals such as retail?
 
 

DISCOVERY PHASE

Learning about our users

The first thing we wanted to do was dive deep into the technicians’ world in order to better understand their problems and pain-points. Contextualizing their world and familiarizing ourselves with it was a crucial step in our process. Creating relationships with our users allowed for us to empathize and have them at the focus of the design process.

We kicked this process off by doing a few things:

1) Problem Interviews:

We spent the first week interviewing a handful of field technicians and their managers. They walked us through their day-to-day which afforded us to dig a little bit deeper into their workflow. Many gave us unprompted remarks about inefficiency or problems they ran into. When enough of these came up, we started seeing a pattern. 

2) Watching field-study ethnographic research videos on the field technicians:

We went through a tonne of footage that was taken with a few field technicians using a Go-Pro rig. This helped us get a detailed understanding of what their day looked like, who they interacted with, and what type of work they did.

3) Learning about their workflow + interactions:

We mapped out our entire understanding of the user into an Experience Map (heavily influenced by Adaptive Path’s experience map model). This helped us consolidate all our learnings and assumptions into one board. Key actions that the user took, important touch points, the emotional state of each interaction, were all posted on a wall. From here, we had the whole team come up with questions, and pinpointed our assumptions, possible opportunities, and areas where we had a gap in knowledge.

4) Quick feedback loop:

We showed the technicians and their managers rough designs of possible ideas like points, leaderboards, weekly/daily goals, etc. This gave a canvas for open ideas and responses we could test with the field technicians. This was a great way for us to get quick ‘knee-jerk’ reactions to things the managers and our clients thought would work. For example, when showing things like a ‘points system’ (something our client thought would work), it got us very mixed responses.

 

Example of an experience map from Adaptive Path.

We used the ‘rainbow spreadsheet’ method of tracking user research. This was a productive way to track numerous participants. It also forced us to think beforehand, what type of behavior we were expecting based on our hypothesis, thus keeping our analysis honest.

Click to learn more about rainbow spreadsheet methodology


Empathizing with our users

Most of our communication and testing were done through voice or video calling. This presented a difficult barrier to both connect with the user and conduct thorough user research. I was reading Steve Krug's book on user testing and a quote really resonated with me at the time:

Watching real users gives them that eureka moment: they’re not all like me, and infact they’re not all like anybody
— Steve Krug, Rocket Surgery Made Easy

While this was a barrier it also presented us with an opportunity to really get immediate feedback from the user within real-world contexts. For example, when we had a technician willing to test with us at the end of his week, we got his immediate feedback and response to seeing his performance numbers literally onsite. We also got to see him react and reflect on his jobs he completed that day while still in his working environment.

Talking to the users about their performance metrics were also a very sensitive subject. I made extra precautions and preparations to make sure that I approached them with an inquisitive mindset and not one that could be perceived as judgemental. For example, instead of asking "Why was your return rates so high?" I would wait for them to react to the numbers and ask "Why are you seeing here?" or "So you're noticing the return rate, what about it? So you think that its too high, why is that?". 


Findings and decisions

After the initial research dive, we distilled all our learnings into an opportunity assessment. Before exploring how gamification can be used in this setting (something our client was pushing for), we communicated the learnings that we gained and attempted to reframe the problem. Instead of slapping on ‘PBLs’ (points, badges, leaderboars) and hoping for increased productivity, we learned that the following were the underlying factors in productivity and the biggest opportunities to find a solution for:

Reactive firefighting:

Managers and technicians had poor visibility into the technician’s on-site performance. Instead of being proactive and being able to tell ahead of time, they would only find out when it was too late. One manager said, “When a customer calls me complaining... it’s like ‘gosh, I wish we could catch it before he gets so mad and calls in and complains,’ you know?”

Babysitting:

Managers spent an exuberant amount of their day tracking and figuring out what their field technicians were doing and how things were going. 

Sisyphus at night: 

Many field technicians shared common problems of isolation, a lack of concrete goals, and the absence of an effective feedback loop.

Suboptimal local maxima:

Without a longterm view of the team’s performance, it’s difficult to spot opportunities for growth, both professionally for workers and strategically for the whole team. Instead, managers rely on instinct, anecdotes, and whatever stands out to them at the moment. For example, maybe a favored worker’s recent “rough patch” isn’t just bad luck; perhaps it’s a larger trend that indicates a need for training? Or worse, perhaps the team’s quiet, but consistent workers don’t get the recognition they deserve - a consistent performer might never be in the top five for a single week, but over the course of the year, they could be a hidden top performer.

So we reframed the problem from ‘how can we incorporate gamification into the technician’s work’ into ‘how might we improve productivity by solving existing painpoints’.

Our main goals were to:

  1. Increase visibility/insight into the job performance of the field technicians to the manager and themselves.
  2. Increase productivity, in terms of reduced return rates, increased jobs completed, and reduced job duration.
  3. Test out a gamification layer to see viability.

Early prototype we used to test and validate our direction.

 
 

HOW WE GOT THERE

Weekly prototyping and interviews

We worked on one-week sprints that generally followed this schedule:

Monday: consolidate research; update our ‘source of truth’ documents about the users, journey, and hypotheses. Determine what we need to learn and what the biggest unknown/risks are. 

Tuesday: ideate on possible experiments / prototypes we can test with based on Monday’s work. 

Wednesday: iterate on previous designs and create new design wireframes/mockups for new ideas/tests

Thursday: build prototype

Friday: test prototype with users.

We stuck with this cadence to keep us flexible and agile. There were so many unknowns and new learnings each week that we had to be in the best position to react to them. 

One of the most important parts of the design process is to highlight risks and assumptions. Testing the biggest risks and assumptions early on will significantly increase the success of the project. In this case, it wasn’t just about telling a field technician how they are performing, but trying to get a deeper understanding of which metrics were important and why.

For example, we tested various metrics early on and through feedback from both the manager and the field technicians, we were able to dwindle the first iteration down to four key metrics: jobs completed, return rates, job duration, and hours worked. 

Some managers wanted to show specific metrics but when we shared those with the field technicians we got a lot of pushback that they were not ‘fair’ or a good representation of their output. It was crucial that we got involvement and buy-in from the technicians about which metrics they felt accountable to. We got to that point because we found that they were much more vocal and self-reflective about metrics they knew they had control over. With one of our ultimate goals being to increase productivity, finding the right metrics the field technicians would rally behind was an important part of the research process.


 

This did not work as the 'points' were too far removed within the context of their performance.

They responded far better to direct data visualization of their performance within context of its metric + timeframe. 

 

We learned that we can’t just show them a stat. It needed to have context, meaning, and motivation behind it. So that’s why we decided to show them four weeks worth of data for each metric so they could see their own progress. We also tested their company’s average to show context of how their work stacks up.


 
 

After another couple rounds of testing we found that while the bar graphs were a good start to visualize their performance, it did not take into account the nuances of their work week. For example, an installation job could take up to a whole day, versus 2-3 hours for a ‘pump job’. If a field technician were assigned to 5 installation jobs for the week, his total jobs completed for the week would be only 5, while someone who is only assigned to ‘pump jobs’ could easily finish 15 in a week. We decided to break the bar graphs into color-coded sections to give a visual indicator of the details of each week.


 

The devil is in the detail

 

There was a recurring pattern for requests to ‘see more information’. While the stacked-bar-graphs helped give a visual representation of the diverse types of work done, it still failed to fully satisfy the field technicians. 

So we dug deeper into not what the field technicians were asking for, but why they needed more information—what were their desired deeper outcomes? Our continual user research cadence allowed us to uncover these shared needs:

  • The need to reduce mistakes that caused returns (i.e. return trips to the same locations). 
  • The emotional aspect of having a tool to assist in conversations with management.
  • To uncover where they might need more training and education - i.e. a desire to improve.

We designed a ‘detail view’ that broke down each major metric into its individual jobs. Within each job, we outlined what type of job it was, how long it took (as well as if it took longer or shorter, on average), if there was a return needed, and grouped them up into categories that made sense to the field technicians (e.g. ‘Construction jobs’, ‘Service-Quick jobs’, etc…). 

This gave them the ability to:

  • Identify training opportunities. Technicians could see at a weekly, monthly, and yearly level which jobs they consistently had returns for. This would bubble up to the surface which jobs they needed more training in.
  • Surface areas of improvement. For example, if their overall job duration time was ‘average’ they might think they’re doing fine. However, the detail view could show them that while they excel at ‘pump jobs’ they could still be struggling with ‘verifone repairs’. If they improved their ‘verifone repair’ jobs, their job duration could go from ‘average’ to well below the company average.
  • Be empowered to advocate for themselves. By giving access to this information, the field technicians are now empowered to have detailed discussions with their managers during their reviews. 
 
 

Connecting managers with technicians

Early flow and direction of the 'Manager View' that we used to test with the Larson managers.

The field technicians were only one half of the equation. The people who managed these field technicians had a different set of needs, personalities, and behaviors. We built them a tool that would integrate all of the information so that they would be able to see high level patterns on their team’s performance, but also be able to zoom in on any individual technician to diagnose a problem in performance.

We did weekly usability and feature tests with these managers. We got very positive responses and they were thrilled that they were so intimately part of the process and that their needs and requests were being taken into account.

We conducted remote user-testing with the managers on a weekly basis.

This is amazing...this lets us be on the site without having to be there!
— Larson Manager

The biggest signal that we were onto something that provided value to the managers were when we tested with managers they would ask their directors if they could get access to the beta or pilot.

 
 

TEAM GOALS

Early on we tested an idea of ‘levels’ or ‘jobs points’ based on key metrics. Our stakeholders loved the idea because they wanted a quick way to quantify each technician. Through user testing, we got major pushback from some of the field technicians while a small handful loved the idea. We dove into it more and discovered some interesting learnings:

  1. Overall, it was very difficult to have a points/level system. If the system was simple enough to understand, it would not be complex enough to be considered ‘fair’ in the eyes of the field technicians. If it were complex enough to account for nuances and the complexity of what constitutes a ‘productive’ day, it was too complicated and the field technicians did not trust it.
  2. Some field technicians were very into the idea. They were generally the newer generation of field technicians who were open to new things and a bit more competitive but also were very high performing. We decided to focus a lot of our user tests with this group. The reasoning for this was two fold: we hypothesized that we would learn more and get more return out of increasing the productivity of the top 10% of field technicians rather than trying to build a tool that would be accepted by everyone; secondly, we hoped that they would be the product’s champion within the organization to help spur engagement with those who were less open to new tools and ways of doing things.

We had a higher level goal from our stakeholders to apply gamification principles. As stated earlier, we thought that just slapping on ‘PBLs’ (points, badges, and leaderboards) without understanding our users, empathizing with them, and giving us a deeper understanding of their behaviors would not have resulted in a successful product.

After all our research and user testing, we then started the process of what a gamification layer to this app could be.

Our first take at this problem was to find out how they responded to the idea of ‘goals’ and what would ‘motivate’ them.

Our research clearly showed that while individual success was important, being part of a larger team and working together was far more desirable to the field technicians. After all, they can go weeks without working with someone else in very remote areas of the country.

So our first take on this was to test the concept of ‘Team Goals’ around a metric the users thought would make the most sense to ‘compete’ on: jobs completed.

This also provided value to the managers as they could communicate team-wide, which metrics needed improvement.

This is an example of different visual treatments and ideas of adding a ‘Team Goals’ layer to jobs completed. 

 
 

THE PILOT + RESULTS

We launched a pilot and gave access to this product to the field technicians. We followed our pilot group against a control group.

One key result was a 19% decrease in return jobs (which were the company’s #1 problematic metric).

Our research and designs heavily influenced another team’s project in applying this product to the retail vertical and later on to a team performance software tool in the UK. 

 
 

CONCLUSION - WHAT I LEARNED

Empathize and test with the user as much as possible : 

I am proud that we were able to have a weekly user-test cadence. This gave us qualitative and quantitative data to help shape the product, communicate with stakeholders when they were clamoring for superficial features, and de-risk concepts early on.

Communicating this to our stakeholders was difficult at times. At one point, one stakeholder took us into a room and berated us for spending a significant portion of our time planning and conducting user interviews/tests. It wasn’t easy, but we stayed true to our user-centered design process.

A good product is not good UI, its one that provides value:

As a designer, it is easy to fall into the trap of making beautiful UI and interactions. But if it doesn’t provide value, then it’s a useless product. Constraining ourselves to  the material-design language afforded us time and resources to do more user research and testing that ultimately lead us down a path of searching for value-add not dribbble-fodder.

To quote a hilarious and truth-filled rant from Tricia Wong

Focus on value creation. Design enhances value, it does not create it. Stop creating shitty startups that look amazing. A product or service that is indispensably useful yet looks like ass is infinitely more likely to be successful than a product that solves zero problems but looks like a work of art.
— Tricia Wong