Role: Project Manager, UX Researcher

Timeline: Jan. – Apr. 2018

---

Overview

As part of the Needs Assessment and Usability Evaluation course at the University of Michigan (SI*622), my team and I worked with the Sonic Research Group at Northwestern University to study the usability of its web app My Dream Team. Throughout the semester, we conducted contextual interviews, competitive analyses, surveys, heuristic evaluations, and usability tests.

Background

Members of the Sonic Research Group leveraged more than 50 years of research on intergroup dynamics to build My Dream Team.

Meant to help people form effective teams, My Dream Team was described as “more akin to eHarmony than Tinder”. That is, in a defined group (like a class), users begin by filling out a survey about themselves and their work preferences. They are then provided with recommendations about who in the group they might work best with.

The Sonic Research Group conducted preliminary usability research on My Dream Team during a pilot at the University of Buenos Aires, so our team conducted a more in-depth suite of studies, providing recommendations for improvement.

Methodology

Interviews

My team and I began by conducting a series of interviews with five people who had recently formed self-assembled teams in academia or for educational purposes. These interviews (and their protocols) were guided by the following research questions:

• What are most important factors that people consider when selecting teammates?
• What are people’s biggest frustrations with current methods of team formation?
• How do people assess the effectiveness of their teams?

After transcribing our interview recordings and analyzing and synthesizing the data, we found that

• Most people who form teams on their own, whether for curricular or extracurricular purposes, value having a social connection with and being able to trust their potential teammates.
• Relatedly, most people leveraged their social connections to find potential teammates; none had used a tool similar to My Dream Team to do this before.
• Most people indicated that taking time to evaluate the success or failure of a team is an important aspect of team formation, but most didn’t regularly do this.

Competitive Analyses

After conducting interviews, we evaluated eight other tools and websites that offer similar team-formation services and categorized them according to the following competitor parameters.

Competitor	Type
Match.com	Analogous
University of Michigan’s IMLeagues	Analogous
/r/INAT	Indirect
Applying & interviewing for a job	Indirect
DreamTeam.gg	Parallel
CATME	Partial
CollabFinder	Partial
CoFoundersLab	Partial

Then, we used the following criteria as points of comparison: target user groups, interaction patterns, and functionality.

Generally, we hoped to answer the following research questions:

• How much interaction or effort is required to effectively use the tool or website?
• To what extent do team-formation tools and websites facilitate or allow team interaction and collaboration on the tool or website?

From our analysis, we found that

• For team-formation tools and websites that have a sign-up process that takes, on average, between 31 and 60 minutes, an explicit save option or an autosave feature and indicator of such is helpful to communicate to users that they don’t have to complete the lengthy sign-up process in one sitting.
• During the team-formation process, the ability to see which people have already joined a team and to join ‘formed-but-not-full’ teams eliminates a lot of guesswork on the part of a user seeking to join a team.

Surveys

Next, we crafted a survey to learn how often people in academia and educational contexts participate in group work, who they contact when forming groups, and what types of information they’re comfortable sharing with others.

We went through two pilot versions of the survey before developing and deploying a final version to open University of Michigan listservs, as well as other targeted groups of graduate or undergraduate students, instructors, and staff members at the University of Michigan.

We received 75 responses and conducted a thorough analysis of the responses. Based upon our analysis, we found that

• More than 75% of respondents had formed teams on their own, and more than 60% formed teams by leveraging existing social connections.
• Most people were willing to share multiple pieces of personal information with their teammates, particularly their schedules and availability.

Heuristic Evaluations

Then, we conducted a heuristic evaluation of My Dream Team (specifically the log-in process and the survey phase) according to 10 heuristics described by Nielsen (1994). We adapted questions from the Heuristic Evaluation System Checklist created by Denise Pierotti to guide our evaluations, using Nielsen’s severity ratings to quantify the severity of any issues uncovered as a result of each question.

Each member of our team conducted an individual evaluation, and we then aggregated our findings to come to a common understanding of the most severe usability problems. Specifically, we found that there was a lack of consistent and clear

• System Status: Though there are forward- and backward-facing arrows/links on either side of each survey question screen, a user is automatically brought forward to the next survey question screen immediately after clicking an answer. Save for the updated progress incrementer near the bottom of each screen and the new descriptor, the next survey question screen looks identical to the previous survey question screen. These two instances of system status feedback are subtle, and thus easily overlooked, as shown in this video:

• Error Feedback: We also identified two specific examples of error feedback that did not suggest the cause of the error or indicate what actions needed to be taken for resolution, as shown below:

Usability Tests

Finally, we conducted a usability test of My Dream Team to explore the following research questions:

• Do users see the process of answering questions about themselves and their network ties as tedious or intrusive?
• Do users intuitively know how to progress through the platform?
• What information displayed on the interface influences a user’s choice in teammate?

We asked each of our five participants, recruited through our team’s personal social networks, to imagine themselves in the following task scenario and then to complete the two tasks that follow using My Dream Team.

Task Scenario

Assume you’re enrolled in a course at the University that requires you to form a three-person group to collaborate on a semester-long app development project. Your instructor has told you to use the My Dream Team platform to determine which of your classmates would be your most compatible teammates.

Task 1: Log in to MDT with the following credentials: [redacted]. Then, complete and submit the survey.

Task 2: Log in to MDT with the following credentials: [redacted]. Go to the class [redacted]. Complete the teammate preferences survey. Then, from the list of recommended teammates that appears, invite two to join your team.

We each moderated one usability test and observed another. As observers, we used a data-coding table and recorded whether each participant achieved success, partial success, or failure during each task and the smaller subtasks that comprised it.

To analyze the results of our usability study, we shared our individual data-coding tables with each other, paying specific attention to tasks and subtasks that were not successfully completed.

We ultimately found that

• Some participants felt specific questions in the survey were too personal: “This feels so weirdly personal for a survey for a course. This is something I’d never actually talk to any of my potential teammates about . . . I wouldn’t feel comfortable sharing this with an entire class of people I don’t know.”
• Others felt that there were too many survey questions (48 in total), especially because the four open-ended questions appeared at the end of the survey felt particularly taxing.

Recommendations

While we did provide the Sonic Research Group with recommendations for improvement after each individual study, we completed our semester with a prioritized list of recommendations, considered from the findings of each of our studies:

No.	From	Finding	Recommendation
1	Interviews	Social connection and trust are some of the most important factors that many people consider when choosing teammates.	Move ‘Social Connection’ into a more prominent position in the Teaming stage of the survey.
2	Heuristic Evaluation	There’s a lack of consistent and clear feedback with regards to system status and errors.	Provide clearer cues when a user has moved forward to a new survey question or caused or stumbled upon an error, with steps for resolution.
3	Usability Tests	There may be some hesitation for users to truthfully answer personal or private questions.	Revise, reframe, or reconsider the need for personal or private questions.

We summed up our semester of studies and our prioritized recommendations in this video: