Planned and launched a series of research studies on a team-formation web app, including interviews, competitive analyses, surveys, heuristic evaluations, and usability tests.
Role: Project Manager, UX Researcher
Timeline: Jan. – Apr. 2018
As part of the Needs Assessment and Usability Evaluation course at the University of Michigan (SI*622), my team and I worked with the Sonic Research Group at Northwestern University to study the usability of its web app My Dream Team. Throughout the semester, we conducted contextual interviews, competitive analyses, surveys, heuristic evaluations, and usability tests.
Members of the Sonic Research Group leveraged more than 50 years of research on intergroup dynamics to build My Dream Team.
Meant to help people form effective teams, My Dream Team was described as “more akin to eHarmony than Tinder”. That is, in a defined group (like a class), users begin by filling out a survey about themselves and their work preferences. They are then provided with recommendations about who in the group they might work best with.
The Sonic Research Group conducted preliminary usability research on My Dream Team during a pilot at the University of Buenos Aires, so our team conducted a more in-depth suite of studies, providing recommendations for improvement.
My team and I began by conducting a series of interviews with five people who had recently formed self-assembled teams in academia or for educational purposes. These interviews (and their protocols) were guided by the following research questions:
After transcribing our interview recordings and analyzing and synthesizing the data, we found that
After conducting interviews, we evaluated eight other tools and websites that offer similar team-formation services and categorized them according to the following competitor parameters.
|University of Michigan’s IMLeagues||Analogous|
|Applying & interviewing for a job||Indirect|
Then, we used the following criteria as points of comparison: target user groups, interaction patterns, and functionality.
Generally, we hoped to answer the following research questions:
From our analysis, we found that
Next, we crafted a survey to learn how often people in academia and educational contexts participate in group work, who they contact when forming groups, and what types of information they’re comfortable sharing with others.
We went through two pilot versions of the survey before developing and deploying a final version to open University of Michigan listservs, as well as other targeted groups of graduate or undergraduate students, instructors, and staff members at the University of Michigan.
We received 75 responses and conducted a thorough analysis of the responses. Based upon our analysis, we found that
Then, we conducted a heuristic evaluation of My Dream Team (specifically the log-in process and the survey phase) according to 10 heuristics described by Nielsen (1994). We adapted questions from the Heuristic Evaluation System Checklist created by Denise Pierotti to guide our evaluations, using Nielsen’s severity ratings to quantify the severity of any issues uncovered as a result of each question.
Each member of our team conducted an individual evaluation, and we then aggregated our findings to come to a common understanding of the most severe usability problems. Specifically, we found that there was a lack of consistent and clear
Finally, we conducted a usability test of My Dream Team to explore the following research questions:
We asked each of our five participants, recruited through our team’s personal social networks, to imagine themselves in the following task scenario and then to complete the two tasks that follow using My Dream Team.
Assume you’re enrolled in a course at the University that requires you to form a three-person group to collaborate on a semester-long app development project. Your instructor has told you to use the My Dream Team platform to determine which of your classmates would be your most compatible teammates.
Task 1: Log in to MDT with the following credentials: [redacted]. Then, complete and submit the survey.
Task 2: Log in to MDT with the following credentials: [redacted]. Go to the class [redacted]. Complete the teammate preferences survey. Then, from the list of recommended teammates that appears, invite two to join your team.
We each moderated one usability test and observed another. As observers, we used a data-coding table and recorded whether each participant achieved success, partial success, or failure during each task and the smaller subtasks that comprised it.
To analyze the results of our usability study, we shared our individual data-coding tables with each other, paying specific attention to tasks and subtasks that were not successfully completed.
We ultimately found that
While we did provide the Sonic Research Group with recommendations for improvement after each individual study, we completed our semester with a prioritized list of recommendations, considered from the findings of each of our studies:
|1||Interviews||Social connection and trust are some of the most important factors that many people consider when choosing teammates.||Move ‘Social Connection’ into a more prominent position in the Teaming stage of the survey.|
|2||Heuristic Evaluation||There’s a lack of consistent and clear feedback with regards to system status and errors.||Provide clearer cues when a user has moved forward to a new survey question or caused or stumbled upon an error, with steps for resolution.|
|3||Usability Tests||There may be some hesitation for users to truthfully answer personal or private questions.||Revise, reframe, or reconsider the need for personal or private questions.|
We summed up our semester of studies and our prioritized recommendations in this video: