Mattel163
Project Background
Mattel163 represents a strategic collaboration between Mattel and NetEase, focusing on the global development and release of mobile games such as UNO!, Phase 10, and Skip-Bo, integrating Mattel's iconic themes. These games appeal to a diverse, international player base. Working together with the director of UX, we faced the challenge of giving 45 million players a voice, defining who they were, and their needs and their motivations to play. As the game design team released builds weekly, we needed to streamline a UX research process and utilize various methodologies.
Project Overview
This overview is for various studies conducted for each of our games.
Research Process
Timeline of UX Research Activities
Average Duration per step while owning multiple studies
Preparation
Research Plan + Defining Participant Criteria
To devise a research plan and define participant criteria, I collaborated with my manager (head of UX), game designers, and project manager to define our research goals and select the most appropriate methods. For defining participant criteria, we relied heavily on input from the game designers who wanted perspectives from a diverse player group. The different types of players mattered, such as competitive players who engage deeply with game mechanics, and casual players who play more sporadically.
Recruiting
We had a creative recruiting process for our research as we had thousands of players globally, we used our social media channels to distribute surveys and utilized a pre-existing database of users who had previously provided feedback, reaching out via email to encourage continued engagement.
To increase participation, we offered incentives such as Amazon gift cards or in-game tokens that players could use to advance their levels. Game designers played a crucial role in specifying the demographics of the players they wanted to target for specific research initiatives, which helped us refine our recruitment strategy through screener surveys. We also considered the various time zones our players were located in, so scheduling was appropriately timed and accessible to encourage the most participation.
Execution
In-Game Surveys
Together with the game designers, I created an intercept survey as part of the game. This was a quick way to gather feedback that included a short series of questions with simple scaled rating questions. The surveys had rewards of a high value number of coins.
Surveys
We were able to collect and analyze data from a large number of players globally, and it was not a heavy time investment for players. We deployed surveys through social media channels and through emails, which were sent to a narrowed down list of players in the game database.
Objectives:
-Gather feedback on design choices, such as themes in the Phase 10 or UNO game
-Discover player motivations, challenges, and future expectations directly from the users
-Discover the majority demographic that are playing the games
Objectives:
Capture real-time insights into gameplay dynamics, user interface usability, difficulty of understanding the gameplay, and overall player engagement.
Setup:
- In-Person: Utilized tripods and company-provided iPhones to capture high-quality video of player interactions.
- Remote: Employed Playtest Cloud and Lookback for iOS and Android recordings, and Zoom for live interactions. This approach ensured participation from geographically distant or busy players.
We combined open-ended questions ("How would you upgrade the pass level?" and "Explain your thoughts on Battle Pass?") with scaled rating questions ("How would you rate the difficulty of understanding Battle Pass?").
Weekly Internal Focus Groups and Phone Interviews
- Weekly phone interviews and internal focus groups were specifically chosen to dive deeper into the qualitative aspects of our game design and development.
- We gathered Mattel employees regularly, to tap into detailed insights about user expectations, preferences, and experiences.
Why internal focus groups?
Advantages:
- These discussions were valuable for testing new game features and updates before wider release.
- The consistent schedule allowed for ongoing feedback that kept our development cycle agile and responsive to user needs.
- Ensured that each iteration of our game was informed by direct observations and reflections from a core group of users who were well-acquainted with the gaming industry’s dynamics.
How did we conduct phone interviews with such a large global pool of players?
I conducted weekly phone interviews pulling participants from surveys and previous play testing sessions. To make sure I had a diverse pool of interviewees, I chose players who were either competitive or beginner players to obtain different perspectives, as well as aiming to have a balanced number of participants across variables like gender, age, and location. Players who consistently participated in research we called “Super Tester Beta Group” and we made sure to keep them on file to reach out to for quick feedback as they were most available and familiar with the research process.
Internal “Street” Interviews
I set up a designated table in the Mattel cafeteria where we invited Mattel employees to play our games, and not only did we observe their immediate reactions but also gathered valuable feedback on the spot. These interviews were incentivized with small rewards, such as candy.
After gameplay, the player filled out a survey with open-ended and scaled rating questions about the feature they were testing. This method proved effective for collecting spontaneous, genuine user responses that are often more reflective of natural game-playing experiences, helping us to quickly identify and address potential issues within the game environment.
Analysis
Synthesizing Insights
-I utilized Google sheets and Excel to categorize, organize, and code themes for qualitative research. For quantitative data, such as average rating scales, results were also calculated through sheets or Excel.
-All our playtest sessions were recorded with the participants’ permission which made it easier to review and polish up notes.
-There was a lot of data to sift through, so this process took quite some time but my manager and I often partnered up to combine our notes and analysis.
Player Persona and Journey Mapping
Persona Development
Based on collected data, we developed detailed personas for different types of players, such as the "competitive player" in Phase 10 research.
Persona Impact
Use in Development: Phil's persona guides game feature development to ensure new updates resonate with competitive players, focusing on strategy and social interaction.
Mapping the Competitive Player’s Journey
The journey map for a competitive player of the Phase 10 game incorporated insights from user testing and direct feedback. The map illustrates five stages of game interaction: Lobby, Tutorial, Gameplay, Leaderboard, and Final Results.
-Data Collection: We gathered feedback through various means, including direct observation, surveys, and in-depth interviews conducted during playtesting sessions.
-Key Findings: Each stage was carefully examined to identify Phil's important touchpoints, emotional reactions, and opportunities for game enhancement.
-Mapping Procedure: The journey was then divided into stages, providing detailed insights into Phil's thoughts ("This rule is unfamiliar"), emotions (from excitement to frustration), and areas for improvement (such as offering more strategic control and clearer rule explanations).
-Outcome: This journey helped guide the game development team to refine game features and interactions, catering to the needs of competitive players.
Final and Weekly Presentations - Socializing the Research
Collaboration with Cross-Functional Teams
Regular presentations and workshops with the game design and development teams ensured that insights were shared and understood across the company. This helped in aligning the product development with the needs of players.
How the insights were shared:
- Weekly presentations where we gave context on the project and updated the team, providing snapshots of interviews.
- Final presentation after each study was completed.
Impact
