The Digital Archive of Literacy Narratives

For this week’s assignment I wanted to do a bit of user testing on the Digital Archive of Literacy Narratives (DALN) archive and blog to determine how it might be changed to improve user experience. The sites serve as a way for users to investigate phenomena associated with literacy development and practice through user uploaded narratives of their experiences with literacy, or in other cases to see developments, events, and how others are using the archive in a variety of contexts. The DALN is currently in the process of undergoing a new cycle of development to enhance user experiences and infrastructure to ensure the longevity of the project. So, to reconfirm what has been discussed by personnel working on the project, I wanted to see if a user new to the archive could make good use of the site based on its current functionality. To this end, I enlisted one of my roommates, a graduate from the neuroscience program at GSU who engages in lab-based research and isn’t familiar with archives to see how they would approach two common tasks that that sites are used for. 

Conducting the Test

I split this 10 minute foray into user testing by splitting the test into two parts: a test to determine how a user would find a literacy narrative about experiences in elementary school as the topic of the narrative, and another asking them to find examples or resources of how to integrate the archive into a class on literacy. I gave them these basic instructions without much additional guidance or explanation of what a literacy narrative was, how to navigate the site, or a specific end goal for the testing. The goal of reducing frontloaded information was to see how a new user would navigate the site independent of guidance. They were able to eventually find what they were looking for, but they struggled to identify which narratives worked perfectly to satisfy the goal of finding a narrative regarding elementary school. It seemed to mostly be their own limitation on what they thought would work (there were a number of narratives that appeared during their search that were related to the topic, but none that were specifically titled with “Literacy Narrative.”) This similarly caused issues when they were trying to find a blog post that was related to classroom utilization, wherein they searched for specific phrasing in the titles of the posts that would make it abundantly clear that it was within the parameters of what I was looking for in the test. After we finished testing, I asked them a few questions to see what they thought of the site and if they had any recommendations:

1. What was a challenge in finding content related to what I asked you to find?
2. Was there anything you would have liked clarified before testing started?
3. How could the search functionality better support your end goal?

It was from these questions that it was easier to understand their thought processes during the test and explained some of the difficulties that I saw when I watched back over the screen recording. 

 

Identifying Issues

In this section I want to discuss some of the issues that arose from the websites’ design, or in some cases issues with the testing. While the test overall felt successful in confirming some of the issues that the DALN development team has already identified as being key areas of future development, it also elucidated some other considerations that we will likely need to make as we move forward. 

With the Site

The main issue that this test showed regarding both sites is a lack of organization – both in the archive and in the blog. While the user was able to find examples of what they were tasked to look for, they struggled to identify what would have been the perfect fit for what I had told them to find. As I asked them the follow up questions, they expressed difficulties that were encountered because the search functionality was limited compared to the databases that they had used in the past. The current search is entirely key-word based, however each narrative lacks distinct keywords to associate them with already known phenomena such as literacy practice, literacy development, literacy sponsorship, TESOL, etc. This seemed to cause issues for the participant as they were trying to identify which narrative was the best example for the test, similar to issues a student or researcher might have in trying to find a narrative for their own projects. 

As we moved onto the blog, similar challenges arose due to the blog being organized chronologically. The user navigated the blog searching for a post that related to classroom application which appeared more explicitly on the second page in the title of one of the blog posts, but it leaves me wondering if there could be a better method of organizing the blog posts in the future. If things are organized by overarching topics, it may be possible to make certain examples easier to find while also creating a more organized history of the archive/project’s development. Currently, blog posts are related to examples of classroom application, introduction of new personnel, updates on web development, and announcements regarding conferences and other events that the DALN will be at. If these were organized along these lines, we could continue testing to determine what a more frequent user would be looking for in their experience and cater to those needs more closely. 

With the Test

The test went mostly well, however it became clear to me how different participants will need different levels of instruction. If my roommate was more familiar with archival research, it might have been possible to complete this test without any guidance, however I didn’t consider what additional information my participant might need before starting. This was especially true in instances where I used terminology that they weren’t familiar with. When I mentioned literacy narratives, they didn’t ask for clarification, but when I asked if they would have liked any clarification post-test they acknowledged that a bit of explanation of what the DALN is for and what literacy narratives were could have been helpful to them in their search. In the future, I think I will need to provide at least some guidance or background information for users, rather than seeing how they approach the test based on instruction alone. 

Reflection

This initial testing experience was remarkably helpful. I found that what I think of user testing needs to meet in the middle ground with users’ own experiences with similar applications and media. In cases where they might be completely new to something, I believe that some background information may be needed for more complex tests or in cases where the app is designed around specialized knowledge (such as an archive on a specific research topic). To this end, I’ll likely be spending more time reconsidering my participants for future tests, focusing on how much they might already know about how to use an application and then from there develop tests that are suitable to their skill levels. While this might make testing easier for participants, I’m left wondering how much information you can realistically give participants without invalidating the results of user testing….

Introducing Quicktime Player

Quicktime Player has been one of the most longstanding screen recording programs available, dating back to the earliest days of MacOS and supported on Windows platforms up until Windows 8 was eventually released a few years ago. As a result, if you ask anyone about how to record your screen, or what desktop video player they use, Quicktime wouldn’t unnecessarily be an uncommon answer you hear. However, when it comes to usability testing in our cases, and general screen recording for others Quicktime has a number of pros and cons that we need to explore to determine if it’s worth utilizing for our purposes as user experience/usability experts.

As I said in the first sentence of this post, Quicktime Player is no longer supported on Windows computers, eliminating it as an option for a wide range of purposes, and makes it even more restrictive in what programs you can do usability testing on since there are a number of programs that never receive MacOS support during their lifecycle. This issue is a paramount concern that needs to be taken into account in the early stages of planning for your usability/UX testing. However, it goes beyond just the MacOS-Windows divide, because even those programs that are supported on MacOS will need independent testing done on that infrastructure in addition to testing on Windows since the different operating systems also means different functionality and often different interfaces than users might be accustomed to. As such, you can’t utilize Quicktime for all of your user testing, you would have to have a separate program selected for Windows testing, and then Quicktime for any Mac testing. This divide makes it an even greater challenge for testers, since they won’t have the luxury of learning how to use one screen recorder before having to learn a completely new one later. For Mac specific programs though, such as the ever popular Garage Band, usability testing using Quicktime makes a lot of sense. If a program is designed specifically for use on MacOS, then you need not worry about having to prepare for any other screen recorders. 

While this may make it seem as if Quicktime doesn’t have any advantages, Quicktime is still one of the most simplistic screen recorders available, preloaded on any modern Apple computer reducing need to find a download for it, has very clear parameters for what you are able to do with it, and it offers one of the most important features for usability testing and videos supporting technical documentation – a clicking tracker. These affordances make Quicktime an effective “quick fix” option if one is struggling with finding the right software for their usability testing, and thus it’s almost an ideal program if conducting quick tests on things like websites, or if recording instructions for users with video supplements. So, we can’t quite yet make a determination on the value of Quicktime without doing a bit of testing ourselves.

Getting Started

Starting Quicktime is straightforward. As with any MacOS app, you simply go to Finder > Applications > Open Quicktime Player, but from here users might have some difficulty in actually starting their screen recording. After opening the app, Quicktime immediately opens its own version of Finder prompting users to open a video file to play – this is its primary functionality but because it does this it may be confusing for users unfamiliar with Quicktime, and even worse for users who are unfamiliar with the way MacOS displays application settings and options. In the navigation bar at the top of the screen, if Quicktime is selected you are given a unique menu where you can start a new screen, movie, or audio recording, and if a video is loaded you can edit playback options. An alternative approach to getting to this point is right clicking the app icon in the bottom navigation bar and starting your recording there. The issue that this presents is that there is no affordances to indicate to users how they can start a new screen recording, and limited documentation/instructions embedded in the app. Instead, users have to utilize a search engine and find instructions on the internet, but even these instructions might prove difficult for those who are usually Windows users. For example, the “File” button is not located within the window that Quicktime opens, rather it’s at the top left of the screen, breaking the usual approach that Windows programs follow. 

Once you finally find this option, though, the options before you start your recording are much easier to follow. It’s a very simplistic interface that comes up at the bottom of the screen, prompting you to select among different screenshotting options on the left, and screen recording options on the right, along with additional options to determine where you would like to save the file, if you’d like a brief timer before the recording starts, if you would like to enable your microphone, and additional options including the option to show mouse clicks. Then a very clear button for “Record” that makes it abundantly clear how to get your recording started. This simplistic design makes Quicktime among the easiest screen recorders to use, and requires almost no expertise to start screen recording. These affordances make Quicktime an appealing option for novice users especially, maintaining its position as a nearly household name despite how many new programs have come out in the last two decades.

 

Once You’re Recording

There’s not much you can do once you actually start your recording. Quicktime doesn’t have any additional tools to enhance the recording in real time, so what you see is what you get. In some regards, this appeals to the same simplistic design that makes Quicktime an extremely accessible option for user testing, but at that same time it struggles to match the capabilities of its competitors in this regard. Where other options allow you to change settings on the fly such as audio levels, scene transitions, video settings, and more, Quicktime only gives you the option to stop your recording. This is also where the native MacOS also might cause additional challenges for users. The stop recording button is placed in the top right of the screen in a relatively small button with the usual stop symbol placed on it. With newer Macbooks (2019+) that have touchbar integration, there is a stop button placed there, much more conveniently. The challenge this poses is that if you’re working from an older model, or don’t look down at your keyboard, you might completely miss the stop button. This creates an annoying (but not program breaking) issue where your recording may just have an additional 5-10 seconds in the recordings prior to getting used to this new type of interface. If doing a lot of screen recording for more substantial usability testing, the impact this has isn’t just on the usability or user experience of the program, but also on your storage capabilities as most MacOS systems screen record at the native resolution of the system – often far exceeding the video quality usually selected for user testing which also increases the size of the video files you will end up with. As I tested this, I exceeded a gigabyte after about 30 minutes of recording. Meaning, user testing could easily exceed a few gigabytes, limiting the ability of the testers to share video files easily afterward.

Determining the Value of Quicktime

At its core, Quicktime is simple, however that is precisely where a lot of the issues lie with Quicktime. Because Apple has placed so many limitations on screenrecording options, it’s exceedingly simple to get started and quickly complete user testing on the fly. Additionally, it boasts the coveted “Show Mouse Clicks” option that does set it apart from some of its other competitors, however this feature is easily supplemented by a secondary program running alongside any other screen recorder. Beyond that though, Quicktime has so many instances where the design is frustrating for new users to MacOS, limited in ability to generate qualitative data from something like a webcam video feed, and excessively large file sizes that limit large scale user testing efforts. As such, I don’t feel that I can confidently recommend Quicktime if there is enough time to learn how to use another option, or if the intention is to do extended user testing across multiple personas and for any longer than a couple of hours. If I had to give a rating out of ten, I’d say Quicktime falls comfortably at 6/10 – a useful, but limited tool that lacks the innovations that have come from far more accessible programs in recent years. 

Does AI Recommend Quicktime?

As AI language models become more prevalent, it’s worth also exploring if they recommend Quicktime as this may lead some users to picking up the platform based on that recommendation. When ChatGPT was asked “What can I use to record my screen?” it provided a list of 7 different programs, including Quicktime due to its ease of use and because it comes pre-installed limiting the need for additional steps. ChatGPT followed this up when I asked it to elaborate (Why should I use Quicktime Player to record my screen?), agreeing with my assessment on a couple key areas: preinstalled, simple to use, basic functionality, MacOS integration, and reliable performance. AI also agreed with my criticisms of the platform, but most notably made note of an aspect of the program that I missed. Quicktime Player saves files in a .MOV format rather than the much more common and support .MP4 format that most modern videos are recorded in. This additional limitation of Quicktime means that if you want to highlight specific cases that appeared in your user testing, you are restricted to a limited number of video editors as well, making Quicktime considerably worse than its competitors if your intention is to do professional user testing and then report on it in a meeting or conference environment. 

TLDR Pros and Cons of Quicktime

Free	Yes / No / Kindof – Because it’s only accessible on MacOS, there is a high barrier to entry due to cost of hardware.
One button install	Yes / No / Kindof
One button launch	Yes / No / Kindof
Lightweight output	Yes / No / Kindof
Editable output	Yes / No / Kindof – There are strong limitations on what you’re able to edit
Tagable output	Yes / No / Kindof
Help	Yes / No / Kindof – Apple Support does have a page explaining how to do screen recording through Quicktime, but there is no help provided within the application itself.
Tutorials	Yes – Available from independent creators on Youtube, or Apple Support Website. / No / Kindof
Positive industry reputations	Yes / No – Most sites when you search for “best available screen recorders” don’t acknowledge Quicktime as an effective option anymore. / Kindof
Easy to use?	Yes / No / Kindof

 

The following notes are in response to testing completed on Dr. Pullman’s to-do list (https://www.gpullman.com/8122/todo2.html) and the ensuing discussion around what was good, bad, and strange about how the application functioned, its usability, and what the user experience of using this application might look like. I have attempted to maintain the nature of my field notes to reflect my experience with the different testing methods sampled in class, showcasing some of the advantages and disadvantages that will be reflected on later in this post, along with a mockup of a potential alternative to the application provided to us in class.

 

5 Second Test Notes

The app seemed to be designed as a schedule organizer. Giving a prompt to add tasks with the ability to sort them. However, there aren’t any organizing features. 

 

Using and Brainstorming Session

To do list is rudimentary, can add tasks, delete them, or mark them as completed, but the option to sort doesn’t do anything, nor are there any additional bites of information that indicate when/where/how something needs to be completed. 

 

The enter key doesn’t add the entry that you’re typing out, making the interface feel even more clunky than it appears.

 

Revision on sorting comment* – once a task is completed and the button is pressed, it sorts based on completed/not completed. This helps get things out of the way once they’re done, but still lacks other sorting principles that might be valuables. I.e. being able to sort by time it needs to be completed, date to be completed, date added, type of task, etc.

 

Looking back, I noticed that because there was a lack of ability to add additional information regarding time/date, one of my criticisms, I ended up putting down tasks for today only that I still needed to complete but that didn’t have a specific time they needed to be completed by. 

 

Heuristic Inventory Form and Personas

Heuristics

Visibility – Yes, if link is broken, site responds to users saying page doesn’t exist or incorrect web address is inputted. 

Use Familiar, real-world language – Yes, the to do list is simplistic in language it uses to prompt users and is therefore easier to use.

Users should be in control – No, users have some control over adding, deleting, and marking as completed/incomplete, but there isn’t anything that allows users to save their to-do list for later reflection. No ability to undo deletion of a task and no real sorting options beyond complete/incomplete.

Follow Industry Standards – Yes, can save web page as a pdf as you can with any other web page.

Don’t let users make mistakes – No, users can delete a task and can’t recover it. Need to type it out again.

Recognition over recall – Yes, simplistic design makes it easy to recognize how a user should be using the app.

Flexible designs – N/A, no real shortcuts or specialized knowledge that allows experienced users to quickly navigate interface. 

Minimalist Design – Yes, avoid overly complex design. 

No error should be fatal – Yes, you can delete tasks you didn’t mean to, but you can just input them again. Can’t delete more than one item at a time.

Provide help – No, app relies on recognition to guide users, doesn’t have additional documentation. Was tempted to say N/A, since it’s just a to do list. 

 

Personas

Low, medium, high frequency

Low, medium, high skilled

Demographic data related to race, ethnicity, gender

Do they use lists?

Paper or digital lists?

 

Reflection

This reflection will hopefully provide some context on each of the above sections, and offer some of the insights gleaned from their completion at each stage of the class. While the methods of testing differed, a constant was that field notes helped to provide an in-the-moment way of documenting the impression that the application left on me, and how my perception of the app changed with each method of testing. The 5 second test, the extended brainstorming session, and the heuristic were especially valuable in communicating the level of information that each type of testing could give and why testing with each of these methods can yield vital insights into the value of your creation/application/service.

 

5 Second Test

The 5 second test was interesting and of all the methods, the only one I was unfamiliar with. I recall in my past run-ins with user experience testing that it was important to document first impressions, and this test does just that without tainting the user’s response with the ability to test any further than that very first impression. In reflecting on my brief notes, it’s obvious to me that while this was successful in documenting first impressions, the class discussion that followed and peoples’ varied answers to what the app was for, and the amount of information they deducted about the app, showed that even among twelve participants a 5 seconds test results in multiple different answers. So, while useful, this doesn’t seem like it should be the preferred testing format under most circumstances, even if it should be utilized if possible. 

 

Brainstorm Session

The brainstorming sessions yielded much more results and gained much more agreement among the class than the 5 second test did. Key issues of the app were brought to light, and enabled us to discuss things like the enter key not doing what we usually expect it to, issues with sorting items added to the list, user errors that caused frustration, the challenges posed by deleting items with no undo button, or how the simplicity of the app resulted in a desire for more features. This consensus among participants demonstrated that while the 5 second test was valuable in capturing a variety of first impressions, it’s in these longer tests that you can determine the genuine issues and challenges you will have as the developer moving forward. However, that doesn’t mean that this consensus is even then accurately portrayed through something like survey data – rather the interview/discussion portion of this assessment of the app was extremely important in coming to an understanding amongst participants. 

 

Heuristics

The survey data did demonstrate, however, the importance of simplicity in how you manage your testing as much as, if not more than maintaining simple language and design in the application itself. Things like industry standards, fatal errors, and communicating system status were foreign concepts for some of the participants showing that a novice user might not be able to respond to questions about these terms specifically – or if you give them examples they might just say that the app either does or doesn’t satisfy the design principle given in that example. For example, the system status communicated by the website that a page doesn’t exist at the given link is much different than a web service telling users that the server is down. These two examples of system status help users understand what’s being discussed, but they have widely different implications for how users actually interact with the app. In the case of the To-do List, there is no server attached to it, instead the only status to report is if the list is functioning or not. Some may argue that communicating things like completed v.s. Incomplete tasks could be another example which is something the list did do. So, if we are to use heuristics as an effective tool for user experience research moving forward, the language used in the questions is extremely important for getting accurate results, especially if no interviews/extended discussion will be conducted afterward.

 

Personas

Personas were an interesting part of our discussion following heuristics. While it can be detrimental to put your potential users into a box, if you have limited access to test with those types of users it makes it possible to go through testing with a couple of assumptions in place that may allow you to see how some users will use your app/service. This is most useful if you’re limited to inhouse testing, caused by various limitations including but not limited to user experience research budget, limited access to actual users, attempting to test very early iterations of what will become the end product, lack of other adequately sized departments in the case of larger organizations, etc. Any of these limitations or any more that appear in the lead up to user testing may indicate that you will need to brainstorm some user personas and test using them rather than your actual users. 

 

To Do List Mock Up

Case Studies

My introduction to case studies as a form of research came a long time ago during my high school years when I was developing an interest in technology based fields as potential avenues for employment, in my AP psychology class, and in a very interesting visit with my Anatomy class to a local cadaver lab to witness how medical students would utilize individual bodies for their educational development. While the cases I witnessed in my Psychology and Anatomy class perhaps aren’t the most applicable to user experience and design (except in some extremely niche scenarios) these early experiences installed in me the confidence that case studies can be a worthwhile endeavor to accelerate development of a product, set of knowledge, or research trajectory. 

For the burgeoning researcher or UI/UX professional, gaining an understanding of these case studies is then a necessary component of their education. Thankfully, there is a long history of publicly available UX case studies, newer AI models that help to consolidate a plethora of advice found on the internet, and additional case studies that while publicly available have struggled to reach popularity with the public. In this post, you will find the methodologies that AI and other case study models recommend to create your own case studies and a brief analysis of a case study that I have found that makes it easier to understand what this advice looks like in practice. 

Practical and AI Case Study Models

• Background/Problem Statement/Explore
  • Identify the problems you’re experiencing with the current model and the needs of the organization. It’s important that you take a moment at the beginning of the process to find what your organization’s needs are or the purpose of your end-product. From there, you might identify very clear problems with your product, those glaring issues that you don’t need user testing to understand, but that user testing might help in finding solutions for. Afterwards, you can begin exploring options for next steps – do you need test new research tools? are there additional needs that you think users might have that you’re currently unaware of? how can you improve upon the user experience? 
  • In your report on user testing, this may be a point where you begin to define your target audiences, unintended audiences, roles of personnel and users, etc.
• Brainstorming/Ideation
  • What I would consider to be the second most important step of your user testing, your brainstorming stage, is intended to provide additional time to expand your user profiles, testing methods, complete preliminary research, and complete low-high fidelity models for users to engage with (through wire frames, mock-ups, drafts of websites or interactive models). 
  • No matter how much brainstorming you do, you can’t plan for everything. You should plan on doing/making surveys/interviews/observations/personas for a variety of different circumstances, but if additional circumstances present themselves you may need to complete a second case study.
• Design
  • The design of your case study, like the brainstorming stage, should take into account a variety of scenarios. Particularly, you need to consider what your users are going to be using your product for, and the variety of uses that it could potentially have. Using your wireframes, mockups, and personas developed during the brainstorming phase, administer these tools to research participants and have them complete tasks that their personas would want to use your product for. 
  • You might have users do a variety of tests, both to determine the efficacy of your design and the usability of the design. Before doing your testing/research, identify ways to generate findings around your products problems, strengths, questionable areas, and variety of uses to get a holistic picture of how users perceive the product at this stage of development. 
• User Research and Methods
  • Once you have completed your brainstorming, drafts, mock-ups, wireframes, personas, and testing design you can move onto the actual usability testing phase. At this stage, follow the research agenda dictated in the design phase, if alterations need to be made to the research plan you can complete additional tests at a later time.
• Evaluate Findings
  • After accumulating data, potentially both quantitative and qualitative, determine how that data might inform later stages of product development. Rather than following intuition on how you might continue to improve your product, allow the data to speak for itself. Your users have given you what you need to implement clear solutions, or you may have asked them directly how they would solve problems they encountered, to create solutions that improve user engagement, satisfaction, conversion rates, etc. 
• Reflection
  • Once all is said and done, you need to reflect on your testing processes. Review each step of the process and decide on whether or not that step was successful, what made it successful, what you might change in future case studies, or what that step failed to communicate to you or users. This might look like you realizing that wireframes simply weren’t enough to understand how users would engage with the final product, so in future testing your might work from a functional draft of the final product.
  • For example, you start creating a new website, so you want to conduct user testing before you get too far into the process and waste important labor. You create a card sorting activity and a wire frame of how you intend to create the website, this testing might yield information about the organization that users would prefer, but it can’t tell you about the functionality of the end product. In this case, you would need to later create a high fidelity model that has some base level functionality to conduct further tests from.

Sources:

Dr. Pullman’s conversations with Chat GPT and Copilot (https://www.gpullman.com/8122/cases.php?cases)

https://uxdesign.cc/airbnb-redesigning-for-the-new-normal-66fb273de769

https://www.behance.net/gallery/126901637/HAVEN-UXUI-Case-Study

Examining an Independent Case Study

To provide some context for this case study, Dan Jenrette discusses the user experience testing that was conducted on what has widely been considered one of the biggest failed launches of a World of Warcraft expansion in the last 20 years. World of Warcraft: Battle for Azeroth was the 8th installment of the franchise, and thus there may be an assumption that Blizzard entertainment already had a clear conception of how they “should” do user experience testing. From this they identified a couple of key areas and metrics that they considered to be of the utmost importance:

Key Areas:

• The “First Hour” experience, wherein players would first be introduced to the story of the expansion, basic gameplay mechanics, and introduction to new areas of the world.
• Dungeon Content – The core of the leveling and end-game experience, has been here since the very start of the game back in 2004.
• Island Expedition Content – New content that is both players versus ai, or players versus other players.
• Warfronts – Reminiscent of the original Warcraft games, where players would build a base and defend or attack against an enemy. Designed to try to emulate the feeling of a larger battle or ongoing war. 

Within these areas, quality assurance personnel were focused on player retention, how “fun” something was, pain points that players might experience during leveling and endgame content, what killed the most players on their journey, what were some common player successes, how players felt about artificial intelligence in island expeditions, etc.

To test these things, they used multistage testing, focusing first on usability, then playtesting, then a post-test survey to understand user reactions. From this testing they were able to adequately measure data around how users felt about various dungeon and raid (long form dungeons at a higher difficulty with more players) bosses, how they felt about AI and specifically how they felt about particular AI they encountered during island expeditions, how they felt that Warfronts were engaging and fun content that emulated large scale battles, and more. However, as Jenrette notes early on, user testing was often limited during the development stages to in-house staff in departments that weren’t working on the game such as accounting, human resources, staff on other games, etc. This caused the downfall of the expansion in the eyes of many of the actual players who would later play the expansion, and Jenrette corroborates this, recognizing that in their reflection that even when they identified crucial issues in gameplay from beta testing (testing done closer to launch of the game to make sure that servers could meet demand and that the game was, at its baseline, functional), it was already too late to go back and make changes to improve player experience. 

The result was heavy criticism from players, notably from longtime players that noted that the new expansion stripped away too many of the systems from the previous expansion that created engaging gameplay, there was a lack of player freedom in their choices to pursue different types of gear (the Azerite armor system was undertested and underwent numerous changes over the course of the expansion), Warfronts and Island Expeditions had their difficulty toned down after internal testing and thus many players thought they were unrewarding, they installed endless endgame “grinds” (endless content that provided meaningful rewards) at multiple stages to drive up user engagement, etc. From this case study, and the ensuing response, it seems clear that it’s not enough to do your user research at the development stages of a project, it’s an ongoing process that needs to consider how every change might impact users’ experiences.

The Reddit thread below consolidates many of these criticisms.

(Content warning for language, mentions of genocide, misogyny, etc.) 

Why is/was BfA one of the most hated expansions?
byu/HerrMatthew inwownoob