Dylan Maroney

As the semester comes to a close and I’ve completed my user experience research regarding the Digital Archive of Literacy Narratives, it’s time to start planning out the process of delivering on the results that I’ve gathered. It’s worth noting though, that as this project developed I realized that me and the other DALN personnel aren’t quite at the stage yet to do investigative case studies, rather there’s a disconnect between how we perceive people using the site and how they’re actually using it. As a result, there are 4 key areas of research to report on: revisiting my original conceptions of who was using the site, the existing site analytics of the Digital Archive of Literacy Narratives, survey results from respondents, and an interview with a professor at the University of the District of Columbia who breaks the mold of our original conceptions of who used our sites. 

Introduction to the Project and Goals

Nearly a year ago now, I took Dr. Gu’s technical writing course where I started my work with the Digital Archive of Literacy Narratives. That original project sought to develop a recommendations report that could provide a framework for website changes that would benefit users based on web design best practices that we discussed in that class. However, that project didn’t incorporate any formal user experience research that would be necessary for determining if those recommendations I made had any actual impact on the ways that users could use the site. Looking back, the recommendations made were mostly in regard to design and failed to account for the functionality and usability of the site. While having a more consistent design philosophy was a positive, there needed to be more work done to account for the different ways that users would use the Digital Archive of Literacy Narratives and how we could change the functions of the sites (the archive and blog) in order to accommodate those needs. 

This project to determine user needs and a clear direction for future development is primarily being done to support a grant application for the Digital Humanities Advancement Grant funded by the National Endowment for the Humanities. Like that original design recommendations report, this research will be used to help provide a thorough list of necessary changes that we will discuss with Slalom, a information technology developer that the DALN originally worked with to get the site up and running. Essentially, this is preliminary testing to determine user needs that will lead to a quote for labor and costs associated with development, which will then help finalize the budgetary needs for the grant application. As such, this user testing is in its very early stages, and despite providing great insights, there will need to be clear ideas of what will need to be done in the future to continue adhering to the needs and wants of DALN users.

Original (Mis)conceptions

At the beginning of this research process, fresh off changing the focus of what I envisioned to be a case study regarding the different ways users were engaging with the site, I believed that survey data and interviews would lead us to clear areas of focus for our planned revitalization of the site. For this purpose, I was envisioning users within fairly stringent categories, graduate student researchers, professionals (teachers, tutors, professors) that were using the site to supplement their teaching and policy, and literacy studies scholars. The idea here was that by understanding this core constituent of users we could revise areas of the site to better serve them, but as I will discuss later those aren’t realistically the only groups of users that we need to consider as we move into development. 

There’s also the issue of misconceptions gained from the previous project that I did in Dr. Gu’s technical writing class. This early stage of trying to improve the site likely led me to having blind spots regarding how the site should be improved. As a result, that original recommendations report was like taking a step forward, while this class aptly informed me that I needed to take two steps back to better assess the actual needs of our users. While best practices are a good consideration, they aren’t realistically going to meet the needs of our users, and not within the funding that we’re shooting for with the Digital Humanities Advancement grant that we’re applying for this Summer. 

Looking to Existing Analytics as an Opportunity for Understanding

The existing analytics structure for the DALN site is due to a shortcoming in restricted user data collected on the site itself or through more robust analytics APIs such as Google Analytics, rather the analytics mostly come from the video hosting service that we use to allow users access to the video interviews that are done at conferences, major events, and in cases where users decide to do a video narrative rather than a text-based one. As a result, these analytics give a small picture of a much wider audience that we need to be considering. As a result, we have little to no analytics for the blog portion of the site, as seen in the image below. Aside from a bit of traffic through other sites, the vast majority of activity regarding engaging with literacy narratives is happening on the primary DALN site.

The differences presented by the two charts shows us that there are quite a lot of people who are visiting the site, and revisiting the site repeatedly, but the number of plays remains fairly consistent. It very well could be that users are having difficulty locating necessary resources and are visiting the site a multitude of times over just to find what they were looking for. Given just the differences between unique impressions and unique plays, less than 50% of new users actually locate a video that they can watch that corresponds with what they were looking for in the archive. Thus, it seems like a major imperative of future development must be to improve users’ ability to locate valuable materials. Interestingly, our most engaged with media reflects this idea. 

The most popular literacy narrative viewed was a narrative about learning English as a second language, but the next two represent a more concerning trend. For the last year, due to a lack of updates to the site, the front page has highlighted Musical Literacy as a popular collection, and the only formal collection on the site. Two of the top 3 coming from this collection indicates that there are users who only come as far as the front page before selecting one of the easily located narratives and interacting with it. The interview with the professor at University of the District of Columbia later may provide insights into why this is, but on the surface it seems likely that the ease of access leads at least some users to engage primarily with the narratives we highlight on the front page – popularity is being tied to accessibility. To get users to engage with more than just these easily accessible narratives, it seems necessary for us to either cycle through narratives we wish to highlight and to improve on the search functionality that currently exists on the site. This leads us directly to the survey that I sent out to the user categories identified earlier: graduate students, scholars, professionals regarding how we can improve the site to meet their needs.

The Survey

In order to get a wider perspective on how people were using the site, the obstacles they were potentially encountering, and to verify our early ideas of what users would need I created a survey using Google Forms and disseminated it to people I believed would have inte

resting insights into how the DALN could be changed to facilitate their research earlier. This proved to be a challenge as many of the people I originally sent the survey to didn’t respond in the 3 weeks I kept the survey open to responses. Instead, at CCCC I was able to connect with other literacy studies scholars and the leader of that special interest group was gracious enough to send the survey on to a wider listserv for people in the discipline. Of those that I sent the survey to at GSU, most of my fellow graduate students responded making up a fairly large group in the respondents (approximately half). 

The first area of focus was on collecting some basic demographic data – where people were in their graduate program or their level of education contingent with their position at their respective institutions.

While most respondents took the opportunity presented by the free response nature of the question, all of my respondents either already had a Ph.D. or were actively pursuing one. From this data, we’re missing information potentially about how public school teachers, administration, and support staff for programmatic administration might utilize the DALN. Most of those represented here were active in research in Rhetoric and Composition, or an adjacent position within their university.

That being said, there was a wide range of research interests. The rise of AI and AI use in pedagogy, graduate student research and writing, archival research and historical preservation, cultural rhetorics, professional development, college student experiences, 2-year colleges and dual enrollment, and the history of rhetoric gave a wide range of area where the DALN could potentially be of value should users be able to find the appropriate materials. From these 7 survey responses, I began to see connections to the original user personas that I was envisioning, college administration, college teachers/professors, graduate students, and literacy studies researchers. However, this question offered an early indication of a gap in my proposed research plan – the college student experience, namely the undergraduate experience and how they might engage with the DALN (more on this later).

 

Of these respondents though, those familiar with archival research was split nearly right down the middle.

The respondents indicated, essentially, that they likely didn’t have the expertise to engage with the archive in the context of other more traditional archival resources. This posed a potential challenge in establishing a baseline among respondents and could potentially skew results. Instead, the baseline presented itself in familiarity with literacy studies.

Every single respondent acknowledged that they had some level of familiarity with literacy studies that could allow them to utilize the site effectively. It’s important to note then, I think, that those users with limited experience in archives could provide an inroad to more accessible updates akin to creating a database rather than a more traditional archive structure. Despite the familiarity they had with the discipline, I thought it be important as well to see how potential users attempted to tackle new research interests more wholeheartedly. 

In most instances, users were actively engaging in new research by cross-referencing their existing research interests, paying attention to growing trends in the wider field of rhetoric and composition, diving in head-first (the classroom detail perhaps indicating that there is an iterative process involved), and starting with basic Google searches before moving onto available databases. Of all of these, there is a clear indication that there is no one way that users will approach the DALN, some may have done their homework and be entirely familiar with figures and their associated ideas such as Ong, Hirsch, Graff, Brandt, Street, Keller, Hamilton, and more, but there is a greater concern for those that may be going into the Digital Archive of Literacy narratives with a limited perspective on how to use the site and why it’s a valuable resource for the research they’re doing. To that end, I wanted to also ask participants if having additional resources available on the site could be valuable, and if so what they would need. 

 

Of the 7 respondents, only one selected that they weren’t interested in having a supplemental resource on the site to facilitate easier engagement with the archive. Whether it be a bibliography of key texts to reference, or the more popular glossary of key terms and their definitions, users saw value in having access to something that would make the research process easier. While there are other literacy studies organizations that have similar resources, if someone were to find themselves at the DALN site because they were interested first in literacy narratives, having a similar resource would be valuable as an additional supplement (and likely not one that would require much effort). Instead, the efforts of users could then be refocused on finding the primary source materials that we believe is the focal point of the archive by eliminating some of the labor that’s normally associated with the research process. That being said, most of the respondents wanted the archive to mirror a traditional database. 

Of the 7 respondents there was most agreement on having well established meta data in the form of key words and enhanced search functionality, while the idea of curation only being a preference by 3 respondents. Because of this, it seems pertinent to focus on building up the metadata for existing and new narratives with the goal of making a more database oriented site where researchers will be met with an interface that is undoubtedly familiar to them. Similarly, the next two most desired features were to have abstracts and uploader information available to further enhance the details regarding a submission before having to actually read the narrative. While having curated collections would keep us more in line with traditional archives, the potential users of the site don’t seem as concerned with archival preservation as they do having a robust research tool that is easy to use. Sadly, that brings us to the DALN blog. 

The current DALN co-directors view the blog as a major component of the DALN’s presence and a necessary component for enhancing the manner in which teachers, researchers, and professionals can see the site being used by others, and any major updates to the archive. Most of the respondents indicated that they weren’t particularly interested in the blog, but may actually be a branding and marketing issue, as I will talk about in regard to the interview I did later in this report. Nonetheless, having continued blog development may be a tertiary need for our userbase, focusing first on functionality and accessibility before moving onto the resources that we can share through that platform. Despite that, when asked about how the blog could be improved, respondents were willing to add the perspectives in valuable ways. 

Above all else, respondents indicated a desire for clear categories of posts and additional sorting options to easily locate specific posts that they’re interested in. For the DALN, this was part of our original plans. The blog currently is sorted chronologically which impedes access to examples of new research, pedagogical materials and reflections, updates to personnel, and updates to the archive and our resources. All of this considered, the fact that most respondents were interested in seeing a more impactful organization of the blog incentivizes us to continue to work on this part of the platform. When asked to elaborate on content that users would be interested, only 3 responded but I want to draw attention to the second response.

While this respondent wasn’t familiar with the blog, they mention a couple of different key considerations for future content and focus areas for organization. Being able to easily see new entries in the archive, reflections on transcription and “choose your own adventure,” and pedagogical examples/materials seem to be a high point for the blog and could provide additional visibility for the DALN as we continue to enhance the site. Being able to point directly to new pedagogical materials in the blog and having them relate to the updates we’re making to the archive could garner positive attention to the enhancements we make. This leads us to the interview I conducted with Dr. Bethany Monea from the University of the District of Columbia where she’s an assistant professor teaching community writing. 

The Interview

The interview with Dr. Monea was an enlightening experience where she provided numerous insights into how an instructor in a first year composition class might use the Digital Archive of Literacy Narratives. When we began our discussion I wanted her to explain her role a bit for me to gain an understanding of how she fit into the conceptions of users that I had identified before creating the survey. She talked briefly about where she stood in the university, a newly hired professor working toward tenure, and in particular her role in the community writing program where she taught a multitude of students ranging from 125 students a semester to 150. In those classes, one of the recurring assignments was a literacy and language narrative wherein students would describe their journey toward literacy or language acquisition. In those classes, she was already using the DALN as a resource to show her students what a literacy narrative was, the genre conventions, some examples of different types of literacy narratives, and from there they would write their own. Thus, Dr. Monea has a unique perspective on the DALN among other respondents in that she’s able to speak about the student experience. 

While aware of the DALN, Dr. Monea didn’t see it as the most valuable tool for this task due to a couple of key restrictions that I’ve already talked about in this report. Where researchers want familiarity with the search functionality and additional ways of narrowing their search, Dr. Monea identified the search function as a major inhibitor on student engagement with the archive. In a genre analysis assignment that scaffolds into the narrative, Dr. Monea asks her students to find a narrative and write a brief response identifying key information – most of these responses focus in on the Musical Literacy narratives that have been on the front page of the DALN site for a long time now as seen in the image below. 

Use Cases

Because I changed the focus of this research about halfway through the semester, I put myself in a position where I wasn’t quite able to establish clear conceptions of personas for different types of users. Instead, having a strong idea of the different use cases for the DALN archive and blog sites will help guide future development and design choices as we move forward. From survey results and the interview there are four clear use cases for the two sites: formal primary research, contributing to the archive through uploading your own narrative, reading the blog to see new developments in the DALN sites, utilizing existing resources to assist in pedagogical endeavors such as creating assignments or providing examples of the genre to students. 

Primary Research

Users engaging with primary research have demonstrated a clear need for updates in the functionality of the archive. In survey responses, the discussion of having revamped search functions, sorting options, updated and consistent metadata, available and accurate abstracts, and the potential offered by curated collections all reflect a desire to have a site that operates similarly to traditional databases and archives. Despite this, users are split along the line of how they would like to engage with the site. Curated collections seem to be preferred for those who are familiar with archival research, but nearly everyone saw the value of updated metadata and search. As such, catering to the needs of everyone interacting with the site while making the layout and function familiar is likely the best course of action. As for Dr. Monea’s discussion of having an easily navigable site for students, adhering to the model presented by traditional databases offers a simple solution that those students will be both familiar with and still satisfying the needs of more vetted researchers. Having curated collections available is still a possibility, but a lower priority based on these results so far.

Contributing to the Archive

If a user isn’t actively searching for primary source materials, then the likely secondary reason for using the site is to upload a narrative created by the user. In the case of the survey results collected, this use case didn’t seem to be a high priority for researchers or graduate students, but in the conversation with Dr. Monea she recognized that having a system in place for students to see ways to publicly sharing their work was valuable nonetheless. The issue here is that to make it easier for students and others to contribute to the archive we will need to update the existing form to reflect the needs of other users. Currently, users have limited avenues to provide their own metadata, so having an updated submission screen will undoubtedly ease the pains of submitting a narrative while simultaneously creating opportunities to have more consistent metadata attached to newly uploaded narratives.

Reading the Blog

While seemingly a less popular part of the website, the blog still has opportunities for users to engage with the DALN in unique ways that are currently underexplored. The lack of organization and structure in the blog (ironic given the state of my own blog that I’m posting this to) creates a difficult to use tool in which users could have access to new projects utilizing the DALN, teaching resources, research resources, and more. Thus, while nearly ever type of user could benefit from looking through the updates that are provided there, it’s hard to find precisely what you’re looking for. As a result, a near overhaul of the blog needs to happen to make these resources more accessible to the average user without the bloat of having to navigate multiple pages to come to your destination. 

Utilizing DALN Resources

Beyond simply reading the blog, users also have the opportunity to download a variety of resources from the DALN Resources page, but this page is both difficult to find and is poorly designed. Documents are able to be downloaded, but they don’t actually provide users explanation about what the resources are or how to use them. Instead, having these resources more properly integrated into the blog and the blog into the main site makes these resources more accessible and provides an opportunity to place them in a frame on the site so that the resource can be viewed without downloading it. Doing this should make a good resource that’s mostly inaccessible in its current form much more accessible for the different users that would benefit from having access to teaching, research, and publication materials. 

Wire Frames

DALN Navigation and Wireframes

As mentioned in the introduction, much of this user research and framing was done with the intention of supporting the DALN’s application for the National Endowment for the Humanities’ Digital Humanities Advancement grant. These wireframes have been developed in part to understand the ways that the site could be updated to enhance user’s experiences based on previous conceptions of enhancements and what respondents have indicated in their responses to the survey, along with Dr. Monea’s consideration of her students and how we can focus on creating a tool that’s useful for them as well, and then sent to Slalom so they can provide us quotes on how much revitalizing the site in this manner would cost). Below is a sort of table of contents as well as a rational for each change from the current site and how it serves user needs as described above. 

1. Current Navigational layout
   1. New pages have been color coded in green while current pages are in red. The new pages added are a part of the DALN co-directors goal of consolidating the site into one singular site that provides easy access to a variety of DALN materials for research and practice. Some links are evergreen (necessary on every page and must stay the same) such as those in the navigation menu on each page, those links have been marked by green lines while red lines indicate a one-way path necessary to reach a destination. 
   2. This navigational layout ideally will help users better find exactly what they’re looking for, with the blog having clear sub categories that users can identify easily. These blog pages also will include additional sorting options that make it possible for users to easily find authors and posts from a specific time frame. 
2. Home Screen
   1. The site analytics data and the interview with Dr. Monea indicated that students and people curious about what the DALN does likely stop at the featured narratives as an easy opportunity to see examples. As such, the featured narratives section of the page has been kept intact but instead of having it rely on admin editing whenever the narratives are changed, they will cycle between pre-selected narratives or randomly from a variety of high profile and/or narratives that provide an example of effective literacy narratives. 
3. Narrative Submission Screen
   1. In order to accommodate the need for improved metadata and tagging in each of the narratives, changes made to the submission form have been made to make it easier for users to provide more consistent metadata during the submission process to make it easier for administrators to review and approve, edit, or deny submissions as they’re made. This will free up time for efforts to improve the metadata in existing narratives.
4. Search Results Screen
   1. To reflect respondents’ desire for improved search results and sorting options, some changes have been made so users can quickly sort according to the metadata categories that will be updated during the submission process and should make it easier to quickly locate specific types of narratives. Much of the metadata is demographic oriented, which will help users who are interested in how literacy appears in certain communities, regions, ages, or educational levels.
5. General Blog Layout 
   1. The blog being split into three categories will make it seem like three separate blogs. This will make it easier for users to find resources that are relevant to their interests, as well as make the resources available through the blog more visible to users who want to see how to use the DALN in a variety of different ways. Having the blog organized in this way also provides an opportunity to have an easily accessible glossary or bibliography for users as well.
6. Curated Collections Selection
   1. Because there was an indication in the survey results that curated collections would be a desire, likely by those familiar with traditional archives, in addition to the updated search functionality and results page having a curated collections area of the site would be beneficial to those researchers looking for specific categories designed by DALN personnel. By creating a screen with all of the collections listed, it makes them easier to locate within the greater framework of the site. In some instances with archival research the collection is first found in the search results page through one artifact and then the researcher tries to find the whole collection, this would allow them to do that. 
7. Navigation Menu Changes
   1. Because there are changes in the overall structure of the site, changing the navigation menu to reflect new sections of the site and including drop down menus will help users see the breadth of the site available to them. 
8. Blog Post Layout
   1. The general layout of the blog post should make it easier for users to engage with content while also providing some measure of where users are within the blog (category + post). Because most of the blog posts are fairly straightforward text-based posts, there doesn’t need to be a large amount of complexity here.
9. Search for Literacy Narratives
   1. In the survey and interview, having updated search functionality was a primary concern for respondents and so the search function has been updated to accommodate the variety of different types of metadata that narratives currently have. Including search functionality for the title, keywords, dates associated with the narrative, and additional drop down for the different available collections provides an easy framework for users to find narratives that correspond with their research topic/assignment. 

Conclusions and Next Steps

Suffice to say that the Digital Archive of Literacy Narratives needs some attention in the future to meet user needs. In our conversations, we’ve been focused primarily on the DALN as a research tool and archive, without accounting for ways to improve the experience for teachers and students as crucial users. This research has demonstrated a clear need to pay additional attention to those two groups while providing updates that account for the intersections of teacher, student, and researcher as we move forward with development. As such, there should be clear priorities for development as we move forward. First, updating the infrastructure of the search function, metadata, and submission process, then making the resources in the blog more accessible and visible to users, and finally creating collections that will make it easier for researchers to engage with the archive. Despite these clear takeaways, this is just the first step in understanding how the site must change to improve users’ experience.

After these changes are implemented, we will need to revisit these current issues. By doing another round of user experience research, this time focusing more clearly on case studies and monitoring user behavior as they carry out the use cases listed above, we will be able to determine if these changes were enough to meet our users’ needs. As we’ve discussed plenty of times this semester, user experience research is an iterative process. As soon as we solve one issue, another may appear that demands new attention and new ideas of how to improve on usability or user experience. So, in the next year, assuming that we are awarded the grant, I will need to revisit this project and determine whether or not the insights gained were worthwhile.