Skip to main content

Usability Tests

Usability tests are essential for identifying UX/UI flaws, and this was especially true for TrailBlazer. These tests took place during IHC classes at the University of Aveiro. Students were tasked with performing various activities on the app and providing feedback through a form.

The primary aim was to evaluate the user interface, features, and overall experience of the application. Additionally, we wanted to see if the sentiment analysis models for reviews and the auto-tagging feature met the students’ expectations. Each participant was guided through specific tasks within the app, and their feedback was collected in real time via the forms. This detailed feedback was crucial for pinpointing both the strengths of the app and areas needing improvement.

The purpose of these usability tests was to assess how intuitive and user-friendly the app is for its intended users and to identify any potential issues that could affect user satisfaction and engagement. Conducting the tests in a classroom environment with a diverse group of students allowed us to better understand the app’s performance under different conditions and how it could be refined to better meet user needs. Additionally, during these tests, the method of providing recommendations to users was also evaluated.

The structure of the test was designed as follows:

  • Part 1 - Introduction and Consent: Begin the session by outlining the participant's tasks and explaining how their data will be utilized. Emphasize that the focus is on evaluating the system, not on assessing their performance. Ensure you obtain their consent to continue at this point.

  • Part 2 - Background Information: Start by asking the participant some demographic and background questions to gauge their familiarity with our type of system. Inquire about their age, profession, and previous experience with similar systems.

  • Part 3 - Tasks: Present the participant with a series of tasks to complete within the app. These tasks should be designed to test the app's core features and functionalities. Observe the participant as they complete each task and take note of any difficulties they encounter.

    • Register a new account as a Tourist and complete the recommendation assessment: The task involves creating a profile and completing the recommendation assessment. This process helps the system understand your preferences. After completing the recommendation assessment, the user evaluates whether the offers align with the preferences and interests indicated during the assessment.
    • Search an offer: The task involves searching for an accommodation offer rated above 4 stars. After filtering for accommodations, the goal is to find the cheapest offer meeting these criteria.
    • Post a Review: The task involves posting a review about the previous offer, expressing either amusement or disappointment. Afterward, the user evaluates whether the attributed score is accurate.
    • Login as a Provider and create an offer for a Restaurant: The task involves logging out of the current account and logging in as a Provider. Once logged in, the user needs to create an offer for a restaurant. After creating the offer, the user should review the tags associated with it to ensure they accurately represent the restaurant’s features and offerings.
    • Go to the dashboard: The task involves navigating to the dashboard and identifying the number of offers predicted for July 2024.

  • Part 4 - Post-Test Questions: After all tasks are completed, ask the participant questions about their overall experience. Ask them to rate their experience out of 5, point out any difficulties they faced, what they liked, and how they think the system could be improved.

Post Task Questionnaire​

After the usability tests, participants were asked to complete a post-task questionnaire to provide feedback on their experience. The questionnaire included questions about the ease of use, clarity of instructions, and overall satisfaction with the system. Participants were also encouraged to provide additional comments and suggestions for improvement.

From that questionnaire we were able to calculate the System Usability Scale (SUS) score, which is a widely used measure of system usability. The SUS score ranges from 0 to 100, with higher scores indicating better usability. The SUS score for TrailBlazer was 80.4, which is considered very good result and indicates that the system is generally user-friendly and easy to use.

We also retrieved some suggestions and comments from the participants, which helped in improving the interface. Those suggesting lead mostly to the changes we referred in the previous section.