ux research

While it’s not my primary skill focus, during my time at D2L and at FairVentures, I’ve run a number of UX research activities. Below are a few highlights of my experience.

problem validation - BRIGHTSPACE PULSE

At the early stages of the Brightspace Pulse project, there was very little information about what students would want in a Brightspace mobile app. We had some hunches in a few different areas (ie. dates, course content, announcements, prioritization) but needed to know which of those areas to prioritize. To do this, we assembled a research team of product designers and product managers and set out to conduct a series of problem interviews with students on campus to understand what aspects of their course management was most challenging.

Format: On-campus short interviews with students.

What we used: Problem validation script, problem ranking sheet, problem intensity sheet, NDA, Consent, Amazon Gift Card compensation.

Research participants: 30 students at multiple Waterloo Region campuses.

Analysis: We compiled ranking data across all student participants to see which problem had the highest rank and “intensity” of challenge. We compiled qualitative data into themes for each problem statement.

Decisions and Action: We presented results to the department team, which enabled us to focus first stage of development on the challenges that were ranked highest by students.

Other problem validation projects: Brightspace Binder, Small Business Insurance with Fairfax companies.

USABILITY TESTING - eportfolio

At the early stages of a new ePortoflio feature, the team wanted to understand how well users understood the feature and perform tasks using the feature. The team used a lean and agile methodology, where we would build the product incremental slices, so each sprint we would have more and more functionality available for testing. We conducted usability testing each week for several weeks with D2L colleagues to learn if we were on the right track with our concept.

Format: In-house moderated task-based interviews using ‘think-aloud’ protocol and SEQ.

What we used: High-fidelity interactive prototype (Axure), usability task script with SEQ, screen recorder.

Research participants: 15 colleagues and co-op students at D2L.

Analysis: Across each participant, we compiled SEQ data for key tasks.

Decisions and Action: Recordings were played with the greater team to create awareness. Tasks that had a low SEQ score were redesigned in later iterations of the feature.

Other usability testing projects: Brightspace Pulse, Brightspace Binder, ePortfolio Accessibility.

Design Sprint testing - Underwriter availability app

Both at D2L and at FairVentures Lab, I’ve conducted several design sprints, each with a concentrated testing phase to understand how users responded to the concept created. In the case of the design sprint for the underwriter availability app, we tested an interactive inVision prototype based on the week of design sprint activities with insurance brokers from the Lloyd’s Market of London.

Format: Prototype testing interviews with team observation room and in-field prototype testing interviews recorded with mobile screen recorder (no observation room).

Research participants: 10 insurance brokers in Lloyd’s Market of London, UK.

What we used: On-person web-cam, observation room, inVision prototype, mobile device, Zoom to share mobile device, interview script, NDA, compensation.

Analysis: For each participant we captured what worked, what didn’t work, and possible fix.

Decisions and Action: Based on the response from brokers, the underwriter availability app was approved and the design concept and test results were shared with a 3rd party development group to be launched in late 2017.

Other design sprint testing projects: Brightspace Pulse, Claims Assist App with Fairfax companies, Micro-insurance in Egypt with Fairfax company.

CONTEXTUAL INQUIRY - CLAIMS DEPARTMENT

To learn how the claims experience could be improved for a Fairfax insurance company, the Lab conducted a two-day contextual inquiry within their claims department. This included interviews and shadowing with various roles through the entire claim process. It also included interviews with risk managers to learn about how claims are handled and where hang-ups occurred. It concluded with recommendations through a mobile application prototype.

Format: 2-day Master-apprentice-style, shadowing of multiple functional roles across a claims department.

What we used: Laptops for note-taking, script, camera for workspace photos.

Research participants: 2 First Notice of Loss Administrators, 2 Claims Adjusters, 2 Claims Managers, and 2 Risk Managers (customers).

Analysis: flow model of claims, artifact model of physical and digital systems.

Decisions and Action: Prototyped and reviewed the Claims Assist App to be shared internally for consideration.

SMall business Field Research

In an effort for a team at a Fairfax company to better understand the small business market, the Lab did research with small business owners in both the Florida and California markets. To do this, we prototyped a fictional landing page and the sign-up process for a mobile app. We conducted some interviews on-site at the place of business as well as in a research facility equipped with one-way mirror and observation room.

Format: Interview with prototype and ‘think-aloud’ protocol.

What we used: Laptop for prototypes, Figma prototype, script, NDA, audio recorder, gift card compensation, Rev voice transcription service.

Research participants: 20 small business owners within Oakland, San Francisco, and Miami.

Analysis: Affinity map with themes for value proposition “proof points”.

Decision and action: Results shared proposal for next steps at culminating project meeting. Based on results, team was created for product conception with results in mind; however, the prototype concept and value proposition tested with business owners not directly continued.

Unmoderated Value proposition testing

To explore new products and business models, FairVentures will often create value proposition prototypes and test them with potential users to gauge response and gather feedback to refine our thought process. We’ve conducted several of these value proposition tests using different product concepts across multiple customer segments.

Format: 20 minute task-based unmoderated test with survey and landing page prototype.

What we used: Validately or UserTesting.com, Figma prototype.

Research participants: >100 users across multiple market segments.

Analysis: NPS (likelihood to tell another), collateral (email addresses), quotes, features rank.

Decision and action: Results of value proposition tests are shared with Fairfax SMEs and business leaders to determine if light concepts gets sponsored for a deeper proof-of-concept.

Analytics and telemetry - Binder

To evaluate where to improve Binder, the direct-to-learner tool in the Brightspace suite of products, I worked with the team to better understand the conversion rates at different stages of product use and churn rates for “active” users. This helped us determine where we could apply design effort for the biggest impact.

Format: Telemetry of user actions within Binder app.

What we used: Excel, Lean Analytics SaSS Customer Lifecycle Model.

Research participants: All Binder users in rolling monthly cohorts.

Analysis: Cohort, churn, other conversions within from the customer lifecycle model.

Decision and action: Based on low activation rates from other tools, design team worked on onboarding experience for Binder from other tools.