Product, Design & Research Leader
The Starling v5.0 platform delivers personalized, adaptive Cognitive Behavioral Therapy education and user self-harm monitoring. As Head of Product, I directed the building of StarlingAdmin to manage multiple aspects the platform.
Project Title: StarlingAdmin v5.0
Company: Starling Minds, Vancouver, BC
Years: 2019/20/21
Role: Head of Product, Design & Research, Chief UX Architect
Design Methodologies: design thinking, rapid prototyping
Research Methodologies: card sorting, lab usability testing, user interviews, survey design
StarlingAdmin v5.0 dashboard view.
Project Summary
With the development of its new v5.0 digital CBT offering, Starling Minds required an administration interface to manage multiple aspects of the platform. As Head of Product, I directed the team building this interface. As Chief UX Architect, I designed many aspects of the architecture and feature set. The interface itself included features for managing customer configurations, user accounts, member referrals, content, translations, and also provided an NLP-powered interface for user mental health monitoring, community moderation tools, and data visualizations.
The Challenges
As we were developing a new, automated and adaptive Cognitive Behavioral Therapy (CBT) app, we also needed to design and build an interface for internal management of the platform. The interface needed to have features basic to any SaaS CMS or EHR system, such as user management, customer configuration, content management, translation and versioning, but there were a number of more challenging requirements as well.
First of all, the system needed to provide personalized CBT training that would adapt to changes in user data and decisions. We had devised an architecture for creating learning experiences based on a heuristic model of Cognitive Behavioral Therapy that would guide the user through reading, videos, exercises, and interactions that would mimic the CBT skill building that they could experience in face-to-face therapy provided by a trained CBT practitioner. We had constructed methods and frameworks for content authoring, but we also needed to build tools to import and compile the designers' work on the platform without needing dedicated, ongoing engineering resources.
Second, because this would be an automated mental health system without human intervention, we also needed to have a way to monitor for potential abuse in the community forums.
Additionally, because the mental health solution was to be deployed in disability management and, potentially, primary care scenarios, a mechanism was needed to give disability case managers or healthcare providers the ability to refer to solution to claimants or patients and track the referrals.
Finally, also because of the lack of a human element to the CBT training, we needed to be able to predict risk of member self harm and suicide so that our Member Support team could intervene, if necessary.
The Solution
Design and development of the Admin interface began in early 2020, several months after design of the CBT app and content began. As this was to be a technical interface used only by internal Starling Minds team members, dismissing branding as a requirement, I opted to select a UI template in order to speed up development time and reduce costs associated with custom UI development. As Chief UX Architect, I worked directly with the Technology team, bypassing the need for a UI designer to be tasked to the project. The interface, like the CBT app, was built with Vue.js on a Java/Spring framework in a secure environment on AWS.
Backend EHR Features & User Communication
StarlingAdmin v5.0 combined basic functions commonly found in EHR (electronic health record) systems. These functions included user management, customer configuration, user progress and health tracking, translation management, and dashboard reporting. The EHR aspects of the UI allowed us to track user activity history and behavior within the app. We could also track an individual's assessment scores, answers to questions posed in the training scenarios, and assessment scores. Data visualizations were available to give our Therapy team the ability to monitor individual user progress and triage potentially risky or challenging cases. In the case that a member needed individual support, our Member Support team could reach out and communicate directly with those users through the app.
Detail of user progress in the EHR.
Content Import & Management / Xmind Integration
In order to build the complex, adaptive learning experiences required by the Therapy team for execution of their CBT methods, the Design team devised a process of building out the traversal maps in a mind-mapping tool called Xmind. These maps included branched options that the user could be guided through based on their data phenotype and decisions, the rules scripting language that determined the automated guidance, variables for conditional parameters, written content that would appear on the screen, and tags pointing to images, video, and other media.
You can read more about that process in this case study.
Initial versions of the maps were printed on a plotter for team collaboration and editing sessions. However, once the CBT experiences were designed and the maps were built, all of the content and rules had to be input manually through a very clunky and unstable prototype content authoring tool. Looking at our tightening production schedule, the looming dates for Beta release, and the exploding need for scalable online mental health tools in the early days of the COVID-19 pandemic, I made the streamlining of the content authoring process an urgent priority. I knew that without some sort of innovation to solve that problem, we would end up being months behind schedule, increasing burn and losing our shot to make an impact in the blossoming metal health technology market. But, I had also witnessed the inventiveness and tenacity of our new Technology team and put my faith in them to deliver a solution that would save us.
Early plans had been to build a JS/HTML5 Canvas application in the Admin interface that would have all of the features of a drag-and-drop mind-mapping tool, content versioning, and content management. Once tasked with the urgent priority of building a tool to shorten and stabilize the content authoring process, the Technology team came up with an inventive and novel solution: to build a middleware application that would import content in the form of the Xmind maps that the design team was already producing and parse that data onto the platform. Since the Xmind maps contained so much of the content and data required, an integration would free them up to focus on net new features, such as media management and custom debugging tools for the millions of possible content paths in the system. This integration reduced content importing to a fraction of the time the manual process took and with an exponentially reduced risk of human error that could result in days or weeks of delays due to troubleshooting and debugging content.
Content map import features.
Media management features
Community Moderation
Because of the lack of human intervention in the mental health approach provided by the app, the in-app user community was a popular source of emotional support needed by participants that could not be provided by the app. Moderation tools were built into the Admin UI, including comment tracking and suspension.
I also prioritized development of a secure in-app communication system that would allow Community Moderators to contact users within the secure environment of the app, rather than communicating with them by email as had been done previously. This mitigated privacy issues and contributed to HIPAA compliance.
One further feature for community moderation was banned content monitoring and flagging system. A list of terms, as indicated by our Terms of Use and Community Code of Conduct, were flagged in the system as potential abuses. If any of those terms were used in community comments, Community Moderators would be notified and could take appropriate action.
Referrals & Referral Tracking
The main use case for our Return to Health product was to help employees on a short or long-term disability claim cope with mental health issues and prepare to return to the workforce. In this workflow, our solution would be referred to claimants as part of their disability claim. The solution was also designed to be referred by healthcare providers in a primary care scenario.
As Chief UX Architect, I designed an EHR and monitoring portal for our customers, as well as features in the Admin UI for tracking and troubleshooting referrals. In the design of this solution, I leaned on my previous experience designing customer referral systems and a Referral CRM years before at RewardStream.
Although it was not considered an early priority on the v5.0 product Roadmap, I prioritized these features to increase case manager referrals and user adoption. Because our pricing model for the Return to Health product was based on referrals, the referral features proved to be an integral part of the user experience and go-to-market strategy. I also prioritized an integration into our backend billing system, streamlining customer billing processes and reducing collection time for receivables.
Referral management features
NLP User Monitoring & ML Risk Prediction
The automated and human-free aspect of the digital mental health solution was intended to increase scalability, but it also raised other issues pertaining to user health and safety. One frequently asked question in early customer research was how the system could spot and support users at risk of self harm or suicide. As a product solution to this problem, I tasked our Data Science team, led by the talented Dr. Hoora Moradian, to devise an automated system that would monitor user activity, flag users at risk of self-harm or suicide, and alert our Member Support and Therapy team for triage and possible intervention.
The resulting solution was a novel monitoring and alerting solution using NLP (natural language processing), ML (machine learning) research, a proprietary risk prediction algorithm, and an alerting system.
The NLP component was a relatively straightforward algorithm that monitored community comments for terms that indicated possible rick of self harm or suicide. The heart of this part of the system was a comprehensive library of terms compiled by Dr. Moradian. At my direction, she mined thousands of comments on Reddit forums containing posts made by people considering suicide. This was harrowing work, but the result was an incredibly comprehensive repository of data that could be scored and incorporated into the ML prediction algorithm.
The terms in the library were initially scored as indicators of risk by our Therapy team of Psychologists. Once we secured funding from Canada's Digital Technology Supercluster, I was able to broker a partnership with the University of British Columbia to aid in the training of the ML system. Through that partnership, we worked with researchers from the UBC and SFU Psychology departments who were experts in suicidal ideation. these researchers provided further scoring of the library of terms and tuning of the ML algorithm.
The risk prediction algorithm would assign a risk score to comments, and those score above the indicated threshold would result in an alert sent to the Member Support team. The initial prototype of the system was built using our Tableau enterprise reporting system, and later would be integrated into the the main platform.
Results
The StarlingAdmin interface was a successful workflow tool that increased product time to market and user safety.
Since release of StarlingAdmin 5.0:
-
98% reduction in content time-to-market compared to initial manual authoring processes,
-
52% reduction in customer implementation time compared to the v4 platform,
-
5x increase in productivity by the Community Moderation team,
-
100x more scalability of member monitoring compared to previous manual processes.