How does MyOutcomes model the true clinical population?
When trying to identify and measure any latent variable, two critical factors will play a major role. The first is sample size. The larger the sample size, the greater the variance and, therefore, the greater the likelihood of extracting the variable. An additional benefit of having a large sample size is that the sample has a higher probability of approximating the true population. The second key factor is the statistical model to be used. The ideal statistical model should be able to detect the variable, as well as be able to model and make predictions about the variable as it truly exists in the population of interest. MyOutcomes can easily meet these two critical factors. As a result, MyOutcomes has the power to predict change.
MyOutcomes' database consists of a sample with well over half a million measurements. This sample can be considered to be fairly representative as the measurements come from a broad range of countries and clinical settings. We are confident that MyOutcomes easily meets the following four sampling conditions necessary for predictive statistical modeling:
– Randomized sampling from a defined population
– Independence of sampling
– Normal distribution
– Population variances being equal
MyOutcomes' statistical modeling uses algorithms that have passed extensive cross-validation analyses. The PCOMS (Partners for Change Outcome Management System) development team was led by Dr. Barry Duncan and included Professor Michael Toland, a statistician at the University of Kentucky. A MyOutcomes' independent analysis was also completed and was led by statistician Douglas L. Steinley at the University of Missouri. The teams used their considerable years of clinical experience to develop and validate the model.
Initially, the development team eliminated extreme outliers from the database. These outliers are viewed as errors resulting from the use of MyOutcomes by inexperienced clinicians. These errors, which disappear as a function of increased use and experience with PCOMS, could impact the algorithms' ability to make accurate and practical predictions.
Based upon clinical experience, theory and research in the clinical field, the development team's a priori assumption was that therapeutic progress and outcomes should be described by a curve following a non-linear growth function. Analyses of models conditional on intake score demonstrated that a cubic model, rather than a quadratic model, provided the best fit for the data.
Considerable in-depth testing was conducted on the statistical model. Using descriptive statistics analyses, the development team evaluated the model's ability to predict trajectories for each intake score, as well as the means across all sessions in the database. Individual scores were plotted and compared to the expected treatment response predicted by the algorithms. The algorithms passed this very extensive testing process.
To evaluate whether the model predicted too much change, the development team used the data sets from published randomized clinical trials (RCT) of PCOMS (Anker, Duncan, & Sparks, 2009; Reese, Norsworthy, & Rowland, 2009; Reese, Toland, Slone, and Norsworthy, 2010) to examine how much change occurred in the feedback conditions. The algorithms used by MyOutcomes were found to be effective in accurately predicting change. The average amount of change across the feedback conditions in all three RCTs was 10.1 points. This value includes all the clients, those who changed and those who did not. Algorithms predicting far less change, not only wouldn't match the feedback RCTs, but would also inflate outcomes. This would ultimately affect the clinician's sense of how effective they really are, which is exactly what PCOMS is designed to prevent.
In practical terms, the increased sensitivity of the PCOMS algorithms to detect change and do a better job of predicting what change to expect translates into providers feeling even more confident that MyOutcomes helps them to provide their clients with the best quality service.
Based on this information, we are confident that the statistical model currently used by MyOutcomes v.12, represents a true clinical population. MyOutcomes has proven to be an effective clinical tool, providing vital information on the status of treatment outcomes. Together with their own theory and knowledge, MyOutcomes helps the therapist identify those clients who are not responding to clinical treatment. MyOutcomes enables the clinician to address a lack of progress in a positive, proactive way that keeps clients engaged while therapists collaboratively seek new directions.
To know more about how MyOutcomes can help you and your clients, call us toll free on 1-877-763-4775