What Are Risk and Reliability Adjustment and Why Do They Matter?
Measuring clinical performance is challenging. Some patients are older or sicker; some have more advanced clinical conditions. Measurement populations are rarely large enough to eliminate chance. To accurately measure clinical performance, you need to control for these confounders and mitigate the role of chance. You need accurate measurement of outcomes, processes, and costs to meaningfully improve the quality and efficiency of care delivery. You also need to be able to compare the performance of different providers to one another and track changes over time.
To that effect, there are three key technologies that are essential to success in measuring and improving performance in healthcare: risk adjustment, reliability adjustment, and on-demand performance feedback.
In this post we will cover:
- Risk and Reliability Adjustment in Healthcare: The Basics
- Risk Adjustment Methods
- Reliability Adjustment Methods
- On-Demand Performance Feedback
Risk and Reliability Adjustment: The Basics
Let’s start out with a simple equation:
Now, let’s break this equation down and define each component.
What Is Risk Adjustment?
Risk adjustment is a process that corrects for the severity of a patient’s illness. In other words, adjusting for risk levels the playing field and ensures that comparisons of hospitals and clinicians are fair and accurate.
Why Is Risk Adjustment Important?
Some hospitals and clinicians treat more high-risk patients than others. Because of that, comparing a metric like mortality rates between two different hospitals isn’t always helpful, because one hospital is working with much more severely ill patients than the other.
In short, it’s challenging to compare outcomes like complications, utilization, and mortality rates for patients with the same condition but different health statuses.
That’s where risk adjustment comes in. Risk adjustment uses statistical models to account for clinical risk factors that differ between patients. For example, an 85-year-old female undergoing cardiac surgery is more likely to suffer adverse outcomes compared to an otherwise healthy 50-year-old male undergoing the same procedure. Comparing the two patients as if they had the same level of risk would be misleading.
We have developed our own risk adjustment methodology, which we’ll discuss in more detail in the next section. In short, we run data through scientifically and clinically validated statistical models that explicitly evaluate all patient factors that may be related to each adverse outcome. Our modeling approach adheres to all of the best practices in risk adjustment, but with significantly improved efficiency.
Reliability adjustment is a statistical technique based on hierarchical modeling that was pioneered by the ArborMetrix founders [1] and is designed to isolate the signal and reduce the noise in large datasets.
When sample sizes for a hospital or clinician are small, it can be difficult to determine if outcomes such as mortality are due to chance or to true differences in quality. Traditional analytical methods often report outcome numbers that are misleading, because clinicians and hospitals can’t determine whether they are truly different from the average.
Additionally, statistical measures like confidence intervals and p-values are often misunderstood, ignored, or relegated to a footnote. This lack of accuracy can lead to care teams addressing problem areas that are, in reality, not problematic — leading to loss of time, effort, and expense.
To optimize the effort of quality improvement work, we provide reliability adjustment in addition to risk adjustment.
Together, these methodologies ensure patient outcomes and other performance metrics are reported accurately and with enough actionable clinical detail to effectively inform improvement efforts. Importantly, these methodologies are essential to building trust in the data.
Next, we’ll take a closer look at the methods used in both types of techniques, starting with risk adjustment.
Risk Adjustment Methods
ArborMetrix has developed a unique, best-in-practice risk adjustment methodology to significantly enhance analytic efficiency. In our methodology, registry data is run through scientifically and clinically validated statistical models that evaluate all patient factors that may be related to each outcome.
Additionally, we have introduced several innovations into our technology, such as real-time model recalibration, that automate the risk-adjustment process and ensure models are well-fit to changing data.
Let’s dive a bit deeper and explore the steps involved in developing a risk model.
In practice, the specific risk factors included in the model will vary by outcome and by the patient populations included in the denominator for the measure. For example, some of the most important comorbidities in a risk model for serious complications of colectomy surgery include factors such as anemia, renal failure, and weight loss.
Risk adjustment corrects for the individual severity of patient illness within a dataset. With our highly efficient methodology, you can be sure that comparisons between clinicians, hospitals, or regions are accurate.
Reliability Adjustment Methods
Pioneered by our founders, reliability adjustment is a statistical technique based on hierarchical modeling that is designed to isolate the signal and reduce the noise in your dataset.
More specifically, when sample sizes for a hospital or clinician are small, the observed rates or rare outcomes may be due to chance and should be considered less precise than rates based on larger sample sizes.
What Is Reliability Adjustment?
Reliability is a measure of this imprecision, where a value of “0” means the outcome is 100% noise (completely unreliable) and a value of “1” means the outcome is 100% signal (perfectly reliable). In practice, most outcomes have reliability between 0 and 1.
Why Is Reliability Adjustment Important?
Reliability adjustment is important in certain circumstances because small sample sizes or rare outcomes create uncertainty for a hospital or clinician around whether differences in outcomes such as mortality were due to chance or true differences in quality.
For example, a hospital with a mortality rate of 50% because 1 patient died in a total of 2 cases should be interpreted very differently than if 100 out of 200 patients died. For most surgical procedures, they would need to have its mortality rate reliability-adjusted to account for the prediction that if they had 200 cases instead of 2, it would be very unlikely that their mortality rate would still be 50%.
We work with our registry partners through a methodical planning process to develop and refine clinically relevant and scientifically accurate reliability models.
Without reliability adjustment, traditional analytical methods often report outcome numbers that are misleading to clinicians and hospitals, who cannot sort out whether their performance rates are truly different from the average. Reliability adjustment helps clinicians and hospitals identify real problem areas and focus improvement activities where the greatest improvement opportunity exists.
In short, reliability adjustment corrects for random error, allowing you to trust that the analytic insights you’re generating are as accurate as possible.
MSQC Relies on Risk and Reliability Adjustment to Improve Performance
The Michigan Surgical Quality Collaborative (MSQC) is a national leader in surgical quality improvement. They help surgeons improve at the individual level by providing surgeon-specific reports to aid in performance self-assessment.
Risk adjustment is crucial to the hospital-to-hospital comparisons done routinely by MSQC, and the risk adjustment modeling and reporting development represent a combined effort between the MSQC analytic team and ArborMetrix. After the process of risk adjustment, the outcomes data is also reliability adjusted.
MSQC’s real-time, risk- and reliability-adjusted reports offer a look at each surgeon’s individual outcomes data, with a detailed drill-down on patient-level data included in the analysis. Surgeons can view their performance relative to their hospital or to the collaborative as a whole. Surgeons can also view their performance in comparison to other contributing surgeons, both within their institution and beyond, in a de-identified manner, providing even more valuable information to help surgeons identify areas for quality improvement.
MSQC is a top example of a clinical registry program that is achieving real-world quality improvement. Its members have collectively achieved outstanding results and greatly enhanced care for their patients. A few highlights include:
• 34% reduction in surgical site infection rates for colectomy procedures.
• 50% reduction in patient opioid consumption across 9 surgical procedures, with no changes to patient-reported satisfaction.
• 40% reduction in morbidity and mortality of non-trauma operations.
• Regularly published research in leading academic journals.
On-Demand Performance Feedback Makes Rapid Healthcare Improvements Possible
Having access to accurate, trusted analytical insights is helpful. Being able to act on those insights as quickly as possible is even better. That’s where on-demand performance feedback comes in.
ArborMetrix’s products deploy rigorous analytics, grounded in scientific best practices, allowing users to quickly and easily measure performance and understand their clinical outcomes and business. Our unique technology allows for reporting and the application of data science in real-time. This gives physicians access to up-to-date risk- and reliability-adjusted reporting, which builds trust and enables rapid-cycle improvements.
Traditional processes for producing reports with risk- and reliability-adjusted measures can take months using manual methods performed by statisticians. As a result, reports on comparative outcomes have traditionally been delivered in static formats that reflect old, stale data that may no longer be useful for quality improvement efforts.
With the interactive ArborMetrix platform, end-users can select different levels of aggregation, patient populations, and time periods to see dynamic risk- and reliability-adjusted reports that reflect the most up-to-date data. The combination of fresh data and flexible exploration of outcomes information means you can get useful feedback quickly.
The visual below shows an example of an interactive report where the user has looking to understand in-hospital mortality across several measures.
Work With Data You and Others Can Trust
Together, risk adjustment, reliability adjustment, and on-demand performance feedback build trust in the data, which trickles down to trust in the evidence, trust in the insights, and trust in the conclusions drawn from the data. Further, having access to all of this information in real-time ensures that quality improvement efforts won’t go to waste.
Ready to get started? Talk to our data science experts today and begin building a registry you can rely on.
Steps to Address Health Inequity: Awareness, Measurement, and Action
There are widespread, longstanding, systemic, and well-documented preventable health disparities among racial and ethnic minorities and amongst those in LGBTQIA+ communities. Minority populations experience a multiplicity of burden in regard to their health including poorer overall health statuses, worse outcomes, and disproportionate diagnosis of health conditions due to the intersection of structural inequity.
Barriers to access, differences in clinical need, and differences in patient treatment exacerbate these poorer health outcomes. Given this unfair burden, healthcare stakeholders have an ethical obligation to change healthcare delivery to address health inequity.
The Council of Medical Specialty Society (CMSS) provides a forum to catalyze healthcare improvements across U.S. medical specialty societies. They held the CMSS Leadership Summit on Equity and Antiracism in April 2021 to focus on methods to reduce health disparities, address issues perpetuated through dominant culture dynamics, and improve equity within medical specialties. Key objectives included:
- Increasing awareness of longstanding, health inequity and health disparities.
- Identifying and measuring disparities.
- Forming action to address disparities.
In this post, we share key learnings from the summit and expand on the role of clinical registries in this imperative work.
Increasing Awareness of Health Inequity and Disparities
The initial step to increasing awareness around inequity and the role of bias in health disparities is acknowledging the extent to which these systematic processes harm health outcomes.
These processes permeate healthcare at every level: from the language we use to talk about racism and bias, to algorithms built on racist data that are used to perpetuate myths about genetic and physiologic differences between patients of different races [1], to the exclusion of minorities in medical schools, to the lack of provider diversity, and ultimately in patient care.
This work begins with the way we speak about bias in general, and racism specifically. The language used to describe processes that exclude people who do not conform to dominant sociocultural norms indemnifies the very groups that perpetuate these processes. [2] This camouflages how detrimental bias is to both wealth-building and health outcomes. In changing how we talk about persons that do not hold power and influence in society, firstly we recognize that there is a difference in treatment based on power dynamics and we open the door to changing that dynamic.
Increasing awareness must be a systematic process occurring at every level of healthcare. This begins with recognizing race as a social construct that has physical implications [3] and treating social data with the same gravity as traditional medical data and enacting policy to support the collection and bidirectional transfer of these data. It also includes creating psychologically safe environments that encourage allyship amongst clinicians of dominant sociocultural groups and marginalized people. [4]
To dismantle racism and bias within healthcare, we must prioritize diverse perspectives amongst clinicians and the healthcare delivery system as a whole and normalize diversity, equity and inclusion initiatives. [5] To increase awareness surrounding health inequity, we are educating and training our team, focusing on ways to integrate the use of social data in registries, and supporting our clients in doing the same. We are also developing partnerships with and learning from experts who can help accelerate our work in measuring and improving health equity so that we can in turn support organizations like CMSS.
Identifying and Measuring Healthcare Disparities
Advancing health equity work in the pursuit of reducing and eliminating health disparities involves detection and measurement in addition to understanding the determinants. Clinical data registries can be instrumental in these tasks by assessing the causes and impact of health disparities on patient outcomes.
According to the Centers for Disease Control and Prevention (CDC), only 10 percent of factors affecting premature death are related to clinical care, and 30 percent of factors relate to genetics. This means that 60 percent of factors impacting premature death are based on a combination of social/environmental factors (20 percent) and behavior (40 percent). Ensuring the delivery of high-quality, patient-centered care requires understanding both the needs and circumstances of the populations served. Registries can collect and analyze patient, clinical, social, and geographic data to provide a more complete view of the type of advantages or disadvantages certain patients have in attempting to achieve optimal health. [6]
Fully identifying and measuring disparities and their root causes necessitates data that goes beyond clinical data elements. This includes the prioritization and comprehensive collection of social data such as:
- ICD-10-CM codes (Z55-Z65) that detail patient social risk factors
- Utilizing public health datasets such as the American Community Survey, Behavioral Risk Factor Surveillance System and CDC National Vital Statistics System
- Patient-reported outcomes data
Integrating comprehensive social data elements in the United States Core Data for Interoperability (USCDI) and enforcing mandatory collection within electronic health records (EHRs) are two steps that make up a meaningful process in measuring the extent and impact of social determinants of health.
Forming Action to Address Healthcare Disparities
Data and measurement are only as valuable as it is usable and actionable. Advancing health equity begins with understanding the sources of disparities, but it can only be achieved through effective action.
Once disparities are assessed and the causes are understood, impactful interventions can be developed using data-driven approaches.
However, implementing informed interventions is only the first step. Next, interventions must be evaluated for effectiveness, changed to address problems as necessary, and finally, best practices must be disseminated to scale.
Leveraging Clinical Registries to Improve Health Equity
To promote optimal patient care, it is our ethical obligation to understand the prevalence and severity of social and health inequities, utilize comprehensive data collection and analysis, and take informed action to end them. We must evaluate and implement interventions that effectively improve outcomes for the most vulnerable populations, which in turn improves conditions for all populations.
Clinical data registries are valuable tools for pursuing these efforts. Registries can collect a high-volume of data from different sources and deliver high-quality, data-driven insights. Reduction and eventually elimination of health disparities require data to detect and measure impact, identify the complex and interconnected determinants, and inform interventions and best practices.
We’re interested in hearing how your organization is addressing these important issues and what challenges or questions you are finding. If you have insights to share or questions to ask, reach out to me at cprice@arbormetrix.com.
I’d like to thank my colleague, Bradley Moore, M.P.H., Director, Registry Solutions, for his significant contributions to this post.