Ask the Experts: KRI Thresholds

Q: We are accelerating implementation of our Centralized Monitoring program due to the COVID-19 pandemic. We’ve started with a small number of Key Risk Indicators (KRIs) including AEs/patient. We have upper and lower threshold limits, so we can look for outlier sites – those that are over- or under-reporting relative to others. But what we’ve noticed is that we are getting lots of false signals that are taking time to investigate. For example, a site that has recently been activated and only has enrolled one patient has no AEs/patient and looks like an outlier. Do you have any suggestions for how to set thresholds to take account of this? (Metric Insights, Apr. 2020)

A: With KRIs such as AEs/patient, you should use leading metrics – metrics that provide information that you can use to course correct on the current study. The problem you describe is not due to the KRI threshold levels – it is the metric that you are tracking. Cumulative AEs/patient is a lagging metric. During the study, as patients progress through their treatment, the likelihood that they will experience AEs increases – in other words, the opportunity for AEs to occur increases. In the example of the site that just enrolled its first patient, there are no AEs reported because the study is just getting started. The leading metric cumulative AEs per patient-week is a better measurement because it accounts for the length of time (number of weeks) each patient has been on the study. Using a version of the measurement that accounts for the time the patient is on the study can significantly impact how early a signal is detected and can help to reduce the number of false signals.

Additional details about defining KRIs are available in the paper we have recently published, Measure the Right Things at the Right Time: Design Key Risk Indicators and Key Performance Indicators that Provide Timely Insights During Study Conduct. We suggest you read the paper and review the definitions of your other KRIs to make sure they are leading metrics too. View the paper ›

Ask the Experts: Answers to your Metric Questions

With more organizations focusing on metrics, we have received an increase of questions ranging from how to use metrics, why some metrics are better than others, which type of metrics is best to use, as well as questions about specific metrics. This column provides a forum for us to share these questions and answers with you. Submit your question ›

Linda Sullivan

Senior Advisor, Metrics & Performance Management

WCG Avoca

Keith Dorricott

Metrics Subject Matter Expert

WCG Avoca