The Measurement Gap States Can't Afford to Miss

Centers for Medicare & Medicaid Services (CMS)has committed $50 billion to transform rural health careEvery state that received funding has laid out goals for what that investment is supposed to accomplish. But there's a problem that doesn't get much attention: do states actually have the data they need to prove those goals are being met? 

The assumption that doesn't hold up

 Most rural health initiatives assume the data needed to measure progress already exists. They assume the data is reliable, complete, and detailed enough to show whether programs are making a difference. In many rural communities, that's simply not the case. Most administrative data systems were built for billing and operations, not for evaluating rural health programs. Those systems weren't designed with small populations, isolated communities, or new models of care in mind.

Three data problems rural health programs run into 

1. Small populations make the numbers unreliable

Many rural areas don't have enough people to produce stable health measures each year. Federal reporting rules also suppress results when there are too few events, making some metrics impossible to report. Even when data is available, annual rates in counties with very small populations can swing dramatically from year to year. Sometimes what looks like improvement, or decline, can just be random variation.

2. Some of the people you're trying to help aren't in the data

Most state reporting relies heavily on Medicaid and Medicare claims and encounters. That leaves out many rural residents who don't interact with those systems, including uninsured farm workers, the privately insured, and people receiving services through community health workers instead of traditional providers. A program may improve access and health outcomes for these communities, but if those people never generate claims data, the impact is largely invisible.

3. New programs don't have a true baseline

ManyRural Health Transformation Program (RHTP) initiatives are creating services that never existed before, such as telepsychiatry programs or community paramedicine. If there was no service before the grant, there isn't a meaningful pre-program baseline to compare against. Measuring outcomes in the first year often tells you where the program started, not how much it improved. That's an important distinction, and one that's easy to overlook.

States need evaluation approaches that reflect the realities of rural communities rather than treating them like larger urban populations.

That usually means:

• Combining multiple years of data to create more reliable estimates in areas with smaller population sizes.Many rural counties don't have the patient volume for single-year estimates to be statistically stable.A handful of behavioral health deaths or EMS calls in one county can swing dramatically year to year, making multi-year pooling the difference between a defensible trend and statistical noise.
• Establishing clear benchmarks for new programs before data collection begins.For a program creating services like telehealth expansion where none existed before, the benchmark has to be set at zero or at a proxy measure. Waiting for a 'baseline year' that will never meaningfully exist risks the same problem: first-year numbers that tell you nothing. Benchmark decisions need to happen at the evaluation-planning stage, not after year-one numbers come in.
• Handling federally protected data consistently across reports and dashboards.A performance indicator defined one way in a quarterly report and another way in a dashboard, say a different date range, denominator, or suppression threshold, leaves stakeholders looking at numbers that don't reconcile. Locking in those definitions early, and holding them steady, is what keeps five years of data telling one consistent story.
• Pairing administrative data with interviews, surveys, and other qualitative research to understand what is happening on the ground.Administrative data alone often cannot explain why metrics such as Specialty Care Access have moved. A spike in specialty referrals could reflect real new access, or it could reflect a single provider hire in one county. Only ground-level interviews with practitioners and staff can distinguish a systematic win from a fragile one-off.
• Comparing results with similar states or national benchmarks when local comparisons aren't meaningful.Pairing state-level indicator movement with cross-state RHTP data gives a credible answer when CMS asks how your state is performing relative to other states, rather than a number floating with no frame of reference.

These approaches are what make rural health evaluations credible.

CMS is tying continued RHTP funding to demonstrated progress. Legislators, community leaders, and residents are watching for proof that these programs work. If a state's evaluation cannot separate real improvement from statistical noise, seasonality, or normal year-to-year variation, it becomes much harder to defend those results during future funding reviews. States need both a credible measurement strategy and the grant management infrastructure to back it up.

The states best positioned for future funding reviews won't be the ones with the biggest numbers. They'll be the ones that built a measurement strategy they can defend, one that shows real and meaningful impact for rural communities.

How North Highland builds measurement strategies that hold up 

North Highland’s public health team has spent years helping states evaluate rural health initiatives, including a long-term partnership with a state Office of Rural Health and work supporting rural hospital quality and operational improvement.That experience has shaped a practical approach to rural health evaluation. It includes methods for handling small sample data, creating defensible benchmarks for new programs, and combining quantitative analysis with qualitative research to fill the gaps administrative data leaves behind. Most evaluation frameworks weren't built for the realities of rural data.

Ours were.

The goal is an honest account of what the data actually shows.

The federal investment in rural health is a rare opportunity. But showing that it made a difference takes more than collecting data. It takes building the right measurement approach from the beginning, before the annual reports are due.

 If your state is navigating these measurement challenges, contact our team to discuss your evaluation approach before the next reporting window opens.