Leapfrog Hospital Survey Results are publicly reported on Leapfrog’s website and used by purchasers and consumers, regional and national health plans, transparency vendors, and researchers. The accuracy of the data we publish and distribute is of utmost importance. Leapfrog has several protocols in place to ensure the accuracy of our results. Each protocol is described in detail below.
Affirmation of Accuracy
At the end of each section of the Survey, the hospital's CEO, or their designee, completes an Affirmation of Accuracy, affirming that the information the hospital has submitted in that section of the Survey is indeed accurate. Leapfrog reserves the right to require documentation before certifying and/or publishing any hospital’s Survey Results and does exercise this right at random.
Data Review Warnings in the Online Survey Tool
Leapfrog’s new Online Survey Tool requires hospitals to “check for data review warnings” before they are able to submit a Survey. When a hospital clicks this button on the Survey dashboard, sections that have been completed and affirmed will be scanned for potential data entry errors and inconsistencies. For example, if a hospital reports licensed and staffed ICU beds in Section 1 Basic Hospital Information, and then reports not caring for patients in an ICU in Section 5 ICU Physician Staffing, a “data review warning” will be generated. The hospital will still be able to submit their Survey but will be contacted via email by the Help Desk to either (1) correct the error or (2) document that the original response was correct. The data review warning system gives hospitals an opportunity to correct potential errors immediately – while they are still in the Online Survey Tool.
If a hospital receives a "data review warning," they can either:
- Review their response and make corrections on the spot. Hospitals will need to re-affirm any updated sections before they attempt to submit them again.
- Print their “data review warning” and investigate the issue. Hospitals are encouraged to contact the Help Desk if they have any questions.
Monthly Data Review
Upon the receipt of a hospital’s submitted Leapfrog Hospital Survey, The Leapfrog Group conducts an extensive data review of the quantitative responses submitted in the Survey.
Quantitative responses are assessed using empirically driven, normative data quality standards. The data quality thresholds are constructed with both high and low benchmarks of acceptability. Data quality thresholds for each question are determined from both external data sources (e.g., state quality reports, other national performance measurement entities) and historical Leapfrog Hospital Survey data. Applying this review methodology, The Leapfrog Group assigns each response into one of three categories:
- Category C: The response provided is plausible and within the data quality thresholds; no follow-up with the hospital is needed on this response.
- Category B: The response provided is plausible, but it falls outside the data quality thresholds; follow-up with the hospital is needed on this response.
- Category A: The response provided is considered implausible, given that the response falls outside of the data quality thresholds and appears to have been provided with the intent to mislead; follow-up with the hospital is needed on this response.
If a response is categorized in either category B or category A, an e-mail is sent to the hospital’s primary Survey contact and system contact (if listed). The e-mail: 1) details responses that Leapfrog would like the hospital to review; and 2) asks the hospital to re-read the associated Survey questions and specifications, and provide updated responses if needed. Hospitals with category A (likely implausible or intentionally misleading) responses are required to either update their initial response with a plausible response or provide a written explanation of their initial response within 30 days. Otherwise, the section containing the category A response is decertified.
The Leapfrog Group conducts this data review monthly throughout the Survey cycle (April 1 – December 31), beginning with Surveys submitted by June 30. In the interim (that is, prior to receipt of a hospital’s response to Leapfrog’s data review e-mail), the original Survey Results are posted publicly on Leapfrog’s consumer website.
When Leapfrog is contacted about a particular survey response or Leapfrog representatives are made aware of a potential error in reporting, Leapfrog does investigate these concerns and the hospital’s Survey is subject to decertification.
Monthly Requests for Documentation
In addition to the pre- and post-submission data review described above, Leapfrog also randomly selects hospitals to submit documentation to verify the accuracy of various responses from Sections 2 CPOE, 4 Maternity Care, 5 ICU Physician Staffing, 6 NQF Safe Practices, 7 Managing Serious Errors, and others. Hospitals must respond to the documentation request within 30 days or risk having the section decertified.
On-site Data Verification
The on-site data verification protocol builds on Leapfrog’s already robust process for reviewing Survey responses, and, at the same time, answers hospitals’ requests for more information on using the Online Survey Tool and Survey Results for quality improvement.
Leapfrog has commissioned DHG Healthcare to oversee and conduct the on-site verification program with a team of expert reviewers. The verification team operates separately from other healthcare consultants within DHG and does not provide consulting services for any of the hospitals selected for verification. Leapfrog partnered with DHG Healthcare to complete both a feasibility study and national pilot of an on-site data verification protocol.
In 2018, DHG Healthcare randomly selected a small number of hospitals for on-site data verification. The hospitals selected for 2018 have been notified. Hospitals that have not received notification will not be selected for on-site data verification this year.
On-site data verification will be conducted by DHG Healthcare, with technical support from The Leapfrog Group. DHG Healthcare is the national healthcare practice of Dixon Hughes Goodman LLP. Please visit their website for more information.
The protocol for the on-site data verification includes the following for each selected hospital:
- Orientation Call: This call is an opportunity for DHG to review the pre-visit documentation request, the Leapfrog Hospital Survey Binder, finalize the agenda for the half-day scheduled visit, provide examples of the methods that will be used to validate Survey responses, and answer any questions the hospital and their team may have. DHG will provide an orientation checklist to help the hospital determine who should be on this call.
- Pre-visit Documentation Request: Hospitals will use the Leapfrog Hospital Survey Binder while they are collecting information for completing the 2017 Leapfrog Hospital Survey so they can concurrently record and track the information needed for the on-site data verification. As a result, retrospective data collection and record finding will not be necessary.
- Half-Day Scheduled On-Site Visit, which includes:
- Introductory Meeting (approx. 20 minutes): DHG will meet with the hospital's pre-selected team to review the agenda for the day and answer any questions the hospital may have.
- Validation of three to four specific measures (approx. 2.5 to 3 hours): DHG will work through the established agenda by meeting with various team members to administer the validation protocol for selected measures.
- Discussion of quality improvement strategies tied to Leapfrog measures (approx. 30 minutes): DHG will meet with the hospital and their team to share examples of quality improvement strategies in use by other hospitals across the country. DHG will also ask the hospital and their team to share ideas for how Leapfrog could make changes to the Survey and/or Survey Results tools so they can be better used in quality improvement.
- Exit Interview (approx. 20 minutes): DHG will wrap up the day by reviewing the validation findings report and outlining any necessary next steps. In addition, the hospital and their team will be asked to share feedback on the tested protocol.
Technical Support from the Leapfrog Hospital Survey Help Desk: If DHG discovers any issues with a hospital’s submitted responses, a summary of those issues will be made available to the hospital (see description below) and the Leapfrog Hospital Survey Help Desk. Help Desk staff will work with the hospital so that they are able to update their Survey appropriately.
Findings Report: A hospital-specific findings report will be shared with the hospital. Findings are not publicly reported.
Leapfrog Hospital Survey Binder
The Leapfrog Hospital Survey Binder was originally designed for those hospitals that have been selected for Leapfrog’s On-Site Data Verification Program administered by DHG Healthcare. Those hospitals are mailed a binder to collect and organize the information (e.g., reports, documentation, notes, etc.) they used to complete their Leapfrog Hospital Survey AND at the same time, they are preparing for their verification. DHG uses the information in the binder during the scheduled visit to verify a hospital’s Leapfrog Hospital Survey responses.
However, the Leapfrog Hospital Survey Binder document (PDF) is available for use by all hospitals to collect, organize, and record information during the completion of the 2018 Leapfrog Hospital Survey. The document, which is divided up into 10 tabs (or sections), one for each section of the Survey, can be printed and placed in a binder. The information is helpful when completing subsequent year’s Surveys, in staff and leadership transitions, and as a historical record. The use of the binder also acquaints hospitals with the elements of the On-Site Data Verification Program.