About the AAFP-PT Evaluation
When to Expect Your Evaluation
Evaluations are typically available approximately 30 days after the result due date. This time allows for processing the data. AAFP-PT recommends marking your calendar to remind you when to expect your evaluation. The estimated evaluation release date will appear on your PT Central home page shortly after the close of the testing period. Participants will receive an email with the evaluations attached.
Participants may view and, if desired, print any current or previous evaluations by logging into their PT Central home page. Once logged in, follow these steps to access your evaluation:
- Choose “Reports” from the menu on the left side of the screen.
- Select the desired event and year from the dropdown menus and click “Apply.”
- Then click “View” next to PT Evaluation Report. This will open the Evaluation as a PDF document.
During each event, AAFP-PT collects data submitted by participating labs. The data is compiled and Peer Group, Method Group, and All Method statistics are generated based upon:
- The Peer (instrument/method), Comparative Method (like methods/technology) and All Method groups (all labs reporting for the specimen and analyte) are formed.
- A data evaluation process (statistical analysis) that is performed to calculate the quantitative statistics.
- The mean, median, and coefficient of variation (CV) are evaluated and outliers are removed from the data. The mean and median are compared to “closeness” in value; the lower the CV, the more “precise” or less variable the data. As a rule, outliers are data points that are less than or greater than three standard deviations from the mean.
- The acceptable range (calculated based on the grading criteria established by the CLIA regulations) and the percent consensus (number of passing labs divided by the total number of labs multiplied by 100) are then calculated.
If there is a sufficient number of labs using the same method/instrument that have reported results, then these labs are compared.
- If the Peer Group consensus is 80% or greater, the results are graded by the Peer Group statistics.
- If the Peer Group consensus is less than 80%, the results may be graded by either referee labs or by All Method statistics – provided that referee or All Method statistics achieves 80% or greater consensus.
- If 80% of greater consensus is not achieved with Peer, Referee, nor All Method statistics, the results are not graded due to lack of consensus.
If there is not a sufficient number of labs using the same method that have reported results, the labs are compared with either Comparative Method (methods determined to have the same methodology or technology) or All Method statistics.
- If the Comparative Method group consensus is 80% or greater, the results are graded by the Comparative Method statistics. If Comparative Method group consensus is less than 80%, the data is reviewed by the All Method statistics.
- If the All Method group consensus is 80% or greater, the results are graded by the All Method statistics unless it is determined that the Peer Group values vary excessively from the All Method values.
- If no Comparative Method group can be established and the All Method group consensus is less than 80% and, the results will not be graded due to lack of consensus.
If there is not a sufficient number of labs reporting at All Method, the results are not graded due to no comparison group.
- If the total analyte consensus is 80% or greater, the results are evaluated.
- If the total analyte consensus is less than 80% referee labs are selected. If the referee consensus is 80% or greater, results are evaluated. If the referee consensus is less than 80%, the results will not be graded due to lack of referee consensus.
Miscellaneous Grading Situation
Occasionally, AAFP-PT, in consultation with advising physicians, determines that a specimen or analyte should not be graded following review of the results. Reasons may include documented specimen problems, result variance, and invalid results. The evaluation clearly states the reason for not grading a specimen.
Labs are required to review and evaluate proficiency testing results according to the CLIA regulations. A “PASS" in the column labeled “Comments” requires no additional follow-up unless the result has passed because the specimen was “Not Graded.” Any analyte (both regulated and non-regulated), specialty, or subspecialty assigned a score that does not reflect the laboratory’s performance must be evaluated for accuracy by the laboratory. This includes a review of actual PT results against the PT provider’s participant summary results. This requirement is now specified in the regulations. Situations requiring this review include:
- 100% score given due to analyte, specialty or subspecialty not being graded because of a lack of consensus or other discrepancy.
- The CLIA grading criteria are applied to establish the acceptable range for each specimen/analyte. When 80% or greater consensus is not achieved with Peer, Referee nor All Method statistics, results are not graded due to lack of consensus.
- For the various "Not Graded" reasons listed in the "Comment" column of your evaluation, compare your results to the acceptable ranges in the Event Summary. Again, document what you have done and how you would have performed if the specimen had been graded.
- 0% score given due to lab’s failure to participate in PT.
- 0% score given due to lab’s failure to return results to PT program in required timeframe.
Non-regulated analytes and analytes for which compatible PT samples are not available must also be evaluated for accuracy; this may be accomplished by split or blind testing of materials with known values or other external assessment programs.
A "FAIL" result always requires follow-up investigation and documented corrective action.
The CMS Performance Summary
A cumulative report indicating your performance status for all regulated analytes, specialties, and/or subspecialties for which you are enrolled. Federal regulations do not require waived or non-regulated analytes to be included on this summary.
- Unsatisfactory performance means failure in a specific event to attain the minimum satisfactory score when compared to the target value or reference mean for the peer group for an analyte, test, subspecialty, or specialty.
- Unsuccessful cumulative performance means a failure to attain the minimum satisfactory score for an analyte, test, subspecialty, or specialty for two consecutive or two of three consecutive testing events.
All Method Statistics – The combined statistics for all labs reporting for the specimen and analyte.
Peer Group Statistics – The combined statistics for labs reporting with the same methodology/instrument.
Comparative Method (Method Group) Statistics – The combined statistics for labs reporting with like methods/instruments.
Not Graded/Lack of Consensus – Less than 80% of participants in the Peer Group reporting results for the analyte are within the established acceptable range.
Not Graded/No Comparison Group Found – Less than five laboratories in Peer Group. Unable to establish a scientifically defensible statistical range for evaluation as defined by CLIA. (Unable to grade by using neither comparative method nor all method statistics.)
Not Graded/Lack of Referee Consensus – Less than 80% of referees reporting results for the analyte are within the established acceptable range.
Not Graded/Specimen Problem/Unable to Obtain Result – Laboratory indicated specimen problem. This typically occurs when the lab is unable to obtain results for the specimen due to specimen performance and no replacement specimens are available.
Not Graded/Excessive Variability Data – The coefficient of variation (CV) is a tool used to express precision of the determination. Precision is the measure of random variability and is defined as the reproducibility of a laboratory determination when it is run repeatedly under similar conditions. An extremely high CV indicates lack of precision and may occur when outliers are included in the statistical calculations. Excessive result variability is also observed when there is a significant difference between the mean and the median indicating no distinct target value.
Not Graded/Exclusion Requested – The lab is unable to report both patient and proficiency testing results during the testing period. An exclusion may be granted if the laboratory meets the requirements in 493.823 (b).
Referee Grading Used – Grading for a specimen is determined by a select group of participating labs (referee labs). The consensus of the referee group must be 80% or higher.
Fail/No Results Received – The laboratory failed to report results. For an analyte or module in which they were enrolled.
Investigating PT Failures
All laboratories, at one time or another, experience a proficiency testing (PT) failure. It is important to promptly investigate all PT failures and initiate corrective action. A systematic approach to PT failure investigation is required and easily adaptable to all laboratory environments. Labs should document completely each step of investigation taken in their effort to uncover the cause of the failure and develop a corrective action plan to prevent reoccurrence. For more information about investigating PT failures and corrective action, see the current AAFP-PT handbook.
Requesting a Corrected Evaluation
Review your evaluation carefully. If the error is not the fault of the participant, your entry may be re-evaluated if AAFP-PT is notified in writing of the problem within 30 days of the evaluation release date for the event in question. The correction deadline date will be on the front page of the evaluation for each event. Correction requests will not be accepted after this date.
Clerical errors made by the laboratory when reporting data cannot be corrected. Failures that result from neglecting to notify AAFP-PT of instrument or method changes are also unable to be corrected. For failures that are the result of a data processing or grading error, complete a Corrected Evaluation Request form. A copy of the form may be found on your PT Central homepage or in the AAFP-PT handbook.