Green Score

Takeaways

• solid motivation : Current radiology report generation (RRG) metrics use NLG, which DOES NOT reflect report factual accuracy.
• Okay solution—train a small LLM to judge the generated report by counting inconsistent findings.
• more investigation required - The LLM preference is very much opposite to that of radiologists. Why? And maybe that’s a metric by itself?

Method

Objective:
Evaluate the accuracy of a candidate radiology report in comparison to a reference
radiology report composed by expert radiologists.
Process Overview:
You will be presented with:
1. The criteria for making a judgment.
2. The reference radiology report.
3. The candidate radiology report.
4. The desired format for your assessment.
1. Criteria for Judgment:
For each candidate report, determine:
- The count of clinically significant errors.
- The count of clinically insignificant errors.
Errors can fall into one of the following categories:
a) A false report of a finding in the candidate.
b) Missing a finding that is present in the reference.
c) Misidentification of a finding's anatomical location/position.
d) Misassessment of the severity of a finding.
e) Mentioning a comparison that isn't in the reference.
f) Omitting a comparison detailing a change from a prior study.

Note: Concentrate on the clinical findings rather than the report's writing style.
Evaluate only the findings that appear in both reports.
2. Reference Report:
**Reference Report**
3. Candidate Report:
**Candidate Report**
4. Reporting Your Assessment:
Follow this specific format for your output, even if no errors are found:
[Explanation]:
<Explanation>
[Clinically Significant Errors]:
(a) <Error Type>: <The number of errors>. <Error 1>; <Error 2>; ...; <Error n>
....
(f) <Error Type>: <The number of errors>. <Error 1>; <Error 2>; ...; <Error n>
[Clinically Insignificant Errors]:
(a) <Error Type>: <The number of errors>. <Error 1>; <Error 2>; ...; <Error n>
....
(f) <Error Type>: <The number of errors>. <Error 1>; <Error 2>; ...; <Error n>
[Matched Findings]:
<The number of matched findings>. <Finding 1>; <Finding 2>; ...; <Finding n>

• Distilling GPT-4’s response of analyzing common findings/errors to a smaller LLM

• The Generative Evaluator Fine-tune LLaMA-2 (7B) and Phi-2 (2.7B) variants (“RadLLaMA-2”, “RadPhi-2”) on 100k report pairs that GPT-4 has annotated for six error categories:

  1. False finding
  2. Missed finding
  3. Wrong location
  4. Wrong severity
  5. Hallucinated comparison
  6. Missed comparison

• GREEN Score Formula: The matched-finding count divided by matched findings plus clinically significant errors (Equation 1). Clinically insignificant errors are described by the evaluator, but are not included in this score.

  $$Green Score=\frac{\text{Matched Findings}}{\text{Sig.\ Errors}+ \text{Matched Findings}}$$
  • Defined as 0 when there are no matched findings
  • Bounds 0-1
  • Higher is better

Result

The Correlation Analysis

Using the ReXVal dataset with assessments from six board-certified radiologists:

• 200 report pairs from 50 MIMIC-CXR (Yu et al., 2023b) test cases
• 0.63 correlation with the mean expert assessment
• Individual expert correlations: 0.48-0.64

• GREEN local models show similar performance to the GPT-4 baseline

The Preference Analysis

This is the unsettling part of the study.

While error counting provides systematic evaluation, true clinical utility depends on alignment with human preferences.

1. GREEN has a higher point estimate for preference agreement than raw error count in Table 7, but the confidence intervals overlap. That table does not establish a significant advantage.
   • Correlation with error counts and agreement with expert preferences are different validation questions; neither automatically establishes clinical utility.
2. Direct preference by GPT-4 is REALLY BAD, but WHY? - If it is what I think it is, using reverse GPT-4 preference actually gives the best metric.
   • This is worth investigating, especially because GPT-4 supplies the evaluation-training targets. Reversing preferences after seeing this table would be a post-hoc hypothesis; it needs independent validation.