# Green Score

An okay metric for radiology report generation.

Canonical: https://astro-blog-bice-chi.vercel.app/blog/green

Edition: en · Language: en · Revision: 1

Archived: 2025-07-22

Updated: 2026-09-21

The archive date records this article’s first appearance in this repository; its original publication date is unknown.

Identifiable source figures and tables were re-extracted from pinned paper versions. Original attachment files remain unavailable; uncertain matches are explicitly marked. Selected source and metric descriptions were corrected.

Updated   2026-09-21

### GREEN: Generative Radiology Report Evaluation and Error Notation

Published May 2024

URL [https://arxiv.org/pdf/2405.03595](https://arxiv.org/pdf/2405.03595)

Tags

RRGMetric

## [Takeaways](https://astro-blog-bice-chi.vercel.app/blog/green#takeaways)

*   solid motivation : Current radiology report generation (RRG) metrics use NLG, which **DOES NOT** reflect report factual accuracy.
*   Okay solution—train a small LLM to judge the generated report by counting inconsistent findings.
*   more investigation required \- The LLM preference is very much opposite to that of radiologists. Why? And maybe that’s a metric by itself?

## [Method](https://astro-blog-bice-chi.vercel.app/blog/green#method)

Prompt

```
Objective:
Evaluate the accuracy of a candidate radiology report in comparison to a reference
radiology report composed by expert radiologists.
Process Overview:
You will be presented with:
1. The criteria for making a judgment.
2. The reference radiology report.
3. The candidate radiology report.
4. The desired format for your assessment.
1. Criteria for Judgment:
For each candidate report, determine:
- The count of clinically significant errors.
- The count of clinically insignificant errors.
Errors can fall into one of the following categories:
a) A false report of a finding in the candidate.
b) Missing a finding that is present in the reference.
c) Misidentification of a finding's anatomical location/position.
d) Misassessment of the severity of a finding.
e) Mentioning a comparison that isn't in the reference.
f) Omitting a comparison detailing a change from a prior study.

Note: Concentrate on the clinical findings rather than the report's writing style.
Evaluate only the findings that appear in both reports.
2. Reference Report:
**Reference Report**
3. Candidate Report:
**Candidate Report**
4. Reporting Your Assessment:
Follow this specific format for your output, even if no errors are found:
[Explanation]:
<Explanation>
[Clinically Significant Errors]:
(a) <Error Type>: <The number of errors>. <Error 1>; <Error 2>; ...; <Error n>
....
(f) <Error Type>: <The number of errors>. <Error 1>; <Error 2>; ...; <Error n>
[Clinically Insignificant Errors]:
(a) <Error Type>: <The number of errors>. <Error 1>; <Error 2>; ...; <Error n>
....
(f) <Error Type>: <The number of errors>. <Error 1>; <Error 2>; ...; <Error n>
[Matched Findings]:
<The number of matched findings>. <Finding 1>; <Finding 2>; ...; <Finding n>
```

*   Distilling GPT-4’s response of analyzing common findings/errors to a smaller LLM

![GREEN synthetic-data and distillation procedure](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2405.03595v2-p3-figure-2.webp)

Source: [2405.03595v2, Figure 2, PDF p. 3](https://arxiv.org/pdf/2405.03595v2#page=3). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

*   The Generative Evaluator Fine-tune LLaMA-2 (7B) and Phi-2 (2.7B) variants (“RadLLaMA-2”, “RadPhi-2”) on **100k** report pairs that GPT-4 has annotated for six error categories:
    
    1.  False finding
    2.  Missed finding
    3.  Wrong location
    4.  Wrong severity
    5.  Hallucinated comparison
    6.  Missed comparison
*   **GREEN Score Formula**: The matched-finding count divided by matched findings plus clinically significant errors (Equation 1). Clinically insignificant errors are described by the evaluator, but are not included in this score.
    
    $$
    Green Score=\frac{\text{Matched Findings}}{\text{Sig.\ Errors}+ \text{Matched Findings}}
    $$
    
    *   Defined as 0 when there are no matched findings
    *   Bounds 0-1
    *   Higher is better

## [Result](https://astro-blog-bice-chi.vercel.app/blog/green#result)

### [The Correlation Analysis](https://astro-blog-bice-chi.vercel.app/blog/green#the-correlation-analysis)

Using the **ReXVal** dataset with assessments from six board-certified radiologists:

*   **200 report pairs** from 50 MIMIC-CXR (Yu et al., 2023b) test cases
*   **0.63 correlation** with the mean expert assessment
*   Individual expert correlations: **0.48-0.64**

![Expert and mean-expert error-count correlation matrix](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2405.03595v2-p17-figure-6.webp)

Source: [2405.03595v2, Figure 6, PDF p. 17](https://arxiv.org/pdf/2405.03595v2#page=17). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

*   **GREEN** local models show similar performance to the **GPT-4** baseline

![GREEN correlation with expert error counts](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2405.03595v2-p8-table-6.webp)

Source: [2405.03595v2, Table 6, PDF p. 8](https://arxiv.org/pdf/2405.03595v2#page=8). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

### [The Preference Analysis](https://astro-blog-bice-chi.vercel.app/blog/green#the-preference-analysis)

This is the unsettling part of the study.

While error counting provides systematic evaluation, true clinical utility depends on alignment with human preferences.

![Accuracy of metric and GPT-4 preferences against experts](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2405.03595v2-p8-table-7.webp)

Source: [2405.03595v2, Table 7, PDF p. 8](https://arxiv.org/pdf/2405.03595v2#page=8). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

1.  GREEN has a higher point estimate for preference agreement than raw error count in Table 7, but the confidence intervals overlap. That table does not establish a significant advantage.
    *   Correlation with error counts and agreement with expert preferences are different validation questions; neither automatically establishes clinical utility.
2.  Direct preference by GPT-4 is REALLY BAD, but WHY? \- If it is what I think it is, using reverse GPT-4 preference actually gives the best metric.
    *   This is worth investigating, especially because GPT-4 supplies the evaluation-training targets. Reversing preferences after seeing this table would be a post-hoc hypothesis; it needs independent validation.