# CT-to-Report Does NOT Work, Yet

It could, just not with current methodologies.

Canonical: https://astro-blog-bice-chi.vercel.app/blog/ct-2-report

Edition: en · Language: en · Revision: 1

Archived: 2025-07-16

Updated: 2026-09-21

The archive date records this article’s first appearance in this repository; its original publication date is unknown.

Identifiable source figures and tables were re-extracted from pinned paper versions. Original attachment files remain unavailable; uncertain matches are explicitly marked. Selected source and metric descriptions were corrected.

Updated   2026-09-21

Observations as of July 2025; things MAY CHANGE

## [TL;DR](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#tldr)

This is a collection of studies on the task of RRG (“CT DICOMs in, Report out”).

*   We finally see a good-sized (~50k) dataset with reports , which further attracts [more “annotations”](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#radgenome-chestct)
*   None of the direct report generation methods are really working
*   Even worse, The numbers DON’T MATCH across papers (CT2Rep)
    *   which shows a lack of a consistent benchmark
    *   perhaps the new challenge will redeem the problem [^1]

## [CT-CLIP](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#ct-clip)

### Developing Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography

Published Mar 2024

URL [https://arxiv.org/pdf/2403.17834](https://arxiv.org/pdf/2403.17834)

Tags

CTDatasetCLIP

> NOTE: The paper and dataset were first published in 03/24, but the figure is drawn from the v3 version in 04/25.

### [Takeaway](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#takeaway)

*   good amount CT data with labels [^2]
*   native, whole-CT abnormalities classification DOES NOT work

### [Dataset (CT-RATE)](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#dataset-ct-rate)

*   50,188 non-contrast 3D chest CT volumes (`series_id`) from 25,692 distinct CT experiments (`study_id`) conducted on 21,304 unique patients (`patient_id`).
*   with reports and parsed into 18 distinct types of abnormalities
*   with extended metadata

Abnormality Distribution

![CT-RATE population and abnormality distribution](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2403.17834v3-p11-figure-2.webp)

Source: [2403.17834v3, Figure 2, PDF p. 11](https://arxiv.org/pdf/2403.17834v3#page=11). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

1.  Medical material
2.  Arterial wall calcification
3.  Cardiomegaly
4.  Pericardial effusion
5.  Coronary artery wall calcification
6.  Hiatal hernia
7.  Lymphadenopathy
8.  Emphysema
9.  Atelectasis
10.  Lung nodule
11.  Lung opacity
12.  Pulmonary fibrotic sequela
13.  Pleural effusion
14.  Mosaic attenuation pattern
15.  Peribronchial thickening
16.  Consolidation
17.  Bronchiectasis
18.  Interlobular septal thickening

### [Method and Results](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#method-and-results)

*   Extract labels from reports with fine-tuned RadBERT
*   CT-ViT with
    *   patch size of 20 × 20 × 10
    *   normalized spacing of 0.75mm x 0.75mm x 1.5mm
    *   normalized resolution of 480 × 480 × 240
    *   ending up with 24^3 = **13,824 patches** compressed to a 512-channel embedding before the text embedding
*   several strategies, such as supervised, linear probing, CT-CLIP, etc.
*   Performance is better than random but **NOT** usable; even worse, it **GENERALIZES POORLY**

![CT-CLIP classification and external-validation results](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2403.17834v3-p13-figure-4.webp)

Source: [2403.17834v3, Figure 4, PDF p. 13](https://arxiv.org/pdf/2403.17834v3#page=13). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

## [RadGenome-ChestCT](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#radgenome-chestct)

### RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis

Published Apr 2024

URL [https://arxiv.org/pdf/2404.16754](https://arxiv.org/pdf/2404.16754)

Tags

CTAnnotationVQA

### [Takeaway](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#takeaway-1)

More annotations for CT-RATE \- segmentation, hierarchical structure identities, and VQA[^3]

### [Method](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#method)

New annotations added for the CT-RATE dataset

*   universal segmenter SAT[^4]: whole body segmentation of 197 categories
*   LLM and NER parsing: GPT-4 annotated hierarchical structure: Breaks each report into an _anatomically-hierarchical_ tree and align every sentence to the matching mask region.
*   Rule-based template: Generates grounded VQA pairs that ask about abnormality presence, location, size, etc.

![RadGenome-Chest CT annotation pipeline](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2404.16754v1-p2-figure-1.webp)

Source: [2404.16754v1, Figure 1, PDF p. 2](https://arxiv.org/pdf/2404.16754v1#page=2). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

## [CT2Rep](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#ct2rep)

### CT2Rep: Automated Radiology Report Generation for 3D Medical Imaging

Published Mar 2024

URL [https://arxiv.org/pdf/2403.06801](https://arxiv.org/pdf/2403.06801)

Tags

CTAnnotationRRG

### [Takeaway](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#takeaway-2)

*   A good first-effort attempt; has been used as a baseline for comparison for CT RRG

### [Method](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#method-1)

*   First direct CT → image encoder → text decoder to generate a report

![CT2Rep architecture](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2403.06801v1-p3-figure-1.webp)

Source: [2403.06801v1, Figure 1, PDF p. 3](https://arxiv.org/pdf/2403.06801v1#page=3). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

*   similar to CT-CLIP preprocessing, but patch size of 12x24x24, ending up with 20^3 = **8,000** vision embeddings
*   Added a follow-up setting, CT2RepLong, but the results did not get better…

### [Result](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#result)

![CT2Rep and longitudinal report-generation results](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2403.06801v1-p6-table-1.webp)

Source: [2403.06801v1, Table 1, PDF p. 6](https://arxiv.org/pdf/2403.06801v1#page=6). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

*   NOTE: RepLong reports lower aggregate precision and recall but a higher aggregate F1. Separate macro-averaging can produce this pattern, so it is not by itself proof of an error. The paper’s averaging convention needs to be checked before comparing these columns.

Detailed Result

*   The per-label table also warrants clarification: for medical material, Table 2 reports precision 0.736, recall 0.752, and F1 0.308. If these come from one confusion matrix, F1 should be approximately 0.744. Separate averaging could change that relationship; the table alone does not establish how these values were aggregated.

![CT2Rep per-abnormality clinical efficacy](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2403.06801v1-p11-table-2.webp)

Source: [2403.06801v1, Table 2, PDF p. 11](https://arxiv.org/pdf/2403.06801v1#page=11). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

![CT2RepLong per-abnormality clinical efficacy](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2403.06801v1-p12-table-3.webp)

Source: [2403.06801v1, Table 3, PDF p. 12](https://arxiv.org/pdf/2403.06801v1#page=12). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

## [CT-AGRG](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#ct-agrg)

### CT-AGRG: AUTOMATED ABNORMALITY-GUIDED REPORT GENERATION FROM 3D

Published Aug 2024

URL [https://arxiv.org/pdf/2408.11965](https://arxiv.org/pdf/2408.11965)

Tags

CTRRG

### [Takeaway](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#takeaway-3)

*   A native baseline to start with
*   CT-Net with CNN architecture > CT-ViT with Transformer architecture
    *   Aligns with my expectation: ViT is limited due to the current setting (full-size CT → sparse supervision)

### [Method](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#method-2)

![CT-AGRG classification and generation pipeline](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2408.11965v3-p2-figure-1.webp)

Source: [2408.11965v3, Figure 1, PDF p. 2](https://arxiv.org/pdf/2408.11965v3#page=2). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

*   Use a visual encoder to train on supervised classification first, then integrate it with a small GPT-2 for report generation
    *   multi-label → multi-task embedding → report sentence

### [result](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#result-1)

![CT-AGRG main results and ablations](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2408.11965v3-p4-table-1.webp)

Source: [2408.11965v3, Table 1, PDF p. 4](https://arxiv.org/pdf/2408.11965v3#page=4). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

Detailed Result

The results of `CT2Rep` are COMPLETELY DIFFERENT from what is reported in the original paper, though the frequencies are the SAME suggesting the dataset is the same

> Version note: The detailed table below is Table 2 in arXiv:2408.11965v3. The comparison here is pinned to that version; later revisions may differ.

![CT-AGRG and CT2Rep per-abnormality results in version 3](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2408.11965v3-p10-table-2.webp)

Source: [2408.11965v3, Table 2, PDF p. 10](https://arxiv.org/pdf/2408.11965v3#page=10). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

## [CT-Agent](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#ct-agent)

### CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answering

Published May 2025

URL [https://arxiv.org/pdf/2505.16229](https://arxiv.org/pdf/2505.16229)

Tags

CTRRGAgent

### [Takeaway](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#takeaway-4)

*   First attempt? To integrate a whole CT volume into an LLM
    
*   Interesting, regionally prompted analysis reveals that the MLLM CANNOT identify some organs well
    

### [Method](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#method-3)

![CT-Agent anatomy-aware reasoning tool architecture](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2505.16229v1-p7-figure-2.webp)

Source: [2505.16229v1, Figure 2, PDF p. 7](https://arxiv.org/pdf/2505.16229v1#page=7). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

*   **frozen 2D CLIP (`ViT-B/16`)**: as vision encoder, each image is 256 tokens = a total of 240 (`slice`) x 256 (`num_token`) x 1024 (`emb_dim`) tensor to start with
*   **Global Token Aggregation (GTA)**: compresses the volume into 256 x 1024 global embedding
*   **Local Token Selection(LTS)**: selects the top K slices, merged by similarity M slices in K+M x 1024 local embedding
*   **Projector**: matches vision tokens to LLM (LLaVAMed-v1.5) embedding dim 1024 → 4096
*   **LORA**: Is trained on **EACH** anatomical region to answer VQA
*   **Frozen DeepSeek V3**: As a planner to orchestrate everything

### [Results](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#results)

*   Shows improvement? Still pretty bad; the `CT2Rep` baseline is again SIGNIFICANTLY WORSE than what’s reported in the original paper

![CT-Agent radiology report-generation results](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2505.16229v1-p15-table-1.webp)

Source: [2505.16229v1, Table 1, PDF p. 15](https://arxiv.org/pdf/2505.16229v1#page=15). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

*   The regionally prompted analysis is by far the most interesting part, which shows that the agent either doesn’t know or can’t identify some organs well…
    *   Surprisingly, bone is hard?

![CT-Agent performance by anatomical region](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2505.16229v1-p15-table-2.webp)

Source: [2505.16229v1, Table 2, PDF p. 15](https://arxiv.org/pdf/2505.16229v1#page=15). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

## [MedRegion-CT](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#medregion-ct)

### MedRegion-CT: Region-Focused Multimodal LLM for Comprehensive 3D CT

Published Jun 2025

URL [https://arxiv.org/pdf/2506.23102](https://arxiv.org/pdf/2506.23102)

Tags

CTLLMVQA

### [Takeaway](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#takeaway-5)

An interesting way to “manage” contexts for an MLLM, though the benefits are limited

### [Method](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#method-4)

Prompt

```
Input:
The global visual information is provided here: <image>.
The following regions of interest have been identified: lung<region1>,
airway <region2>, .... abdomen <region6>
Additional clinical attribute information is provided as follows:
Organ Volumes:
Lung:

- Right Upper Lobe:{right_upper_lobe_volume}ml
- Right Middle Lobe:{right_middle_lobe_volume}m
  Right Lower Lobe:{right_lower_lobe_volume}ml
- Left Upper Lobe:{left_upper_lobe_volume}ml
- Left Lower Lobe:{left_lower_lobe_volume}m
  Heart:
- Left Atrium:{left_atrium_volume}ml
- Right Atrium:{right_atrium_volume}ml
  Left Ventricle:{left_ventricle_volume}ml
- Right Ventricle:{right_ventricle_volume}ml
  Liver:{liver_volume}ml
  Kidney:
- Left:{left_kidney_volume}ml
  Right:{right_kidney_volume}ml
  Lesion Details:
  Nodule:
- count:{nodule_count}
  diameter (mm):{nodule_diameter}
  location:{nodule_location}
  Cyst:
  count:{cyst_count}
  diameter (mm):{cyst_diameter}
- location:{cyst_location}
  Effusion:
  count:{effusion_count}
  diameter (mm):{effusion_diameter}
- location:{effusion_location}
  Describe this medical scan with findings.

Output:
[Lung]:{lung_findings]
[Airways]: {airways_findings]
[Mediastinum]: {mediastinum_findings}
[Heart]:{heart_findings]
[Osseous]:{osseous_findings]
[Abdomen]:{abdomen_findings}
```

![MedRegion-CT architecture with region tokens and attributes](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2506.23102v1-p4-figure-2.webp)

Source: [2506.23102v1, Figure 2, PDF p. 4](https://arxiv.org/pdf/2506.23102v1#page=4). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

1.  Leverage a pre-trained **2D** Vision encoder (RAD-DINO) and heuristics to train global slice and region slice tokens
2.  Leverage a 3D segmentation model (SAT) and add masks of 6 major organs and attributes (size of organ and lesions)

full setup:

$$
R = LLM(T_{vision}, T_{seg}, T_{attr}, I)
$$

### [Result](https://astro-blog-bice-chi.vercel.app/blog/ct-2-report#result-2)

![MedRegion-CT report-generation metrics and ablations](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2506.23102v1-p6-table-1.webp)

Source: [2506.23102v1, Table 1, PDF p. 6](https://arxiv.org/pdf/2506.23102v1#page=6). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.

*   Minor improvements over NLG metrics
*   Greater degradation in the [Green score](https://astro-blog-bice-chi.vercel.app/blog/green) may suggest a loss of medical findings…

Sources & further reading

## References

[^1]: VLM3D Challenge – Task 2: Multi-Abnormality Classification: [https://abnclass.vlm3dchallenge.com/evaluation/test/leaderboard/](https://abnclass.vlm3dchallenge.com/evaluation/test/leaderboard/)

[^2]: [https://huggingface.co/datasets/ibrahimhamamci/CT-RATE/](https://huggingface.co/datasets/ibrahimhamamci/CT-RATE/)

[^3]: [https://huggingface.co/datasets/RadGenome/RadGenome-ChestCT](https://huggingface.co/datasets/RadGenome/RadGenome-ChestCT)

[^4]: Large-Vocabulary Segmentation for Medical Images with Text Prompts: [https://arxiv.org/pdf/2312.17183](https://arxiv.org/pdf/2312.17183)