# The Diagnostic Agent

Introducing a medical diagnosis agent that leverages reasoning to overcome the scalability issues in medical imaging analysis.

Canonical: https://astro-blog-bice-chi.vercel.app/blog/the-diagnostic-agent

Edition: en · Language: en · Revision: 1

Archived: 2025-07-11

Status: unfinished

The archive date records this article’s first appearance in this repository; its original publication date is unknown.

This article is still being developed and may be incomplete.

This is the final piece of the [med](https://astro-blog-bice-chi.vercel.app/tags/med) series.

## [TL;DR](https://astro-blog-bice-chi.vercel.app/blog/the-diagnostic-agent#tldr)

Yes, I am throwing the buzzword “reasoning” at you, and on top of that, I claim I want to build a fancy “medical diagnosis agent” with it. The ideology presented here is largely inspired by three things:

1.  The necessary condition: [Zhang Xiangyu’s talk on multimodality & reasoning](https://astro-blog-bice-chi.vercel.app/blog/image-reasoning), which had me believe reasoning is necessary for image understanding.
2.  The sufficient condition: OpenAI’s O3 results, which convinced me that an LLM has the ability to reason and plan at least OpenAI's O3
3.  The approach: [O3 place detective blog](https://simonwillison.net/2025/Apr/26/o3-photo-locations/), which made me realize this approach will work

The following are just my logical steps and specifics of the claim.

The logic is pretty straightforward:

1.  The current way of “solving” imaging tasks is running into scalability issues in two ways: a. The data scarcity issue: As the performance of the model increases, it’s actually harder to curate more diverse data and to label it. b. No matter how well we develop a model in one domain, it’s pretty much a one-time deal; its transferability to other applications is rather limited.

## [How?](https://astro-blog-bice-chi.vercel.app/blog/the-diagnostic-agent#how)

The approach involves building a multimodal reasoning system that combines specialized vision models with language model planning and reasoning capabilities.

## [What is an agent?](https://astro-blog-bice-chi.vercel.app/blog/the-diagnostic-agent#what-is-an-agent)

The term “agent” has been overused and misused, just like “AGI”. I would rather follow Lilian Weng’s definition[^1]. She framed the agent as an autonomous problem-solver with four major components: memory, planning, tool use, and action.

![agent overview](https://lilianweng.github.io/posts/2023-06-23-agent/agent-overview.png)

*   `Memory` is still an active area of research. Most working solutions choose to use a simpler solution of filtering, summarizing, or concatenating the previous context. I am very much interested in a more generic/elegant solution, but I do not have one to offer here.
    
*   `Planning` seems to be a more settled topic, as reasoning and a long context really enable an LLM to craft a list of executable steps.
    
*   `Tools` enable the agent to call upon external resources to perform actions and were popularized by MCP.
    
*   `Action` used to be troublesome but has gotten much better with reasoning and better models.
    

## [How does it work?](https://astro-blog-bice-chi.vercel.app/blog/the-diagnostic-agent#how-does-it-work)

The diagnostic agent combines vision models and language models in a structured workflow:

1.  **Vision tools** extract specific findings from medical images
2.  **Language model** plans diagnostic reasoning steps
3.  **Memory** maintains context across a multi-step analysis
4.  **Actions** execute diagnostic protocols and follow-ups

## [Paradigm Comparison](https://astro-blog-bice-chi.vercel.app/blog/the-diagnostic-agent#paradigm-comparison)

| Paradigm | Medical Relevance | Data Format | Annotation Challenge | Advantage | Disadvantage |
| --- | --- | --- | --- | --- | --- |
| Vision-Only Model | Medical findings | Raw image | Labeling quality | Works well and runs fast | Hard to transfer and scale |
| MLLM | Medical findings and impressions | Image and report pair | Curating/synthesizing reports | Scales well across tasks and domains | Not very robust; has been reported to have hallucinations and be short-sighted |
| Agent | Medical findings, impressions, and diagnosis | Image, CoT/RL with diagnosis | Cold start with report-synthetic CoT -> self-adapted RL | Scales well across tasks and domains | Complex to implement and validate |

### [Benefits](https://astro-blog-bice-chi.vercel.app/blog/the-diagnostic-agent#benefits)

The agent approach combines the best of both worlds:

*   To show: How and what
*   To tell: Contexts and system prompts allow changes in behavior. [https://www.dbreunig.com/2025/05/07/claude-s-system-prompt-chatbots-are-more-than-just-models.html](https://www.dbreunig.com/2025/05/07/claude-s-system-prompt-chatbots-are-more-than-just-models.html)

### [Watch-Outs](https://astro-blog-bice-chi.vercel.app/blog/the-diagnostic-agent#watch-outs)

We are making the assumption that “knowledge” and “skills” are composable and complement each other. This requires careful validation in clinical settings.

## [Final Thoughts](https://astro-blog-bice-chi.vercel.app/blog/the-diagnostic-agent#final-thoughts)

By this point, I hope you understand what I am trying to build:

*   **Vision models alone are limited**: Data and annotation constraints force narrow specialization
*   **Language models provide reasoning**: They excel at planning, memory, and action coordination
*   **CV models become tools**: Repositioned as specialized instruments within a broader system

The beauty of this approach:

*   **Extensible**: More modalities = more tools
*   **Transferable**: Cross-checks between regions, modalities, and follow-ups
*   **Scalable**: Potential to move from diagnosis to prognosis

To excel at doing ONE task, we may have to do well in ALL tasks.

Sources & further reading

## References

[^1]: [https://lilianweng.github.io/posts/2023-06-23-agent/](https://lilianweng.github.io/posts/2023-06-23-agent/)