# GEMeX-ThinkVG Using RL to get the bbox right. Canonical: https://astro-blog-bice-chi.vercel.app/blog/GEMeX-ThinkVG Edition: en · Language: en · Revision: 1 Archived: 2025-07-23 Updated: 2026-09-21 The archive date records this article’s first appearance in this repository; its original publication date is unknown. Identifiable source figures and tables were re-extracted from pinned paper versions. Original attachment files remain unavailable; uncertain matches are explicitly marked. Selected source and metric descriptions were corrected. Updated 2026-09-21 ### GEMeX-ThinkVG: Towards Thinking with Visual Grounding in Medical VQA via Reinforcement Learning Published Jun 2025 URL [https://arxiv.org/pdf/2506.17939v1](https://arxiv.org/pdf/2506.17939v1) Data [https://huggingface.co/datasets/BoKelvin/GEMeX-ThinkVG](https://huggingface.co/datasets/BoKelvin/GEMeX-ThinkVG) Tags RRGDatasetX-Ray Successor to [GEMeX](https://astro-blog-bice-chi.vercel.app/blog/GEMeX-ThinkVG#gemex) ## [Data](https://astro-blog-bice-chi.vercel.app/blog/GEMeX-ThinkVG#data) * **206,071 ThinkVG instances for 21,994 images** * Open-ended = 62,712 * Closed-ended = 37,340 * Single-choice = 53,905 * Multi-choice = 52,114 * Generated with the **Deepseek R1 text-only** reasoning model * Generated with the following prompt, optimized for 50 x 4 question types > messages = \[ “role”: “user”, “content”: f“‘Suppose you are viewing a CXR that shows the following: “The hilar contours are normal \[visual location: bilateral hilar structures (\[116, 112, 227, 182\])\] …”. Given the question: “YOUR QUESTION”, provide a detailed thinking process (around 100 words), including a specific visual location (e.g., (region \[x1,y1,x2,y2\])) about how to solve this question with answer “ANSWER”. You must assume that you are viewing the CXR image rather than reading the textual findings, thus, do not output words like “observe the report” or “from report” or “report states” or “given findings” or “provided findings” or “described findings”."" * Verified **text-reasoning** ![ThinkVG generation and text-based verification pipeline](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2506.17939v1-p3-figure-2.webp) Source: [2506.17939v1, Figure 2, PDF p. 3](https://arxiv.org/pdf/2506.17939v1#page=3). Re-extracted from the cited paper for this discussion; the original attachment is unavailable. ## [RL](https://astro-blog-bice-chi.vercel.app/blog/GEMeX-ThinkVG#rl) Semantic Judge Prompt ``` messages = ["role":"user", "content": f"'We would like to request your feedback on the performance of two AI assistants in response to the user question displayed above. For your reference, the visual content in the image is represented with a caption describing the same image. Please rate the accuracy (most important), relevance of their responses, considering both answer and reason (if any). Each assistant receives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance. Please output both the scores and your reason in JSON format { assistant1: score, assistant2: score, reason:your reason }.""" ``` * **GRPO** * **Semantic reward** * Uses **OpenBio-LLM-70B** to grade the generated answer vs. ground truth on a 1-10 scale * If scores differ < 2 → reward = 1, else 0 * Treats free-text and multiple choice uniformly * **Grounding reward** * Checks that the number of predicted boxes matches GT * Computes mean IoU; if > 0.75 → reward = 1 ## [Results](https://astro-blog-bice-chi.vercel.app/blog/GEMeX-ThinkVG#results) ![ThinkVG supervised and reinforcement fine-tuning results](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2506.17939v1-p6-table-4.webp) Source: [2506.17939v1, Table 4, PDF p. 6](https://arxiv.org/pdf/2506.17939v1#page=6). Re-extracted from the cited paper for this discussion; the original attachment is unavailable. > **Figure unavailable:** The imported screenshot could not be recovered or matched confidently. See the paper’s [results section](https://arxiv.org/pdf/2506.17939v1#page=6). * **Training is significantly useful** (RL or SFT) * **RL helps with accuracy and data utilization** * **V-score still relies on more data** * With or without **ThinkVG** doesn’t help (row 2 vs 3) * **RL also helps** (last 2 rows) * **A-score is higher than previous versions almost across the board**; perhaps a model improvement? ### [Robustness](https://astro-blog-bice-chi.vercel.app/blog/GEMeX-ThinkVG#robustness) ![ThinkVG robustness under question perturbations](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2506.17939v1-p6-table-5.webp) Source: [2506.17939v1, Table 5, PDF p. 6](https://arxiv.org/pdf/2506.17939v1#page=6). Re-extracted from the cited paper for this discussion; the original attachment is unavailable. * Two checkpoints with the same amount of SFT data, so the only difference is the thinking trace. * **Closed** = closed-ended question, swap (is vs. isn’t), (normal vs. abnormal) * **Single** = single-choice QA, shuffle choice order ## [Personal Takes](https://astro-blog-bice-chi.vercel.app/blog/GEMeX-ThinkVG#personal-takes) * **Reasoning and verification done in text-only space** * Not sure if region grounding actually helps (e.g., do not predict bbox) * Not sure if the visual description actually derived from the region proposed; they are all just decoded messages from **ALL visual embeddings** * “**Losing textual-visual fidelity**” mentioned in [Reason Like a Radiologist](https://astro-blog-bice-chi.vercel.app/blog/reason-like-radiologist) ## [GEMeX](https://astro-blog-bice-chi.vercel.app/blog/GEMeX-ThinkVG#gemex) ### GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosis Published Mar 2025 URL [https://arxiv.org/pdf/2411.16778](https://arxiv.org/pdf/2411.16778) Data [https://huggingface.co/datasets/BoKelvin/GEMeX-VQA](https://huggingface.co/datasets/BoKelvin/GEMeX-VQA) Tags RRGDatasetX-Ray A dataset for VQA/VLM created by cleaning **Chest ImaGenome** for: * **Normalized locations** (Table 8) * Deleted small parts (carina - right clavicle, left clavicle, aortic arch) * Merged multiple parts → left mid + left lower = left mid-to-lower lung zone * **One-to-one sentence-location match** * **Leveraged GPT-4o to generate large and diverse QA** ![GEMeX grounded-report and question construction pipeline](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2411.16778v2-p3-figure-2.webp) Source: [2411.16778v2, Figure 2, PDF p. 3](https://arxiv.org/pdf/2411.16778v2#page=3). Re-extracted from the cited paper for this discussion; the original attachment is unavailable. * **Total 1.6M QA** > **Figure unavailable:** The imported screenshot could not be recovered or matched confidently. See the paper’s [dataset section](https://arxiv.org/pdf/2411.16778v2#page=5). > **Figure unavailable:** The imported screenshot could not be recovered or matched confidently. See the paper’s [dataset section](https://arxiv.org/pdf/2411.16778v2#page=5). * **Test set is human-verified** * 300 images from the MIMIC-CXR test set * Initially accompanied by 3,291 questions automatically generated by GPT-4o * Radiologists reviewed 10 incorrect answers and adjusted 3 inaccurate location annotations. * Contributed approximately 600 new questions. * **Results on fine-tuning** * **A-score** = accuracy * **V-score** = mIoU of Pred vs. GT (for multiple choice) with Hungarian matching * Seems low, but maybe instance mismatch really hurts the metric? * **AR-score** = using GPT-4o as a judge * Model has high AR but low accuracy = format issue? ![GEMeX model results across question types](https://astro-blog-bice-chi.vercel.app/assets/blog/paper-reading/2411.16778v2-p6-table-4.webp) Source: [2411.16778v2, Table 4, PDF p. 6](https://arxiv.org/pdf/2411.16778v2#page=6). Re-extracted from the cited paper for this discussion; the original attachment is unavailable.