Sophie Ostmeier

I am a PhD student in the SPY Lab, supported by the ETH AI Center. I completed a master's degree in CS at Stanford University, where I researched at the AIMI Center. My research was generously supported by the German Research Foundation (DFG) (2023-2025). I completed an M.D. and Dr.med. (Prof. Daniela Pfeiffer) at the Technical University of Munich.

Outside the lab you will find me running, cycling, bikepacking, swimming, playing pickleball, or hiking. I follow a wide range of sports, especially tennis, triathlon, and track and field, but I can appreciate almost any activity with movement.

Contact  /  Google Scholar  /  Github  /  LinkedIn

profile photo

Research

My research focuses on how machine-learning models learn, how they make predictions, and how we measure that, with a current emphasis on safety and security aspects. I am also curious about a path toward reliable intelligent decision-making systems and wonder how they would practice medicine.

Some recent papers (Google Scholar)

Gradient-Stable Attention Heads Signal LLM Correctness
Sophie Ostmeier, Brian Axelrod, Maya Varma, Asad Ali, Yabin Zhang, Magda Paschali, Sanmi Koyejo, Curtis Langlotz, Akshay Chaudhari
EMNLP, Findings track, 2026 arXiv / code

We investigated if a method can estimate answer correctness with access only to the attention-based model, the prompt, and its answer. Our method, HeadEntropy, uses a simple statistic that measures how sensitive each attention pattern would be to gradient updates. With no training, HeadEntropy reaches 0.74 AUROC over 5 models and 5 benchmarks, beats all 6 training-free baselines, and matches a trained hidden-state probe out-of-domain.

LieRE: Lie Rotational Positional Encodings
Sophie Ostmeier, Brian Axelrod, Maya Varma, Michael Moseley, Akshay Chaudhari, Curtis Langlotz
ICML, 2025
arXiv / code

We investigate how to increase a transformer's capacity to learn positional information for n-dimensional inputs. We generalized rotary position encodings (RoPE) from fixed 2D rotation blocks to learned, high-dimensional rotation matrices by using their Lie group structure, and we tested this approach on 2D and 3D vision tasks.

GREEN: Generative Radiology Report Evaluation and Error Notation
Sophie Ostmeier, Justin Xu, Zhihong Chen, Maya Varma, Louis Blankemeier, Christian Bluethgen, Arne Edward Michalson, Michael Moseley, Curtis Langlotz, Akshay Chaudhari, Jean-Benoit Delbrouck
EMNLP, Findings, 2024
project website / data / model / code

We investigate how we can use the skills of a language model to evaluate the difference between a model generated and the human generated radiology report. We report present GREEN, an open-source metric that employs language models to spot and explain clinically significant errors in radiology reports, providing interpretable feedback, and commercial-grade performance.

USE-Evaluator: Performance metrics for medical image segmentation models supervised by uncertain, small or empty reference annotations
Sophie Ostmeier, Brian Axelrod, Fabian Isensee, Jeroen Bertels, Michael Mlynash, Soren Christensen, Maarten G Lansberg, Gregory W Albers, Rajen Sheth, Benjamin FJ Verhaaren, Abdelkader Mahammedi, Li-Jia Li, Greg Zaharchuk, Jeremy J Heit
Medical Image Analysis, 2023
arXiv / code

We investigate evaluation metrics for medical image segmentation that account for uncertainty, small structures, and empty reference annotations. We find that model performance generalizes better to clinical practice when evaluated by the Surface Dice at Tolerance. We further proposed to set a volume threshold for very small volumes or empty reference masks and then use image-level classification metrics such as AUC

Miscellaneous

Academic Service

Co-Organizer of Workshop on Multimodal LLMs in Clinical Practice (MICCAI 2025),
Speaker at Foundation to Multimodal Models for Medical Imaging (FMLLM) Tutorial (MICCAI 2025),
Reviewer (ICLR 2026, ICML 2026, NeurIPS 2026)

Teaching

Lecturer, Stanford BioE 224, "AI in Medical Imaging" (2024, 2025),
Stanford AIMI Center Summer Camp Mentor (2024, 2025),
Stanford Small Science Groups (2023)

Website Template source code.