Sophie Ostmeier

I'm a physician turned AI researcher, now a CS PhD student in the SPY Lab at ETH Zurich. I work on telling when large language models are wrong, so people can trust AI in high-stakes fields like medicine. Before ETH, I did my master's at Stanford and was a radiology resident. Outside work, I run marathons, go bikepacking and ski.

Some recent papers (Google Scholar)

We investigate if a method can estimate answer correctness with access only to the attention-based model, the prompt, and its answer. Our method, HeadEntropy, uses a simple statistic that measures how sensitive each attention pattern would be to gradient updates. With no training, HeadEntropy reaches 0.74 AUROC over 5 models and 5 benchmarks, beats all 6 training-free baselines, and matches a trained hidden-state probe out-of-domain.

We investigate how to increase a transformer's capacity to learn positional information for n-dimensional inputs. We generalized rotary position encodings (RoPE) from fixed 2D rotation blocks to learned, high-dimensional rotation matrices by using their Lie group structure, and we tested this approach on 2D and 3D vision tasks.

We investigate how we can use the skills of a language model to evaluate the difference between a model generated and the human generated radiology report. We present GREEN, an open-source metric that employs language models to spot and explain clinically significant errors in radiology reports, providing interpretable feedback, and commercial-grade performance.

We investigate evaluation metrics for medical image segmentation that account for uncertainty, small structures, and empty reference annotations. We find that model performance generalizes better to clinical practice when evaluated by the Surface Dice at Tolerance. We further proposed to set a volume threshold for very small volumes or empty reference masks and then use image-level classification metrics such as AUC.

Miscellaneous

Website Template source code.