Reverse-Engineering Visualizations: Recovering Visual Encodings from Chart Images
Jorge Poco, Jeffrey Heer
Teaching machines to read charts, from recovering encodings to auditing claims about them
Chart Understanding studies how machines can read visualizations the way people do. The line began by reverse-engineering chart images: recovering visual encodings from bitmap charts, extracting color mappings and scalar fields from map charts, and linking the text of a document to the charts it refers to (ChartText). It now turns to the claims people make about charts. Multimodal models can verify whether a statement is supported by a chart, but they collapse interpretation, evidence, and verdict into a single answer. Our current work makes that process auditable: systems that expose the interpretive choices behind a verdict, trace the evidence back to marks and pixels, and present verification as a sequence of data-preserving transformations a reader can inspect, contest, and replay. Together with AI for Mathematics, this line is part of our broader interest in auditable AI.
Jorge Poco, Jeffrey Heer
Angela Mayhua, Erick Gomez Nieto, Jeffrey Heer, Jorge Poco
Jorge Poco, Angela Mayhua, Jeffrey Heer