Training AIs to Solve CAPTCHAs

Interesting research: Abstract: Vision-language models (VLMs) can solve visual CAPTCHAs without task-specific training, but the agents built on them approach every challenge from scratch.

For such an agent, the hundredth instance of a familiar puzzle costs as much time and compute as the first.

Specialized detectors invert the trade-off, answering in milliseconds but only for categories they were trained on.

Neither improves with exposure.

We study what changes when a solver improves with use.

Our system pairs a fine-tuned YOLOv8 detector with an open-weight VLM behind a confidence-based router, and runs entirely from screenshots and operating-system input events, with no browser automation or DOM access.

It reaches 85.4% overall and 84.2% macro accuracy across 16 classes, exceeding either component alone.

Every answer VLM produces also serves as a training label, so the detector absorbs categories it was never trained for, typically after one or two encounters and without human annotation.

The same loop also repairs it.

A CAPTCHA operator can perturb images against the publicly released detector and drive its accuracy to 0%, but the perturbations leave VLM untouched, and its labels let the detector recover.

Continue reading at the original source →