ScarfBench tests agents on Java migrations

IBM researchers introduced ScarfBench to evaluate AI agents migrating enterprise Java applications between frameworks. These tasks involve connected decisions about dependencies, configuration, and application behavior, putting project-level modernization at the center of the assessment.

Context

This resembles engineering work more closely than a short function with a self-contained specification. Success should mean preserving the required behavior after changing the application’s foundations. File counts and syntax checks alone cannot establish that. A migration also needs to survive build, runtime, and behavioral validation.