JFinBench

A continuously refreshable Japanese financial benchmark built from the latest editions of six professional certification exams.

Leaderboard

JFinBench overall score versus cost per question Scatter plot of model scores versus cost or output tokens per question.

About JFinBench

JFinBench is a Japanese financial benchmark comprising 321 examples from the 2026 editions of six certification exams. Constructing a benchmark from publicly available exam PDFs is challenging: the documents have heterogeneous layouts, complex tables and figures, and diverse question and answer formats. We address these challenges with a human-in-the-loop agentic pipeline: a vision-language model transcribes PDFs into layout-preserving Markdown, a coding agent writes exam-specific conversion code and review interfaces, and a human verifies every example and provides feedback for the agent to make corrections. The pipeline requires roughly two hours of human verification per exam, making it practical to refresh the benchmark as new exams are released.

Evaluating 12 LLMs, we find that the leading proprietary models score above 93%, while the best open-weight model trails by 12 percentage points, primarily due to gaps in specialized knowledge. Giving models web and terminal access improves every model's performance and narrows this gap, but about 4 percentage points remain, and the ranking of models changes little (Spearman ρ = 0.87). This suggests that knowledge stored in the model itself remains important even when models can look things up.

264 of the 321 examples are hosted on Hugging Face under the publishers' terms. The rest (the Nissho and Zensho Bookkeeping questions and answers, and the answer keys of the Tax Accountant exam, Nissho, and the FP applied section) is not redistributed: recipes in the repository rebuild it, byte for byte, from the PDFs you download from the publishers, so the full benchmark can be evaluated.

Human-in-the-loop agentic construction pipeline: generic VLM transcription of exam PDFs, exam-specific code written by a coding agent, and human verification with feedback to the agent.
Human-in-the-loop agentic construction pipeline. Generic scripts transcribe each exam PDF with a VLM, a coding agent writes the exam-specific code that turns the transcripts into structured examples, and a human verifies every example against the source page and gives feedback to the agent.
ExamExam dateQuestion typesExamples
CPA short-answer examMay 24, 2026MCQ93
Certified Public Tax Accountant examAug 4–6, 2026Answer forms, essays40
FP skill test Level 1 (written)May 24, 2026MCQ, short answer65
Financial counter services skill test Level 1May 2026 (CBT)MCQ, short answer66
Nissho Bookkeeping Level 1Jun 14, 2026Short answer, answer form24
Zensho Bookkeeping Level 1Jun 28, 2026Short answer33

Task examples

Citation

@misc{jfinbench2026,
  title         = {JFinBench: A Continuously Refreshable Japanese Financial Benchmark from the Latest Certification Exams},
  author        = {Issa Sugiura and Taro Makino and Takashi Ishida},
  year          = {2026},
  eprint        = {2611.18130},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2611.18130}
}