LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting
Abstract
A benchmark of dated U.S. financial facts shows that large language models often understate their knowledge and that common safeguards cannot reliably certify that historical backtests are free of information leakage.
Financial backtests of large language models (LLMs) are meaningful only if the model did not already know the outcomes it is asked to predict. Practitioners guard against this by trusting a vendor's stated knowledge cutoff, by asking the model what it knows, or by instructing it to answer as of a past date. To verify whether any of these safeguards can be trusted, we introduce FinDendro, a benchmark of 2,226 dated U.S. facts from 1999 to 2026, covering monthly macroeconomic series, month-end stock prices and Federal Reserve decisions. Each fact yields four prompts: a value question, Recall and Predict questions that pair each candidate value with a claim of knowing or guessing without a declared date, and a Predict question with a historical declared date. Controls for persistence and dominant outcomes keep chance accuracy near 0.27. Results show that for most frontier models, self-reports understate knowledge. GPT-5 answers 76% of controlled facts correctly but claims to know 38%, and answers 61% of the facts it says it does not know. Knowledge claims can persist beyond the measured knowledge boundary, by up to seven quarters, and a declared date removes most of the claims of prior knowledge while leaving accuracy unchanged. Backtests inside a model's training window therefore cannot be certified leak-free by self-report, instruction or stated cutoff. These findings motivate behavioral measurement of temporal knowledge boundaries, and FinDendro provides a benchmark and procedure for selecting backtest periods based on demonstrated factual knowledge.
Community
🚀 LLMs Don’t Know What They Know. Can We Trust Their Financial Predictions?
Excited to share our latest research:
📄 LLMs Don’t Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting
Large Language Models (LLMs) are increasingly used for financial forecasting. But when asked to predict the past, are they forecasting—or recalling outcomes they already know? Simply asking the model may not reveal the answer.
Common safeguards seem reassuring: trust the vendor’s knowledge cutoff, ask the model whether it knows the outcome, or instruct it to answer as of a historical date. But do these safeguards actually prevent hindsight leakage—or merely change how the model describes its answer?
We introduce 🌳 FinDendro, a benchmark of 2,226 dated U.S. financial and macroeconomic facts spanning 1999–2026, covering macroeconomic indicators, month-end stock prices, and Federal Reserve decisions.
🔍 Key findings:
- GPT-5 answers 76% of controlled facts correctly but claims to know only 38%.
- It correctly answers 61% of the facts it says it does not know.
- Knowledge claims can extend up to seven quarters beyond the measured knowledge boundary.
- Historical-date instructions remove most claims of prior knowledge while leaving accuracy unchanged.
💡 Reliable backtesting requires measuring what a model knows—not just trusting what it says. FinDendro provides a behavioral procedure for identifying temporal knowledge boundaries and selecting backtest periods based on demonstrated factual knowledge.
🏆 Interactive leaderboard — compare model knowledge and claims, explore temporal boundaries, and replay real benchmark questions.
đź“„ Read the paper
We’d love to hear your thoughts, especially if you work on LLM evaluation, forecasting, or financial AI!
This stopped my scroll: asking a model what it knows is exactly how many people check a cutoff, and you tested it against dated facts. Did the 'answer as of a past date' instruction help at all, or did the models leak anyway?
Get this paper in your agent:
hf papers read 2611.06808 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper