What does raw OHLCV cost an LLM in tokens?

An asserted number is not a measured one.

The original pitch for patternfetch is a compression argument: a ticker and a timeframe become a token-compact read, so a model never has to swallow a wall of candles. The site states it as ~500 tokens of market state instead of 400,000 of raw candles, roughly 800× fewer, at ≈$6 a call saved. Good marketing. But nowhere is it measured, and an unmeasured number is a liability the moment someone checks it.

This page checks it. We fetch both sides for real — the raw OHLCV a developer would pull from the Yahoo chart endpoint, and the finished brief from POST /v1/demo — and count the tokens of each with gpt-tokenizer (cl100k_base, the GPT-3.5/4 tokenizer), across eight tickers and timeframes. Where the claim holds, we show by how much. Where it is generous, we say so.

Two things make this worth publishing rather than burying. First, a page that measures a token cost is the kind of citable data asset that earns real references — which is the one growth lever that works beyond one person's own reach. Second, it carries the product's actual advantage without a sales voice: the number, checkable, is the argument.

The method, so you can repeat it.

For eight ticker/timeframe pairs — crypto and US equities, across daily and hourly — we did the same thing:

• The brief. POST /v1/demo {ticker, timeframe} (keyless, rate-limited). The demo always returns 120 candles of analysis, so 120 bars is the window every comparison is pinned to.
• The raw candles. The Yahoo chart endpoint for the same timeframe, sliced to the last 120 bars — the exact window the brief covers. We serialize them two ways a careful developer actually would: minimal CSV rows (time,open,high,low,close,volume) and a JSON array of rounded objects. Both are generous to the raw side; the untrimmed provider response is larger still.
• The count. Every string is tokenized with cl100k_base. We re-ran the headline figures through o200k_base (the GPT-4o tokenizer) as a cross-check — on this mostly-numeric content the two agree exactly.
• The rate. To project beyond one window we measured a per-candle token rate from large real pulls (up to ~3,500 candles): ~24.9 tokens/candle (CSV), ~36.9 (JSON).

It is not only cheaper. Long context also reads worse.

Cost is the easy argument. The harder, more honest one is quality: a 400,000-token context does not just cost more, it tends to be reasoned over worse. Several strands of published work point the same way — and because it matters, here is what is well-supported and what is not.

What is well-supported

• Position matters — "lost in the middle." Liu et al. (TACL 2024) find a U-shaped curve: models use information best when it sits at the start or end of the context and measurably worse when it is buried in the middle — even in models built for long contexts. arXiv:2307.03172
• Advertised context ≠ usable context. RULER (Hsieh et al., NVIDIA, COLM 2024) tests 17 long-context models and finds nearly all degrade sharply as length grows; only about half of those claiming ≥32K hold a real task at 32K. arXiv:2404.06654
• Removing keyword overlap breaks it faster. NoLiMa (Modarressi et al., ICML 2025) shows that when the question can't lexically match the answer, GPT-4o falls from a 99.3% short-context baseline to 69.7% at 32K, and 11 of the tested models drop below half their own baseline. Raw candles do not lexically match "is this overbought?". arXiv:2502.05167
• Irrelevant tokens hurt the relevant sub-task. Shi et al. (ICML 2023) add irrelevant sentences to grade-school math and watch accuracy fall even with explicit "ignore it" prompting. A wall of candles the model does not need is exactly this kind of distractor. arXiv:2302.00093
• Arithmetic over long numeric sequences degrades. "Faith and Fate" (Dziri et al., NeurIPS 2023) shows transformers reduce multi-step arithmetic to pattern-matching, and accuracy collapses as operand size grows. RSI, EMA and MACD are running sums and windowed averages over long number arrays — the operation class this describes. arXiv:2305.18654

Where the evidence is thinner — say so

• The broadest sweep across 2025 frontier models — "Context Rot" (Chroma, 2025), 18 models, degradation even on trivial tasks with length — is an industry technical report, not peer-reviewed, from a vendor that sells retrieval. We cite it as corroboration, labeled as such, not as proof. trychroma.com/research/context-rot
• Almost all of this literature uses synthetic retrieval tasks, not financial reasoning. The application to OHLCV is analogical, not measured.
• Newer frontier models degrade less than the 2023 models in the early papers. The effect is a persistent tendency, not a hard cliff — and it is shrinking.
• The specific claim "a model miscomputes RSI over raw candles" is a mechanistic inference from the two threads above. No published study measures indicator errors over raw candle tables directly. We do not put a number on it.

Net: a compact brief plausibly sidesteps a documented failure mode, on top of the measured cost saving. We present it as suggestive, not settled — which is the honest register for a claim the literature supports by analogy rather than by direct test.

Where the brief is the wrong tool, and where our numbers bend.

A study that only flatters the product is an advertisement. Here is what the measurement does not say.

• The brief is an interpretation. It computes regime, levels and indicators and throws the raw ticks away. A model that wanted to do something we didn't anticipate — a bespoke indicator, a pattern we don't detect, its own arithmetic — cannot, because the numbers are gone. Compression is lossy by design. The raw ticks still ship in codec.rows and via POST /v1/candles if you want them; that is a deliberate escape hatch, not a contradiction.
• The full demo payload is larger than a single raw window. The keyless demo averages ~12,300 tokens because it ships the compact candle codec plus base-rate evidence for every detected pattern — the transparency layer from our base-rate study. The clean "~300 tokens" is the state, the part a model acts on; the full payload is the state plus its receipts. If token budget is the whole game and you don't need the evidence, the state read is the number that matters — but we won't pretend the wire payload is 300 tokens.
• The counts are from a GPT-family tokenizer. cl100k_base and o200k_base agree here, but Claude and Gemini will report somewhat different absolute numbers. The ratio is robust; the absolute tokens are a baseline, not a universal constant.
• The raw side is a fair, not a worst, case. We serialized clean CSV and rounded JSON — what a careful developer builds. A naive pass-through of the untrimmed provider response is heavier still, so if anything we are generous to raw.
• The quality argument is analogical. As §05 states, the context-degradation literature is mostly synthetic retrieval, the OHLCV link is inferred, and modern models degrade less than older ones. Believe the cost numbers as measured; treat the quality benefit as plausible and directional.

So who is it for? An agent that repeatedly asks "where is this market and what's printing" across many tickers and timeframes — polling, screening, multi-asset watch — is exactly where a flat ~300-token read pays off against a history that would otherwise grow without bound. An agent that needs the raw series for a custom computation, or asks once about one small window, should just send the candles: at that scale the gap is ~10×, real but not decisive. The brief earns its keep by volume and history, not by magic.

Two requests and any tokenizer.

Nothing here is behind a key. Fetch both sides and count them yourself.

# 1) the brief — keyless, returns 120 candles of analysis
curl -X POST https://patternfetch.com/v1/demo \
  -H "Content-Type: application/json" \
  -d '{"ticker":"AAPL","timeframe":"1d"}'

# 2) the raw candles — the public Yahoo chart endpoint, same window
curl "https://query1.finance.yahoo.com/v8/finance/chart/AAPL?range=1y&interval=1d"

# 3) count both with a real tokenizer (Node)
npm i gpt-tokenizer
node -e 'import("gpt-tokenizer").then(t=>{
  const csv = /* last 120 rows: time,open,high,low,close,volume */ "";
  const read = /* brief.analysis.nl + regime + levels + indicators */ "";
  console.log("raw csv:", t.encode(csv).length, " read:", t.encode(read).length);
})'

The brief object is documented in the API reference; the market-state read is analysis.nl, analysis.regime, analysis.levels, analysis.indicators plus the pattern names and their one-line base-rate signal — everything except the per-pattern evidence tables and the candle codec.