diff --git a/docs/architecture/adr/0006-prompt-cache-ttl-program.md b/docs/architecture/adr/0006-prompt-cache-ttl-program.md new file mode 100644 index 0000000000..d419515a5b --- /dev/null +++ b/docs/architecture/adr/0006-prompt-cache-ttl-program.md @@ -0,0 +1,75 @@ +# ADR-0006: Prompt-cache TTL program — catalog-embedded per-TTL pricing, opt-in knob, fail-closed capability gates + +> RECONSTRUCTED 2026-10-31: the original file (written 261031 ~01:58) was lost in a docs-folder deletion incident (~01:40-02:10 window). This copy is rebuilt verbatim from the session transcript. Report folders are session artifacts; a live backup also exists at C:/Users/k1yt/ZooCode_docs_rebuild/. + +## Status + +Proposed (2026-10-31). Deciders: @myk1yt (program owner), upstream maintainers for merge. Design produced by a 3-seat double-blind architect swarm; per-seat plans in `docs/261031_0001_session_caching-diet-wave/architect-report-ttl-adr-{mlarge,glm,deep}.md`, reconciliation in `ttl-adr-pick.md` (session artifacts, not committed). Supersedes nothing. + +## Context + +Zoo-Code emits only bare `{ type: "ephemeral" }` cache markers — no TTL anywhere (`src/api/providers/anthropic.ts:67`, four `src/api/transform/caching/*.ts` transforms, `openai.ts:121,150`, `minimax.ts:85`, Bedrock `{ cachePoint: { type: "default" } }` at `bedrock.ts:1040` + `base-strategy.ts:53-55`). Per-TTL usage splits (`usage.cache_creation.{ephemeral_5m_input_tokens, ephemeral_1h_input_tokens}`; OpenAI `cache_write_tokens_details.{ttl_30m,ttl_12h}_tokens`) are typed in the pinned SDKs (`messages.d.ts:166-174`; AWS `CacheTTL` enum) but read nowhere — grep over `src/` returns zero hits. `src/shared/cost.ts:50` prices writes at one rate. + +Economics (verified by the 261030 caching validation swarm): Anthropic 5m write = 1.25x base input, 1h write = 2.0x. Shipping a TTL knob without per-TTL write splits under-reports every 1h write by **37.5%** (50% for OpenAI's 2.5x 12h class). Break-even for 1h is a single avoided re-write per hour (premium 0.75xP vs saving 1.15xP per expiry event; ~0.65 events/hour). Bedrock hard-fails (`ValidationException`) on a TTL a model does not support. + +## Decision Drivers + +- Billing correctness (a knob that misprices is worse than no knob; the repo already under-reports DeepSeek-style shape bugs once — PR #2252). +- Single source of truth for price data (catalog), not derived multipliers. +- Zero behavior change by default (byte-identical requests unless the user opts in). +- Hard provider failures are unacceptable (Bedrock gate must fail closed). +- Mutation-diff budget (`MAX_CHANGED_LINES=500` per package) and per-PR reviewability. + +## Decision + +Adopt **Option A — catalog-embedded per-TTL write prices + a provider-settings opt-in knob** — executed in a staged order so the split-accounting half lands **before** any request can emit 1h: + +1. **PR-C5a (accounting only, zero request-shape change):** `ModelInfo` gains optional `cacheWritesPrice1h`/`cacheWritesPrice12h` (populated from a single `deriveCacheWritesPrice1h(inputPrice)` ratio helper, never retyped per entry); usage chunks gain **count-split fields** (`cacheWriteTokens1h`, `cacheWriteTokens12h`) accumulated from the vendor's reported split, with `writes5m = total - 1h - 12h` clamped at 0; `cost.ts` gains optional trailing split params (backward-compatible; existing 55 call sites untouched). +2. **PR-C5b (mechanism):** `promptCacheTtl?: "5m" | "1h"` added to `baseProviderSettingsShape` (provider-settings, **not** global-settings — `ApiHandlerOptions = Omit`, `src/shared/api.ts:14`, means a global key is invisible to handlers). Emission gated by a catalog capability array `promptCacheTtlSupport?: ("5m"|"1h")[]`; unset knob emits **no `ttl` key at all** (not `"5m"`). +3. **PR-C6 (settings/UI):** the knob ships last and rides the provider-profile round-trip (`apiConfiguration`), satisfying the Persisted Setting Checklist in AGENTS.md with `awsUsePromptCache`/`anthropicBeta1MContext` as precedents; UI disables 1h for models without capability. +4. **PR-C7 (Bedrock per-model gate):** capability comes from the catalog flag on resolved base-model ids (never ARN/custom-id string inference); both cache-point emitters (`base-strategy.ts:54`, `bedrock.ts:1040`) are patched together; requests carry one uniform TTL per request, making AWS's longer-before-shorter ordering hazard structurally unreachable. + +Invariants binding all PRs: **I1** price from reported counts, never from the requested TTL, never summing two shapes of the same quantity; **I2** absent capability metadata means 5m-only (fail closed); **I3** default request bytes unchanged; **I4** one uniform TTL per request. + +## Alternatives + +- **Option B — multiplier table / overrides map outside the catalog (fastest, ~260-280 lines):** dismissed. It creates a second source of truth for write prices; this repo has already lived the drift (Moonshot `cacheWritesPrice: 0 "free for limited time"` staleness, MiMo precedent). Rejected 3/3 seats. +- **Option C — full staging behind a temporary `experiments` flag:** dismissed as shipped design. The flag is a throwaway surface with its own removal cost; C5b's knob is already the opt-in, so a second gate adds nothing. Its *ordering* (accounting first) is adopted into this decision. +- **Global-settings knob location (mlarge variant):** dismissed; provider-settings route verified independently by two seats with the `ApiHandlerOptions` citation. +- **Static Bedrock 1h model-id allowlist const:** dismissed; duplicates capability the catalog already expresses (drift surface), though the repo does carry the `BEDROCK_1M_CONTEXT_MODEL_IDS` precedent for simpler cases. + +## Consequences + +Positive: the 37.5% under-report becomes unrepresentable before the knob can ship; C5a is a pure reporting fix mergeable on its own; capability and price live in the catalog where pricing reviewers already look; Bedrock cannot be sent an unsupported TTL from the shipped code path. + +Negative: ~51 ungated catalog lines across `packages/types/src/providers/{anthropic,bedrock,openai}.ts` must be arithmetically reviewed by hand; `Task.ts` accumulator plumbing (~8 sites) is touched (a missed site silently falls back to the lump rate — mitigated by the catalog-invariant and split tests); a per-provider knob means users with many profiles set TTL per profile, not once globally; OpenRouter pass-through exposes a single `cache_write` figure, so its 1h pricing keeps the documented fallback rather than a split. + +## Risks and Mitigations + +- **Two Bedrock emitters diverge (mixed TTL request -> 400):** both-emitter test asserting every `cachePoint` carries the ttl (C7 plan). +- **Silent fallback re-hides the bug for a forgotten catalog entry:** invariant test — every Anthropic/Bedrock entry with `supportsPromptCache: true` gets a defined 1h price/capability. +- **Dynamic catalogs (OpenRouter fetcher, custom ARNs) lack capability data:** fail-closed to 5m; if the invariant cannot be enforced, C5a re-specs to omit 1h attribution rather than misprice. +- **Pinned SDK does not actually return the split on some path:** C5a reads via optional chain; if the Anthropic split proves unverifiable without a live call, gate C5b behind one CEO-key probe (same protocol as PR-C8) and ship C5a's OpenAI 12h half only. +- **`applyLongContextPricing` forgetting the new field (>200k-context 1h writes at half rate):** dedicated long-context split test. + +## Validation + +- The 37.5% tripwire test: 1,000 1h-write tokens at `inputPrice 3.0` must price **0.006**, not 0.00375 (`src/utils/__tests__/cost.spec.ts`). +- Default-parity: knob unset => serialized request contains no `"ttl"` key (snapshot/grep assertions per provider spec). +- Fail-closed gate: `promptCacheTtl:"1h"` + non-capable model => no `ttl` emitted; Bedrock Nova-class row asserted against `ValidationException` shape. +- Settings round-trip: saved `"1h"` survives `getStateToPostToWebview()`; unset stays `undefined` (default applied by consumers). +- Mixed-shape non-additivity fixtures (DeepSeek lesson, PR #2252) on every split reader. + +## Revisit Triggers + +- AWS publishes/changes the per-model 1h list, or a live probe returns `ValidationException` for 2+ allowlisted models -> re-pin or defer C7 permanently (Bedrock stays 5m). +- Vendors reprice TTL classes away from the 1.25x/2.0x/2.5x ratios -> catalog edit; if per-request mixed TTLs are ever wanted, the uniform-TTL invariant (I4) must gain ordering enforcement first. +- `packages/types` enters the mutation-gate `PACKAGE_CONFIGS` -> re-check C5a line budget. (UPDATE 261031: C3 verified `types` IS in PACKAGE_CONFIGS at scripts/stryker-diff.mjs:40-44 — C5a's catalog lines DO count against the gate; coder measured own 217/500, cumulative 281/500.) +- PR-C8's live Kimi/Moonshot probe returns per-TTL write fields -> extend C5a's readers with the Moonshot lever rather than a new design. + +## References + +- `docs/261030_0005_session_diet-caching-validation/FINAL-PLAN.md` (K5/K8 rows, DO-NOT list) +- Architect seat reports + pick record (session folder `docs/261031_0001_session_caching-diet-wave/`) +- PR #2252 (wire-shape lesson: fixtures must match vendor schema), PR #2260 (C5a implementation) +- ADR-0003, ADR-0005 (repo ADR numbering/format precedent) diff --git a/packages/types/src/providers/bedrock.ts b/packages/types/src/providers/bedrock.ts index 3d6efa0d8c..6447f0d73f 100644 --- a/packages/types/src/providers/bedrock.ts +++ b/packages/types/src/providers/bedrock.ts @@ -21,12 +21,21 @@ export const bedrockDefaultPromptRouterModelId: BedrockModelId = "anthropic.clau // https://aws.amazon.com/bedrock/pricing, including older models that are part // of the default prompt routers AWS enabled for GA of the promot router // feature. +// +// `promptCacheTtlSupport` marks the models whose Converse cache points accept an +// explicit `ttl` (ADR-0006 PR-C7). The 1h boundary is not a version rule: Opus 4.1 +// and Opus 4 are 5m-only while Opus 4.6/4.7/4.8 are 1h-capable, so the flag is set +// per exact model id from the AWS "Supported TTL" table and never inferred from a +// pattern. Rows without it are 5m-only, which makes Bedrock reject an explicit ttl +// (ValidationException), so absence must stay the fail-closed default. +// https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html export const bedrockModels = { "anthropic.claude-sonnet-4-5-20250929-v1:0": { maxTokens: 64_000, contextWindow: 200_000, supportsImages: true, supportsPromptCache: true, + promptCacheTtlSupport: ["5m", "1h"], supportsReasoningBudget: true, inputPrice: 3.0, outputPrice: 15.0, @@ -42,6 +51,7 @@ export const bedrockModels = { contextWindow: 200_000, // Default 200K, extendable to 1M with beta flag 'context-1m-2025-08-07' supportsImages: true, supportsPromptCache: true, + promptCacheTtlSupport: ["5m", "1h"], supportsReasoningBudget: true, inputPrice: 3.0, // $3 per million input tokens (≤200K context) outputPrice: 15.0, // $15 per million output tokens (≤200K context) @@ -69,6 +79,7 @@ export const bedrockModels = { contextWindow: 1_000_000, // 1M context window native (no beta header required) supportsImages: true, supportsPromptCache: true, + promptCacheTtlSupport: ["5m", "1h"], supportsReasoningBudget: true, supportsReasoningBinary: true, supportsTemperature: false, @@ -182,6 +193,7 @@ export const bedrockModels = { contextWindow: 200_000, // Default 200K, extendable to 1M with beta flag 'context-1m-2025-08-07' supportsImages: true, supportsPromptCache: true, + promptCacheTtlSupport: ["5m", "1h"], supportsReasoningBudget: true, inputPrice: 5.0, // $5 per million input tokens (≤200K context) outputPrice: 25.0, // $25 per million output tokens (≤200K context) @@ -208,6 +220,7 @@ export const bedrockModels = { contextWindow: 200_000, // Default 200K, extendable to 1M with beta flag 'context-1m-2025-08-07' supportsImages: true, supportsPromptCache: true, + promptCacheTtlSupport: ["5m", "1h"], supportsReasoningBudget: true, inputPrice: 5.0, // $5 per million input tokens (≤200K context) — verify against Bedrock console outputPrice: 25.0, // $25 per million output tokens (≤200K context) — verify against Bedrock console @@ -234,6 +247,7 @@ export const bedrockModels = { contextWindow: 200_000, // Default 200K, extendable to 1M with beta flag 'context-1m-2025-08-07' supportsImages: true, supportsPromptCache: true, + promptCacheTtlSupport: ["5m", "1h"], supportsReasoningBudget: true, inputPrice: 5.0, // $5 per million input tokens (≤200K context) — verify against Bedrock console outputPrice: 25.0, // $25 per million output tokens (≤200K context) — verify against Bedrock console @@ -263,6 +277,7 @@ export const bedrockModels = { contextWindow: 1_000_000, // 1M context window native (no beta header required) supportsImages: true, supportsPromptCache: true, + promptCacheTtlSupport: ["5m", "1h"], supportsReasoningBudget: true, supportsReasoningBinary: true, supportsTemperature: false, @@ -284,6 +299,7 @@ export const bedrockModels = { contextWindow: 1_000_000, // 1M context window native (no beta header required) supportsImages: true, supportsPromptCache: true, + promptCacheTtlSupport: ["5m", "1h"], supportsReasoningBudget: true, supportsReasoningBinary: true, supportsTemperature: false, @@ -309,6 +325,7 @@ export const bedrockModels = { contextWindow: 1_000_000, // 1M context window native (no beta header required) supportsImages: true, supportsPromptCache: true, + promptCacheTtlSupport: ["5m", "1h"], supportsReasoningBudget: true, supportsReasoningBinary: true, supportsTemperature: false, @@ -335,6 +352,7 @@ export const bedrockModels = { contextWindow: 1_000_000, supportsImages: true, supportsPromptCache: true, + promptCacheTtlSupport: ["5m", "1h"], supportsReasoningBudget: true, supportsReasoningBinary: true, supportsTemperature: false, @@ -354,6 +372,7 @@ export const bedrockModels = { contextWindow: 1_000_000, supportsImages: true, supportsPromptCache: true, + promptCacheTtlSupport: ["5m", "1h"], supportsReasoningBudget: true, supportsReasoningBinary: true, supportsTemperature: false, @@ -373,6 +392,7 @@ export const bedrockModels = { contextWindow: 200_000, supportsImages: true, supportsPromptCache: true, + promptCacheTtlSupport: ["5m", "1h"], supportsReasoningBudget: true, inputPrice: 5.0, outputPrice: 25.0, @@ -446,6 +466,7 @@ export const bedrockModels = { contextWindow: 200_000, supportsImages: true, supportsPromptCache: true, + promptCacheTtlSupport: ["5m", "1h"], supportsReasoningBudget: true, inputPrice: 1.0, outputPrice: 5.0, diff --git a/src/api/providers/__tests__/anthropic-vertex.spec.ts b/src/api/providers/__tests__/anthropic-vertex.spec.ts index f4860e9914..bae29755f1 100644 --- a/src/api/providers/__tests__/anthropic-vertex.spec.ts +++ b/src/api/providers/__tests__/anthropic-vertex.spec.ts @@ -8,6 +8,7 @@ import { VERTEX_1M_CONTEXT_MODEL_IDS } from "@roo-code/types" import { AnthropicVertexHandler } from "../anthropic-vertex" import { MAX_TIMEOUT_MS } from "../utils/abort-signal" +import { calculateApiCostAnthropic } from "../../../shared/cost" import { asyncStreamFrom, collectStream } from "../../../test-utils/stream" import { makeCreateMessageMetadata } from "../../../test-utils/api" import type { ApiStreamChunk } from "../../../api/transform/stream" @@ -1108,6 +1109,77 @@ describe("VertexHandler", () => { ) }) + it("completePrompt carries ttl 1h on the user marker when the model is capable", async () => { + handler = new AnthropicVertexHandler({ + apiModelId: "claude-3-5-sonnet-v2@20241022", + vertexProjectId: "test-project", + vertexRegion: "us-central1", + promptCacheTtl: "1h", + }) + + // The Vertex catalog carries no TTL capability yet, so inject one to + // exercise the emission path this PR adds. + const model = handler.getModel() + vitest.spyOn(handler, "getModel").mockReturnValue({ + ...model, + info: { ...model.info, promptCacheTtlSupport: ["5m", "1h"] }, + }) + + const mockCreate = vitest.fn().mockResolvedValue({ content: [{ type: "text", text: "ok" }] }) + const client = handler["client"] as unknown as { messages: { create: unknown } } + client.messages.create = mockCreate + + await handler.completePrompt("Test prompt") + + const body = mockCreate.mock.calls[0][0] + const controls: Array> = [] + const walk = (node: unknown) => { + if (Array.isArray(node)) { + node.forEach(walk) + return + } + if (node && typeof node === "object") { + for (const [key, child] of Object.entries(node as Record)) { + if (key === "cache_control" && child && typeof child === "object") { + controls.push(child as Record) + } + walk(child) + } + } + } + walk(JSON.parse(JSON.stringify(body))) + expect(controls).toHaveLength(1) + expect(controls[0]).toEqual({ type: "ephemeral", ttl: "1h" }) + }) + + it("completePrompt emits the plain ephemeral literal when the knob is unset", async () => { + handler = new AnthropicVertexHandler({ + apiModelId: "claude-3-5-sonnet-v2@20241022", + vertexProjectId: "test-project", + vertexRegion: "us-central1", + }) + + const mockCreate = vitest.fn().mockResolvedValue({ content: [{ type: "text", text: "ok" }] }) + const client = handler["client"] as unknown as { messages: { create: unknown } } + client.messages.create = mockCreate + + await handler.completePrompt("Test prompt") + + // Byte-parity with the pre-knob literal (ADR-0006 I3): no ttl key anywhere. + expect(JSON.stringify(mockCreate.mock.calls[0][0])).not.toContain("ttl") + expect(mockCreate).toHaveBeenCalledWith( + expect.objectContaining({ + messages: [ + { + role: "user", + content: [{ type: "text", text: "Test prompt", cache_control: { type: "ephemeral" } }], + }, + ], + }), + undefined, + ) + }) + it("should handle API errors for Claude", async () => { handler = new AnthropicVertexHandler({ apiModelId: "claude-3-5-sonnet-v2@20241022", @@ -1378,6 +1450,28 @@ describe("VertexHandler", () => { expect(model.betas).toContain("context-1m-2025-08-07") }) + it("bills a 1M-context 1h cache write at the tier-derived rate, not the 200K rate", () => { + // Review pin: the 1M merge carries the tier's TTL rate, but the vertex + // catalog declares no explicit cacheWritesPrice1h, so the effective 1h + // rate is derived from the applied input price (2x per the shared + // resolver). If the tier merge regressed to the entry-level input price, + // the derived 1h rate would drop from 12.0 to 6.0 (50% under-billing). + const handler = new AnthropicVertexHandler({ + apiModelId: "claude-sonnet-4-6", + vertexProjectId: "test-project", + vertexRegion: "us-central1", + vertex1MContext: true, + }) + + const model = handler.getModel() + const { totalCost } = calculateApiCostAnthropic(model.info, 0, 0, 100_000, 0, 100_000) + + // Derived tier 1h rate 12.0: (12.0 / 1_000_000) * 100_000 = 1.2 + // The entry-derived rate 6.0 would bill 0.6 (50% under). + expect(totalCost).toBeCloseTo(1.2, 9) + expect(totalCost).not.toBeCloseTo(0.6, 9) + }) + it("should enable 1M context for Claude Opus 4.7 when beta flag is set", () => { const handler = new AnthropicVertexHandler({ apiModelId: "claude-opus-4-7", diff --git a/src/api/providers/__tests__/anthropic.spec.ts b/src/api/providers/__tests__/anthropic.spec.ts index 4f15564323..415b6f8bfa 100644 --- a/src/api/providers/__tests__/anthropic.spec.ts +++ b/src/api/providers/__tests__/anthropic.spec.ts @@ -310,6 +310,32 @@ describe("AnthropicHandler", () => { const body = mockCreate.mock.calls[mockCreate.mock.calls.length - 1]?.[0] expect(JSON.stringify(body)).not.toContain("ttl") }) + + it("emits no ttl key for a 1h knob on a model without 1h capability", async () => { + // Every catalog model now advertises 1h (1h cache writes are GA on all + // active Claude models), so the reject path is exercised by resolving a + // 5m-only info — the shape a dynamic or stale catalog would produce. + // The knob must fail closed here too (ADR-0006 I2), not only in ttl.spec.ts. + const staleCatalogHandler = new AnthropicHandler({ + ...mockOptions, + promptCacheTtl: "1h", + }) + + const model = staleCatalogHandler.getModel() + vitest.spyOn(staleCatalogHandler, "getModel").mockReturnValue({ + ...model, + info: { ...model.info, promptCacheTtlSupport: ["5m"] }, + }) + + await collectStream(staleCatalogHandler.createMessage(systemPrompt, threeTurnMessages())) + + const body = mockCreate.mock.calls[mockCreate.mock.calls.length - 1]?.[0] + expect(JSON.stringify(body)).not.toContain("ttl") + + const controls = requestCacheControls() + expect(controls).toHaveLength(3) + expect(controls.every((control) => control.ttl === undefined)).toBe(true) + }) }) it("should include 1M context beta header for Claude Sonnet 4.6 when enabled", async () => { diff --git a/src/api/providers/__tests__/bedrock-cache-ttl.spec.ts b/src/api/providers/__tests__/bedrock-cache-ttl.spec.ts new file mode 100644 index 0000000000..45a7a7db46 --- /dev/null +++ b/src/api/providers/__tests__/bedrock-cache-ttl.spec.ts @@ -0,0 +1,577 @@ +// npx vitest run src/api/providers/__tests__/bedrock-cache-ttl.spec.ts + +const mockCaptureException = vi.fn() + +vi.mock("@roo-code/telemetry", () => ({ + TelemetryService: { + instance: { + captureException: (...args: unknown[]) => mockCaptureException(...args), + }, + }, +})) + +vi.mock("@aws-sdk/credential-providers", () => { + const mockFromIni = vi.fn().mockReturnValue({ + accessKeyId: "profile-access-key", + secretAccessKey: "profile-secret-key", + }) + return { fromIni: mockFromIni } +}) + +vi.mock("../../../utils/networkProxy", () => ({ + getSystemProxyUrl: vi.fn().mockReturnValue(undefined), +})) + +vi.mock("@smithy/node-http-handler", () => ({ + NodeHttpHandler: vi.fn(), +})) + +vi.mock("http-proxy-agent", () => ({ + HttpProxyAgent: vi.fn(), +})) + +vi.mock("https-proxy-agent", () => ({ + HttpsProxyAgent: vi.fn(), +})) + +vi.mock("@aws-sdk/client-bedrock-runtime", async () => { + const actual = await vi.importActual( + "@aws-sdk/client-bedrock-runtime", + ) + const mockSend = vi.fn().mockResolvedValue({ stream: [] }) + const mockConverseStreamCommand = vi.fn() + + return { + ...actual, + BedrockRuntimeClient: vi.fn().mockImplementation(function () { + return { send: mockSend } + }), + ConverseStreamCommand: mockConverseStreamCommand, + ConverseCommand: vi.fn(), + } +}) + +import { AwsBedrockHandler } from "../bedrock" +import { ConverseStreamCommand } from "@aws-sdk/client-bedrock-runtime" +import { bedrockModels, BEDROCK_CUSTOM_ARN_OTHER_BASE_MODEL, type ProviderSettings } from "@roo-code/types" + +import type { Anthropic } from "@anthropic-ai/sdk" +import { clearAllMocks } from "../../../test-utils/reset" +import { asyncStreamFrom, collectStream } from "../../../test-utils/stream" + +const mockConverseStreamCommand = vi.mocked(ConverseStreamCommand) + +/** + * Bedrock Converse cache-point TTL emission (ADR-0006 PR-C7). + * + * Every assertion here runs on the serialized `ConverseStreamCommand` input, i.e. + * the bytes that would reach AWS. Bedrock rejects an explicit `ttl` for a model + * that does not support it (`ValidationException`, 400, no fallback), and it also + * rejects a shorter TTL appearing before a longer one across tools -> system -> + * messages, so both "did we emit it everywhere" and "did we emit it nowhere" have + * to be checked by counting cache points rather than by spot-checking one. + */ + +/** Collect every `cachePoint` object reachable in a serialized request. */ +const collectCachePoints = (value: unknown): Array> => { + const found: Array> = [] + + const walk = (node: unknown) => { + if (Array.isArray(node)) { + node.forEach(walk) + return + } + + if (node && typeof node === "object") { + for (const [key, child] of Object.entries(node as Record)) { + if (key === "cachePoint" && child && typeof child === "object") { + found.push(child as Record) + } + walk(child) + } + } + } + + walk(value) + return found +} + +/** True when any `ttl` key exists anywhere in the serialized request. */ +const containsTtlKey = (value: unknown): boolean => { + if (Array.isArray(value)) return value.some(containsTtlKey) + if (value && typeof value === "object") { + return Object.entries(value as Record).some( + ([key, child]) => key === "ttl" || containsTtlKey(child), + ) + } + return false +} + +/** The serialized command input, so assertions run on the bytes that would be sent. */ +const serialize = (value: unknown) => JSON.parse(JSON.stringify(value)) as Record + +// The system prompt must clear minTokensPerCachePoint (1024 for the Claude rows) so +// the strategy actually places a system cache point, and the message tail must clear +// it too so a message-level cache point is placed. Both emitters then appear in one +// request, which is the case a single-emitter assertion would miss. +const longSystemPrompt = "You are a helpful assistant. ".repeat(200) +const longUserMessage = "Please review this file. ".repeat(200) + +const messages: Anthropic.Messages.MessageParam[] = [ + { role: "user", content: longUserMessage }, + { role: "assistant", content: "Here is the review. ".repeat(200) }, + { role: "user", content: longUserMessage }, +] + +const createHandler = (overrides: Partial = {}) => + new AwsBedrockHandler({ + apiModelId: "anthropic.claude-sonnet-4-5-20250929-v1:0", + awsAccessKey: "test-access-key", + awsSecretKey: "test-secret-key", + awsRegion: "us-east-1", + ...overrides, + }) + +/** Run one createMessage and return the serialized ConverseStreamCommand input. */ +const captureConverseInput = async (handler: AwsBedrockHandler): Promise> => { + mockConverseStreamCommand.mockClear() + const stream = handler.createMessage(longSystemPrompt, messages) + await collectStream(stream) + + expect(mockConverseStreamCommand).toHaveBeenCalledTimes(1) + return serialize(mockConverseStreamCommand.mock.calls[0][0]) +} + +describe("Bedrock cache-point TTL emission", () => { + beforeEach(() => { + clearAllMocks() + mockCaptureException.mockClear() + }) + + describe("1h-capable model with the knob at 1h", () => { + it("carries ttl 1h on every cache point in the request", async () => { + const handler = createHandler({ + apiModelId: "anthropic.claude-sonnet-4-5-20250929-v1:0", + promptCacheTtl: "1h", + }) + handler["client"].send = vi.fn().mockResolvedValue({ stream: asyncStreamFrom([]) }) + + const input = await captureConverseInput(handler) + const cachePoints = collectCachePoints(input) + + // Both emitters must be present, otherwise the "every cache point" claim is vacuous. + const systemCachePoints = collectCachePoints(input.system) + const messageCachePoints = collectCachePoints(input.messages) + expect(systemCachePoints.length).toBeGreaterThan(0) + expect(messageCachePoints.length).toBeGreaterThan(0) + expect(cachePoints.length).toBe(systemCachePoints.length + messageCachePoints.length) + + for (const cachePoint of cachePoints) { + expect(cachePoint).toEqual({ type: "default", ttl: "1h" }) + } + }) + + it("emits one uniform TTL per request, so the tools -> system -> messages ordering hazard stays unreachable", async () => { + const handler = createHandler({ + apiModelId: "anthropic.claude-sonnet-4-5-20250929-v1:0", + promptCacheTtl: "1h", + }) + handler["client"].send = vi.fn().mockResolvedValue({ stream: asyncStreamFrom([]) }) + + const input = await captureConverseInput(handler) + const ttls = new Set(collectCachePoints(input).map((cachePoint) => cachePoint.ttl)) + + // A set of size 1 is the whole invariant: with one value there is no + // "shorter before longer" pair to reject, regardless of placement order. + expect(ttls).toEqual(new Set(["1h"])) + }) + + it("keeps the cache-point placement identical to the unset-knob request", async () => { + const capable = { apiModelId: "anthropic.claude-sonnet-4-5-20250929-v1:0" } as const + + const withoutKnob = createHandler(capable) + withoutKnob["client"].send = vi.fn().mockResolvedValue({ stream: asyncStreamFrom([]) }) + const baseInput = await captureConverseInput(withoutKnob) + + const withKnob = createHandler({ ...capable, promptCacheTtl: "1h" }) + withKnob["client"].send = vi.fn().mockResolvedValue({ stream: asyncStreamFrom([]) }) + const ttlInput = await captureConverseInput(withKnob) + + // Strip the ttl the knob adds and the two requests must be byte-identical: + // adding a TTL must not move a single cache point. + const stripTtl = (value: unknown): unknown => + JSON.parse(JSON.stringify(value, (key, child) => (key === "ttl" ? undefined : child))) + + expect(stripTtl(ttlInput)).toEqual(baseInput) + expect(collectCachePoints(baseInput).length).toBe(collectCachePoints(ttlInput).length) + }) + }) + + describe("5m-only models with the knob at 1h (ValidationException tripwire)", () => { + // AWS rejects an explicit ttl for these with a 400 that kills the turn, so the + // assertion is that no `ttl` key exists anywhere in the serialized request. + it.each([ + ["Opus 4.1 (4.x generation, 5m-only)", "anthropic.claude-opus-4-1-20250805-v1:0"], + ["Opus 4 (pre-4.1, 5m-only)", "anthropic.claude-opus-4-20250514-v1:0"], + ["Claude 3.7 Sonnet (3.x, 5m-only)", "anthropic.claude-3-7-sonnet-20250219-v1:0"], + ["Nova Pro (non-Anthropic, 5m-only)", "amazon.nova-pro-v1:0"], + ])("emits no ttl for %s", async (_label, apiModelId) => { + const handler = createHandler({ apiModelId, promptCacheTtl: "1h" }) + handler["client"].send = vi.fn().mockResolvedValue({ stream: asyncStreamFrom([]) }) + + const input = await captureConverseInput(handler) + + expect(containsTtlKey(input)).toBe(false) + // The cache points themselves must still be there: the gate suppresses the + // ttl, not the caching. + for (const cachePoint of collectCachePoints(input)) { + expect(cachePoint).toEqual({ type: "default" }) + } + }) + }) + + describe("knob unset (default parity)", () => { + it("serializes a byte-identical request to a pre-knob build for a capable model", async () => { + const handler = createHandler({ apiModelId: "anthropic.claude-sonnet-4-5-20250929-v1:0" }) + handler["client"].send = vi.fn().mockResolvedValue({ stream: asyncStreamFrom([]) }) + + const input = await captureConverseInput(handler) + + expect(containsTtlKey(input)).toBe(false) + expect(collectCachePoints(input).length).toBeGreaterThan(0) + for (const cachePoint of collectCachePoints(input)) { + expect(cachePoint).toEqual({ type: "default" }) + } + }) + + it("treats an explicit 5m knob as unset so the wire shape does not change", async () => { + const capable = { apiModelId: "anthropic.claude-sonnet-4-5-20250929-v1:0" } as const + + const unset = createHandler(capable) + unset["client"].send = vi.fn().mockResolvedValue({ stream: asyncStreamFrom([]) }) + const unsetInput = await captureConverseInput(unset) + + const explicit5m = createHandler({ ...capable, promptCacheTtl: "5m" }) + explicit5m["client"].send = vi.fn().mockResolvedValue({ stream: asyncStreamFrom([]) }) + const explicit5mInput = await captureConverseInput(explicit5m) + + expect(explicit5mInput).toEqual(unsetInput) + expect(containsTtlKey(explicit5mInput)).toBe(false) + }) + }) + + describe("custom ARN / inference-profile resolution (fail closed)", () => { + it("emits 1h for an inference profile whose resolved base id is exactly listed", async () => { + // `us.` is a cross-region prefix, not part of the catalog id, so this only + // works if the prefix is stripped before the capability lookup. + const handler = createHandler({ + apiModelId: "us.anthropic.claude-sonnet-4-5-20250929-v1:0", + promptCacheTtl: "1h", + }) + handler["client"].send = vi.fn().mockResolvedValue({ stream: asyncStreamFrom([]) }) + + const input = await captureConverseInput(handler) + + expect(collectCachePoints(input).length).toBeGreaterThan(0) + for (const cachePoint of collectCachePoints(input)) { + expect(cachePoint).toEqual({ type: "default", ttl: "1h" }) + } + }) + + it("emits no ttl for an inference-profile ARN that names a 5m-only model", async () => { + // The ARN names Opus 4.1, which is 5m-only despite being a 4.x generation. + // Resolution succeeds (so cache points are placed), but the capability + // lookup must still refuse the TTL. + const handler = createHandler({ + awsCustomArn: + "arn:aws:bedrock:us-east-1:123456789012:inference-profile/us.anthropic.claude-opus-4-1-20250805-v1:0", + promptCacheTtl: "1h", + }) + handler["client"].send = vi.fn().mockResolvedValue({ stream: asyncStreamFrom([]) }) + + const input = await captureConverseInput(handler) + + // Cache points exist, so the assertion is about the gate rather than about + // caching being off for an unrelated reason. + expect(collectCachePoints(input).length).toBeGreaterThan(0) + expect(containsTtlKey(input)).toBe(false) + }) + + it("places no cache points at all when the ARN resolves to no catalog model", async () => { + // The deepest fail-closed path: an application inference profile the user + // marked as "other" resolves to no catalog entry, so the guessed ModelInfo + // reports no prompt-cache support and no capability can exist. + const handler = createHandler({ + awsCustomArn: "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/abc123", + awsCustomArnBaseModelId: BEDROCK_CUSTOM_ARN_OTHER_BASE_MODEL, + promptCacheTtl: "1h", + }) + handler["client"].send = vi.fn().mockResolvedValue({ stream: asyncStreamFrom([]) }) + + const input = await captureConverseInput(handler) + + expect(collectCachePoints(input)).toEqual([]) + expect(containsTtlKey(input)).toBe(false) + }) + + it("emits no ttl for an unlisted model that the heuristic fallback guesses", async () => { + // The fallback spreads the default row (which is 1h-capable) and then overlays + // the guess, so the capability is inherited unless it is cleared. An unlisted + // id is exactly the case where we cannot prove the model accepts 1h, and Bedrock + // answers an unsupported ttl with a ValidationException that kills the turn. + const handler = createHandler({ + awsCustomArn: + "arn:aws:bedrock:us-east-1:123456789012:inference-profile/us.anthropic.claude-3-5-sonnet-20241022-v1:0", + promptCacheTtl: "1h", + }) + handler["client"].send = vi.fn().mockResolvedValue({ stream: asyncStreamFrom([]) }) + + const input = await captureConverseInput(handler) + + // The guess still reports caching support, so cache points exist and the + // assertion is about the capability gate rather than caching being off. + expect(collectCachePoints(input).length).toBeGreaterThan(0) + expect(containsTtlKey(input)).toBe(false) + }) + + it("emits 1h for a foundation-model ARN that names a listed model", async () => { + const handler = createHandler({ + awsCustomArn: "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-sonnet-4-5-20250929-v1:0", + promptCacheTtl: "1h", + }) + handler["client"].send = vi.fn().mockResolvedValue({ stream: asyncStreamFrom([]) }) + + const input = await captureConverseInput(handler) + + expect(collectCachePoints(input).length).toBeGreaterThan(0) + for (const cachePoint of collectCachePoints(input)) { + expect(cachePoint).toEqual({ type: "default", ttl: "1h" }) + } + }) + }) + + describe("usage cacheDetails split", () => { + it("reports the 1h share of the cache writes when Converse returns a split", async () => { + const handler = createHandler({ + apiModelId: "anthropic.claude-sonnet-4-5-20250929-v1:0", + promptCacheTtl: "1h", + }) + handler["client"].send = vi.fn().mockResolvedValue({ + stream: asyncStreamFrom([ + { + metadata: { + usage: { + inputTokens: 10, + outputTokens: 5, + cacheWriteInputTokens: 3000, + cacheDetails: [ + { ttl: "1h", inputTokens: 2000 }, + { ttl: "5m", inputTokens: 1000 }, + ], + }, + }, + }, + ]), + }) + + const chunks = await collectStream(handler.createMessage(longSystemPrompt, messages)) + const usage = chunks.find((chunk) => chunk.type === "usage") + + expect(usage).toMatchObject({ + type: "usage", + cacheWriteTokens: 3000, + cacheWrite1hTokens: 2000, + }) + }) + + it("omits the 1h share when the response carries no cacheDetails", async () => { + const handler = createHandler({ apiModelId: "anthropic.claude-sonnet-4-5-20250929-v1:0" }) + handler["client"].send = vi.fn().mockResolvedValue({ + stream: asyncStreamFrom([ + { metadata: { usage: { inputTokens: 10, outputTokens: 5, cacheWriteInputTokens: 3000 } } }, + ]), + }) + + const chunks = await collectStream(handler.createMessage(longSystemPrompt, messages)) + const usage = chunks.find((chunk) => chunk.type === "usage") + + expect(usage).toMatchObject({ type: "usage", cacheWriteTokens: 3000 }) + expect((usage as { cacheWrite1hTokens?: number }).cacheWrite1hTokens).toBeUndefined() + }) + + it.each([ + ["a 5m entry and no 1h entry", [{ ttl: "5m", inputTokens: 1000 }]], + ["a negative 1h count", [{ ttl: "1h", inputTokens: -5 }]], + ["a zero 1h count", [{ ttl: "1h", inputTokens: 0 }]], + ])("omits the 1h share when the split carries %s", async (_label, cacheDetails) => { + // Each row is a distinct malformed shape the reader must absorb rather than + // bill: an absent 1h entry (optional chain), a negative count (the `<= 0` + // guard, which `!oneHour` alone does not cover), and a zero count. + const handler = createHandler({ apiModelId: "anthropic.claude-sonnet-4-5-20250929-v1:0" }) + handler["client"].send = vi.fn().mockResolvedValue({ + stream: asyncStreamFrom([ + { + metadata: { + usage: { + inputTokens: 10, + outputTokens: 5, + cacheWriteInputTokens: 3000, + cacheDetails, + }, + }, + }, + ]), + }) + + const chunks = await collectStream(handler.createMessage(longSystemPrompt, messages)) + const usage = chunks.find((chunk) => chunk.type === "usage") + + expect(usage).toMatchObject({ type: "usage", cacheWriteTokens: 3000 }) + expect((usage as { cacheWrite1hTokens?: number }).cacheWrite1hTokens).toBeUndefined() + }) + + it("reads the 1h entry wherever it sits in the split, not just the first entry", async () => { + // Converse orders the split longest-TTL-first today, but the reader must match + // on the ttl value rather than trusting position: a 5m entry ahead of the 1h + // entry would otherwise report the 5m count as the 1h share. + const handler = createHandler({ apiModelId: "anthropic.claude-sonnet-4-5-20250929-v1:0" }) + handler["client"].send = vi.fn().mockResolvedValue({ + stream: asyncStreamFrom([ + { + metadata: { + usage: { + inputTokens: 10, + outputTokens: 5, + cacheWriteInputTokens: 3000, + cacheDetails: [ + { ttl: "5m", inputTokens: 1000 }, + { ttl: "1h", inputTokens: 2000 }, + ], + }, + }, + }, + ]), + }) + + const chunks = await collectStream(handler.createMessage(longSystemPrompt, messages)) + const usage = chunks.find((chunk) => chunk.type === "usage") + + expect(usage).toMatchObject({ type: "usage", cacheWriteTokens: 3000, cacheWrite1hTokens: 2000 }) + }) + + it("clamps the 1h share to the reported aggregate so a malformed split cannot over-bill", async () => { + const handler = createHandler({ apiModelId: "anthropic.claude-sonnet-4-5-20250929-v1:0" }) + handler["client"].send = vi.fn().mockResolvedValue({ + stream: asyncStreamFrom([ + { + metadata: { + usage: { + inputTokens: 10, + outputTokens: 5, + cacheWriteInputTokens: 500, + cacheDetails: [{ ttl: "1h", inputTokens: 900 }], + }, + }, + }, + ]), + }) + + const chunks = await collectStream(handler.createMessage(longSystemPrompt, messages)) + const usage = chunks.find((chunk) => chunk.type === "usage") + + expect(usage).toMatchObject({ type: "usage", cacheWriteTokens: 500, cacheWrite1hTokens: 500 }) + }) + }) +}) + +describe("Bedrock cache TTL capability catalog", () => { + const oneHourIds = Object.entries(bedrockModels) + .filter(([, info]) => (info as { promptCacheTtlSupport?: string[] }).promptCacheTtlSupport?.includes("1h")) + .map(([id]) => id) + .sort() + + it("marks exactly the 1h-capable Claude rows the AWS TTL table lists", () => { + // Pinned by exact id, not by pattern: Opus 4.1 and Opus 4 are 5m-only while + // Opus 4.6/4.7/4.8 are 1h-capable, so a version heuristic mis-classifies the + // boundary in both directions. Adding a row here means adding it to the AWS + // table first. + expect(oneHourIds).toEqual( + [ + "anthropic.claude-fable-5", + "anthropic.claude-fable-5-1", + "anthropic.claude-haiku-4-5-20251001-v1:0", + "anthropic.claude-haiku-5-5", + "anthropic.claude-opus-4-5-20251101-v1:0", + "anthropic.claude-opus-4-6-v1", + "anthropic.claude-opus-4-7", + "anthropic.claude-opus-4-8", + "anthropic.claude-opus-5", + "anthropic.claude-opus-5-5", + "anthropic.claude-sonnet-4-5-20250929-v1:0", + "anthropic.claude-sonnet-4-6", + "anthropic.claude-sonnet-5", + ].sort(), + ) + }) + + it("never marks a Nova row, because Nova rejects an extended TTL outright", () => { + // The observed error for a Nova model is "Extended TTL prompt caching is only + // supported for Anthropic models", and Nova Premier is unresolved in the AWS + // table, so no Nova row may claim 1h. + const novaRows = Object.entries(bedrockModels) + .filter(([id]) => id.startsWith("amazon.nova")) + .filter(([, info]) => (info as { promptCacheTtlSupport?: string[] }).promptCacheTtlSupport !== undefined) + .map(([id]) => id) + + expect(novaRows).toEqual([]) + }) + + it("leaves the 5m-only Claude rows without capability so they fail closed", () => { + const fiveMinuteOnly = [ + "anthropic.claude-3-5-haiku-20241022-v1:0", + "anthropic.claude-3-5-sonnet-20241022-v2:0", + "anthropic.claude-3-7-sonnet-20250219-v1:0", + "anthropic.claude-opus-4-1-20250805-v1:0", + "anthropic.claude-opus-4-20250514-v1:0", + "anthropic.claude-sonnet-4-20250514-v1:0", + ] + + for (const id of fiveMinuteOnly) { + const info = (bedrockModels as Record)[id] + expect(info, id).toBeDefined() + expect(info.promptCacheTtlSupport, id).toBeUndefined() + } + }) + + it("keys every 1h cache-write rate off the same tier's input price", () => { + // The tier-shadow bug class: a tier that overrides inputPrice must override + // cacheWritesPrice1h too, otherwise a long-context 1h write bills at the base + // tier's rate. Checked per entry and per tier, not just on the top-level row. + for (const [id, info] of Object.entries(bedrockModels)) { + const entry = info as { + inputPrice?: number + cacheWritesPrice1h?: number + tiers?: Array<{ inputPrice?: number; cacheWritesPrice1h?: number }> + } + if (entry.cacheWritesPrice1h !== undefined && entry.inputPrice !== undefined) { + expect(entry.cacheWritesPrice1h, id).toBeCloseTo(entry.inputPrice * 2, 9) + } + + for (const [index, tier] of (entry.tiers ?? []).entries()) { + // A tier that reprices input without repricing the 1h write is exactly the + // shadow this test exists to catch, so it must fail rather than be skipped. + // The exemption is only for rows that price no 1h write at all: a 5m-only + // model's tier has no 1h rate to shadow. + if (tier.cacheWritesPrice1h === undefined) { + expect( + tier.inputPrice === undefined || entry.cacheWritesPrice1h === undefined, + `${id} tier ${index} reprices input but not the 1h write`, + ).toBe(true) + continue + } + expect(tier.inputPrice, `${id} tier ${index}`).toBeDefined() + expect(tier.cacheWritesPrice1h, `${id} tier ${index}`).toBeCloseTo((tier.inputPrice as number) * 2, 9) + } + } + }) +}) diff --git a/src/api/providers/__tests__/bedrock.spec.ts b/src/api/providers/__tests__/bedrock.spec.ts index 0d8e5a784a..f21f0cd1e4 100644 --- a/src/api/providers/__tests__/bedrock.spec.ts +++ b/src/api/providers/__tests__/bedrock.spec.ts @@ -1159,6 +1159,25 @@ describe("AwsBedrockHandler", () => { expect(model.info.outputPrice).toBe(22.5) }) + it("carries the tier's own cacheWritesPrice1h when the 1M context tier applies", () => { + // Review pin: the 1M merge overrode inputPrice but must also carry the + // tier's TTL rate. The entry-level 1h rate was derived from the 200K input + // price, so a fallback to it bills a 1M-context 1h write at half the rate. + const handler = new AwsBedrockHandler({ + apiModelId: "anthropic.claude-sonnet-4-6", + awsAccessKey: "test", + awsSecretKey: "test", + awsRegion: "us-east-1", + awsBedrock1MContext: true, + }) + + const model = handler.getModel() + + // Tier input price 6.0 => 1h rate must be 2x that, not the 200K entry's 6.0. + expect(model.info.cacheWritesPrice1h).toBe(12.0) + expect(model.info.cacheWritesPrice1h).not.toBe(6.0) + }) + it("should use default context window when awsBedrock1MContext is false for Claude Sonnet 4", () => { const handler = new AwsBedrockHandler({ apiModelId: BEDROCK_1M_CONTEXT_MODEL_IDS[0], diff --git a/src/api/providers/__tests__/zoo-gateway.spec.ts b/src/api/providers/__tests__/zoo-gateway.spec.ts index a9beaada1c..09460f099c 100644 --- a/src/api/providers/__tests__/zoo-gateway.spec.ts +++ b/src/api/providers/__tests__/zoo-gateway.spec.ts @@ -67,6 +67,24 @@ const DEFAULT_MODEL_CATALOG = vitest.hoisted(() => ({ cacheReadsPrice: 0.1, description: "Claude 3.5 Haiku", }, + // The runtime gateway catalog never sets `promptCacheTtlSupport`, so this row + // exists only to inject the capability the accept path needs. It is kept + // separate from the shared haiku row so the reject-path tests keep a model + // with no TTL capability. + "anthropic/claude-3.5-sonnet": { + maxTokens: 8192, + contextWindow: 200000, + supportsImages: true, + supportsPromptCache: true, + // `as const` keeps the literal a "1h" tuple; a plain ["1h"] widens to + // string[] and fails the ModelRecord index signature. + promptCacheTtlSupport: ["1h" as const], + inputPrice: 3, + outputPrice: 15, + cacheWritesPrice: 3.75, + cacheReadsPrice: 0.3, + description: "Claude 3.5 Sonnet", + }, })) vitest.mock("../fetchers/modelCache", () => ({ @@ -421,6 +439,40 @@ describe("ZooGatewayHandler", () => { expect(addCacheBreakpoints).toHaveBeenCalled() }) + it("does not emit a ttl for a gateway model with no TTL capability", async () => { + // The runtime gateway catalog carries no TTL capability, so a 1h knob must + // fail closed (ADR-0006 I2) even though the model is prompt-cache capable. + const { addCacheBreakpoints } = await import("../../transform/caching/vercel-ai-gateway") + const handler = new ZooGatewayHandler({ + ...mockOptions, + zooGatewayModelId: "anthropic/claude-3.5-haiku", + promptCacheTtl: "1h", + }) + + await handler.createMessage("prompt", [{ role: "user", content: "Hi" }]).next() + + expect(addCacheBreakpoints).toHaveBeenCalled() + const lastCall = vitest.mocked(addCacheBreakpoints).mock.calls.at(-1) + expect(lastCall).toBeDefined() + expect(lastCall![2]).toBeUndefined() + }) + + it("passes the resolved ttl for a gateway model with 1h TTL capability", async () => { + // The runtime catalog never sets `promptCacheTtlSupport`, so the accept path is + // only reachable with an injected capability. The dedicated catalog row keeps + // the reject-path model capability-free. + const { addCacheBreakpoints } = await import("../../transform/caching/vercel-ai-gateway") + const handler = new ZooGatewayHandler({ + ...mockOptions, + zooGatewayModelId: "anthropic/claude-3.5-sonnet", + promptCacheTtl: "1h", + }) + + await handler.createMessage("prompt", [{ role: "user", content: "Hi" }]).next() + + expect(vitest.mocked(addCacheBreakpoints).mock.calls.at(-1)![2]).toBe("1h") + }) + it("yields tool_call_partial chunks when streaming tool calls", async () => { mockCreate.mockImplementation(async () => asyncStreamFrom([ diff --git a/src/api/providers/bedrock.ts b/src/api/providers/bedrock.ts index d11f5f7dba..c72dc8f360 100644 --- a/src/api/providers/bedrock.ts +++ b/src/api/providers/bedrock.ts @@ -3,12 +3,13 @@ import { ConverseStreamCommand, ConverseCommand, BedrockRuntimeClientConfig, - ContentBlock, Message, SystemContentBlock, Tool, ToolConfiguration, ToolChoice, + CacheTTL, + type CacheDetail, } from "@aws-sdk/client-bedrock-runtime" import { NodeHttpHandler } from "@smithy/node-http-handler" import OpenAI from "openai" @@ -22,6 +23,7 @@ import { type ProviderSettings, type BedrockModelId, type BedrockServiceTier, + type PromptCacheTtl, bedrockDefaultModelId, bedrockModels, bedrockDefaultPromptRouterModelId, @@ -47,6 +49,7 @@ import { logger } from "../../utils/logging" import { Package } from "../../shared/package" import { MultiPointStrategy } from "../transform/cache-strategy/multi-point-strategy" import { ModelInfo as CacheModelInfo } from "../transform/cache-strategy/types" +import { resolveCacheTtl } from "../transform/caching/ttl" import { convertToBedrockConverseMessages as sharedConverter } from "../transform/bedrock-converse-format" import { getModelParams } from "../transform/model-params" import { shouldUseReasoningBudget } from "../../shared/api" @@ -179,6 +182,9 @@ export interface StreamEvent { cacheWriteInputTokens?: number cacheReadInputTokenCount?: number cacheWriteInputTokenCount?: number + // Per-TTL split of the cache writes. Converse returns one entry per TTL + // class that received writes, longest TTL first. + cacheDetails?: CacheDetail[] } metrics?: { latencyMs: number @@ -210,6 +216,25 @@ export type UsageType = { cacheWriteInputTokens?: number cacheReadInputTokenCount?: number cacheWriteInputTokenCount?: number + cacheDetails?: CacheDetail[] +} + +/** + * Read the 1h share of a Converse `cacheDetails` split. + * + * Best effort by construction: the field is optional in the pinned SDK and absent + * on every pre-1h response, so a missing split means "no long-lived write was + * reported" and the cost layer prices the whole aggregate at the 5m rate. The + * counts are a decomposition of `cacheWriteInputTokens`, never an addition to it, + * so the value is clamped to the aggregate to keep a malformed response from + * billing more writes than were reported. + */ +function readCacheWrite1hTokens(usage: UsageType, cacheWriteTokens: number): number | undefined { + const oneHour = usage.cacheDetails?.find((detail) => detail.ttl === CacheTTL.ONE_HOUR)?.inputTokens + if (!oneHour || oneHour <= 0) { + return undefined + } + return Math.min(oneHour, cacheWriteTokens) } /************************************************************************************ @@ -448,6 +473,7 @@ export class AwsBedrockHandler extends BaseProvider implements SingleCompletionH usePromptCache, modelConfig.info, conversationId, + resolveCacheTtl(this.options, modelConfig.info), ) let additionalModelRequestFields: BedrockAdditionalModelFields | undefined @@ -642,6 +668,7 @@ export class AwsBedrockHandler extends BaseProvider implements SingleCompletionH outputTokens: usage.outputTokens || 0, cacheReadTokens: cacheReadTokens, cacheWriteTokens: cacheWriteTokens, + cacheWrite1hTokens: readCacheWrite1hTokens(usage, cacheWriteTokens), } continue } @@ -974,7 +1001,11 @@ export class AwsBedrockHandler extends BaseProvider implements SingleCompletionH } /** - * Convert Anthropic messages to Bedrock Converse format + * Convert Anthropic messages to Bedrock Converse format. + * + * `cacheTtl` is resolved once by the caller and threaded into the strategy, so + * both cache-point emitters in this method (the strategy's system/message + * blocks and the message-level literal below) carry the same value. */ private convertToBedrockConverseMessages( anthropicMessages: Anthropic.Messages.MessageParam[] | { role: string; content: string }[], @@ -982,6 +1013,7 @@ export class AwsBedrockHandler extends BaseProvider implements SingleCompletionH usePromptCache: boolean = false, modelInfo?: any, conversationId?: string, // Optional conversation ID to track cache points across messages + cacheTtl?: PromptCacheTtl, ): { system: SystemContentBlock[]; messages: Message[] } { // First convert messages using shared converter for proper image handling const convertedMessages = sharedConverter(anthropicMessages as Anthropic.Messages.MessageParam[]) @@ -1017,6 +1049,7 @@ export class AwsBedrockHandler extends BaseProvider implements SingleCompletionH messages: anthropicMessages as Anthropic.Messages.MessageParam[], usePromptCache, previousCachePointPlacements: previousPlacements, + cacheTtl, } // Get cache point placements @@ -1028,13 +1061,15 @@ export class AwsBedrockHandler extends BaseProvider implements SingleCompletionH this.previousCachePointPlacements[conversationId] = cacheResult.messageCachePointPlacements } - // Apply cache points to the properly converted messages + // Apply cache points to the properly converted messages. The strategy owns the + // cache-point shape, so the message-level emitter asks it rather than repeating + // the literal: that is what keeps both emitters on one TTL. const messagesWithCache = convertedMessages.map((msg, index) => { const placement = cacheResult.messageCachePointPlacements?.find((p) => p.index === index) if (placement) { return { ...msg, - content: [...(msg.content || []), { cachePoint: { type: "default" } } as ContentBlock], + content: [...(msg.content || []), strategy.createCachePoint()], } } return msg @@ -1212,13 +1247,15 @@ export class AwsBedrockHandler extends BaseProvider implements SingleCompletionH } else { // Use heuristics for model info, then allow overrides from ProviderSettings const guessed = this.guessModelInfoFromId(modelId) - model = { - id: bedrockDefaultModelId, - info: { - ...JSON.parse(JSON.stringify(bedrockModels[bedrockDefaultModelId])), - ...guessed, - }, + const info: ModelInfo = { + ...JSON.parse(JSON.stringify(bedrockModels[bedrockDefaultModelId])), + ...guessed, } + // The default row is 1h-capable, but an id we cannot resolve is not. Inheriting + // the capability would send `ttl: "1h"` to a model that may reject it with a + // ValidationException that kills the turn, so an unlisted id fails closed (I2). + delete info.promptCacheTtlSupport + model = { id: bedrockDefaultModelId, info } } // Always allow user to override detected/guessed maxTokens and contextWindow diff --git a/src/api/transform/cache-strategy/__tests__/cache-strategy-ttl.spec.ts b/src/api/transform/cache-strategy/__tests__/cache-strategy-ttl.spec.ts new file mode 100644 index 0000000000..e295d9ef6a --- /dev/null +++ b/src/api/transform/cache-strategy/__tests__/cache-strategy-ttl.spec.ts @@ -0,0 +1,121 @@ +// npx vitest run src/api/transform/cache-strategy/__tests__/cache-strategy-ttl.spec.ts + +import type { ContentBlock, SystemContentBlock } from "@aws-sdk/client-bedrock-runtime" + +import { MultiPointStrategy } from "../multi-point-strategy" +import type { CacheStrategyConfig, ModelInfo } from "../types" + +/** + * TTL threading through the cache strategy (ADR-0006 PR-C7). + * + * The strategy is the single owner of the cache-point shape, so these tests pin the + * two properties the provider relies on: a configured TTL reaches every cache point + * it emits, and an unset TTL leaves the block byte-identical to a pre-knob build. + */ + +const modelInfo: ModelInfo = { + maxTokens: 8192, + contextWindow: 200_000, + supportsPromptCache: true, + maxCachePoints: 4, + minTokensPerCachePoint: 50, + cachableFields: ["system", "messages", "tools"], +} + +// The strategy estimates tokens by counting whitespace-separated words, so the +// fixtures need real words rather than one long run of characters. +const longText = (tokens: number) => "word ".repeat(tokens) + +const createConfig = (overrides: Partial = {}): CacheStrategyConfig => ({ + modelInfo, + systemPrompt: longText(200), + messages: [ + { role: "user", content: longText(200) }, + { role: "assistant", content: longText(200) }, + { role: "user", content: longText(200) }, + ], + usePromptCache: true, + ...overrides, +}) + +const collectCachePoints = (value: unknown): Array> => { + const found: Array> = [] + + const walk = (node: unknown) => { + if (Array.isArray(node)) { + node.forEach(walk) + return + } + + if (node && typeof node === "object") { + for (const [key, child] of Object.entries(node as Record)) { + if (key === "cachePoint" && child && typeof child === "object") { + found.push(child as Record) + } + walk(child) + } + } + } + + walk(value) + return found +} + +const serialize = (value: unknown) => JSON.parse(JSON.stringify(value)) + +describe("MultiPointStrategy cache TTL", () => { + it("carries the configured TTL on the system and message cache points", () => { + const strategy = new MultiPointStrategy(createConfig({ cacheTtl: "1h" })) + const result = strategy.determineOptimalCachePoints() + + const cachePoints = collectCachePoints(serialize({ system: result.system, messages: result.messages })) + + expect(collectCachePoints(serialize(result.system)).length).toBeGreaterThan(0) + expect(collectCachePoints(serialize(result.messages)).length).toBeGreaterThan(0) + for (const cachePoint of cachePoints) { + expect(cachePoint).toEqual({ type: "default", ttl: "1h" }) + } + }) + + it("omits the ttl key entirely when no TTL is configured", () => { + const strategy = new MultiPointStrategy(createConfig()) + const result = strategy.determineOptimalCachePoints() + + const cachePoints = collectCachePoints(serialize({ system: result.system, messages: result.messages })) + + expect(cachePoints.length).toBeGreaterThan(0) + for (const cachePoint of cachePoints) { + // Deep-equality against the pre-knob literal, not just "no ttl": a stray + // extra key would change the request bytes just as surely. + expect(cachePoint).toEqual({ type: "default" }) + } + }) + + it("places the same cache points with and without a TTL", () => { + const withoutTtl = new MultiPointStrategy(createConfig()).determineOptimalCachePoints() + const withTtl = new MultiPointStrategy(createConfig({ cacheTtl: "1h" })).determineOptimalCachePoints() + + expect(withTtl.messageCachePointPlacements).toEqual(withoutTtl.messageCachePointPlacements) + expect(collectCachePoints(serialize(withTtl.system)).length).toBe( + collectCachePoints(serialize(withoutTtl.system)).length, + ) + }) + + it("keeps createCachePoint public so the provider's second emitter can reuse it", () => { + // The provider's message-level emitter used to build its own literal, which is + // how two emitters drift apart. Asserting the method is reachable from outside + // the class is the cheap structural guard against it going back to private. + const strategy = new MultiPointStrategy(createConfig({ cacheTtl: "1h" })) + const block = strategy.createCachePoint() as unknown as { cachePoint: Record } + + expect(block.cachePoint).toEqual({ type: "default", ttl: "1h" }) + expect(typeof (block as unknown as ContentBlock)).toBe("object") + }) + + it("returns an untyped-compatible block for the system union as well", () => { + const strategy = new MultiPointStrategy(createConfig({ cacheTtl: "1h" })) + const block = strategy.createCachePoint() as unknown as SystemContentBlock + + expect(collectCachePoints(serialize(block))).toEqual([{ type: "default", ttl: "1h" }]) + }) +}) diff --git a/src/api/transform/cache-strategy/base-strategy.ts b/src/api/transform/cache-strategy/base-strategy.ts index 1bc05cdb84..740940d805 100644 --- a/src/api/transform/cache-strategy/base-strategy.ts +++ b/src/api/transform/cache-strategy/base-strategy.ts @@ -48,10 +48,21 @@ export abstract class CacheStrategy { } /** - * Create a cache point content block + * Create a cache point content block. + * + * `ttl` is omitted, not set to "5m", when no TTL was resolved: AWS documents + * omitting the field as the way to get the default 5-minute behaviour, and an + * explicit "5m" would change the bytes of every existing request. + * + * Public so the provider's message-level emitter can reuse it instead of + * building its own literal: one factory means one TTL per request (I4) and + * makes a mixed-TTL request unrepresentable rather than merely unlikely. */ - protected createCachePoint(): ContentBlock { - return { cachePoint: { type: "default" } } as unknown as ContentBlock + public createCachePoint(): ContentBlock { + const { cacheTtl } = this.config + return { + cachePoint: cacheTtl ? { type: "default", ttl: cacheTtl } : { type: "default" }, + } as ContentBlock } /** diff --git a/src/api/transform/cache-strategy/types.ts b/src/api/transform/cache-strategy/types.ts index 4215295d6a..94a02f9c44 100644 --- a/src/api/transform/cache-strategy/types.ts +++ b/src/api/transform/cache-strategy/types.ts @@ -1,5 +1,6 @@ import { Anthropic } from "@anthropic-ai/sdk" import { SystemContentBlock, Message } from "@aws-sdk/client-bedrock-runtime" +import type { PromptCacheTtl } from "@roo-code/types" /** * Information about a model's capabilities and constraints @@ -57,4 +58,12 @@ export interface CacheStrategyConfig { usePromptCache: boolean /** Previous cache point placements (for maintaining consistency across consecutive messages) */ previousCachePointPlacements?: CachePointPlacement[] + /** + * Cache TTL every cache point in this request carries, already resolved against + * the model's catalog capability. One value per request is the point: Bedrock + * Converse rejects a shorter TTL appearing before a longer one across + * tools -> system -> messages, so a per-point TTL would be a 400 waiting for the + * first mixed placement. Leaving it unset omits `ttl` from the wire entirely. + */ + cacheTtl?: PromptCacheTtl }