DX Audit app / stripe
demo
major open stripe:test-tokens-not-fetchable

Declined-card fixtures were not surfaced in the audit's WebFetch output

Neither model received the declined-card fixture table in its WebFetch output. Opus tried three URL variants, then sent a raw card number that the API rejected before inferring pm_card_chargeDeclined. Sonnet tried ten URL and fragment variants, skipped the raw-number attempt, and inferred the same fixture name. A verified repository repro found that Stripe's documentation did contain the fixtures: the long Markdown/HTML response was truncated before the relevant table, and fragment anchors did not change the server response. Dynamic rendering was not the cause. The measured major friction is that repeated first-party documentation fetches failed to surface a required deterministic test value, forcing fallback to learned or inferred naming.

How can I reproduce this?

Requires: None — only a programmatic HTTP client (curl, fetch, or equivalent)

  1. Fetch https://docs.stripe.com/llms.txt and find the link to the testing page (docs.stripe.com/testing.md).
  2. Fetch https://docs.stripe.com/testing.md and search the response for 4000000000000002 (the generic decline card number) or pm_card_chargeDeclined.
  3. Try the HTML variant: fetch https://docs.stripe.com/testing and search for the same tokens.
  4. Try fragment anchors (#declined-payments, #cards).

Expected: Neither token appears in any variant: the .md content truncates in the international-cards section before the declined-payments table, the HTML page is too long for the section to be reached, and fragment anchors are client-side only — they do not change the server response.

Evidence · transcript moments

Opus Audit run

Show all 3

Sonnet Audit run

Show all 3