Skip to content

read sources version by version: list, read at, extract - #209

Open
adssib wants to merge 2 commits into
pl-r3-nolistsfrom
pl-r4-versions
Open

adssib wants to merge 2 commits into
pl-r3-nolistsfrom
pl-r4-versions

Conversation

@adssib

@adssib adssib commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #208. Refs #179 (I19.8), part of #171.

What this changes

Additive interface in core/plugin (versions.go), next to Plugin{Name, Run}, which keeps working unchanged:

type Version  struct{ ID string; Released time.Time; Ref, URL string }
type Snapshot struct{ Version Version; Content []byte; SourceURL string }

type VersionLister interface{ ListVersions(ctx, env) ([]Version, error) }       // oldest first
type Reader        interface{ ReadAt(ctx, env, Version) (Snapshot, error) }
type Extractor     interface{ Extract(ctx, env, from, to Snapshot) (Result, error) } // previous_version = from's
type Versioned     interface{ Plugin; VersionLister; Reader; Extractor }

Extract stores nothing (no baselines); Run adds its own, so a backfill of old pairs never moves the normal baseline.

Shared version sources in core/versions: GitHubReleases (drafts and prereleases out), GitHubTags (by pattern), GitHubCommits (the commits that changed a file, dated), PyPIReleases (final, not yanked), NPMReleases (final, not deprecated), WaybackAt (the Web Archive's capture of a page nearest a date), plus Compare/Final/Major/Pairs. Each takes a getter (env.Get or github.Get) and pages through GitHub lists (max 1,000 entries). Unit tests for each.

Implemented for:

  • openapi (all 15 sources): Run and Replay are now ReadAt + Extract. adapter.Lister (optional) lists every version: Stripe, Datadog, Vercel (canaries out), Flagsmith from GitHub releases; Supabase, Neon, Convex, PlanetScale from the commits that changed the spec. GitHub, GitLab, Elasticsearch, Sentry, Algolia and Appwrite list their newest only, for now.
  • changelog (urllib3, Textual): no pinned tag. File has a TagPrefix; releases come from PyPI, each read at TagPrefix + version, and Extract is that release's section. File.At(release) pins one; Textual0860 / Urllib3200 are now Textual.At("0.86.0") / Urllib3.At("2.0.0"), the goldens' cases. Registered sources are unpinned.
  • migrationguide (Anthropic SDK): no pinned major. Releases from PyPI; only the first release of a major (X.0.0) has a guide to read (MIGRATION.md at vX.0.0), every other pair has no events. sdkguide.Anthropic is unpinned; AnthropicV1 = Anthropic.At(1).
  • removedmembers (NumPy, PostHog, New Relic): Guide.Tag is gone: the tag is derived (TagPrefix + release, v by default, posthog-v for PostHog); New Relic's guide lives in a docs repo with no tags, so it keeps that repo's commit as DocsRef. Releases from PyPI; only the release a guide documents has events.

Living pages (retirements, feed, eol, catalog, apichangelog) keep Run only: patch-it's hosted ingestion service reads them as dated versions of a page.

Golden proof

Every golden is byte-identical (36 files, sha256 compared with main). The pinned cases still produce them through Run, and the new tests check that the pair into the pinned release gives the same fingerprints with previous_version set:

  • openapi: three consecutive pairs, each with only its own change and the right version/previous_version.
  • changelog: urllib3 1.26.15 → 2.0.0 == the 2.0.0 golden's events, previous_version 1.26.15.
  • migrationguide: 0.75.0 → 1.0.0 has the three removed parameters, 1.0.0 → 1.0.1 none; the unpinned Run reproduces the golden byte for byte (CompareGolden).
  • removedmembers: 1.26.4 → 2.0.0 == the NumPy golden's events; 2.0.0 → 2.0.1 none.

Checklist

  • Metadata complete in the registration: the changelog and migration-guide sources' Watches and URLs now describe every release (no pinned URL); I read them
  • Every commit is signed off (DCO)
  • Golden tests: byte-identical
  • testdata/README.md: rows for the three small PyPI-shaped files the listing tests use (written for the test, labelled as such, read by no golden)
  • Drop tests: unchanged
  • providers/textual.md, urllib3.md, anthropic.md updated (no longer "pinned to that release")
  • Fetches only through env.Get / core/kit/ (openapi adapters through github.Get, as before)
  • make check passes locally
  • Nothing private

What patch-it's hosted ingestion service must change later (not in this PR)

  • Nothing to compile: a copy of its code builds and vets against this branch.
  • Behavior on bump: the registered changelog and migration-guide sources are unpinned, so Run reads the newest release from PyPI instead of the pinned one. Expect new events (and, for a release whose changelog has no section under its exact version, a run error naming it) until the worker drives ListVersions → ReadAt → Extract per consecutive pair.
  • To use the new parts: list versions, upsert them, ReadAt each version of a pair it hasn't done, Extract, store events; Run can stay for living pages.
  • The golden-replay test's saved-page paths moved in move shared code to core/ and readers under their kind #183.

Calls I made (please check)

  • Extract for changelog reads only to's section: correct for consecutive pairs (what the worker diffs), not for a jump across several releases.
  • removedmembers keeps Release per guide: each guide documents one release (NumPy's 2.0 guide, PostHog's 7→8 guide); a later major needs its own guide entry (or I19.4's merged kind).
  • Version IDs for openapi are what the spec declares (as events already say), not the tag; the tag is the Ref.

Related PRs

🤖 Generated with Claude Code

A versioned source now has three parts next to Run: ListVersions (every
version, oldest first), ReadAt (the source at one version) and Extract (the
changes between two snapshots, with previous_version set). Run keeps working
and is built from them.

core/versions lists versions from GitHub releases, tags and commits, PyPI,
npm and the Web Archive at a date, with one version ordering.

openapi: every adapter whose LatestTag was releases/latest or the newest
commit on its spec can list all of them (Stripe, Datadog, Vercel, Flagsmith,
Supabase, Neon, Convex, PlanetScale); the rest list their newest.
changelog, migrationguide and removedmembers no longer pin a tag: releases
come from PyPI and each is read at its tag; the pinned releases stay as the
goldens' cases. Every golden is byte-identical.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: adssib <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant