Skip to content

every version for six more spec adapters; living pages by Wayback capture (I19.8, I19.11) - #216

Merged
adssib merged 1 commit into
mainfrom
pl2-versions
Nov 21, 2026
Merged

adssib merged 1 commit into
mainfrom
pl2-versions

Conversation

@adssib

@adssib adssib commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #215

Closes #182 · Part of #179

What this changes

I19.8, the OpenAPI adapters that read only their newest: every adapter is now an adapter.Lister (every version, oldest first):

Adapter Versions
GitHub every supported dated API version (/versions)
GitLab every stable release from v18.10.0-ee (the first with openapi_v3.yaml), paged until the listing passes it; release candidates out; version order
Elasticsearch every released minor from 8.14 (the first with the OpenAPI output) to the newest released one; never the next, unreleased branch
Sentry the archived sentry-api-schema release tags, then getsentry/sentry's api@ releases with the spec attached, by date (IDs are dates, as Fetch's version)
Algolia every release tag; several on one day are ordered by commit date (one lookup per tag on such a day only)
Appwrite every commit that changed specs/

Tested on recorded API responses (recorded 2026-11-20, trimmed; plugins/openapi/adapter/testdata/README.md), no network.

I19.11, Wayback-at-date: versions.WaybackCaptures(page, from, to) lists a living page's captures from the Web Archive's CDX API, one version per distinct content (repeated captures collapse; a change back is a new version; one per day), the ref the capture's original bytes, the URL its citation. The notices kind implements plugin.Versioned with it: ListVersions = the captures, ReadAt = the page as captured, Extract = the notices the newer capture has that the older didn't, citing that capture. versions.GitHubPages is exported for other GitHub lists.

Acceptance criteria

  • I19.11: a fake CDX API where duplicate captures collapse into one version (TestWaybackCapturesCollapseDuplicates)
  • I19.11: a notices source backfills a year from captures alone (TestBackfillAYearFromCaptures: four captures, three versions, every retirement once, each citing the capture where it first appeared)
  • I19.11 polite use: one request (the CDX listing), made through the caller's fetch, which spaces requests and names the client; anchoring to another source's dates is WaybackAt
  • I19.8: every golden byte-identical (no golden changes in this PR)
  • I19.8 "no release tag or version string hardcoded in plugin code (a test checks)": not done; three migration guides still name their release (follow-up)
  • I19.8 "version sources: Go proxy, Maven": not in this PR

Calls I made (please check)

  • Sentry lists release tags only, not the schema repo's commits before its first tag (2020 on): thousands of 10 MB specs; a SHA is still readable.
  • WaybackCaptures keeps the last distinct capture of a day (versions are dated by day).
  • notices events from a capture pair keep the notice's own version (a date) and no previous_version.

Checks

  • make check passes (tidy, build, gofmt, vet, import allowlist, net/http guard, go test -race, no TODO(new-plugin), leak guard); make contract against patch-it's main fixtures passes (11 files).
  • Goldens: no golden changed

Related PRs

🤖 Generated with Claude Code

GitHub, GitLab, Elasticsearch, Sentry, Algolia and Appwrite list every
version of their spec, tested on recorded responses. versions.WaybackCaptures
lists a living page's distinct Web Archive captures (CDX), and the notices
kind reads and extracts them pair by pair, so a page's history can be
backfilled by date.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: adssib <[email protected]>
@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @adssib's task in 2m 8s —— View job


✅ Ready to merge: I found no blocking bugs. Every check I ran passes, and no golden file, registration or contract fixture changed. One thing is worth checking before you run a full Algolia backfill (see the first inline comment).

Rules

Rule Status Evidence
Plugin list (Rule 1) ✅ No RegisterProvider call changed, and no hand-kept list (docs/plugins.md or plugins.json) was added. The new notices ListVersions/ReadAt/Extract and the six adapter.Listers change how existing sources are read, not their metadata.
Provider page ✅ No new provider. No provider page says a source reads only its newest version, so no page is now out of date.
Golden test and quote ✅ No goldens changed. The recorded API responses have a new plugins/openapi/adapter/testdata/README.md: URL, date recorded (2026-11-20), how each was trimmed, and a note on licenses. These are API metadata rather than quoted source text, so a URL and date per file is the right shape.
Drop test ✅ n/a No new plugin. TestBackfillAYearFromCaptures checks that a repeated capture adds no events (duplicate digests collapse into one version, and each retirement is emitted only once).
Fetch only via env ✅ WaybackCaptures and notices.ReadAt use env.Get. The adapters use fetch.Get / github.Get inside plugins/openapi/adapter/, which is allowed. net/http appears only in versions_test.go (an httptest server).
Nothing private (Rule 2) ✅ Nothing private in the code, tests, docs or PR body. leak_guard.py --all is clean.

Bugs

None found that block merging.

One thing to check (unverified, inline on versions.go:211): Algolia.Versions gives ingestion and recommend the same tag list as search, back to 2023-08-22. If those spec files didn't exist yet at the oldest tags, ReadAt will 404 and a full backfill of that source fails. GitLab and Elasticsearch have a first-version floor; Algolia, and possibly Appwrite's oldest commits, don't.

Verified

  • go build ./...: OK
  • go vet ./...: OK
  • gofmt -l .: printed nothing
  • go test -race ./...: all packages pass, including core/versions, plugins/notices and plugins/openapi/adapter
  • python3 scripts/leak_guard.py --all: "no banned terms (8 hashed terms checked)"
  • I couldn't run make check itself: running it needed approval this job doesn't have. The steps above cover most of it; I didn't run make imports, make nethttp or make todos separately.

I read the logic of WaybackCaptures (identical captures collapse, a change back counts as a new version, the last capture of a day wins, output sorted), GitLab's paging stop, the Elasticsearch released-minor bounds, Sentry's two repos, Algolia's commit-date lookup only on days with several tags, and the LatestTag refactors in algolia.go and elasticsearch.go. Behaviour is unchanged: the m == nil case that was removed is now filtered out in algoliaTags.

Non-blocking

  • notices.Extract has no shape guard (inline, notices.go:260). A challenge page or JS-only shell captured as HTTP 200 reads as 0 notices, so the next pair emits every notice again. The fingerprints are the same, so it's only noise for a consumer that dedupes by fingerprint.
  • Elasticsearch versions have no Released date (inline, versions.go:170), so a date-windowed backfill can't place them.
  • Appwrite and Sentry listings are capped silently at MaxPages (1,000 entries) and read newest first, so if there are more, the oldest are dropped without notice. That's fine for now; a log line when the cap is hit would make it visible.
  • Doc comment line length: the package comment in core/versions/versions.go:4 now has one very long line; you could rewrap it.

// Versions are every release tag, oldest first. A day with several releases
// is ordered by each tag's commit date (LatestTag's rule), one lookup per
// tag on such a day only.
func (a Algolia) Versions(ctx context.Context) ([]plugin.Version, error) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check (I couldn't verify this offline): all three Algolia sources (search, ingestion, recommend) get the same tag list, back to released-2023-08-22. If specs/bundled/ingestion.yml (or recommend.yml) wasn't in the repo yet at the oldest tags, Fetch will 404 on them. Then ReadAt fails on the first version and a full backfill of that source stops before it gets anywhere. GitLab and Elasticsearch avoid this with a first-version floor (gitlabFirstSpec, esFirstSpec). Algolia may need the same, set per spec, or a check that drops tags where the file is missing. Appwrite's oldest specs/ commits may hit the same thing if they come from before the specs/<X.Y>.x/ layout (Fetch returns "has no specs/<X.Y>.x directory").

func (p Plugin) events(s plugin.Snapshot, now time.Time) ([]event.EventV2, []error, error) {
found, err := p.Page.Find(s.Content)
if err != nil {
return nil, nil, fmt.Errorf("reading %s: %w", s.SourceURL, err)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: Run uses ShapeGuard to refuse a read with far fewer notices, but Extract doesn't. Archived captures sometimes hold a challenge or JS-shell page (still HTTP 200), so Find reads 0 notices from them. If one of those falls between two good captures, the next pair sees every notice as new and emits all of them again, citing the later capture. The fingerprints are the same, so a consumer that dedupes by fingerprint will be fine. Still, an option is to treat a from whose read fails the shape guard as unreadable: skip the pair, or compare against the last good snapshot.

if slices.Compare(v[:], esFirstSpec[:]) < 0 || slices.Compare(v[:], newest[:]) > 0 {
continue
}
out = append(out, plugin.Version{ID: b + ".0", Ref: b, URL: "https://github.com/elastic/elasticsearch-specification/tree/" + b})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: these versions leave Released empty. That's allowed (plugin.Version says zero means unknown), but a caller that "keeps the ones in its window" by date can't place Elasticsearch minors in it. If the branch-cut or vX.Y.0 release date is cheap to get from es-releases.json (which is already fetched), filling it in would make date-windowed backfills work here too.

Base automatically changed from pl2-notices to main November 21, 2026 00:57
@adssib
adssib merged commit d502df1 into main Nov 21, 2026
11 checks passed
@adssib
adssib deleted the pl2-versions branch November 21, 2026 00:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

I19.11 · Wayback-at-date version source

1 participant