Repository navigation
every version for six more spec adapters; living pages by Wayback capture (I19.8, I19.11) - #216
Conversation
GitHub, GitLab, Elasticsearch, Sentry, Algolia and Appwrite list every version of their spec, tested on recorded responses. versions.WaybackCaptures lists a living page's distinct Web Archive captures (CDX), and the notices kind reads and extracts them pair by pair, so a page's history can be backfilled by date. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: adssib <[email protected]>
|
Claude finished @adssib's task in 2m 8s —— View job ✅ Ready to merge: I found no blocking bugs. Every check I ran passes, and no golden file, registration or contract fixture changed. One thing is worth checking before you run a full Algolia backfill (see the first inline comment). Rules
BugsNone found that block merging. One thing to check (unverified, inline on Verified
I read the logic of Non-blocking
|
| // Versions are every release tag, oldest first. A day with several releases | ||
| // is ordered by each tag's commit date (LatestTag's rule), one lookup per | ||
| // tag on such a day only. | ||
| func (a Algolia) Versions(ctx context.Context) ([]plugin.Version, error) { |
There was a problem hiding this comment.
Please check (I couldn't verify this offline): all three Algolia sources (search, ingestion, recommend) get the same tag list, back to released-2023-08-22. If specs/bundled/ingestion.yml (or recommend.yml) wasn't in the repo yet at the oldest tags, Fetch will 404 on them. Then ReadAt fails on the first version and a full backfill of that source stops before it gets anywhere. GitLab and Elasticsearch avoid this with a first-version floor (gitlabFirstSpec, esFirstSpec). Algolia may need the same, set per spec, or a check that drops tags where the file is missing. Appwrite's oldest specs/ commits may hit the same thing if they come from before the specs/<X.Y>.x/ layout (Fetch returns "has no specs/<X.Y>.x directory").
| func (p Plugin) events(s plugin.Snapshot, now time.Time) ([]event.EventV2, []error, error) { | ||
| found, err := p.Page.Find(s.Content) | ||
| if err != nil { | ||
| return nil, nil, fmt.Errorf("reading %s: %w", s.SourceURL, err) |
There was a problem hiding this comment.
Non-blocking: Run uses ShapeGuard to refuse a read with far fewer notices, but Extract doesn't. Archived captures sometimes hold a challenge or JS-shell page (still HTTP 200), so Find reads 0 notices from them. If one of those falls between two good captures, the next pair sees every notice as new and emits all of them again, citing the later capture. The fingerprints are the same, so a consumer that dedupes by fingerprint will be fine. Still, an option is to treat a from whose read fails the shape guard as unreadable: skip the pair, or compare against the last good snapshot.
| if slices.Compare(v[:], esFirstSpec[:]) < 0 || slices.Compare(v[:], newest[:]) > 0 { | ||
| continue | ||
| } | ||
| out = append(out, plugin.Version{ID: b + ".0", Ref: b, URL: "https://github.com/elastic/elasticsearch-specification/tree/" + b}) |
There was a problem hiding this comment.
Non-blocking: these versions leave Released empty. That's allowed (plugin.Version says zero means unknown), but a caller that "keeps the ones in its window" by date can't place Elasticsearch minors in it. If the branch-cut or vX.Y.0 release date is cheap to get from es-releases.json (which is already fetched), filling it in would make date-windowed backfills work here too.
Stacked on #215
Closes #182 · Part of #179
What this changes
I19.8, the OpenAPI adapters that read only their newest: every adapter is now an
adapter.Lister(every version, oldest first):/versions)openapi_v3.yaml), paged until the listing passes it; release candidates out; version ordersentry-api-schemarelease tags, thengetsentry/sentry'sapi@releases with the spec attached, by date (IDs are dates, asFetch's version)specs/Tested on recorded API responses (recorded 2026-11-20, trimmed;
plugins/openapi/adapter/testdata/README.md), no network.I19.11, Wayback-at-date:
versions.WaybackCaptures(page, from, to)lists a living page's captures from the Web Archive's CDX API, one version per distinct content (repeated captures collapse; a change back is a new version; one per day), the ref the capture's original bytes, the URL its citation. Thenoticeskind implementsplugin.Versionedwith it:ListVersions= the captures,ReadAt= the page as captured,Extract= the notices the newer capture has that the older didn't, citing that capture.versions.GitHubPagesis exported for other GitHub lists.Acceptance criteria
TestWaybackCapturesCollapseDuplicates)TestBackfillAYearFromCaptures: four captures, three versions, every retirement once, each citing the capture where it first appeared)WaybackAtCalls I made (please check)
WaybackCaptureskeeps the last distinct capture of a day (versions are dated by day).noticesevents from a capture pair keep the notice's own version (a date) and noprevious_version.Checks
make checkpasses (tidy, build, gofmt, vet, import allowlist, net/http guard,go test -race, noTODO(new-plugin), leak guard);make contractagainst patch-it'smainfixtures passes (11 files).Related PRs
apps/plugins-repo.md)v0.3.0(see "What the next bump changes" in the combined PR); the contract change this stack prepares for: patch-it-dev/patch-it#58 (adopted in a separate PR after it merges)🤖 Generated with Claude Code