diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 30b6631..5e14f73 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -4,14 +4,14 @@ ## Checklist -- [ ] **Added to / updated in `docs/plugins.md`** (new plugin: a row; changed id, provider, source, what it watches, format or status: the row updated; removed: the row removed). Not applicable: +- [ ] **Metadata complete in the registration** (`plugin.RegisterProvider`: provider id, name, category, homepage; each source's id, layer, what it watches, URLs, page), and I read the words. There is no list to update. Not applicable: - [ ] Every commit is signed off (`git commit -s`, DCO) - [ ] Golden test over a real saved copy of the source (`testdata/golden/.json`) - [ ] `testdata/README.md` lists each file: URL, date fetched, license, quote of the part the plugin reads - [ ] One pinned drop test: a realistic input that produces no event - [ ] `providers/.md` written or updated from `providers/_template.md`, including **Why this source** -- [ ] Fetches only through `env.Get` / `kit/` -- [ ] `make check` passes locally (and `make plugins-json` was run if a plugin or its row changed) +- [ ] Fetches only through `env.Get` / `core/kit/` +- [ ] `make check` passes locally (`make plugins` prints the source as you expect) - [ ] Nothing private: no credentials, internal hosts, private links, customer names ## Notes for the reviewer diff --git a/.github/workflows/CI_contract.yml b/.github/workflows/CI_contract.yml index 7818684..0ddd5cb 100644 --- a/.github/workflows/CI_contract.yml +++ b/.github/workflows/CI_contract.yml @@ -1,5 +1,5 @@ # contract/fixtures/ is a copy of patch-it's fixtures/ (the change-event contract -# patch-it owns). This fails when the two drift apart, so event/ and migrate/ test +# patch-it owns). This fails when the two drift apart, so core/event/ and core/migrate/ test # the same events the patch-it client does. Only top-level "_comment" fields may # differ (scripts/check_contract.py). # diff --git a/.github/workflows/CI_go.yml b/.github/workflows/CI_go.yml index 8f21644..82162c1 100644 --- a/.github/workflows/CI_go.yml +++ b/.github/workflows/CI_go.yml @@ -1,7 +1,7 @@ # The plugins build, are formatted and vetted, and their tests pass with the race # detector. Two guards keep plugins self-contained: # - imports: only this module, the standard library and the approved dependencies; -# - network: net/http only in the fetch layer (fetch/, kit/, adapter/). +# - network: net/http only in the fetch layer (core/fetch/, core/kit/, plugins/openapi/adapter/). # The steps are Makefile targets, so `make check` runs exactly this locally. # Until the Go code lands there is no go.mod, and every step after the first is skipped # with a notice, so the check passes. @@ -80,7 +80,6 @@ jobs: run: make todos # -race: the fetcher and the tests' HTTP servers run on other goroutines. - # Also fails when docs/plugins.json is out of date (make plugins-json). - name: go test if: steps.go.outputs.present == 'true' run: make test diff --git a/.github/workflows/claude-code-review.yml b/.github/workflows/claude-code-review.yml index 2a41f67..cb7b4f9 100644 --- a/.github/workflows/claude-code-review.yml +++ b/.github/workflows/claude-code-review.yml @@ -93,12 +93,12 @@ jobs: any issue it references with `gh issue view `. Read the diff with `gh pr diff `, and changed files in full where the diff isn't enough context. - 2. THE PLUGIN LIST (AGENTS.md Rule 1). Every plugin added, removed, or with a - changed id, provider, source, what it watches, format plugin or status has - its row added, removed or updated in docs/plugins.md, between the - `` and `` markers, as a plain - Markdown table with absolute links only. Compare the registered plugins with - the table yourself; a missing row is blocking. + 2. THE METADATA (AGENTS.md Rule 1). Every source added or changed is + registered with plugin.RegisterProvider with complete, accurate metadata: + the provider's id, name, category and homepage; each source's id, layer, + what it watches, the URLs it reads and its page. There is no hand-kept list + (no docs/plugins.md, no plugins.json in the repo): one reappearing is + blocking. 3. PROVIDER PAGE. Every new provider has providers/.md copied from providers/_template.md, with a real **Why this source**: the link, the date it was checked, why it's the best source, the alternatives not used, and a @@ -110,7 +110,7 @@ jobs: the source URL and a quote. A golden file edited without the PR explaining why the new output is right is blocking. 5. PLUGIN CODE. Fetch only through env.Get or kit/: no net/http outside the - fetch layer (fetch/, kit/, adapter/), no filesystem outside testdata/, no + fetch layer (core/fetch/, core/kit/, plugins/openapi/adapter/), no filesystem outside testdata/, no database. Precision over recall: flag any path that could emit an event for something that isn't a real change; a wrong event becomes a wrong pull request in someone's repo. Don't change contract/fixtures/ here first (the diff --git a/.github/workflows/claude.yml b/.github/workflows/claude.yml index 4fc0a79..3040e8f 100644 --- a/.github/workflows/claude.yml +++ b/.github/workflows/claude.yml @@ -68,4 +68,4 @@ jobs: --permission-mode default --allowedTools "mcp__github_inline_comment__create_inline_comment,Read,Grep,Glob,Bash(gh pr view:*),Bash(gh pr diff:*),Bash(gh issue view:*)" --disallowedTools "Edit,Write,MultiEdit,NotebookEdit,WebFetch,WebSearch,Bash(git push:*),Bash(git commit:*),Bash(git add:*),Bash(git rm:*),mcp__github_file_ops__commit_files,mcp__github_file_ops__delete_files" - --append-system-prompt "This is patch-it-plugins, a public repo of source plugins. Everything you write is public. You can only read and comment; you can't change files or run code, so if asked to, say so and describe the change instead. When asked to review a pull request (for example '@claude review'), check it against AGENTS.md: (1) every new, changed or removed plugin has its row added, updated or removed in docs/plugins.md between the plugins:start and plugins:end markers, absolute links only; (2) every new provider has providers/.md from providers/_template.md with a real 'Why this source' (link, date checked, why it's best, alternatives, quote); (3) a golden test over a real saved source, a testdata/README.md row (URL, date fetched, license, quote), one pinned drop test, every event carries its source URL and a quote, and golden edits are explained; (4) fetch only through env.Get or kit/, no net/http outside fetch/, kit/ and adapter/, no filesystem outside testdata/; (5) precision over recall: flag anything that could emit an event for a non-change; (6) nothing private: no credentials, internal hostnames or service names, links to private repos or tickets, customer names, or links to AI sessions. You can't run the tests; say that CI_go covers them. Start with one verdict line ('✅ Ready to merge', '⚠️ Merge after fixes' or '❌ Not ready'), then a table of rule | status | evidence, blocking bugs, and clearly marked non-blocking suggestions. Be kind and specific: many authors are outside contributors. Never include a Claude session ID or claude.ai link." + --append-system-prompt "This is patch-it-plugins, a public repo of source plugins. Everything you write is public. You can only read and comment; you can't change files or run code, so if asked to, say so and describe the change instead. When asked to review a pull request (for example '@claude review'), check it against AGENTS.md: (1) every new or changed source is registered with plugin.RegisterProvider with complete, accurate metadata (provider id, name, category, homepage; source id, layer, what it watches, URLs, page), and no hand-kept list of plugins comes back; (2) every new provider has providers/.md from providers/_template.md with a real 'Why this source' (link, date checked, why it's best, alternatives, quote); (3) a golden test over a real saved source, a testdata/README.md row (URL, date fetched, license, quote), one pinned drop test, every event carries its source URL and a quote, and golden edits are explained; (4) fetch only through env.Get or core/kit/, no net/http outside core/fetch/, core/kit/ and plugins/openapi/adapter/, no filesystem outside testdata/; (5) precision over recall: flag anything that could emit an event for a non-change; (6) nothing private: no credentials, internal hostnames or service names, links to private repos or tickets, customer names, or links to AI sessions. You can't run the tests; say that CI_go covers them. Start with one verdict line ('✅ Ready to merge', '⚠️ Merge after fixes' or '❌ Not ready'), then a table of rule | status | evidence, blocking bugs, and clearly marked non-blocking suggestions. Be kind and specific: many authors are outside contributors. Never include a Claude session ID or claude.ai link." diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 8e1d14c..5cc4f74 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -172,9 +172,18 @@ jobs: git tag -a "$TAG" "$SHA" -m "$TAG" -m "Released by the Release workflow ($by)." git push origin "refs/tags/$TAG" + # The list of providers and sources, from the registry at the tag: the + # release's plugins.json asset. Never committed; the docs site reads the + # latest release's copy. + - name: list the providers and sources + if: steps.version.outputs.skip != 'true' + env: + TAG: ${{ steps.version.outputs.tag }} + run: go run ./cmd/list-plugins -ref "$TAG" -o "$RUNNER_TEMP/plugins.json" + - name: publish the GitHub release if: steps.version.outputs.skip != 'true' env: GH_TOKEN: ${{ github.token }} TAG: ${{ steps.version.outputs.tag }} - run: gh release create "$TAG" --verify-tag --generate-notes --title "$TAG" + run: gh release create "$TAG" --verify-tag --generate-notes --title "$TAG" "$RUNNER_TEMP/plugins.json" diff --git a/AGENTS.md b/AGENTS.md index 9c6764a..24b5996 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,13 +2,11 @@ Rules for AI coding agents (and humans) working in this repo. `CLAUDE.md` imports this file. -## Rule 1: every new or changed plugin goes in `docs/plugins.md` +## Rule 1: every new or changed provider or source has complete metadata in its registration -**If you add a plugin, add its row to [`docs/plugins.md`](docs/plugins.md) in the same PR. If you change a plugin's id, provider, source, what it watches, format plugin or status, update its row in the same PR. If you remove one, remove its row.** +**A source exists when its kind's package registers it with `plugin.RegisterProvider`, and its metadata lives only there: the provider's id, name, category and homepage, and each source's id, layer, what it watches, the URLs it reads and its page, `providers/.md`.** Adding, changing or removing a source is a change to that registration (and to its page) in the same PR. There is no list to update: nothing in this repo, the docs site or anywhere else keeps one by hand. -The list of supported APIs on docs.patch-it.dev is generated from that file, so a plugin missing from it doesn't exist as far as users can tell. Then run `make plugins-json`: it regenerates [`docs/plugins.json`](docs/plugins.json) from the plugin registry and the table (and the plugin count on the README's badge), and fails if a registered plugin has no row (or a row has no plugin, a wrong format or no provider page). `go test` fails while `docs/plugins.json` is out of date. The check can't tell whether a row's words are right: you are that check. Say in the PR that you compared them. - -Keep the table between the `` and `` markers a plain Markdown table, one row per plugin, absolute links only (the docs site copies it verbatim). +`go run ./cmd/list-plugins` prints the list from the registry (JSON), and the Release workflow attaches it to every release as `plugins.json`; the docs site and patch-it's hosted ingestion service read it from there. `RegisterProvider` refuses incomplete metadata and `go test` fails when a source's page is missing, but no check can tell whether the words are right: you are that check. Say in the PR that you read them. ## Rule 2: this repo is public @@ -26,7 +24,7 @@ Refer to the service that runs these plugins as "patch-it's hosted ingestion ser ## Rules for plugin code - **Start a new provider with `go run ./cmd/new-plugin --api --provider --format `**, then fill in every `TODO(new-plugin)`; `make todos` (part of `make check` and CI_go) fails while one is left. -- **Fetch only through the environment** (`env.Get` or `kit/`). No `net/http` outside the fetch layer, no filesystem outside `testdata/`, no database. +- **Fetch only through the environment** (`env.Get` or `core/kit/`). No `net/http` outside the fetch layer, no filesystem outside `testdata/`, no database. - **Golden test + dated source quote**: a saved copy of the real source in `testdata/`, a row in `testdata/README.md` (URL, date fetched, license, quote), and `testdata/golden/.json` with the exact events. - **Precision over recall.** When unsure whether something is a change, don't emit an event. - **One pinned drop test** per plugin: a realistic input that must produce no event. @@ -41,7 +39,7 @@ Refer to the service that runs these plugins as "patch-it's hosted ingestion ser ```sh make check # tidy, build, gofmt, vet, the import allowlist, the net/http guard, go test -race, no TODO(new-plugin) left, leak guard make check PATCH_IT=../patch-it # also CI_contract: contract/fixtures/ against a patch-it checkout -make plugins-json # regenerate docs/plugins.json after adding or changing a plugin or its row +make plugins # print the list of providers and sources from the registry (what each release attaches as plugins.json) ``` One target at a time: `make tidy`, `make build`, `make fmt`, `make vet`, `make imports`, `make nethttp`, `make test`, `make todos`, `make leak-guard`, `make contract`. Use `make GO=/path/to/go check` for another Go. @@ -51,19 +49,23 @@ One target at a time: `make tidy`, `make build`, `make fmt`, `make vet`, `make i Go module `github.com/patch-it-dev/patch-it-plugins`; the only dependency is `gopkg.in/yaml.v3`. ``` -plugin/ the Plugin interface, the registry, Env, Result, Spec (baselines) -cmd/list-plugins/ writes docs/plugins.json (make plugins-json); internal/pluginlist/ is its logic +core/plugin/ the Plugin interface (Run; ListVersions, ReadAt, Extract), the registry of providers and sources, Env, Result +core/fetch/ the HTTP fetch with retries and the ETag / Last-Modified cache +core/github/ the newest release / commit on GitHub, and a commit's SHA and date (one place) +core/versions/ every version of a source: GitHub releases, tags and commits, PyPI, npm, the Web Archive at a date +core/kit/ core/kit/kittest/ Unchanged, Cite, Wayback; the test kit (Mem, Pages, Ingest, Golden) +core/event/ core/migrate/ the change-event types from patch-it's contract (v1 + v2), v1 -> v2 +core/diff/ core/normalize/ core/hint/ the OpenAPI diff engine and its helpers +plugins// one package per kind, with its providers: openapi (+ adapter/, one spec adapter + per provider), retirements (+ retire/), migrationguide (+ sdkguide/), + removedmembers, changelog, feed, apichangelog, catalog, eol; + plugins/all imports every one +plugin/ event/ fetch/ migrate/ kit/ retire/ sdkguide/ deprecated old import paths: they only + forward to the packages above; don't add to them +cmd/list-plugins/ prints the list from the registry (make plugins); internal/pluginlist/ is its logic cmd/new-plugin/ scaffolds a provider on a format plugin (go run ./cmd/new-plugin --help) -plugins// one package per format plugin: openapi, retirements, migrationguide, - removedmembers, changelog, feed, apichangelog, catalog; plugins/all imports every one -kit/ kit/kittest/ Unchanged, Cite, Wayback; the test kit (Mem, Pages, Ingest, Golden) -event/ migrate/ the change-event types from patch-it's contract (v1 + v2), v1 -> v2 -fetch/ the HTTP fetch with retries and the ETag / Last-Modified cache -diff/ normalize/ hint/ adapter/ the OpenAPI diff engine and the spec adapters -retire/ sdkguide/ deprecations-page and migration-guide readers contract/fixtures/ a copy of patch-it's change-event fixtures (CI_contract checks it) -docs/plugins.md the list of supported plugins (Rule 1); docs/plugins.json is generated from it -Makefile make check (what CI_go runs, plus the leak guard), make plugins-json +Makefile make check (what CI_go runs, plus the leak guard), make plugins docs/ contributor docs, including writing-a-plugin.md providers/.md one page per provider scripts/ leak_guard.py (hashed internal names), check_contract.py @@ -77,7 +79,7 @@ The [change-event contract](https://github.com/patch-it-dev/patch-it/tree/main/d - Sign off every commit: `git commit -s` (DCO; `CI_dco` checks it). - Subjects short, lowercase, imperative: `add the foo plugin`. -- One plugin or fix per PR; fill in every checkbox of the PR template, including "added to docs/plugins.md". +- One plugin or fix per PR; fill in every checkbox of the PR template, including "metadata complete in the registration". - AI-assisted commits keep their co-author trailer. - Never re-run, dispatch or retrigger CI to "see if it passes"; fix the cause and push. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 69d5d46..8554f02 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,12 +2,12 @@ Thanks for helping patch-it watch more APIs. This file is the rulebook; [docs/writing-a-plugin.md](docs/writing-a-plugin.md) is the how-to. -**Start a new provider with the starter:** `go run ./cmd/new-plugin --api --provider --format ` writes the definition, its registration, a test over placeholder sources, the testdata rows, the provider page and the `docs/plugins.md` row; you fill in the `TODO(new-plugin)`s ([how](docs/writing-a-plugin.md#1-scaffold-it-with-new-plugin)). +**Start a new provider with the starter:** `go run ./cmd/new-plugin --api --provider --format ` writes the definition, its registration, a test over placeholder sources, the testdata rows, the provider page, and the source's metadata in its `RegisterProvider` block; you fill in the `TODO(new-plugin)`s ([how](docs/writing-a-plugin.md#1-scaffold-it-with-new-plugin)). ## The rules -1. **Add your plugin to [docs/plugins.md](docs/plugins.md).** Every new plugin gets a row; every change to a plugin's id, provider, source, what it watches or status updates its row. The list on docs.patch-it.dev is built from that file, so a plugin that isn't listed is invisible to users. Then run `make plugins-json` to regenerate [`docs/plugins.json`](docs/plugins.json); it fails if the registry and the table disagree, and the tests fail while the JSON is out of date. -2. **Fetch only through the environment.** Plugins get their sources from `env.Get` or the helpers in `kit/`, never from `net/http` directly, the filesystem or a database. That keeps tests offline and lets the hosted service cache, retry and rate-limit politely. +1. **Give every provider and source complete metadata in its registration.** A source exists when its kind's package registers it with `plugin.RegisterProvider`: the provider's id, name, category and homepage, and the source's id, layer, what it watches, the URLs it reads and its page. That registration is the only list: `make plugins` prints it, each release carries it as `plugins.json`, and docs.patch-it.dev reads it from there. Nothing else to update. +2. **Fetch only through the environment.** Plugins get their sources from `env.Get` or the helpers in `core/kit/`, never from `net/http` directly, the filesystem or a database. That keeps tests offline and lets the hosted service cache, retry and rate-limit politely. 3. **Every plugin has a golden test** over a real saved copy of its source: `testdata/` in, `testdata/golden/.json` out. `testdata/README.md` lists each file with its URL, the date you fetched it, its license, and a short quote of the part your plugin reads. Keep openly licensed files whole; trim vendor documentation pages to what the test needs. 4. **Precision over recall.** A missed change is a bug we can fix later. A wrong change becomes a wrong pull request in someone's repo. When in doubt, don't emit. 5. **One pinned drop test per plugin**: a realistic input your plugin must *not* turn into an event (an unrelated section, a renamed heading, a non-breaking addition). @@ -36,7 +36,7 @@ That adds `Signed-off-by: Your Name `, matching the commit's au | Workflow | What it checks | |---|---| -| `CI_go` | `go mod tidy`, build, vet, gofmt, `go test -race` (including that `docs/plugins.json` is up to date); plugins import only this module, the standard library and approved dependencies; `net/http` only in the fetch layer. Each step is a [Makefile](Makefile) target: `make check` runs them all locally | +| `CI_go` | `go mod tidy`, build, vet, gofmt, `go test -race` (including that every registered source has its metadata and page); plugins import only this module, the standard library and approved dependencies; `net/http` only in the fetch layer. Each step is a [Makefile](Makefile) target: `make check` runs them all locally | | `CI_dco` | Every commit in the PR is signed off by its author | | `CI_leak_guard` | gitleaks over the PR's commits, and a check for known internal names (stored as hashes only). Also runs weekly over the whole repo | | `Claude Code Review` | Claude reviews the PR against the rules above, once (PRs from this repo's branches; forks: a maintainer comments `@claude review`). Advisory, not a required check | @@ -55,7 +55,7 @@ Releases are tags on `main` (`v0.x.y`), made only by the **Release** workflow. N - **patch** (`v0.4.1` → `v0.4.2`): fixes, a new provider on an existing format plugin, docs; - **minor** (`v0.4.1` → `v0.5.0`): a new format plugin, new exported API, anything users should read about. 3. The workflow refuses if `main`'s HEAD is already tagged, then runs, on that exact commit, the internal-names check over every file, gitleaks over the whole history, `go build`, `go vet`, `gofmt` and `go test -race`. Any failure stops it before a tag exists. -4. When everything passes it pushes an annotated tag and publishes a GitHub release with generated notes (grouped by the PR's label: `plugin`/`enhancement`, `bug`, `documentation`; see [`.github/release.yml`](.github/release.yml)). +4. When everything passes it pushes an annotated tag and publishes a GitHub release with generated notes (grouped by the PR's label: `plugin`/`enhancement`, `bug`, `documentation`; see [`.github/release.yml`](.github/release.yml)). The release carries one asset, `plugins.json`: the list of providers and sources printed from the registry at that tag (`go run ./cmd/list-plugins`). It is the only published list; nothing in the repo keeps one, and docs.patch-it.dev reads `https://github.com/patch-it-dev/patch-it-plugins/releases/latest/download/plugins.json`. `v1.0.0` (and any later major) is not a button: it is a deliberate decision a maintainer makes and tags by hand. diff --git a/Makefile b/Makefile index 23ddf94..5aa7bcd 100644 --- a/Makefile +++ b/Makefile @@ -5,7 +5,7 @@ # # make check everything below, in CI's order # make check PATCH_IT=../patch-it also compare contract/fixtures/ with patch-it's -# make plugins-json regenerate docs/plugins.json from the registry and docs/plugins.md +# make plugins print the providers and sources from the registry (JSON) # # The `::error` lines are GitHub annotations; locally they're plain messages. @@ -17,7 +17,7 @@ PYTHON ?= python3 MODULE := github.com/patch-it-dev/patch-it-plugins ALLOWED := ^($(MODULE)(/|$$)|gopkg\.in/yaml\.v3$$) -.PHONY: check tidy build fmt vet imports nethttp test todos leak-guard contract plugins-json +.PHONY: check tidy build fmt vet imports nethttp test todos leak-guard contract plugins check: tidy build fmt vet imports nethttp test todos leak-guard contract @echo "make check: all passed" @@ -52,19 +52,18 @@ imports: nethttp: @bad=$$($(GO) list -f '{{.ImportPath}} {{join .Imports " "}}' ./... \ | grep -E '(^| )net/http( |$$)' | cut -d' ' -f1 \ - | grep -vE '^$(MODULE)/(fetch|kit|adapter)(/|$$)' || true); \ + | grep -vE '^$(MODULE)/(core/fetch|core/kit|plugins/openapi/adapter)(/|$$)' || true); \ if [ -n "$$bad" ]; then \ echo "::error::these packages import net/http; fetch through env.Get or kit instead:"; echo "$$bad"; exit 1; \ fi # -race: the fetcher and the tests' HTTP servers run on other goroutines. -# Includes the check that docs/plugins.json is up to date (internal/pluginlist). test: $(GO) test -race -count=1 ./... # A provider started with new-plugin is done when nothing it wrote still says # TODO(new-plugin). -TODO_DIRS := plugins adapter retire sdkguide migrate providers docs/plugins.md +TODO_DIRS := plugins core/migrate providers todos: @left=$$(grep -rl 'TODO(new-plugin)' $(TODO_DIRS) || true); \ if [ -n "$$left" ]; then \ @@ -87,5 +86,7 @@ contract: echo "contract: skipped (make check PATCH_IT= compares contract/fixtures/)"; \ fi -plugins-json: - $(GO) run ./cmd/list-plugins +# The list of what patch-it watches, from the registry. Never committed: the +# Release workflow attaches it to each release as plugins.json. +plugins: + @$(GO) run ./cmd/list-plugins diff --git a/README.md b/README.md index 99b99a4..ab32e15 100644 --- a/README.md +++ b/README.md @@ -2,133 +2,36 @@ # patch-it-plugins -**How patch-it notices, on its own, that something your code depends on changed.** Each plugin watches one provider's own source (an API spec, a changelog, a shutdown notice, a release feed) and turns what changed into a change event, with the source linked and quoted. - -**Why:** when anything your code depends on changes (APIs, packages, SDKs, runtimes, CI, AI models), [patch-it](https://patch-it.dev) finds the code that breaks and opens the pull request that fixes it. In your CI, on your key. You get the fix, not the alert. These plugins are the trigger: nobody has to assign an alert or open a ticket. +**How patch-it notices, on its own, that something your code depends on changed.** Each provider (Stripe, OpenAI, GitHub…) has sources (a spec, a changelog, a shutdown notice, a feed); a plugin reads every version of a source and turns what changed into a change event, with the source linked and quoted. [![CI](https://github.com/patch-it-dev/patch-it-plugins/actions/workflows/CI_go.yml/badge.svg)](https://github.com/patch-it-dev/patch-it-plugins/actions/workflows/CI_go.yml) [![Go Reference](https://pkg.go.dev/badge/github.com/patch-it-dev/patch-it-plugins.svg)](https://pkg.go.dev/github.com/patch-it-dev/patch-it-plugins) [![Go version](https://img.shields.io/github/go-mod/go-version/patch-it-dev/patch-it-plugins)](go.mod) [![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-3f5bf0)](LICENSE) -[![Plugins](https://img.shields.io/badge/plugins-29-3f5bf0)](docs/plugins.md) [![Docs](https://img.shields.io/badge/docs-docs.patch--it.dev-3f5bf0)](https://docs.patch-it.dev) [![Discussions](https://img.shields.io/github/discussions/patch-it-dev/patch-it-plugins)](https://github.com/patch-it-dev/patch-it-plugins/discussions) -```mermaid -flowchart LR - A["Provider's own source
spec · changelog · shutdown notice · feed"] -->|"read daily"| B["Plugin
(this repo)"] - B -->|"what changed, quoted"| C["Change event
breaking · deprecation · date · source"] - C --> D["patch-it in your CI
finds the lines, an AI model writes the change"] - D --> E["Pull request in your repo
your CI runs your tests, you merge it"] -``` - -## What patch-it does - -1. **It notices on its own.** A plugin here sees the change where the provider announces it, and patch-it picks it up from there: no alert to triage, no ticket to assign. That includes changes that never bump a version, like an AI model or an API shut down on a date. -2. **It writes the fix.** An AI model (on your own key) reads the change and the lines it affects and writes the code change. It arrives as a pull request: your CI runs your tests on it, like any PR, and you review and merge it. The code change, not just the version. -3. **Proof:** 8 pull requests opened by the early version have been merged by outside maintainers. -4. **What it covers:** - - | Layer | Noticed (a plugin here watches it) | Fixed (patch-it writes the PR) | - |---|---|---| - | APIs: Stripe, GitHub, GitLab, Datadog, Vercel, Supabase, Algolia, MongoDB Atlas, Cloudflare | live | live | - | AI models: OpenAI and Anthropic model retirements | live | live | - | AI models: Vertex AI model retirements | live | planned | - | Packages and SDKs: NumPy, Textual, urllib3, the Anthropic Python SDK | live | in testing | - | Runtimes: Redis | live | building | - | CI: GitHub Actions runner retirements | live | planned | - - As of 2026-11-19; every plugin is in [docs/plugins.md](docs/plugins.md). - -5. **Start:** [docs.patch-it.dev](https://docs.patch-it.dev) sets up the GitHub Action in your repo. To have it watch a provider it doesn't cover yet, add a plugin (below). - ## Add a provider in 15 minutes -You need Go (the version in `go.mod`), `make` and Python 3. Pick the format that reads your provider's source ([which one](docs/writing-a-plugin.md#3-pick-the-format-plugin)), then: - -**1. Scaffold it.** - -```sh -URL=https://acme.example/changelog.xml # your provider's source -go run ./cmd/new-plugin --api acme --provider Acme --format feed --source "$URL" -``` -``` -Scaffolded acme (Acme, format feed): - docs/plugins.md - plugins/feed/acme.go (new) - plugins/feed/acme_test.go (new) - ... -Next steps (every place to fill in says TODO(new-plugin): ...) -``` - -**2. Save a real copy of the source and write the golden file.** Fill in the `TODO(new-plugin)`s: the reader in `plugins/feed/acme.go`, the saved file's row in `testdata/README.md`, and "Why this source" in `providers/acme.md`. - -```sh -curl -fsSL -o plugins/feed/testdata/acme.rss.xml "$URL" -go test ./plugins/feed -run TestAcme -update # writes testdata/golden/acme.json: read it like code -``` - -**3. Check it, the way CI will.** - -```sh -make plugins-json && make check -``` -``` -... -make check: all passed -``` - -**4. Open the pull request** (`git commit -s`, one plugin per PR). Details: [docs/writing-a-plugin.md](docs/writing-a-plugin.md). Stuck? Ask in [Discussions](https://github.com/patch-it-dev/patch-it-plugins/discussions). Looking for a first change? [Good first issues](https://github.com/patch-it-dev/patch-it-plugins/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22). - -## How a plugin works - -1. **Fetch.** The plugin asks its environment for the source (`env.Get`); it never opens connections itself. Pages that change in place are also cited through a Web Archive snapshot, so the link in an event keeps showing what the plugin read. -2. **Compare.** It compares the source to the last version it saw (the baseline): two OpenAPI specs, two versions of a deprecations page, a new release's changelog section. -3. **Emit events.** Each change becomes one event in [patch-it's change-event format](https://github.com/patch-it-dev/patch-it/tree/main/docs/contract): what changed, how bad it is (breaking, deprecation, behavior change, non-breaking), the date it takes effect, and the source URL with a quote. **No source, no event.** When unsure, a plugin emits nothing: a wrong event would become a wrong pull request. - -Most providers are an entry for a shared format plugin (`openapi`, `retirements`, `migrationguide`, `removedmembers`, `changelog`, `feed`, `apichangelog`, `catalog`), not new code. patch-it's hosted ingestion service runs every plugin daily and serves the events to the patch-it GitHub Action and CLI. - -## Supported plugins - -**[docs/plugins.md](docs/plugins.md)** lists every plugin: provider, layer, what it watches, its source. The same list is on [docs.patch-it.dev](https://docs.patch-it.dev/supported-apis/plugins/), and as JSON in [docs/plugins.json](docs/plugins.json) (`make plugins-json`). - -## Layout - -| Path | What's there | -|---|---| -| [`plugin/`](plugin/) | The `Plugin` interface, the registry (`Register`, `Lookup`, `Names`), `Env`, `Result` | -| [`plugins/`](plugins/) | The format plugins; `plugins/all` imports every one | -| [`cmd/new-plugin/`](cmd/new-plugin/), [`cmd/list-plugins/`](cmd/list-plugins/) | The starter for a new provider; the generator of `docs/plugins.json` | -| [`kit/`](kit/), [`kit/kittest/`](kit/kittest/) | What plugins share (the unchanged-page check, Wayback citing) and the test kit (in-memory store, saved pages, golden files) | -| [`event/`](event/), [`migrate/`](migrate/) | The change-event types of [patch-it's contract](https://github.com/patch-it-dev/patch-it/tree/main/docs/contract), and the v1 → v2 conversion | -| [`diff/`](diff/), [`normalize/`](normalize/), [`hint/`](hint/), [`adapter/`](adapter/) | The OpenAPI diff engine: compare two specs, turn the changes into events, attach changelog hints; one adapter per spec provider | -| [`retire/`](retire/), [`sdkguide/`](sdkguide/), [`fetch/`](fetch/) | The deprecations-page and migration-guide readers; the HTTP fetch and cache | -| [`contract/fixtures/`](contract/) | A copy of patch-it's change-event fixtures | -| [`providers/`](providers/) | One page per provider: why that source, what the plugin reads | -| [`docs/`](docs/) | [The plugin list](docs/plugins.md), [how to write a plugin](docs/writing-a-plugin.md), the [README template](docs/README-template.md) patch-it's public repos share | - -## Run the checks - -```sh -make check # what CI runs: tidy, build, gofmt, vet, import and net/http guards, go test -race, no TODO(new-plugin) left, leak guard -``` - -The tests are offline: every plugin's test reads a saved copy of its source from `testdata/` and compares the events with a golden file. After an intended change to a plugin's output, `go test ./plugins/ -update` rewrites its golden files; review that diff like code. - -`GITHUB_TOKEN` (optional, no scopes) only raises GitHub's rate limit when the `openapi` adapters look up release tags; the tests never use it. Nothing else is read from the environment. - -## Releases - -Releases are `v0.x.y` tags on `main` with a [GitHub release](https://github.com/patch-it-dev/patch-it-plugins/releases) and generated notes. The **Release** workflow tags a patch release on its own every Monday and Thursday at 14:00 UTC when something new was merged, and maintainers can run it by hand ([how](CONTRIBUTING.md#releasing)). patch-it's hosted ingestion service runs the latest release, so a merged plugin is running within a few days. Tags are never moved or deleted. - -```sh -go get github.com/patch-it-dev/patch-it-plugins@latest -``` - -## Contributing - -New providers and fixes are welcome: [CONTRIBUTING.md](CONTRIBUTING.md) has the rules (DCO sign-off, golden tests, the plugin list), [docs/writing-a-plugin.md](docs/writing-a-plugin.md) the how-to. AI coding agents: [AGENTS.md](AGENTS.md). Questions and ideas: [Discussions](https://github.com/patch-it-dev/patch-it-plugins/discussions). - -## License - -[Apache-2.0](LICENSE), copyright patch-it contributors. +You need Go (the version in `go.mod`), `make` and Python 3. + +1. **Scaffold it**, on the kind that reads its source ([which kind](docs/writing-a-plugin.md#3-pick-the-kind)): + ```sh + go run ./cmd/new-plugin --api acme --provider Acme --format feed --source https://acme.example/changelog.xml + ``` +2. **Fill in every `TODO(new-plugin)`**: the reader, the provider and source metadata in its `plugin.RegisterProvider` block, a saved copy of the source in `testdata/`, and "Why this source" in `providers/acme.md`. Then write the golden file and read it like code: + ```sh + go test ./plugins/feed -run TestAcme -update + ``` +3. **Check it the way CI will, and open the PR** (`git commit -s`, one provider per PR): + ```sh + make check + ``` + +## More + +- [How to write a plugin](docs/writing-a-plugin.md): providers, sources, kinds, versions, the test kit, step by step. +- [How it works](docs/how-it-works.md): what a plugin does, the layout, the checks, releases. +- What's covered: `make plugins` prints every provider and source from the code; each [release](https://github.com/patch-it-dev/patch-it-plugins/releases) carries it as `plugins.json`, and [docs.patch-it.dev](https://docs.patch-it.dev/supported-apis/plugins/) shows it. +- [CONTRIBUTING.md](CONTRIBUTING.md) (the rules), [AGENTS.md](AGENTS.md) (for AI coding agents), [Discussions](https://github.com/patch-it-dev/patch-it-plugins/discussions), [good first issues](https://github.com/patch-it-dev/patch-it-plugins/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22). +- What patch-it does with the events: [patch-it.dev](https://patch-it.dev) and [docs.patch-it.dev](https://docs.patch-it.dev). [Apache-2.0](LICENSE), copyright patch-it contributors. diff --git a/cmd/list-plugins/main.go b/cmd/list-plugins/main.go index 55504d7..0718165 100644 --- a/cmd/list-plugins/main.go +++ b/cmd/list-plugins/main.go @@ -1,42 +1,34 @@ -// Command list-plugins writes docs/plugins.json: every registered plugin -// joined with its row in docs/plugins.md (id, provider, what it watches, -// format plugin, source URL, status, provider page), and sets the plugin -// count on the README's badge. It fails when the -// registry and the table disagree. Run it from the repo root: +// Command list-plugins prints what patch-it watches, as JSON: every registered +// provider with its sources (id, old slug, kind, layer, what it watches, the +// URLs it reads, its page), built from the plugin registry at run time. The +// Release workflow attaches its output to each release as plugins.json; it is +// never committed. // -// go run ./cmd/list-plugins # rewrite docs/plugins.json (make plugins-json) -// go run ./cmd/list-plugins -check # fail if docs/plugins.json is out of date (make check) -// go run ./cmd/list-plugins -o - # print it +// go run ./cmd/list-plugins # print it (make plugins) +// go run ./cmd/list-plugins -ref v0.3.0 -o f.json # link the pages at a tag, write a file package main import ( - "bytes" "flag" "fmt" "os" - "path/filepath" "github.com/patch-it-dev/patch-it-plugins/internal/pluginlist" _ "github.com/patch-it-dev/patch-it-plugins/plugins/all" ) func main() { - root := flag.String("root", ".", "the repo root") - out := flag.String("o", pluginlist.JSON, "where to write the list, relative to -root; - for stdout") - check := flag.Bool("check", false, "don't write: fail if the file differs from what would be written") + ref := flag.String("ref", "main", "the git ref the provider pages are linked at: a release tag, or main") + out := flag.String("o", "-", "where to write the list; - for stdout") flag.Parse() - if err := run(*root, *out, *check); err != nil { + if err := run(*ref, *out); err != nil { fmt.Fprintln(os.Stderr, "list-plugins:", err) os.Exit(1) } } -func run(root, out string, check bool) error { - plugins, err := pluginlist.List(root) - if err != nil { - return err - } - b, err := pluginlist.Marshal(plugins) +func run(ref, out string) error { + b, err := pluginlist.Marshal(pluginlist.Build(ref)) if err != nil { return err } @@ -44,24 +36,5 @@ func run(root, out string, check bool) error { _, err := os.Stdout.Write(b) return err } - readmePath := filepath.Join(root, pluginlist.Readme) - readme, err := os.ReadFile(readmePath) - if err != nil { - return err - } - badged, _ := pluginlist.Badge(readme, len(plugins)) - path := filepath.Join(root, out) - if check { - have, err := os.ReadFile(path) - if err != nil || !bytes.Equal(have, b) || !bytes.Equal(readme, badged) { - return fmt.Errorf("%s or the README's plugins badge is out of date: run `make plugins-json` and commit them", out) - } - return nil - } - if !bytes.Equal(readme, badged) { - if err := os.WriteFile(readmePath, badged, 0o644); err != nil { - return err - } - } - return os.WriteFile(path, b, 0o644) + return os.WriteFile(out, b, 0o644) } diff --git a/cmd/new-plugin/formats.go b/cmd/new-plugin/formats.go index e6a0b75..7c88291 100644 --- a/cmd/new-plugin/formats.go +++ b/cmd/new-plugin/formats.go @@ -4,7 +4,7 @@ package main // definition goes, the line that registers it, the placeholder sources its // test reads, and how the test runs it. type formatSpec struct { - layer string // default layer for docs/plugins.md + layer string // the source's default layer (plugin.Layers()) defDir string // the package directory the definition goes in def string // the definition file (template) register string // the plugin.Register line added to plugins/'s init (template) @@ -30,7 +30,7 @@ const todo = "TODO(new-plugin)" var formats = map[string]formatSpec{ "feed": { - layer: "API", + layer: "api", defDir: "plugins/feed", def: `package feed @@ -76,12 +76,12 @@ func {{.Lower}}Ingest(t *testing.T, file string) []event.EventV2 { return kittest.IngestWith(t, kittest.Env(t, saved), Plugin{Feed: {{.Ident}}, Archiver: archive}) } `, - imports: []string{"github.com/patch-it-dev/patch-it-plugins/event"}, + imports: []string{"github.com/patch-it-dev/patch-it-plugins/core/event"}, }, "retirements": { - layer: "AI models", - defDir: "retire", + layer: "ai-models", + defDir: "plugins/retirements/retire", def: `package retire import "strings" @@ -104,7 +104,7 @@ var {{.Ident}} = Page{ } `, register: `plugin.Register(Plugin{Page: retire.{{.Ident}}, Archiver: kit.Wayback{}})`, - mapEntry: &mapEntry{"migrate/migrate.go", "providers", `retire.{{.Ident}}.API: "{{.Slug}}"`}, + mapEntry: &mapEntry{"core/migrate/migrate.go", "providers", `retire.{{.Ident}}.API: "{{.Slug}}"`}, pluginOf: `Plugin{Page: retire.{{.Ident}}}`, fixtures: map[string]string{"{{.File}}-deprecations.md": `# Placeholder deprecations page @@ -149,12 +149,12 @@ func {{.Lower}}Ingest(t *testing.T, file string) []event.EventV2 { return kittest.IngestWith(t, kittest.Env(t, saved), Plugin{Page: retire.{{.Ident}}, Archiver: archive}) } `, - imports: []string{"encoding/json", "strings", "github.com/patch-it-dev/patch-it-plugins/event", "github.com/patch-it-dev/patch-it-plugins/retire"}, + imports: []string{"encoding/json", "strings", "github.com/patch-it-dev/patch-it-plugins/core/event", "github.com/patch-it-dev/patch-it-plugins/plugins/retirements/retire"}, }, "migrationguide": { - layer: "SDK", - defDir: "sdkguide", + layer: "sdk", + defDir: "plugins/migrationguide/sdkguide", def: `package sdkguide // {{.Ident}} is {{.Provider}}'s SDK guide to a major release (MIGRATION.md at @@ -173,7 +173,7 @@ var {{.Ident}} = Guide{ } `, register: `plugin.Register(Plugin{Guide: sdkguide.{{.Ident}}})`, - mapEntry: &mapEntry{"migrate/migrate.go", "guides", `sdkguide.{{.Ident}}.API: sdkguide.{{.Ident}}`}, + mapEntry: &mapEntry{"core/migrate/migrate.go", "guides", `sdkguide.{{.Ident}}.API: sdkguide.{{.Ident}}`}, pluginOf: `Plugin{Guide: sdkguide.{{.Ident}}}`, fixtures: map[string]string{"{{.File}}-MIGRATION.md": `# Migrating to v1 @@ -215,7 +215,7 @@ func {{.Lower}}Ingest(t *testing.T, file string) []event.EventV2 { return kittest.Ingest(t, Plugin{Guide: sdkguide.{{.Ident}}}, kittest.Pages{sdkguide.{{.Ident}}.URL(): file}) } `, - imports: []string{"encoding/json", "strings", "github.com/patch-it-dev/patch-it-plugins/event", "github.com/patch-it-dev/patch-it-plugins/sdkguide"}, + imports: []string{"encoding/json", "strings", "github.com/patch-it-dev/patch-it-plugins/core/event", "github.com/patch-it-dev/patch-it-plugins/plugins/migrationguide/sdkguide"}, }, "removedmembers": { @@ -290,7 +290,7 @@ func {{.Lower}}Ingest(t *testing.T, file string) []event.EventV2 { return kittest.Ingest(t, Plugin{Guide: {{.Ident}}}, kittest.Pages{ {{.Ident}}.URL(): file}) } `, - imports: []string{"slices", "github.com/patch-it-dev/patch-it-plugins/event"}, + imports: []string{"slices", "github.com/patch-it-dev/patch-it-plugins/core/event"}, }, "changelog": { @@ -298,15 +298,19 @@ func {{.Lower}}Ingest(t *testing.T, file string) []event.EventV2 { defDir: "plugins/changelog", def: `package changelog -// {{.Ident}} is {{.Provider}}'s changelog at one release. +// {{.Ident}} is {{.Provider}}'s changelog, read release by release (its +// releases listed from the ecosystem's registry, each at TagPrefix+version). // -// ` + todo + `: the ecosystem, the import name, the repo, the release's tag, -// the release as the changelog heads it, and the changelog's path. Say what -// you checked, with the date, as changelog.go does for Textual and urllib3. +// ` + todo + `: the ecosystem, the import name, the repo, the tag prefix and +// the changelog's path. Say what you checked, with the date, as changelog.go +// does for Textual and urllib3. var {{.Ident}} = File{ Package: "{{.ID}}", Ecosystem: "pypi", Module: "{{.ID}}", - Repo: "TODO-owner/TODO-repo", Tag: "v1.0.0", Release: "1.0.0", Path: "CHANGELOG.md", + Repo: "TODO-owner/TODO-repo", TagPrefix: "v", Path: "CHANGELOG.md", } + +// {{.Ident}}Golden is the release the golden test pins. +var {{.Ident}}Golden = {{.Ident}}.At("1.0.0") `, register: `plugin.Register(Plugin{File: {{.Ident}}})`, pluginOf: `Plugin{File: {{.Ident}}}`, @@ -324,14 +328,14 @@ var {{.Ident}} = File{ ingest: `// {{.Lower}}Ingest runs the plugin over a saved copy of the changelog. func {{.Lower}}Ingest(t *testing.T, file string) []event.EventV2 { t.Helper() - return kittest.Ingest(t, Plugin{File: {{.Ident}}}, kittest.Pages{ {{.Ident}}.URL(): file}) + return kittest.Ingest(t, Plugin{File: {{.Ident}}Golden}, kittest.Pages{ {{.Ident}}Golden.URL(): file}) } `, - imports: []string{"github.com/patch-it-dev/patch-it-plugins/event"}, + imports: []string{"github.com/patch-it-dev/patch-it-plugins/core/event"}, }, "apichangelog": { - layer: "API", + layer: "api", defDir: "plugins/apichangelog", def: `package apichangelog @@ -364,7 +368,7 @@ func {{.Lower}}Ingest(t *testing.T, file string) []event.EventV2 { }) } `, - imports: []string{"github.com/patch-it-dev/patch-it-plugins/event"}, + imports: []string{"github.com/patch-it-dev/patch-it-plugins/core/event"}, }, "catalog": { @@ -413,12 +417,12 @@ func {{.Lower}}Ingest(t *testing.T, file string) []event.EventV2 { return kittest.Stored(t, sink) } `, - imports: []string{"context", "github.com/patch-it-dev/patch-it-plugins/event"}, + imports: []string{"context", "github.com/patch-it-dev/patch-it-plugins/core/event"}, }, "openapi": { - layer: "API", - defDir: "adapter", + layer: "api", + defDir: "plugins/openapi/adapter", def: `package adapter import ( @@ -440,12 +444,12 @@ func ({{.Ident}}) API() string { return "{{.ID}}" } // LatestTag is the newest release of the spec. func ({{.Ident}}) LatestTag(ctx context.Context) (string, error) { - return "", fmt.Errorf("{{.ID}}: LatestTag isn't written yet (adapter/{{.File}}.go)") + return "", fmt.Errorf("{{.ID}}: LatestTag isn't written yet (plugins/openapi/adapter/{{.File}}.go)") } // Fetch downloads the spec at tag. func ({{.Ident}}) Fetch(ctx context.Context, tag string) (Spec, error) { - return Spec{}, fmt.Errorf("{{.ID}}: Fetch isn't written yet (adapter/{{.File}}.go)") + return Spec{}, fmt.Errorf("{{.ID}}: Fetch isn't written yet (plugins/openapi/adapter/{{.File}}.go)") } `, register: `plugin.Register(Plugin{Adapter: adapter.{{.Ident}}{}})`, @@ -487,8 +491,8 @@ func {{.Lower}}Ingest(t *testing.T, file string) []event.EventV2 { return kittest.Stored(t, sink) } `, - imports: []string{"context", "fmt", "os", "github.com/patch-it-dev/patch-it-plugins/adapter", "github.com/patch-it-dev/patch-it-plugins/event"}, - next: "Write LatestTag and Fetch in adapter/{{.File}}.go (fetch.Get only); add Diff/Keep to the Register line if the spec needs them (see vercel.go).", + imports: []string{"context", "fmt", "os", "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter", "github.com/patch-it-dev/patch-it-plugins/core/event"}, + next: "Write LatestTag and Fetch in plugins/openapi/adapter/{{.File}}.go (fetch.Get only); add Diff/Keep to the Register line if the spec needs them (see vercel.go).", }, } diff --git a/cmd/new-plugin/main.go b/cmd/new-plugin/main.go index 7e4dd1d..12e21bc 100644 --- a/cmd/new-plugin/main.go +++ b/cmd/new-plugin/main.go @@ -5,15 +5,16 @@ // go run ./cmd/new-plugin --api acme --provider Acme --format openapi // // It writes the provider's definition in the format's package (or the reader -// package the format uses: adapter/, retire/, sdkguide/), adds its -// plugin.Register line to plugins/'s init, a test (registered, golden, -// drop) over placeholder sources with an empty golden file, a row in that -// testdata/README.md, providers/.md from providers/_template.md and a -// row in docs/plugins.md, then regenerates docs/plugins.json. `go test ./...` +// package the format uses: plugins/openapi/adapter/, plugins/retirements/retire/, +// plugins/migrationguide/sdkguide/), adds its +// plugin.RegisterProvider block (the source's metadata) to plugins/'s +// init, a test (registered, golden, drop) over placeholder sources with an +// empty golden file, a row in that testdata/README.md, and providers/.md +// from providers/_template.md. `go test ./...` // passes right after; every place to fill in says TODO(new-plugin). // -// It changes nothing when the id is taken (registered, in docs/plugins.md, or -// a file it would write exists). +// It changes nothing when the id is taken (registered, or a file it would +// write exists). package main import ( @@ -35,8 +36,7 @@ import ( "time" "unicode" - "github.com/patch-it-dev/patch-it-plugins/internal/pluginlist" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" _ "github.com/patch-it-dev/patch-it-plugins/plugins/all" ) @@ -46,13 +46,13 @@ type Options struct { API string // the plugin id: --api and every event's api Provider string // whose API, SDK or package Format string // the format plugin - Layer string // the docs/plugins.md layer; "" is the format's default + Layer string // the source's layer (plugin.Layers()); "" is the format's default Source string // the provider's source URL; "" is a placeholder - Watches string // the docs/plugins.md "what it watches" cell + Watches string // what the source watches, in a few words Page string // the provider page, providers/.md; "" is the id. An existing page is shared, not rewritten Today time.Time // Go runs the go command afterwards: the golden file from the placeholders - // (go test -update) and docs/plugins.json (go run ./cmd/list-plugins). + // (go test -update). Go bool } @@ -62,7 +62,7 @@ func main() { flag.StringVar(&o.API, "api", "", "the plugin id, e.g. acme or acme-models (lowercase letters, digits, hyphens)") flag.StringVar(&o.Provider, "provider", "", "the provider's name, e.g. Acme") flag.StringVar(&o.Format, "format", "", "the format plugin: "+strings.Join(formatNames(), ", ")) - flag.StringVar(&o.Layer, "layer", "", "the layer in docs/plugins.md: API, SDK, package, runtime, CI or \"AI models\" (default: the format's)") + flag.StringVar(&o.Layer, "layer", "", "the source's layer: api, sdk, package, runtime, ci, ai-models or infra (default: the format's)") flag.StringVar(&o.Source, "source", "", "the provider's source URL (default: a placeholder to fill in)") flag.StringVar(&o.Watches, "watches", "", "what it watches, in a few words (default: a placeholder)") flag.StringVar(&o.Page, "page", "", "the provider page, providers/.md (default: the --api id); one page per provider, so a second plugin of a provider names the existing page") @@ -115,8 +115,8 @@ func Run(o Options, out io.Writer) error { if o.Layer == "" { o.Layer = f.layer } - if _, ok := pluginlist.Layers[o.Layer]; !ok { - return fmt.Errorf("--layer %q: want API, SDK, package, runtime, CI or \"AI models\"", o.Layer) + if !slices.Contains(plugin.Layers(), plugin.Layer(o.Layer)) { + return fmt.Errorf("--layer %q: want one of %v", o.Layer, plugin.Layers()) } if o.Source == "" { o.Source = "https://example.com/TODO-" + o.API + "-source" @@ -142,20 +142,6 @@ func Run(o Options, out io.Writer) error { if _, ok := plugin.Lookup(o.API); ok { return fmt.Errorf("a plugin named %q is already registered: pick another --api", o.API) } - tablePath := filepath.Join(o.Root, pluginlist.Table) - table, err := os.ReadFile(tablePath) - if err != nil { - return fmt.Errorf("%v (run from the repo root, or pass --root)", err) - } - rows, err := pluginlist.ReadTable(table) - if err != nil { - return err - } - for _, r := range rows { - if r.ID == o.API { - return fmt.Errorf("%s already has a row for %q: pick another --api", pluginlist.Table, o.API) - } - } d := data{ID: o.API, Provider: o.Provider, Source: o.Source, Format: o.Format} d.Ident = ident(o.API) @@ -167,7 +153,7 @@ func Run(o Options, out io.Writer) error { d.PluginOf = fill(f.pluginOf, d) d.Ingest = fill(f.ingest, d) d.Drop = fill(f.drop, d) - d.Imports = append(slices.Clone(f.imports), "testing", "github.com/patch-it-dev/patch-it-plugins/kit/kittest") + d.Imports = append(slices.Clone(f.imports), "testing", "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest") slices.Sort(d.Imports) pkgDir := "plugins/" + o.Format @@ -216,7 +202,7 @@ func Run(o Options, out io.Writer) error { } // The edits to existing files. - mainFile, registered, err := register(o.Root, pkgDir, fill(f.register, d)) + mainFile, registered, err := register(o.Root, pkgDir, providerBlock(o, d, fill(f.register, d))) if err != nil { return err } @@ -234,9 +220,6 @@ func Run(o Options, out io.Writer) error { if edits[readmePath], err = readmeRows(readme, []string{d.SourceFile, d.DropFile}, o.Source); err != nil { return fmt.Errorf("%s: %w", readmePath, err) } - if edits[pluginlist.Table], err = tableRow(table, o, d); err != nil { - return err - } // Write. var paths []string @@ -264,7 +247,6 @@ func Run(o Options, out io.Writer) error { if o.Go { for _, args := range [][]string{ {"test", "./" + pkgDir, "-run", "^Test" + d.Ident + "Golden$", "-update"}, - {"run", "./cmd/list-plugins"}, } { cmd := exec.Command("go", args...) cmd.Dir = o.Root @@ -272,7 +254,7 @@ func Run(o Options, out io.Writer) error { return fmt.Errorf("the files are written, but `go %s` failed: %w\n%s", strings.Join(args, " "), err, b) } } - fmt.Fprintf(out, " %s (from the placeholders)\n docs/plugins.json (regenerated)\n", filepath.Join(pkgDir, "testdata", "golden", o.API+".json")) + fmt.Fprintf(out, " %s (from the placeholders)\n", filepath.Join(pkgDir, "testdata", "golden", o.API+".json")) } fmt.Fprint(out, fill(nextSteps, struct { data @@ -292,8 +274,9 @@ Next steps (every place to fill in says TODO(new-plugin): grep -rn 'TODO(new-plu {{.Extra}}{{end}} 3. Write the golden file, then read it like code: go test ./plugins/{{.Format}} -run Test{{.Ident}} -update - 4. {{if .Shared}}Add {{.ID}} to providers/{{.Page}}.md (shared with the provider's other plugins){{else}}Fill in providers/{{.Page}}.md{{end}}, above all "Why this source", and the row in docs/plugins.md - (what it watches, the source; a plugin in the code is live), then: make plugins-json + 4. {{if .Shared}}Add {{.ID}} to providers/{{.Page}}.md (shared with the provider's other plugins){{else}}Fill in providers/{{.Page}}.md{{end}}, above all "Why this source", and the source's + metadata in its RegisterProvider block (what it watches, its URLs; a new provider's category and + homepage). go run ./cmd/list-plugins prints the list with it; there is no list to edit. 5. make check, then open the PR (git commit -s). ` @@ -377,6 +360,32 @@ func declNames(decl ast.Decl) []string { return nil } +// providerBlock turns a format's "plugin.Register()" line into the +// provider's registration with its metadata: a provider already registered +// (a second source of Algolia) keeps its id, category and homepage; a new one +// gets placeholders that say TODO(new-plugin). +func providerBlock(o Options, d data, line string) string { + expr := strings.TrimSuffix(strings.TrimPrefix(line, "plugin.Register("), ")") + id, category, homepage := d.Slug, "plugin.CategoryDevTools, // TODO(new-plugin): the provider's category, one of plugin.Categories()\n\t\t", `"https://example.com/TODO-`+d.Slug+`-homepage", // TODO(new-plugin): the provider's own site`+"\n\t\t" + for _, p := range plugin.Providers() { + if strings.EqualFold(p.Name, o.Provider) { + id, category, homepage = p.ID, fmt.Sprintf("%q,", p.Category), fmt.Sprintf("%q,", p.Homepage) + category = "plugin.Category(" + strings.TrimSuffix(category, ",") + ")," + } + } + source := strings.TrimPrefix(o.API, id+"-") + return fmt.Sprintf(`plugin.RegisterProvider(plugin.Provider{ + ID: %q, Name: %q, + Category: %s Homepage: %s + Sources: []plugin.Source{{ + ID: %q, Layer: %q, Page: %q, + Watches: %q, + URLs: []string{%q}, + Plugin: %s, + }}, + })`, id, o.Provider, category, homepage, source, o.Layer, "providers/"+o.Page+".md", o.Watches, o.Source, expr) +} + // register adds line at the end of the init func that registers pkgDir's // plugins, next to the others. func register(root, pkgDir, line string) (string, []byte, error) { @@ -399,7 +408,7 @@ func register(root, pkgDir, line string) (string, []byte, error) { } for _, decl := range file.Decls { fn, ok := decl.(*ast.FuncDecl) - if !ok || fn.Name.Name != "init" || fn.Recv != nil || !bytes.Contains(src[fn.Pos()-1:fn.End()-1], []byte("plugin.Register(")) { + if !ok || fn.Name.Name != "init" || fn.Recv != nil || !bytes.Contains(src[fn.Pos()-1:fn.End()-1], []byte("plugin.Register")) { continue } at := fset.Position(fn.Body.Rbrace).Offset @@ -415,7 +424,7 @@ func register(root, pkgDir, line string) (string, []byte, error) { return rel, out, nil } } - return "", nil, fmt.Errorf("found no init in %s that calls plugin.Register", pkgDir) + return "", nil, fmt.Errorf("found no init in %s that calls plugin.RegisterProvider", pkgDir) } // addMapEntry adds entry to the map literal varName is set to in path. @@ -502,21 +511,6 @@ func readmeRows(readme []byte, files []string, source string) ([]byte, error) { return []byte(strings.Join(lines, "\n")), nil } -// tableRow adds the plugin's row at the end of docs/plugins.md's table. -func tableRow(table []byte, o Options, d data) ([]byte, error) { - end := bytes.Index(table, []byte(pluginlist.EndMarker)) - if end < 0 { - return nil, fmt.Errorf("%s: no %s", pluginlist.Table, pluginlist.EndMarker) - } - row := fmt.Sprintf("| [`%s`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/%s.md) | %s | %s | %s | `%s` | [%s](%s) | live |\n", - o.API, o.Page, o.Provider, o.Layer, o.Watches, o.Format, linkText(o.Source), o.Source) - var b bytes.Buffer - b.Write(table[:end]) - b.WriteString(row) - b.Write(table[end:]) - return b.Bytes(), nil -} - func linkText(url string) string { if strings.Contains(url, "example.com/TODO") { return "TODO: the source" diff --git a/cmd/new-plugin/main_test.go b/cmd/new-plugin/main_test.go index 4fbbecd..3092ef5 100644 --- a/cmd/new-plugin/main_test.go +++ b/cmd/new-plugin/main_test.go @@ -17,7 +17,7 @@ import ( var today = time.Date(2026, 11, 19, 0, 0, 0, 0, time.UTC) // A plugin of every format, scaffolded into a copy of the repo: the copy -// builds, vets and passes its tests, and docs/plugins.json lists them all. +// builds, vets and passes its tests, and list-plugins lists them all. // Then the same command again changes nothing. func TestScaffoldEveryFormat(t *testing.T) { if testing.Short() { @@ -38,26 +38,28 @@ func TestScaffoldEveryFormat(t *testing.T) { t.Errorf("%s: no next steps in\n%s", f, out.String()) } } - for _, args := range [][]string{{"build", "./..."}, {"vet", "./..."}, {"test", "-count=1", "./plugins/...", "./internal/...", "./migrate/..."}} { + for _, args := range [][]string{{"build", "./..."}, {"vet", "./..."}, {"test", "-count=1", "./plugins/...", "./internal/...", "./core/migrate/..."}} { cmd := exec.Command(goCmd, args...) cmd.Dir = root if b, err := cmd.CombinedOutput(); err != nil { t.Fatalf("go %s: %v\n%s", strings.Join(args, " "), err, b) } } - listed, err := os.ReadFile(filepath.Join(root, "docs", "plugins.json")) + list := exec.Command(goCmd, "run", "./cmd/list-plugins") + list.Dir = root + listed, err := list.Output() if err != nil { - t.Fatal(err) + t.Fatalf("go run ./cmd/list-plugins: %v", err) } for _, f := range formatNames() { - if !bytes.Contains(listed, []byte(`"id": "acme-`+f+`"`)) { - t.Errorf("docs/plugins.json doesn't list acme-%s", f) + if !bytes.Contains(listed, []byte(`"alias": "acme-`+f+`"`)) { + t.Errorf("list-plugins doesn't list acme-%s", f) } } before := tree(t, root) err = Run(Options{Root: root, API: "acme-feed", Provider: "Acme Corp", Format: "feed", Today: today, Go: true}, io.Discard) - if err == nil || !strings.Contains(err.Error(), "already has a row") { + if err == nil || !strings.Contains(err.Error(), "already exists") { t.Errorf("second run: err = %v, want the id taken", err) } if after := tree(t, root); !slices.Equal(before, after) { @@ -65,8 +67,8 @@ func TestScaffoldEveryFormat(t *testing.T) { } } -// A second plugin of a provider shares its page: no new page, and the row -// links the existing one. +// A second source of a provider shares its page and its provider: no new +// page, and the registration names the existing ones. func TestScaffoldSharesTheProviderPage(t *testing.T) { root := copyRepo(t) o := Options{Root: root, API: "algolia-analytics", Provider: "Algolia", Format: "openapi", Page: "algolia", Today: today} @@ -76,9 +78,11 @@ func TestScaffoldSharesTheProviderPage(t *testing.T) { if _, err := os.Stat(filepath.Join(root, "providers", "algolia-analytics.md")); err == nil { t.Error("wrote providers/algolia-analytics.md; want the shared providers/algolia.md") } - table, _ := os.ReadFile(filepath.Join(root, "docs", "plugins.md")) - if !bytes.Contains(table, []byte("[`algolia-analytics`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/algolia.md)")) { - t.Error("the row doesn't link providers/algolia.md") + reg, _ := os.ReadFile(filepath.Join(root, "plugins", "openapi", "openapi.go")) + for _, want := range []string{`ID: "algolia", Name: "Algolia"`, `ID: "analytics"`, `Page: "providers/algolia.md"`, `plugin.Category("search")`} { + if !bytes.Contains(reg, []byte(want)) { + t.Errorf("the registration has no %s", want) + } } } diff --git a/diff/diff.go b/core/diff/diff.go similarity index 99% rename from diff/diff.go rename to core/diff/diff.go index c012569..e675458 100644 --- a/diff/diff.go +++ b/core/diff/diff.go @@ -19,7 +19,7 @@ import ( "sort" "strings" - "github.com/patch-it-dev/patch-it-plugins/event" + "github.com/patch-it-dev/patch-it-plugins/core/event" ) // Kind is the shape of one structural difference. It is deliberately finer diff --git a/diff/diff_test.go b/core/diff/diff_test.go similarity index 100% rename from diff/diff_test.go rename to core/diff/diff_test.go diff --git a/diff/options_test.go b/core/diff/options_test.go similarity index 100% rename from diff/options_test.go rename to core/diff/options_test.go diff --git a/core/event/event.go b/core/event/event.go new file mode 100644 index 0000000..d17f1c9 --- /dev/null +++ b/core/event/event.go @@ -0,0 +1,215 @@ +package event + +import ( + "crypto/sha256" + "encoding/hex" + "fmt" + "regexp" + "strings" + "time" + "unicode/utf8" +) + +type ( + ChangeType string + AffectedKind string + Side string +) + +const ( + ChangeTypeBreaking ChangeType = "breaking" + ChangeTypeDeprecation ChangeType = "deprecation" + ChangeTypeNewFeature ChangeType = "new_feature" + ChangeTypeFix ChangeType = "fix" +) + +const ( + AffectedKindOperation AffectedKind = "operation" + AffectedKindRequestParameter AffectedKind = "request_parameter" + AffectedKindSchema AffectedKind = "schema" + AffectedKindSchemaProperty AffectedKind = "schema_property" +) + +const ( + SideRequest Side = "request" + SideResponse Side = "response" +) + +type Affected struct { + Kind AffectedKind `json:"kind"` // kind of change + Side Side `json:"side"` // which side is breaking + Operation string `json:"operation,omitempty"` // if operation change, what changed + Schema string `json:"schema,omitempty"` // if schema change, what changed + Property string `json:"property,omitempty"` // if poperty change, what changed + // Value is the one value of Property the change is about, when it's about one: + // the retired model name for a model retirement ("gpt-4"). Optional. + Value string `json:"value,omitempty"` +} + +type ChangeEvent struct { + ID int64 `json:"id"` + Fingerprint string `json:"fingerprint"` + API string `json:"api"` + Version string `json:"version"` + PreviousVersion string `json:"previous_version,omitempty"` + ChangeType ChangeType `json:"change_type"` + Breaking bool `json:"breaking"` + Affected Affected `json:"affected"` + Description string `json:"description"` + MigrationHint string `json:"migration_hint,omitempty"` + SourceURL string `json:"source_url"` + SourcePointer string `json:"source_pointer,omitempty"` + SourceExcerpt string `json:"source_excerpt"` + Confidence float64 `json:"confidence"` + DetectedAt time.Time `json:"detected_at"` +} + +// Fingerprint identifies one change at one location, so re-running ingest over +// the same specs updates nothing instead of duplicating events. detail is what +// separates two changes at the same place: the enum value added, the old and +// new type, "added as required". Absent fields are empty strings, and the parts +// are joined with a separator that cannot appear in a path or a schema name. +func Fingerprint(api, version string, changeType ChangeType, affected Affected, detail string) string { + sum := sha256.Sum256([]byte(strings.Join([]string{ + api, + version, + string(changeType), + string(affected.Kind), + string(affected.Side), + affected.Operation, + affected.Schema, + affected.Property, + detail, + }, "|"))) + return "sha256:" + hex.EncodeToString(sum[:]) +} + +// fingerprintPattern is the contract's shape for a fingerprint (the schema's +// pattern in the change event contract). patch-it checks the same pattern. +var fingerprintPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) + +func (e ChangeEvent) Validate() error { + if err := checkCommon(e.SourceURL, e.SourceExcerpt, e.Fingerprint, e.API, e.Version, + e.Description, e.DetectedAt, e.ChangeType, e.Breaking, e.Confidence); err != nil { + return err + } + + if err := e.Affected.validate(); err != nil { + return err + } + + // Rule 5 (continued): a value is a value of a property. + if e.Affected.Value != "" && e.Affected.Property == "" { + return fmt.Errorf("affected.value needs affected.property: it is a value of that property") + } + + return nil +} + +// checkCommon is the contract's rules for every field outside affected (v1) or +// subject (v2): the two versions differ only in what changed, not in the rest. +func checkCommon(sourceURL, excerpt, fingerprint, api, version, description string, + detectedAt time.Time, changeType ChangeType, breaking bool, confidence float64) error { + // Rule 1: provenance is mandatory — the anti-hallucination mechanism. + if sourceURL == "" { + return fmt.Errorf("source_url is required") + } + if excerpt == "" { + return fmt.Errorf("source_excerpt is required") + } + + // Rule 2: other required fields must be non-empty. + if fingerprint == "" { + return fmt.Errorf("fingerprint is required") + } + if !fingerprintPattern.MatchString(fingerprint) { + return fmt.Errorf("fingerprint must look like sha256:<64 lowercase hex>, got %q", fingerprint) + } + if api == "" { + return fmt.Errorf("api is required") + } + if version == "" { + return fmt.Errorf("version is required") + } + if description == "" { + return fmt.Errorf("description is required") + } + if detectedAt.IsZero() { + return fmt.Errorf("detected_at is required") + } + + // Rule 3: change_type must be one of the known enum values. + if changeType != ChangeTypeBreaking && + changeType != ChangeTypeDeprecation && + changeType != ChangeTypeNewFeature && + changeType != ChangeTypeFix { + return fmt.Errorf("change_type must be a valid enum value, got %q", changeType) + } + + // Rule 4: breaking must agree with change_type. + if breaking != (changeType == ChangeTypeBreaking) { + return fmt.Errorf("breaking (%v) must equal change_type == breaking", breaking) + } + + // Rule 6: confidence must be in [0, 1]. + if confidence < 0 || confidence > 1 { + return fmt.Errorf("confidence must be between 0 and 1, got %v", confidence) + } + + // Rule 7: source_excerpt is capped at 2000 characters. Count runes, not + // bytes: spec text is full of non-ASCII (Stripe writes "invoice\u2019s"), and + // len() would reject excerpts well under the documented limit. + if n := utf8.RuneCountInString(excerpt); n > 2000 { + return fmt.Errorf("source_excerpt must be at most 2000 characters, got %d", n) + } + return nil +} + +// validate is the contract's rules for an API location: v1's affected, and v2's +// api_operation subject. Kind decides the side and the required fields. +func (a Affected) validate() error { + // Rule 3 (continued): affected.kind and affected.side must be valid too. + if a.Kind != AffectedKindOperation && + a.Kind != AffectedKindRequestParameter && + a.Kind != AffectedKindSchema && + a.Kind != AffectedKindSchemaProperty { + return fmt.Errorf("affected.kind must be a valid enum value, got %q", a.Kind) + } + if a.Side != SideRequest && a.Side != SideResponse { + return fmt.Errorf("affected.side must be a valid enum value, got %q", a.Side) + } + + // Rule 5: affected's required sub-fields, and its side, depend on its kind. + // Kind decides side: requests live under paths, responses under schemas. + switch a.Kind { + case AffectedKindOperation: + if a.Operation == "" { + return fmt.Errorf("affected.operation is required when kind is operation") + } + if a.Side != SideRequest { + return fmt.Errorf("affected.side must be %q when kind is operation, got %q", SideRequest, a.Side) + } + case AffectedKindRequestParameter: + if a.Operation == "" || a.Property == "" { + return fmt.Errorf("affected.operation and affected.property are required when kind is request_parameter") + } + if a.Side != SideRequest { + return fmt.Errorf("affected.side must be %q when kind is request_parameter, got %q", SideRequest, a.Side) + } + case AffectedKindSchema: + if a.Schema == "" { + return fmt.Errorf("affected.schema is required when kind is schema") + } + if a.Side != SideResponse { + return fmt.Errorf("affected.side must be %q when kind is schema, got %q", SideResponse, a.Side) + } + case AffectedKindSchemaProperty: + if a.Schema == "" || a.Property == "" { + return fmt.Errorf("affected.schema and affected.property are required when kind is schema_property") + } + if a.Side != SideResponse { + return fmt.Errorf("affected.side must be %q when kind is schema_property, got %q", SideResponse, a.Side) + } + } + return nil +} diff --git a/event/event_test.go b/core/event/event_test.go similarity index 97% rename from event/event_test.go rename to core/event/event_test.go index e492dc5..6f0b3f6 100644 --- a/event/event_test.go +++ b/core/event/event_test.go @@ -176,7 +176,7 @@ func TestAValueNeedsTheProperty(t *testing.T) { // contract, one of the two tests fails. Reading a shared fixture is not an // import: the two still share only the contract. func TestSharedFixtureMatchesTheContract(t *testing.T) { - raw, err := os.ReadFile("../contract/fixtures/change-events.json") + raw, err := os.ReadFile("../../contract/fixtures/change-events.json") if err != nil { t.Fatalf("reading the shared fixture: %v", err) } @@ -199,7 +199,7 @@ func TestSharedFixtureMatchesTheContract(t *testing.T) { // The model retirements fixture is the hosted service's real output, the first to // carry affected.value. Ingest and the client both check it. func TestRetirementFixtureMatchesTheContract(t *testing.T) { - raw, err := os.ReadFile("../contract/fixtures/change-events-model-retirements-2026-10-13.json") + raw, err := os.ReadFile("../../contract/fixtures/change-events-model-retirements-2026-10-13.json") if err != nil { t.Fatalf("reading the fixture: %v", err) } @@ -222,7 +222,7 @@ func TestRetirementFixtureMatchesTheContract(t *testing.T) { // The negative half of the contract test: every event in the shared invalid // fixture breaks one rule, and both loops must reject all of them. func TestSharedInvalidFixtureIsRejected(t *testing.T) { - raw, err := os.ReadFile("../contract/fixtures/change-events-invalid.json") + raw, err := os.ReadFile("../../contract/fixtures/change-events-invalid.json") if err != nil { t.Fatalf("reading the shared invalid fixture: %v", err) } diff --git a/event/v2.go b/core/event/v2.go similarity index 100% rename from event/v2.go rename to core/event/v2.go diff --git a/event/v2_test.go b/core/event/v2_test.go similarity index 99% rename from event/v2_test.go rename to core/event/v2_test.go index 5b1c58c..0d2dab3 100644 --- a/event/v2_test.go +++ b/core/event/v2_test.go @@ -12,7 +12,7 @@ import ( // go test ./event -update var update = flag.Bool("update", false, "rewrite fingerprints in change-events-v2-types.json") -const v2Fixtures = "../contract/fixtures/" +const v2Fixtures = "../../contract/fixtures/" func rawEvents(t *testing.T, name string) []json.RawMessage { t.Helper() diff --git a/fetch/cache.go b/core/fetch/cache.go similarity index 100% rename from fetch/cache.go rename to core/fetch/cache.go diff --git a/fetch/cache_test.go b/core/fetch/cache_test.go similarity index 100% rename from fetch/cache_test.go rename to core/fetch/cache_test.go diff --git a/core/fetch/fetch.go b/core/fetch/fetch.go new file mode 100644 index 0000000..8cb228c --- /dev/null +++ b/core/fetch/fetch.go @@ -0,0 +1,135 @@ +// Package fetch does plain HTTP GETs with retries and a timeout. It knows +// nothing about specs, Stripe, or any other caller — just how to reliably +// download bytes from a URL. +package fetch + +import ( + "context" + "errors" + "fmt" + "io" + "net/http" + "time" +) + +const ( + maxAttempts = 3 + attemptTimeout = 30 * time.Second +) + +// baseBackoff is a variable only so tests can shorten it. +var baseBackoff = 1 * time.Second + +// StatusError is returned when a request completes but with a non-200 status, +// so callers can check the exact code (e.g. a 404 meaning "try elsewhere") +// instead of matching on an error string. +type StatusError struct { + URL string + StatusCode int +} + +func (e *StatusError) Error() string { + return fmt.Sprintf("unexpected status %d from %s", e.StatusCode, e.URL) +} + +// requestError marks a failure to even build the HTTP request (e.g. a +// malformed URL). It's never transient, so Get fails fast on it instead of +// retrying like it does for network errors and 5xx responses. +type requestError struct{ err error } + +func (e *requestError) Error() string { return e.err.Error() } +func (e *requestError) Unwrap() error { return e.err } + +// Get downloads url, retrying on network errors and 5xx responses up to +// maxAttempts times with exponential backoff (1s, 2s, 4s...). A 4xx response +// means the request itself is wrong, so it fails immediately instead of +// retrying. headers are set on every attempt; nil is fine for none. +func Get(ctx context.Context, url string, headers map[string]string) ([]byte, error) { + r, err := get(ctx, url, headers) + if err == nil && r.status == http.StatusNotModified { + // Get sends no validators of its own, so a 304 is the server's error, + // never an empty page. + return nil, &StatusError{URL: url, StatusCode: r.status} + } + return r.body, err +} + +// response is one successful attempt: 200 with its body, or 304 Not Modified +// to a conditional request (no body). +type response struct { + status int + body []byte + header http.Header +} + +// get is Get's retry loop, returning the whole response. +func get(ctx context.Context, url string, headers map[string]string) (response, error) { + var lastErr error + + for attempt := 1; attempt <= maxAttempts; attempt++ { + r, err := attemptGet(ctx, url, headers) + status := r.status + if err == nil { + return r, nil + } + lastErr = err + + var reqErr *requestError + if errors.As(err, &reqErr) { + return response{}, err // a malformed request won't build any differently on retry + } + + if status >= 400 && status < 500 { + return response{}, err // retrying a bad request never helps + } + + if attempt < maxAttempts { + backoff := baseBackoff * time.Duration(1<<(attempt-1)) + select { + case <-time.After(backoff): + case <-ctx.Done(): + return response{}, ctx.Err() + } + } + } + + return response{}, fmt.Errorf("fetching %s failed after %d attempts: %w", url, maxAttempts, lastErr) +} + +// attemptGet makes one attempt, bounded by its own timeout so a single hung +// request can't block forever. status is 0 when the failure happened before +// a response arrived (a bad request, a network error). A 304 is a success: it +// only comes back to a request that asked for it (If-None-Match, +// If-Modified-Since). +func attemptGet(ctx context.Context, url string, headers map[string]string) (response, error) { + ctx, cancel := context.WithTimeout(ctx, attemptTimeout) + defer cancel() + + req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) + if err != nil { + return response{}, &requestError{err: fmt.Errorf("could not build the request: %w", err)} + } + for key, value := range headers { + req.Header.Set(key, value) + } + + resp, err := http.DefaultClient.Do(req) + if err != nil { + return response{}, fmt.Errorf("could not retrieve the content from given url: %w", err) + } + defer resp.Body.Close() + + r := response{status: resp.StatusCode, header: resp.Header} + if resp.StatusCode == http.StatusNotModified { + return r, nil + } + if resp.StatusCode != http.StatusOK { + return r, &StatusError{URL: url, StatusCode: resp.StatusCode} + } + + r.body, err = io.ReadAll(resp.Body) + if err != nil { + return r, fmt.Errorf("could not read the response body: %w", err) + } + return r, nil +} diff --git a/fetch/fetch_test.go b/core/fetch/fetch_test.go similarity index 100% rename from fetch/fetch_test.go rename to core/fetch/fetch_test.go diff --git a/core/github/github.go b/core/github/github.go new file mode 100644 index 0000000..b056470 --- /dev/null +++ b/core/github/github.go @@ -0,0 +1,126 @@ +// Package github is the one place a plugin asks GitHub's REST API where a +// source's newest version is: the newest release, the newest commit that +// changed a path, and one commit's full SHA and date. Every plugin that +// follows a GitHub repo calls these instead of decoding GitHub's responses +// itself. +// +// Each function takes how to fetch (Get, or a plugin's env.Get) and the full +// API URL, so a test serves it from saved pages or a local server. +package github + +import ( + "context" + "encoding/json" + "fmt" + "os" + "time" + + "github.com/patch-it-dev/patch-it-plugins/core/fetch" +) + +// API is GitHub's REST API. +const API = "https://api.github.com" + +// Raw serves a repo's files at a ref, with no auth. +const Raw = "https://raw.githubusercontent.com" + +// Headers adds GITHUB_TOKEN as a bearer token when set, which raises the +// GitHub API's rate limit. Never required. +func Headers() map[string]string { + token := os.Getenv("GITHUB_TOKEN") + if token == "" { + return nil + } + return map[string]string{"Authorization": "Bearer " + token} +} + +// Getter fetches one URL: Get, or plugin.Env.Get. +type Getter func(ctx context.Context, url string) ([]byte, error) + +// Get is fetch.Get with Headers: GITHUB_TOKEN, when set, raises the rate limit +// (never required). +func Get(ctx context.Context, url string) ([]byte, error) { + return fetch.Get(ctx, url, Headers()) +} + +// LatestRelease reads a repo's releases/latest (url) and returns its tag. +// GitHub's "latest" skips drafts and prereleases. +func LatestRelease(ctx context.Context, get Getter, url string) (string, error) { + body, err := get(ctx, url) + if err != nil { + return "", err + } + var release struct { + TagName string `json:"tag_name"` + } + if err := json.Unmarshal(body, &release); err != nil { + return "", fmt.Errorf("parsing github release response: %w", err) + } + if release.TagName == "" { + return "", fmt.Errorf("github release response had no tag_name") + } + return release.TagName, nil +} + +// LatestCommit reads a commit list (url: commits?path=&per_page=1, newest +// first) and returns the newest commit's SHA, or an error when it lists none. +func LatestCommit(ctx context.Context, get Getter, url string) (string, error) { + body, err := get(ctx, url) + if err != nil { + return "", err + } + var commits []struct { + SHA string `json:"sha"` + } + if err := json.Unmarshal(body, &commits); err != nil { + return "", fmt.Errorf("parsing github commits response: %w", err) + } + if len(commits) == 0 || commits[0].SHA == "" { + return "", fmt.Errorf("github lists no commit") + } + return commits[0].SHA, nil +} + +// Commit reads one commit (url: commits/) and returns its full SHA and +// its committer date. A response with no date is an error; one with no SHA +// returns "" (a caller that pins by SHA checks it). +func Commit(ctx context.Context, get Getter, url string) (sha string, date time.Time, err error) { + body, err := get(ctx, url) + if err != nil { + return "", time.Time{}, err + } + var c struct { + SHA string `json:"sha"` + Commit struct { + Committer struct { + Date time.Time `json:"date"` + } `json:"committer"` + } `json:"commit"` + } + if err := json.Unmarshal(body, &c); err != nil { + return "", time.Time{}, fmt.Errorf("parsing github commit: %w", err) + } + if c.Commit.Committer.Date.IsZero() { + return "", time.Time{}, fmt.Errorf("github commit has no date") + } + return c.SHA, c.Commit.Committer.Date, nil +} + +// Head reads one commit (url: commits/) and returns its full +// SHA: the newest commit on a branch, when the source has no releases. +func Head(ctx context.Context, get Getter, url string) (string, error) { + body, err := get(ctx, url) + if err != nil { + return "", err + } + var c struct { + SHA string `json:"sha"` + } + if err := json.Unmarshal(body, &c); err != nil { + return "", fmt.Errorf("parsing github commit: %w", err) + } + if c.SHA == "" { + return "", fmt.Errorf("the commit lookup named no sha") + } + return c.SHA, nil +} diff --git a/core/github/github_test.go b/core/github/github_test.go new file mode 100644 index 0000000..5c4d44a --- /dev/null +++ b/core/github/github_test.go @@ -0,0 +1,58 @@ +package github + +import ( + "context" + "fmt" + "testing" + "time" +) + +// pages serves canned API responses by URL. +func pages(m map[string]string) Getter { + return func(_ context.Context, url string) ([]byte, error) { + b, ok := m[url] + if !ok { + return nil, fmt.Errorf("no page %s", url) + } + return []byte(b), nil + } +} + +func TestLatestRelease(t *testing.T) { + get := pages(map[string]string{"r": `{"tag_name":"v2468"}`, "empty": `{}`, "bad": ``}) + if tag, err := LatestRelease(context.Background(), get, "r"); err != nil || tag != "v2468" { + t.Errorf("got %q, %v", tag, err) + } + for _, url := range []string{"empty", "bad", "missing"} { + if _, err := LatestRelease(context.Background(), get, url); err == nil { + t.Errorf("%s: no error", url) + } + } +} + +func TestLatestCommit(t *testing.T) { + get := pages(map[string]string{"c": `[{"sha":"abc"},{"sha":"def"}]`, "none": `[]`}) + if sha, err := LatestCommit(context.Background(), get, "c"); err != nil || sha != "abc" { + t.Errorf("got %q, %v", sha, err) + } + if _, err := LatestCommit(context.Background(), get, "none"); err == nil { + t.Error("an empty list gave no error") + } +} + +func TestCommitAndHead(t *testing.T) { + get := pages(map[string]string{ + "c": `{"sha":"abc","commit":{"committer":{"date":"2026-11-19T10:00:00Z"}}}`, + "nodate": `{"sha":"abc"}`, + }) + sha, date, err := Commit(context.Background(), get, "c") + if err != nil || sha != "abc" || !date.Equal(time.Date(2026, 11, 19, 10, 0, 0, 0, time.UTC)) { + t.Errorf("got %q %v %v", sha, date, err) + } + if _, _, err := Commit(context.Background(), get, "nodate"); err == nil { + t.Error("a commit with no date gave no error") + } + if sha, err := Head(context.Background(), get, "nodate"); err != nil || sha != "abc" { + t.Errorf("Head: %q %v", sha, err) + } +} diff --git a/hint/hint.go b/core/hint/hint.go similarity index 99% rename from hint/hint.go rename to core/hint/hint.go index 60fe8aa..7039941 100644 --- a/hint/hint.go +++ b/core/hint/hint.go @@ -22,7 +22,7 @@ import ( "strings" "unicode/utf8" - "github.com/patch-it-dev/patch-it-plugins/event" + "github.com/patch-it-dev/patch-it-plugins/core/event" ) // hintLimit caps a hint at the contract's excerpt cap, counted in characters. diff --git a/hint/hint_test.go b/core/hint/hint_test.go similarity index 99% rename from hint/hint_test.go rename to core/hint/hint_test.go index 52020fd..00ba8f0 100644 --- a/hint/hint_test.go +++ b/core/hint/hint_test.go @@ -9,7 +9,7 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/event" + "github.com/patch-it-dev/patch-it-plugins/core/event" ) // fakeDocs serves the saved changelog pages under testdata the way diff --git a/hint/stripe.go b/core/hint/stripe.go similarity index 99% rename from hint/stripe.go rename to core/hint/stripe.go index a94d676..88eaa10 100644 --- a/hint/stripe.go +++ b/core/hint/stripe.go @@ -9,7 +9,7 @@ import ( "strings" "unicode" - "github.com/patch-it-dev/patch-it-plugins/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" ) // Stripe reads docs.stripe.com's changelog, which every page serves as diff --git a/hint/testdata/README.md b/core/hint/testdata/README.md similarity index 100% rename from hint/testdata/README.md rename to core/hint/testdata/README.md diff --git a/hint/testdata/endive.md b/core/hint/testdata/endive.md similarity index 100% rename from hint/testdata/endive.md rename to core/hint/testdata/endive.md diff --git a/hint/testdata/polymorphic-billing-cycle-anchor.md b/core/hint/testdata/polymorphic-billing-cycle-anchor.md similarity index 100% rename from hint/testdata/polymorphic-billing-cycle-anchor.md rename to core/hint/testdata/polymorphic-billing-cycle-anchor.md diff --git a/hint/testdata/remove-payment-method-types-checkout-sessions.md b/core/hint/testdata/remove-payment-method-types-checkout-sessions.md similarity index 100% rename from hint/testdata/remove-payment-method-types-checkout-sessions.md rename to core/hint/testdata/remove-payment-method-types-checkout-sessions.md diff --git a/hint/testdata/removed-payto-fields.md b/core/hint/testdata/removed-payto-fields.md similarity index 100% rename from hint/testdata/removed-payto-fields.md rename to core/hint/testdata/removed-payto-fields.md diff --git a/hint/testdata/removes-the-payment-method-types-parameter-from-payment-intents-and-setup-intents.md b/core/hint/testdata/removes-the-payment-method-types-parameter-from-payment-intents-and-setup-intents.md similarity index 100% rename from hint/testdata/removes-the-payment-method-types-parameter-from-payment-intents-and-setup-intents.md rename to core/hint/testdata/removes-the-payment-method-types-parameter-from-payment-intents-and-setup-intents.md diff --git a/hint/testdata/trial-offer-ga.md b/core/hint/testdata/trial-offer-ga.md similarity index 100% rename from hint/testdata/trial-offer-ga.md rename to core/hint/testdata/trial-offer-ga.md diff --git a/core/kit/kit.go b/core/kit/kit.go new file mode 100644 index 0000000..4f16c7b --- /dev/null +++ b/core/kit/kit.go @@ -0,0 +1,48 @@ +// Package kit is what source plugins share beyond the core: the Wayback +// Machine client a plugin cites a live page through (Wayback, Cite) and the +// "same page as last run" check (Unchanged). Fetching with ETag and +// Last-Modified caching is fetch's Cache, handed to every plugin as +// plugin.Env.Get; the test kit is kittest. +// +// kit is not a plugin: plugins import it, it imports no plugin. +package kit + +import ( + "context" + "fmt" + "reflect" + + "github.com/patch-it-dev/patch-it-plugins/core/plugin" +) + +// Unchanged reports whether body is what api's last stored baseline hashed +// to, and returns that baseline (nil on a first run). A plugin reading a live +// page returns early on true: an unchanged page costs one GET and no snapshot. +func Unchanged(env plugin.Env, api string, body []byte) (last *plugin.Spec, same bool, err error) { + last, err = env.Baselines.LastSpec(api) + if err != nil { + return nil, false, fmt.Errorf("loading the last baseline of %s: %w", api, err) + } + return last, last != nil && last.SHA256 == plugin.SHA256(body), nil +} + +// Cite snapshots a live page and checks the snapshot holds what the plugin +// read from the live page: read turns bytes into whatever the plugin makes +// events from, and the snapshot's must deep-equal want. The events are then +// made from the snapshot and cite snapURL, so every excerpt is on the page it +// cites. No snapshot, or one that says something else (the page changed in +// between), is an error: no provenance, no event. +func Cite[T any](ctx context.Context, a Archiver, pageURL string, read func([]byte) (T, error), want T) (snapURL string, snap []byte, got T, err error) { + snapURL, snap, err = a.Snapshot(ctx, pageURL) + if err != nil { + return "", nil, got, fmt.Errorf("no events without a snapshot to cite: %w", err) + } + got, err = read(snap) + if err != nil { + return "", nil, got, fmt.Errorf("reading the snapshot %s: %w", snapURL, err) + } + if !reflect.DeepEqual(got, want) { + return "", nil, got, fmt.Errorf("the snapshot %s doesn't hold what %s says now; run again", snapURL, pageURL) + } + return snapURL, snap, got, nil +} diff --git a/kit/kit_test.go b/core/kit/kit_test.go similarity index 94% rename from kit/kit_test.go rename to core/kit/kit_test.go index 17f8fa1..a27881e 100644 --- a/kit/kit_test.go +++ b/core/kit/kit_test.go @@ -7,8 +7,8 @@ import ( "testing" "time" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" ) type archive struct { diff --git a/kit/kittest/checksink.go b/core/kit/kittest/checksink.go similarity index 97% rename from kit/kittest/checksink.go rename to core/kit/kittest/checksink.go index b991b6a..548881d 100644 --- a/kit/kittest/checksink.go +++ b/core/kit/kittest/checksink.go @@ -5,8 +5,8 @@ import ( "testing" "time" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" ) // CheckSink checks that the Sinks open returns store as patch-it's hosted diff --git a/core/kit/kittest/kittest.go b/core/kit/kittest/kittest.go new file mode 100644 index 0000000..d134f14 --- /dev/null +++ b/core/kit/kittest/kittest.go @@ -0,0 +1,214 @@ +// Package kittest is the plugin test kit: a saved copy of a source in, the +// events a plugin stores out, compared with a golden file. Nothing reaches the +// network or a database: a URL the test didn't save fails the test, events and +// baselines are kept in memory (Mem), and the clock is fixed, so the same +// pages always give byte-identical events. +// +// got := kittest.Ingest(t, plugin, kittest.Pages{url: "testdata/page.md"}) +// kittest.Golden(t, "testdata/golden/numpy.json", got) +// +// `go test ./plugins/ -update` rewrites a plugin's golden files; +// review the diff like code. +package kittest + +import ( + "bytes" + "context" + "encoding/json" + "flag" + "fmt" + "os" + "path/filepath" + "reflect" + "strconv" + "testing" + "time" + + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" +) + +var update = flag.Bool("update", false, "rewrite the golden files from what the plugins store now") + +// Clock is the time every kit run detects its changes at. +var Clock = time.Date(2026, 11, 14, 12, 0, 0, 0, time.UTC) + +// Pages are the saved copies a run may fetch: URL to file, relative to the +// test's package. +type Pages map[string]string + +// Get serves Pages, and fails the test on any other URL. +func (p Pages) Get(t testing.TB) func(context.Context, string) ([]byte, error) { + return func(_ context.Context, url string) ([]byte, error) { + file, ok := p[url] + if !ok { + t.Errorf("fetched %s, which the test didn't save", url) + return nil, fmt.Errorf("no saved copy of %s", url) + } + return Read(t, file), nil + } +} + +// Read is a saved file, or the test fails. +func Read(t testing.TB, file string) []byte { + t.Helper() + b, err := os.ReadFile(file) + if err != nil { + t.Fatal(err) + } + return b +} + +// Env is an empty Mem for the baselines and the events, Pages for the network +// and Clock for the time. Nothing outside the test is touched: no database, no +// network. +func Env(t testing.TB, pages Pages) plugin.Env { + t.Helper() + return plugin.Env{Baselines: NewMem(), Get: pages.Get(t), Now: func() time.Time { return Clock }} +} + +// Ingest runs p twice and returns what the first run stored. Any rejected +// event or error fails the test, and so does a second run over the same pages +// that stores anything new: the fingerprints must dedupe. +func Ingest(t testing.TB, p plugin.Plugin, pages Pages) []event.EventV2 { + t.Helper() + env := Env(t, pages) + return IngestWith(t, env, p) +} + +// IngestWith is Ingest over an Env the test built (its own archive, a baseline +// already saved). env.Baselines is where the events are stored too, so it must +// be a Sink, as Env's Mem is. +func IngestWith(t testing.TB, env plugin.Env, p plugin.Plugin) []event.EventV2 { + t.Helper() + sink, ok := env.Baselines.(Sink) + if !ok { + t.Fatalf("env.Baselines is a %T, not a kittest.Sink to store the events in", env.Baselines) + } + if rejected, err := Run(context.Background(), env, sink, p); err != nil || rejected { + t.Fatalf("%s: rejected=%v err=%v", p.Name(), rejected, err) + } + first := Stored(t, sink) + if rejected, err := Run(context.Background(), env, sink, p); err != nil || rejected { + t.Fatalf("%s, second run: rejected=%v err=%v", p.Name(), rejected, err) + } + if again := Stored(t, sink); len(again) != len(first) { + t.Errorf("%s: a second run over the same pages stored %d more events", p.Name(), len(again)-len(first)) + } + return first +} + +// Stored is every event in s, in feed order, or the test fails. +func Stored(t testing.TB, s Sink) []event.EventV2 { + t.Helper() + all, err := s.All() + if err != nil { + t.Fatal(err) + } + return all +} + +// Golden compares events with the JSON file at path, or rewrites it with +// -update. Every event must also be valid v2, or the golden would pin a bug. +func Golden(t testing.TB, path string, events []event.EventV2) { + t.Helper() + golden(t, path, events, *update) +} + +// CompareGolden is Golden that never rewrites the file, even with -update: +// for a test that checks someone else's golden file. +func CompareGolden(t testing.TB, path string, events []event.EventV2) { + t.Helper() + golden(t, path, events, false) +} + +func golden(t testing.TB, path string, events []event.EventV2, rewrite bool) { + t.Helper() + for _, e := range events { + if err := e.Validate(); err != nil { + t.Errorf("event %d (%s) is invalid: %v", e.ID, e.Description, err) + } + } + if events == nil { + events = []event.EventV2{} + } + var buf bytes.Buffer + enc := json.NewEncoder(&buf) + enc.SetIndent("", " ") + enc.SetEscapeHTML(false) + if err := enc.Encode(events); err != nil { + t.Fatal(err) + } + if rewrite { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, buf.Bytes(), 0o644); err != nil { + t.Fatal(err) + } + return + } + want, err := os.ReadFile(path) + if err != nil { + t.Fatalf("%v (run the test with -update to create it)", err) + } + if !bytes.Equal(want, buf.Bytes()) { + t.Errorf("%s: the stored events differ from the golden file (%d events now); if the change is intended, rerun with -update and review the diff\n%s", + path, len(events), firstDiff(want, buf.Bytes())) + } +} + +// firstDiff is the first line that differs, for the failure message. +func firstDiff(want, got []byte) string { + w, g := bytes.Split(want, []byte("\n")), bytes.Split(got, []byte("\n")) + for i := 0; i < len(w) || i < len(g); i++ { + var wl, gl []byte + if i < len(w) { + wl = w[i] + } + if i < len(g) { + gl = g[i] + } + if !bytes.Equal(wl, gl) { + return "line " + strconv.Itoa(i+1) + ":\n want " + string(wl) + "\n got " + string(gl) + } + } + return "" +} + +// Archive is the Wayback Machine without the network: it serves a saved copy +// as the snapshot of a page, under a fixed timestamp, and counts the +// snapshots it was asked for. +type Archive struct { + T testing.TB + Pages Pages // the page's URL to the file the snapshot holds + Timestamp string // 14 digits; default 20261114120000 + Calls int +} + +func (a *Archive) Snapshot(_ context.Context, page string) (string, []byte, error) { + a.Calls++ + file, ok := a.Pages[page] + if !ok { + return "", nil, fmt.Errorf("no snapshot of %s in the test archive", page) + } + ts := a.Timestamp + if ts == "" { + ts = "20261114120000" + } + return "https://web.archive.org/web/" + ts + "/" + page, Read(a.T, file), nil +} + +// Registered fails the test unless p's package registered a plugin of p's +// type under p's name: each plugin package checks its own registration. +func Registered(t testing.TB, p plugin.Plugin) { + t.Helper() + got, ok := plugin.Lookup(p.Name()) + if !ok { + t.Errorf("no plugin registered as %q", p.Name()) + return + } + if reflect.TypeOf(got) != reflect.TypeOf(p) { + t.Errorf("%q is a %T, want a %T", p.Name(), got, p) + } +} diff --git a/kit/kittest/mem.go b/core/kit/kittest/mem.go similarity index 97% rename from kit/kittest/mem.go rename to core/kit/kittest/mem.go index dba9b6b..8a04593 100644 --- a/kit/kittest/mem.go +++ b/core/kit/kittest/mem.go @@ -9,8 +9,8 @@ import ( "sync" "time" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" ) // Sink is where a kit run stores what a plugin found, and where the plugin diff --git a/kit/kittest/mem_test.go b/core/kit/kittest/mem_test.go similarity index 91% rename from kit/kittest/mem_test.go rename to core/kit/kittest/mem_test.go index 148767b..c673613 100644 --- a/kit/kittest/mem_test.go +++ b/core/kit/kittest/mem_test.go @@ -3,8 +3,8 @@ package kittest import ( "testing" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" ) func TestMemStoresAsIngestDoes(t *testing.T) { diff --git a/kit/wayback.go b/core/kit/wayback.go similarity index 98% rename from kit/wayback.go rename to core/kit/wayback.go index 74cca51..6745126 100644 --- a/kit/wayback.go +++ b/core/kit/wayback.go @@ -11,7 +11,7 @@ import ( "strings" "time" - "github.com/patch-it-dev/patch-it-plugins/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" ) // Archiver takes a snapshot of a page that never changes afterwards, and hands diff --git a/kit/wayback_test.go b/core/kit/wayback_test.go similarity index 100% rename from kit/wayback_test.go rename to core/kit/wayback_test.go diff --git a/core/migrate/migrate.go b/core/migrate/migrate.go new file mode 100644 index 0000000..3ffa523 --- /dev/null +++ b/core/migrate/migrate.go @@ -0,0 +1,120 @@ +// Package migrate converts stored v1 change events to the proposed v2 contract +// (the change event contract): `affected` becomes a typed +// `subject`. It runs once, at the cut-over, over every event in the store. +// +// A Stripe event keeps its fingerprint: v2's api_operation formula is v1's. +// A model retirement or an SDK release gets a new one, and `supersedes` names +// the old one, so the patch-it client, which finds its open issues and pull +// requests by the fingerprint in their marker, knows they're the same change +// and opens nothing twice. +package migrate + +import ( + "fmt" + "regexp" + "strings" + + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/plugins/migrationguide/sdkguide" + "github.com/patch-it-dev/patch-it-plugins/plugins/retirements/retire" +) + +// providers are the model retirement slugs and who retires the models. +var providers = map[string]string{ + retire.OpenAI.API: "openai", + retire.Anthropic.API: "anthropic", +} + +// guides are the SDKs whose v1 events came from a migration guide. +var guides = map[string]sdkguide.Guide{sdkguide.AnthropicV1.API: sdkguide.AnthropicV1} + +// ToV2 converts one v1 event. It refuses an event it can't place rather than +// guess: an unknown slug with a value, or an SDK event that isn't a removed +// request parameter. +func ToV2(e event.ChangeEvent) (event.EventV2, error) { + v2 := event.EventV2{ + ID: e.ID, API: e.API, Version: e.Version, PreviousVersion: e.PreviousVersion, + ChangeType: e.ChangeType, Breaking: e.Breaking, Description: e.Description, + MigrationHint: e.MigrationHint, SourceURL: e.SourceURL, SourcePointer: e.SourcePointer, + SourceExcerpt: e.SourceExcerpt, Confidence: e.Confidence, DetectedAt: e.DetectedAt, + } + a := e.Affected + if provider, ok := providers[e.API]; ok { + if a.Value == "" { + return event.EventV2{}, fmt.Errorf("event %d: a %s event without the model in affected.value", e.ID, e.API) + } + v2.Subject = event.Subject{ + Type: event.SubjectModel, Provider: provider, Name: a.Value, + Replacement: replacement(e.MigrationHint), + } + return rekeyed(v2, e.Fingerprint), nil + } + if g, ok := guides[e.API]; ok { + if a.Kind != event.AffectedKindRequestParameter || a.Property == "" { + return event.EventV2{}, fmt.Errorf("event %d: an %s SDK event that isn't a removed parameter", e.ID, e.API) + } + v2.Subject = event.Subject{ + Type: event.SubjectPackageRelease, Ecosystem: "pypi", Package: g.API, + Change: event.PackageRemovedParameter, Parameter: a.Property, Symbols: g.Symbols, + } + return rekeyed(v2, e.Fingerprint), nil + } + if a.Value != "" { + return event.EventV2{}, fmt.Errorf("event %d: affected.value on %s, which isn't a model retirement", e.ID, e.API) + } + v2.Subject = event.Subject{ + Type: event.SubjectAPIOperation, Kind: a.Kind, Side: a.Side, + Operation: a.Operation, Schema: a.Schema, Property: a.Property, + } + v2.Fingerprint = e.Fingerprint // the same formula: unchanged by construction + return v2, nil +} + +// rekeyed gives the event its v2 fingerprint and says which v1 one it replaces. +func rekeyed(v2 event.EventV2, old string) event.EventV2 { + v2.Fingerprint = event.FingerprintV2(v2.API, v2.Version, v2.ChangeType, v2.Subject, "") + if v2.Fingerprint != old { + v2.Supersedes = old + } + return v2 +} + +// useInstead is the hint retire.Events writes: "Use instead: …". +var useInstead = regexp.MustCompile("^Use (.+?) instead: ") + +// replacement is the one model the provider names, when its replacement cell +// is exactly one model name: not none ("names no replacement model"), not +// several, not a model with a setting (`gpt-5.6-sol` (`reasoning.mode: pro`)). +// The patch-it client opens an issue for the rest, as it does today. +func replacement(hint string) string { + m := useInstead.FindStringSubmatch(hint) + if m == nil { + return "" + } + name := retire.First(m[1]) + if name == "" || strings.TrimSpace(m[1]) != "`"+name+"`" { + return "" + } + return name +} + +// Fresh converts events a source has just produced in the v1 shape. Unlike a +// stored row, a fresh event replaces nothing, so it carries no `supersedes`. An +// event that can't be placed is returned as an error, like a failed validation. +func Fresh(events []event.ChangeEvent) ([]event.EventV2, []error) { + var out []event.EventV2 + var errs []error + for _, e := range events { + v2, err := ToV2(e) + if err == nil { + v2.Supersedes = "" + err = v2.Validate() + } + if err != nil { + errs = append(errs, err) + continue + } + out = append(out, v2) + } + return out, errs +} diff --git a/migrate/migrate_test.go b/core/migrate/migrate_test.go similarity index 97% rename from migrate/migrate_test.go rename to core/migrate/migrate_test.go index 84646c5..c6fccb3 100644 --- a/migrate/migrate_test.go +++ b/core/migrate/migrate_test.go @@ -8,13 +8,13 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/event" + "github.com/patch-it-dev/patch-it-plugins/core/event" ) // -update rewrites the golden v2 fixture from the v1 ones: go test ./migrate -update var update = flag.Bool("update", false, "rewrite contract/fixtures/change-events-v2.json") -const fixtures = "../contract/fixtures/" +const fixtures = "../../contract/fixtures/" // The v1 fixtures the golden v2 file is converted from: the shared contract // fixture, the hosted service's model retirements and the anthropic SDK release. diff --git a/normalize/golden_test.go b/core/normalize/golden_test.go similarity index 98% rename from normalize/golden_test.go rename to core/normalize/golden_test.go index f3d25ab..6fb92a1 100644 --- a/normalize/golden_test.go +++ b/core/normalize/golden_test.go @@ -9,7 +9,7 @@ import ( "testing" "time" - "github.com/patch-it-dev/patch-it-plugins/diff" + "github.com/patch-it-dev/patch-it-plugins/core/diff" ) // Golden-file tests for the whole pipeline: each directory under diff --git a/normalize/normalize.go b/core/normalize/normalize.go similarity index 98% rename from normalize/normalize.go rename to core/normalize/normalize.go index 9fb8097..9279fc9 100644 --- a/normalize/normalize.go +++ b/core/normalize/normalize.go @@ -14,8 +14,8 @@ import ( "time" "unicode/utf8" - "github.com/patch-it-dev/patch-it-plugins/diff" - "github.com/patch-it-dev/patch-it-plugins/event" + "github.com/patch-it-dev/patch-it-plugins/core/diff" + "github.com/patch-it-dev/patch-it-plugins/core/event" ) // excerptLimit matches the contract's cap, counted in characters. diff --git a/normalize/normalize_test.go b/core/normalize/normalize_test.go similarity index 97% rename from normalize/normalize_test.go rename to core/normalize/normalize_test.go index 28aae0d..89d8025 100644 --- a/normalize/normalize_test.go +++ b/core/normalize/normalize_test.go @@ -6,8 +6,8 @@ import ( "time" "unicode/utf8" - "github.com/patch-it-dev/patch-it-plugins/diff" - "github.com/patch-it-dev/patch-it-plugins/event" + "github.com/patch-it-dev/patch-it-plugins/core/diff" + "github.com/patch-it-dev/patch-it-plugins/core/event" ) const ( diff --git a/normalize/testdata/README.md b/core/normalize/testdata/README.md similarity index 100% rename from normalize/testdata/README.md rename to core/normalize/testdata/README.md diff --git a/normalize/testdata/golden/breaking-property-removed/events.json b/core/normalize/testdata/golden/breaking-property-removed/events.json similarity index 100% rename from normalize/testdata/golden/breaking-property-removed/events.json rename to core/normalize/testdata/golden/breaking-property-removed/events.json diff --git a/normalize/testdata/golden/breaking-property-removed/new.json b/core/normalize/testdata/golden/breaking-property-removed/new.json similarity index 100% rename from normalize/testdata/golden/breaking-property-removed/new.json rename to core/normalize/testdata/golden/breaking-property-removed/new.json diff --git a/normalize/testdata/golden/breaking-property-removed/old.json b/core/normalize/testdata/golden/breaking-property-removed/old.json similarity index 100% rename from normalize/testdata/golden/breaking-property-removed/old.json rename to core/normalize/testdata/golden/breaking-property-removed/old.json diff --git a/normalize/testdata/golden/deprecation/events.json b/core/normalize/testdata/golden/deprecation/events.json similarity index 100% rename from normalize/testdata/golden/deprecation/events.json rename to core/normalize/testdata/golden/deprecation/events.json diff --git a/normalize/testdata/golden/deprecation/new.json b/core/normalize/testdata/golden/deprecation/new.json similarity index 100% rename from normalize/testdata/golden/deprecation/new.json rename to core/normalize/testdata/golden/deprecation/new.json diff --git a/normalize/testdata/golden/deprecation/old.json b/core/normalize/testdata/golden/deprecation/old.json similarity index 100% rename from normalize/testdata/golden/deprecation/old.json rename to core/normalize/testdata/golden/deprecation/old.json diff --git a/normalize/testdata/golden/identical-specs/events.json b/core/normalize/testdata/golden/identical-specs/events.json similarity index 100% rename from normalize/testdata/golden/identical-specs/events.json rename to core/normalize/testdata/golden/identical-specs/events.json diff --git a/normalize/testdata/golden/identical-specs/new.json b/core/normalize/testdata/golden/identical-specs/new.json similarity index 100% rename from normalize/testdata/golden/identical-specs/new.json rename to core/normalize/testdata/golden/identical-specs/new.json diff --git a/normalize/testdata/golden/identical-specs/old.json b/core/normalize/testdata/golden/identical-specs/old.json similarity index 100% rename from normalize/testdata/golden/identical-specs/old.json rename to core/normalize/testdata/golden/identical-specs/old.json diff --git a/normalize/testdata/golden/new-endpoint/events.json b/core/normalize/testdata/golden/new-endpoint/events.json similarity index 100% rename from normalize/testdata/golden/new-endpoint/events.json rename to core/normalize/testdata/golden/new-endpoint/events.json diff --git a/normalize/testdata/golden/new-endpoint/new.json b/core/normalize/testdata/golden/new-endpoint/new.json similarity index 100% rename from normalize/testdata/golden/new-endpoint/new.json rename to core/normalize/testdata/golden/new-endpoint/new.json diff --git a/normalize/testdata/golden/new-endpoint/old.json b/core/normalize/testdata/golden/new-endpoint/old.json similarity index 100% rename from normalize/testdata/golden/new-endpoint/old.json rename to core/normalize/testdata/golden/new-endpoint/old.json diff --git a/core/plugin/plugin.go b/core/plugin/plugin.go new file mode 100644 index 0000000..203f092 --- /dev/null +++ b/core/plugin/plugin.go @@ -0,0 +1,148 @@ +// Package plugin is what a change source is: every source is a plugin behind +// one interface, registered by name. A plugin reads its source and returns a +// Result; it stores nothing itself. Whoever runs it (patch-it's hosted +// ingestion service, or the test kit) stores the events, then the baselines. +// Adding a source is a new package under plugins/ that calls Register; nothing +// here changes. +package plugin + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "fmt" + "slices" + "sync" + "time" + + "github.com/patch-it-dev/patch-it-plugins/core/event" +) + +// Plugin is one place changes come from. +type Plugin interface { + // Name is the slug the command line selects it by and its events carry in + // `api`: "stripe", "openai-models". + Name() string + // Run reads the source as it is now and returns what changed since the + // baseline it saved last (env.Baselines.LastSpec). An error stores nothing. + Run(ctx context.Context, env Env) (Result, error) +} + +// Replayer is a plugin that can also diff two given releases (--from, --to): +// how a historical change is recreated without moving the normal baseline. +type Replayer interface { + Replay(ctx context.Context, env Env, from, to string) (Result, error) +} + +// Env is everything a plugin may reach outside itself, so a test can hand it +// baselines, the network and a clock of its own. A plugin that fetches through +// its own client (the OpenAPI plugin's adapter) ignores Get; its tests fake +// that client instead. +type Env struct { + Baselines BaselineReader + // Get fetches one URL. In both the hosted ingestion service and the test + // kit, this is fetch.Cache.Get (kit/kittest's Pages.Get wraps saved files + // the same way): a non-2xx, non-304 response fails with a *fetch.StatusError + // naming the status, returned as is, so errors.As finds it. A plugin that + // branches on the status (eol, when the provider's own site blocks a plain + // GET) can rely on that. + Get func(ctx context.Context, url string) ([]byte, error) + Now func() time.Time +} + +// BaselineReader is the one thing a plugin reads back: the baseline its last +// run saved. Ingest's store implements it; so does the test kit. +type BaselineReader interface { + // LastSpec is api's latest baseline, or nil (and no error) on a first run. + LastSpec(api string) (*Spec, error) +} + +// Spec is one baseline: a fetched copy of what a source said, kept only so a +// later run can diff against it, or tell in one hash comparison that nothing +// changed. +type Spec struct { + API string + Version string + Tag string + SHA256 string + SourceURL string + FetchedAt time.Time + Replay bool + Content []byte +} + +// Result is what one run found. +type Result struct { + Label string // what was read, for the summary: a tag, a snapshot, a URL + Events []event.EventV2 // valid v2 events, to store + Rejected []error // changes that failed to become valid events + Baselines []Spec // what the run read, saved after the events +} + +var ( + mu sync.Mutex + registry = map[string]Plugin{} +) + +// Register adds a plugin. A second plugin with the same name is a programming +// error: two sources would write the same `api`. So is the name "all", which +// `--api all` means. +func Register(p Plugin) { + mu.Lock() + defer mu.Unlock() + if p.Name() == "all" { + panic(`plugin: "all" means every plugin; a plugin can't be named it`) + } + if _, dup := registry[p.Name()]; dup { + panic(fmt.Sprintf("plugin: a plugin named %q is already registered", p.Name())) + } + registry[p.Name()] = p +} + +// Lookup finds a registered plugin by name: its old slug ("algolia-search") +// or its source's full id ("algolia/search-api"). +func Lookup(name string) (Plugin, bool) { + mu.Lock() + defer mu.Unlock() + if p, ok := registry[name]; ok { + return p, true + } + if s, ok := sources[name]; ok { + return s.Source.Plugin, true + } + return nil, false +} + +// Names are the registered plugins' old slugs, sorted: what `--api all` runs. +// Sources lists them with their full ids and metadata. +func Names() []string { + mu.Lock() + defer mu.Unlock() + names := make([]string, 0, len(registry)) + for n := range registry { + names = append(names, n) + } + slices.Sort(names) + return names +} + +// Baseline is what a plugin saves as the last thing it read, hashed so the +// next run can tell an unchanged source in one comparison. +func Baseline(api, version, tag, sourceURL string, content []byte, replay bool, at time.Time) Spec { + return Spec{ + API: api, + Version: version, + Tag: tag, + SHA256: SHA256(content), + SourceURL: sourceURL, + FetchedAt: at, + Replay: replay, + Content: content, + } +} + +// SHA256 is the hex hash a baseline is compared by. +func SHA256(data []byte) string { + sum := sha256.Sum256(data) + return hex.EncodeToString(sum[:]) +} diff --git a/plugin/plugin_test.go b/core/plugin/plugin_test.go similarity index 97% rename from plugin/plugin_test.go rename to core/plugin/plugin_test.go index 6595548..34f3b12 100644 --- a/plugin/plugin_test.go +++ b/core/plugin/plugin_test.go @@ -5,7 +5,7 @@ import ( "slices" "testing" - "github.com/patch-it-dev/patch-it-plugins/event" + "github.com/patch-it-dev/patch-it-plugins/core/event" ) // testPlugin is a source defined only here: registering it and running it diff --git a/core/plugin/provider.go b/core/plugin/provider.go new file mode 100644 index 0000000..4546d40 --- /dev/null +++ b/core/plugin/provider.go @@ -0,0 +1,272 @@ +package plugin + +import ( + "cmp" + "fmt" + "net/url" + "path" + "reflect" + "regexp" + "slices" + "strings" +) + +// A provider is whose API, SDK or package it is (Stripe, OpenAI, Kubernetes), +// in one category. It has one or more sources: each one place patch-it reads +// (a spec repo, a deprecations page, a changelog), read by one kind (the +// plugins/ package whose plugin runs it), at one layer of a customer's +// stack. A source's id is "/": "stripe/api", +// "algolia/search-api", "openai/models". +// +// Every source is registered with its metadata, in code, by the kind's +// package (RegisterProvider). That registration is the only list of what +// patch-it watches: `go run ./cmd/list-plugins` prints it, the Release +// workflow attaches it to each release as plugins.json, and patch-it's hosted +// ingestion service serves it. There is no list to keep by hand. + +// Category is what a provider is, for grouping: one per provider. +type Category string + +// The categories. A new one is added here, in review. +const ( + CategoryAI Category = "ai" + CategoryPayments Category = "payments" + CategoryInfrastructure Category = "infrastructure" + CategoryDatabases Category = "databases" + CategoryMonitoring Category = "monitoring" + CategoryAnalytics Category = "analytics" + CategoryAuth Category = "auth" + CategoryMessaging Category = "messaging" + CategoryHosting Category = "hosting" + CategorySearch Category = "search" + CategoryDevTools Category = "devtools" + CategoryCRM Category = "crm" + CategoryPackages Category = "packages" +) + +// Categories are every category, in display order. +func Categories() []Category { + return []Category{ + CategoryAI, CategoryPayments, CategoryInfrastructure, CategoryDatabases, + CategoryMonitoring, CategoryAnalytics, CategoryAuth, CategoryMessaging, + CategoryHosting, CategorySearch, CategoryDevTools, CategoryCRM, CategoryPackages, + } +} + +// Layer is which part of a customer's stack a source covers. +type Layer string + +// The layers. +const ( + LayerAPI Layer = "api" + LayerSDK Layer = "sdk" + LayerPackage Layer = "package" + LayerRuntime Layer = "runtime" + LayerCI Layer = "ci" + LayerAIModels Layer = "ai-models" + LayerInfra Layer = "infra" +) + +// Layers are every layer. +func Layers() []Layer { + return []Layer{LayerAPI, LayerSDK, LayerPackage, LayerRuntime, LayerCI, LayerAIModels, LayerInfra} +} + +// Provider is one provider and the sources a kind's package registers for it. +// A provider with sources in several kinds (Anthropic: a deprecations page and +// an SDK migration guide) is registered by each, with the same ID, Name, +// Category and Homepage; the registry merges them. +type Provider struct { + ID string // lowercase slug: "algolia" + Name string // as the provider writes it: "Algolia" + Category Category // what it is + Homepage string // the provider's own site + Sources []Source +} + +// Source is one place a provider's changes are read from. +type Source struct { + // ID is the source's part of its id, unique within the provider: + // "search-api" in "algolia/search-api". + ID string + // Kind is the kind that reads it: the plugins/ package Plugin is + // defined in. RegisterProvider fills it in; a value set by hand must + // match. + Kind string + // Layer is which part of a customer's stack it covers. + Layer Layer + // Watches is what changes it reports, in a few words. + Watches string + // URLs are the provider's own pages or files it reads, absolute; the + // first is the one a person should look at. + URLs []string + // Page is the source's page in this repo: "providers/algolia.md" (why + // this source, what it reads and ignores, its known gaps). + Page string + // Plugin runs it. Its Name is the source's old slug ("algolia-search"): + // what `--api` selects and every event's `api` carries until patch-it's + // change-event contract switches to source ids. + Plugin Plugin +} + +// SourceInfo is a registered source with its provider: what the list prints. +type SourceInfo struct { + Provider Provider // without Sources + Source Source +} + +// FullID is "/". +func (s SourceInfo) FullID() string { return s.Provider.ID + "/" + s.Source.ID } + +// Alias is the source's old slug, its Plugin's Name: "algolia-search". +func (s SourceInfo) Alias() string { return s.Source.Plugin.Name() } + +var ( + providers = map[string]Provider{} // by provider ID, Sources merged + sources = map[string]SourceInfo{} // by full ID + aliases = map[string]string{} // old slug to full ID + slug = regexp.MustCompile(`^[a-z0-9][a-z0-9-]*$`) +) + +// RegisterProvider registers p's sources, each under its full id, and each +// source's Plugin under its old slug (Register), so Lookup finds it by +// either. Incomplete metadata is a programming error, as a duplicate is. +func RegisterProvider(p Provider) { + if err := p.check(); err != nil { + panic("plugin: " + err.Error()) + } + for i := range p.Sources { + p.Sources[i].Kind = KindOf(p.Sources[i].Plugin) + } + mu.Lock() + if have, ok := providers[p.ID]; ok { + if have.Name != p.Name || have.Category != p.Category || have.Homepage != p.Homepage { + mu.Unlock() + panic(fmt.Sprintf("plugin: provider %q registered twice with different metadata", p.ID)) + } + } + for _, s := range p.Sources { + id := p.ID + "/" + s.ID + if _, dup := sources[id]; dup { + mu.Unlock() + panic(fmt.Sprintf("plugin: source %q is already registered", id)) + } + } + mu.Unlock() + + for _, s := range p.Sources { + Register(s.Plugin) // panics on a duplicate old slug + } + + mu.Lock() + defer mu.Unlock() + have := providers[p.ID] + merged := Provider{ID: p.ID, Name: p.Name, Category: p.Category, Homepage: p.Homepage, Sources: append(have.Sources, p.Sources...)} + providers[p.ID] = merged + bare := merged + bare.Sources = nil + for _, s := range p.Sources { + id := p.ID + "/" + s.ID + sources[id] = SourceInfo{Provider: bare, Source: s} + aliases[s.Plugin.Name()] = id + } +} + +// check is the metadata every provider and source must have. +func (p Provider) check() error { + switch { + case !slug.MatchString(p.ID): + return fmt.Errorf("provider id %q isn't a lowercase slug", p.ID) + case p.Name == "": + return fmt.Errorf("provider %q has no Name", p.ID) + case !slices.Contains(Categories(), p.Category): + return fmt.Errorf("provider %q: category %q isn't one of plugin.Categories()", p.ID, p.Category) + case !absolute(p.Homepage): + return fmt.Errorf("provider %q: Homepage %q isn't an absolute URL", p.ID, p.Homepage) + case len(p.Sources) == 0: + return fmt.Errorf("provider %q has no sources", p.ID) + } + seen := map[string]bool{} + for _, s := range p.Sources { + id := p.ID + "/" + s.ID + switch { + case !slug.MatchString(s.ID): + return fmt.Errorf("source id %q isn't a lowercase slug", id) + case seen[s.ID]: + return fmt.Errorf("source %q is listed twice", id) + case s.Plugin == nil: + return fmt.Errorf("source %q has no Plugin", id) + case !slices.Contains(Layers(), s.Layer): + return fmt.Errorf("source %q: layer %q isn't one of plugin.Layers()", id, s.Layer) + case strings.TrimSpace(s.Watches) == "": + return fmt.Errorf("source %q: Watches is empty", id) + case len(s.URLs) == 0: + return fmt.Errorf("source %q has no URLs", id) + case !strings.HasPrefix(s.Page, "providers/") || !strings.HasSuffix(s.Page, ".md"): + return fmt.Errorf("source %q: Page %q isn't providers/.md", id, s.Page) + case s.Kind != "" && s.Kind != KindOf(s.Plugin): + return fmt.Errorf("source %q: Kind %q, but its plugin is in plugins/%s", id, s.Kind, KindOf(s.Plugin)) + } + for _, u := range s.URLs { + if !absolute(u) { + return fmt.Errorf("source %q: URL %q isn't absolute", id, u) + } + } + seen[s.ID] = true + } + return nil +} + +func absolute(u string) bool { + parsed, err := url.Parse(u) + return err == nil && (parsed.Scheme == "https" || parsed.Scheme == "http") && parsed.Host != "" +} + +// KindOf is the kind a plugin is: the name of the package its type is +// defined in (plugins/openapi -> "openapi"). +func KindOf(p Plugin) string { + t := reflect.TypeOf(p) + for t.Kind() == reflect.Pointer { + t = t.Elem() + } + return path.Base(t.PkgPath()) +} + +// Providers are the registered providers with their sources, sorted by id +// (sources by id within each). +func Providers() []Provider { + mu.Lock() + defer mu.Unlock() + out := make([]Provider, 0, len(providers)) + for _, p := range providers { + p.Sources = slices.Clone(p.Sources) + slices.SortFunc(p.Sources, func(a, b Source) int { return cmp.Compare(a.ID, b.ID) }) + out = append(out, p) + } + slices.SortFunc(out, func(a, b Provider) int { return cmp.Compare(a.ID, b.ID) }) + return out +} + +// Sources are the registered sources, sorted by full id. +func Sources() []SourceInfo { + mu.Lock() + defer mu.Unlock() + out := make([]SourceInfo, 0, len(sources)) + for _, s := range sources { + out = append(out, s) + } + slices.SortFunc(out, func(a, b SourceInfo) int { return cmp.Compare(a.FullID(), b.FullID()) }) + return out +} + +// LookupSource finds a source by its full id ("algolia/search-api") or its +// old slug ("algolia-search"). +func LookupSource(id string) (SourceInfo, bool) { + mu.Lock() + defer mu.Unlock() + if full, ok := aliases[id]; ok { + id = full + } + s, ok := sources[id] + return s, ok +} diff --git a/core/plugin/provider_test.go b/core/plugin/provider_test.go new file mode 100644 index 0000000..e890f1d --- /dev/null +++ b/core/plugin/provider_test.go @@ -0,0 +1,84 @@ +package plugin + +import ( + "context" + "strings" + "testing" +) + +type fake struct{ name string } + +func (f fake) Name() string { return f.name } +func (fake) Run(context.Context, Env) (Result, error) { return Result{}, nil } + +func src(id, slug string) Source { + return Source{ID: id, Layer: LayerAPI, Watches: "w", URLs: []string{"https://example.com/spec"}, Page: "providers/acme.md", Plugin: fake{slug}} +} + +func acme(sources ...Source) Provider { + return Provider{ID: "acme-test", Name: "Acme", Category: CategoryPayments, Homepage: "https://example.com", Sources: sources} +} + +func TestRegisterProviderResolvesBothIDs(t *testing.T) { + RegisterProvider(acme(src("api", "acme-test-api"))) + RegisterProvider(acme(src("sdk", "acme-test-sdk"))) // a second kind adds a source + for _, id := range []string{"acme-test/api", "acme-test-api"} { + s, ok := LookupSource(id) + if !ok || s.FullID() != "acme-test/api" || s.Alias() != "acme-test-api" || s.Source.Kind != "plugin" { + t.Errorf("LookupSource(%q) = %+v, %v", id, s, ok) + } + if p, ok := Lookup(id); !ok || p.Name() != "acme-test-api" { + t.Errorf("Lookup(%q) = %v, %v", id, p, ok) + } + } + var got []string + for _, p := range Providers() { + if p.ID == "acme-test" { + for _, s := range p.Sources { + got = append(got, s.ID) + } + } + } + if strings.Join(got, ",") != "api,sdk" { + t.Errorf("acme-test's sources = %v", got) + } +} + +func TestRegisterProviderRefusesIncompleteMetadata(t *testing.T) { + for name, p := range map[string]Provider{ + "no category": {ID: "x1", Name: "X", Homepage: "https://x.example", Sources: []Source{src("api", "x1")}}, + "no sources": {ID: "x2", Name: "X", Category: CategoryAI, Homepage: "https://x.example"}, + "bad homepage": {ID: "x3", Name: "X", Category: CategoryAI, Homepage: "x.example", Sources: []Source{src("api", "x3")}}, + "no watches": {ID: "x4", Name: "X", Category: CategoryAI, Homepage: "https://x.example", Sources: []Source{func() Source { + s := src("api", "x4") + s.Watches = "" + return s + }()}}, + "bad layer": {ID: "x5", Name: "X", Category: CategoryAI, Homepage: "https://x.example", Sources: []Source{func() Source { + s := src("api", "x5") + s.Layer = "API" + return s + }()}}, + "wrong kind": {ID: "x6", Name: "X", Category: CategoryAI, Homepage: "https://x.example", Sources: []Source{func() Source { + s := src("api", "x6") + s.Kind = "openapi" + return s + }()}}, + } { + if err := p.check(); err == nil { + t.Errorf("%s: accepted", name) + } + } +} + +func TestRegisterProviderRefusesConflictingProvider(t *testing.T) { + RegisterProvider(acme(src("one", "acme-test-one"))) + defer func() { + if recover() == nil { + t.Error("a provider registered again with another name didn't panic") + } + }() + p := acme(src("two", "acme-test-two")) + p.Name = "Acme Corp" + RegisterProvider(p) +} diff --git a/core/plugin/versions.go b/core/plugin/versions.go new file mode 100644 index 0000000..5971490 --- /dev/null +++ b/core/plugin/versions.go @@ -0,0 +1,70 @@ +package plugin + +import ( + "context" + "time" +) + +// A source that has versions (a spec repo's tags, a package's releases on +// PyPI or npm, a guide per major) is read in three parts, so whoever runs it +// can cover every version, not only the newest: +// +// versions, _ := s.ListVersions(ctx, env) // oldest first +// a, _ := s.ReadAt(ctx, env, versions[i]) +// b, _ := s.ReadAt(ctx, env, versions[i+1]) +// result, _ := s.Extract(ctx, env, a, b) // what changed from a to b +// +// Every consecutive pair is one Extract; a released version never changes, so +// a pair is done once. Run keeps working: it reads the newest version and +// extracts against the last baseline, as before. +// +// A source with no versions of its own (a deprecations page, a feed) has only +// Run: patch-it's hosted ingestion service reads it as a living page. + +// Version is one version of a source. +type Version struct { + // ID is the version as events name it: "2.0.0", "2026-11-19", "2025-03-31.basil". + ID string + // Released is when it came out; zero when the source doesn't say. + Released time.Time + // Ref is what to read it at: a tag, a commit SHA, a branch, a date. + Ref string + // URL is where a person sees this version (a release page, a commit). + URL string +} + +// Snapshot is a source as it was at one version: what ReadAt read. +type Snapshot struct { + Version Version + Content []byte + SourceURL string // the exact document read, pinned to the version +} + +// IsZero is true for the snapshot before the first version: nothing. +func (s Snapshot) IsZero() bool { return s.Version.ID == "" && s.Content == nil } + +// VersionLister lists every version a source has, oldest first. +type VersionLister interface { + ListVersions(ctx context.Context, env Env) ([]Version, error) +} + +// Reader reads a source at one version. +type Reader interface { + ReadAt(ctx context.Context, env Env, v Version) (Snapshot, error) +} + +// Extractor turns two snapshots into the changes between them. from may be +// zero (no earlier version read). The events carry to's version and, when from +// isn't zero, previous_version set to from's. It stores nothing: Result has no +// Baselines (Run adds its own). +type Extractor interface { + Extract(ctx context.Context, env Env, from, to Snapshot) (Result, error) +} + +// Versioned is a plugin read version by version. +type Versioned interface { + Plugin + VersionLister + Reader + Extractor +} diff --git a/core/versions/versions.go b/core/versions/versions.go new file mode 100644 index 0000000..77ebddf --- /dev/null +++ b/core/versions/versions.go @@ -0,0 +1,287 @@ +// Package versions lists a source's versions from where they're published: +// a GitHub repo's releases, tags or the commits that changed a file, a +// package's releases on PyPI or npm, and the Web Archive's capture of a page +// at a date. Every list comes back oldest first, as plugin.Version. +// +// Each function takes how to fetch (a plugin's env.Get, or github.Get for the +// GitHub API with its token), so tests serve saved responses. +package versions + +import ( + "cmp" + "context" + "encoding/json" + "fmt" + "net/url" + "regexp" + "slices" + "strconv" + "strings" + "time" + + "github.com/patch-it-dev/patch-it-plugins/core/plugin" +) + +// Getter fetches one URL: plugin.Env.Get, or github.Get. +type Getter func(ctx context.Context, url string) ([]byte, error) + +// The services' base URLs; variables only so a test can point them elsewhere. +var ( + GitHubAPI = "https://api.github.com" + PyPI = "https://pypi.org" + NPM = "https://registry.npmjs.org" + Wayback = "https://archive.org" +) + +// MaxPages caps how many pages of 100 a GitHub list reads (1,000 entries). +const MaxPages = 10 + +// pages reads a GitHub list endpoint page by page until a short page. +func pages[T any](ctx context.Context, get Getter, base string) ([]T, error) { + sep := "?" + if strings.Contains(base, "?") { + sep = "&" + } + var all []T + for page := 1; page <= MaxPages; page++ { + body, err := get(ctx, fmt.Sprintf("%s%sper_page=100&page=%d", base, sep, page)) + if err != nil { + return nil, err + } + var got []T + if err := json.Unmarshal(body, &got); err != nil { + return nil, fmt.Errorf("parsing %s: %w", base, err) + } + all = append(all, got...) + if len(got) < 100 { + break + } + } + return all, nil +} + +// GitHubReleases are a repo's published releases ("owner/name"), drafts and +// prereleases left out, oldest first by publication date. keep, when set, +// keeps only the tags it accepts. +func GitHubReleases(ctx context.Context, get Getter, repo string, keep func(tag string) bool) ([]plugin.Version, error) { + rs, err := pages[struct { + Tag string `json:"tag_name"` + Draft bool `json:"draft"` + Prerelease bool `json:"prerelease"` + Published time.Time `json:"published_at"` + URL string `json:"html_url"` + }](ctx, get, GitHubAPI+"/repos/"+repo+"/releases") + if err != nil { + return nil, fmt.Errorf("listing %s's releases: %w", repo, err) + } + var out []plugin.Version + for _, r := range rs { + if r.Draft || r.Prerelease || r.Tag == "" || (keep != nil && !keep(r.Tag)) { + continue + } + out = append(out, plugin.Version{ID: r.Tag, Released: r.Published, Ref: r.Tag, URL: r.URL}) + } + slices.SortStableFunc(out, func(a, b plugin.Version) int { return a.Released.Compare(b.Released) }) + return out, nil +} + +// GitHubTags are a repo's tags that match pattern; its first submatch, when +// it has one, is the version's ID ("^v(\d+\.\d+\.\d+)$" makes "v1.2.3" 1.2.3). +// GitHub's tag list has no dates, so they're sorted by Compare on the ID. +func GitHubTags(ctx context.Context, get Getter, repo string, pattern *regexp.Regexp) ([]plugin.Version, error) { + ts, err := pages[struct { + Name string `json:"name"` + }](ctx, get, GitHubAPI+"/repos/"+repo+"/tags") + if err != nil { + return nil, fmt.Errorf("listing %s's tags: %w", repo, err) + } + var out []plugin.Version + for _, t := range ts { + m := pattern.FindStringSubmatch(t.Name) + if m == nil { + continue + } + id := t.Name + if len(m) > 1 && m[1] != "" { + id = m[1] + } + out = append(out, plugin.Version{ID: id, Ref: t.Name, URL: "https://github.com/" + repo + "/releases/tag/" + url.PathEscape(t.Name)}) + } + slices.SortStableFunc(out, func(a, b plugin.Version) int { return Compare(a.ID, b.ID) }) + return out, nil +} + +// GitHubCommits are the commits on the default branch that changed path, +// oldest first: a version per change, for a file the repo never tags. The ID +// is the commit's date (YYYY-MM-DD), the Ref its SHA. +func GitHubCommits(ctx context.Context, get Getter, repo, path string) ([]plugin.Version, error) { + cs, err := pages[struct { + SHA string `json:"sha"` + URL string `json:"html_url"` + Commit struct { + Committer struct { + Date time.Time `json:"date"` + } `json:"committer"` + } `json:"commit"` + }](ctx, get, GitHubAPI+"/repos/"+repo+"/commits?path="+url.QueryEscape(path)) + if err != nil { + return nil, fmt.Errorf("listing the commits that changed %s in %s: %w", path, repo, err) + } + var out []plugin.Version + for _, c := range cs { + if c.SHA == "" || c.Commit.Committer.Date.IsZero() { + continue + } + at := c.Commit.Committer.Date.UTC() + out = append(out, plugin.Version{ID: at.Format("2006-01-02"), Released: at, Ref: c.SHA, URL: c.URL}) + } + slices.SortStableFunc(out, func(a, b plugin.Version) int { return a.Released.Compare(b.Released) }) + return out, nil +} + +// PyPIReleases are a package's final releases on PyPI (no pre-, dev- or +// yanked releases, none without files), oldest first by version. The Ref is +// the version; a plugin that reads a repo at a tag maps it. +func PyPIReleases(ctx context.Context, get Getter, pkg string) ([]plugin.Version, error) { + body, err := get(ctx, PyPI+"/pypi/"+url.PathEscape(pkg)+"/json") + if err != nil { + return nil, fmt.Errorf("listing %s's releases on PyPI: %w", pkg, err) + } + var doc struct { + Releases map[string][]struct { + Uploaded time.Time `json:"upload_time_iso_8601"` + Yanked bool `json:"yanked"` + } `json:"releases"` + } + if err := json.Unmarshal(body, &doc); err != nil { + return nil, fmt.Errorf("parsing PyPI's %s: %w", pkg, err) + } + var out []plugin.Version + for v, files := range doc.Releases { + if len(files) == 0 || !Final(v) { + continue + } + var first time.Time + yanked := true + for _, f := range files { + yanked = yanked && f.Yanked + if first.IsZero() || f.Uploaded.Before(first) { + first = f.Uploaded + } + } + if yanked { + continue + } + out = append(out, plugin.Version{ID: v, Released: first.UTC(), Ref: v, URL: "https://pypi.org/project/" + pkg + "/" + v + "/"}) + } + slices.SortStableFunc(out, func(a, b plugin.Version) int { return Compare(a.ID, b.ID) }) + return out, nil +} + +// NPMReleases are a package's published versions on npm, prereleases and +// deprecated versions left out, oldest first by version. +func NPMReleases(ctx context.Context, get Getter, pkg string) ([]plugin.Version, error) { + body, err := get(ctx, NPM+"/"+strings.Replace(url.PathEscape(pkg), "%40", "@", 1)) + if err != nil { + return nil, fmt.Errorf("listing %s's versions on npm: %w", pkg, err) + } + var doc struct { + Versions map[string]struct { + Deprecated any `json:"deprecated"` + } `json:"versions"` + Time map[string]time.Time `json:"time"` + } + if err := json.Unmarshal(body, &doc); err != nil { + return nil, fmt.Errorf("parsing npm's %s: %w", pkg, err) + } + var out []plugin.Version + for v, meta := range doc.Versions { + if !Final(v) || meta.Deprecated != nil { + continue + } + out = append(out, plugin.Version{ID: v, Released: doc.Time[v].UTC(), Ref: v, URL: "https://www.npmjs.com/package/" + pkg + "/v/" + v}) + } + slices.SortStableFunc(out, func(a, b plugin.Version) int { return Compare(a.ID, b.ID) }) + return out, nil +} + +// WaybackAt is the Web Archive's capture of page closest to at: the version +// of a page that only shows its newest state, as it was on a release's date. +// The Ref is the capture's URL (web.archive.org/web//). +func WaybackAt(ctx context.Context, get Getter, page string, at time.Time) (plugin.Version, error) { + q := url.Values{"url": {page}, "timestamp": {at.UTC().Format("20060102150405")}} + body, err := get(ctx, Wayback+"/wayback/available?"+q.Encode()) + if err != nil { + return plugin.Version{}, fmt.Errorf("asking the Web Archive for %s at %s: %w", page, at.Format(time.DateOnly), err) + } + var doc struct { + Snapshots struct { + Closest struct { + Available bool `json:"available"` + URL string `json:"url"` + Timestamp string `json:"timestamp"` + Status string `json:"status"` + } `json:"closest"` + } `json:"archived_snapshots"` + } + if err := json.Unmarshal(body, &doc); err != nil { + return plugin.Version{}, fmt.Errorf("parsing the Web Archive's answer for %s: %w", page, err) + } + c := doc.Snapshots.Closest + if !c.Available || c.URL == "" || (c.Status != "" && c.Status != "200") { + return plugin.Version{}, fmt.Errorf("the Web Archive has no capture of %s near %s", page, at.Format(time.DateOnly)) + } + taken, err := time.Parse("20060102150405", c.Timestamp) + if err != nil { + return plugin.Version{}, fmt.Errorf("the Web Archive's capture of %s has timestamp %q", page, c.Timestamp) + } + return plugin.Version{ID: taken.Format(time.DateOnly), Released: taken, Ref: c.URL, URL: c.URL}, nil +} + +var ( + final = regexp.MustCompile(`^v?\d+(\.\d+)*$`) + number = regexp.MustCompile(`\d+`) +) + +// Final is true for a plain release number ("2.0.0", "v0.86.0", "11.0"), false +// for a pre-release, dev, post or local version ("2.0.0rc1", "1.0.0-beta.2"). +func Final(v string) bool { return final.MatchString(v) } + +// Compare orders two version IDs: their numbers part by part ("1.10" after +// "1.9"), dates as numbers too ("2026-01-02" after "2025-12-31"), then as +// strings when the numbers tie. +func Compare(a, b string) int { + na, nb := number.FindAllString(a, -1), number.FindAllString(b, -1) + for i := 0; i < len(na) && i < len(nb); i++ { + x, _ := strconv.Atoi(na[i]) + y, _ := strconv.Atoi(nb[i]) + if c := cmp.Compare(x, y); c != 0 { + return c + } + } + if c := cmp.Compare(len(na), len(nb)); c != 0 { + return c + } + return strings.Compare(a, b) +} + +// Major is a version's first number ("2" for "2.0.0", "8" for "posthog-v8.1"), +// or -1. +func Major(v string) int { + m := number.FindString(v) + if m == "" { + return -1 + } + n, _ := strconv.Atoi(m) + return n +} + +// Pairs are the consecutive pairs of vs, oldest first: what a backfill +// extracts, one pair at a time. +func Pairs(vs []plugin.Version) [][2]plugin.Version { + var out [][2]plugin.Version + for i := 1; i < len(vs); i++ { + out = append(out, [2]plugin.Version{vs[i-1], vs[i]}) + } + return out +} diff --git a/core/versions/versions_test.go b/core/versions/versions_test.go new file mode 100644 index 0000000..ba1981a --- /dev/null +++ b/core/versions/versions_test.go @@ -0,0 +1,170 @@ +package versions + +import ( + "context" + "fmt" + "regexp" + "strings" + "testing" + "time" + + "github.com/patch-it-dev/patch-it-plugins/core/plugin" +) + +// serve answers saved (here: written out) responses by URL; anything else +// fails the test. +func serve(t *testing.T, pages map[string]string) Getter { + return func(_ context.Context, u string) ([]byte, error) { + b, ok := pages[u] + if !ok { + t.Errorf("fetched %s, which the test didn't save", u) + return nil, fmt.Errorf("no page %s", u) + } + return []byte(b), nil + } +} + +func ids(vs []plugin.Version) string { + var out []string + for _, v := range vs { + out = append(out, v.ID+"@"+v.Ref) + } + return strings.Join(out, " ") +} + +func TestGitHubReleasesSkipsDraftsAndPrereleasesOldestFirst(t *testing.T) { + get := serve(t, map[string]string{ + "https://api.github.com/repos/acme/spec/releases?per_page=100&page=1": `[ + {"tag_name":"v3","published_at":"2026-03-01T00:00:00Z","html_url":"https://github.com/acme/spec/releases/tag/v3"}, + {"tag_name":"v3-rc","prerelease":true,"published_at":"2026-02-20T00:00:00Z"}, + {"tag_name":"v4","draft":true}, + {"tag_name":"v2","published_at":"2026-02-01T00:00:00Z"}, + {"tag_name":"canary-1","published_at":"2026-02-15T00:00:00Z"} + ]`, + }) + vs, err := GitHubReleases(context.Background(), get, "acme/spec", func(tag string) bool { return !strings.HasPrefix(tag, "canary") }) + if err != nil { + t.Fatal(err) + } + if got := ids(vs); got != "v2@v2 v3@v3" { + t.Errorf("got %s", got) + } +} + +func TestGitHubReleasesReadsEveryPage(t *testing.T) { + var full []string + for i := 0; i < 100; i++ { + full = append(full, fmt.Sprintf(`{"tag_name":"v%d","published_at":"2025-01-01T00:00:%02dZ"}`, i, i%60)) + } + get := serve(t, map[string]string{ + "https://api.github.com/repos/acme/spec/releases?per_page=100&page=1": "[" + strings.Join(full, ",") + "]", + "https://api.github.com/repos/acme/spec/releases?per_page=100&page=2": `[{"tag_name":"v100","published_at":"2026-01-01T00:00:00Z"}]`, + }) + vs, err := GitHubReleases(context.Background(), get, "acme/spec", nil) + if err != nil || len(vs) != 101 || vs[100].ID != "v100" { + t.Fatalf("got %d versions, last %v, err %v", len(vs), vs[len(vs)-1], err) + } +} + +func TestGitHubTagsMatchesAndSortsByVersion(t *testing.T) { + get := serve(t, map[string]string{ + "https://api.github.com/repos/acme/sdk/tags?per_page=100&page=1": `[{"name":"v1.10.0"},{"name":"v1.9.0"},{"name":"v2.0.0rc1"},{"name":"nightly"},{"name":"v1.2.0"}]`, + }) + vs, err := GitHubTags(context.Background(), get, "acme/sdk", regexp.MustCompile(`^v(\d+\.\d+\.\d+)$`)) + if err != nil { + t.Fatal(err) + } + if got := ids(vs); got != "1.2.0@v1.2.0 1.9.0@v1.9.0 1.10.0@v1.10.0" { + t.Errorf("got %s", got) + } +} + +func TestGitHubCommitsAreDatedVersions(t *testing.T) { + get := serve(t, map[string]string{ + "https://api.github.com/repos/acme/spec/commits?path=spec%2Fopenapi.json&per_page=100&page=1": `[ + {"sha":"bbb","commit":{"committer":{"date":"2026-11-19T10:00:00Z"}}}, + {"sha":"aaa","commit":{"committer":{"date":"2026-10-01T09:00:00Z"}}} + ]`, + }) + vs, err := GitHubCommits(context.Background(), get, "acme/spec", "spec/openapi.json") + if err != nil { + t.Fatal(err) + } + if got := ids(vs); got != "2026-10-01@aaa 2026-11-19@bbb" { + t.Errorf("got %s", got) + } +} + +func TestPyPIReleasesAreFinalAndNotYanked(t *testing.T) { + get := serve(t, map[string]string{ + "https://pypi.org/pypi/acme/json": `{"releases":{ + "1.0.0":[{"upload_time_iso_8601":"2024-01-01T00:00:00Z"}], + "1.10.0":[{"upload_time_iso_8601":"2025-01-01T00:00:00Z"}], + "1.9.0":[{"upload_time_iso_8601":"2024-06-01T00:00:00Z"}], + "2.0.0rc1":[{"upload_time_iso_8601":"2025-02-01T00:00:00Z"}], + "1.9.1":[{"upload_time_iso_8601":"2024-07-01T00:00:00Z","yanked":true}], + "0.9":[] + }}`, + }) + vs, err := PyPIReleases(context.Background(), get, "acme") + if err != nil { + t.Fatal(err) + } + if got := ids(vs); got != "1.0.0@1.0.0 1.9.0@1.9.0 1.10.0@1.10.0" { + t.Errorf("got %s", got) + } + if !vs[0].Released.Equal(time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC)) { + t.Errorf("released %v", vs[0].Released) + } +} + +func TestNPMReleases(t *testing.T) { + get := serve(t, map[string]string{ + "https://registry.npmjs.org/@acme%2Fsdk": `{"versions":{"1.0.0":{},"1.1.0":{"deprecated":"use 1.1.1"},"1.1.1":{},"2.0.0-beta.1":{}}, + "time":{"1.0.0":"2024-01-01T00:00:00Z","1.1.1":"2024-02-01T00:00:00Z"}}`, + }) + vs, err := NPMReleases(context.Background(), get, "@acme/sdk") + if err != nil { + t.Fatal(err) + } + if got := ids(vs); got != "1.0.0@1.0.0 1.1.1@1.1.1" { + t.Errorf("got %s", got) + } +} + +func TestWaybackAt(t *testing.T) { + get := serve(t, map[string]string{ + "https://archive.org/wayback/available?timestamp=20250301000000&url=https%3A%2F%2Facme.example%2Fdeprecations": `{"archived_snapshots":{"closest":{"available":true,"status":"200","timestamp":"20250228120000","url":"http://web.archive.org/web/20250228120000/https://acme.example/deprecations"}}}`, + "https://archive.org/wayback/available?timestamp=20000101000000&url=https%3A%2F%2Facme.example%2Fdeprecations": `{"archived_snapshots":{}}`, + }) + v, err := WaybackAt(context.Background(), get, "https://acme.example/deprecations", time.Date(2025, 3, 1, 0, 0, 0, 0, time.UTC)) + if err != nil || v.ID != "2025-02-28" || !strings.Contains(v.Ref, "/web/20250228120000/") { + t.Errorf("got %+v, %v", v, err) + } + if _, err := WaybackAt(context.Background(), get, "https://acme.example/deprecations", time.Date(2000, 1, 1, 0, 0, 0, 0, time.UTC)); err == nil { + t.Error("no capture gave no error") + } +} + +func TestCompareFinalMajorPairs(t *testing.T) { + for _, c := range []struct { + a, b string + want int + }{{"1.9", "1.10", -1}, {"2.0.0", "2.0.0", 0}, {"2026-01-02", "2025-12-31", 1}, {"v2", "v10", -1}, {"1.0", "1.0.1", -1}} { + if got := Compare(c.a, c.b); got != c.want { + t.Errorf("Compare(%q, %q) = %d, want %d", c.a, c.b, got, c.want) + } + } + for v, want := range map[string]bool{"2.0.0": true, "v0.86.0": true, "2.0.0rc1": false, "1.0.0-beta.1": false, "1.0.post1": false} { + if Final(v) != want { + t.Errorf("Final(%q) = %v", v, !want) + } + } + if Major("posthog-v8.1.0") != 8 || Major("none") != -1 { + t.Error("Major") + } + ps := Pairs([]plugin.Version{{ID: "1"}, {ID: "2"}, {ID: "3"}}) + if len(ps) != 2 || ps[0][0].ID != "1" || ps[1][1].ID != "3" { + t.Errorf("Pairs = %v", ps) + } +} diff --git a/docs/README-template.md b/docs/README-template.md index 43f9bd5..53e37dd 100644 --- a/docs/README-template.md +++ b/docs/README-template.md @@ -4,15 +4,16 @@ Every public patch-it repo's README has the same shape, so a developer landing o ## The sections, in order +A README is **one screen**. Everything else goes in `docs/` and is linked. + | # | Section | What goes in it | |---|---|---| -| 1 | **What** (one line, bold, under the title) | What *this repo* is, in one sentence, in the product's words. For a part of patch-it, say which part of the story it is ("how patch-it notices, on its own, that something changed"). | -| 2 | **Why** (one short paragraph) | The one-liner for patch-it: when anything your code depends on changes (APIs, packages, SDKs, runtimes, CI, AI models), patch-it finds the code that breaks and opens the pull request that fixes it. In your CI, on your key. "You get the fix, not the alert." Then one sentence on why this repo matters to that. | -| 3 | **Badges** (one line of them) | The badge set below, in that order. | -| 4 | **Visual** | One picture of the flow: a mermaid diagram (GitHub renders it) until there's a GIF or screenshot. Left to right, ending in "pull request in your repo". | -| 5 | **The story** ("What patch-it does") | In this order, one or two lines each: (1) **it notices on its own** (it watches the change itself; no one assigns an alert; also changes with no version bump, like a model or API shut down on a date); (2) **it writes the fix** (an AI model on your key writes the code change, your CI runs your tests on the pull request like any PR, you review and merge it; never say patch-it tests the fix); (3) **proof** ("8 pull requests merged by outside maintainers", no other counts); (4) **layers covered**, as a table with two status columns, **Noticed** (a plugin watches it) and **Fixed** (patch-it writes the PR), each `live`, `in testing`, `building` or `planned`, never ahead of the truth; (5) **how to start** (a link to docs.patch-it.dev). | -| 6 | **Quickstart** (three or four steps, one screen) | The shortest path from a clean clone to the repo's main job, each step a command that runs as written, with the output to expect. Details live in linked docs. | -| 7 | **How it works, and links** | A short "how it works", the layout if the repo has code, how to run its checks, releases, contributing (with Discussions and good first issues), license, and docs.patch-it.dev. | +| 1 | **What** (two lines at most, bold first sentence, under the title) | What *this repo* is, in the product's words. For a part of patch-it, say which part of the story it is ("how patch-it notices, on its own, that something changed"). | +| 2 | **Badges** (one line of them) | The badge set below, in that order. | +| 3 | **Quickstart** (three steps) | The shortest path from a clean clone to the repo's main job, each step a command that runs as written. | +| 4 | **Links** | `docs/` pages (how-to, how it works), CONTRIBUTING, AGENTS, Discussions, good first issues, license, docs.patch-it.dev. | + +**Never a list of what's covered** (providers, plugins, layers with statuses, counts): those change at run time and the code is their only source. Say how to find them (the docs site, `make plugins`) instead. ## The badge set @@ -23,7 +24,7 @@ In this order; drop one only if it can't be honest for the repo (no release yet: 3. **Language version**: e.g. `https://img.shields.io/github/go-mod/go-version/patch-it-dev/`. 4. **Latest release**, once there is one: `https://img.shields.io/github/v/release/patch-it-dev/`. 5. **License**: `https://img.shields.io/badge/license--3f5bf0`. -6. **A count the repo is about**, only if a test keeps it right (patch-it-plugins: the plugins badge, set by `make plugins-json` and checked by `go test`). +6. **No counts.** A badge that counts plugins, providers or anything else that changes at run time is a hand-kept list in disguise: link the docs instead. 7. **Docs**: `https://img.shields.io/badge/docs-docs.patch--it.dev-3f5bf0`. 8. **Discussions**: `https://img.shields.io/github/discussions/patch-it-dev/`. @@ -48,52 +49,17 @@ Use only images that are already public, linked by URL, never copied in: Patchy # -**** - -**Why:** when anything your code depends on changes (APIs, packages, SDKs, runtimes, CI, AI models), [patch-it](https://patch-it.dev) finds the code that breaks and opens the pull request that fixes it. In your CI, on your key. You get the fix, not the alert. +**** -```mermaid -flowchart LR - A["A dependency changes"] --> B[""] --> C["Pull request in your repo"] -``` - -## What patch-it does - -1. **It notices on its own.** ... -2. **It writes the fix.** ... -3. **Proof:** 8 pull requests opened by the early version have been merged by outside maintainers. -4. **What it covers:** - - | Layer | Noticed | Fixed | - |---|---|---| - | APIs | live | live | - | AI models | live | live | - | Packages and SDKs | live | in testing | - | Runtimes | live | building | - | CI | live | planned | +## -5. **Start:** [docs.patch-it.dev](https://docs.patch-it.dev). +1. ... +2. ... +3. ... -## +## More -## How it works - -## Contributing - -## License +- ```` - -Keep the layer statuses the same in every repo, and change them everywhere at once when one moves. - -## When a repo goes public - -The README is half of it. The same day, in the repo's settings: - -1. **Description**: one line in the same words (what the repo is, the layers, "You get the fix, not the alert"), under 350 characters, no version-bot names, never API-only. -2. **Website**: `https://docs.patch-it.dev` (or the page in the docs about this repo). -3. **Topics**: from `dependencies`, `breaking-changes`, `api-changes`, `api`, `sdk`, `ai-models`, `ci`, `pull-requests`, `automation`, plus the repo's language and what it is (`go`, `plugins`, `documentation`, …). -4. **Discussions** on, if people will ask questions there; keep the categories Q&A, Ideas, Show and tell and Announcements. -5. **Social preview** (Settings → General → Social preview; GitHub has no API for it): a 1280×640 PNG, dark background from the brand tokens, the plain wordmark, Patchy on a cream tile, IBM Plex, the repo's one line. No gradients or glows. -6. Check a shared link to the repo shows its own card. diff --git a/docs/README.md b/docs/README.md index fbfc72d..f5b8552 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,10 +2,10 @@ | Page | What's in it | |---|---| -| [plugins.md](plugins.md) | Every supported plugin: id, provider, what it watches, source. Also published on [docs.patch-it.dev](https://docs.patch-it.dev/supported-apis/plugins/). | -| [plugins.json](plugins.json) | The same list as JSON, generated by `make plugins-json` from the plugin registry and plugins.md. Don't edit by hand. | +| [plugins.schema.json](plugins.schema.json) | The shape of `plugins.json`, the list of providers and sources that `go run ./cmd/list-plugins` prints from the registry and each release attaches. No list is committed. | +| [how-it-works.md](how-it-works.md) | Providers, sources and kinds; what a plugin does, version by version; the layout, the checks, releases. | | [README-template.md](README-template.md) | The shape every public patch-it repo's README follows: sections, badge set, words, images. | -| [writing-a-plugin.md](writing-a-plugin.md) | How to add a provider: the `Plugin` interface, fetching through `env.Get` and `kit/`, saving the source, the golden test with `kittest`, the provider page and the plugin list. | +| [writing-a-plugin.md](writing-a-plugin.md) | How to add a provider: the `Plugin` interface, fetching through `env.Get` and `kit/`, saving the source, the golden test with `kittest`, the provider page and the source's metadata in its registration. | | [../providers/](../providers/) | One page per provider: why that source, what the plugin reads and ignores. | | [../contract/README.md](../contract/README.md) | The change-event fixtures copied from patch-it. | diff --git a/docs/how-it-works.md b/docs/how-it-works.md new file mode 100644 index 0000000..9a30707 --- /dev/null +++ b/docs/how-it-works.md @@ -0,0 +1,51 @@ +# How it works + +A **provider** (Stripe, OpenAI, GitHub…) has one category and one or more **sources**: each one place it announces changes (an OpenAPI spec repo, a changelog, a deprecations page, a feed), read by one **kind** of plugin (`plugins//`), at one **layer** of a customer's stack. A source's id is `provider/source` (`stripe/api`, `openai/models`); its old slug (`stripe`, `openai-models`) is what events carry in `api` until patch-it's change-event contract switches to source ids. Every provider and source is registered in code with its metadata (`plugin.RegisterProvider`): that registration is the only list. + +## What a plugin does + +0. **List the versions.** A source with versions (a spec's releases, a package on PyPI or npm, a guide per major) lists every one, oldest first (`ListVersions`, from `core/versions`: GitHub releases, tags and commits, PyPI, npm, the Web Archive at a date). Each consecutive pair is read and compared once (`ReadAt`, `Extract`), so a customer several versions behind gets every step of the upgrade. A living page with no versions (a deprecations page, a feed) is read as it is now; each change to it is a dated version. +1. **Fetch.** The plugin asks its environment for the source (`env.Get`); it never opens connections itself. Pages that change in place are also cited through a Web Archive snapshot, so the link in an event keeps showing what the plugin read. +2. **Compare.** It compares two versions of the source: two OpenAPI specs, two versions of a deprecations page, a release's changelog section against the release before. `Run` does the newest version against the last one it saw (the baseline); `Extract` does any pair. +3. **Emit events.** Each change becomes one event in [patch-it's change-event format](https://github.com/patch-it-dev/patch-it/tree/main/docs/contract): what changed, how bad it is (breaking, deprecation, behavior change, non-breaking), the date it takes effect, and the source URL with a quote. **No source, no event.** When unsure, a plugin emits nothing: a wrong event would become a wrong pull request. + +Most providers are an entry for an existing kind, not new code. patch-it's hosted ingestion service runs every source on a schedule and serves the events to the patch-it GitHub Action and CLI. + +## What's covered + +There's no list in this repo: `make plugins` prints every provider and source from the registry as JSON, every release attaches it as `plugins.json`, and [docs.patch-it.dev](https://docs.patch-it.dev/supported-apis/plugins/) shows it. + +## Layout + +| Path | What's there | +|---|---| +| [`core/plugin/`](../core/plugin/) | The `Plugin` interface (`Run`; `ListVersions`, `ReadAt`, `Extract`), the registry of providers and sources (`RegisterProvider`, `Lookup`, `LookupSource`, `Providers`), `Env`, `Result` | +| [`core/versions/`](../core/versions/) | Every version of a source: GitHub releases, tags and commits, PyPI, npm, the Web Archive at a date | +| [`plugins/`](../plugins/) | One package per kind, with its providers; `plugins/all` imports every one | +| [`cmd/new-plugin/`](../cmd/new-plugin/), [`cmd/list-plugins/`](../cmd/list-plugins/) | The starter for a new provider; the list of providers and sources, printed from the registry | +| [`core/kit/`](../core/kit/), [`core/kit/kittest/`](../core/kit/kittest/), [`core/github/`](../core/github/) | What plugins share (the unchanged-page check, Wayback citing, the newest release or commit on GitHub) and the test kit (in-memory store, saved pages, golden files) | +| [`core/event/`](../core/event/), [`core/migrate/`](../core/migrate/) | The change-event types of [patch-it's contract](https://github.com/patch-it-dev/patch-it/tree/main/docs/contract), and the v1 → v2 conversion | +| [`core/diff/`](../core/diff/), [`core/normalize/`](../core/normalize/), [`core/hint/`](../core/hint/), [`plugins/openapi/adapter/`](../plugins/openapi/adapter/) | The OpenAPI diff engine: compare two specs, turn the changes into events, attach changelog hints; one adapter per spec provider | +| [`plugins/retirements/retire/`](../plugins/retirements/retire/), [`plugins/migrationguide/sdkguide/`](../plugins/migrationguide/sdkguide/), [`core/fetch/`](../core/fetch/) | The deprecations-page and migration-guide readers; the HTTP fetch and cache | +| [`contract/fixtures/`](../contract/) | A copy of patch-it's change-event fixtures | +| `plugin/`, `event/`, `fetch/`, `migrate/`, `kit/`, `kit/kittest/`, `retire/`, `sdkguide/` | Deprecated: the old import paths, forwarding to the packages above until code that imports them moves | +| [`providers/`](../providers/) | One page per provider: why that source, what the plugin reads | +| [`docs/`](./) | [How to write a plugin](writing-a-plugin.md), the [README template](README-template.md) patch-it's public repos share | + +## Run the checks + +```sh +make check # what CI runs: tidy, build, gofmt, vet, import and net/http guards, go test -race, no TODO(new-plugin) left, leak guard +``` + +The tests are offline: every plugin's test reads a saved copy of its source from `testdata/` and compares the events with a golden file. After an intended change to a plugin's output, `go test ./plugins/ -update` rewrites its golden files; review that diff like code. + +`GITHUB_TOKEN` (optional, no scopes) only raises GitHub's rate limit when the `openapi` adapters look up release tags; the tests never use it. Nothing else is read from the environment. + +## Releases + +Releases are `v0.x.y` tags on `main` with a [GitHub release](https://github.com/patch-it-dev/patch-it-plugins/releases) and generated notes. The **Release** workflow tags a patch release on its own every Monday and Thursday at 14:00 UTC when something new was merged, and maintainers can run it by hand ([how](../CONTRIBUTING.md#releasing)). patch-it's hosted ingestion service runs the latest release, so a merged plugin is running within a few days. Each release carries `plugins.json`, the list of providers and sources printed from the registry at that tag. Tags are never moved or deleted. + +```sh +go get github.com/patch-it-dev/patch-it-plugins@latest +``` diff --git a/docs/plugins.json b/docs/plugins.json deleted file mode 100644 index dba6b2a..0000000 --- a/docs/plugins.json +++ /dev/null @@ -1,307 +0,0 @@ -{ - "_comment": "Generated from the plugin registry and docs/plugins.md by `make plugins-json` (go run ./cmd/list-plugins). Don't edit by hand.", - "$schema": "./plugins.schema.json", - "count": 29, - "plugins": [ - { - "id": "algolia-ingestion", - "provider": "Algolia", - "layer": "api", - "watches": "Ingestion API operations deprecated, and deprecated operations removed, between releases of the spec", - "format": "openapi", - "source_url": "https://github.com/algolia/api-clients-automation", - "status": "live", - "provider_page": "providers/algolia.md" - }, - { - "id": "algolia-recommend", - "provider": "Algolia", - "layer": "api", - "watches": "Recommend API operations deprecated, and deprecated operations removed, between releases of the spec", - "format": "openapi", - "source_url": "https://github.com/algolia/api-clients-automation", - "status": "live", - "provider_page": "providers/algolia.md" - }, - { - "id": "algolia-search", - "provider": "Algolia", - "layer": "api", - "watches": "Search API operations deprecated, and deprecated operations removed, between releases of the spec", - "format": "openapi", - "source_url": "https://github.com/algolia/api-clients-automation", - "status": "live", - "provider_page": "providers/algolia.md" - }, - { - "id": "anthropic", - "provider": "Anthropic", - "layer": "sdk", - "watches": "Parameters removed from `messages.create()` and friends in the Python SDK 1.0", - "format": "migrationguide", - "source_url": "https://github.com/anthropics/anthropic-sdk-python/blob/v1.0.0/MIGRATION.md", - "status": "live", - "provider_page": "providers/anthropic.md" - }, - { - "id": "anthropic-models", - "provider": "Anthropic", - "layer": "ai-models", - "watches": "Model deprecations and retirement dates (Messages API models)", - "format": "retirements", - "source_url": "https://platform.claude.com/docs/en/about-claude/model-deprecations", - "status": "live", - "provider_page": "providers/anthropic-models.md" - }, - { - "id": "appwrite", - "provider": "Appwrite", - "layer": "api", - "watches": "REST API operations a server SDK calls that are deprecated between commits of the spec, SDK method renames left out", - "format": "openapi", - "source_url": "https://github.com/appwrite/specs", - "status": "live", - "provider_page": "providers/appwrite.md" - }, - { - "id": "cloudflare", - "provider": "Cloudflare", - "layer": "api", - "watches": "REST API operations Cloudflare deprecates, breaking on their end-of-life date when it gives one", - "format": "feed", - "source_url": "https://developers.cloudflare.com/fundamentals/api/reference/deprecations/index.xml", - "status": "live", - "provider_page": "providers/cloudflare.md" - }, - { - "id": "convex", - "provider": "Convex", - "layer": "api", - "watches": "Deployment API operations deprecated, and deprecated operations removed, between commits of the spec", - "format": "openapi", - "source_url": "https://github.com/get-convex/convex-backend", - "status": "live", - "provider_page": "providers/convex.md" - }, - { - "id": "datadog", - "provider": "Datadog", - "layer": "api", - "watches": "API v2 changes between OpenAPI spec releases", - "format": "openapi", - "source_url": "https://github.com/DataDog/datadog-api-client-python", - "status": "live", - "provider_page": "providers/datadog.md" - }, - { - "id": "elasticsearch", - "provider": "Elastic", - "layer": "api", - "watches": "Elasticsearch REST API operations deprecated, and deprecated operations removed, between released minor versions of the spec", - "format": "openapi", - "source_url": "https://github.com/elastic/elasticsearch-specification", - "status": "live", - "provider_page": "providers/elasticsearch.md" - }, - { - "id": "flagsmith", - "provider": "Flagsmith", - "layer": "api", - "watches": "REST API changes between releases of Flagsmith's own OpenAPI spec (removed endpoints and request parameters)", - "format": "openapi", - "source_url": "https://github.com/Flagsmith/flagsmith/blob/main/openapi.yaml", - "status": "live", - "provider_page": "providers/flagsmith.md" - }, - { - "id": "github", - "provider": "GitHub", - "layer": "api", - "watches": "REST API changes between dated API versions", - "format": "openapi", - "source_url": "https://github.com/github/rest-api-description", - "status": "live", - "provider_page": "providers/github.md" - }, - { - "id": "github-actions", - "provider": "GitHub", - "layer": "ci", - "watches": "GitHub Actions retirements, such as runner images and the `runs-on` labels they remove", - "format": "feed", - "source_url": "https://github.blog/changelog/label/actions/feed/", - "status": "live", - "provider_page": "providers/github-actions.md" - }, - { - "id": "gitlab", - "provider": "GitLab", - "layer": "api", - "watches": "REST API changes between releases, request side only (what callers send)", - "format": "openapi", - "source_url": "https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/api/openapi", - "status": "live", - "provider_page": "providers/gitlab.md" - }, - { - "id": "mongodb-atlas", - "provider": "MongoDB", - "layer": "api", - "watches": "Atlas Administration API versions deprecated and removed at their sunset date, per endpoint, from MongoDB's own API changelog", - "format": "apichangelog", - "source_url": "https://github.com/mongodb/openapi/blob/main/changelog/changelog.json", - "status": "live", - "provider_page": "providers/mongodb-atlas.md" - }, - { - "id": "neon", - "provider": "Neon", - "layer": "api", - "watches": "Neon API operations deprecated, and deprecated operations removed, between commits of the spec", - "format": "openapi", - "source_url": "https://github.com/neondatabase/neon-pkgs", - "status": "live", - "provider_page": "providers/neon.md" - }, - { - "id": "newrelic", - "provider": "New Relic", - "layer": "sdk", - "watches": "Members removed in the New Relic Python agent 11.0.0", - "format": "removedmembers", - "source_url": "https://github.com/newrelic/docs-website/blob/f0b4f817bf9a7b76a78e26609369a8a7ce764c8e/src/content/docs/release-notes/agent-release-notes/python-release-notes/python-agent-110000.mdx", - "status": "live", - "provider_page": "providers/newrelic.md" - }, - { - "id": "numpy", - "provider": "NumPy", - "layer": "package", - "watches": "Members removed or renamed in the main namespace in NumPy 2.0", - "format": "removedmembers", - "source_url": "https://github.com/numpy/numpy/blob/v2.0.0/doc/source/numpy_2_0_migration_guide.rst", - "status": "live", - "provider_page": "providers/numpy.md" - }, - { - "id": "openai-models", - "provider": "OpenAI", - "layer": "ai-models", - "watches": "Model deprecations and shutdown dates (Chat Completions and Responses text models)", - "format": "retirements", - "source_url": "https://developers.openai.com/api/docs/deprecations", - "status": "live", - "provider_page": "providers/openai-models.md" - }, - { - "id": "planetscale", - "provider": "PlanetScale", - "layer": "api", - "watches": "API operations deprecated, and deprecated operations removed, between commits of the spec, said in the operation's description", - "format": "openapi", - "source_url": "https://github.com/planetscale/terraform-provider-planetscale", - "status": "live", - "provider_page": "providers/planetscale.md" - }, - { - "id": "posthog", - "provider": "PostHog", - "layer": "sdk", - "watches": "Members removed in the posthog Python SDK 8.0.0", - "format": "removedmembers", - "source_url": "https://github.com/PostHog/posthog-python/blob/posthog-v8.0.0/docs/migration-7.x-to-8.0.md", - "status": "live", - "provider_page": "providers/posthog.md" - }, - { - "id": "redis", - "provider": "Redis", - "layer": "runtime", - "watches": "Commands removed entirely from Redis's own command catalog between two snapshots", - "format": "catalog", - "source_url": "https://github.com/redis/redis-doc/blob/master/commands.json", - "status": "live", - "provider_page": "providers/redis.md" - }, - { - "id": "salesforce", - "provider": "Salesforce", - "layer": "api", - "watches": "REST API version ranges deprecated and retired, from Salesforce's own API end-of-life policy page", - "format": "eol", - "source_url": "https://developer.salesforce.com/docs/platform/api-rest/guide/api-rest-eol.html", - "status": "live", - "provider_page": "providers/salesforce.md" - }, - { - "id": "sentry", - "provider": "Sentry", - "layer": "api", - "watches": "REST API operations deprecated between releases of the spec, most of them in the operation's description", - "format": "openapi", - "source_url": "https://github.com/getsentry/sentry/releases", - "status": "live", - "provider_page": "providers/sentry.md" - }, - { - "id": "stripe", - "provider": "Stripe", - "layer": "api", - "watches": "REST API changes between OpenAPI spec releases (removed and renamed endpoints, parameters and fields), with migration hints from Stripe's API changelog", - "format": "openapi", - "source_url": "https://github.com/stripe/openapi", - "status": "live", - "provider_page": "providers/stripe.md" - }, - { - "id": "supabase", - "provider": "Supabase", - "layer": "api", - "watches": "Management API operations deprecated, and deprecated operations removed, between commits of the spec", - "format": "openapi", - "source_url": "https://github.com/supabase/supabase", - "status": "live", - "provider_page": "providers/supabase.md" - }, - { - "id": "textual", - "provider": "Textual", - "layer": "package", - "watches": "Removals and breaking changes in release 0.86.0", - "format": "changelog", - "source_url": "https://github.com/Textualize/textual/blob/v0.86.0/CHANGELOG.md", - "status": "live", - "provider_page": "providers/textual.md" - }, - { - "id": "urllib3", - "provider": "urllib3", - "layer": "package", - "watches": "Removals and deprecations in release 2.0.0", - "format": "changelog", - "source_url": "https://github.com/urllib3/urllib3/blob/2.0.0/CHANGES.rst", - "status": "live", - "provider_page": "providers/urllib3.md" - }, - { - "id": "vercel", - "provider": "Vercel", - "layer": "api", - "watches": "REST API operations deprecated, and deprecated operations removed, between SDK releases of the spec", - "format": "openapi", - "source_url": "https://github.com/vercel/sdk", - "status": "live", - "provider_page": "providers/vercel.md" - }, - { - "id": "vertex-ai-models", - "provider": "Google", - "layer": "ai-models", - "watches": "Generative AI model deprecations on Vertex AI", - "format": "feed", - "source_url": "https://docs.cloud.google.com/feeds/generative-ai-on-vertex-ai-release-notes.xml", - "status": "live", - "provider_page": "providers/vertex-ai-models.md" - } - ] -} diff --git a/docs/plugins.md b/docs/plugins.md deleted file mode 100644 index 9d6f05f..0000000 --- a/docs/plugins.md +++ /dev/null @@ -1,56 +0,0 @@ -# Supported plugins - -Every source patch-it watches is one plugin. This is the list of them: **if a plugin isn't here, it doesn't exist yet**. Add a row when you add a plugin, and update it when you change one (see [AGENTS.md](../AGENTS.md)). - -Each plugin's id links to its provider page in [`providers/`](../providers/): why that source, what the plugin reads and ignores, and its known gaps. How to add one: [writing-a-plugin.md](writing-a-plugin.md). - -The table between the `plugins:start` and `plugins:end` markers is also published on [docs.patch-it.dev](https://docs.patch-it.dev/supported-apis/plugins/): the docs site reads this file from `main` when it builds. Keep it a plain Markdown table, one row per plugin, with absolute links only. [`plugins.json`](plugins.json) is the same list as JSON (id, provider, layer, format, source URL, status, provider page; schema in [`plugins.schema.json`](plugins.schema.json)), generated by `make plugins-json`. - - -| Plugin (`--api`) | Provider | Layer | What it watches | Format plugin | Source | Status | -|---|---|---|---|---|---|---| -| [`stripe`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/stripe.md) | Stripe | API | REST API changes between OpenAPI spec releases (removed and renamed endpoints, parameters and fields), with migration hints from Stripe's API changelog | `openapi` | [stripe/openapi](https://github.com/stripe/openapi) | live | -| [`datadog`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/datadog.md) | Datadog | API | API v2 changes between OpenAPI spec releases | `openapi` | [DataDog/datadog-api-client-python](https://github.com/DataDog/datadog-api-client-python) (`.generator/schemas/v2/openapi.yaml`) | live | -| [`github`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/github.md) | GitHub | API | REST API changes between dated API versions | `openapi` | [github/rest-api-description](https://github.com/github/rest-api-description) | live | -| [`gitlab`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/gitlab.md) | GitLab | API | REST API changes between releases, request side only (what callers send) | `openapi` | [gitlab-org/gitlab `doc/api/openapi`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/api/openapi) | live | -| [`vercel`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/vercel.md) | Vercel | API | REST API operations deprecated, and deprecated operations removed, between SDK releases of the spec | `openapi` | [vercel/sdk](https://github.com/vercel/sdk) (`vercel-spec.json`) | live | -| [`supabase`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/supabase.md) | Supabase | API | Management API operations deprecated, and deprecated operations removed, between commits of the spec | `openapi` | [supabase/supabase](https://github.com/supabase/supabase) (`apps/docs/spec/api_v1_openapi.json`) | live | -| [`algolia-search`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/algolia.md) | Algolia | API | Search API operations deprecated, and deprecated operations removed, between releases of the spec | `openapi` | [algolia/api-clients-automation](https://github.com/algolia/api-clients-automation) (`specs/bundled/search.yml`) | live | -| [`algolia-ingestion`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/algolia.md) | Algolia | API | Ingestion API operations deprecated, and deprecated operations removed, between releases of the spec | `openapi` | [algolia/api-clients-automation](https://github.com/algolia/api-clients-automation) (`specs/bundled/ingestion.yml`) | live | -| [`algolia-recommend`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/algolia.md) | Algolia | API | Recommend API operations deprecated, and deprecated operations removed, between releases of the spec | `openapi` | [algolia/api-clients-automation](https://github.com/algolia/api-clients-automation) (`specs/bundled/recommend.yml`) | live | -| [`elasticsearch`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/elasticsearch.md) | Elastic | API | Elasticsearch REST API operations deprecated, and deprecated operations removed, between released minor versions of the spec | `openapi` | [elastic/elasticsearch-specification](https://github.com/elastic/elasticsearch-specification) (`output/openapi/elasticsearch-openapi.json`) | live | -| [`sentry`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/sentry.md) | Sentry | API | REST API operations deprecated between releases of the spec, most of them in the operation's description | `openapi` | [getsentry/sentry releases](https://github.com/getsentry/sentry/releases) (`openapi-derefed.json` on each `api@` release) | live | -| [`neon`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/neon.md) | Neon | API | Neon API operations deprecated, and deprecated operations removed, between commits of the spec | `openapi` | [neondatabase/neon-pkgs](https://github.com/neondatabase/neon-pkgs) (`packages/sdk/spec/neon-openapi.json`) | live | -| [`convex`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/convex.md) | Convex | API | Deployment API operations deprecated, and deprecated operations removed, between commits of the spec | `openapi` | [get-convex/convex-backend](https://github.com/get-convex/convex-backend) (`npm-packages/@convex-dev/platform/deployment-openapi.json`) | live | -| [`planetscale`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/planetscale.md) | PlanetScale | API | API operations deprecated, and deprecated operations removed, between commits of the spec, said in the operation's description | `openapi` | [planetscale/terraform-provider-planetscale](https://github.com/planetscale/terraform-provider-planetscale) (`schemas/openapi.yaml`) | live | -| [`mongodb-atlas`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/mongodb-atlas.md) | MongoDB | API | Atlas Administration API versions deprecated and removed at their sunset date, per endpoint, from MongoDB's own API changelog | `apichangelog` | [mongodb/openapi `changelog/changelog.json`](https://github.com/mongodb/openapi/blob/main/changelog/changelog.json) | live | -| [`openai-models`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/openai-models.md) | OpenAI | AI models | Model deprecations and shutdown dates (Chat Completions and Responses text models) | `retirements` | [OpenAI deprecations](https://developers.openai.com/api/docs/deprecations) | live | -| [`anthropic-models`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/anthropic-models.md) | Anthropic | AI models | Model deprecations and retirement dates (Messages API models) | `retirements` | [Anthropic model deprecations](https://platform.claude.com/docs/en/about-claude/model-deprecations) | live | -| [`anthropic`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/anthropic.md) | Anthropic | SDK | Parameters removed from `messages.create()` and friends in the Python SDK 1.0 | `migrationguide` | [anthropic-sdk-python MIGRATION.md](https://github.com/anthropics/anthropic-sdk-python/blob/v1.0.0/MIGRATION.md) | live | -| [`numpy`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/numpy.md) | NumPy | package | Members removed or renamed in the main namespace in NumPy 2.0 | `removedmembers` | [NumPy 2.0 migration guide](https://github.com/numpy/numpy/blob/v2.0.0/doc/source/numpy_2_0_migration_guide.rst) | live | -| [`textual`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/textual.md) | Textual | package | Removals and breaking changes in release 0.86.0 | `changelog` | [Textualize/textual CHANGELOG.md](https://github.com/Textualize/textual/blob/v0.86.0/CHANGELOG.md) | live | -| [`urllib3`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/urllib3.md) | urllib3 | package | Removals and deprecations in release 2.0.0 | `changelog` | [urllib3/urllib3 CHANGES.rst](https://github.com/urllib3/urllib3/blob/2.0.0/CHANGES.rst) | live | -| [`github-actions`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/github-actions.md) | GitHub | CI | GitHub Actions retirements, such as runner images and the `runs-on` labels they remove | `feed` | [GitHub changelog, Actions label (RSS)](https://github.blog/changelog/label/actions/feed/) | live | -| [`vertex-ai-models`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/vertex-ai-models.md) | Google | AI models | Generative AI model deprecations on Vertex AI | `feed` | [Vertex AI generative AI release notes (Atom)](https://docs.cloud.google.com/feeds/generative-ai-on-vertex-ai-release-notes.xml) | live | -| [`redis`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/redis.md) | Redis | runtime | Commands removed entirely from Redis's own command catalog between two snapshots | `catalog` | [redis/redis-doc `commands.json`](https://github.com/redis/redis-doc/blob/master/commands.json) | live | -| [`cloudflare`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/cloudflare.md) | Cloudflare | API | REST API operations Cloudflare deprecates, breaking on their end-of-life date when it gives one | `feed` | [Cloudflare API deprecations (RSS)](https://developers.cloudflare.com/fundamentals/api/reference/deprecations/index.xml) | live | -| [`posthog`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/posthog.md) | PostHog | SDK | Members removed in the posthog Python SDK 8.0.0 | `removedmembers` | [posthog-python 7.x to 8.0 migration guide](https://github.com/PostHog/posthog-python/blob/posthog-v8.0.0/docs/migration-7.x-to-8.0.md) | live | -| [`newrelic`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/newrelic.md) | New Relic | SDK | Members removed in the New Relic Python agent 11.0.0 | `removedmembers` | [Python agent 11.0.0 release notes](https://github.com/newrelic/docs-website/blob/f0b4f817bf9a7b76a78e26609369a8a7ce764c8e/src/content/docs/release-notes/agent-release-notes/python-release-notes/python-agent-110000.mdx) | live | -| [`appwrite`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/appwrite.md) | Appwrite | API | REST API operations a server SDK calls that are deprecated between commits of the spec, SDK method renames left out | `openapi` | [appwrite/specs](https://github.com/appwrite/specs) (`specs/.x/open-api3-.x.json`, the newest minor) | live | -| [`salesforce`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/salesforce.md) | Salesforce | API | REST API version ranges deprecated and retired, from Salesforce's own API end-of-life policy page | `eol` | [Salesforce API End-of-Life Policy](https://developer.salesforce.com/docs/platform/api-rest/guide/api-rest-eol.html) | live | -| [`flagsmith`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/flagsmith.md) | Flagsmith | API | REST API changes between releases of Flagsmith's own OpenAPI spec (removed endpoints and request parameters) | `openapi` | [github.com/Flagsmith/flagsmith/blob/main/openapi.yaml](https://github.com/Flagsmith/flagsmith/blob/main/openapi.yaml) | live | - - -## The columns - -| Column | Meaning | -|---|---| -| Plugin | The id you pass as `--api` and the `api` field on every event it emits. Lowercase, unique. | -| Provider | Whose API, SDK or package it is. | -| Layer | Which kind of dependency it covers: `API`, `SDK`, `package`, `runtime`, `CI` or `AI models`. | -| What it watches | The kind of change it reports, in a few words. | -| Format plugin | Which shared reader it is built on: `openapi` (a spec per release, diffed), `retirements` (a deprecations page), `migrationguide` (an SDK's migration guide), `removedmembers` (a package's table of removed members, in RST or Markdown), `changelog` (a release changelog file), `feed` (an RSS or Atom feed), `catalog` (a structured JSON catalog of names, diffed between two snapshots), `apichangelog` (a vendor's machine-readable API changelog, in JSON), `eol` (a vendor's own table of whole API version ranges and their support status). | -| Source | The provider's own public page or file the plugin reads. Every event links back to it. | -| Status | `live`: in the code on `main`. patch-it's hosted ingestion service runs it daily from the next release, which is cut automatically. `planned`: accepted, not written yet. `paused`: in the code but deliberately switched off, with the reason in the row. | - -Some plugins read one pinned release (`anthropic` 1.0.0, `newrelic` 11.0.0, `numpy` 2.0.0, `posthog` 8.0.0, `textual` 0.86.0, `urllib3` 2.0.0): a newer release of the same package is a new entry in that plugin, not a new row, unless it changes the format. diff --git a/docs/plugins.schema.json b/docs/plugins.schema.json index 35b1cad..2033d81 100644 --- a/docs/plugins.schema.json +++ b/docs/plugins.schema.json @@ -2,30 +2,49 @@ "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://github.com/patch-it-dev/patch-it-plugins/blob/main/docs/plugins.schema.json", "title": "patch-it plugins", - "description": "docs/plugins.json: every registered plugin, generated by `make plugins-json` from the plugin registry and docs/plugins.md.", + "description": "plugins.json: every registered provider and its sources, printed by `go run ./cmd/list-plugins` from the plugin registry and attached to each release. Never committed.", "type": "object", - "required": ["_comment", "$schema", "count", "plugins"], + "required": ["_comment", "$schema", "version", "provider_count", "source_count", "categories", "providers"], "additionalProperties": false, "properties": { "_comment": { "type": "string" }, "$schema": { "type": "string" }, - "count": { "type": "integer", "description": "How many plugins are listed." }, - "plugins": { + "version": { "type": "string", "minLength": 1, "description": "The release tag it was built from, or main." }, + "provider_count": { "type": "integer" }, + "source_count": { "type": "integer" }, + "categories": { + "type": "array", + "description": "Every category, in display order.", + "items": { "type": "string", "enum": ["ai", "payments", "infrastructure", "databases", "monitoring", "analytics", "auth", "messaging", "hosting", "search", "devtools", "crm", "packages"] } + }, + "providers": { "type": "array", "items": { "type": "object", - "required": ["id", "provider", "layer", "watches", "format", "source_url", "status", "provider_page"], + "required": ["id", "name", "category", "homepage", "sources"], "additionalProperties": false, "properties": { - "id": { "type": "string", "pattern": "^[a-z0-9][a-z0-9.-]*$", "description": "The plugin's --api id and every event's api field." }, - "provider": { "type": "string", "minLength": 1 }, - "layer": { "type": "string", "enum": ["api", "sdk", "package", "runtime", "ci", "ai-models"] }, - "watches": { "type": "string", "minLength": 1 }, - "format": { "type": "string", "enum": ["openapi", "retirements", "migrationguide", "removedmembers", "changelog", "feed", "apichangelog", "catalog", "eol"] }, - "source_url": { "type": "string", "pattern": "^https://" }, - "status": { "type": "string", "enum": ["live", "paused", "planned"] }, - "status_note": { "type": "string", "description": "Why, for a paused plugin." }, - "provider_page": { "type": "string", "pattern": "^providers/[a-z0-9][a-z0-9.-]*\\.md$", "description": "One page per provider: plugins of the same provider may share it." } + "id": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]*$" }, + "name": { "type": "string", "minLength": 1 }, + "category": { "type": "string", "enum": ["ai", "payments", "infrastructure", "databases", "monitoring", "analytics", "auth", "messaging", "hosting", "search", "devtools", "crm", "packages"] }, + "homepage": { "type": "string", "pattern": "^https://" }, + "sources": { + "type": "array", + "items": { + "type": "object", + "required": ["id", "alias", "kind", "layer", "watches", "urls", "page"], + "additionalProperties": false, + "properties": { + "id": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]*/[a-z0-9][a-z0-9-]*$", "description": "provider/source" }, + "alias": { "type": "string", "pattern": "^[a-z0-9][a-z0-9.-]*$", "description": "The old slug: --api and every event's api field until patch-it's change-event contract switches to source ids." }, + "kind": { "type": "string", "enum": ["openapi", "retirements", "migrationguide", "removedmembers", "changelog", "feed", "apichangelog", "catalog", "eol"] }, + "layer": { "type": "string", "enum": ["api", "sdk", "package", "runtime", "ci", "ai-models", "infra"] }, + "watches": { "type": "string", "minLength": 1 }, + "urls": { "type": "array", "items": { "type": "string", "pattern": "^https://" } }, + "page": { "type": "string", "pattern": "^https://github\\.com/patch-it-dev/patch-it-plugins/blob/[^/]+/providers/[a-z0-9][a-z0-9.-]*\\.md$" } + } + } + } } } } diff --git a/docs/writing-a-plugin.md b/docs/writing-a-plugin.md index 9e08c71..58aab2b 100644 --- a/docs/writing-a-plugin.md +++ b/docs/writing-a-plugin.md @@ -1,29 +1,28 @@ # Writing a plugin -A plugin watches one provider source and turns what changed into change events. This guide walks through adding one, from picking a source to the PR. The rules are in [CONTRIBUTING.md](../CONTRIBUTING.md); this is the how-to. +A **provider** (Stripe, OpenAI, Kubernetes…) has a **category** and one or more **sources**. A source is one place the provider announces changes; it is read by one **kind** of plugin (the `plugins/` package: `openapi`, `changelog`, `feed`, …), covers one **layer** (`api`, `sdk`, `package`, `runtime`, `ci`, `ai-models`, `infra`), and has **versions**. Its id is `provider/source` (`stripe/api`, `algolia/search-api`). This guide walks through adding a source, from picking it to the PR. The rules are in [CONTRIBUTING.md](../CONTRIBUTING.md); this is the how-to. How the pieces fit: [how-it-works.md](how-it-works.md). -**Most new providers don't need new code.** They are a new entry for an existing format plugin: a new `changelog.File`, a new `retire.Page`, a new `feed.Feed` reader, a new OpenAPI adapter. Write a new format plugin only when none of them reads your source. +**Most new sources don't need new code.** They are a new entry for an existing kind: a new `changelog.File`, a new `retire.Page`, a new `feed.Feed` reader, a new OpenAPI adapter. Write a new kind only when none of them reads your source. There is no "Kubernetes kind" or "Terraform kind": Kubernetes API removals are a migration-guide or notices source, a client SDK is a package source, a Terraform provider is a changelog source. ## 1. Scaffold it with `new-plugin` -Pick the source and the format plugin that reads it (steps 2 and 3), then let the starter write everything around the provider: +Pick the source and the kind that reads it (steps 2 and 3), then let the starter write everything around the provider: ```sh go run ./cmd/new-plugin --api acme --provider Acme --format openapi # --format: openapi, retirements, migrationguide, removedmembers, changelog, feed, apichangelog, catalog -# optional: --source https://... (the provider's source), --watches "...", --layer API|SDK|package|runtime|CI|"AI models", -# --page acme (a second plugin of a provider shares its page in providers/) +# optional: --source https://... (the provider's source), --watches "...", --layer api|sdk|package|runtime|ci|ai-models|infra, +# --page acme (a second source of a provider shares its page in providers/) ``` It writes, and lists: -- the provider's definition, in the package the format reads it from (`adapter/`, `retire/`, `sdkguide/` or `plugins//`), and its `plugin.Register` line in that format's `init` (plus the `migrate/` map entry that `retirements` and `migrationguide` need); +- the provider's definition, in the package the format reads it from (`plugins/openapi/adapter/`, `plugins/retirements/retire/`, `plugins/migrationguide/sdkguide/` or `plugins//`), and its `plugin.RegisterProvider` block (the provider and the source with their metadata) in that format's `init` (plus the `core/migrate/` map entry that `retirements` and `migrationguide` need); - `plugins//_test.go`: the registration test, a golden test and a pinned drop test, over placeholder sources in `testdata/`, with the golden file written from them; - a row per saved file in that `testdata/README.md` (URL, date, license, quote to fill in); - `providers/.md` from the template (or nothing, with `--page`, when the provider already has one); -- the row in `docs/plugins.md`, and `docs/plugins.json` regenerated. -`go test ./...` passes right after; every place to fill in says `TODO(new-plugin)` (`grep -rn 'TODO(new-plugin)' .`), and `make check` (its `todos` step, also in CI_go) fails until none is left. It refuses an id that's taken (registered, in `docs/plugins.md`, or a file it would write exists) and then changes nothing. The steps below are what you fill in. +`go test ./...` passes right after; every place to fill in says `TODO(new-plugin)` (`grep -rn 'TODO(new-plugin)' .`), and `make check` (its `todos` step, also in CI_go) fails until none is left. It refuses an id that's taken (registered, or a file it would write exists) and then changes nothing. The steps below are what you fill in. ## 2. Pick the source @@ -60,7 +59,7 @@ Then check what's left against the provider's own words: Write the numbers on the provider page, as `providers/vercel.md` does for its spec's history, so a reviewer can check the rule against them. -### Pin, and pick a version a repo pins +### Cite a version, and pick one a repo pins - **Cite what never changes:** a release tag, or the commit SHA of the release or of a branch's head. A moving branch or a live URL isn't a citation. - **"Newest" needs care:** @@ -71,15 +70,15 @@ Write the numbers on the provider page, as `providers/vercel.md` does for its sp - The patch-it client gates a package version as a package. It keeps a date-versioned API's events only if the API is in its list of date-versioned APIs, so **a new date-versioned API needs an entry there too**. - Two specs of one version (fixes on a release branch) aren't a release: `RemovalsNeedANewVersion` drops removals between them. -## 3. Pick the format plugin +## 3. Pick the kind -| Format plugin | Reads | Add a provider by | +| Kind | Reads | Add a source by | |---|---|---| -| `plugins/openapi` | An OpenAPI spec per release, diffed with `diff/` and turned into events by `normalize/` | An `adapter.Adapter` in `adapter/` (latest tag, fetch a tag's spec) and a `plugin.Register(Plugin{Adapter: …})` line | -| `plugins/retirements` | A model provider's deprecations page (a table of date, model, replacement) | A `retire.Page` in `retire/providers.go` | -| `plugins/migrationguide` | An SDK's migration guide at a release tag (removed request parameters) | A `sdkguide.Guide` in `sdkguide/` | +| `plugins/openapi` | An OpenAPI spec per release, diffed with `core/diff/` and turned into events by `core/normalize/` | An `adapter.Adapter` in `plugins/openapi/adapter/` (latest tag, fetch a tag's spec; `Versions` to list every release) and a `plugin.RegisterProvider` block | +| `plugins/retirements` | A model provider's deprecations page (a table of date, model, replacement) | A `retire.Page` in `plugins/retirements/retire/providers.go` | +| `plugins/migrationguide` | An SDK's migration guide at the tag of each new major (removed request parameters) | A `sdkguide.Guide` in `plugins/migrationguide/sdkguide/` | | `plugins/removedmembers` | A package's table of removed members at a release tag (an RST simple table or a Markdown pipe table) | A `removedmembers.Guide` | -| `plugins/changelog` | A `CHANGELOG.md` / `CHANGES.rst` / `HISTORY.rst` at a release tag | A `changelog.File` | +| `plugins/changelog` | A `CHANGELOG.md` / `CHANGES.rst` / `HISTORY.rst`, each release's section at its tag (releases from PyPI or npm) | A `changelog.File` with its `TagPrefix` | | `plugins/feed` | An RSS or Atom feed | A `feed.Feed` with a reader for that vendor's entries | | `plugins/catalog` | A flat JSON catalog, one entry per name, with no release tags to pin — a commit SHA stands in | A `catalog.Catalog`, diffed between two snapshots for names that disappeared entirely | | `plugins/apichangelog` | A vendor's machine-readable API changelog in JSON (per day, endpoint and API version, oasdiff change codes), at the commit that last changed it | An `apichangelog.Log` | @@ -87,17 +86,12 @@ Write the numbers on the provider page, as `providers/vercel.md` does for its sp ## 4. The interface -A plugin implements `plugin.Plugin` (package [`plugin/`](../plugin/plugin.go)): +A plugin implements `plugin.Plugin` (package [`core/plugin/`](../core/plugin/plugin.go)): ```go type Plugin interface { - Name() string // "stripe": the --api id and every event's `api` - Run(ctx context.Context, env Env) (Result, error) // what changed since the baseline it saved last -} - -// Optional: diff two given releases, to recreate a historical change. -type Replayer interface { - Replay(ctx context.Context, env Env, from, to string) (Result, error) + Name() string // the source's old slug: --api and every event's `api` + Run(ctx context.Context, env Env) (Result, error) // the newest version against the baseline it saved last } type Env struct { @@ -114,33 +108,61 @@ type Result struct { } ``` -A plugin **stores nothing**. It reads its source, compares it with its last baseline (`env.Baselines.LastSpec(api)`), and returns events and the new baseline. Whoever runs it (patch-it's hosted ingestion service, or the test kit) stores the events first and the baseline after, so a crash in between re-reads the same change next time and the fingerprints dedupe it. +A plugin **stores nothing**. Whoever runs it (patch-it's hosted ingestion service, or the test kit) stores the events first and the baseline after, so a crash in between re-reads the same change next time and the fingerprints dedupe it. + +### Every version, not only the newest + +A source that has versions (a spec's releases, a package's releases, a guide per major) also implements the three parts of `plugin.Versioned` ([`core/plugin/versions.go`](../core/plugin/versions.go)), so a customer several versions behind gets every step: + +```go +ListVersions(ctx, env) ([]Version, error) // every version, oldest first +ReadAt(ctx, env, v Version) (Snapshot, error) // the source at one version +Extract(ctx, env, from, to Snapshot) (Result, error) // the changes from one to the next +``` + +- **List** with [`core/versions`](../core/versions/versions.go): `GitHubReleases`, `GitHubTags`, `GitHubCommits` (the commits that changed a file), `PyPIReleases`, `NPMReleases`, `WaybackAt` (a page as it was on a date). Never pin a tag in code: a version you need for a golden test is a pinned *case* in the test (`changelog.Urllib3.At("2.0.0")`), not the registered source. +- **Read at** a version's `Ref` (a tag, a commit SHA, an archive URL) and return the exact URL read as the snapshot's `SourceURL`: it's the events' citation. +- **Extract** sets `previous_version` to `from`'s version; `from` may be zero (nothing before). It returns no baselines. +- `Run` is built from the same parts: the newest version, extracted against the last baseline. + +A living page with no versions of its own (a deprecations page, a feed) only has `Run`: it's read as it is now, and each change to it is a dated version. + +### Register the provider and its source -Register the plugin in its package's `init`: +In the kind's package `init`, with the metadata ([AGENTS.md](../AGENTS.md) rule 1): ```go func init() { - plugin.Register(Plugin{File: Textual0860}) + plugin.RegisterProvider(plugin.Provider{ + ID: "urllib3", Name: "urllib3", Category: plugin.CategoryPackages, Homepage: "https://urllib3.readthedocs.io", + Sources: []plugin.Source{{ + ID: "urllib3", Layer: plugin.LayerPackage, Page: "providers/urllib3.md", + Watches: "Names removed or deprecated in each release, …", + URLs: []string{"https://github.com/urllib3/urllib3/blob/main/CHANGES.rst"}, + Plugin: Plugin{File: Urllib3}, + }}, + }) } ``` -`Register` panics on a duplicate name or the name `all`. +`RegisterProvider` panics on missing metadata, a duplicate id, or a provider registered twice with a different name, category or homepage (a provider with sources in two kinds repeats the same ones). `plugin.Lookup` and `plugin.LookupSource` find a source by `provider/source` or by its old slug. -### Fetching: `env.Get` and `kit/` +### Fetching: `env.Get` and `core/kit/` -- Fetch **only** with `env.Get`. It caches by `ETag` / `Last-Modified`, retries, and in tests serves saved files. `CI_go` fails a package outside `fetch/`, `kit/` and `adapter/` that imports `net/http`. +- Fetch **only** with `env.Get`. It caches by `ETag` / `Last-Modified`, retries, and in tests serves saved files. `CI_go` fails a package outside `core/fetch/`, `core/kit/` and `plugins/openapi/adapter/` that imports `net/http`. - `kit.Unchanged(env, api, body)`: true when the page hashes to the last baseline. Return early: an unchanged source costs one GET and nothing else. - `kit.Cite(ctx, archiver, url, read, want)`: for a **live page**. Takes a Wayback Machine snapshot, reads it with your own reader and checks it says exactly what the live page said. Make the events from the snapshot and cite its URL. No snapshot, no events. -- `plugin.Baseline(api, version, tag, sourceURL, content, replay, now)`: the baseline to return. -- Older format plugins build v1 `event.ChangeEvent`s and convert them with `migrate.Fresh`; new code builds `event.EventV2` directly. +- `plugin.Baseline(api, version, tag, sourceURL, content, replay, now)`: the baseline `Run` returns. +- `github.LatestRelease`, `github.LatestCommit`, `github.Commit`, `github.Head` ([`core/github`](../core/github/github.go)): the newest release or commit on GitHub, one place; don't decode GitHub's responses yourself. +- Older kinds build v1 `event.ChangeEvent`s and convert them with `migrate.Fresh`; new code builds `event.EventV2` directly. ### Events -The event format is [patch-it's change-event contract](https://github.com/patch-it-dev/patch-it/tree/main/docs/contract); `event/` is its Go binding. Every event has a typed `subject` (`api_operation`, `model`, `package_release`, `ci_runtime`), a `change_type` (`breaking`, `deprecation`, …), a `version`, a `fingerprint` (`event.FingerprintV2`), and **a `source_url` and `source_excerpt`: no source, no event**. Call `Validate()`; an invalid event goes in `Result.Rejected`, not `Events`. +The event format is [patch-it's change-event contract](https://github.com/patch-it-dev/patch-it/tree/main/docs/contract); `core/event/` is its Go binding. Every event has a typed `subject` (`api_operation`, `model`, `package_release`, `ci_runtime`), a `change_type` (`breaking`, `deprecation`, …), a `version`, a `fingerprint` (`event.FingerprintV2`), and **a `source_url` and `source_excerpt`: no source, no event**. Call `Validate()`; an invalid event goes in `Result.Rejected`, not `Events`. ## 5. Save the source -Save a real copy of what the plugin reads under your package's `testdata/` (or the reader package's, like `retire/testdata/`). Add a row to that directory's `README.md`: file, URL, date fetched, license, and a short quote of the part the plugin reads. +Save a real copy of what the plugin reads under your package's `testdata/` (or the reader package's, like `plugins/retirements/retire/testdata/`). Add a row to that directory's `README.md`: file, URL, date fetched, license, and a short quote of the part the plugin reads. - **Openly licensed files** (MIT, BSD, Apache, CC BY): keep them whole. - **Vendor documentation pages**: trim to what the plugin reads (the tables, the sections), copy every kept line unchanged, mark it "excerpt for testing; © the vendor", and say what you left out. @@ -150,7 +172,7 @@ Save a real copy of what the plugin reads under your package's `testdata/` (or t ## 6. Write the golden test -The test kit, `kit/kittest`, runs the plugin with no network, no database and a fixed clock: +The test kit, `core/kit/kittest`, runs the plugin with no network, no database and a fixed clock: ```go func TestGolden(t *testing.T) { @@ -169,15 +191,15 @@ func TestGolden(t *testing.T) { - **One drop test**: a realistic input the plugin must *not* turn into an event (an unrelated section, a behavior change written as prose, a product name in a sentence). Precision over recall: a missed change is a bug we fix later; a wrong change is a wrong pull request in someone's repo. Make it prove something: also run the same input with the filter off, and assert the noise it drops is there. - **`registered_test.go`**: `kittest.Registered(t, Plugin{…})` checks the package registered that plugin under that name. -- **A new format plugin package**: add its blank import to [`plugins/all/all.go`](../plugins/all/all.go). `TestEveryPluginIsImported` fails until you do; the hosted service runs exactly what `plugins/all` imports. +- **A new kind (a new package under plugins/)**: add its blank import to [`plugins/all/all.go`](../plugins/all/all.go). `TestEveryPluginIsImported` fails until you do; the hosted service runs exactly what `plugins/all` imports. ## 8. Write the provider page Copy [`providers/_template.md`](../providers/_template.md) to `providers/.md` and fill in every section: what it watches; **why this source** (the link, the date you checked it, why it's the best place this provider announces breaking changes, the alternatives you didn't use and why, one short quote); how the provider announces changes; what the plugin reads and ignores; known gaps. -## 9. Add the row to `docs/plugins.md` +## 9. Fill in the metadata in the registration -**Not optional** ([AGENTS.md](../AGENTS.md) rule 1). Add one row between the `plugins:start` and `plugins:end` markers: the id linked to its provider page, the provider, what it watches, the format plugin, the source, the status. Absolute links only: the docs site copies the table as it is. Changing a plugin's id, source or what it watches updates its row in the same PR. Then run `make plugins-json` to regenerate `docs/plugins.json`; it fails, naming the problem, if the registry and the table disagree. +**Not optional** ([AGENTS.md](../AGENTS.md) rule 1). The `plugin.RegisterProvider` block the starter wrote is the only place the source is listed: the provider's id, name, category (`plugin.Categories()`) and homepage, and the source's id (`/`), layer (`plugin.Layers()`), what it watches, the URLs it reads and its page. A second source of a provider repeats the provider's id, name, category and homepage exactly. `make plugins` prints the list with it; each release attaches that list as `plugins.json`, which docs.patch-it.dev and patch-it's hosted ingestion service read. There's nothing else to update. ## 10. Check and open the PR diff --git a/event/event.go b/event/event.go index d17f1c9..02bec52 100644 --- a/event/event.go +++ b/event/event.go @@ -1,215 +1,50 @@ +// Package event is the old import path of github.com/patch-it-dev/patch-it-plugins/core/event. +// +// Deprecated: import github.com/patch-it-dev/patch-it-plugins/core/event +// instead. This package only forwards to it, so code written against the old +// path keeps building; it will be removed in a later release. package event -import ( - "crypto/sha256" - "encoding/hex" - "fmt" - "regexp" - "strings" - "time" - "unicode/utf8" -) +import moved "github.com/patch-it-dev/patch-it-plugins/core/event" +// Deprecated: the types live in github.com/patch-it-dev/patch-it-plugins/core/event. type ( - ChangeType string - AffectedKind string - Side string -) - -const ( - ChangeTypeBreaking ChangeType = "breaking" - ChangeTypeDeprecation ChangeType = "deprecation" - ChangeTypeNewFeature ChangeType = "new_feature" - ChangeTypeFix ChangeType = "fix" + Affected = moved.Affected + AffectedKind = moved.AffectedKind + ChangeEvent = moved.ChangeEvent + ChangeType = moved.ChangeType + EventV2 = moved.EventV2 + PackageChange = moved.PackageChange + Side = moved.Side + Subject = moved.Subject + SubjectType = moved.SubjectType ) +// Deprecated: the constants live in github.com/patch-it-dev/patch-it-plugins/core/event. const ( - AffectedKindOperation AffectedKind = "operation" - AffectedKindRequestParameter AffectedKind = "request_parameter" - AffectedKindSchema AffectedKind = "schema" - AffectedKindSchemaProperty AffectedKind = "schema_property" + AffectedKindOperation = moved.AffectedKindOperation + AffectedKindRequestParameter = moved.AffectedKindRequestParameter + AffectedKindSchema = moved.AffectedKindSchema + AffectedKindSchemaProperty = moved.AffectedKindSchemaProperty + ChangeTypeBreaking = moved.ChangeTypeBreaking + ChangeTypeDeprecation = moved.ChangeTypeDeprecation + ChangeTypeFix = moved.ChangeTypeFix + ChangeTypeNewFeature = moved.ChangeTypeNewFeature + PackageChangedDefault = moved.PackageChangedDefault + PackageRemovedParameter = moved.PackageRemovedParameter + PackageRemovedSymbol = moved.PackageRemovedSymbol + PackageRenamedSymbol = moved.PackageRenamedSymbol + SideRequest = moved.SideRequest + SideResponse = moved.SideResponse + SubjectAPIOperation = moved.SubjectAPIOperation + SubjectCIRuntime = moved.SubjectCIRuntime + SubjectModel = moved.SubjectModel + SubjectPackageRelease = moved.SubjectPackageRelease ) -const ( - SideRequest Side = "request" - SideResponse Side = "response" +// Deprecated: the functions and values live in github.com/patch-it-dev/patch-it-plugins/core/event. +var ( + DecodeV2 = moved.DecodeV2 + Fingerprint = moved.Fingerprint + FingerprintV2 = moved.FingerprintV2 ) - -type Affected struct { - Kind AffectedKind `json:"kind"` // kind of change - Side Side `json:"side"` // which side is breaking - Operation string `json:"operation,omitempty"` // if operation change, what changed - Schema string `json:"schema,omitempty"` // if schema change, what changed - Property string `json:"property,omitempty"` // if poperty change, what changed - // Value is the one value of Property the change is about, when it's about one: - // the retired model name for a model retirement ("gpt-4"). Optional. - Value string `json:"value,omitempty"` -} - -type ChangeEvent struct { - ID int64 `json:"id"` - Fingerprint string `json:"fingerprint"` - API string `json:"api"` - Version string `json:"version"` - PreviousVersion string `json:"previous_version,omitempty"` - ChangeType ChangeType `json:"change_type"` - Breaking bool `json:"breaking"` - Affected Affected `json:"affected"` - Description string `json:"description"` - MigrationHint string `json:"migration_hint,omitempty"` - SourceURL string `json:"source_url"` - SourcePointer string `json:"source_pointer,omitempty"` - SourceExcerpt string `json:"source_excerpt"` - Confidence float64 `json:"confidence"` - DetectedAt time.Time `json:"detected_at"` -} - -// Fingerprint identifies one change at one location, so re-running ingest over -// the same specs updates nothing instead of duplicating events. detail is what -// separates two changes at the same place: the enum value added, the old and -// new type, "added as required". Absent fields are empty strings, and the parts -// are joined with a separator that cannot appear in a path or a schema name. -func Fingerprint(api, version string, changeType ChangeType, affected Affected, detail string) string { - sum := sha256.Sum256([]byte(strings.Join([]string{ - api, - version, - string(changeType), - string(affected.Kind), - string(affected.Side), - affected.Operation, - affected.Schema, - affected.Property, - detail, - }, "|"))) - return "sha256:" + hex.EncodeToString(sum[:]) -} - -// fingerprintPattern is the contract's shape for a fingerprint (the schema's -// pattern in the change event contract). patch-it checks the same pattern. -var fingerprintPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`) - -func (e ChangeEvent) Validate() error { - if err := checkCommon(e.SourceURL, e.SourceExcerpt, e.Fingerprint, e.API, e.Version, - e.Description, e.DetectedAt, e.ChangeType, e.Breaking, e.Confidence); err != nil { - return err - } - - if err := e.Affected.validate(); err != nil { - return err - } - - // Rule 5 (continued): a value is a value of a property. - if e.Affected.Value != "" && e.Affected.Property == "" { - return fmt.Errorf("affected.value needs affected.property: it is a value of that property") - } - - return nil -} - -// checkCommon is the contract's rules for every field outside affected (v1) or -// subject (v2): the two versions differ only in what changed, not in the rest. -func checkCommon(sourceURL, excerpt, fingerprint, api, version, description string, - detectedAt time.Time, changeType ChangeType, breaking bool, confidence float64) error { - // Rule 1: provenance is mandatory — the anti-hallucination mechanism. - if sourceURL == "" { - return fmt.Errorf("source_url is required") - } - if excerpt == "" { - return fmt.Errorf("source_excerpt is required") - } - - // Rule 2: other required fields must be non-empty. - if fingerprint == "" { - return fmt.Errorf("fingerprint is required") - } - if !fingerprintPattern.MatchString(fingerprint) { - return fmt.Errorf("fingerprint must look like sha256:<64 lowercase hex>, got %q", fingerprint) - } - if api == "" { - return fmt.Errorf("api is required") - } - if version == "" { - return fmt.Errorf("version is required") - } - if description == "" { - return fmt.Errorf("description is required") - } - if detectedAt.IsZero() { - return fmt.Errorf("detected_at is required") - } - - // Rule 3: change_type must be one of the known enum values. - if changeType != ChangeTypeBreaking && - changeType != ChangeTypeDeprecation && - changeType != ChangeTypeNewFeature && - changeType != ChangeTypeFix { - return fmt.Errorf("change_type must be a valid enum value, got %q", changeType) - } - - // Rule 4: breaking must agree with change_type. - if breaking != (changeType == ChangeTypeBreaking) { - return fmt.Errorf("breaking (%v) must equal change_type == breaking", breaking) - } - - // Rule 6: confidence must be in [0, 1]. - if confidence < 0 || confidence > 1 { - return fmt.Errorf("confidence must be between 0 and 1, got %v", confidence) - } - - // Rule 7: source_excerpt is capped at 2000 characters. Count runes, not - // bytes: spec text is full of non-ASCII (Stripe writes "invoice\u2019s"), and - // len() would reject excerpts well under the documented limit. - if n := utf8.RuneCountInString(excerpt); n > 2000 { - return fmt.Errorf("source_excerpt must be at most 2000 characters, got %d", n) - } - return nil -} - -// validate is the contract's rules for an API location: v1's affected, and v2's -// api_operation subject. Kind decides the side and the required fields. -func (a Affected) validate() error { - // Rule 3 (continued): affected.kind and affected.side must be valid too. - if a.Kind != AffectedKindOperation && - a.Kind != AffectedKindRequestParameter && - a.Kind != AffectedKindSchema && - a.Kind != AffectedKindSchemaProperty { - return fmt.Errorf("affected.kind must be a valid enum value, got %q", a.Kind) - } - if a.Side != SideRequest && a.Side != SideResponse { - return fmt.Errorf("affected.side must be a valid enum value, got %q", a.Side) - } - - // Rule 5: affected's required sub-fields, and its side, depend on its kind. - // Kind decides side: requests live under paths, responses under schemas. - switch a.Kind { - case AffectedKindOperation: - if a.Operation == "" { - return fmt.Errorf("affected.operation is required when kind is operation") - } - if a.Side != SideRequest { - return fmt.Errorf("affected.side must be %q when kind is operation, got %q", SideRequest, a.Side) - } - case AffectedKindRequestParameter: - if a.Operation == "" || a.Property == "" { - return fmt.Errorf("affected.operation and affected.property are required when kind is request_parameter") - } - if a.Side != SideRequest { - return fmt.Errorf("affected.side must be %q when kind is request_parameter, got %q", SideRequest, a.Side) - } - case AffectedKindSchema: - if a.Schema == "" { - return fmt.Errorf("affected.schema is required when kind is schema") - } - if a.Side != SideResponse { - return fmt.Errorf("affected.side must be %q when kind is schema, got %q", SideResponse, a.Side) - } - case AffectedKindSchemaProperty: - if a.Schema == "" || a.Property == "" { - return fmt.Errorf("affected.schema and affected.property are required when kind is schema_property") - } - if a.Side != SideResponse { - return fmt.Errorf("affected.side must be %q when kind is schema_property, got %q", SideResponse, a.Side) - } - } - return nil -} diff --git a/fetch/fetch.go b/fetch/fetch.go index 8cb228c..8d79a32 100644 --- a/fetch/fetch.go +++ b/fetch/fetch.go @@ -1,135 +1,19 @@ -// Package fetch does plain HTTP GETs with retries and a timeout. It knows -// nothing about specs, Stripe, or any other caller — just how to reliably -// download bytes from a URL. +// Package fetch is the old import path of github.com/patch-it-dev/patch-it-plugins/core/fetch. +// +// Deprecated: import github.com/patch-it-dev/patch-it-plugins/core/fetch +// instead. This package only forwards to it, so code written against the old +// path keeps building; it will be removed in a later release. package fetch -import ( - "context" - "errors" - "fmt" - "io" - "net/http" - "time" -) +import moved "github.com/patch-it-dev/patch-it-plugins/core/fetch" -const ( - maxAttempts = 3 - attemptTimeout = 30 * time.Second +// Deprecated: the types live in github.com/patch-it-dev/patch-it-plugins/core/fetch. +type ( + Cache = moved.Cache + StatusError = moved.StatusError ) -// baseBackoff is a variable only so tests can shorten it. -var baseBackoff = 1 * time.Second - -// StatusError is returned when a request completes but with a non-200 status, -// so callers can check the exact code (e.g. a 404 meaning "try elsewhere") -// instead of matching on an error string. -type StatusError struct { - URL string - StatusCode int -} - -func (e *StatusError) Error() string { - return fmt.Sprintf("unexpected status %d from %s", e.StatusCode, e.URL) -} - -// requestError marks a failure to even build the HTTP request (e.g. a -// malformed URL). It's never transient, so Get fails fast on it instead of -// retrying like it does for network errors and 5xx responses. -type requestError struct{ err error } - -func (e *requestError) Error() string { return e.err.Error() } -func (e *requestError) Unwrap() error { return e.err } - -// Get downloads url, retrying on network errors and 5xx responses up to -// maxAttempts times with exponential backoff (1s, 2s, 4s...). A 4xx response -// means the request itself is wrong, so it fails immediately instead of -// retrying. headers are set on every attempt; nil is fine for none. -func Get(ctx context.Context, url string, headers map[string]string) ([]byte, error) { - r, err := get(ctx, url, headers) - if err == nil && r.status == http.StatusNotModified { - // Get sends no validators of its own, so a 304 is the server's error, - // never an empty page. - return nil, &StatusError{URL: url, StatusCode: r.status} - } - return r.body, err -} - -// response is one successful attempt: 200 with its body, or 304 Not Modified -// to a conditional request (no body). -type response struct { - status int - body []byte - header http.Header -} - -// get is Get's retry loop, returning the whole response. -func get(ctx context.Context, url string, headers map[string]string) (response, error) { - var lastErr error - - for attempt := 1; attempt <= maxAttempts; attempt++ { - r, err := attemptGet(ctx, url, headers) - status := r.status - if err == nil { - return r, nil - } - lastErr = err - - var reqErr *requestError - if errors.As(err, &reqErr) { - return response{}, err // a malformed request won't build any differently on retry - } - - if status >= 400 && status < 500 { - return response{}, err // retrying a bad request never helps - } - - if attempt < maxAttempts { - backoff := baseBackoff * time.Duration(1<<(attempt-1)) - select { - case <-time.After(backoff): - case <-ctx.Done(): - return response{}, ctx.Err() - } - } - } - - return response{}, fmt.Errorf("fetching %s failed after %d attempts: %w", url, maxAttempts, lastErr) -} - -// attemptGet makes one attempt, bounded by its own timeout so a single hung -// request can't block forever. status is 0 when the failure happened before -// a response arrived (a bad request, a network error). A 304 is a success: it -// only comes back to a request that asked for it (If-None-Match, -// If-Modified-Since). -func attemptGet(ctx context.Context, url string, headers map[string]string) (response, error) { - ctx, cancel := context.WithTimeout(ctx, attemptTimeout) - defer cancel() - - req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) - if err != nil { - return response{}, &requestError{err: fmt.Errorf("could not build the request: %w", err)} - } - for key, value := range headers { - req.Header.Set(key, value) - } - - resp, err := http.DefaultClient.Do(req) - if err != nil { - return response{}, fmt.Errorf("could not retrieve the content from given url: %w", err) - } - defer resp.Body.Close() - - r := response{status: resp.StatusCode, header: resp.Header} - if resp.StatusCode == http.StatusNotModified { - return r, nil - } - if resp.StatusCode != http.StatusOK { - return r, &StatusError{URL: url, StatusCode: resp.StatusCode} - } - - r.body, err = io.ReadAll(resp.Body) - if err != nil { - return r, fmt.Errorf("could not read the response body: %w", err) - } - return r, nil -} +// Deprecated: the functions and values live in github.com/patch-it-dev/patch-it-plugins/core/fetch. +var ( + Get = moved.Get +) diff --git a/internal/pluginlist/pluginlist.go b/internal/pluginlist/pluginlist.go index 7ad543f..e9a9aa8 100644 --- a/internal/pluginlist/pluginlist.go +++ b/internal/pluginlist/pluginlist.go @@ -1,239 +1,87 @@ -// Package pluginlist joins the plugin registry with the table in -// docs/plugins.md: the registry says which plugins exist and which format -// plugin each is built on, the table says who the provider is, what it -// watches, its source and its status. List checks that the two agree, so a -// plugin missing from the table (AGENTS.md rule 1) fails `make check`. +// Package pluginlist is the list of what patch-it watches, as JSON, built at +// run time from the plugin registry: every registered provider with its +// sources and their metadata. Nothing is kept by hand and nothing is +// committed: `go run ./cmd/list-plugins` prints it, and the Release workflow +// attaches it to each release as plugins.json (docs/plugins.schema.json +// describes it). package pluginlist import ( "bytes" "encoding/json" - "errors" - "fmt" - "os" - "path" - "path/filepath" - "reflect" - "regexp" - "slices" - "strconv" - "strings" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" ) -// Table is the plugin list, relative to the repo root. -const Table = "docs/plugins.md" +// Repo is where a source's page is published. +const Repo = "https://github.com/patch-it-dev/patch-it-plugins" -// JSON is the generated list, relative to the repo root. -const JSON = "docs/plugins.json" - -// The markers around the table in Table. -const ( - StartMarker = "" - EndMarker = "" -) - -// Row is one row of the table, as written. -type Row struct { - ID string // the plugin id, without the backticks and link - Page string // the provider page it links to: "algolia" for providers/algolia.md - Provider string - Layer string // as written: "API", "AI models" - Watches string - Format string - Source string // the cell, as written - Status string // the cell, as written: "live", "paused: why" -} - -// Plugin is one entry of docs/plugins.json. -type Plugin struct { - ID string `json:"id"` - Provider string `json:"provider"` - Layer string `json:"layer"` // api, sdk, package, runtime, ci or ai-models - Watches string `json:"watches"` - Format string `json:"format"` - SourceURL string `json:"source_url"` - Status string `json:"status"` // live, paused or planned - StatusNote string `json:"status_note,omitempty"` // why, for paused - ProviderPage string `json:"provider_page"` // relative to the repo root -} - -// File is docs/plugins.json. +// File is plugins.json. type File struct { - Comment string `json:"_comment"` - Schema string `json:"$schema"` - Count int `json:"count"` - Plugins []Plugin `json:"plugins"` + Comment string `json:"_comment"` + Schema string `json:"$schema"` + Version string `json:"version"` // the release it was built from; "main" between releases + Providers int `json:"provider_count"` + Sources int `json:"source_count"` + Categories []string `json:"categories"` // every category, in display order + List []Provider `json:"providers"` // sorted by id } -var ( - cellID = regexp.MustCompile("^\\[`([a-z0-9][a-z0-9.-]*)`\\]\\(https://github\\.com/patch-it-dev/patch-it-plugins/blob/main/providers/([^)]+)\\.md\\)$") - cellFmt = regexp.MustCompile("^`([a-z]+)`$") - link = regexp.MustCompile(`\]\((https?://[^)\s]+)\)`) - statuses = []string{"live", "paused", "planned"} -) - -// Layers are the kinds of dependency a plugin covers: the table's word to -// the JSON's. -var Layers = map[string]string{ - "API": "api", - "SDK": "sdk", - "package": "package", - "runtime": "runtime", - "CI": "ci", - "AI models": "ai-models", +// Provider is one provider and its sources. +type Provider struct { + ID string `json:"id"` + Name string `json:"name"` + Category string `json:"category"` + Homepage string `json:"homepage"` + Sources []Source `json:"sources"` // sorted by id } -// Statuses are the values of a plugin's status. -func Statuses() []string { return slices.Clone(statuses) } - -// ReadTable reads the rows between the markers of the plugin list. -func ReadTable(md []byte) ([]Row, error) { - start := bytes.Index(md, []byte(StartMarker)) - end := bytes.Index(md, []byte(EndMarker)) - if start < 0 || end < start { - return nil, fmt.Errorf("%s: no %s ... %s block", Table, StartMarker, EndMarker) - } - var rows []Row - lines := strings.Split(string(md[start+len(StartMarker):end]), "\n") - for _, l := range lines { - l = strings.TrimSpace(l) - if !strings.HasPrefix(l, "|") || strings.HasPrefix(l, "|---") || strings.HasPrefix(l, "| Plugin") { - continue - } - cells := splitRow(l) - if len(cells) != 7 { - return nil, fmt.Errorf("%s: row %q has %d cells, want 7", Table, l, len(cells)) - } - m := cellID.FindStringSubmatch(cells[0]) - if m == nil { - return nil, fmt.Errorf("%s: plugin cell %q is not [`id`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/.md)", Table, cells[0]) - } - f := cellFmt.FindStringSubmatch(cells[4]) - if f == nil { - return nil, fmt.Errorf("%s: %s: format cell %q is not `format`", Table, m[1], cells[4]) - } - rows = append(rows, Row{ID: m[1], Page: m[2], Provider: cells[1], Layer: cells[2], Watches: cells[3], Format: f[1], Source: cells[5], Status: cells[6]}) - } - return rows, nil +// Source is one source. +type Source struct { + ID string `json:"id"` // provider/source + Alias string `json:"alias"` // the old slug: --api and every event's api until the contract switches to source ids + Kind string `json:"kind"` + Layer string `json:"layer"` + Watches string `json:"watches"` + URLs []string `json:"urls"` + Page string `json:"page"` // absolute, at Version } -// splitRow splits "| a | b |" into its cells. A cell holding a pipe would -// have to escape it (\|); none does, and the docs site copies the table as is. -func splitRow(l string) []string { - l = strings.TrimSuffix(strings.TrimPrefix(l, "|"), "|") - var cells []string - for _, c := range strings.Split(l, "|") { - cells = append(cells, strings.TrimSpace(c)) +// Build is the list of everything registered, with pages linked at ref (a +// release tag, or "main"). +func Build(ref string) File { + if ref == "" { + ref = "main" } - return cells -} - -// Format is the format plugin a registered plugin is built on: the name of -// the package under plugins/ its type is defined in. -func Format(p plugin.Plugin) string { - t := reflect.TypeOf(p) - for t.Kind() == reflect.Pointer { - t = t.Elem() + f := File{ + Comment: "Generated from the plugin registry by `go run ./cmd/list-plugins`. Not committed: each release carries it as an asset.", + Schema: Repo + "/blob/" + ref + "/docs/plugins.schema.json", + Version: ref, } - return path.Base(t.PkgPath()) -} - -// List joins the registered plugins with the table under root and checks -// they agree: every registered plugin has a row with its format, every row is -// registered (unless it's planned), and every plugin has a provider page. -// The plugins come out sorted by id. -func List(root string) ([]Plugin, error) { - md, err := os.ReadFile(filepath.Join(root, Table)) - if err != nil { - return nil, err - } - rows, err := ReadTable(md) - if err != nil { - return nil, err - } - var errs []error - byID := map[string]Row{} - for _, r := range rows { - if _, dup := byID[r.ID]; dup { - errs = append(errs, fmt.Errorf("%s: %s has two rows", Table, r.ID)) - } - byID[r.ID] = r + for _, c := range plugin.Categories() { + f.Categories = append(f.Categories, string(c)) } - registered := map[string]string{} - for _, name := range plugin.Names() { - p, _ := plugin.Lookup(name) - registered[name] = Format(p) - if _, ok := byID[name]; !ok { - errs = append(errs, fmt.Errorf("%s is registered but has no row in %s: add one (AGENTS.md rule 1), then run `make plugins-json`", name, Table)) - } - } - var out []Plugin - for _, r := range rows { - status, note, _ := strings.Cut(r.Status, ":") - status, note = strings.TrimSpace(status), strings.TrimSpace(note) - if !slices.Contains(statuses, status) { - errs = append(errs, fmt.Errorf("%s: %s: status %q is not live, paused or planned", Table, r.ID, r.Status)) - } - layer, known := Layers[r.Layer] - if !known { - errs = append(errs, fmt.Errorf("%s: %s: layer %q is not one of API, SDK, package, runtime, CI, AI models", Table, r.ID, r.Layer)) - } - format, ok := registered[r.ID] - switch { - case !ok && status != "planned": - errs = append(errs, fmt.Errorf("%s: %s has a row but no plugin registers it: mark it planned, or remove the row", Table, r.ID)) - case ok && format != r.Format: - errs = append(errs, fmt.Errorf("%s: %s says format %q, but the plugin is in plugins/%s", Table, r.ID, r.Format, format)) + for _, p := range plugin.Providers() { + out := Provider{ID: p.ID, Name: p.Name, Category: string(p.Category), Homepage: p.Homepage} + for _, s := range p.Sources { + out.Sources = append(out.Sources, Source{ + ID: p.ID + "/" + s.ID, Alias: s.Plugin.Name(), Kind: s.Kind, Layer: string(s.Layer), + Watches: s.Watches, URLs: s.URLs, Page: Repo + "/blob/" + ref + "/" + s.Page, + }) } - // One page per provider: the algolia-* plugins share providers/algolia.md. - page := "providers/" + r.Page + ".md" - if _, err := os.Stat(filepath.Join(root, page)); err != nil { - errs = append(errs, fmt.Errorf("%s: %s links to %s, which doesn't exist (copy providers/_template.md)", Table, r.ID, page)) - } - src := link.FindStringSubmatch(r.Source) - if src == nil { - errs = append(errs, fmt.Errorf("%s: %s: the source cell has no absolute link", Table, r.ID)) - src = []string{"", ""} - } - out = append(out, Plugin{ - ID: r.ID, Provider: r.Provider, Layer: layer, Watches: r.Watches, Format: r.Format, - SourceURL: src[1], Status: status, StatusNote: note, ProviderPage: page, - }) - } - if len(errs) > 0 { - return nil, errors.Join(errs...) - } - slices.SortFunc(out, func(a, b Plugin) int { return strings.Compare(a.ID, b.ID) }) - return out, nil -} - -// Readme is the README, relative to the repo root: its plugins badge is a -// static count that `make plugins-json` keeps right. -const Readme = "README.md" - -var badge = regexp.MustCompile(`(img\.shields\.io/badge/plugins-)(\d+)(-)`) - -// Badge is readme with its plugins badge set to n. ok is false when it has none. -func Badge(readme []byte, n int) (out []byte, ok bool) { - if !badge.Match(readme) { - return readme, false + f.Sources += len(out.Sources) + f.List = append(f.List, out) } - return badge.ReplaceAll(readme, []byte("${1}"+strconv.Itoa(n)+"${3}")), true + f.Providers = len(f.List) + return f } -// Marshal is docs/plugins.json for plugins, ending in a newline. -func Marshal(plugins []Plugin) ([]byte, error) { +// Marshal is f as indented JSON, ending in a newline. +func Marshal(f File) ([]byte, error) { var buf bytes.Buffer enc := json.NewEncoder(&buf) enc.SetIndent("", " ") enc.SetEscapeHTML(false) - err := enc.Encode(File{ - Comment: "Generated from the plugin registry and docs/plugins.md by `make plugins-json` (go run ./cmd/list-plugins). Don't edit by hand.", - Schema: "./plugins.schema.json", - Count: len(plugins), - Plugins: plugins, - }) + err := enc.Encode(f) return buf.Bytes(), err } diff --git a/internal/pluginlist/pluginlist_test.go b/internal/pluginlist/pluginlist_test.go index 895d93f..21dfb84 100644 --- a/internal/pluginlist/pluginlist_test.go +++ b/internal/pluginlist/pluginlist_test.go @@ -1,109 +1,72 @@ package pluginlist import ( - "bytes" - "os" - "path/filepath" + "encoding/json" "strings" "testing" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" _ "github.com/patch-it-dev/patch-it-plugins/plugins/all" ) const root = "../.." -// The real table agrees with the registry, and docs/plugins.json is what -// `make plugins-json` writes from them. This is how CI_go (go test) catches a -// plugin added without its row, or a row changed without regenerating. -func TestPluginsJSONIsUpToDate(t *testing.T) { - plugins, err := List(root) - if err != nil { - t.Fatal(err) - } - want, err := Marshal(plugins) - if err != nil { - t.Fatal(err) - } - have, err := os.ReadFile(filepath.Join(root, JSON)) - if err != nil { - t.Fatalf("%v (run `make plugins-json`)", err) - } - if !bytes.Equal(have, want) { - t.Errorf("%s is out of date: run `make plugins-json` and commit it", JSON) - } -} - -func TestReadTable(t *testing.T) { - md := "intro\n" + StartMarker + "\n" + - "| Plugin (`--api`) | Provider | Layer | What it watches | Format plugin | Source | Status |\n" + - "|---|---|---|---|---|---|---|\n" + - "| [`foo-api`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/foo-api.md) | Foo | API | Removed endpoints | `openapi` | [foo/spec](https://github.com/foo/spec) (`openapi.json`) | paused: not released yet |\n" + - EndMarker + "\n" - rows, err := ReadTable([]byte(md)) - if err != nil { - t.Fatal(err) - } - want := Row{ID: "foo-api", Page: "foo-api", Provider: "Foo", Layer: "API", Watches: "Removed endpoints", Format: "openapi", - Source: "[foo/spec](https://github.com/foo/spec) (`openapi.json`)", Status: "paused: not released yet"} - if len(rows) != 1 || rows[0] != want { - t.Errorf("rows = %+v, want [%+v]", rows, want) - } -} - -func TestReadTableRejectsABadRow(t *testing.T) { - for name, row := range map[string]string{ - "relative link": "| [`foo`](../providers/foo.md) | Foo | API | x | `openapi` | [s](https://s.example) | live |", - "not a provider page": "| [`foo`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/docs/foo.md) | Foo | API | x | `openapi` | [s](https://s.example) | live |", - "missing a cell": "| [`foo`](https://github.com/patch-it-dev/patch-it-plugins/blob/main/providers/foo.md) | Foo | `openapi` | [s](https://s.example) | live |", - } { - md := StartMarker + "\n" + row + "\n" + EndMarker - if _, err := ReadTable([]byte(md)); err == nil { - t.Errorf("%s: no error", name) +// The list is the registry: every registered source once, under its provider, +// with its old slug, and the counts agree. +func TestBuildListsEveryRegisteredSource(t *testing.T) { + f := Build("") + if f.Version != "main" { + t.Errorf("version %q, want main", f.Version) + } + seen := map[string]bool{} + n := 0 + for _, p := range f.List { + for _, s := range p.Sources { + n++ + if !strings.HasPrefix(s.ID, p.ID+"/") { + t.Errorf("%s is listed under %s", s.ID, p.ID) + } + if !strings.HasPrefix(s.Page, Repo+"/blob/main/providers/") { + t.Errorf("%s: page %s", s.ID, s.Page) + } + seen[s.Alias] = true } } -} - -// A registered plugin with no row fails the list, naming the rule. -func TestListWantsARowPerPlugin(t *testing.T) { - dir := t.TempDir() - if err := os.MkdirAll(filepath.Join(dir, "docs"), 0o755); err != nil { - t.Fatal(err) - } - if err := os.WriteFile(filepath.Join(dir, Table), []byte(StartMarker+"\n"+EndMarker+"\n"), 0o644); err != nil { - t.Fatal(err) + if n != f.Sources || len(f.List) != f.Providers || n != len(plugin.Sources()) { + t.Errorf("counts: %d sources listed, source_count %d, %d registered; %d providers, provider_count %d", n, f.Sources, len(plugin.Sources()), len(f.List), f.Providers) } - _, err := List(dir) - if err == nil || !strings.Contains(err.Error(), "stripe is registered but has no row") { - t.Errorf("err = %v, want stripe's missing row", err) + for _, name := range plugin.Names() { + if !seen[name] { + t.Errorf("registered plugin %s isn't listed", name) + } } } -// The README's plugins badge is a static number: it must be the count in -// docs/plugins.json (make plugins-json sets it), or the badge lies. -func TestReadmeBadgeCountsThePlugins(t *testing.T) { - plugins, err := List(root) +// One provider, several sources: Algolia's three APIs. +func TestAlgoliaIsOneProviderWithThreeSources(t *testing.T) { + b, err := Marshal(Build("v1.2.3")) if err != nil { t.Fatal(err) } - readme, err := os.ReadFile(filepath.Join(root, Readme)) - if err != nil { + var f File + if err := json.Unmarshal(b, &f); err != nil { t.Fatal(err) } - want, ok := Badge(readme, len(plugins)) - if !ok { - t.Fatal("README.md has no plugins badge (img.shields.io/badge/plugins--...)") - } - if !bytes.Equal(readme, want) { - t.Errorf("README.md's plugins badge doesn't say %d: run `make plugins-json`", len(plugins)) - } -} - -func TestBadge(t *testing.T) { - got, ok := Badge([]byte("[![Plugins](https://img.shields.io/badge/plugins-20-3f5bf0)](docs/plugins.md)"), 21) - if !ok || string(got) != "[![Plugins](https://img.shields.io/badge/plugins-21-3f5bf0)](docs/plugins.md)" { - t.Errorf("Badge = %s, %v", got, ok) - } - if _, ok := Badge([]byte("no badge"), 1); ok { - t.Error("found a badge in none") + for _, p := range f.List { + if p.ID != "algolia" { + continue + } + var ids []string + for _, s := range p.Sources { + ids = append(ids, s.ID+"="+s.Alias) + if !strings.Contains(s.Page, "/blob/v1.2.3/") { + t.Errorf("%s: page %s isn't at the release", s.ID, s.Page) + } + } + if got := strings.Join(ids, " "); got != "algolia/ingestion-api=algolia-ingestion algolia/recommend-api=algolia-recommend algolia/search-api=algolia-search" { + t.Errorf("algolia's sources: %s", got) + } + return } + t.Error("no algolia provider") } diff --git a/internal/pluginlist/schema_test.go b/internal/pluginlist/schema_test.go index 6cbc729..4b7df50 100644 --- a/internal/pluginlist/schema_test.go +++ b/internal/pluginlist/schema_test.go @@ -8,66 +8,95 @@ import ( "regexp" "slices" "testing" + + "github.com/patch-it-dev/patch-it-plugins/core/plugin" ) -// docs/plugins.json validates against docs/plugins.schema.json. The checker -// below knows only the keywords that schema uses (type, required, +// What list-plugins prints validates against docs/plugins.schema.json. The +// checker below knows only the keywords that schema uses (type, required, // properties, additionalProperties, items, enum, pattern, minLength): the // module takes no dependency for a JSON Schema library. -func TestPluginsJSONMatchesTheSchema(t *testing.T) { +func TestTheListMatchesTheSchema(t *testing.T) { var schema, doc any - for file, v := range map[string]*any{"docs/plugins.schema.json": &schema, JSON: &doc} { - b, err := os.ReadFile(filepath.Join(root, file)) + b, err := os.ReadFile(filepath.Join(root, "docs/plugins.schema.json")) + if err != nil { + t.Fatal(err) + } + if err := json.Unmarshal(b, &schema); err != nil { + t.Fatal(err) + } + for _, ref := range []string{"main", "v9.9.9"} { + out, err := Marshal(Build(ref)) if err != nil { t.Fatal(err) } - if err := json.Unmarshal(b, v); err != nil { - t.Fatalf("%s: %v", file, err) + if err := json.Unmarshal(out, &doc); err != nil { + t.Fatal(err) + } + for _, err := range validate(schema.(map[string]any), doc, "$") { + t.Error(err) } - } - for _, err := range validate(schema.(map[string]any), doc, "$") { - t.Error(err) } } -// The schema's enums are the values the generator writes. -func TestSchemaEnumsMatchTheGenerator(t *testing.T) { +// The schema's enums are the values the registry allows, and every kind. +func TestSchemaEnumsMatchTheRegistry(t *testing.T) { b, err := os.ReadFile(filepath.Join(root, "docs/plugins.schema.json")) if err != nil { t.Fatal(err) } var s struct { Properties struct { - Plugins struct { + Categories struct { Items struct { - Properties map[string]struct { - Enum []string `json:"enum"` + Enum []string `json:"enum"` + } `json:"items"` + } `json:"categories"` + Providers struct { + Items struct { + Properties struct { + Category struct { + Enum []string `json:"enum"` + } `json:"category"` + Sources struct { + Items struct { + Properties map[string]struct { + Enum []string `json:"enum"` + } `json:"properties"` + } `json:"items"` + } `json:"sources"` } `json:"properties"` } `json:"items"` - } `json:"plugins"` + } `json:"providers"` } `json:"properties"` } if err := json.Unmarshal(b, &s); err != nil { t.Fatal(err) } - props := s.Properties.Plugins.Items.Properties - var layers []string - for _, l := range Layers { - layers = append(layers, l) + var categories, layers []string + for _, c := range plugin.Categories() { + categories = append(categories, string(c)) + } + for _, l := range plugin.Layers() { + layers = append(layers, string(l)) } - for name, want := range map[string][]string{"layer": layers, "status": Statuses()} { - got := slices.Sorted(slices.Values(props[name].Enum)) - if !slices.Equal(got, slices.Sorted(slices.Values(want))) { - t.Errorf("schema enum for %s = %v, want %v", name, got, want) + src := s.Properties.Providers.Items.Properties.Sources.Items.Properties + for name, c := range map[string]struct{ got, want []string }{ + "categories": {s.Properties.Categories.Items.Enum, categories}, + "providers.category": {s.Properties.Providers.Items.Properties.Category.Enum, categories}, + "sources.layer": {src["layer"].Enum, layers}, + } { + if !slices.Equal(slices.Sorted(slices.Values(c.got)), slices.Sorted(slices.Values(c.want))) { + t.Errorf("schema enum %s = %v, want %v", name, c.got, c.want) } } - formats, err := filepath.Glob(filepath.Join(root, "plugins", "*")) + kinds, err := filepath.Glob(filepath.Join(root, "plugins", "*")) if err != nil { t.Fatal(err) } - for _, f := range formats { - if name := filepath.Base(f); name != "all" && !slices.Contains(props["format"].Enum, name) { - t.Errorf("format plugin %s is missing from the schema's format enum", name) + for _, k := range kinds { + if name := filepath.Base(k); name != "all" && !slices.Contains(src["kind"].Enum, name) { + t.Errorf("kind %s is missing from the schema's kind enum", name) } } } diff --git a/kit/kit.go b/kit/kit.go index 47920d0..5575062 100644 --- a/kit/kit.go +++ b/kit/kit.go @@ -1,48 +1,30 @@ -// Package kit is what source plugins share beyond the core: the Wayback -// Machine client a plugin cites a live page through (Wayback, Cite) and the -// "same page as last run" check (Unchanged). Fetching with ETag and -// Last-Modified caching is fetch's Cache, handed to every plugin as -// plugin.Env.Get; the test kit is kittest. +// Package kit is the old import path of github.com/patch-it-dev/patch-it-plugins/core/kit. // -// kit is not a plugin: plugins import it, it imports no plugin. +// Deprecated: import github.com/patch-it-dev/patch-it-plugins/core/kit +// instead. This package only forwards to it, so code written against the old +// path keeps building; it will be removed in a later release. package kit import ( "context" - "fmt" - "reflect" - "github.com/patch-it-dev/patch-it-plugins/plugin" + moved "github.com/patch-it-dev/patch-it-plugins/core/kit" ) -// Unchanged reports whether body is what api's last stored baseline hashed -// to, and returns that baseline (nil on a first run). A plugin reading a live -// page returns early on true: an unchanged page costs one GET and no snapshot. -func Unchanged(env plugin.Env, api string, body []byte) (last *plugin.Spec, same bool, err error) { - last, err = env.Baselines.LastSpec(api) - if err != nil { - return nil, false, fmt.Errorf("loading the last baseline of %s: %w", api, err) - } - return last, last != nil && last.SHA256 == plugin.SHA256(body), nil -} +// Deprecated: the types live in github.com/patch-it-dev/patch-it-plugins/core/kit. +type ( + Archiver = moved.Archiver + Wayback = moved.Wayback +) -// Cite snapshots a live page and checks the snapshot holds what the plugin -// read from the live page: read turns bytes into whatever the plugin makes -// events from, and the snapshot's must deep-equal want. The events are then -// made from the snapshot and cite snapURL, so every excerpt is on the page it -// cites. No snapshot, or one that says something else (the page changed in -// between), is an error: no provenance, no event. +// Deprecated: the functions and values live in github.com/patch-it-dev/patch-it-plugins/core/kit. +var ( + Unchanged = moved.Unchanged +) + +// Cite forwards to core/kit.Cite. +// +// Deprecated: use github.com/patch-it-dev/patch-it-plugins/core/kit.Cite. func Cite[T any](ctx context.Context, a Archiver, pageURL string, read func([]byte) (T, error), want T) (snapURL string, snap []byte, got T, err error) { - snapURL, snap, err = a.Snapshot(ctx, pageURL) - if err != nil { - return "", nil, got, fmt.Errorf("no events without a snapshot to cite: %w", err) - } - got, err = read(snap) - if err != nil { - return "", nil, got, fmt.Errorf("reading the snapshot %s: %w", snapURL, err) - } - if !reflect.DeepEqual(got, want) { - return "", nil, got, fmt.Errorf("the snapshot %s doesn't hold what %s says now; run again", snapURL, pageURL) - } - return snapURL, snap, got, nil + return moved.Cite(ctx, a, pageURL, read, want) } diff --git a/kit/kittest/kittest.go b/kit/kittest/kittest.go index 4680623..39f0e4e 100644 --- a/kit/kittest/kittest.go +++ b/kit/kittest/kittest.go @@ -1,214 +1,33 @@ -// Package kittest is the plugin test kit: a saved copy of a source in, the -// events a plugin stores out, compared with a golden file. Nothing reaches the -// network or a database: a URL the test didn't save fails the test, events and -// baselines are kept in memory (Mem), and the clock is fixed, so the same -// pages always give byte-identical events. +// Package kittest is the old import path of github.com/patch-it-dev/patch-it-plugins/core/kit/kittest. // -// got := kittest.Ingest(t, plugin, kittest.Pages{url: "testdata/page.md"}) -// kittest.Golden(t, "testdata/golden/numpy.json", got) -// -// `go test ./plugins/ -update` rewrites a plugin's golden files; -// review the diff like code. +// Deprecated: import github.com/patch-it-dev/patch-it-plugins/core/kit/kittest +// instead. This package only forwards to it, so code written against the old +// path keeps building; it will be removed in a later release. package kittest -import ( - "bytes" - "context" - "encoding/json" - "flag" - "fmt" - "os" - "path/filepath" - "reflect" - "strconv" - "testing" - "time" +import moved "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/plugin" +// Deprecated: the types live in github.com/patch-it-dev/patch-it-plugins/core/kit/kittest. +type ( + Archive = moved.Archive + Mem = moved.Mem + Pages = moved.Pages + Sink = moved.Sink ) -var update = flag.Bool("update", false, "rewrite the golden files from what the plugins store now") - -// Clock is the time every kit run detects its changes at. -var Clock = time.Date(2026, 11, 14, 12, 0, 0, 0, time.UTC) - -// Pages are the saved copies a run may fetch: URL to file, relative to the -// test's package. -type Pages map[string]string - -// Get serves Pages, and fails the test on any other URL. -func (p Pages) Get(t testing.TB) func(context.Context, string) ([]byte, error) { - return func(_ context.Context, url string) ([]byte, error) { - file, ok := p[url] - if !ok { - t.Errorf("fetched %s, which the test didn't save", url) - return nil, fmt.Errorf("no saved copy of %s", url) - } - return Read(t, file), nil - } -} - -// Read is a saved file, or the test fails. -func Read(t testing.TB, file string) []byte { - t.Helper() - b, err := os.ReadFile(file) - if err != nil { - t.Fatal(err) - } - return b -} - -// Env is an empty Mem for the baselines and the events, Pages for the network -// and Clock for the time. Nothing outside the test is touched: no database, no -// network. -func Env(t testing.TB, pages Pages) plugin.Env { - t.Helper() - return plugin.Env{Baselines: NewMem(), Get: pages.Get(t), Now: func() time.Time { return Clock }} -} - -// Ingest runs p twice and returns what the first run stored. Any rejected -// event or error fails the test, and so does a second run over the same pages -// that stores anything new: the fingerprints must dedupe. -func Ingest(t testing.TB, p plugin.Plugin, pages Pages) []event.EventV2 { - t.Helper() - env := Env(t, pages) - return IngestWith(t, env, p) -} - -// IngestWith is Ingest over an Env the test built (its own archive, a baseline -// already saved). env.Baselines is where the events are stored too, so it must -// be a Sink, as Env's Mem is. -func IngestWith(t testing.TB, env plugin.Env, p plugin.Plugin) []event.EventV2 { - t.Helper() - sink, ok := env.Baselines.(Sink) - if !ok { - t.Fatalf("env.Baselines is a %T, not a kittest.Sink to store the events in", env.Baselines) - } - if rejected, err := Run(context.Background(), env, sink, p); err != nil || rejected { - t.Fatalf("%s: rejected=%v err=%v", p.Name(), rejected, err) - } - first := Stored(t, sink) - if rejected, err := Run(context.Background(), env, sink, p); err != nil || rejected { - t.Fatalf("%s, second run: rejected=%v err=%v", p.Name(), rejected, err) - } - if again := Stored(t, sink); len(again) != len(first) { - t.Errorf("%s: a second run over the same pages stored %d more events", p.Name(), len(again)-len(first)) - } - return first -} - -// Stored is every event in s, in feed order, or the test fails. -func Stored(t testing.TB, s Sink) []event.EventV2 { - t.Helper() - all, err := s.All() - if err != nil { - t.Fatal(err) - } - return all -} - -// Golden compares events with the JSON file at path, or rewrites it with -// -update. Every event must also be valid v2, or the golden would pin a bug. -func Golden(t testing.TB, path string, events []event.EventV2) { - t.Helper() - golden(t, path, events, *update) -} - -// CompareGolden is Golden that never rewrites the file, even with -update: -// for a test that checks someone else's golden file. -func CompareGolden(t testing.TB, path string, events []event.EventV2) { - t.Helper() - golden(t, path, events, false) -} - -func golden(t testing.TB, path string, events []event.EventV2, rewrite bool) { - t.Helper() - for _, e := range events { - if err := e.Validate(); err != nil { - t.Errorf("event %d (%s) is invalid: %v", e.ID, e.Description, err) - } - } - if events == nil { - events = []event.EventV2{} - } - var buf bytes.Buffer - enc := json.NewEncoder(&buf) - enc.SetIndent("", " ") - enc.SetEscapeHTML(false) - if err := enc.Encode(events); err != nil { - t.Fatal(err) - } - if rewrite { - if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { - t.Fatal(err) - } - if err := os.WriteFile(path, buf.Bytes(), 0o644); err != nil { - t.Fatal(err) - } - return - } - want, err := os.ReadFile(path) - if err != nil { - t.Fatalf("%v (run the test with -update to create it)", err) - } - if !bytes.Equal(want, buf.Bytes()) { - t.Errorf("%s: the stored events differ from the golden file (%d events now); if the change is intended, rerun with -update and review the diff\n%s", - path, len(events), firstDiff(want, buf.Bytes())) - } -} - -// firstDiff is the first line that differs, for the failure message. -func firstDiff(want, got []byte) string { - w, g := bytes.Split(want, []byte("\n")), bytes.Split(got, []byte("\n")) - for i := 0; i < len(w) || i < len(g); i++ { - var wl, gl []byte - if i < len(w) { - wl = w[i] - } - if i < len(g) { - gl = g[i] - } - if !bytes.Equal(wl, gl) { - return "line " + strconv.Itoa(i+1) + ":\n want " + string(wl) + "\n got " + string(gl) - } - } - return "" -} - -// Archive is the Wayback Machine without the network: it serves a saved copy -// as the snapshot of a page, under a fixed timestamp, and counts the -// snapshots it was asked for. -type Archive struct { - T testing.TB - Pages Pages // the page's URL to the file the snapshot holds - Timestamp string // 14 digits; default 20261114120000 - Calls int -} - -func (a *Archive) Snapshot(_ context.Context, page string) (string, []byte, error) { - a.Calls++ - file, ok := a.Pages[page] - if !ok { - return "", nil, fmt.Errorf("no snapshot of %s in the test archive", page) - } - ts := a.Timestamp - if ts == "" { - ts = "20261114120000" - } - return "https://web.archive.org/web/" + ts + "/" + page, Read(a.T, file), nil -} - -// Registered fails the test unless p's package registered a plugin of p's -// type under p's name: each plugin package checks its own registration. -func Registered(t testing.TB, p plugin.Plugin) { - t.Helper() - got, ok := plugin.Lookup(p.Name()) - if !ok { - t.Errorf("no plugin registered as %q", p.Name()) - return - } - if reflect.TypeOf(got) != reflect.TypeOf(p) { - t.Errorf("%q is a %T, want a %T", p.Name(), got, p) - } -} +// Deprecated: the functions and values live in github.com/patch-it-dev/patch-it-plugins/core/kit/kittest. +var ( + CheckSink = moved.CheckSink + CompareGolden = moved.CompareGolden + Env = moved.Env + Golden = moved.Golden + Ingest = moved.Ingest + IngestWith = moved.IngestWith + NewMem = moved.NewMem + Read = moved.Read + Registered = moved.Registered + Replay = moved.Replay + Run = moved.Run + Stored = moved.Stored + Clock = moved.Clock +) diff --git a/migrate/migrate.go b/migrate/migrate.go index c1dbcce..cf6865d 100644 --- a/migrate/migrate.go +++ b/migrate/migrate.go @@ -1,120 +1,14 @@ -// Package migrate converts stored v1 change events to the proposed v2 contract -// (the change event contract): `affected` becomes a typed -// `subject`. It runs once, at the cut-over, over every event in the store. +// Package migrate is the old import path of github.com/patch-it-dev/patch-it-plugins/core/migrate. // -// A Stripe event keeps its fingerprint: v2's api_operation formula is v1's. -// A model retirement or an SDK release gets a new one, and `supersedes` names -// the old one, so the patch-it client, which finds its open issues and pull -// requests by the fingerprint in their marker, knows they're the same change -// and opens nothing twice. +// Deprecated: import github.com/patch-it-dev/patch-it-plugins/core/migrate +// instead. This package only forwards to it, so code written against the old +// path keeps building; it will be removed in a later release. package migrate -import ( - "fmt" - "regexp" - "strings" +import moved "github.com/patch-it-dev/patch-it-plugins/core/migrate" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/retire" - "github.com/patch-it-dev/patch-it-plugins/sdkguide" +// Deprecated: the functions and values live in github.com/patch-it-dev/patch-it-plugins/core/migrate. +var ( + Fresh = moved.Fresh + ToV2 = moved.ToV2 ) - -// providers are the model retirement slugs and who retires the models. -var providers = map[string]string{ - retire.OpenAI.API: "openai", - retire.Anthropic.API: "anthropic", -} - -// guides are the SDKs whose v1 events came from a migration guide. -var guides = map[string]sdkguide.Guide{sdkguide.AnthropicV1.API: sdkguide.AnthropicV1} - -// ToV2 converts one v1 event. It refuses an event it can't place rather than -// guess: an unknown slug with a value, or an SDK event that isn't a removed -// request parameter. -func ToV2(e event.ChangeEvent) (event.EventV2, error) { - v2 := event.EventV2{ - ID: e.ID, API: e.API, Version: e.Version, PreviousVersion: e.PreviousVersion, - ChangeType: e.ChangeType, Breaking: e.Breaking, Description: e.Description, - MigrationHint: e.MigrationHint, SourceURL: e.SourceURL, SourcePointer: e.SourcePointer, - SourceExcerpt: e.SourceExcerpt, Confidence: e.Confidence, DetectedAt: e.DetectedAt, - } - a := e.Affected - if provider, ok := providers[e.API]; ok { - if a.Value == "" { - return event.EventV2{}, fmt.Errorf("event %d: a %s event without the model in affected.value", e.ID, e.API) - } - v2.Subject = event.Subject{ - Type: event.SubjectModel, Provider: provider, Name: a.Value, - Replacement: replacement(e.MigrationHint), - } - return rekeyed(v2, e.Fingerprint), nil - } - if g, ok := guides[e.API]; ok { - if a.Kind != event.AffectedKindRequestParameter || a.Property == "" { - return event.EventV2{}, fmt.Errorf("event %d: an %s SDK event that isn't a removed parameter", e.ID, e.API) - } - v2.Subject = event.Subject{ - Type: event.SubjectPackageRelease, Ecosystem: "pypi", Package: g.API, - Change: event.PackageRemovedParameter, Parameter: a.Property, Symbols: g.Symbols, - } - return rekeyed(v2, e.Fingerprint), nil - } - if a.Value != "" { - return event.EventV2{}, fmt.Errorf("event %d: affected.value on %s, which isn't a model retirement", e.ID, e.API) - } - v2.Subject = event.Subject{ - Type: event.SubjectAPIOperation, Kind: a.Kind, Side: a.Side, - Operation: a.Operation, Schema: a.Schema, Property: a.Property, - } - v2.Fingerprint = e.Fingerprint // the same formula: unchanged by construction - return v2, nil -} - -// rekeyed gives the event its v2 fingerprint and says which v1 one it replaces. -func rekeyed(v2 event.EventV2, old string) event.EventV2 { - v2.Fingerprint = event.FingerprintV2(v2.API, v2.Version, v2.ChangeType, v2.Subject, "") - if v2.Fingerprint != old { - v2.Supersedes = old - } - return v2 -} - -// useInstead is the hint retire.Events writes: "Use instead: …". -var useInstead = regexp.MustCompile("^Use (.+?) instead: ") - -// replacement is the one model the provider names, when its replacement cell -// is exactly one model name: not none ("names no replacement model"), not -// several, not a model with a setting (`gpt-5.6-sol` (`reasoning.mode: pro`)). -// The patch-it client opens an issue for the rest, as it does today. -func replacement(hint string) string { - m := useInstead.FindStringSubmatch(hint) - if m == nil { - return "" - } - name := retire.First(m[1]) - if name == "" || strings.TrimSpace(m[1]) != "`"+name+"`" { - return "" - } - return name -} - -// Fresh converts events a source has just produced in the v1 shape. Unlike a -// stored row, a fresh event replaces nothing, so it carries no `supersedes`. An -// event that can't be placed is returned as an error, like a failed validation. -func Fresh(events []event.ChangeEvent) ([]event.EventV2, []error) { - var out []event.EventV2 - var errs []error - for _, e := range events { - v2, err := ToV2(e) - if err == nil { - v2.Supersedes = "" - err = v2.Validate() - } - if err != nil { - errs = append(errs, err) - continue - } - out = append(out, v2) - } - return out, errs -} diff --git a/plugin/plugin.go b/plugin/plugin.go index 14e919c..4b0fb6e 100644 --- a/plugin/plugin.go +++ b/plugin/plugin.go @@ -1,141 +1,27 @@ -// Package plugin is what a change source is: every source is a plugin behind -// one interface, registered by name. A plugin reads its source and returns a -// Result; it stores nothing itself. Whoever runs it (patch-it's hosted -// ingestion service, or the test kit) stores the events, then the baselines. -// Adding a source is a new package under plugins/ that calls Register; nothing -// here changes. +// Package plugin is the old import path of github.com/patch-it-dev/patch-it-plugins/core/plugin. +// +// Deprecated: import github.com/patch-it-dev/patch-it-plugins/core/plugin +// instead. This package only forwards to it, so code written against the old +// path keeps building; it will be removed in a later release. package plugin -import ( - "context" - "crypto/sha256" - "encoding/hex" - "fmt" - "slices" - "sync" - "time" +import moved "github.com/patch-it-dev/patch-it-plugins/core/plugin" - "github.com/patch-it-dev/patch-it-plugins/event" +// Deprecated: the types live in github.com/patch-it-dev/patch-it-plugins/core/plugin. +type ( + BaselineReader = moved.BaselineReader + Env = moved.Env + Plugin = moved.Plugin + Replayer = moved.Replayer + Result = moved.Result + Spec = moved.Spec ) -// Plugin is one place changes come from. -type Plugin interface { - // Name is the slug the command line selects it by and its events carry in - // `api`: "stripe", "openai-models". - Name() string - // Run reads the source as it is now and returns what changed since the - // baseline it saved last (env.Baselines.LastSpec). An error stores nothing. - Run(ctx context.Context, env Env) (Result, error) -} - -// Replayer is a plugin that can also diff two given releases (--from, --to): -// how a historical change is recreated without moving the normal baseline. -type Replayer interface { - Replay(ctx context.Context, env Env, from, to string) (Result, error) -} - -// Env is everything a plugin may reach outside itself, so a test can hand it -// baselines, the network and a clock of its own. A plugin that fetches through -// its own client (the OpenAPI plugin's adapter) ignores Get; its tests fake -// that client instead. -type Env struct { - Baselines BaselineReader - // Get fetches one URL. In both the hosted ingestion service and the test - // kit, this is fetch.Cache.Get (kit/kittest's Pages.Get wraps saved files - // the same way): a non-2xx, non-304 response fails with a *fetch.StatusError - // naming the status, returned as is, so errors.As finds it. A plugin that - // branches on the status (eol, when the provider's own site blocks a plain - // GET) can rely on that. - Get func(ctx context.Context, url string) ([]byte, error) - Now func() time.Time -} - -// BaselineReader is the one thing a plugin reads back: the baseline its last -// run saved. Ingest's store implements it; so does the test kit. -type BaselineReader interface { - // LastSpec is api's latest baseline, or nil (and no error) on a first run. - LastSpec(api string) (*Spec, error) -} - -// Spec is one baseline: a fetched copy of what a source said, kept only so a -// later run can diff against it, or tell in one hash comparison that nothing -// changed. -type Spec struct { - API string - Version string - Tag string - SHA256 string - SourceURL string - FetchedAt time.Time - Replay bool - Content []byte -} - -// Result is what one run found. -type Result struct { - Label string // what was read, for the summary: a tag, a snapshot, a URL - Events []event.EventV2 // valid v2 events, to store - Rejected []error // changes that failed to become valid events - Baselines []Spec // what the run read, saved after the events -} - +// Deprecated: the functions and values live in github.com/patch-it-dev/patch-it-plugins/core/plugin. var ( - mu sync.Mutex - registry = map[string]Plugin{} + Baseline = moved.Baseline + Lookup = moved.Lookup + Names = moved.Names + Register = moved.Register + SHA256 = moved.SHA256 ) - -// Register adds a plugin. A second plugin with the same name is a programming -// error: two sources would write the same `api`. So is the name "all", which -// `--api all` means. -func Register(p Plugin) { - mu.Lock() - defer mu.Unlock() - if p.Name() == "all" { - panic(`plugin: "all" means every plugin; a plugin can't be named it`) - } - if _, dup := registry[p.Name()]; dup { - panic(fmt.Sprintf("plugin: a plugin named %q is already registered", p.Name())) - } - registry[p.Name()] = p -} - -// Lookup finds a registered plugin by name. -func Lookup(name string) (Plugin, bool) { - mu.Lock() - defer mu.Unlock() - p, ok := registry[name] - return p, ok -} - -// Names are the registered plugins, sorted. -func Names() []string { - mu.Lock() - defer mu.Unlock() - names := make([]string, 0, len(registry)) - for n := range registry { - names = append(names, n) - } - slices.Sort(names) - return names -} - -// Baseline is what a plugin saves as the last thing it read, hashed so the -// next run can tell an unchanged source in one comparison. -func Baseline(api, version, tag, sourceURL string, content []byte, replay bool, at time.Time) Spec { - return Spec{ - API: api, - Version: version, - Tag: tag, - SHA256: SHA256(content), - SourceURL: sourceURL, - FetchedAt: at, - Replay: replay, - Content: content, - } -} - -// SHA256 is the hex hash a baseline is compared by. -func SHA256(data []byte) string { - sum := sha256.Sum256(data) - return hex.EncodeToString(sum[:]) -} diff --git a/plugins/all/all_test.go b/plugins/all/all_test.go index 5b4f030..e4402a0 100644 --- a/plugins/all/all_test.go +++ b/plugins/all/all_test.go @@ -10,7 +10,7 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" ) // Every package next to this one that registers a plugin is imported here, so @@ -44,7 +44,8 @@ func TestEveryPluginIsImported(t *testing.T) { } } -// registers reports whether a package's non-test code calls plugin.Register. +// registers reports whether a package's non-test code calls +// plugin.RegisterProvider (or plugin.Register). func registers(t *testing.T, dir string) bool { t.Helper() files, err := filepath.Glob(filepath.Join(dir, "*.go")) @@ -59,7 +60,7 @@ func registers(t *testing.T, dir string) bool { if err != nil { t.Fatal(err) } - if strings.Contains(string(b), "plugin.Register(") { + if strings.Contains(string(b), "plugin.Register(") || strings.Contains(string(b), "plugin.RegisterProvider(") { return true } } @@ -76,3 +77,30 @@ func TestImportingAllRegistersThePlugins(t *testing.T) { t.Errorf("names %v: want sorted and unique", names) } } + +// Every registered plugin is a source with complete metadata (RegisterProvider +// checks the fields), reachable by its full id and its old slug, and its page +// exists. +func TestEveryPluginIsASourceWithMetadata(t *testing.T) { + covered := map[string]bool{} + for _, s := range plugin.Sources() { + covered[s.Alias()] = true + if _, err := os.Stat(filepath.Join("..", "..", s.Source.Page)); err != nil { + t.Errorf("%s: its page %s doesn't exist (copy providers/_template.md)", s.FullID(), s.Source.Page) + } + for _, id := range []string{s.FullID(), s.Alias()} { + got, ok := plugin.LookupSource(id) + if !ok || got.FullID() != s.FullID() { + t.Errorf("LookupSource(%q) = %q, %v; want %s", id, got.FullID(), ok, s.FullID()) + } + if p, ok := plugin.Lookup(id); !ok || p.Name() != s.Alias() { + t.Errorf("Lookup(%q) didn't find %s", id, s.Alias()) + } + } + } + for _, name := range plugin.Names() { + if !covered[name] { + t.Errorf("%s is registered with plugin.Register only: register it as a source of its provider (plugin.RegisterProvider) with its metadata", name) + } + } +} diff --git a/plugins/apichangelog/apichangelog.go b/plugins/apichangelog/apichangelog.go index fb251e3..271129f 100644 --- a/plugins/apichangelog/apichangelog.go +++ b/plugins/apichangelog/apichangelog.go @@ -23,13 +23,22 @@ import ( "log" "sort" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/kit" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/github" + "github.com/patch-it-dev/patch-it-plugins/core/kit" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" ) func init() { - plugin.Register(Plugin{Log: MongoDBAtlas}) + plugin.RegisterProvider(plugin.Provider{ + ID: "mongodb", Name: "MongoDB", Category: plugin.CategoryDatabases, Homepage: "https://www.mongodb.com", + Sources: []plugin.Source{{ + ID: "atlas-admin-api", Layer: plugin.LayerAPI, Page: "providers/mongodb-atlas.md", + Watches: "Atlas Administration API versions deprecated and removed at their sunset date, per endpoint, from MongoDB's own API changelog", + URLs: []string{"https://github.com/mongodb/openapi/blob/main/changelog/changelog.json"}, + Plugin: Plugin{Log: MongoDBAtlas}, + }}, + }) } // Log is one vendor's changelog file. @@ -209,20 +218,10 @@ func (p Plugin) Name() string { return p.Log.API } // Run reads the file at the newest commit that changed it. func (p Plugin) Run(ctx context.Context, env plugin.Env) (plugin.Result, error) { l := p.Log - body, err := env.Get(ctx, l.CommitsURL()) + sha, err := github.LatestCommit(ctx, env.Get, l.CommitsURL()) if err != nil { - return plugin.Result{}, fmt.Errorf("looking up %s's newest changelog commit: %w", l.API, err) + return plugin.Result{}, fmt.Errorf("looking up %s's newest changelog commit (%s in %s): %w", l.API, l.Path, l.Repo, err) } - var commits []struct { - SHA string `json:"sha"` - } - if err := json.Unmarshal(body, &commits); err != nil { - return plugin.Result{}, fmt.Errorf("parsing github's commits response: %w", err) - } - if len(commits) == 0 || commits[0].SHA == "" { - return plugin.Result{}, fmt.Errorf("github lists no commit that changed %s in %s", l.Path, l.Repo) - } - sha := commits[0].SHA url := l.URL(sha) page, err := env.Get(ctx, url) if err != nil { diff --git a/plugins/apichangelog/apichangelog_test.go b/plugins/apichangelog/apichangelog_test.go index 21b1f04..21a0eea 100644 --- a/plugins/apichangelog/apichangelog_test.go +++ b/plugins/apichangelog/apichangelog_test.go @@ -7,8 +7,8 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" ) const ( diff --git a/plugins/catalog/catalog.go b/plugins/catalog/catalog.go index a89743e..9c257e4 100644 --- a/plugins/catalog/catalog.go +++ b/plugins/catalog/catalog.go @@ -22,12 +22,21 @@ import ( "sort" "time" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/github" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" ) func init() { - plugin.Register(Plugin{Catalog: Redis}) + plugin.RegisterProvider(plugin.Provider{ + ID: "redis", Name: "Redis", Category: plugin.CategoryDatabases, Homepage: "https://redis.io", + Sources: []plugin.Source{{ + ID: "commands", Layer: plugin.LayerRuntime, Page: "providers/redis.md", + Watches: "Commands removed entirely from Redis's own command catalog between two snapshots", + URLs: []string{"https://github.com/redis/redis-doc/blob/master/commands.json"}, + Plugin: Plugin{Catalog: Redis}, + }}, + }) } // Catalog is one provider's command or endpoint catalog: a single JSON @@ -154,20 +163,11 @@ func (p Plugin) Replay(ctx context.Context, env plugin.Env, from, to string) (pl // latestRef asks GitHub for the catalog repo's newest commit. func (p Plugin) latestRef(ctx context.Context, env plugin.Env) (string, error) { - body, err := env.Get(ctx, p.Catalog.RefsURL) + sha, err := github.Head(ctx, env.Get, p.Catalog.RefsURL) if err != nil { return "", fmt.Errorf("looking up the newest commit: %w", err) } - var commit struct { - SHA string `json:"sha"` - } - if err := json.Unmarshal(body, &commit); err != nil { - return "", fmt.Errorf("parsing the commit lookup: %w", err) - } - if commit.SHA == "" { - return "", fmt.Errorf("the commit lookup named no sha") - } - return commit.SHA, nil + return sha, nil } // events diffs two parsed snapshots into events. oldContent is empty on a diff --git a/plugins/catalog/catalog_test.go b/plugins/catalog/catalog_test.go index 5d22323..ad636b2 100644 --- a/plugins/catalog/catalog_test.go +++ b/plugins/catalog/catalog_test.go @@ -5,9 +5,9 @@ import ( "testing" "time" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" ) // The two commits redis-doc's own history shows the removal happen between: diff --git a/plugins/catalog/registered_test.go b/plugins/catalog/registered_test.go index 6a078fa..66cb74d 100644 --- a/plugins/catalog/registered_test.go +++ b/plugins/catalog/registered_test.go @@ -3,7 +3,7 @@ package catalog import ( "testing" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" ) func TestRegistersItself(t *testing.T) { diff --git a/plugins/changelog/changelog.go b/plugins/changelog/changelog.go index 94a77ac..25bea03 100644 --- a/plugins/changelog/changelog.go +++ b/plugins/changelog/changelog.go @@ -20,42 +20,75 @@ import ( "log" "strings" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/versions" ) func init() { - plugin.Register(Plugin{File: Textual0860}) - plugin.Register(Plugin{File: Urllib3200}) + plugin.RegisterProvider(plugin.Provider{ + ID: "textual", Name: "Textual", Category: plugin.CategoryPackages, Homepage: "https://textual.textualize.io", + Sources: []plugin.Source{{ + ID: "textual", Layer: plugin.LayerPackage, Page: "providers/textual.md", + Watches: "Names removed, renamed or deprecated in each release, from the release's section of CHANGELOG.md at its tag", + URLs: []string{"https://github.com/Textualize/textual/blob/main/CHANGELOG.md", "https://pypi.org/project/textual/"}, + Plugin: Plugin{File: Textual}, + }}, + }) + plugin.RegisterProvider(plugin.Provider{ + ID: "urllib3", Name: "urllib3", Category: plugin.CategoryPackages, Homepage: "https://urllib3.readthedocs.io", + Sources: []plugin.Source{{ + ID: "urllib3", Layer: plugin.LayerPackage, Page: "providers/urllib3.md", + Watches: "Names removed or deprecated in each release, from the release's section of CHANGES.rst at its tag", + URLs: []string{"https://github.com/urllib3/urllib3/blob/main/CHANGES.rst", "https://pypi.org/project/urllib3/"}, + Plugin: Plugin{File: Urllib3}, + }}, + }) } -// File is one package's changelog at one release. +// File is one package's changelog. Unpinned (Tag and Release empty), it is +// read release by release: every version on the package registry, the file at +// that release's tag (TagPrefix + version). Pinned (At), it is one release. type File struct { Package string // the installed name and the events' api: "urllib3" - Ecosystem string // "pypi" + Ecosystem string // where its releases are listed: "pypi" Module string // the import name, stripped from a symbol's front: "urllib3" Repo string // owner/name on GitHub - Tag string // the release's tag: "2.0.0", "v0.86.0" - Release string // the release as the changelog heads it: "2.0.0" + TagPrefix string // a release's git tag is TagPrefix + its version: "v" for v0.86.0 Path string // the changelog's path in the repo + Tag string // pinned: the release's tag: "2.0.0", "v0.86.0" + Release string // pinned: the release as the changelog heads it: "2.0.0" } -// Textual0860 is Textual's Keep a Changelog CHANGELOG.md at v0.86.0 (checked -// 2026-11-14: "## [0.86.0] - 2024-11-16", "### Removed", and "Breaking -// change:" bullets under "### Changed"). -var Textual0860 = File{ +// At is f pinned to one release. +func (f File) At(release string) File { + f.Release, f.Tag = release, f.TagPrefix+release + return f +} + +// Textual is Textual's Keep a Changelog CHANGELOG.md (checked 2026-11-14 at +// v0.86.0: "## [0.86.0] - 2024-11-16", "### Removed", and "Breaking change:" +// bullets under "### Changed"), tagged v. +var Textual = File{ Package: "textual", Ecosystem: "pypi", Module: "textual", - Repo: "Textualize/textual", Tag: "v0.86.0", Release: "0.86.0", Path: "CHANGELOG.md", + Repo: "Textualize/textual", TagPrefix: "v", Path: "CHANGELOG.md", } -// Urllib3200 is urllib3's CHANGES.rst at 2.0.0 (checked 2026-11-14: "2.0.0 -// (2023-04-26)" over "=====", "Removed" and "Deprecated" over "-----"). -var Urllib3200 = File{ +// Urllib3 is urllib3's CHANGES.rst (checked 2026-11-14 at 2.0.0: "2.0.0 +// (2023-04-26)" over "=====", "Removed" and "Deprecated" over "-----"), +// tagged . +var Urllib3 = File{ Package: "urllib3", Ecosystem: "pypi", Module: "urllib3", - Repo: "urllib3/urllib3", Tag: "2.0.0", Release: "2.0.0", Path: "CHANGES.rst", + Repo: "urllib3/urllib3", Path: "CHANGES.rst", } -// URL is the changelog as it was at the tag: its own provenance. +// The releases the golden files pin. +var ( + Textual0860 = Textual.At("0.86.0") + Urllib3200 = Urllib3.At("2.0.0") +) + +// URL is the changelog as it was at the (pinned) tag: its own provenance. func (f File) URL() string { return fmt.Sprintf("https://raw.githubusercontent.com/%s/%s/%s", f.Repo, f.Tag, f.Path) } @@ -114,20 +147,86 @@ func (f File) added(sections []Section) map[string]bool { return out } -// Plugin reads one changelog at one release. +// Plugin reads a package's changelog: release by release (ListVersions, +// ReadAt, Extract), or, with a pinned File, that one release. type Plugin struct { File File } func (p Plugin) Name() string { return p.File.Package } +// Run reads the pinned release, or the newest one on the registry, and +// stores its events. A release's tag never changes, so a rerun stores nothing +// new: the fingerprints dedupe. func (p Plugin) Run(ctx context.Context, env plugin.Env) (plugin.Result, error) { + v := plugin.Version{ID: p.File.Release, Ref: p.File.Tag} + if p.File.Tag == "" { + vs, err := p.ListVersions(ctx, env) + if err != nil { + return plugin.Result{}, err + } + if len(vs) == 0 { + return plugin.Result{}, fmt.Errorf("%s: no release listed", p.File.Package) + } + v = vs[len(vs)-1] + } + to, err := p.ReadAt(ctx, env, v) + if err != nil { + return plugin.Result{}, err + } + res, err := p.Extract(ctx, env, plugin.Snapshot{}, to) + if err != nil { + return plugin.Result{}, err + } + res.Baselines = []plugin.Spec{plugin.Baseline(p.File.Package, v.ID, v.Ref, to.SourceURL, to.Content, false, env.Now().UTC())} + return res, nil +} + +// ListVersions is every final release of the package on its registry, oldest +// first, each read at TagPrefix + version; a pinned File has its one release. +func (p Plugin) ListVersions(ctx context.Context, env plugin.Env) ([]plugin.Version, error) { f := p.File + if f.Tag != "" { + return []plugin.Version{{ID: f.Release, Ref: f.Tag}}, nil + } + var vs []plugin.Version + var err error + switch f.Ecosystem { + case "pypi": + vs, err = versions.PyPIReleases(ctx, env.Get, f.Package) + case "npm": + vs, err = versions.NPMReleases(ctx, env.Get, f.Package) + default: + return nil, fmt.Errorf("%s: no registry to list %q releases from", f.Package, f.Ecosystem) + } + if err != nil { + return nil, err + } + for i := range vs { + vs[i].Ref = f.TagPrefix + vs[i].ID + } + return vs, nil +} + +// ReadAt fetches the changelog at the release's tag. +func (p Plugin) ReadAt(ctx context.Context, env plugin.Env, v plugin.Version) (plugin.Snapshot, error) { + f := p.File + f.Release, f.Tag = v.ID, v.Ref page, err := env.Get(ctx, f.URL()) if err != nil { - return plugin.Result{}, fmt.Errorf("fetching %s: %w", f.URL(), err) + return plugin.Snapshot{}, fmt.Errorf("fetching %s: %w", f.URL(), err) } - heading, findings, err := f.Read(page) + return plugin.Snapshot{Version: v, Content: page, SourceURL: f.URL()}, nil +} + +// Extract is the to release's section of the changelog: an event per bullet +// that plainly removes, renames or deprecates a name, with previous_version +// from's release. Releases are extracted pair by pair, so from is the release +// right before to and to's section is everything between them. +func (p Plugin) Extract(_ context.Context, env plugin.Env, from, to plugin.Snapshot) (plugin.Result, error) { + f := p.File + f.Release, f.Tag = to.Version.ID, to.Version.Ref + heading, findings, err := f.Read(to.Content) if err != nil { return plugin.Result{}, fmt.Errorf("reading %s: %w", f.URL(), err) } @@ -145,17 +244,18 @@ func (p Plugin) Run(ctx context.Context, env plugin.Env) (plugin.Result, error) Change: c.Change, Symbols: c.Symbols, Replacement: c.Replacement, } e := event.EventV2{ - API: f.Package, - Version: f.Release, - ChangeType: c.ChangeType, - Breaking: c.ChangeType == event.ChangeTypeBreaking, - Subject: subject, - Description: describe(f, c), - MigrationHint: cap2000(fmt.Sprintf("%s (%s's changelog for %s)", fd.Bullet.Text, f.Package, f.Release)), - SourceURL: f.URL(), - SourceExcerpt: cap2000(heading + "\n" + fd.Section + "\n" + fd.Bullet.Raw), - Confidence: 1.0, - DetectedAt: now, + API: f.Package, + Version: f.Release, + PreviousVersion: from.Version.ID, + ChangeType: c.ChangeType, + Breaking: c.ChangeType == event.ChangeTypeBreaking, + Subject: subject, + Description: describe(f, c), + MigrationHint: cap2000(fmt.Sprintf("%s (%s's changelog for %s)", fd.Bullet.Text, f.Package, f.Release)), + SourceURL: f.URL(), + SourceExcerpt: cap2000(heading + "\n" + fd.Section + "\n" + fd.Bullet.Raw), + Confidence: 1.0, + DetectedAt: now, } e.Fingerprint = event.FingerprintV2(e.API, e.Version, e.ChangeType, subject, "") if err := e.Validate(); err != nil { @@ -164,14 +264,11 @@ func (p Plugin) Run(ctx context.Context, env plugin.Env) (plugin.Result, error) } events = append(events, e) } - return plugin.Result{ - Label: f.URL(), - Events: events, - Rejected: rejected, - Baselines: []plugin.Spec{plugin.Baseline(f.Package, f.Release, f.Tag, f.URL(), page, false, now)}, - }, nil + return plugin.Result{Label: f.URL(), Events: events, Rejected: rejected}, nil } +var _ plugin.Versioned = Plugin{} + func describe(f File, c Change) string { names := "`" + strings.Join(c.Symbols, "`, `") + "`" verb := "was" diff --git a/plugins/changelog/changelog_test.go b/plugins/changelog/changelog_test.go index 07a184e..eeb0491 100644 --- a/plugins/changelog/changelog_test.go +++ b/plugins/changelog/changelog_test.go @@ -4,7 +4,7 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" ) // Two real changelogs at their release tags (testdata/README.md) in, every diff --git a/plugins/changelog/classify.go b/plugins/changelog/classify.go index 1343455..3b50675 100644 --- a/plugins/changelog/classify.go +++ b/plugins/changelog/classify.go @@ -6,7 +6,7 @@ import ( "slices" "strings" - "github.com/patch-it-dev/patch-it-plugins/event" + "github.com/patch-it-dev/patch-it-plugins/core/event" ) // Kind is what a section (or a bullet's own prefix) says its changes are. diff --git a/plugins/changelog/registered_test.go b/plugins/changelog/registered_test.go index 424d6be..0a182c8 100644 --- a/plugins/changelog/registered_test.go +++ b/plugins/changelog/registered_test.go @@ -3,7 +3,7 @@ package changelog import ( "testing" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" ) func TestRegistersItself(t *testing.T) { diff --git a/plugins/changelog/testdata/README.md b/plugins/changelog/testdata/README.md index 0f33ed9..8196285 100644 --- a/plugins/changelog/testdata/README.md +++ b/plugins/changelog/testdata/README.md @@ -8,3 +8,9 @@ golden files under `golden/` are what the plugin stores from them |---|---|---|---|---|---| | `textual-v0.86.0-CHANGELOG.md` | https://raw.githubusercontent.com/Textualize/textual/v0.86.0/CHANGELOG.md | `5a738fa` | 2026-11-14 08:36 UTC | MIT (Textual), kept whole | "Breaking change: Removed `App.dark` reactive attribute" | | `urllib3-2.0.0-CHANGES.rst` | https://raw.githubusercontent.com/urllib3/urllib3/2.0.0/CHANGES.rst | `303ba81` | 2026-11-14 08:35 UTC | MIT (urllib3), kept whole | "Removed support for Python 2.7, 3.5, and 3.6" | + +## Version lists + +| File | URL | Saved | License | Use | +|---|---|---|---|---| +| `pypi-urllib3-excerpt.json` | https://pypi.org/pypi/urllib3/json (shape only) | written for the test, 2026-11-20 | not a saved copy: four releases in the shape of PyPI's JSON API (`releases` → files with `upload_time_iso_8601`, `yanked`), dates approximate, a pre-release among them | for the version-listing test only (ListVersions, consecutive pairs); no golden reads it | diff --git a/plugins/changelog/testdata/pypi-urllib3-excerpt.json b/plugins/changelog/testdata/pypi-urllib3-excerpt.json new file mode 100644 index 0000000..43e63ae --- /dev/null +++ b/plugins/changelog/testdata/pypi-urllib3-excerpt.json @@ -0,0 +1,6 @@ +{"releases":{ + "1.26.15":[{"upload_time_iso_8601":"2023-03-10T18:31:28.000000Z","yanked":false}], + "2.0.0a1":[{"upload_time_iso_8601":"2022-11-15T19:43:59.000000Z","yanked":false}], + "2.0.0":[{"upload_time_iso_8601":"2023-04-26T16:43:58.000000Z","yanked":false}], + "2.0.1":[{"upload_time_iso_8601":"2023-04-30T18:42:25.000000Z","yanked":false}] +}} diff --git a/plugins/changelog/versions_test.go b/plugins/changelog/versions_test.go new file mode 100644 index 0000000..803fc34 --- /dev/null +++ b/plugins/changelog/versions_test.go @@ -0,0 +1,73 @@ +package changelog + +import ( + "context" + "strings" + "testing" + + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/versions" +) + +// Unpinned, a changelog's versions are the package's final releases on PyPI, +// each read at its tag; the 1.26.15 -> 2.0.0 pair gives the same events as +// the pinned golden, with previous_version set. +func TestReleasesAreListedAndExtractedPairByPair(t *testing.T) { + pages := kittest.Pages{ + "https://pypi.org/pypi/urllib3/json": "testdata/pypi-urllib3-excerpt.json", + Urllib3.At("1.26.15").URL(): "testdata/urllib3-2.0.0-CHANGES.rst", // only 2.0.0's section is read + Urllib3200.URL(): "testdata/urllib3-2.0.0-CHANGES.rst", + } + env := kittest.Env(t, pages) + p := Plugin{File: Urllib3} + ctx := context.Background() + vs, err := p.ListVersions(ctx, env) + if err != nil { + t.Fatal(err) + } + var ids []string + for _, v := range vs { + ids = append(ids, v.ID+"@"+v.Ref) + } + if got := strings.Join(ids, " "); got != "1.26.15@1.26.15 2.0.0@2.0.0 2.0.1@2.0.1" { + t.Fatalf("versions %s", got) + } + pair := versions.Pairs(vs)[0] + from, err := p.ReadAt(ctx, env, pair[0]) + if err != nil { + t.Fatal(err) + } + to, err := p.ReadAt(ctx, env, pair[1]) + if err != nil { + t.Fatal(err) + } + res, err := p.Extract(ctx, env, from, to) + if err != nil || len(res.Rejected) > 0 { + t.Fatalf("%v %v", err, res.Rejected) + } + pinned := kittest.Ingest(t, Plugin{File: Urllib3200}, kittest.Pages{Urllib3200.URL(): "testdata/urllib3-2.0.0-CHANGES.rst"}) + if len(res.Events) == 0 || len(res.Events) != len(pinned) { + t.Fatalf("pair gave %d events, pinned run %d", len(res.Events), len(pinned)) + } + for i, e := range res.Events { + if e.PreviousVersion != "1.26.15" || e.Version != "2.0.0" || e.Fingerprint != pinned[i].Fingerprint { + t.Errorf("event %d: %s<-%s %s, pinned %s", i, e.Version, e.PreviousVersion, e.Fingerprint, pinned[i].Fingerprint) + } + } +} + +// Run on an unpinned file reads the newest release on PyPI. +func TestRunReadsTheNewestRelease(t *testing.T) { + pages := kittest.Pages{ + "https://pypi.org/pypi/urllib3/json": "testdata/pypi-urllib3-excerpt.json", + Urllib3.At("2.0.1").URL(): "testdata/urllib3-2.0.0-CHANGES.rst", + } + _, err := Plugin{File: Urllib3}.Run(context.Background(), kittest.Env(t, pages)) + // The saved file is 2.0.0's, which has no 2.0.1 section: the run says so + // rather than inventing events. + if err == nil || !strings.Contains(err.Error(), "2.0.1") { + t.Errorf("err = %v, want one naming 2.0.1", err) + } + var _ plugin.Versioned = Plugin{} +} diff --git a/plugins/eol/eol.go b/plugins/eol/eol.go index e6e415d..4bd1e31 100644 --- a/plugins/eol/eol.go +++ b/plugins/eol/eol.go @@ -31,14 +31,22 @@ import ( "strconv" "strings" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/fetch" - "github.com/patch-it-dev/patch-it-plugins/kit" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/kit" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" ) func init() { - plugin.Register(Plugin{Page: Salesforce, Archiver: kit.Wayback{}}) + plugin.RegisterProvider(plugin.Provider{ + ID: "salesforce", Name: "Salesforce", Category: plugin.CategoryCRM, Homepage: "https://www.salesforce.com", + Sources: []plugin.Source{{ + ID: "rest-api", Layer: plugin.LayerAPI, Page: "providers/salesforce.md", + Watches: "REST API version ranges deprecated and retired, from Salesforce's own API end-of-life policy page", + URLs: []string{"https://developer.salesforce.com/docs/platform/api-rest/guide/api-rest-eol.html"}, + Plugin: Plugin{Page: Salesforce, Archiver: kit.Wayback{}}, + }}, + }) } // Page is one vendor's end-of-life table. diff --git a/plugins/eol/eol_test.go b/plugins/eol/eol_test.go index fbdf736..6ceac40 100644 --- a/plugins/eol/eol_test.go +++ b/plugins/eol/eol_test.go @@ -6,9 +6,9 @@ import ( "testing" "time" - "github.com/patch-it-dev/patch-it-plugins/fetch" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" ) func TestRegistersItself(t *testing.T) { diff --git a/plugins/feed/cloudflare_test.go b/plugins/feed/cloudflare_test.go index 9f56fe0..83d23fe 100644 --- a/plugins/feed/cloudflare_test.go +++ b/plugins/feed/cloudflare_test.go @@ -4,8 +4,8 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" ) const cloudflareFeed = "testdata/cloudflare-api-deprecations.rss.xml" diff --git a/plugins/feed/feed.go b/plugins/feed/feed.go index e5383a4..dba0f77 100644 --- a/plugins/feed/feed.go +++ b/plugins/feed/feed.go @@ -25,15 +25,39 @@ import ( "fmt" "log" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/kit" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/kit" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" ) func init() { - plugin.Register(Plugin{Feed: GitHubActions, Archiver: kit.Wayback{}}) - plugin.Register(Plugin{Feed: VertexAIModels, Archiver: kit.Wayback{}}) - plugin.Register(Plugin{Feed: CloudflareAPI, Archiver: kit.Wayback{}}) + plugin.RegisterProvider(plugin.Provider{ + ID: "github", Name: "GitHub", Category: plugin.CategoryDevTools, Homepage: "https://github.com", + Sources: []plugin.Source{{ + ID: "actions", Layer: plugin.LayerCI, Page: "providers/github-actions.md", + Watches: "GitHub Actions retirements, such as runner images and the `runs-on` labels they remove", + URLs: []string{"https://github.blog/changelog/label/actions/feed/"}, + Plugin: Plugin{Feed: GitHubActions, Archiver: kit.Wayback{}}, + }}, + }) + plugin.RegisterProvider(plugin.Provider{ + ID: "google", Name: "Google", Category: plugin.CategoryAI, Homepage: "https://cloud.google.com", + Sources: []plugin.Source{{ + ID: "vertex-ai-models", Layer: plugin.LayerAIModels, Page: "providers/vertex-ai-models.md", + Watches: "Generative AI model deprecations on Vertex AI", + URLs: []string{"https://docs.cloud.google.com/feeds/generative-ai-on-vertex-ai-release-notes.xml"}, + Plugin: Plugin{Feed: VertexAIModels, Archiver: kit.Wayback{}}, + }}, + }) + plugin.RegisterProvider(plugin.Provider{ + ID: "cloudflare", Name: "Cloudflare", Category: plugin.CategoryInfrastructure, Homepage: "https://www.cloudflare.com", + Sources: []plugin.Source{{ + ID: "api", Layer: plugin.LayerAPI, Page: "providers/cloudflare.md", + Watches: "REST API operations Cloudflare deprecates, breaking on their end-of-life date when it gives one", + URLs: []string{"https://developers.cloudflare.com/fundamentals/api/reference/deprecations/index.xml"}, + Plugin: Plugin{Feed: CloudflareAPI, Archiver: kit.Wayback{}}, + }}, + }) } // Feed is one vendor feed and how to read its entries. diff --git a/plugins/feed/feed_test.go b/plugins/feed/feed_test.go index e361285..fe58a89 100644 --- a/plugins/feed/feed_test.go +++ b/plugins/feed/feed_test.go @@ -4,7 +4,7 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" ) // The two feeds as saved on 2026-11-14 (testdata/README.md), each through the diff --git a/plugins/feed/readers.go b/plugins/feed/readers.go index d94baa2..b343708 100644 --- a/plugins/feed/readers.go +++ b/plugins/feed/readers.go @@ -6,7 +6,7 @@ import ( "strings" "time" - "github.com/patch-it-dev/patch-it-plugins/event" + "github.com/patch-it-dev/patch-it-plugins/core/event" ) // GitHubActions is the GitHub changelog's Actions label, RSS 2.0 (checked diff --git a/plugins/feed/registered_test.go b/plugins/feed/registered_test.go index e548423..83c9671 100644 --- a/plugins/feed/registered_test.go +++ b/plugins/feed/registered_test.go @@ -3,7 +3,7 @@ package feed import ( "testing" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" ) func TestRegistersItself(t *testing.T) { diff --git a/plugins/migrationguide/golden_test.go b/plugins/migrationguide/golden_test.go index 91c37be..84124b6 100644 --- a/plugins/migrationguide/golden_test.go +++ b/plugins/migrationguide/golden_test.go @@ -3,14 +3,14 @@ package migrationguide import ( "testing" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" - "github.com/patch-it-dev/patch-it-plugins/sdkguide" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/plugins/migrationguide/sdkguide" ) // The guide at tag v1.0.0 (saved 2026-10-17) in, every event out. func TestGolden(t *testing.T) { got := kittest.Ingest(t, Plugin{Guide: sdkguide.AnthropicV1}, kittest.Pages{ - sdkguide.AnthropicV1.URL(): "../../sdkguide/testdata/anthropic-MIGRATION-v1.0.0.md", + sdkguide.AnthropicV1.URL(): "sdkguide/testdata/anthropic-MIGRATION-v1.0.0.md", }) kittest.Golden(t, "testdata/golden/anthropic-1.0.0.json", got) } diff --git a/plugins/migrationguide/migrationguide.go b/plugins/migrationguide/migrationguide.go index cf0da2f..825a8b4 100644 --- a/plugins/migrationguide/migrationguide.go +++ b/plugins/migrationguide/migrationguide.go @@ -1,46 +1,126 @@ // Package migrationguide is the source plugin for an SDK's migration guide, -// read at the release tag of the major it describes: an event per request -// parameter the release removed (sdkguide/). The tag never changes, so -// a rerun stores nothing new: the fingerprints dedupe. +// read at the release tag of each major it describes: an event per request +// parameter the release removed (sdkguide/). A tag never changes, so a rerun +// stores nothing new: the fingerprints dedupe. package migrationguide import ( "context" "fmt" - "github.com/patch-it-dev/patch-it-plugins/migrate" - "github.com/patch-it-dev/patch-it-plugins/plugin" - "github.com/patch-it-dev/patch-it-plugins/sdkguide" + "github.com/patch-it-dev/patch-it-plugins/core/migrate" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/versions" + "github.com/patch-it-dev/patch-it-plugins/plugins/migrationguide/sdkguide" ) func init() { - plugin.Register(Plugin{Guide: sdkguide.AnthropicV1}) + plugin.RegisterProvider(plugin.Provider{ + ID: "anthropic", Name: "Anthropic", Category: plugin.CategoryAI, Homepage: "https://www.anthropic.com", + Sources: []plugin.Source{{ + ID: "python-sdk", Layer: plugin.LayerSDK, Page: "providers/anthropic.md", + Watches: "Request parameters each new major of the Python SDK removes from `messages.create()` and friends, from MIGRATION.md at the major's tag", + URLs: []string{"https://github.com/anthropics/anthropic-sdk-python/blob/main/MIGRATION.md", "https://pypi.org/project/anthropic/"}, + Plugin: Plugin{Guide: sdkguide.Anthropic}, + }}, + }) } -// Plugin reads one SDK's guide to one major release. +// Plugin reads an SDK's migration guide: at the first release of each major +// (ListVersions, ReadAt, Extract), or, with a pinned Guide, that one major. type Plugin struct { Guide sdkguide.Guide } func (p Plugin) Name() string { return p.Guide.API } +// Run reads the pinned major's guide, or the newest major's. func (p Plugin) Run(ctx context.Context, env plugin.Env) (plugin.Result, error) { g := p.Guide + v := plugin.Version{ID: g.Version(), Ref: "v" + g.Version()} + if g.Major == 0 { + vs, err := p.ListVersions(ctx, env) + if err != nil { + return plugin.Result{}, err + } + v = plugin.Version{} + for _, c := range vs { + if firstOfMajor(c.ID) { + v = c + } + } + if v.ID == "" { + return plugin.Result{}, fmt.Errorf("%s: no major release listed", g.API) + } + } + to, err := p.ReadAt(ctx, env, v) + if err != nil { + return plugin.Result{}, err + } + res, err := p.Extract(ctx, env, plugin.Snapshot{}, to) + if err != nil { + return plugin.Result{}, err + } + res.Baselines = []plugin.Spec{plugin.Baseline(g.API, v.ID, v.Ref, to.SourceURL, to.Content, false, env.Now().UTC())} + return res, nil +} + +// ListVersions is every final release of the SDK on PyPI, oldest first; a +// pinned Guide has its one major. +func (p Plugin) ListVersions(ctx context.Context, env plugin.Env) ([]plugin.Version, error) { + g := p.Guide + if g.Major != 0 { + return []plugin.Version{{ID: g.Version(), Ref: "v" + g.Version()}}, nil + } + vs, err := versions.PyPIReleases(ctx, env.Get, g.API) + if err != nil { + return nil, err + } + for i := range vs { + vs[i].Ref = "v" + vs[i].ID + } + return vs, nil +} + +// ReadAt fetches the guide at the release's tag, when the release is the +// first of its major (X.0.0): only a new major has a guide to read. Any other +// release reads as nothing (a snapshot with no content). +func (p Plugin) ReadAt(ctx context.Context, env plugin.Env, v plugin.Version) (plugin.Snapshot, error) { + if !firstOfMajor(v.ID) { + return plugin.Snapshot{Version: v}, nil + } + g := p.Guide.At(versions.Major(v.ID)) page, err := env.Get(ctx, g.URL()) if err != nil { - return plugin.Result{}, fmt.Errorf("fetching %s: %w", g.URL(), err) + return plugin.Snapshot{}, fmt.Errorf("fetching %s: %w", g.URL(), err) } - removals, err := sdkguide.Parse(page, g) + return plugin.Snapshot{Version: v, Content: page, SourceURL: g.URL()}, nil +} + +// Extract is an event per request parameter the to major removed, read from +// its guide, with previous_version from's release. A release that isn't a new +// major has no guide and no events. +func (p Plugin) Extract(_ context.Context, env plugin.Env, from, to plugin.Snapshot) (plugin.Result, error) { + if to.Content == nil { + return plugin.Result{Label: to.Version.ID}, nil + } + g := p.Guide.At(versions.Major(to.Version.ID)) + removals, err := sdkguide.Parse(to.Content, g) if err != nil { return plugin.Result{}, fmt.Errorf("reading %s: %w", g.URL(), err) } - now := env.Now().UTC() - events, errs := sdkguide.Events(g, removals, now) + events, errs := sdkguide.Events(g, removals, env.Now().UTC()) + for i := range events { + events[i].PreviousVersion = from.Version.ID + } converted, invalid := migrate.Fresh(events) - return plugin.Result{ - Label: g.URL(), - Events: converted, - Rejected: append(errs, invalid...), - Baselines: []plugin.Spec{plugin.Baseline(g.API, g.Version(), "v"+g.Version(), g.URL(), page, false, now)}, - }, nil + return plugin.Result{Label: g.URL(), Events: converted, Rejected: append(errs, invalid...)}, nil } + +// firstOfMajor is true for X.0.0, the release a major's guide is tagged at. +func firstOfMajor(v string) bool { + m := versions.Major(v) + return m > 0 && v == fmt.Sprintf("%d.0.0", m) +} + +var _ plugin.Versioned = Plugin{} diff --git a/plugins/migrationguide/migrationguide_test.go b/plugins/migrationguide/migrationguide_test.go index ff2be52..d72a334 100644 --- a/plugins/migrationguide/migrationguide_test.go +++ b/plugins/migrationguide/migrationguide_test.go @@ -4,15 +4,15 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" - "github.com/patch-it-dev/patch-it-plugins/sdkguide" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/plugins/migrationguide/sdkguide" ) func TestAMigrationGuideStoresOneEventPerRemovedParameter(t *testing.T) { // Only the v1.0.0 tag is saved: fetching anything else fails the test, and // kittest's second run (the tag never changes) must store nothing new. events := kittest.Ingest(t, Plugin{Guide: sdkguide.AnthropicV1}, kittest.Pages{ - sdkguide.AnthropicV1.URL(): "../../sdkguide/testdata/anthropic-MIGRATION-v1.0.0.md", + sdkguide.AnthropicV1.URL(): "sdkguide/testdata/anthropic-MIGRATION-v1.0.0.md", }) var params []string for _, e := range events { diff --git a/plugins/migrationguide/registered_test.go b/plugins/migrationguide/registered_test.go index 4d07e5c..acd1ca5 100644 --- a/plugins/migrationguide/registered_test.go +++ b/plugins/migrationguide/registered_test.go @@ -3,8 +3,8 @@ package migrationguide import ( "testing" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" - "github.com/patch-it-dev/patch-it-plugins/sdkguide" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/plugins/migrationguide/sdkguide" ) func TestRegistersItself(t *testing.T) { diff --git a/plugins/migrationguide/sdkguide/sdkguide.go b/plugins/migrationguide/sdkguide/sdkguide.go new file mode 100644 index 0000000..903d194 --- /dev/null +++ b/plugins/migrationguide/sdkguide/sdkguide.go @@ -0,0 +1,183 @@ +// Package sdkguide reads an SDK's own migration guide for the request parameters +// a major release removed: a third kind of source, next to the OpenAPI differ and +// the deprecations pages. The guide is read at the release tag of the major it +// describes (`v1.0.0` for "Migrating to v1"), a URL that never changes, so it is +// its own provenance. +// +// The first is anthropic-sdk-python 1.0.0 (2026-08-20): its MIGRATION.md lists +// `temperature`, `top_p` and `top_k` as removed from `messages.create()` and its +// siblings, passing them is a `TypeError`, and it says what to do instead (checked +// 2026-10-17). A repo with `anthropic` unpinned gets 1.x on a fresh install. +package sdkguide + +import ( + "fmt" + "regexp" + "strings" + "time" + "unicode/utf8" + + "github.com/patch-it-dev/patch-it-plugins/core/event" +) + +// Guide is one SDK's migration guide for one major release. +type Guide struct { + API string // the SDK's slug in change events and its PyPI name, e.g. "anthropic" + Repo string // its GitHub repository, owner/name + Major int // pinned: the release the guide describes, 1 for "Migrating to v1"; 0 lists them + Operation string // the API operation the removed parameters were sent to + // Methods is what a table row's method cell must name for its parameters to + // count: the plain call, so a row about beta-only methods (where the same + // name can still be valid on a helper) isn't read as a removal. + Methods string + // Symbols are the calls the removed parameters can no longer be passed to, as + // code writes them after the import: the plain methods the guide's table names + // first (`messages.create()`, `messages.stream()`, `messages.parse()`). A v2 + // event carries them (the change event contract). + Symbols []string +} + +// Anthropic is anthropic-sdk-python's MIGRATION.md, unpinned: read at the +// first release of each major (Major 0; At pins it). +var Anthropic = Guide{ + API: "anthropic", + Repo: "anthropics/anthropic-sdk-python", + Operation: "POST /v1/messages", + Methods: "`messages.create()`", + Symbols: []string{"messages.create", "messages.stream", "messages.parse"}, +} + +// AnthropicV1 is anthropic-sdk-python's guide to 1.0.0: what the golden pins. +var AnthropicV1 = Anthropic.At(1) + +// At is g pinned to one major. +func (g Guide) At(major int) Guide { + g.Major = major + return g +} + +// Version is the release that removed the parameters. +func (g Guide) Version() string { return fmt.Sprintf("%d.0.0", g.Major) } + +// URL is the guide as it was at that release's tag. +func (g Guide) URL() string { + return fmt.Sprintf("https://raw.githubusercontent.com/%s/v%s/MIGRATION.md", g.Repo, g.Version()) +} + +// Page is the same guide for a person to read. +func (g Guide) Page() string { + return fmt.Sprintf("https://github.com/%s/blob/v%s/MIGRATION.md", g.Repo, g.Version()) +} + +// Removal is one parameter the release removed, with what the guide says to do. +type Removal struct { + Param string + Instead string // the guide's "use instead" cell, as written + Excerpt string // the section heading, the table header and the row +} + +var ticked = regexp.MustCompile("`([^`]+)`") + +// Parse reads the guide's removed-parameter tables: every row whose method cell +// names g.Methods gives one Removal per backticked parameter. A guide for another +// major, or one with no such row, is an error, never zero removals. +func Parse(page []byte, g Guide) ([]Removal, error) { + text := strings.ReplaceAll(string(page), "\r\n", "\n") + if !strings.Contains(text, fmt.Sprintf("# Migrating to v%d", g.Major)) { + return nil, fmt.Errorf("not the guide to v%d", g.Major) + } + lines := strings.Split(text, "\n") + var out []Removal + heading := "" + for i := 0; i < len(lines); i++ { + line := strings.TrimSpace(lines[i]) + if strings.HasPrefix(line, "#") { + heading = line + continue + } + header := cells(line) + param, instead := column(header, "removed parameter"), column(header, "instead") + if param < 0 || instead < 0 { + continue + } + i++ // the |---| row + for i+1 < len(lines) && strings.HasPrefix(strings.TrimSpace(lines[i+1]), "|") { + i++ + row := strings.TrimSpace(lines[i]) + c := cells(row) + if len(c) <= param || len(c) <= instead || !strings.Contains(c[0], g.Methods) { + continue + } + excerpt := cut(strings.Join([]string{heading, line, row}, "\n"), 2000) + for _, m := range ticked.FindAllStringSubmatch(c[param], -1) { + out = append(out, Removal{Param: m[1], Instead: c[instead], Excerpt: excerpt}) + } + } + } + if len(out) == 0 { + return nil, fmt.Errorf("no removed-parameter row naming %s: the guide's shape may have changed", g.Methods) + } + return out, nil +} + +func cells(row string) []string { + if !strings.HasPrefix(row, "|") { + return nil + } + parts := strings.Split(strings.Trim(row, "|"), "|") + for i := range parts { + parts[i] = strings.TrimSpace(parts[i]) + } + return parts +} + +func column(header []string, word string) int { + for i, h := range header { + if strings.Contains(strings.ToLower(h), word) { + return i + } + } + return -1 +} + +func cut(s string, n int) string { + if utf8.RuneCountInString(s) <= n { + return s + } + return string([]rune(s)[:n-1]) + "…" +} + +// Events turns removals into change events: one `breaking` request_parameter +// event per parameter, on the operation the SDK's methods send it to. +func Events(g Guide, rs []Removal, now time.Time) ([]event.ChangeEvent, []error) { + var events []event.ChangeEvent + var errs []error + for _, r := range rs { + affected := event.Affected{ + Kind: event.AffectedKindRequestParameter, + Side: event.SideRequest, + Operation: g.Operation, + Property: r.Param, + } + e := event.ChangeEvent{ + Fingerprint: event.Fingerprint(g.API, g.Version(), event.ChangeTypeBreaking, affected, "removed"), + API: g.API, + Version: g.Version(), + ChangeType: event.ChangeTypeBreaking, + Breaking: true, + Affected: affected, + Description: fmt.Sprintf("Parameter %s removed from %s and its sibling methods in the %s SDK %s: passing it raises a TypeError.", r.Param, g.Methods, g.API, g.Version()), + MigrationHint: fmt.Sprintf("%s Source: %s", r.Instead, g.Page()), + SourceURL: g.URL(), + SourceExcerpt: r.Excerpt, + Confidence: 1.0, + DetectedAt: now, + } + if err := e.Validate(); err != nil { + errs = append(errs, fmt.Errorf("%s %s: %w", g.API, r.Param, err)) + continue + } + events = append(events, e) + } + return events, errs +} diff --git a/sdkguide/sdkguide_test.go b/plugins/migrationguide/sdkguide/sdkguide_test.go similarity index 97% rename from sdkguide/sdkguide_test.go rename to plugins/migrationguide/sdkguide/sdkguide_test.go index 043b90d..d0c3fcb 100644 --- a/sdkguide/sdkguide_test.go +++ b/plugins/migrationguide/sdkguide/sdkguide_test.go @@ -6,7 +6,7 @@ import ( "testing" "time" - "github.com/patch-it-dev/patch-it-plugins/event" + "github.com/patch-it-dev/patch-it-plugins/core/event" ) func guide(t *testing.T) []byte { diff --git a/sdkguide/testdata/README.md b/plugins/migrationguide/sdkguide/testdata/README.md similarity index 100% rename from sdkguide/testdata/README.md rename to plugins/migrationguide/sdkguide/testdata/README.md diff --git a/sdkguide/testdata/anthropic-MIGRATION-v1.0.0.md b/plugins/migrationguide/sdkguide/testdata/anthropic-MIGRATION-v1.0.0.md similarity index 100% rename from sdkguide/testdata/anthropic-MIGRATION-v1.0.0.md rename to plugins/migrationguide/sdkguide/testdata/anthropic-MIGRATION-v1.0.0.md diff --git a/plugins/migrationguide/testdata/README.md b/plugins/migrationguide/testdata/README.md index 7adb466..7646c57 100644 --- a/plugins/migrationguide/testdata/README.md +++ b/plugins/migrationguide/testdata/README.md @@ -3,3 +3,9 @@ `golden/anthropic-1.0.0.json` is what the `anthropic` plugin stores from anthropic-sdk-python's MIGRATION.md at v1.0.0. The saved guide itself is shared with the `sdkguide` package: see [`sdkguide/testdata/README.md`](../../../sdkguide/testdata/README.md) for its URL, date and license. Rewrite it after an intended change with `go test ./plugins/migrationguide -update`, and say in the PR why the new output is right. + +## Version lists + +| File | URL | Saved | License | Use | +|---|---|---|---|---| +| `pypi-anthropic-excerpt.json` | https://pypi.org/pypi/anthropic/json (shape only) | written for the test, 2026-11-20 | not a saved copy: four releases in the shape of PyPI's JSON API (`releases` → files with `upload_time_iso_8601`, `yanked`), dates approximate, a pre-release among them | for the version-listing test only (ListVersions, consecutive pairs); no golden reads it | diff --git a/plugins/migrationguide/testdata/pypi-anthropic-excerpt.json b/plugins/migrationguide/testdata/pypi-anthropic-excerpt.json new file mode 100644 index 0000000..85fa85b --- /dev/null +++ b/plugins/migrationguide/testdata/pypi-anthropic-excerpt.json @@ -0,0 +1,6 @@ +{"releases":{ + "0.75.0":[{"upload_time_iso_8601":"2026-08-01T00:00:00Z","yanked":false}], + "1.0.0":[{"upload_time_iso_8601":"2026-08-20T00:00:00Z","yanked":false}], + "1.0.1":[{"upload_time_iso_8601":"2026-08-27T00:00:00Z","yanked":false}], + "1.1.0b1":[{"upload_time_iso_8601":"2026-09-01T00:00:00Z","yanked":false}] +}} diff --git a/plugins/migrationguide/versions_test.go b/plugins/migrationguide/versions_test.go new file mode 100644 index 0000000..9aa8f0a --- /dev/null +++ b/plugins/migrationguide/versions_test.go @@ -0,0 +1,62 @@ +package migrationguide + +import ( + "context" + "fmt" + "strings" + "testing" + + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/versions" + "github.com/patch-it-dev/patch-it-plugins/plugins/migrationguide/sdkguide" +) + +// Unpinned, the SDK's versions are its releases on PyPI; only the pair that +// crosses into a new major reads a guide (at that major's tag) and has +// events, with previous_version set; the others read nothing. +func TestOnlyANewMajorHasAGuide(t *testing.T) { + env := kittest.Env(t, kittest.Pages{ + "https://pypi.org/pypi/anthropic/json": "testdata/pypi-anthropic-excerpt.json", + sdkguide.AnthropicV1.URL(): "sdkguide/testdata/anthropic-MIGRATION-v1.0.0.md", + }) + p := Plugin{Guide: sdkguide.Anthropic} + ctx := context.Background() + vs, err := p.ListVersions(ctx, env) + if err != nil { + t.Fatal(err) + } + var got []string + for _, pair := range versions.Pairs(vs) { + from, err := p.ReadAt(ctx, env, pair[0]) + if err != nil { + t.Fatal(err) + } + to, err := p.ReadAt(ctx, env, pair[1]) + if err != nil { + t.Fatal(err) + } + res, err := p.Extract(ctx, env, from, to) + if err != nil || len(res.Rejected) > 0 { + t.Fatalf("%v %v", err, res.Rejected) + } + var params []string + for _, e := range res.Events { + params = append(params, e.PreviousVersion+"->"+e.Version+":"+e.Subject.Parameter) + } + got = append(got, fmt.Sprintf("%s->%s [%s]", pair[0].ID, pair[1].ID, strings.Join(params, " "))) + } + want := "0.75.0->1.0.0 [0.75.0->1.0.0:temperature 0.75.0->1.0.0:top_p 0.75.0->1.0.0:top_k] | 1.0.0->1.0.1 []" + if strings.Join(got, " | ") != want { + t.Errorf("got %s\nwant %s", strings.Join(got, " | "), want) + } +} + +// Run on the unpinned guide reads the newest major's: the same events as the +// pinned golden. +func TestRunReadsTheNewestMajor(t *testing.T) { + got := kittest.Ingest(t, Plugin{Guide: sdkguide.Anthropic}, kittest.Pages{ + "https://pypi.org/pypi/anthropic/json": "testdata/pypi-anthropic-excerpt.json", + sdkguide.AnthropicV1.URL(): "sdkguide/testdata/anthropic-MIGRATION-v1.0.0.md", + }) + kittest.CompareGolden(t, "testdata/golden/anthropic-1.0.0.json", got) +} diff --git a/adapter/adapter.go b/plugins/openapi/adapter/adapter.go similarity index 78% rename from adapter/adapter.go rename to plugins/openapi/adapter/adapter.go index 9131d6b..871d23e 100644 --- a/adapter/adapter.go +++ b/plugins/openapi/adapter/adapter.go @@ -4,7 +4,11 @@ // fetch/, and diffing/normalizing/storing are shared, not adapter code. package adapter -import "context" +import ( + "context" + + "github.com/patch-it-dev/patch-it-plugins/core/plugin" +) // Spec is one fetched spec: its raw bytes, the API version it declares, and // the pinned URL it came from. It is not yet parsed or stored — that is @@ -31,3 +35,11 @@ type Adapter interface { type Reshaper interface { SourcePointer(content []byte) func(ptr string) string } + +// Lister is an adapter that can list every version of its spec, oldest first +// (a version's Ref is what Fetch takes): what lets a spec be diffed release by +// release, not only newest against last. One without it is read at LatestTag +// only. +type Lister interface { + Versions(ctx context.Context) ([]plugin.Version, error) +} diff --git a/adapter/algolia.go b/plugins/openapi/adapter/algolia.go similarity index 88% rename from adapter/algolia.go rename to plugins/openapi/adapter/algolia.go index dcb18d0..37a70d0 100644 --- a/adapter/algolia.go +++ b/plugins/openapi/adapter/algolia.go @@ -8,7 +8,8 @@ import ( "strings" "time" - "github.com/patch-it-dev/patch-it-plugins/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/github" "gopkg.in/yaml.v3" ) @@ -42,7 +43,7 @@ var algoliaTag = regexp.MustCompile(`^released-(\d{4}-\d{2}-\d{2})(-[0-9a-f]+)?$ // have several releases (three on 2024-09-19) whose commit suffix says // nothing about their order, so those are told apart by their commit's date. func (a Algolia) LatestTag(ctx context.Context) (string, error) { - body, err := fetch.Get(ctx, algoliaRefsURL, githubHeaders()) + body, err := fetch.Get(ctx, algoliaRefsURL, github.Headers()) if err != nil { return "", fmt.Errorf("listing algolia's release tags: %w", err) } @@ -87,21 +88,11 @@ func (a Algolia) LatestTag(ctx context.Context) (string, error) { } func algoliaCommitDate(ctx context.Context, tag string) (time.Time, error) { - body, err := fetch.Get(ctx, fmt.Sprintf(algoliaCommitURL, tag), githubHeaders()) + _, date, err := github.Commit(ctx, github.Get, fmt.Sprintf(algoliaCommitURL, tag)) if err != nil { - return time.Time{}, fmt.Errorf("looking up algolia tag %s: %w", tag, err) + return time.Time{}, fmt.Errorf("algolia tag %s: no commit date: %w", tag, err) } - var c struct { - Commit struct { - Committer struct { - Date time.Time `json:"date"` - } `json:"committer"` - } `json:"commit"` - } - if err := json.Unmarshal(body, &c); err != nil || c.Commit.Committer.Date.IsZero() { - return time.Time{}, fmt.Errorf("algolia tag %s: no commit date", tag) - } - return c.Commit.Committer.Date, nil + return date, nil } // Fetch downloads the spec at tag and converts it from YAML to JSON, which is diff --git a/adapter/algolia_test.go b/plugins/openapi/adapter/algolia_test.go similarity index 100% rename from adapter/algolia_test.go rename to plugins/openapi/adapter/algolia_test.go diff --git a/adapter/appwrite.go b/plugins/openapi/adapter/appwrite.go similarity index 88% rename from adapter/appwrite.go rename to plugins/openapi/adapter/appwrite.go index 1243072..c458936 100644 --- a/adapter/appwrite.go +++ b/plugins/openapi/adapter/appwrite.go @@ -10,9 +10,9 @@ import ( "slices" "strconv" "strings" - "time" - "github.com/patch-it-dev/patch-it-plugins/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/github" ) var ( @@ -68,20 +68,14 @@ func (a Appwrite) api(path string) string { // LatestTag is the newest commit on appwrite/specs' default branch that // changed specs/. func (a Appwrite) LatestTag(ctx context.Context) (string, error) { - body, err := fetch.Get(ctx, a.api("/commits?path=specs&per_page=1"), githubHeaders()) + sha, err := github.LatestCommit(ctx, github.Get, a.api("/commits?path=specs&per_page=1")) if err != nil { return "", fmt.Errorf("looking up appwrite's latest spec commit: %w", err) } - var commits []struct { - SHA string `json:"sha"` - } - if err := json.Unmarshal(body, &commits); err != nil { - return "", fmt.Errorf("parsing github commits response: %w", err) - } - if len(commits) == 0 || !appwriteCommit.MatchString(commits[0].SHA) { + if !appwriteCommit.MatchString(sha) { return "", fmt.Errorf("github lists no commit that changed appwrite's specs") } - return commits[0].SHA, nil + return sha, nil } // Fetch downloads the spec at a commit: the newest minor's directory, or the @@ -102,23 +96,10 @@ func (a Appwrite) Fetch(ctx context.Context, tag string) (Spec, error) { if !appwriteCommit.MatchString(sha) || (named && !appwriteDir.MatchString(dir)) { return Spec{}, fmt.Errorf("%q is neither a full appwrite/specs commit sha nor .x@", tag) } - meta, err := fetch.Get(ctx, a.api("/commits/"+sha), githubHeaders()) + _, date, err := github.Commit(ctx, github.Get, a.api("/commits/"+sha)) if err != nil { return Spec{}, fmt.Errorf("looking up appwrite/specs commit %s: %w", sha, err) } - var commit struct { - Commit struct { - Committer struct { - Date time.Time `json:"date"` - } `json:"committer"` - } `json:"commit"` - } - if err := json.Unmarshal(meta, &commit); err != nil { - return Spec{}, fmt.Errorf("parsing github commit %s: %w", sha, err) - } - if commit.Commit.Committer.Date.IsZero() { - return Spec{}, fmt.Errorf("github commit %s has no date", sha) - } if dir == "" { dirs, err := a.list(ctx, "specs", sha) if err != nil { @@ -159,7 +140,7 @@ func (a Appwrite) Fetch(ctx context.Context, tag string) (Spec, error) { if err != nil { return Spec{}, fmt.Errorf("appwrite spec %s: %w", tag, err) } - return Spec{Content: content, Version: commit.Commit.Committer.Date.UTC().Format("2006-01-02"), SourceURL: url}, nil + return Spec{Content: content, Version: date.UTC().Format("2006-01-02"), SourceURL: url}, nil } // SourcePointer: see BasePathPointer. @@ -169,7 +150,7 @@ var _ Reshaper = Appwrite{} // list is the names in a directory of appwrite/specs at a commit. func (a Appwrite) list(ctx context.Context, path, sha string) ([]string, error) { - body, err := fetch.Get(ctx, a.api("/contents/"+path+"?ref="+sha), githubHeaders()) + body, err := fetch.Get(ctx, a.api("/contents/"+path+"?ref="+sha), github.Headers()) if err != nil { return nil, fmt.Errorf("listing appwrite/specs %s at %s: %w", path, sha, err) } diff --git a/adapter/appwrite_test.go b/plugins/openapi/adapter/appwrite_test.go similarity index 100% rename from adapter/appwrite_test.go rename to plugins/openapi/adapter/appwrite_test.go diff --git a/adapter/convex.go b/plugins/openapi/adapter/convex.go similarity index 76% rename from adapter/convex.go rename to plugins/openapi/adapter/convex.go index 31cc11c..fe0e020 100644 --- a/adapter/convex.go +++ b/plugins/openapi/adapter/convex.go @@ -4,9 +4,9 @@ import ( "context" "encoding/json" "fmt" - "time" - "github.com/patch-it-dev/patch-it-plugins/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/github" ) // Variables, not constants, only so tests can point them at a local server. @@ -41,20 +41,11 @@ func (Convex) API() string { return "convex" } // LatestTag is the SHA of the newest commit that changed the spec. func (Convex) LatestTag(ctx context.Context) (string, error) { - body, err := fetch.Get(ctx, convexCommitsURL, githubHeaders()) + sha, err := github.LatestCommit(ctx, github.Get, convexCommitsURL) if err != nil { return "", fmt.Errorf("looking up convex's latest spec commit: %w", err) } - var commits []struct { - SHA string `json:"sha"` - } - if err := json.Unmarshal(body, &commits); err != nil { - return "", fmt.Errorf("parsing github commits response: %w", err) - } - if len(commits) == 0 || commits[0].SHA == "" { - return "", fmt.Errorf("github lists no commit that changed the convex spec") - } - return commits[0].SHA, nil + return sha, nil } // Fetch downloads the spec at commit sha. @@ -64,27 +55,16 @@ func (Convex) LatestTag(ctx context.Context) (string, error) { // every commit, checked 2026-11-19), so the date the spec changed is the // closest thing to "the API version the change arrived in". func (Convex) Fetch(ctx context.Context, sha string) (Spec, error) { - meta, err := fetch.Get(ctx, fmt.Sprintf(convexCommitURL, sha), githubHeaders()) + fullSHA, date, err := github.Commit(ctx, github.Get, fmt.Sprintf(convexCommitURL, sha)) if err != nil { return Spec{}, fmt.Errorf("looking up convex spec commit %s: %w", sha, err) } - var commit struct { - SHA string `json:"sha"` - Commit struct { - Committer struct { - Date time.Time `json:"date"` - } `json:"committer"` - } `json:"commit"` - } - if err := json.Unmarshal(meta, &commit); err != nil { - return Spec{}, fmt.Errorf("parsing github commit %s: %w", sha, err) - } - if commit.SHA == "" || commit.Commit.Committer.Date.IsZero() { + if fullSHA == "" { return Spec{}, fmt.Errorf("github commit %s has no sha or date", sha) } // Pin to the full SHA even when given a short one: the citation must // name exactly one document. - url := fmt.Sprintf(convexSpecURLTemplate, commit.SHA) + url := fmt.Sprintf(convexSpecURLTemplate, fullSHA) body, err := fetch.Get(ctx, url, nil) // raw.githubusercontent.com needs no auth if err != nil { return Spec{}, fmt.Errorf("fetching convex spec %s: %w", sha, err) @@ -101,7 +81,7 @@ func (Convex) Fetch(ctx context.Context, sha string) (Spec, error) { if err != nil { return Spec{}, fmt.Errorf("convex spec %s: %w", sha, err) } - return Spec{Content: content, Version: commit.Commit.Committer.Date.UTC().Format("2006-01-02"), SourceURL: url}, nil + return Spec{Content: content, Version: date.UTC().Format("2006-01-02"), SourceURL: url}, nil } // SourcePointer: see BasePathPointer. diff --git a/adapter/convex_test.go b/plugins/openapi/adapter/convex_test.go similarity index 100% rename from adapter/convex_test.go rename to plugins/openapi/adapter/convex_test.go diff --git a/adapter/datadog.go b/plugins/openapi/adapter/datadog.go similarity index 86% rename from adapter/datadog.go rename to plugins/openapi/adapter/datadog.go index 573f429..8a5f9d3 100644 --- a/adapter/datadog.go +++ b/plugins/openapi/adapter/datadog.go @@ -5,7 +5,8 @@ import ( "encoding/json" "fmt" - "github.com/patch-it-dev/patch-it-plugins/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/github" "gopkg.in/yaml.v3" ) @@ -33,20 +34,11 @@ func (Datadog) API() string { return "datadog" } // repo, so the client that's generated from it is the only thing that's // tagged. func (Datadog) LatestTag(ctx context.Context) (string, error) { - body, err := fetch.Get(ctx, datadogReleasesURL, githubHeaders()) + tag, err := github.LatestRelease(ctx, github.Get, datadogReleasesURL) if err != nil { return "", fmt.Errorf("looking up datadog's latest release: %w", err) } - var release struct { - TagName string `json:"tag_name"` - } - if err := json.Unmarshal(body, &release); err != nil { - return "", fmt.Errorf("parsing github release response: %w", err) - } - if release.TagName == "" { - return "", fmt.Errorf("github release response had no tag_name") - } - return release.TagName, nil + return tag, nil } // Fetch downloads the spec pinned to tag and converts it from YAML to JSON, diff --git a/adapter/datadog_test.go b/plugins/openapi/adapter/datadog_test.go similarity index 100% rename from adapter/datadog_test.go rename to plugins/openapi/adapter/datadog_test.go diff --git a/adapter/elasticsearch.go b/plugins/openapi/adapter/elasticsearch.go similarity index 89% rename from adapter/elasticsearch.go rename to plugins/openapi/adapter/elasticsearch.go index 9a7c36f..e14b571 100644 --- a/adapter/elasticsearch.go +++ b/plugins/openapi/adapter/elasticsearch.go @@ -8,7 +8,8 @@ import ( "strconv" "strings" - "github.com/patch-it-dev/patch-it-plugins/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/github" ) // Variables, not constants, only so tests can point them at a local server. @@ -48,7 +49,7 @@ var ( // 2026-11-19, when the newest release was 9.5.6), whose spec can still // change before it ships. func (Elasticsearch) LatestTag(ctx context.Context) (string, error) { - body, err := fetch.Get(ctx, esReleasesURL, githubHeaders()) + body, err := fetch.Get(ctx, esReleasesURL, github.Headers()) if err != nil { return "", fmt.Errorf("listing elasticsearch releases: %w", err) } @@ -75,7 +76,7 @@ func (Elasticsearch) LatestTag(ctx context.Context) (string, error) { return "", fmt.Errorf("github lists no elasticsearch release") } - body, err = fetch.Get(ctx, esBranchesURL, githubHeaders()) + body, err = fetch.Get(ctx, esBranchesURL, github.Headers()) if err != nil { return "", fmt.Errorf("listing elasticsearch-specification branches: %w", err) } @@ -109,17 +110,11 @@ func (Elasticsearch) Fetch(ctx context.Context, branch string) (Spec, error) { if !esBranch.MatchString(branch) { return Spec{}, fmt.Errorf("%q isn't an elasticsearch-specification minor branch", branch) } - meta, err := fetch.Get(ctx, fmt.Sprintf(esCommitURL, branch), githubHeaders()) + sha, err := github.Head(ctx, github.Get, fmt.Sprintf(esCommitURL, branch)) if err != nil { return Spec{}, fmt.Errorf("looking up the head of elasticsearch-specification %s: %w", branch, err) } - var commit struct { - SHA string `json:"sha"` - } - if err := json.Unmarshal(meta, &commit); err != nil || commit.SHA == "" { - return Spec{}, fmt.Errorf("elasticsearch-specification %s: no head commit", branch) - } - url := fmt.Sprintf(esSpecURLTemplate, commit.SHA) + url := fmt.Sprintf(esSpecURLTemplate, sha) body, err := fetch.Get(ctx, url, nil) // raw.githubusercontent.com needs no auth if err != nil { return Spec{}, fmt.Errorf("fetching the elasticsearch %s spec: %w", branch, err) diff --git a/adapter/elasticsearch_test.go b/plugins/openapi/adapter/elasticsearch_test.go similarity index 100% rename from adapter/elasticsearch_test.go rename to plugins/openapi/adapter/elasticsearch_test.go diff --git a/adapter/flagsmith.go b/plugins/openapi/adapter/flagsmith.go similarity index 87% rename from adapter/flagsmith.go rename to plugins/openapi/adapter/flagsmith.go index b491d8e..1c5c182 100644 --- a/adapter/flagsmith.go +++ b/plugins/openapi/adapter/flagsmith.go @@ -6,7 +6,8 @@ import ( "fmt" "strings" - "github.com/patch-it-dev/patch-it-plugins/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/github" "gopkg.in/yaml.v3" ) @@ -38,20 +39,11 @@ func (Flagsmith) API() string { return "flagsmith" } // Flagsmith tags a release on (almost) every merge to main via release-please, // so releases/latest is always close to HEAD. func (Flagsmith) LatestTag(ctx context.Context) (string, error) { - body, err := fetch.Get(ctx, flagsmithReleasesURL, githubHeaders()) + tag, err := github.LatestRelease(ctx, github.Get, flagsmithReleasesURL) if err != nil { return "", fmt.Errorf("looking up flagsmith's latest release: %w", err) } - var release struct { - TagName string `json:"tag_name"` - } - if err := json.Unmarshal(body, &release); err != nil { - return "", fmt.Errorf("parsing github release response: %w", err) - } - if release.TagName == "" { - return "", fmt.Errorf("github release response had no tag_name") - } - return release.TagName, nil + return tag, nil } // Fetch downloads the spec pinned to tag and converts it from YAML to JSON, diff --git a/adapter/github.go b/plugins/openapi/adapter/github.go similarity index 94% rename from adapter/github.go rename to plugins/openapi/adapter/github.go index f0fc108..f8c29f6 100644 --- a/adapter/github.go +++ b/plugins/openapi/adapter/github.go @@ -6,7 +6,8 @@ import ( "fmt" "sort" - "github.com/patch-it-dev/patch-it-plugins/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/github" ) // Variables, not constants, only so tests can point them at a local server. @@ -38,7 +39,7 @@ func (GitHub) API() string { return "github" } // GitHub returns the list newest-first, but that isn't documented, so this // doesn't rely on it. func (GitHub) LatestTag(ctx context.Context) (string, error) { - body, err := fetch.Get(ctx, githubVersionsURL, githubHeaders()) + body, err := fetch.Get(ctx, githubVersionsURL, github.Headers()) if err != nil { return "", fmt.Errorf("looking up github's supported api versions: %w", err) } diff --git a/adapter/github_test.go b/plugins/openapi/adapter/github_test.go similarity index 100% rename from adapter/github_test.go rename to plugins/openapi/adapter/github_test.go diff --git a/adapter/gitlab.go b/plugins/openapi/adapter/gitlab.go similarity index 99% rename from adapter/gitlab.go rename to plugins/openapi/adapter/gitlab.go index 54ec41f..b3ebb94 100644 --- a/adapter/gitlab.go +++ b/plugins/openapi/adapter/gitlab.go @@ -10,7 +10,7 @@ import ( "strconv" "strings" - "github.com/patch-it-dev/patch-it-plugins/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" "gopkg.in/yaml.v3" ) diff --git a/adapter/gitlab_test.go b/plugins/openapi/adapter/gitlab_test.go similarity index 100% rename from adapter/gitlab_test.go rename to plugins/openapi/adapter/gitlab_test.go diff --git a/adapter/neon.go b/plugins/openapi/adapter/neon.go similarity index 74% rename from adapter/neon.go rename to plugins/openapi/adapter/neon.go index 920b3a2..b767b2b 100644 --- a/adapter/neon.go +++ b/plugins/openapi/adapter/neon.go @@ -4,9 +4,9 @@ import ( "context" "encoding/json" "fmt" - "time" - "github.com/patch-it-dev/patch-it-plugins/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/github" ) // Variables, not constants, only so tests can point them at a local server. @@ -36,20 +36,11 @@ func (Neon) API() string { return "neon" } // LatestTag is the SHA of the newest commit that changed the spec. func (Neon) LatestTag(ctx context.Context) (string, error) { - body, err := fetch.Get(ctx, neonCommitsURL, githubHeaders()) + sha, err := github.LatestCommit(ctx, github.Get, neonCommitsURL) if err != nil { return "", fmt.Errorf("looking up neon's latest spec commit: %w", err) } - var commits []struct { - SHA string `json:"sha"` - } - if err := json.Unmarshal(body, &commits); err != nil { - return "", fmt.Errorf("parsing github commits response: %w", err) - } - if len(commits) == 0 || commits[0].SHA == "" { - return "", fmt.Errorf("github lists no commit that changed the neon spec") - } - return commits[0].SHA, nil + return sha, nil } // Fetch downloads the spec at commit sha and moves the server URL's path into @@ -60,27 +51,16 @@ func (Neon) LatestTag(ctx context.Context) (string, error) { // 2026-11-19), so the date the spec changed is the closest thing to "the API // version the change arrived in". func (Neon) Fetch(ctx context.Context, sha string) (Spec, error) { - meta, err := fetch.Get(ctx, fmt.Sprintf(neonCommitURL, sha), githubHeaders()) + fullSHA, date, err := github.Commit(ctx, github.Get, fmt.Sprintf(neonCommitURL, sha)) if err != nil { return Spec{}, fmt.Errorf("looking up neon spec commit %s: %w", sha, err) } - var commit struct { - SHA string `json:"sha"` - Commit struct { - Committer struct { - Date time.Time `json:"date"` - } `json:"committer"` - } `json:"commit"` - } - if err := json.Unmarshal(meta, &commit); err != nil { - return Spec{}, fmt.Errorf("parsing github commit %s: %w", sha, err) - } - if commit.SHA == "" || commit.Commit.Committer.Date.IsZero() { + if fullSHA == "" { return Spec{}, fmt.Errorf("github commit %s has no sha or date", sha) } // Pin to the full SHA even when given a short one: the citation must // name exactly one document. - url := fmt.Sprintf(neonSpecURLTemplate, commit.SHA) + url := fmt.Sprintf(neonSpecURLTemplate, fullSHA) body, err := fetch.Get(ctx, url, nil) // raw.githubusercontent.com needs no auth if err != nil { return Spec{}, fmt.Errorf("fetching neon spec %s: %w", sha, err) @@ -97,7 +77,7 @@ func (Neon) Fetch(ctx context.Context, sha string) (Spec, error) { if err != nil { return Spec{}, fmt.Errorf("neon spec %s: %w", sha, err) } - return Spec{Content: content, Version: commit.Commit.Committer.Date.UTC().Format("2006-01-02"), SourceURL: url}, nil + return Spec{Content: content, Version: date.UTC().Format("2006-01-02"), SourceURL: url}, nil } // SourcePointer: see BasePathPointer. diff --git a/adapter/neon_test.go b/plugins/openapi/adapter/neon_test.go similarity index 100% rename from adapter/neon_test.go rename to plugins/openapi/adapter/neon_test.go diff --git a/adapter/planetscale.go b/plugins/openapi/adapter/planetscale.go similarity index 77% rename from adapter/planetscale.go rename to plugins/openapi/adapter/planetscale.go index cb245ee..5c83d61 100644 --- a/adapter/planetscale.go +++ b/plugins/openapi/adapter/planetscale.go @@ -5,9 +5,9 @@ import ( "context" "encoding/json" "fmt" - "time" - "github.com/patch-it-dev/patch-it-plugins/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/github" "gopkg.in/yaml.v3" ) @@ -42,20 +42,11 @@ func (p PlanetScale) api(path string) string { // LatestTag is the SHA of the newest commit that changed the spec. func (p PlanetScale) LatestTag(ctx context.Context) (string, error) { - body, err := fetch.Get(ctx, p.api("/commits?path="+planetscaleSpecPath+"&per_page=1"), githubHeaders()) + sha, err := github.LatestCommit(ctx, github.Get, p.api("/commits?path="+planetscaleSpecPath+"&per_page=1")) if err != nil { return "", fmt.Errorf("looking up planetscale's latest spec commit: %w", err) } - var commits []struct { - SHA string `json:"sha"` - } - if err := json.Unmarshal(body, &commits); err != nil { - return "", fmt.Errorf("parsing github commits response: %w", err) - } - if len(commits) == 0 || commits[0].SHA == "" { - return "", fmt.Errorf("github lists no commit that changed the planetscale spec") - } - return commits[0].SHA, nil + return sha, nil } // Fetch downloads the spec at commit sha and converts it from YAML to JSON, @@ -73,27 +64,16 @@ func (p PlanetScale) LatestTag(ctx context.Context) (string, error) { // error. A file served 200 that has no paths (emptied, or a stub page like // "404: Not Found", which YAML reads as a one-key mapping) is refused here. func (p PlanetScale) Fetch(ctx context.Context, sha string) (Spec, error) { - meta, err := fetch.Get(ctx, p.api("/commits/"+sha), githubHeaders()) + fullSHA, date, err := github.Commit(ctx, github.Get, p.api("/commits/"+sha)) if err != nil { return Spec{}, fmt.Errorf("looking up planetscale spec commit %s: %w", sha, err) } - var commit struct { - SHA string `json:"sha"` - Commit struct { - Committer struct { - Date time.Time `json:"date"` - } `json:"committer"` - } `json:"commit"` - } - if err := json.Unmarshal(meta, &commit); err != nil { - return Spec{}, fmt.Errorf("parsing github commit %s: %w", sha, err) - } - if commit.SHA == "" || commit.Commit.Committer.Date.IsZero() { + if fullSHA == "" { return Spec{}, fmt.Errorf("github commit %s has no sha or date", sha) } // Pin to the full SHA even when given a short one: the citation must // name exactly one document. - url := cmp.Or(p.RawBase, "https://raw.githubusercontent.com") + "/" + planetscaleRepo + "/" + commit.SHA + "/" + planetscaleSpecPath + url := cmp.Or(p.RawBase, "https://raw.githubusercontent.com") + "/" + planetscaleRepo + "/" + fullSHA + "/" + planetscaleSpecPath body, err := fetch.Get(ctx, url, nil) // raw.githubusercontent.com needs no auth if err != nil { return Spec{}, fmt.Errorf("fetching planetscale spec %s: %w", sha, err) @@ -117,7 +97,7 @@ func (p PlanetScale) Fetch(ctx context.Context, sha string) (Spec, error) { if content, err = ServerBasePath(content); err != nil { return Spec{}, fmt.Errorf("planetscale spec %s: %w", sha, err) } - return Spec{Content: content, Version: commit.Commit.Committer.Date.UTC().Format("2006-01-02"), SourceURL: url}, nil + return Spec{Content: content, Version: date.UTC().Format("2006-01-02"), SourceURL: url}, nil } // SourcePointer: see BasePathPointer. diff --git a/adapter/planetscale_test.go b/plugins/openapi/adapter/planetscale_test.go similarity index 100% rename from adapter/planetscale_test.go rename to plugins/openapi/adapter/planetscale_test.go diff --git a/adapter/sentry.go b/plugins/openapi/adapter/sentry.go similarity index 97% rename from adapter/sentry.go rename to plugins/openapi/adapter/sentry.go index 1c4c97f..4b8bda6 100644 --- a/adapter/sentry.go +++ b/plugins/openapi/adapter/sentry.go @@ -12,7 +12,8 @@ import ( "strings" "time" - "github.com/patch-it-dev/patch-it-plugins/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/github" ) // sentryAsset is the spec's file name in both places it's published: @@ -64,7 +65,7 @@ func (s Sentry) api(path string) string { // "scraps@0.5.0"). GitHub lists releases in an order that isn't version // order, so the newest is computed from the version. func (s Sentry) LatestTag(ctx context.Context) (string, error) { - body, err := fetch.Get(ctx, s.api("/repos/getsentry/sentry/releases?per_page=100"), githubHeaders()) + body, err := fetch.Get(ctx, s.api("/repos/getsentry/sentry/releases?per_page=100"), github.Headers()) if err != nil { return "", fmt.Errorf("listing sentry releases: %w", err) } @@ -148,7 +149,7 @@ func (s Sentry) Fetch(ctx context.Context, tag string) (Spec, error) { } url, date = raw, r.PublishedAt case sentrySchemaCommit.MatchString(tag): - meta, err := fetch.Get(ctx, s.api("/repos/getsentry/sentry-api-schema/commits/"+tag), githubHeaders()) + meta, err := fetch.Get(ctx, s.api("/repos/getsentry/sentry-api-schema/commits/"+tag), github.Headers()) if err != nil { return Spec{}, fmt.Errorf("looking up sentry-api-schema commit %s: %w", tag, err) } @@ -181,7 +182,7 @@ func (s Sentry) Fetch(ctx context.Context, tag string) (Spec, error) { } func sentryReleaseMeta(ctx context.Context, url, tag string) (sentryReleaseInfo, error) { - meta, err := fetch.Get(ctx, url, githubHeaders()) + meta, err := fetch.Get(ctx, url, github.Headers()) if err != nil { return sentryReleaseInfo{}, fmt.Errorf("looking up sentry release %s: %w", tag, err) } diff --git a/adapter/sentry_test.go b/plugins/openapi/adapter/sentry_test.go similarity index 100% rename from adapter/sentry_test.go rename to plugins/openapi/adapter/sentry_test.go diff --git a/adapter/stripe.go b/plugins/openapi/adapter/stripe.go similarity index 76% rename from adapter/stripe.go rename to plugins/openapi/adapter/stripe.go index 869053e..171f736 100644 --- a/adapter/stripe.go +++ b/plugins/openapi/adapter/stripe.go @@ -6,9 +6,9 @@ import ( "errors" "fmt" "net/http" - "os" - "github.com/patch-it-dev/patch-it-plugins/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/github" ) // Variables, not constants, only so tests can point them at a local server. @@ -33,21 +33,11 @@ func (Stripe) API() string { return "stripe" } // LatestTag asks GitHub for Stripe's newest openapi release tag, e.g. "v2468". func (Stripe) LatestTag(ctx context.Context) (string, error) { - body, err := fetch.Get(ctx, stripeReleasesURL, githubHeaders()) + tag, err := github.LatestRelease(ctx, github.Get, stripeReleasesURL) if err != nil { return "", fmt.Errorf("looking up stripe's latest release: %w", err) } - - var release struct { - TagName string `json:"tag_name"` - } - if err := json.Unmarshal(body, &release); err != nil { - return "", fmt.Errorf("parsing github release response: %w", err) - } - if release.TagName == "" { - return "", fmt.Errorf("github release response had no tag_name") - } - return release.TagName, nil + return tag, nil } // Fetch downloads the spec pinned to tag and reads its declared API version @@ -84,13 +74,3 @@ func (Stripe) Fetch(ctx context.Context, tag string) (Spec, error) { SourceURL: url, }, nil } - -// githubHeaders adds GITHUB_TOKEN as a bearer token when set, which raises -// the GitHub API's rate limit for looking up release tags. Never required. -func githubHeaders() map[string]string { - token := os.Getenv("GITHUB_TOKEN") - if token == "" { - return nil - } - return map[string]string{"Authorization": "Bearer " + token} -} diff --git a/adapter/stripe_test.go b/plugins/openapi/adapter/stripe_test.go similarity index 100% rename from adapter/stripe_test.go rename to plugins/openapi/adapter/stripe_test.go diff --git a/adapter/supabase.go b/plugins/openapi/adapter/supabase.go similarity index 69% rename from adapter/supabase.go rename to plugins/openapi/adapter/supabase.go index d6cb188..bd4ab86 100644 --- a/adapter/supabase.go +++ b/plugins/openapi/adapter/supabase.go @@ -4,9 +4,9 @@ import ( "context" "encoding/json" "fmt" - "time" - "github.com/patch-it-dev/patch-it-plugins/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/github" ) // Variables, not constants, only so tests can point them at a local server. @@ -35,20 +35,11 @@ func (Supabase) API() string { return "supabase" } // LatestTag is the SHA of the newest commit that changed the spec. func (Supabase) LatestTag(ctx context.Context) (string, error) { - body, err := fetch.Get(ctx, supabaseCommitsURL, githubHeaders()) + sha, err := github.LatestCommit(ctx, github.Get, supabaseCommitsURL) if err != nil { return "", fmt.Errorf("looking up supabase's latest spec commit: %w", err) } - var commits []struct { - SHA string `json:"sha"` - } - if err := json.Unmarshal(body, &commits); err != nil { - return "", fmt.Errorf("parsing github commits response: %w", err) - } - if len(commits) == 0 || commits[0].SHA == "" { - return "", fmt.Errorf("github lists no commit that changed the supabase spec") - } - return commits[0].SHA, nil + return sha, nil } // Fetch downloads the spec at commit sha. @@ -58,27 +49,16 @@ func (Supabase) LatestTag(ctx context.Context) (string, error) { // 2026-11-19), so the date the spec changed is the closest thing to "the API // version the change arrived in". func (Supabase) Fetch(ctx context.Context, sha string) (Spec, error) { - meta, err := fetch.Get(ctx, fmt.Sprintf(supabaseCommitURL, sha), githubHeaders()) + fullSHA, date, err := github.Commit(ctx, github.Get, fmt.Sprintf(supabaseCommitURL, sha)) if err != nil { return Spec{}, fmt.Errorf("looking up supabase spec commit %s: %w", sha, err) } - var commit struct { - SHA string `json:"sha"` - Commit struct { - Committer struct { - Date time.Time `json:"date"` - } `json:"committer"` - } `json:"commit"` - } - if err := json.Unmarshal(meta, &commit); err != nil { - return Spec{}, fmt.Errorf("parsing github commit %s: %w", sha, err) - } - if commit.SHA == "" || commit.Commit.Committer.Date.IsZero() { + if fullSHA == "" { return Spec{}, fmt.Errorf("github commit %s has no sha or date", sha) } // Pin to the full SHA even when given a short one: the citation must // name exactly one document. - url := fmt.Sprintf(supabaseSpecURLTemplate, commit.SHA) + url := fmt.Sprintf(supabaseSpecURLTemplate, fullSHA) body, err := fetch.Get(ctx, url, nil) // raw.githubusercontent.com needs no auth if err != nil { return Spec{}, fmt.Errorf("fetching supabase spec %s: %w", sha, err) @@ -86,5 +66,5 @@ func (Supabase) Fetch(ctx context.Context, sha string) (Spec, error) { if !json.Valid(body) { return Spec{}, fmt.Errorf("supabase spec %s is not valid json", sha) } - return Spec{Content: body, Version: commit.Commit.Committer.Date.UTC().Format("2006-01-02"), SourceURL: url}, nil + return Spec{Content: body, Version: date.UTC().Format("2006-01-02"), SourceURL: url}, nil } diff --git a/adapter/supabase_test.go b/plugins/openapi/adapter/supabase_test.go similarity index 100% rename from adapter/supabase_test.go rename to plugins/openapi/adapter/supabase_test.go diff --git a/adapter/vercel.go b/plugins/openapi/adapter/vercel.go similarity index 83% rename from adapter/vercel.go rename to plugins/openapi/adapter/vercel.go index 97dfbaa..ae0d5e1 100644 --- a/adapter/vercel.go +++ b/plugins/openapi/adapter/vercel.go @@ -6,7 +6,8 @@ import ( "fmt" "strings" - "github.com/patch-it-dev/patch-it-plugins/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/fetch" + "github.com/patch-it-dev/patch-it-plugins/core/github" ) // Variables, not constants, only so tests can point them at a local server. @@ -32,20 +33,11 @@ func (Vercel) API() string { return "vercel" } // The canary tags (v14.0.0-canary.9) aren't releases, so releases/latest never // returns them (checked 2026-11-18). func (Vercel) LatestTag(ctx context.Context) (string, error) { - body, err := fetch.Get(ctx, vercelReleasesURL, githubHeaders()) + tag, err := github.LatestRelease(ctx, github.Get, vercelReleasesURL) if err != nil { return "", fmt.Errorf("looking up vercel's latest release: %w", err) } - var release struct { - TagName string `json:"tag_name"` - } - if err := json.Unmarshal(body, &release); err != nil { - return "", fmt.Errorf("parsing github release response: %w", err) - } - if release.TagName == "" { - return "", fmt.Errorf("github release response had no tag_name") - } - return release.TagName, nil + return tag, nil } // Fetch downloads the spec pinned to tag. It's JSON already, so it's passed diff --git a/adapter/vercel_test.go b/plugins/openapi/adapter/vercel_test.go similarity index 100% rename from adapter/vercel_test.go rename to plugins/openapi/adapter/vercel_test.go diff --git a/plugins/openapi/adapter/versions.go b/plugins/openapi/adapter/versions.go new file mode 100644 index 0000000..9e6238e --- /dev/null +++ b/plugins/openapi/adapter/versions.go @@ -0,0 +1,75 @@ +package adapter + +import ( + "context" + "strings" + + "github.com/patch-it-dev/patch-it-plugins/core/github" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/versions" +) + +// Every version of each spec that can be listed, oldest first (Lister): the +// repo's releases for the adapters whose LatestTag is releases/latest, the +// commits that changed the spec for those that follow a file. The others +// (GitHub's dated versions, GitLab, Elasticsearch, Sentry, Algolia, Appwrite) +// are read at LatestTag only, for now. + +func releases(ctx context.Context, repo string, keep func(string) bool) ([]plugin.Version, error) { + return versions.GitHubReleases(ctx, github.Get, repo, keep) +} + +func commits(ctx context.Context, repo, path string) ([]plugin.Version, error) { + return versions.GitHubCommits(ctx, github.Get, repo, path) +} + +// Versions are stripe/openapi's releases. +func (Stripe) Versions(ctx context.Context) ([]plugin.Version, error) { + return releases(ctx, "stripe/openapi", nil) +} + +// Versions are datadog-api-client-python's releases. +func (Datadog) Versions(ctx context.Context) ([]plugin.Version, error) { + return releases(ctx, "DataDog/datadog-api-client-python", nil) +} + +// Versions are vercel/sdk's releases, canaries left out (as LatestTag does). +func (Vercel) Versions(ctx context.Context) ([]plugin.Version, error) { + return releases(ctx, "vercel/sdk", func(tag string) bool { return !strings.Contains(tag, "canary") }) +} + +// Versions are Flagsmith's releases. +func (Flagsmith) Versions(ctx context.Context) ([]plugin.Version, error) { + return releases(ctx, "Flagsmith/flagsmith", nil) +} + +// Versions are the commits that changed the Management API spec. +func (Supabase) Versions(ctx context.Context) ([]plugin.Version, error) { + return commits(ctx, "supabase/supabase", "apps/docs/spec/api_v1_openapi.json") +} + +// Versions are the commits that changed Neon's spec. +func (Neon) Versions(ctx context.Context) ([]plugin.Version, error) { + return commits(ctx, "neondatabase/neon-pkgs", "packages/sdk/spec/neon-openapi.json") +} + +// Versions are the commits that changed Convex's deployment API spec. +func (Convex) Versions(ctx context.Context) ([]plugin.Version, error) { + return commits(ctx, "get-convex/convex-backend", "npm-packages/@convex-dev/platform/deployment-openapi.json") +} + +// Versions are the commits that changed PlanetScale's spec. +func (PlanetScale) Versions(ctx context.Context) ([]plugin.Version, error) { + return commits(ctx, planetscaleRepo, planetscaleSpecPath) +} + +var ( + _ Lister = Stripe{} + _ Lister = Datadog{} + _ Lister = Vercel{} + _ Lister = Flagsmith{} + _ Lister = Supabase{} + _ Lister = Neon{} + _ Lister = Convex{} + _ Lister = PlanetScale{} +) diff --git a/plugins/openapi/algolia.go b/plugins/openapi/algolia.go index 2e1503f..9362faa 100644 --- a/plugins/openapi/algolia.go +++ b/plugins/openapi/algolia.go @@ -1,6 +1,6 @@ package openapi -import "github.com/patch-it-dev/patch-it-plugins/diff" +import "github.com/patch-it-dev/patch-it-plugins/core/diff" // algoliaDiff reads Algolia's specs like Supabase's: deprecations by the // `deprecated` flag, quoting the whole operation, and path templates matched. diff --git a/plugins/openapi/algolia_test.go b/plugins/openapi/algolia_test.go index 7635650..6689e85 100644 --- a/plugins/openapi/algolia_test.go +++ b/plugins/openapi/algolia_test.go @@ -8,10 +8,10 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/adapter" - "github.com/patch-it-dev/patch-it-plugins/diff" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/diff" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter" "gopkg.in/yaml.v3" ) diff --git a/plugins/openapi/appwrite.go b/plugins/openapi/appwrite.go index 0b81d0d..7055d9f 100644 --- a/plugins/openapi/appwrite.go +++ b/plugins/openapi/appwrite.go @@ -1,6 +1,6 @@ package openapi -import "github.com/patch-it-dev/patch-it-plugins/diff" +import "github.com/patch-it-dev/patch-it-plugins/core/diff" // appwriteDiff is how Appwrite's spec says what it says: // diff --git a/plugins/openapi/appwrite_test.go b/plugins/openapi/appwrite_test.go index a4bf6f3..04f84cc 100644 --- a/plugins/openapi/appwrite_test.go +++ b/plugins/openapi/appwrite_test.go @@ -9,10 +9,10 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/adapter" - "github.com/patch-it-dev/patch-it-plugins/diff" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/diff" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter" ) // The appwrite/specs commits the tests read (checked 2026-11-19), and their diff --git a/plugins/openapi/convex.go b/plugins/openapi/convex.go index e57101e..3a52769 100644 --- a/plugins/openapi/convex.go +++ b/plugins/openapi/convex.go @@ -1,6 +1,6 @@ package openapi -import "github.com/patch-it-dev/patch-it-plugins/diff" +import "github.com/patch-it-dev/patch-it-plugins/core/diff" // convexDiff is how the Convex Deployment API spec says what it says: // diff --git a/plugins/openapi/convex_test.go b/plugins/openapi/convex_test.go index 4d81e48..21c7613 100644 --- a/plugins/openapi/convex_test.go +++ b/plugins/openapi/convex_test.go @@ -7,9 +7,9 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/adapter" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter" ) // convexCommits are the spec commits saved under testdata/ (trimmed to the diff --git a/plugins/openapi/datadog_test.go b/plugins/openapi/datadog_test.go index fcc1ea6..908ba5c 100644 --- a/plugins/openapi/datadog_test.go +++ b/plugins/openapi/datadog_test.go @@ -7,7 +7,7 @@ import ( "os" "testing" - "github.com/patch-it-dev/patch-it-plugins/adapter" + "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter" "gopkg.in/yaml.v3" ) diff --git a/plugins/openapi/elasticsearch.go b/plugins/openapi/elasticsearch.go index 1f43b2b..c8c9090 100644 --- a/plugins/openapi/elasticsearch.go +++ b/plugins/openapi/elasticsearch.go @@ -1,6 +1,6 @@ package openapi -import "github.com/patch-it-dev/patch-it-plugins/diff" +import "github.com/patch-it-dev/patch-it-plugins/core/diff" // elasticsearchDiff reads Elastic's spec like Supabase's: deprecations by the // `deprecated` flag, quoting the whole operation, and path templates matched. diff --git a/plugins/openapi/elasticsearch_test.go b/plugins/openapi/elasticsearch_test.go index 0a57679..402ad23 100644 --- a/plugins/openapi/elasticsearch_test.go +++ b/plugins/openapi/elasticsearch_test.go @@ -7,9 +7,9 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/adapter" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter" ) // esHeads are the elasticsearch-specification branch heads saved under diff --git a/plugins/openapi/flagsmith_test.go b/plugins/openapi/flagsmith_test.go index 3cce606..eaea1df 100644 --- a/plugins/openapi/flagsmith_test.go +++ b/plugins/openapi/flagsmith_test.go @@ -8,8 +8,8 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/adapter" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter" "gopkg.in/yaml.v3" ) diff --git a/plugins/openapi/github_test.go b/plugins/openapi/github_test.go index 6ff7d86..4496f7a 100644 --- a/plugins/openapi/github_test.go +++ b/plugins/openapi/github_test.go @@ -6,7 +6,7 @@ import ( "os" "testing" - "github.com/patch-it-dev/patch-it-plugins/adapter" + "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter" ) // fakeGitHubAdapter serves the real, trimmed spec excerpts saved under diff --git a/plugins/openapi/gitlab_test.go b/plugins/openapi/gitlab_test.go index a79dca4..a61851f 100644 --- a/plugins/openapi/gitlab_test.go +++ b/plugins/openapi/gitlab_test.go @@ -10,7 +10,7 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/adapter" + "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter" "gopkg.in/yaml.v3" ) diff --git a/plugins/openapi/neon.go b/plugins/openapi/neon.go index 41a70ee..3facc5b 100644 --- a/plugins/openapi/neon.go +++ b/plugins/openapi/neon.go @@ -1,6 +1,6 @@ package openapi -import "github.com/patch-it-dev/patch-it-plugins/diff" +import "github.com/patch-it-dev/patch-it-plugins/core/diff" // neonDiff is how the Neon API spec says what it says: // diff --git a/plugins/openapi/neon_test.go b/plugins/openapi/neon_test.go index 8a5c483..ac726f6 100644 --- a/plugins/openapi/neon_test.go +++ b/plugins/openapi/neon_test.go @@ -10,9 +10,9 @@ import ( "gopkg.in/yaml.v3" - "github.com/patch-it-dev/patch-it-plugins/adapter" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter" ) // neonCommits are the spec commits saved under testdata/ (trimmed to the diff --git a/plugins/openapi/openapi.go b/plugins/openapi/openapi.go index 3d943d6..61eeda4 100644 --- a/plugins/openapi/openapi.go +++ b/plugins/openapi/openapi.go @@ -11,64 +11,176 @@ import ( "fmt" "log" - "github.com/patch-it-dev/patch-it-plugins/adapter" - "github.com/patch-it-dev/patch-it-plugins/diff" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/hint" - "github.com/patch-it-dev/patch-it-plugins/migrate" - "github.com/patch-it-dev/patch-it-plugins/normalize" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/diff" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/hint" + "github.com/patch-it-dev/patch-it-plugins/core/migrate" + "github.com/patch-it-dev/patch-it-plugins/core/normalize" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" + "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter" ) func init() { - plugin.Register(Plugin{Adapter: adapter.Stripe{}, Hints: hint.Stripe{}}) + plugin.RegisterProvider(plugin.Provider{ + ID: "stripe", Name: "Stripe", Category: plugin.CategoryPayments, Homepage: "https://stripe.com", + Sources: []plugin.Source{{ + ID: "api", Layer: plugin.LayerAPI, Page: "providers/stripe.md", + Watches: "REST API changes between OpenAPI spec releases (removed and renamed endpoints, parameters and fields), with migration hints from Stripe's API changelog", + URLs: []string{"https://github.com/stripe/openapi"}, + Plugin: Plugin{Adapter: adapter.Stripe{}, Hints: hint.Stripe{}}, + }}, + }) // No Hints yet: Datadog's changelog isn't Stripe's per-parameter table // shape, so there's no hint.Source for it to read. - plugin.Register(Plugin{Adapter: adapter.Datadog{}}) + plugin.RegisterProvider(plugin.Provider{ + ID: "datadog", Name: "Datadog", Category: plugin.CategoryMonitoring, Homepage: "https://www.datadoghq.com", + Sources: []plugin.Source{{ + ID: "api", Layer: plugin.LayerAPI, Page: "providers/datadog.md", + Watches: "API v2 changes between OpenAPI spec releases", + URLs: []string{"https://github.com/DataDog/datadog-api-client-python"}, + Plugin: Plugin{Adapter: adapter.Datadog{}}, + }}, + }) // No Hints: GitLab's deprecations are prose per feature, not a changelog // keyed by API version, so there's nothing for hint.Source to look up. - plugin.Register(Plugin{Adapter: adapter.GitLab{}, Keep: requestSide}) + plugin.RegisterProvider(plugin.Provider{ + ID: "gitlab", Name: "GitLab", Category: plugin.CategoryDevTools, Homepage: "https://about.gitlab.com", + Sources: []plugin.Source{{ + ID: "rest-api", Layer: plugin.LayerAPI, Page: "providers/gitlab.md", + Watches: "REST API changes between releases, request side only (what callers send)", + URLs: []string{"https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/api/openapi"}, + Plugin: Plugin{Adapter: adapter.GitLab{}, Keep: requestSide}, + }}, + }) // No Hints yet: GitHub's per-version CHANGELOG.md is prose with no table - // at all, so there's nothing for hint.Source to key on. - plugin.Register(Plugin{Adapter: adapter.GitHub{}}) + // at all, so there's nothing for hint.Source to key on. (GitHub's Actions + // source is registered by plugins/feed.) + plugin.RegisterProvider(plugin.Provider{ + ID: "github", Name: "GitHub", Category: plugin.CategoryDevTools, Homepage: "https://github.com", + Sources: []plugin.Source{{ + ID: "rest-api", Layer: plugin.LayerAPI, Page: "providers/github.md", + Watches: "REST API changes between dated API versions", + URLs: []string{"https://github.com/github/rest-api-description"}, + Plugin: Plugin{Adapter: adapter.GitHub{}}, + }}, + }) // No Hints: Vercel's changelog is product news, not keyed by API change. // Only operation deprecations and removals are kept: see vercelKeep. - plugin.Register(Plugin{Adapter: adapter.Vercel{}, Diff: vercelDiff, Keep: vercelKeep}) + plugin.RegisterProvider(plugin.Provider{ + ID: "vercel", Name: "Vercel", Category: plugin.CategoryHosting, Homepage: "https://vercel.com", + Sources: []plugin.Source{{ + ID: "api", Layer: plugin.LayerAPI, Page: "providers/vercel.md", + Watches: "REST API operations deprecated, and deprecated operations removed, between SDK releases of the spec", + URLs: []string{"https://github.com/vercel/sdk"}, + Plugin: Plugin{Adapter: adapter.Vercel{}, Diff: vercelDiff, Keep: vercelKeep}, + }}, + }) // No Hints: Supabase's changelog is platform news, not keyed by // operation. Only operation deprecations and removals: see supabase.go. - plugin.Register(Plugin{Adapter: adapter.Supabase{}, Diff: supabaseDiff, Keep: supabaseKeep}) + plugin.RegisterProvider(plugin.Provider{ + ID: "supabase", Name: "Supabase", Category: plugin.CategoryDatabases, Homepage: "https://supabase.com", + Sources: []plugin.Source{{ + ID: "management-api", Layer: plugin.LayerAPI, Page: "providers/supabase.md", + Watches: "Management API operations deprecated, and deprecated operations removed, between commits of the spec", + URLs: []string{"https://github.com/supabase/supabase"}, + Plugin: Plugin{Adapter: adapter.Supabase{}, Diff: supabaseDiff, Keep: supabaseKeep}, + }}, + }) // No Hints: Algolia's changelog is a hosted product feed, not keyed by - // operation. One plugin per API spec: see algolia.go. - for _, spec := range []string{"search", "ingestion", "recommend"} { - plugin.Register(Plugin{Adapter: adapter.Algolia{Spec: spec}, Diff: algoliaDiff, Keep: algoliaKeep}) + // operation. One source per API spec: see algolia.go. + algolia := plugin.Provider{ID: "algolia", Name: "Algolia", Category: plugin.CategorySearch, Homepage: "https://www.algolia.com"} + for _, spec := range []struct{ name, title string }{{"search", "Search"}, {"ingestion", "Ingestion"}, {"recommend", "Recommend"}} { + algolia.Sources = append(algolia.Sources, plugin.Source{ + ID: spec.name + "-api", Layer: plugin.LayerAPI, Page: "providers/algolia.md", + Watches: spec.title + " API operations deprecated, and deprecated operations removed, between releases of the spec", + URLs: []string{"https://github.com/algolia/api-clients-automation", "https://github.com/algolia/api-clients-automation/blob/main/specs/bundled/" + spec.name + ".yml"}, + Plugin: Plugin{Adapter: adapter.Algolia{Spec: spec.name}, Diff: algoliaDiff, Keep: algoliaKeep}, + }) } + plugin.RegisterProvider(algolia) // No Hints: Elastic's breaking-changes page is prose per area, not keyed // by operation. See elasticsearch.go. - plugin.Register(Plugin{Adapter: adapter.Elasticsearch{}, Diff: elasticsearchDiff, Keep: elasticsearchKeep, RemovalsNeedANewVersion: true}) + plugin.RegisterProvider(plugin.Provider{ + ID: "elastic", Name: "Elastic", Category: plugin.CategorySearch, Homepage: "https://www.elastic.co", + Sources: []plugin.Source{{ + ID: "elasticsearch-api", Layer: plugin.LayerAPI, Page: "providers/elasticsearch.md", + Watches: "Elasticsearch REST API operations deprecated, and deprecated operations removed, between released minor versions of the spec", + URLs: []string{"https://github.com/elastic/elasticsearch-specification"}, + Plugin: Plugin{Adapter: adapter.Elasticsearch{}, Diff: elasticsearchDiff, Keep: elasticsearchKeep, RemovalsNeedANewVersion: true}, + }}, + }) // No Hints: Sentry's changelog is product news, not keyed by operation. // Operation deprecations only, most said in the description: see // sentry.go. - plugin.Register(Plugin{Adapter: adapter.Sentry{}, Diff: sentryDiff, Keep: sentryKeep}) + plugin.RegisterProvider(plugin.Provider{ + ID: "sentry", Name: "Sentry", Category: plugin.CategoryMonitoring, Homepage: "https://sentry.io", + Sources: []plugin.Source{{ + ID: "api", Layer: plugin.LayerAPI, Page: "providers/sentry.md", + Watches: "REST API operations deprecated between releases of the spec, most of them in the operation's description", + URLs: []string{"https://github.com/getsentry/sentry/releases"}, + Plugin: Plugin{Adapter: adapter.Sentry{}, Diff: sentryDiff, Keep: sentryKeep}, + }}, + }) // No Hints: Neon's changelog is a weekly digest, not keyed by operation. // Only operation deprecations and removals: see neon.go. - plugin.Register(Plugin{Adapter: adapter.Neon{}, Diff: neonDiff, Keep: neonKeep}) + plugin.RegisterProvider(plugin.Provider{ + ID: "neon", Name: "Neon", Category: plugin.CategoryDatabases, Homepage: "https://neon.com", + Sources: []plugin.Source{{ + ID: "api", Layer: plugin.LayerAPI, Page: "providers/neon.md", + Watches: "Neon API operations deprecated, and deprecated operations removed, between commits of the spec", + URLs: []string{"https://github.com/neondatabase/neon-pkgs"}, + Plugin: Plugin{Adapter: adapter.Neon{}, Diff: neonDiff, Keep: neonKeep}, + }}, + }) // No Hints: Convex has no API changelog; its news feed is product news. // Only operation deprecations and removals: see convex.go. - plugin.Register(Plugin{Adapter: adapter.Convex{}, Diff: convexDiff, Keep: convexKeep}) + plugin.RegisterProvider(plugin.Provider{ + ID: "convex", Name: "Convex", Category: plugin.CategoryDatabases, Homepage: "https://www.convex.dev", + Sources: []plugin.Source{{ + ID: "deployment-api", Layer: plugin.LayerAPI, Page: "providers/convex.md", + Watches: "Deployment API operations deprecated, and deprecated operations removed, between commits of the spec", + URLs: []string{"https://github.com/get-convex/convex-backend"}, + Plugin: Plugin{Adapter: adapter.Convex{}, Diff: convexDiff, Keep: convexKeep}, + }}, + }) // No Hints: PlanetScale's changelog is product news, not keyed by // operation. Operation deprecations only, said in the description: see // planetscale.go. - plugin.Register(Plugin{Adapter: adapter.PlanetScale{}, Diff: planetscaleDiff, Keep: planetscaleKeep}) + plugin.RegisterProvider(plugin.Provider{ + ID: "planetscale", Name: "PlanetScale", Category: plugin.CategoryDatabases, Homepage: "https://planetscale.com", + Sources: []plugin.Source{{ + ID: "api", Layer: plugin.LayerAPI, Page: "providers/planetscale.md", + Watches: "API operations deprecated, and deprecated operations removed, between commits of the spec, said in the operation's description", + URLs: []string{"https://github.com/planetscale/terraform-provider-planetscale"}, + Plugin: Plugin{Adapter: adapter.PlanetScale{}, Diff: planetscaleDiff, Keep: planetscaleKeep}, + }}, + }) // No Hints: Appwrite's changelog is product news, not keyed by operation. // Operation deprecations only, SDK method renames left out: see // appwrite.go. - plugin.Register(Plugin{Adapter: adapter.Appwrite{}, Diff: appwriteDiff, Keep: appwriteKeep}) + plugin.RegisterProvider(plugin.Provider{ + ID: "appwrite", Name: "Appwrite", Category: plugin.CategoryHosting, Homepage: "https://appwrite.io", + Sources: []plugin.Source{{ + ID: "api", Layer: plugin.LayerAPI, Page: "providers/appwrite.md", + Watches: "REST API operations a server SDK calls that are deprecated between commits of the spec, SDK method renames left out", + URLs: []string{"https://github.com/appwrite/specs"}, + Plugin: Plugin{Adapter: adapter.Appwrite{}, Diff: appwriteDiff, Keep: appwriteKeep}, + }}, + }) // No Hints: Flagsmith's release notes are product news, not keyed by // operation. requestSide, same reason as GitLab's: a named request-body // schema shared by several operations (FeatureValue, SegmentOverride) // reads as its own removal, side "response" by the differ's default, // whether or not any operation that used it is actually gone. - plugin.Register(Plugin{Adapter: adapter.Flagsmith{}, Keep: requestSide}) + plugin.RegisterProvider(plugin.Provider{ + ID: "flagsmith", Name: "Flagsmith", Category: plugin.CategoryDevTools, Homepage: "https://www.flagsmith.com", + Sources: []plugin.Source{{ + ID: "api", Layer: plugin.LayerAPI, Page: "providers/flagsmith.md", + Watches: "REST API changes between releases of Flagsmith's own OpenAPI spec (removed endpoints and request parameters)", + URLs: []string{"https://github.com/Flagsmith/flagsmith/blob/main/openapi.yaml"}, + Plugin: Plugin{Adapter: adapter.Flagsmith{}, Keep: requestSide}, + }}, + }) } // requestSide keeps the changes to what callers send. GitLab's response @@ -102,106 +214,117 @@ type Plugin struct { func (p Plugin) Name() string { return p.Adapter.API() } -// Run diffs the newest release against whatever was last stored. +// Run diffs the newest release against whatever was last stored: the newest +// version (LatestTag), ReadAt it, and Extract against the last baseline. func (p Plugin) Run(ctx context.Context, env plugin.Env) (plugin.Result, error) { a := p.Adapter tag, err := a.LatestTag(ctx) if err != nil { return plugin.Result{}, fmt.Errorf("finding latest tag: %w", err) } - newSpec, err := a.Fetch(ctx, tag) + to, err := p.ReadAt(ctx, env, plugin.Version{ID: tag, Ref: tag}) if err != nil { - return plugin.Result{}, fmt.Errorf("fetching %s %s: %w", a.API(), tag, err) + return plugin.Result{}, err } last, err := env.Baselines.LastSpec(a.API()) if err != nil { return plugin.Result{}, fmt.Errorf("loading last spec: %w", err) } - if last != nil && last.SHA256 == plugin.SHA256(newSpec.Content) { + if last != nil && last.SHA256 == plugin.SHA256(to.Content) { log.Printf("%s: no change (%s)", a.API(), tag) return plugin.Result{}, nil } - - var oldParsed *diff.Spec - var previousVersion, oldSourceURL string + var from plugin.Snapshot if last != nil { - oldParsed, err = diff.Parse(last.Content) - if err != nil { - return plugin.Result{}, fmt.Errorf("parsing stored spec: %w", err) - } - previousVersion = last.Version - oldSourceURL = last.SourceURL + from = plugin.Snapshot{Version: plugin.Version{ID: last.Version, Ref: last.Tag}, Content: last.Content, SourceURL: last.SourceURL} } - newParsed, err := diff.Parse(newSpec.Content) + res, err := p.Extract(ctx, env, from, to) if err != nil { - return plugin.Result{}, fmt.Errorf("parsing fetched spec: %w", err) + return plugin.Result{}, err } - - var oldContent []byte - if last != nil { - oldContent = last.Content - } - events, errs := p.events(oldParsed, newParsed, oldContent, newSpec.Content, normalize.Source{ - API: a.API(), - Version: newSpec.Version, - PreviousVersion: previousVersion, - NewSpecURL: newSpec.SourceURL, - OldSpecURL: oldSourceURL, - DetectedAt: env.Now().UTC(), - }) - p.attachHints(ctx, newSpec.Version, events) - converted, invalid := migrate.Fresh(events) - return plugin.Result{ - Label: tag, - Events: converted, - Rejected: append(errs, invalid...), - Baselines: []plugin.Spec{plugin.Baseline(a.API(), newSpec.Version, tag, newSpec.SourceURL, newSpec.Content, false, env.Now())}, - }, nil + res.Label = tag + res.Baselines = []plugin.Spec{plugin.Baseline(a.API(), to.Version.ID, tag, to.SourceURL, to.Content, false, env.Now())} + return res, nil } // Replay diffs two given releases. Both specs are saved marked as replays, so // the normal-run baseline never moves; the events are stored like any others: // that's how a historical change becomes something the feed can serve. func (p Plugin) Replay(ctx context.Context, env plugin.Env, from, to string) (plugin.Result, error) { - a := p.Adapter - oldSpec, err := a.Fetch(ctx, from) + old, err := p.ReadAt(ctx, env, plugin.Version{ID: from, Ref: from}) + if err != nil { + return plugin.Result{}, err + } + next, err := p.ReadAt(ctx, env, plugin.Version{ID: to, Ref: to}) + if err != nil { + return plugin.Result{}, err + } + res, err := p.Extract(ctx, env, old, next) if err != nil { - return plugin.Result{}, fmt.Errorf("fetching %s %s: %w", a.API(), from, err) + return plugin.Result{}, err + } + now := env.Now() + res.Label = to + res.Baselines = []plugin.Spec{ + plugin.Baseline(p.Adapter.API(), old.Version.ID, from, old.SourceURL, old.Content, true, now), + plugin.Baseline(p.Adapter.API(), next.Version.ID, to, next.SourceURL, next.Content, true, now), + } + return res, nil +} + +// ListVersions is every release of the spec, oldest first, when the adapter +// can list them (adapter.Lister); otherwise only the newest (LatestTag). +func (p Plugin) ListVersions(ctx context.Context, _ plugin.Env) ([]plugin.Version, error) { + if l, ok := p.Adapter.(adapter.Lister); ok { + return l.Versions(ctx) } - newSpec, err := a.Fetch(ctx, to) + tag, err := p.Adapter.LatestTag(ctx) if err != nil { - return plugin.Result{}, fmt.Errorf("fetching %s %s: %w", a.API(), to, err) + return nil, fmt.Errorf("finding latest tag: %w", err) } - oldParsed, err := diff.Parse(oldSpec.Content) + return []plugin.Version{{ID: tag, Ref: tag}}, nil +} + +// ReadAt fetches the spec at v.Ref. The snapshot's version ID is the version +// the spec itself declares (what events carry), which may differ from the tag. +func (p Plugin) ReadAt(ctx context.Context, _ plugin.Env, v plugin.Version) (plugin.Snapshot, error) { + s, err := p.Adapter.Fetch(ctx, v.Ref) if err != nil { - return plugin.Result{}, fmt.Errorf("parsing %s: %w", from, err) + return plugin.Snapshot{}, fmt.Errorf("fetching %s %s: %w", p.Adapter.API(), v.Ref, err) + } + v.ID = s.Version + return plugin.Snapshot{Version: v, Content: s.Content, SourceURL: s.SourceURL}, nil +} + +// Extract diffs two snapshots of the spec into events, with to's version and +// from's as previous_version. A zero from is an empty spec before the first. +func (p Plugin) Extract(ctx context.Context, env plugin.Env, from, to plugin.Snapshot) (plugin.Result, error) { + var oldParsed *diff.Spec + if !from.IsZero() { + var err error + if oldParsed, err = diff.Parse(from.Content); err != nil { + return plugin.Result{}, fmt.Errorf("parsing the %s spec at %s: %w", p.Adapter.API(), from.Version.ID, err) + } } - newParsed, err := diff.Parse(newSpec.Content) + newParsed, err := diff.Parse(to.Content) if err != nil { - return plugin.Result{}, fmt.Errorf("parsing %s: %w", to, err) + return plugin.Result{}, fmt.Errorf("parsing the %s spec at %s: %w", p.Adapter.API(), to.Version.ID, err) } - events, errs := p.events(oldParsed, newParsed, oldSpec.Content, newSpec.Content, normalize.Source{ - API: a.API(), - Version: newSpec.Version, - PreviousVersion: oldSpec.Version, - NewSpecURL: newSpec.SourceURL, - OldSpecURL: oldSpec.SourceURL, + events, errs := p.events(oldParsed, newParsed, from.Content, to.Content, normalize.Source{ + API: p.Adapter.API(), + Version: to.Version.ID, + PreviousVersion: from.Version.ID, + NewSpecURL: to.SourceURL, + OldSpecURL: from.SourceURL, DetectedAt: env.Now().UTC(), }) - p.attachHints(ctx, newSpec.Version, events) - now := env.Now() + p.attachHints(ctx, to.Version.ID, events) converted, invalid := migrate.Fresh(events) - return plugin.Result{ - Label: to, - Events: converted, - Rejected: append(errs, invalid...), - Baselines: []plugin.Spec{ - plugin.Baseline(a.API(), oldSpec.Version, from, oldSpec.SourceURL, oldSpec.Content, true, now), - plugin.Baseline(a.API(), newSpec.Version, to, newSpec.SourceURL, newSpec.Content, true, now), - }, - }, nil + return plugin.Result{Label: to.Version.Ref, Events: converted, Rejected: append(errs, invalid...)}, nil } +var _ plugin.Versioned = Plugin{} + // events diffs two parsed specs into events, applying Keep, and pointing each // event's source_pointer into the published file when the adapter reshaped it. func (p Plugin) events(oldParsed, newParsed *diff.Spec, oldContent, newContent []byte, src normalize.Source) ([]event.ChangeEvent, []error) { diff --git a/plugins/openapi/openapi_test.go b/plugins/openapi/openapi_test.go index 4f05499..d76c60c 100644 --- a/plugins/openapi/openapi_test.go +++ b/plugins/openapi/openapi_test.go @@ -6,11 +6,11 @@ import ( "testing" "time" - "github.com/patch-it-dev/patch-it-plugins/adapter" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/hint" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/hint" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" + "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter" ) // fakeHints stands in for a changelog: an index built from the entries given, diff --git a/plugins/openapi/planetscale.go b/plugins/openapi/planetscale.go index a922223..7531942 100644 --- a/plugins/openapi/planetscale.go +++ b/plugins/openapi/planetscale.go @@ -3,7 +3,7 @@ package openapi import ( "regexp" - "github.com/patch-it-dev/patch-it-plugins/diff" + "github.com/patch-it-dev/patch-it-plugins/core/diff" ) // planetscaleDiff is how the PlanetScale API spec says what it says: diff --git a/plugins/openapi/planetscale_test.go b/plugins/openapi/planetscale_test.go index 5cf0b0f..ecf51f3 100644 --- a/plugins/openapi/planetscale_test.go +++ b/plugins/openapi/planetscale_test.go @@ -9,10 +9,10 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/adapter" - "github.com/patch-it-dev/patch-it-plugins/diff" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/diff" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter" ) // planetscaleCommits are the planetscale/terraform-provider-planetscale diff --git a/plugins/openapi/registered_test.go b/plugins/openapi/registered_test.go index faa4a3d..489b0c1 100644 --- a/plugins/openapi/registered_test.go +++ b/plugins/openapi/registered_test.go @@ -3,8 +3,8 @@ package openapi import ( "testing" - "github.com/patch-it-dev/patch-it-plugins/adapter" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter" ) func TestRegistersItself(t *testing.T) { diff --git a/plugins/openapi/sentry.go b/plugins/openapi/sentry.go index a702fd5..dc0fa5a 100644 --- a/plugins/openapi/sentry.go +++ b/plugins/openapi/sentry.go @@ -3,7 +3,7 @@ package openapi import ( "regexp" - "github.com/patch-it-dev/patch-it-plugins/diff" + "github.com/patch-it-dev/patch-it-plugins/core/diff" ) // sentryDiff is how Sentry's spec says what it says: diff --git a/plugins/openapi/sentry_test.go b/plugins/openapi/sentry_test.go index 03b69c8..c435b28 100644 --- a/plugins/openapi/sentry_test.go +++ b/plugins/openapi/sentry_test.go @@ -11,10 +11,10 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/adapter" - "github.com/patch-it-dev/patch-it-plugins/diff" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/diff" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter" ) // The getsentry/sentry-api-schema commits and release tags saved under diff --git a/plugins/openapi/supabase.go b/plugins/openapi/supabase.go index 840261f..20d4f3a 100644 --- a/plugins/openapi/supabase.go +++ b/plugins/openapi/supabase.go @@ -1,6 +1,6 @@ package openapi -import "github.com/patch-it-dev/patch-it-plugins/diff" +import "github.com/patch-it-dev/patch-it-plugins/core/diff" // supabaseDiff is how the Supabase Management API spec says what it says: // diff --git a/plugins/openapi/supabase_test.go b/plugins/openapi/supabase_test.go index b6c549f..a9ae173 100644 --- a/plugins/openapi/supabase_test.go +++ b/plugins/openapi/supabase_test.go @@ -7,9 +7,9 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/adapter" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter" ) // supabaseCommits are the spec commits saved under testdata/ (trimmed to the diff --git a/plugins/openapi/vercel.go b/plugins/openapi/vercel.go index 0c50690..191be09 100644 --- a/plugins/openapi/vercel.go +++ b/plugins/openapi/vercel.go @@ -3,7 +3,7 @@ package openapi import ( "regexp" - "github.com/patch-it-dev/patch-it-plugins/diff" + "github.com/patch-it-dev/patch-it-plugins/core/diff" ) // vercelDiff is how Vercel's spec says what it says: diff --git a/plugins/openapi/vercel_test.go b/plugins/openapi/vercel_test.go index 6d0ebed..2c1e357 100644 --- a/plugins/openapi/vercel_test.go +++ b/plugins/openapi/vercel_test.go @@ -10,10 +10,10 @@ import ( "testing" "time" - "github.com/patch-it-dev/patch-it-plugins/adapter" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" + "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter" ) // fakeVercelAdapter serves the trimmed, unedited excerpts of vercel/sdk's @@ -212,20 +212,27 @@ func TestVercelPathParameterRenameIsNoChange(t *testing.T) { } func TestVercelDeprecatedSummary(t *testing.T) { - for summary, want := range map[string]bool{ - // Every word-only deprecation form in vercel/sdk's history. - "Check the price for a domain (deprecated)": true, - "Retrieves a list of all the Log Drains (deprecated)": true, - "Deprecated: Use DELETE /v1/drives/:name. Requires ...": true, - "Deprecated: true. Update Resource Secrets (Deprecated)": true, - "Get a list of deprecated Node.js versions": false, - "Undeprecated thing": false, - "Deprecated runtimes for a project": false, - "Update a project (deprecated fields are ignored)": false, - "": false, + // Every word-only deprecation form in vercel/sdk's history. + for _, summary := range []string{ + "Check the price for a domain (deprecated)", + "Retrieves a list of all the Log Drains (deprecated)", + "Deprecated: Use DELETE /v1/drives/:name. Requires ...", + "Deprecated: true. Update Resource Secrets (Deprecated)", } { - if got := vercelDeprecatedSummary(summary); got != want { - t.Errorf("%q: got %v, want %v", summary, got, want) + if !vercelDeprecatedSummary(summary) { + t.Errorf("%q: not read as deprecated", summary) + } + } + // Mentions of the word that aren't a deprecation, and no summary at all. + for _, summary := range []string{ + "Get a list of deprecated Node.js versions", + "Undeprecated thing", + "Deprecated runtimes for a project", + "Update a project (deprecated fields are ignored)", + "", + } { + if vercelDeprecatedSummary(summary) { + t.Errorf("%q: read as deprecated", summary) } } } diff --git a/plugins/openapi/versions_test.go b/plugins/openapi/versions_test.go new file mode 100644 index 0000000..e204766 --- /dev/null +++ b/plugins/openapi/versions_test.go @@ -0,0 +1,118 @@ +package openapi + +import ( + "context" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/versions" + "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter" +) + +// listingAdapter is fakeAdapter that can also list its versions. +type listingAdapter struct { + fakeAdapter + list []plugin.Version +} + +func (l listingAdapter) Versions(context.Context) ([]plugin.Version, error) { return l.list, nil } + +var _ adapter.Lister = listingAdapter{} + +// Three releases, read version by version: each consecutive pair is its own +// Extract, with to's version and from's as previous_version, and only what +// changed in that pair. +func TestEveryConsecutivePairIsExtracted(t *testing.T) { + spec := func(v string, props ...string) string { + var ps []string + for _, p := range props { + ps = append(ps, fmt.Sprintf("%q:{\"type\":\"string\"}", p)) + } + return fmt.Sprintf(`{"openapi":"3.0.0","info":{"version":"version-%s"},"paths":{},"components":{"schemas":{"charge":{"type":"object","properties":{%s}}}}}`, v, strings.Join(ps, ",")) + } + a := listingAdapter{ + fakeAdapter: fakeAdapter{latest: "v4", specs: map[string]string{ + "v1": spec("v1", "id", "a", "b", "c"), + "v2": spec("v2", "id", "b", "c"), // a removed + "v3": spec("v3", "id", "c"), // b removed + "v4": spec("v4", "id"), // c removed + }}, + list: []plugin.Version{{ID: "v1", Ref: "v1"}, {ID: "v2", Ref: "v2"}, {ID: "v3", Ref: "v3"}, {ID: "v4", Ref: "v4"}}, + } + p := newPlugin(a, noChangelog) + ctx, e := context.Background(), env(kittest.NewMem()) + vs, err := p.ListVersions(ctx, e) + if err != nil || len(vs) != 4 { + t.Fatalf("ListVersions = %v, %v", vs, err) + } + var got []string + for _, pair := range versions.Pairs(vs) { + from, err := p.ReadAt(ctx, e, pair[0]) + if err != nil { + t.Fatal(err) + } + to, err := p.ReadAt(ctx, e, pair[1]) + if err != nil { + t.Fatal(err) + } + res, err := p.Extract(ctx, e, from, to) + if err != nil || len(res.Rejected) > 0 || len(res.Baselines) > 0 { + t.Fatalf("Extract: %v %v baselines=%d", err, res.Rejected, len(res.Baselines)) + } + for _, ev := range res.Events { + got = append(got, fmt.Sprintf("%s<-%s %s %s", ev.Version, ev.PreviousVersion, ev.ChangeType, ev.Subject.Property)) + } + } + want := "version-v2<-version-v1 breaking a|version-v3<-version-v2 breaking b|version-v4<-version-v3 breaking c" + if strings.Join(got, "|") != want { + t.Errorf("got %s\nwant %s", strings.Join(got, "|"), want) + } +} + +// An adapter that can't list is read at its newest version only. +func TestListVersionsFallsBackToTheNewest(t *testing.T) { + vs, err := newPlugin(fakeAdapter{latest: "v2", specs: specs}, noChangelog).ListVersions(context.Background(), plugin.Env{}) + if err != nil || len(vs) != 1 || vs[0].Ref != "v2" { + t.Errorf("got %v, %v", vs, err) + } +} + +// The real adapters list GitHub releases or the commits that changed their +// spec, through core/versions. +func TestAdaptersListReleasesAndCommits(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch { + case r.URL.Path == "/repos/stripe/openapi/releases": + fmt.Fprint(w, `[{"tag_name":"v2","published_at":"2026-02-01T00:00:00Z"},{"tag_name":"v1","published_at":"2026-01-01T00:00:00Z"}]`) + case r.URL.Path == "/repos/supabase/supabase/commits" && r.URL.Query().Get("path") == "apps/docs/spec/api_v1_openapi.json": + fmt.Fprint(w, `[{"sha":"b","commit":{"committer":{"date":"2026-02-01T00:00:00Z"}}},{"sha":"a","commit":{"committer":{"date":"2026-01-01T00:00:00Z"}}}]`) + default: + http.NotFound(w, r) + } + })) + defer srv.Close() + old := versions.GitHubAPI + versions.GitHubAPI = srv.URL + defer func() { versions.GitHubAPI = old }() + for _, c := range []struct { + a adapter.Adapter + want string + }{{adapter.Stripe{}, "v1 v2"}, {adapter.Supabase{}, "a b"}} { + vs, err := Plugin{Adapter: c.a}.ListVersions(context.Background(), plugin.Env{}) + if err != nil { + t.Fatalf("%s: %v", c.a.API(), err) + } + var refs []string + for _, v := range vs { + refs = append(refs, v.Ref) + } + if strings.Join(refs, " ") != c.want { + t.Errorf("%s: refs %v, want %s", c.a.API(), refs, c.want) + } + } +} diff --git a/plugins/removedmembers/golden_test.go b/plugins/removedmembers/golden_test.go index dd6bd52..ac62dbd 100644 --- a/plugins/removedmembers/golden_test.go +++ b/plugins/removedmembers/golden_test.go @@ -3,7 +3,7 @@ package removedmembers import ( "testing" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" ) // The guide at tag v2.0.0 (saved 2026-11-04) in, every event out. diff --git a/plugins/removedmembers/newrelic.go b/plugins/removedmembers/newrelic.go index 842197d..1fb3cd5 100644 --- a/plugins/removedmembers/newrelic.go +++ b/plugins/removedmembers/newrelic.go @@ -65,7 +65,7 @@ var NewRelic11 = Guide{ Package: "newrelic", Module: "newrelic", Release: "11.0.0", - Tag: "f0b4f817bf9a7b76a78e26609369a8a7ce764c8e", + DocsRef: "f0b4f817bf9a7b76a78e26609369a8a7ce764c8e", Repo: "newrelic/docs-website", Path: "src/content/docs/release-notes/agent-release-notes/python-release-notes/python-agent-110000.mdx", Section: "Removed features", diff --git a/plugins/removedmembers/newrelic_test.go b/plugins/removedmembers/newrelic_test.go index 9102714..220052b 100644 --- a/plugins/removedmembers/newrelic_test.go +++ b/plugins/removedmembers/newrelic_test.go @@ -7,8 +7,8 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" ) // An excerpt of the release note at commit f0b4f817 (fetched 2026-11-19): the diff --git a/plugins/removedmembers/posthog.go b/plugins/removedmembers/posthog.go index 7f6b8ce..9250e5c 100644 --- a/plugins/removedmembers/posthog.go +++ b/plugins/removedmembers/posthog.go @@ -56,16 +56,16 @@ package removedmembers // fields, retries, dropped events, options, and "Rename deprecated settings", // whose old names still work) aren't under the heading, so they're never read. var PostHog8 = Guide{ - Package: "posthog", - Module: "posthog", - Release: "8.0.0", - Tag: "posthog-v8.0.0", - Repo: "PostHog/posthog-python", - Path: "docs/migration-7.x-to-8.0.md", - Section: "Fix removed APIs", - Format: FormatMarkdown, - Header: "Removed", - EnvVars: []string{"POSTHOG_CAPTURE_MODE"}, + Package: "posthog", + Module: "posthog", + Release: "8.0.0", + TagPrefix: "posthog-v", + Repo: "PostHog/posthog-python", + Path: "docs/migration-7.x-to-8.0.md", + Section: "Fix removed APIs", + Format: FormatMarkdown, + Header: "Removed", + EnvVars: []string{"POSTHOG_CAPTURE_MODE"}, Paths: map[string]string{ "CaptureV1Error": "capture_v1.CaptureV1Error", "async_batch_post": "_async_request.async_batch_post", diff --git a/plugins/removedmembers/posthog_test.go b/plugins/removedmembers/posthog_test.go index cccf046..8b7d70a 100644 --- a/plugins/removedmembers/posthog_test.go +++ b/plugins/removedmembers/posthog_test.go @@ -5,8 +5,8 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" ) // The guide as published at tag posthog-v8.0.0 (fetched 2026-11-19), whole. diff --git a/plugins/removedmembers/registered_test.go b/plugins/removedmembers/registered_test.go index b0b7a81..20af0b9 100644 --- a/plugins/removedmembers/registered_test.go +++ b/plugins/removedmembers/registered_test.go @@ -3,7 +3,7 @@ package removedmembers import ( "testing" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" ) func TestRegistersItself(t *testing.T) { diff --git a/plugins/removedmembers/removedmembers.go b/plugins/removedmembers/removedmembers.go index 5dc6ccb..ec78e71 100644 --- a/plugins/removedmembers/removedmembers.go +++ b/plugins/removedmembers/removedmembers.go @@ -27,14 +27,39 @@ import ( "slices" "strings" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" + "github.com/patch-it-dev/patch-it-plugins/core/versions" ) func init() { - plugin.Register(Plugin{Guide: NumPy2}) - plugin.Register(Plugin{Guide: PostHog8}) - plugin.Register(Plugin{Guide: NewRelic11}) + plugin.RegisterProvider(plugin.Provider{ + ID: "numpy", Name: "NumPy", Category: plugin.CategoryPackages, Homepage: "https://numpy.org", + Sources: []plugin.Source{{ + ID: "numpy", Layer: plugin.LayerPackage, Page: "providers/numpy.md", + Watches: "Members removed or renamed in the main namespace in NumPy 2.0", + URLs: []string{"https://github.com/numpy/numpy/blob/v2.0.0/doc/source/numpy_2_0_migration_guide.rst"}, + Plugin: Plugin{Guide: NumPy2}, + }}, + }) + plugin.RegisterProvider(plugin.Provider{ + ID: "posthog", Name: "PostHog", Category: plugin.CategoryAnalytics, Homepage: "https://posthog.com", + Sources: []plugin.Source{{ + ID: "python-sdk", Layer: plugin.LayerSDK, Page: "providers/posthog.md", + Watches: "Members removed in the posthog Python SDK 8.0.0", + URLs: []string{"https://github.com/PostHog/posthog-python/blob/posthog-v8.0.0/docs/migration-7.x-to-8.0.md"}, + Plugin: Plugin{Guide: PostHog8}, + }}, + }) + plugin.RegisterProvider(plugin.Provider{ + ID: "newrelic", Name: "New Relic", Category: plugin.CategoryMonitoring, Homepage: "https://newrelic.com", + Sources: []plugin.Source{{ + ID: "python-agent", Layer: plugin.LayerSDK, Page: "providers/newrelic.md", + Watches: "Members removed in the New Relic Python agent 11.0.0", + URLs: []string{"https://github.com/newrelic/docs-website/blob/f0b4f817bf9a7b76a78e26609369a8a7ce764c8e/src/content/docs/release-notes/agent-release-notes/python-release-notes/python-agent-110000.mdx"}, + Plugin: Plugin{Guide: NewRelic11}, + }}, + }) } // Guide is one package's guide to one release. @@ -46,8 +71,13 @@ type Guide struct { Path string // the guide's path in the repo at the release tag Section string // the heading the table is under - // Tag is the release tag; "" is "v" + Release. - Tag string + // TagPrefix makes a release's git tag: TagPrefix + version ("" is "v": + // v2.0.0; PostHog's is "posthog-v"). The guide is read at Release's tag. + TagPrefix string + // DocsRef, when the guide isn't in the package's own repo but in a docs + // repo with no release tags (New Relic's release notes), is the commit + // of Repo the guide is read at. + DocsRef string // Format is how the table is written: FormatRST (the default) or // FormatMarkdown. Format Format @@ -108,11 +138,17 @@ const ( FormatMarkdown Format = "markdown" ) -func (g Guide) tag() string { - if g.Tag != "" { - return g.Tag +func (g Guide) tag() string { return g.tagOf(g.Release) } + +// tagOf is the git ref a release of the package is read at. +func (g Guide) tagOf(release string) string { + if g.DocsRef != "" { + return g.DocsRef + } + if g.TagPrefix != "" { + return g.TagPrefix + release } - return "v" + g.Release + return "v" + release } func (g Guide) doc() string { @@ -239,15 +275,58 @@ type Plugin struct { func (p Plugin) Name() string { return p.Guide.Package } -// Run reads the guide at its tag. The tag never changes, so a rerun stores -// nothing new: the fingerprints dedupe. +// Run reads the guide at its release's tag. The tag never changes, so a +// rerun stores nothing new: the fingerprints dedupe. func (p Plugin) Run(ctx context.Context, env plugin.Env) (plugin.Result, error) { g := p.Guide + v := plugin.Version{ID: g.Release, Ref: g.tag()} + to, err := p.ReadAt(ctx, env, v) + if err != nil { + return plugin.Result{}, err + } + res, err := p.Extract(ctx, env, plugin.Snapshot{}, to) + if err != nil { + return plugin.Result{}, err + } + res.Baselines = []plugin.Spec{plugin.Baseline(g.Package, g.Release, g.tag(), g.URL(), to.Content, false, env.Now().UTC())} + return res, nil +} + +// ListVersions is every final release of the package on PyPI, oldest first, +// each with the ref its guide would be read at. +func (p Plugin) ListVersions(ctx context.Context, env plugin.Env) ([]plugin.Version, error) { + vs, err := versions.PyPIReleases(ctx, env.Get, p.Guide.Package) + if err != nil { + return nil, err + } + for i := range vs { + vs[i].Ref = p.Guide.tagOf(vs[i].ID) + } + return vs, nil +} + +// ReadAt fetches the guide when v is the release it documents; any other +// release has no guide and reads as nothing (a snapshot with no content). +func (p Plugin) ReadAt(ctx context.Context, env plugin.Env, v plugin.Version) (plugin.Snapshot, error) { + g := p.Guide + if v.ID != g.Release { + return plugin.Snapshot{Version: v}, nil + } page, err := env.Get(ctx, g.URL()) if err != nil { - return plugin.Result{}, fmt.Errorf("fetching %s: %w", g.URL(), err) + return plugin.Snapshot{}, fmt.Errorf("fetching %s: %w", g.URL(), err) } - removals, err := Parse(page, g) + return plugin.Snapshot{Version: v, Content: page, SourceURL: g.URL()}, nil +} + +// Extract is an event per removed member in the to release's guide, with +// previous_version from's release; a release with no guide has none. +func (p Plugin) Extract(_ context.Context, env plugin.Env, from, to plugin.Snapshot) (plugin.Result, error) { + g := p.Guide + if to.Content == nil { + return plugin.Result{Label: to.Version.ID}, nil + } + removals, err := Parse(to.Content, g) if err != nil { return plugin.Result{}, fmt.Errorf("reading %s: %w", g.URL(), err) } @@ -271,17 +350,18 @@ func (p Plugin) Run(ctx context.Context, env plugin.Env) (plugin.Result, error) subject.Change, subject.Replacement = event.PackageRenamedSymbol, r.Replacement } e := event.EventV2{ - API: g.Package, - Version: g.Release, - ChangeType: event.ChangeTypeBreaking, - Breaking: true, - Subject: subject, - Description: desc, - MigrationHint: fmt.Sprintf("%s (%s's %s for %s)", g.absoluteLinks(r.Guideline), g.Package, g.doc(), g.Release), - SourceURL: g.URL(), - SourceExcerpt: r.Excerpt, - Confidence: 1.0, - DetectedAt: now, + API: g.Package, + PreviousVersion: from.Version.ID, + Version: g.Release, + ChangeType: event.ChangeTypeBreaking, + Breaking: true, + Subject: subject, + Description: desc, + MigrationHint: fmt.Sprintf("%s (%s's %s for %s)", g.absoluteLinks(r.Guideline), g.Package, g.doc(), g.Release), + SourceURL: g.URL(), + SourceExcerpt: r.Excerpt, + Confidence: 1.0, + DetectedAt: now, } e.Fingerprint = event.FingerprintV2(e.API, e.Version, e.ChangeType, subject, "") if err := e.Validate(); err != nil { @@ -290,14 +370,11 @@ func (p Plugin) Run(ctx context.Context, env plugin.Env) (plugin.Result, error) } events = append(events, e) } - return plugin.Result{ - Label: g.URL(), - Events: events, - Rejected: rejected, - Baselines: []plugin.Spec{plugin.Baseline(g.Package, g.Release, g.tag(), g.URL(), page, false, now)}, - }, nil + return plugin.Result{Label: g.URL(), Events: events, Rejected: rejected}, nil } +var _ plugin.Versioned = Plugin{} + // absoluteLinks prefixes LinkBase to every Markdown link target in s that // starts with "/". func (g Guide) absoluteLinks(s string) string { diff --git a/plugins/removedmembers/removedmembers_test.go b/plugins/removedmembers/removedmembers_test.go index 8a23c8f..afab21b 100644 --- a/plugins/removedmembers/removedmembers_test.go +++ b/plugins/removedmembers/removedmembers_test.go @@ -4,8 +4,8 @@ import ( "strings" "testing" - "github.com/patch-it-dev/patch-it-plugins/event" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/event" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" ) // The guide as published at tag v2.0.0 (fetched 2026-11-04). diff --git a/plugins/removedmembers/testdata/README.md b/plugins/removedmembers/testdata/README.md index eed475a..05e4aa5 100644 --- a/plugins/removedmembers/testdata/README.md +++ b/plugins/removedmembers/testdata/README.md @@ -7,3 +7,4 @@ The golden files under `golden/` are what the plugin stores from them (`go test | `numpy_2_0_migration_guide-v2.0.0.rst` | https://github.com/numpy/numpy/blob/v2.0.0/doc/source/numpy_2_0_migration_guide.rst | 2026-11-04 | BSD-3-Clause (NumPy), kept whole | "The table below provides replacements for the removed members:" | | `migration-7.x-to-8.0-posthog-v8.0.0.md` | https://github.com/PostHog/posthog-python/blob/posthog-v8.0.0/docs/migration-7.x-to-8.0.md (commit 6edff41425188056e48f6119457dc9bc0669ca99) | 2026-11-19 | MIT (the repo's LICENSE is the MIT text, with two copyright lines, so GitHub's detector says NOASSERTION; PyPI says MIT), kept whole, byte for byte | "8.x.x removes these options and APIs:" (Fix removed APIs), with rows such as "\| `CaptureV1Error` \| `CaptureError` \|" | | `python-agent-110000-f0b4f817.mdx` | https://github.com/newrelic/docs-website/blob/f0b4f817bf9a7b76a78e26609369a8a7ce764c8e/src/content/docs/release-notes/agent-release-notes/python-release-notes/python-agent-110000.mdx (the repo has no tags; the file's latest commit) | 2026-11-19 | Excerpt for testing; © New Relic, CC BY-NC-SA 4.0. Kept: the "## Removed features" heading, the bullets that introduce its two tables, the "Legacy API" table and the "Legacy Setting" table (lines 30-31 and 36-66 of 136), every line unchanged. Left out: the front matter, "Notes", the Python 3.7 bullet, and the "Deprecated features", "New features", "Bug fixes" and "Support statement" sections. The plugin finds the same 8 rows and 7 events in the excerpt as in the full file | "The following APIs have been removed:", with rows such as "\| `record_exception` \| [`notice_error()`](/docs/agents/python-agent/python-agent-api/notice_error) \|" | +| `pypi-numpy-excerpt.json` | https://pypi.org/pypi/numpy/json (shape only) | written for the test, 2026-11-20 | not a saved copy: four releases in the shape of PyPI's JSON API (`releases` → files with `upload_time_iso_8601`, `yanked`), dates approximate, a pre-release among them | for the version-listing test only (ListVersions, consecutive pairs); no golden reads it | diff --git a/plugins/removedmembers/testdata/pypi-numpy-excerpt.json b/plugins/removedmembers/testdata/pypi-numpy-excerpt.json new file mode 100644 index 0000000..01ccf4d --- /dev/null +++ b/plugins/removedmembers/testdata/pypi-numpy-excerpt.json @@ -0,0 +1,6 @@ +{"releases":{ + "1.26.4":[{"upload_time_iso_8601":"2024-02-05T00:00:00Z","yanked":false}], + "2.0.0rc2":[{"upload_time_iso_8601":"2024-05-01T00:00:00Z","yanked":false}], + "2.0.0":[{"upload_time_iso_8601":"2024-06-16T00:00:00Z","yanked":false}], + "2.0.1":[{"upload_time_iso_8601":"2024-07-21T00:00:00Z","yanked":false}] +}} diff --git a/plugins/removedmembers/versions_test.go b/plugins/removedmembers/versions_test.go new file mode 100644 index 0000000..a48552f --- /dev/null +++ b/plugins/removedmembers/versions_test.go @@ -0,0 +1,61 @@ +package removedmembers + +import ( + "context" + "fmt" + "strings" + "testing" + + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/versions" +) + +// The package's releases come from PyPI; only the pair into the release the +// guide documents has events (the golden's, with previous_version set). +func TestOnlyTheGuidesReleaseHasEvents(t *testing.T) { + env := kittest.Env(t, kittest.Pages{ + "https://pypi.org/pypi/numpy/json": "testdata/pypi-numpy-excerpt.json", + NumPy2.URL(): "testdata/numpy_2_0_migration_guide-v2.0.0.rst", + }) + p := Plugin{Guide: NumPy2} + ctx := context.Background() + vs, err := p.ListVersions(ctx, env) + if err != nil { + t.Fatal(err) + } + pinned := kittest.Ingest(t, p, kittest.Pages{NumPy2.URL(): "testdata/numpy_2_0_migration_guide-v2.0.0.rst"}) + var got []string + for _, pair := range versions.Pairs(vs) { + from, err := p.ReadAt(ctx, env, pair[0]) + if err != nil { + t.Fatal(err) + } + to, err := p.ReadAt(ctx, env, pair[1]) + if err != nil { + t.Fatal(err) + } + res, err := p.Extract(ctx, env, from, to) + if err != nil || len(res.Rejected) > 0 { + t.Fatalf("%v %v", err, res.Rejected) + } + for i, e := range res.Events { + if e.PreviousVersion != pair[0].ID || e.Fingerprint != pinned[i].Fingerprint { + t.Errorf("event %d: previous %q, fingerprint differs from the golden's: %v", i, e.PreviousVersion, e.Fingerprint != pinned[i].Fingerprint) + } + } + got = append(got, fmt.Sprintf("%s->%s:%d", pair[0].ID, pair[1].ID, len(res.Events))) + } + want := fmt.Sprintf("1.26.4->2.0.0:%d 2.0.0->2.0.1:0", len(pinned)) + if strings.Join(got, " ") != want || len(pinned) == 0 { + t.Errorf("got %s, want %s", strings.Join(got, " "), want) + } +} + +// Each release's ref follows the guide's tag shape. +func TestTagsFollowTheGuide(t *testing.T) { + for g, want := range map[*Guide]string{&NumPy2: "v2.0.0", &PostHog8: "posthog-v8.0.0", &NewRelic11: "f0b4f817bf9a7b76a78e26609369a8a7ce764c8e"} { + if got := g.tag(); got != want { + t.Errorf("%s: tag %q, want %q", g.Package, got, want) + } + } +} diff --git a/plugins/retirements/golden_test.go b/plugins/retirements/golden_test.go index b4c3a51..b4095c9 100644 --- a/plugins/retirements/golden_test.go +++ b/plugins/retirements/golden_test.go @@ -3,8 +3,8 @@ package retirements import ( "testing" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" - "github.com/patch-it-dev/patch-it-plugins/retire" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/plugins/retirements/retire" ) // Both providers' saved pages (2026-10-13) in, every event out. The archive @@ -16,7 +16,7 @@ func TestGolden(t *testing.T) { "anthropic-deprecations.md": retire.Anthropic, } { t.Run(page.API, func(t *testing.T) { - saved := kittest.Pages{page.URL: "../../retire/testdata/" + file} + saved := kittest.Pages{page.URL: "retire/testdata/" + file} archive := &kittest.Archive{T: t, Pages: saved} got := kittest.IngestWith(t, kittest.Env(t, saved), Plugin{Page: page, Archiver: archive}) if archive.Calls != 1 { diff --git a/plugins/retirements/registered_test.go b/plugins/retirements/registered_test.go index a123690..c5be64c 100644 --- a/plugins/retirements/registered_test.go +++ b/plugins/retirements/registered_test.go @@ -3,8 +3,8 @@ package retirements import ( "testing" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" - "github.com/patch-it-dev/patch-it-plugins/retire" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/plugins/retirements/retire" ) func TestRegistersItself(t *testing.T) { diff --git a/retire/providers.go b/plugins/retirements/retire/providers.go similarity index 100% rename from retire/providers.go rename to plugins/retirements/retire/providers.go diff --git a/plugins/retirements/retire/retire.go b/plugins/retirements/retire/retire.go new file mode 100644 index 0000000..1c3d673 --- /dev/null +++ b/plugins/retirements/retire/retire.go @@ -0,0 +1,245 @@ +// Package retire reads a model provider's deprecations page and turns every +// retired model into change events: a second kind of source next to the +// OpenAPI differ. There is no spec to diff, only the provider's own table of +// shutdown date, model and named replacement, so this reads the table. +// +// Both providers serve the page as markdown at its URL plus ".md" (checked +// 2026-10-13), and both write each announcement as a table whose first column +// is the date the model stops working. The event cites a Wayback Machine +// snapshot of that markdown (kit/), never the live page: the contract +// requires a source that can't change under the event. +package retire + +import ( + "fmt" + "regexp" + "strings" + "time" + + "github.com/patch-it-dev/patch-it-plugins/core/event" +) + +// Page is one provider's deprecations page and what its models are sent to. +type Page struct { + API string // the api slug the events carry, e.g. "openai-models" + Vendor string // for sentences: "OpenAI" + URL string // the page as markdown + Operation string // the provider's main call that takes a model name + // Keep says whether a model name belongs to this source: the operations + // above take text models, not image, audio or embedding ones. + Keep func(name string) bool +} + +// Retirement is one model name the provider stops serving on Date, with the +// replacement it names. An alias gets its own Retirement, with the row it +// came from. +type Retirement struct { + Model string + Date string // YYYY-MM-DD + Replacement string // the replacement cell, as written ("`gpt-5.6-sol` (`reasoning.mode: pro`)") + Via []string + Excerpt string // the section heading, the table header and the row +} + +var ( + dateHeader = regexp.MustCompile(`(?i)^(shutdown|retirement) date$`) + ticked = regexp.MustCompile("`([^`]+)`") +) + +var dateLayouts = []string{"January 2, 2006", "Jan 2, 2006", "2006-01-02"} + +// Parse reads every retirement table on the page. A model listed more than +// once keeps its first row: both pages list the most recent announcement +// first, and a later one supersedes an earlier date, including a later row +// that withdraws the date ("To be announced"). A replacement that is +// itself retired is followed to the model that replaces it, so a hint never +// points at a model that no longer works. +func Parse(page []byte, keep func(string) bool) ([]Retirement, error) { + lines := strings.Split(strings.ReplaceAll(string(page), "\r\n", "\n"), "\n") + var out []Retirement + seen := map[string]bool{} + heading := "" + for i := 0; i < len(lines); i++ { + line := strings.TrimSpace(lines[i]) + if strings.HasPrefix(line, "#") { + heading = line + continue + } + header := cells(line) + if len(header) < 3 || !dateHeader.MatchString(header[0]) { + continue + } + model, repl := column(header, "model"), column(header, "replacement", "substitute") + if model < 0 || repl < 0 { + continue // an endpoint ("System") or a training-price table: no model to replace + } + i++ // the |---| row + for i+1 < len(lines) && strings.HasPrefix(strings.TrimSpace(lines[i+1]), "|") { + i++ + row := strings.TrimSpace(lines[i]) + c := cells(row) + if len(c) <= model || len(c) <= repl { + return nil, fmt.Errorf("a short row under %q: %s", heading, row) + } + names := ticked.FindAllStringSubmatch(c[model], -1) + date, err := parseDate(c[0]) + if err != nil { + // "To be announced": not a retirement yet, and the newer word on + // these names, so an older row's date for them doesn't count. + for _, m := range names { + seen[m[1]] = true + } + continue + } + excerpt := strings.Join([]string{heading, strings.TrimSpace(lines[headerLine(lines, i)]), row}, "\n") + for _, m := range names { + name := m[1] + if seen[name] || !keep(name) { + continue + } + seen[name] = true + out = append(out, Retirement{Model: name, Date: date, Replacement: c[repl], Excerpt: excerpt}) + } + } + } + if len(out) == 0 { + return nil, fmt.Errorf("no retirement tables found: the page's shape may have changed") + } + follow(out) + return out, nil +} + +// headerLine walks back from a row to its table's header. +func headerLine(lines []string, i int) int { + for i > 0 && strings.HasPrefix(strings.TrimSpace(lines[i-1]), "|") { + i-- + } + return i +} + +// follow replaces a replacement that is itself retired with its own +// replacement, recording the hops: gpt-3.5-turbo-0613's named replacement is +// gpt-3.5-turbo, which shuts down on 2026-10-23 for gpt-5.6-terra. +func follow(rs []Retirement) { + by := map[string]Retirement{} + for _, r := range rs { + by[r.Model] = r + } + for i := range rs { + for hops := 0; hops < 10; hops++ { + next, ok := by[First(rs[i].Replacement)] + if !ok || next.Model == rs[i].Model { + break + } + rs[i].Via = append(rs[i].Via, next.Model) + rs[i].Replacement = next.Replacement + } + } +} + +// First is the first model name a replacement cell gives, "" when it names +// none ("The most capable cyber model available to you."). +func First(replacement string) string { + if m := ticked.FindStringSubmatch(replacement); m != nil { + return m[1] + } + return "" +} + +// cells splits a markdown table row on its unescaped pipes. A cell can hold +// an escaped one: "`gpt-4-0613` \| `gpt-4`". +func cells(row string) []string { + if !strings.HasPrefix(row, "|") { + return nil + } + row = strings.Trim(row, "|") + var out []string + var cur strings.Builder + for i := 0; i < len(row); i++ { + if row[i] == '\\' && i+1 < len(row) && row[i+1] == '|' { + cur.WriteString(`\|`) + i++ + continue + } + if row[i] == '|' { + out = append(out, strings.TrimSpace(cur.String())) + cur.Reset() + continue + } + cur.WriteByte(row[i]) + } + return append(out, strings.TrimSpace(cur.String())) +} + +func column(header []string, words ...string) int { + for i, h := range header { + for _, w := range words { + if strings.Contains(strings.ToLower(h), w) { + return i + } + } + } + return -1 +} + +func parseDate(s string) (string, error) { + for _, layout := range dateLayouts { + if t, err := time.Parse(layout, strings.TrimSpace(s)); err == nil { + return t.Format("2006-01-02"), nil + } + } + return "", fmt.Errorf("not a date: %q", s) +} + +// Events turns retirements into change events, one per model name: the +// `model` parameter can no longer take that name, in any of the provider's +// calls. affected.operation is the provider's main one; the patch-it client +// looks for the name wherever it reaches the provider's SDK, so one event per +// name gives one fix per repo, not one per operation for the same line. +// version is the shutdown date, so a provider moving the date is a new event. +// sourceURL is the snapshot the retirements were read from. +func Events(p Page, rs []Retirement, sourceURL string, now time.Time) ([]event.ChangeEvent, []error) { + var events []event.ChangeEvent + var errs []error + for _, r := range rs { + affected := event.Affected{ + Kind: event.AffectedKindRequestParameter, + Side: event.SideRequest, + Operation: p.Operation, + Property: "model", + Value: r.Model, + } + e := event.ChangeEvent{ + Fingerprint: event.Fingerprint(p.API, r.Date, event.ChangeTypeBreaking, affected, r.Model), + API: p.API, + Version: r.Date, + ChangeType: event.ChangeTypeBreaking, + Breaking: true, + Affected: affected, + Description: fmt.Sprintf("Model %q is shut down on %s: %s requests naming it fail from that date.", r.Model, r.Date, p.Vendor), + MigrationHint: hint(p.Vendor, r), + SourceURL: sourceURL, + SourceExcerpt: r.Excerpt, + Confidence: 1.0, + DetectedAt: now, + } + if err := e.Validate(); err != nil { + errs = append(errs, fmt.Errorf("%s %s: %w", p.API, r.Model, err)) + continue + } + events = append(events, e) + } + return events, errs +} + +func hint(vendor string, r Retirement) string { + repl := strings.TrimSpace(r.Replacement) + if First(repl) == "" { + return fmt.Sprintf("%s names no replacement model: %s", vendor, repl) + } + h := fmt.Sprintf("Use %s instead: %s's recommended replacement.", repl, vendor) + if len(r.Via) > 0 { + h += fmt.Sprintf(" (%s first named %s, which is itself retired.)", vendor, strings.Join(r.Via, ", then ")) + } + return h +} diff --git a/retire/retire_test.go b/plugins/retirements/retire/retire_test.go similarity index 99% rename from retire/retire_test.go rename to plugins/retirements/retire/retire_test.go index 4860956..02f3c20 100644 --- a/retire/retire_test.go +++ b/plugins/retirements/retire/retire_test.go @@ -6,7 +6,7 @@ import ( "testing" "time" - "github.com/patch-it-dev/patch-it-plugins/event" + "github.com/patch-it-dev/patch-it-plugins/core/event" ) func read(t *testing.T, name string) []byte { diff --git a/retire/testdata/README.md b/plugins/retirements/retire/testdata/README.md similarity index 100% rename from retire/testdata/README.md rename to plugins/retirements/retire/testdata/README.md diff --git a/retire/testdata/anthropic-deprecations.md b/plugins/retirements/retire/testdata/anthropic-deprecations.md similarity index 100% rename from retire/testdata/anthropic-deprecations.md rename to plugins/retirements/retire/testdata/anthropic-deprecations.md diff --git a/retire/testdata/openai-deprecations.md b/plugins/retirements/retire/testdata/openai-deprecations.md similarity index 100% rename from retire/testdata/openai-deprecations.md rename to plugins/retirements/retire/testdata/openai-deprecations.md diff --git a/plugins/retirements/retirements.go b/plugins/retirements/retirements.go index 6f77f98..d4d200a 100644 --- a/plugins/retirements/retirements.go +++ b/plugins/retirements/retirements.go @@ -9,15 +9,31 @@ import ( "fmt" "log" - "github.com/patch-it-dev/patch-it-plugins/kit" - "github.com/patch-it-dev/patch-it-plugins/migrate" - "github.com/patch-it-dev/patch-it-plugins/plugin" - "github.com/patch-it-dev/patch-it-plugins/retire" + "github.com/patch-it-dev/patch-it-plugins/core/kit" + "github.com/patch-it-dev/patch-it-plugins/core/migrate" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" + "github.com/patch-it-dev/patch-it-plugins/plugins/retirements/retire" ) func init() { - plugin.Register(Plugin{Page: retire.OpenAI, Archiver: kit.Wayback{}}) - plugin.Register(Plugin{Page: retire.Anthropic, Archiver: kit.Wayback{}}) + plugin.RegisterProvider(plugin.Provider{ + ID: "openai", Name: "OpenAI", Category: plugin.CategoryAI, Homepage: "https://openai.com", + Sources: []plugin.Source{{ + ID: "models", Layer: plugin.LayerAIModels, Page: "providers/openai-models.md", + Watches: "Model deprecations and shutdown dates (Chat Completions and Responses text models)", + URLs: []string{"https://developers.openai.com/api/docs/deprecations"}, + Plugin: Plugin{Page: retire.OpenAI, Archiver: kit.Wayback{}}, + }}, + }) + plugin.RegisterProvider(plugin.Provider{ + ID: "anthropic", Name: "Anthropic", Category: plugin.CategoryAI, Homepage: "https://www.anthropic.com", + Sources: []plugin.Source{{ + ID: "models", Layer: plugin.LayerAIModels, Page: "providers/anthropic-models.md", + Watches: "Model deprecations and retirement dates (Messages API models)", + URLs: []string{"https://platform.claude.com/docs/en/about-claude/model-deprecations"}, + Plugin: Plugin{Page: retire.Anthropic, Archiver: kit.Wayback{}}, + }}, + }) } // Plugin reads one provider's page. diff --git a/plugins/retirements/retirements_test.go b/plugins/retirements/retirements_test.go index 0f3e9d8..fe89d4d 100644 --- a/plugins/retirements/retirements_test.go +++ b/plugins/retirements/retirements_test.go @@ -7,9 +7,9 @@ import ( "testing" "time" - "github.com/patch-it-dev/patch-it-plugins/kit/kittest" - "github.com/patch-it-dev/patch-it-plugins/plugin" - "github.com/patch-it-dev/patch-it-plugins/retire" + "github.com/patch-it-dev/patch-it-plugins/core/kit/kittest" + "github.com/patch-it-dev/patch-it-plugins/core/plugin" + "github.com/patch-it-dev/patch-it-plugins/plugins/retirements/retire" ) const deprecations = `### 2026-04-22: Legacy GPT model snapshots diff --git a/providers/README.md b/providers/README.md index d345a58..550b4fa 100644 --- a/providers/README.md +++ b/providers/README.md @@ -8,35 +8,4 @@ One page per plugin id: `providers/.md`. Start from [`_template.md`](_templ - **What the plugin reads** from it, and what it deliberately ignores (and why). - **Known gaps**: changes the provider makes that this plugin can't see yet. -| Page | Plugin | Format plugin | -|---|---|---| -| [stripe.md](stripe.md) | `stripe` | `openapi` | -| [datadog.md](datadog.md) | `datadog` | `openapi` | -| [github.md](github.md) | `github` | `openapi` | -| [gitlab.md](gitlab.md) | `gitlab` | `openapi` | -| [vercel.md](vercel.md) | `vercel` | `openapi` | -| [supabase.md](supabase.md) | `supabase` | `openapi` | -| [algolia.md](algolia.md) | `algolia-search`, `algolia-ingestion`, `algolia-recommend` | `openapi` | -| [elasticsearch.md](elasticsearch.md) | `elasticsearch` | `openapi` | -| [sentry.md](sentry.md) | `sentry` | `openapi` | -| [neon.md](neon.md) | `neon` | `openapi` | -| [convex.md](convex.md) | `convex` | `openapi` | -| [planetscale.md](planetscale.md) | `planetscale` | `openapi` | -| [appwrite.md](appwrite.md) | `appwrite` | `openapi` | -| [openai-models.md](openai-models.md) | `openai-models` | `retirements` | -| [anthropic-models.md](anthropic-models.md) | `anthropic-models` | `retirements` | -| [anthropic.md](anthropic.md) | `anthropic` | `migrationguide` | -| [numpy.md](numpy.md) | `numpy` | `removedmembers` | -| [posthog.md](posthog.md) | `posthog` | `removedmembers` | -| [newrelic.md](newrelic.md) | `newrelic` | `removedmembers` | -| [textual.md](textual.md) | `textual` | `changelog` | -| [urllib3.md](urllib3.md) | `urllib3` | `changelog` | -| [github-actions.md](github-actions.md) | `github-actions` | `feed` | -| [vertex-ai-models.md](vertex-ai-models.md) | `vertex-ai-models` | `feed` | -| [redis.md](redis.md) | `redis` | `catalog` | -| [cloudflare.md](cloudflare.md) | `cloudflare` | `feed` | -| [mongodb-atlas.md](mongodb-atlas.md) | `mongodb-atlas` | `apichangelog` | -| [salesforce.md](salesforce.md) | `salesforce` | `eol` | -| [flagsmith.md](flagsmith.md) | `flagsmith` | `openapi` | - -The list of plugins, with their status, is [docs/plugins.md](../docs/plugins.md). +Each source's registration names its page (`Page: "providers/.md"`); a provider's sources may share one. There's no list of pages here: `go run ./cmd/list-plugins` prints every provider and source with its page, and each release carries that list as `plugins.json`. diff --git a/providers/anthropic-models.md b/providers/anthropic-models.md index 8249bb9..77b50db 100644 --- a/providers/anthropic-models.md +++ b/providers/anthropic-models.md @@ -4,7 +4,7 @@ Claude models that are retired or have a retirement date on the Claude API, with the replacement Anthropic recommends. One breaking event per model name. -Format plugin: `retirements` (`plugins/retirements`), reading with `retire/`. +Format plugin: `retirements` (`plugins/retirements`), reading with `plugins/retirements/retire/`. ## Why this source diff --git a/providers/anthropic.md b/providers/anthropic.md index 131031f..cac9358 100644 --- a/providers/anthropic.md +++ b/providers/anthropic.md @@ -4,11 +4,11 @@ Request parameters removed from `messages.create()` and its siblings in anthropic-sdk-python 1.0.0 (`temperature`, `top_p`, `top_k`): passing one is a `TypeError`. One breaking event per parameter. Pinned to that release. -Format plugin: `migrationguide` (`plugins/migrationguide`), reading with `sdkguide/`. +Format plugin: `migrationguide` (`plugins/migrationguide`), reading with `plugins/migrationguide/sdkguide/`. ## Why this source -- **Source:** [MIGRATION.md at v1.0.0](https://github.com/anthropics/anthropic-sdk-python/blob/v1.0.0/MIGRATION.md), read raw at the tag (`https://raw.githubusercontent.com/anthropics/anthropic-sdk-python/v1.0.0/MIGRATION.md`). Checked on 2026-10-17. +- **Source:** MIGRATION.md at the first release of each major ([at v1.0.0](https://github.com/anthropics/anthropic-sdk-python/blob/v1.0.0/MIGRATION.md)), the majors listed from PyPI; read raw at the tag (`https://raw.githubusercontent.com/anthropics/anthropic-sdk-python/v1.0.0/MIGRATION.md`). Checked on 2026-10-17. - **Why this one:** it's the SDK maintainers' own guide to the major release, with a table of the methods, the removed parameter and what to use instead. A tag never changes, so the URL is its own permanent citation. - **Alternatives considered:** the GitHub release notes and `CHANGELOG.md` (list commits, not "what to use instead"); diffing the SDK's code between versions (finds signature changes but not the maintainers' advice, and is a lot more machinery); Anthropic's model deprecations page (mentions the change, but it's the models page). - **Quote:** "Removed: deprecated request parameters" (section heading), with the row "`temperature`, `top_p`, `top_k` | Remove them." diff --git a/providers/flagsmith.md b/providers/flagsmith.md index ac87987..40ff7ae 100644 --- a/providers/flagsmith.md +++ b/providers/flagsmith.md @@ -26,7 +26,7 @@ Format plugin: `openapi` (`plugins/openapi`), with `adapter.Flagsmith` and a req ## What the plugin reads and ignores -- **Reads** whole operations removed between two tags, and the request-side changes `diff/` already tracks for every `openapi`-format provider (removed or newly-required parameters and request fields). +- **Reads** whole operations removed between two tags, and the request-side changes `core/diff/` already tracks for every `openapi`-format provider (removed or newly-required parameters and request fields). - **Drops response-side changes**, same reason as GitLab (`requestSide`): a request body's schema is a named, shared component (`UpdateFlagV2`, `FeatureValue`), not inlined like GitLab's hash-named bodies, so the differ reports its removal as its own "schema removed" change, side `response` by default, whether or not the operations that used it are actually gone. Of the 11 changes between `v2.286.0` and `v2.287.0`, 8 are these schema-level echoes of the 3 real operation removals; the filter drops them and keeps the 3 operations. - The replay fixture: `v2.286.0` to `v2.287.0` (2026-10-15 to 2026-10-20), where PR [#8736](https://github.com/Flagsmith/flagsmith/pull/8736) ("Remove the experimental flag update endpoints") removed `POST /api/experiments/environments/{environment_key}/delete-segment-override/`, `.../update-flag-v1/` and `.../update-flag-v2/` and their eight request-body schemas, listed under "Features" in the [v2.287.0 release](https://github.com/Flagsmith/flagsmith/releases/tag/v2.287.0). All three were marked experimental from the day they shipped; `/api/__future__/...` is where Flagsmith puts their eventual, stable replacements. diff --git a/providers/openai-models.md b/providers/openai-models.md index 586828e..ae73355 100644 --- a/providers/openai-models.md +++ b/providers/openai-models.md @@ -4,7 +4,7 @@ OpenAI models that are being shut down: every text model name code sends to the Chat Completions or Responses API, with its shutdown date and the replacement OpenAI recommends. One breaking event per model name. -Format plugin: `retirements` (`plugins/retirements`), reading with `retire/`. +Format plugin: `retirements` (`plugins/retirements`), reading with `plugins/retirements/retire/`. ## Why this source diff --git a/providers/stripe.md b/providers/stripe.md index 582214c..6c94bd1 100644 --- a/providers/stripe.md +++ b/providers/stripe.md @@ -22,7 +22,7 @@ Format plugin: `openapi` (`plugins/openapi`), with `adapter.Stripe` and the chan ## What the plugin reads and ignores -- Reads: the whole spec at the newest tag, diffed against the last one it saw (`diff/`, `normalize/`). `$ref`s are never followed: a change inside a shared schema is reported once, on that schema, not once per endpoint that uses it. +- Reads: the whole spec at the newest tag, diffed against the last one it saw (`core/diff/`, `core/normalize/`). `$ref`s are never followed: a change inside a shared schema is reported once, on that schema, not once per endpoint that uses it. - Hints: for each breaking event, the changelog entry for that version whose "REST API" table names the same parameter and operation; the entry's "What's new" text and link become `migration_hint`. An entry describing an addition never explains a removal, and the reverse. - Replays a historical pair (`Replay`, from `v207` to `v208`) and recreates the 2022-11-15 change (38 events, 36 breaking). - `latest/openapi.spec3.json` can lag a version bump by several tags; the plugin follows the file, not the release notes. diff --git a/providers/textual.md b/providers/textual.md index dc06f77..4b13089 100644 --- a/providers/textual.md +++ b/providers/textual.md @@ -2,7 +2,7 @@ ## What it watches -Removals and breaking changes in Textual 0.86.0, read from its changelog: symbols removed (`App.dark`, …) or renamed. One `package_release` event per changelog bullet the plugin can read with certainty. Pinned to that release. +Removals and breaking changes in each Textual release, read from that release's section of its changelog: symbols removed (`App.dark`, …) or renamed. One `package_release` event per changelog bullet the plugin can read with certainty. The releases are listed from PyPI (final releases only) and each is read at its tag (`v`); 0.86.0 is the release the golden test pins. Format plugin: `changelog` (`plugins/changelog`). @@ -26,5 +26,5 @@ Format plugin: `changelog` (`plugins/changelog`). ## Known gaps -- Only 0.86.0. Following each new release is planned. +- A release is read at its tag; one whose tag or changelog section is missing fails that pair, it isn't guessed. - Breaking behavior changes written as prose aren't events. diff --git a/providers/urllib3.md b/providers/urllib3.md index c962ac1..df1088c 100644 --- a/providers/urllib3.md +++ b/providers/urllib3.md @@ -2,7 +2,7 @@ ## What it watches -Removals and deprecations in urllib3 2.0.0, read from its changelog: classes, methods and modules removed or renamed. One `package_release` event per bullet the plugin can read with certainty. Pinned to that release. +Removals and deprecations in each urllib3 release, read from that release's section of its changelog: classes, methods and modules removed or renamed. One `package_release` event per bullet the plugin can read with certainty. The releases are listed from PyPI (final releases only) and each is read at its git tag (``); 2.0.0 is the release the golden test pins. Format plugin: `changelog` (`plugins/changelog`). @@ -26,5 +26,5 @@ Format plugin: `changelog` (`plugins/changelog`). ## Known gaps -- Only 2.0.0. +- A release is read at its tag; one whose tag or changelog section is missing fails that pair, it isn't guessed. - Changed defaults (TLS versions, retries) are behavior changes and aren't events. diff --git a/retire/retire.go b/retire/retire.go index 1cb3ab8..e519a3e 100644 --- a/retire/retire.go +++ b/retire/retire.go @@ -1,245 +1,23 @@ -// Package retire reads a model provider's deprecations page and turns every -// retired model into change events: a second kind of source next to the -// OpenAPI differ. There is no spec to diff, only the provider's own table of -// shutdown date, model and named replacement, so this reads the table. +// Package retire is the old import path of github.com/patch-it-dev/patch-it-plugins/plugins/retirements/retire. // -// Both providers serve the page as markdown at its URL plus ".md" (checked -// 2026-10-13), and both write each announcement as a table whose first column -// is the date the model stops working. The event cites a Wayback Machine -// snapshot of that markdown (kit/), never the live page: the contract -// requires a source that can't change under the event. +// Deprecated: import github.com/patch-it-dev/patch-it-plugins/plugins/retirements/retire +// instead. This package only forwards to it, so code written against the old +// path keeps building; it will be removed in a later release. package retire -import ( - "fmt" - "regexp" - "strings" - "time" +import moved "github.com/patch-it-dev/patch-it-plugins/plugins/retirements/retire" - "github.com/patch-it-dev/patch-it-plugins/event" +// Deprecated: the types live in github.com/patch-it-dev/patch-it-plugins/plugins/retirements/retire. +type ( + Page = moved.Page + Retirement = moved.Retirement ) -// Page is one provider's deprecations page and what its models are sent to. -type Page struct { - API string // the api slug the events carry, e.g. "openai-models" - Vendor string // for sentences: "OpenAI" - URL string // the page as markdown - Operation string // the provider's main call that takes a model name - // Keep says whether a model name belongs to this source: the operations - // above take text models, not image, audio or embedding ones. - Keep func(name string) bool -} - -// Retirement is one model name the provider stops serving on Date, with the -// replacement it names. An alias gets its own Retirement, with the row it -// came from. -type Retirement struct { - Model string - Date string // YYYY-MM-DD - Replacement string // the replacement cell, as written ("`gpt-5.6-sol` (`reasoning.mode: pro`)") - Via []string - Excerpt string // the section heading, the table header and the row -} - +// Deprecated: the functions and values live in github.com/patch-it-dev/patch-it-plugins/plugins/retirements/retire. var ( - dateHeader = regexp.MustCompile(`(?i)^(shutdown|retirement) date$`) - ticked = regexp.MustCompile("`([^`]+)`") + Events = moved.Events + First = moved.First + Parse = moved.Parse + Anthropic = moved.Anthropic + OpenAI = moved.OpenAI ) - -var dateLayouts = []string{"January 2, 2006", "Jan 2, 2006", "2006-01-02"} - -// Parse reads every retirement table on the page. A model listed more than -// once keeps its first row: both pages list the most recent announcement -// first, and a later one supersedes an earlier date, including a later row -// that withdraws the date ("To be announced"). A replacement that is -// itself retired is followed to the model that replaces it, so a hint never -// points at a model that no longer works. -func Parse(page []byte, keep func(string) bool) ([]Retirement, error) { - lines := strings.Split(strings.ReplaceAll(string(page), "\r\n", "\n"), "\n") - var out []Retirement - seen := map[string]bool{} - heading := "" - for i := 0; i < len(lines); i++ { - line := strings.TrimSpace(lines[i]) - if strings.HasPrefix(line, "#") { - heading = line - continue - } - header := cells(line) - if len(header) < 3 || !dateHeader.MatchString(header[0]) { - continue - } - model, repl := column(header, "model"), column(header, "replacement", "substitute") - if model < 0 || repl < 0 { - continue // an endpoint ("System") or a training-price table: no model to replace - } - i++ // the |---| row - for i+1 < len(lines) && strings.HasPrefix(strings.TrimSpace(lines[i+1]), "|") { - i++ - row := strings.TrimSpace(lines[i]) - c := cells(row) - if len(c) <= model || len(c) <= repl { - return nil, fmt.Errorf("a short row under %q: %s", heading, row) - } - names := ticked.FindAllStringSubmatch(c[model], -1) - date, err := parseDate(c[0]) - if err != nil { - // "To be announced": not a retirement yet, and the newer word on - // these names, so an older row's date for them doesn't count. - for _, m := range names { - seen[m[1]] = true - } - continue - } - excerpt := strings.Join([]string{heading, strings.TrimSpace(lines[headerLine(lines, i)]), row}, "\n") - for _, m := range names { - name := m[1] - if seen[name] || !keep(name) { - continue - } - seen[name] = true - out = append(out, Retirement{Model: name, Date: date, Replacement: c[repl], Excerpt: excerpt}) - } - } - } - if len(out) == 0 { - return nil, fmt.Errorf("no retirement tables found: the page's shape may have changed") - } - follow(out) - return out, nil -} - -// headerLine walks back from a row to its table's header. -func headerLine(lines []string, i int) int { - for i > 0 && strings.HasPrefix(strings.TrimSpace(lines[i-1]), "|") { - i-- - } - return i -} - -// follow replaces a replacement that is itself retired with its own -// replacement, recording the hops: gpt-3.5-turbo-0613's named replacement is -// gpt-3.5-turbo, which shuts down on 2026-10-23 for gpt-5.6-terra. -func follow(rs []Retirement) { - by := map[string]Retirement{} - for _, r := range rs { - by[r.Model] = r - } - for i := range rs { - for hops := 0; hops < 10; hops++ { - next, ok := by[First(rs[i].Replacement)] - if !ok || next.Model == rs[i].Model { - break - } - rs[i].Via = append(rs[i].Via, next.Model) - rs[i].Replacement = next.Replacement - } - } -} - -// First is the first model name a replacement cell gives, "" when it names -// none ("The most capable cyber model available to you."). -func First(replacement string) string { - if m := ticked.FindStringSubmatch(replacement); m != nil { - return m[1] - } - return "" -} - -// cells splits a markdown table row on its unescaped pipes. A cell can hold -// an escaped one: "`gpt-4-0613` \| `gpt-4`". -func cells(row string) []string { - if !strings.HasPrefix(row, "|") { - return nil - } - row = strings.Trim(row, "|") - var out []string - var cur strings.Builder - for i := 0; i < len(row); i++ { - if row[i] == '\\' && i+1 < len(row) && row[i+1] == '|' { - cur.WriteString(`\|`) - i++ - continue - } - if row[i] == '|' { - out = append(out, strings.TrimSpace(cur.String())) - cur.Reset() - continue - } - cur.WriteByte(row[i]) - } - return append(out, strings.TrimSpace(cur.String())) -} - -func column(header []string, words ...string) int { - for i, h := range header { - for _, w := range words { - if strings.Contains(strings.ToLower(h), w) { - return i - } - } - } - return -1 -} - -func parseDate(s string) (string, error) { - for _, layout := range dateLayouts { - if t, err := time.Parse(layout, strings.TrimSpace(s)); err == nil { - return t.Format("2006-01-02"), nil - } - } - return "", fmt.Errorf("not a date: %q", s) -} - -// Events turns retirements into change events, one per model name: the -// `model` parameter can no longer take that name, in any of the provider's -// calls. affected.operation is the provider's main one; the patch-it client -// looks for the name wherever it reaches the provider's SDK, so one event per -// name gives one fix per repo, not one per operation for the same line. -// version is the shutdown date, so a provider moving the date is a new event. -// sourceURL is the snapshot the retirements were read from. -func Events(p Page, rs []Retirement, sourceURL string, now time.Time) ([]event.ChangeEvent, []error) { - var events []event.ChangeEvent - var errs []error - for _, r := range rs { - affected := event.Affected{ - Kind: event.AffectedKindRequestParameter, - Side: event.SideRequest, - Operation: p.Operation, - Property: "model", - Value: r.Model, - } - e := event.ChangeEvent{ - Fingerprint: event.Fingerprint(p.API, r.Date, event.ChangeTypeBreaking, affected, r.Model), - API: p.API, - Version: r.Date, - ChangeType: event.ChangeTypeBreaking, - Breaking: true, - Affected: affected, - Description: fmt.Sprintf("Model %q is shut down on %s: %s requests naming it fail from that date.", r.Model, r.Date, p.Vendor), - MigrationHint: hint(p.Vendor, r), - SourceURL: sourceURL, - SourceExcerpt: r.Excerpt, - Confidence: 1.0, - DetectedAt: now, - } - if err := e.Validate(); err != nil { - errs = append(errs, fmt.Errorf("%s %s: %w", p.API, r.Model, err)) - continue - } - events = append(events, e) - } - return events, errs -} - -func hint(vendor string, r Retirement) string { - repl := strings.TrimSpace(r.Replacement) - if First(repl) == "" { - return fmt.Sprintf("%s names no replacement model: %s", vendor, repl) - } - h := fmt.Sprintf("Use %s instead: %s's recommended replacement.", repl, vendor) - if len(r.Via) > 0 { - h += fmt.Sprintf(" (%s first named %s, which is itself retired.)", vendor, strings.Join(r.Via, ", then ")) - } - return h -} diff --git a/scripts/check_contract.py b/scripts/check_contract.py index d898c6d..5bbc0fd 100644 --- a/scripts/check_contract.py +++ b/scripts/check_contract.py @@ -2,8 +2,8 @@ """Check contract/fixtures/ against patch-it's fixtures/. patch-it (https://github.com/patch-it-dev/patch-it) owns the change-event -contract; this repo carries a copy of its change-event fixtures so event/ and -migrate/ test against the same events the patch-it client does. The copies +contract; this repo carries a copy of its change-event fixtures so core/event/ and +core/migrate/ test against the same events the patch-it client does. The copies must hold the same JSON as patch-it's. Only the top-level "_comment" fields may differ: they describe where a fixture came from, and this repo words them for a public audience. diff --git a/sdkguide/sdkguide.go b/sdkguide/sdkguide.go index 1653501..7b1a75f 100644 --- a/sdkguide/sdkguide.go +++ b/sdkguide/sdkguide.go @@ -1,174 +1,21 @@ -// Package sdkguide reads an SDK's own migration guide for the request parameters -// a major release removed: a third kind of source, next to the OpenAPI differ and -// the deprecations pages. The guide is read at the release tag of the major it -// describes (`v1.0.0` for "Migrating to v1"), a URL that never changes, so it is -// its own provenance. +// Package sdkguide is the old import path of github.com/patch-it-dev/patch-it-plugins/plugins/migrationguide/sdkguide. // -// The first is anthropic-sdk-python 1.0.0 (2026-08-20): its MIGRATION.md lists -// `temperature`, `top_p` and `top_k` as removed from `messages.create()` and its -// siblings, passing them is a `TypeError`, and it says what to do instead (checked -// 2026-10-17). A repo with `anthropic` unpinned gets 1.x on a fresh install. +// Deprecated: import github.com/patch-it-dev/patch-it-plugins/plugins/migrationguide/sdkguide +// instead. This package only forwards to it, so code written against the old +// path keeps building; it will be removed in a later release. package sdkguide -import ( - "fmt" - "regexp" - "strings" - "time" - "unicode/utf8" +import moved "github.com/patch-it-dev/patch-it-plugins/plugins/migrationguide/sdkguide" - "github.com/patch-it-dev/patch-it-plugins/event" +// Deprecated: the types live in github.com/patch-it-dev/patch-it-plugins/plugins/migrationguide/sdkguide. +type ( + Guide = moved.Guide + Removal = moved.Removal ) -// Guide is one SDK's migration guide for one major release. -type Guide struct { - API string // the SDK's slug in change events, e.g. "anthropic" - Repo string // its GitHub repository, owner/name - Major int // the release the guide describes: 1 for "Migrating to v1" - Operation string // the API operation the removed parameters were sent to - // Methods is what a table row's method cell must name for its parameters to - // count: the plain call, so a row about beta-only methods (where the same - // name can still be valid on a helper) isn't read as a removal. - Methods string - // Symbols are the calls the removed parameters can no longer be passed to, as - // code writes them after the import: the plain methods the guide's table names - // first (`messages.create()`, `messages.stream()`, `messages.parse()`). A v2 - // event carries them (the change event contract). - Symbols []string -} - -// AnthropicV1 is anthropic-sdk-python's guide to 1.0.0. -var AnthropicV1 = Guide{ - API: "anthropic", - Repo: "anthropics/anthropic-sdk-python", - Major: 1, - Operation: "POST /v1/messages", - Methods: "`messages.create()`", - Symbols: []string{"messages.create", "messages.stream", "messages.parse"}, -} - -// Version is the release that removed the parameters. -func (g Guide) Version() string { return fmt.Sprintf("%d.0.0", g.Major) } - -// URL is the guide as it was at that release's tag. -func (g Guide) URL() string { - return fmt.Sprintf("https://raw.githubusercontent.com/%s/v%s/MIGRATION.md", g.Repo, g.Version()) -} - -// Page is the same guide for a person to read. -func (g Guide) Page() string { - return fmt.Sprintf("https://github.com/%s/blob/v%s/MIGRATION.md", g.Repo, g.Version()) -} - -// Removal is one parameter the release removed, with what the guide says to do. -type Removal struct { - Param string - Instead string // the guide's "use instead" cell, as written - Excerpt string // the section heading, the table header and the row -} - -var ticked = regexp.MustCompile("`([^`]+)`") - -// Parse reads the guide's removed-parameter tables: every row whose method cell -// names g.Methods gives one Removal per backticked parameter. A guide for another -// major, or one with no such row, is an error, never zero removals. -func Parse(page []byte, g Guide) ([]Removal, error) { - text := strings.ReplaceAll(string(page), "\r\n", "\n") - if !strings.Contains(text, fmt.Sprintf("# Migrating to v%d", g.Major)) { - return nil, fmt.Errorf("not the guide to v%d", g.Major) - } - lines := strings.Split(text, "\n") - var out []Removal - heading := "" - for i := 0; i < len(lines); i++ { - line := strings.TrimSpace(lines[i]) - if strings.HasPrefix(line, "#") { - heading = line - continue - } - header := cells(line) - param, instead := column(header, "removed parameter"), column(header, "instead") - if param < 0 || instead < 0 { - continue - } - i++ // the |---| row - for i+1 < len(lines) && strings.HasPrefix(strings.TrimSpace(lines[i+1]), "|") { - i++ - row := strings.TrimSpace(lines[i]) - c := cells(row) - if len(c) <= param || len(c) <= instead || !strings.Contains(c[0], g.Methods) { - continue - } - excerpt := cut(strings.Join([]string{heading, line, row}, "\n"), 2000) - for _, m := range ticked.FindAllStringSubmatch(c[param], -1) { - out = append(out, Removal{Param: m[1], Instead: c[instead], Excerpt: excerpt}) - } - } - } - if len(out) == 0 { - return nil, fmt.Errorf("no removed-parameter row naming %s: the guide's shape may have changed", g.Methods) - } - return out, nil -} - -func cells(row string) []string { - if !strings.HasPrefix(row, "|") { - return nil - } - parts := strings.Split(strings.Trim(row, "|"), "|") - for i := range parts { - parts[i] = strings.TrimSpace(parts[i]) - } - return parts -} - -func column(header []string, word string) int { - for i, h := range header { - if strings.Contains(strings.ToLower(h), word) { - return i - } - } - return -1 -} - -func cut(s string, n int) string { - if utf8.RuneCountInString(s) <= n { - return s - } - return string([]rune(s)[:n-1]) + "…" -} - -// Events turns removals into change events: one `breaking` request_parameter -// event per parameter, on the operation the SDK's methods send it to. -func Events(g Guide, rs []Removal, now time.Time) ([]event.ChangeEvent, []error) { - var events []event.ChangeEvent - var errs []error - for _, r := range rs { - affected := event.Affected{ - Kind: event.AffectedKindRequestParameter, - Side: event.SideRequest, - Operation: g.Operation, - Property: r.Param, - } - e := event.ChangeEvent{ - Fingerprint: event.Fingerprint(g.API, g.Version(), event.ChangeTypeBreaking, affected, "removed"), - API: g.API, - Version: g.Version(), - ChangeType: event.ChangeTypeBreaking, - Breaking: true, - Affected: affected, - Description: fmt.Sprintf("Parameter %s removed from %s and its sibling methods in the %s SDK %s: passing it raises a TypeError.", r.Param, g.Methods, g.API, g.Version()), - MigrationHint: fmt.Sprintf("%s Source: %s", r.Instead, g.Page()), - SourceURL: g.URL(), - SourceExcerpt: r.Excerpt, - Confidence: 1.0, - DetectedAt: now, - } - if err := e.Validate(); err != nil { - errs = append(errs, fmt.Errorf("%s %s: %w", g.API, r.Param, err)) - continue - } - events = append(events, e) - } - return events, errs -} +// Deprecated: the functions and values live in github.com/patch-it-dev/patch-it-plugins/plugins/migrationguide/sdkguide. +var ( + Events = moved.Events + Parse = moved.Parse + AnthropicV1 = moved.AnthropicV1 +)