Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
dde9f09
merge removedmembers into the migrationguide kind
adssib Nov 20, 2026
3dd09e7
merge apichangelog into the changelog kind as its json parser
adssib Nov 20, 2026
7ff927b
merge retirements, feed and eol into one notices kind
adssib Nov 20, 2026
d502df1
list every version: six more spec adapters and living pages by capture
adssib Nov 20, 2026
04190eb
registry: packages, first name and aliases per source
adssib Nov 20, 2026
ab212be
add the apisurface kind: a package's public API between releases
adssib Nov 20, 2026
3d0b90a
new-plugin scaffolds a provider's source on a kind; which-kind guide
adssib Nov 20, 2026
99a99ac
remove the top-level forwarding packages
adssib Nov 20, 2026
00cf574
apisurface: no rename on self alone, no removal from a skipped file o…
adssib Nov 20, 2026
4938576
Merge branch 'pl2-apisurface' into pl2-scaffold
adssib Nov 20, 2026
916ebda
Merge branch 'pl2-scaffold' into pl2-toplevel
adssib Nov 20, 2026
999aced
delete the merged kinds' forwarding packages and CanonicalKind
adssib Nov 20, 2026
a36eb22
one event type, sources named by their id, no replay, no legacy flags
adssib Nov 20, 2026
04813a8
docs: ids only, no forwarders, no v1, no replay
adssib Nov 20, 2026
f901430
apisurface: index every class, so a leaf class's new attribute is found
adssib Nov 21, 2026
11ea233
Merge branch 'pl2-apisurface' into pl2-scaffold
adssib Nov 21, 2026
75f9ea9
Merge branch 'pl2-scaffold' into pl2-toplevel
adssib Nov 21, 2026
d33aac1
no replay wording: tests, comments and provider pages say version pair
adssib Nov 21, 2026
ad2ecef
the last two replay comments
adssib Nov 21, 2026
ed940de
catalog test comment reads as a sentence
adssib Nov 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@

## Checklist

- [ ] **Metadata complete in the registration** (`plugin.RegisterProvider`: provider id, name, category, homepage; each source's id, layer, what it watches, URLs, page), and I read the words. There is no list to update. Not applicable: <!-- say why -->
- [ ] **Metadata complete in the registration** (`plugin.RegisterProvider`: provider id, name, category, homepage; each source's id, kind, layer, what it watches, URLs, packages, page), and I read the words. There is no list to update. Not applicable: <!-- say why -->
- [ ] Every commit is signed off (`git commit -s`, DCO)
- [ ] Golden test over a real saved copy of the source (`testdata/golden/<api>.json`)
- [ ] Golden test over a real saved copy of the source (`testdata/golden/<provider>-<source>.json`)
- [ ] `testdata/README.md` lists each file: URL, date fetched, license, quote of the part the plugin reads
- [ ] One pinned drop test: a realistic input that produces no event
- [ ] `providers/<api>.md` written or updated from `providers/_template.md`, including **Why this source**
- [ ] `providers/<provider>-<source>.md` written or updated from `providers/_template.md`, including **Why this source**
- [ ] Fetches only through `env.Get` / `core/kit/`
- [ ] `make check` passes locally (`make plugins` prints the source as you expect)
- [ ] Nothing private: no credentials, internal hosts, private links, customer names
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/CI_contract.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# contract/fixtures/ is a copy of patch-it's fixtures/ (the change-event contract
# patch-it owns). This fails when the two drift apart, so core/event/ and core/migrate/ test
# patch-it owns). This fails when the two drift apart, so core/event/ tests
# the same events the patch-it client does. Only top-level "_comment" fields may
# differ (scripts/check_contract.py).
#
Expand Down
26 changes: 13 additions & 13 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Rules for AI coding agents (and humans) working in this repo. `CLAUDE.md` import

## Rule 1: every new or changed provider or source has complete metadata in its registration

**A source exists when its kind's package registers it with `plugin.RegisterProvider`, and its metadata lives only there: the provider's id, name, category and homepage, and each source's id, layer, what it watches, the URLs it reads and its page, `providers/<name>.md`.** Adding, changing or removing a source is a change to that registration (and to its page) in the same PR. There is no list to update: nothing in this repo, the docs site or anywhere else keeps one by hand.
**A source exists when its kind's package registers it with `plugin.RegisterProvider`, and its metadata lives only there: the provider's id, name, category and homepage, and each source's id, layer, what it watches, the URLs it reads, the packages it covers (`pypi:stripe`) and its page, `providers/<name>.md`.** Adding, changing or removing a source is a change to that registration (and to its page) in the same PR. There is no list to update: nothing in this repo, the docs site or anywhere else keeps one by hand.

`go run ./cmd/list-plugins` prints the list from the registry (JSON), and the Release workflow attaches it to every release as `plugins.json`; the docs site and patch-it's hosted ingestion service read it from there. `RegisterProvider` refuses incomplete metadata and `go test` fails when a source's page is missing, but no check can tell whether the words are right: you are that check. Say in the PR that you read them.

Expand All @@ -23,12 +23,12 @@ Refer to the service that runs these plugins as "patch-it's hosted ingestion ser

## Rules for plugin code

- **Start a new provider with `go run ./cmd/new-plugin --api <id> --provider <Name> --format <format>`**, then fill in every `TODO(new-plugin)`; `make todos` (part of `make check` and CI_go) fails while one is left.
- **Start a new source with `make new-plugin ARGS='--provider <id> --name <Name> --source <id> --kind <kind>'`** (`go run ./cmd/new-plugin`; [which kind](docs/which-kind.md)), then fill in every `TODO(new-plugin)`; `make todos` (part of `make check` and CI_go) fails while one is left.
- **Fetch only through the environment** (`env.Get` or `core/kit/`). No `net/http` outside the fetch layer, no filesystem outside `testdata/`, no database.
- **Golden test + dated source quote**: a saved copy of the real source in `testdata/`, a row in `testdata/README.md` (URL, date fetched, license, quote), and `testdata/golden/<api>.json` with the exact events.
- **Golden test + dated source quote**: a saved copy of the real source in `testdata/`, a row in `testdata/README.md` (URL, date fetched, license, quote), and `testdata/golden/<provider>-<source>.json` with the exact events.
- **Precision over recall.** When unsure whether something is a change, don't emit an event.
- **One pinned drop test** per plugin: a realistic input that must produce no event.
- **`providers/<api>.md`** for every provider, copied from `providers/_template.md`, including **Why this source**: the link, the date it was checked, why it's the best source, the alternatives not used, and a short quote.
- **`providers/<provider>-<source>.md`** for every provider, copied from `providers/_template.md`, including **Why this source**: the link, the date it was checked, why it's the best source, the alternatives not used, and a short quote.
- **No source, no event**: every event carries the source URL and a quote.
- Don't edit golden files to make a test pass without explaining in the PR why the new output is right.

Expand All @@ -40,6 +40,7 @@ Refer to the service that runs these plugins as "patch-it's hosted ingestion ser
make check # tidy, build, gofmt, vet, the import allowlist, the net/http guard, go test -race, no TODO(new-plugin) left, leak guard
make check PATCH_IT=../patch-it # also CI_contract: contract/fixtures/ against a patch-it checkout
make plugins # print the list of providers and sources from the registry (what each release attaches as plugins.json)
make new-plugin ARGS='--provider acme --source api --kind openapi' # scaffold a source (asks for what's missing in a terminal)
```

One target at a time: `make tidy`, `make build`, `make fmt`, `make vet`, `make imports`, `make nethttp`, `make test`, `make todos`, `make leak-guard`, `make contract`. Use `make GO=/path/to/go check` for another Go.
Expand All @@ -52,22 +53,21 @@ Go module `github.com/patch-it-dev/patch-it-plugins`; the only dependency is `go
core/plugin/ the Plugin interface (Run; ListVersions, ReadAt, Extract), the registry of providers and sources, Env, Result
core/fetch/ the HTTP fetch with retries and the ETag / Last-Modified cache
core/github/ the newest release / commit on GitHub, and a commit's SHA and date (one place)
core/versions/ every version of a source: GitHub releases, tags and commits, PyPI, npm, the Web Archive at a date
core/kit/ core/kit/kittest/ Unchanged, Cite, Wayback; the test kit (Mem, Pages, Ingest, Golden)
core/event/ core/migrate/ the change-event types from patch-it's contract (v1 + v2), v1 -> v2
core/versions/ every version of a source: GitHub releases, tags and commits, PyPI, npm, Web Archive captures
core/kit/ core/kit/kittest/ Unchanged, Cite, Wayback, Notices; the test kit (Mem, Pages, Ingest, Golden)
core/extract/ what readers of prose share: RSS/Atom entries, HTML as text, dates, code spans
core/event/ the change event of patch-it's contract: types, fingerprint, validation
core/diff/ core/normalize/ core/hint/ the OpenAPI diff engine and its helpers
plugins/<kind>/ one package per kind, with its providers: openapi (+ adapter/, one spec adapter
per provider), retirements (+ retire/), migrationguide (+ sdkguide/),
removedmembers, changelog, feed, apichangelog, catalog, eol;
per provider), notices (+ retire/, feed/, eol/), migrationguide (+ sdkguide/,
members/), changelog (Markdown, RST or JSON), catalog, apisurface (+ pysurface/);
plugins/all imports every one
plugin/ event/ fetch/ migrate/ kit/ retire/ sdkguide/ deprecated old import paths: they only
forward to the packages above; don't add to them
cmd/list-plugins/ prints the list from the registry (make plugins); internal/pluginlist/ is its logic
cmd/new-plugin/ scaffolds a provider on a format plugin (go run ./cmd/new-plugin --help)
cmd/new-plugin/ scaffolds a provider's source on a kind (make new-plugin ARGS=...; go run ./cmd/new-plugin --help)
contract/fixtures/ a copy of patch-it's change-event fixtures (CI_contract checks it)
Makefile make check (what CI_go runs, plus the leak guard), make plugins
docs/ contributor docs, including writing-a-plugin.md
providers/<api>.md one page per provider
providers/<provider>-<source>.md one page per provider
scripts/ leak_guard.py (hashed internal names), check_contract.py
.github/ CODEOWNERS, PR template, release-notes categories, workflows
(CI_go, CI_dco, CI_leak_guard, CI_contract, Release)
Expand Down
12 changes: 6 additions & 6 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,16 @@

Thanks for helping patch-it watch more APIs. This file is the rulebook; [docs/writing-a-plugin.md](docs/writing-a-plugin.md) is the how-to.

**Start a new provider with the starter:** `go run ./cmd/new-plugin --api <id> --provider <Name> --format <format>` writes the definition, its registration, a test over placeholder sources, the testdata rows, the provider page, and the source's metadata in its `RegisterProvider` block; you fill in the `TODO(new-plugin)`s ([how](docs/writing-a-plugin.md#1-scaffold-it-with-new-plugin)).
**Start a new provider with the starter:** `make new-plugin ARGS='--provider <id> --source <id> --kind <kind>'` ([which kind?](docs/which-kind.md); in a terminal it asks for the rest) writes the definition, its registration, a test over placeholder sources, the testdata rows, the provider page, and the source's metadata in its `RegisterProvider` block; you fill in the `TODO(new-plugin)`s ([how](docs/writing-a-plugin.md#1-scaffold-it-with-new-plugin)).

## The rules

1. **Give every provider and source complete metadata in its registration.** A source exists when its kind's package registers it with `plugin.RegisterProvider`: the provider's id, name, category and homepage, and the source's id, layer, what it watches, the URLs it reads and its page. That registration is the only list: `make plugins` prints it, each release carries it as `plugins.json`, and docs.patch-it.dev reads it from there. Nothing else to update.
2. **Fetch only through the environment.** Plugins get their sources from `env.Get` or the helpers in `core/kit/`, never from `net/http` directly, the filesystem or a database. That keeps tests offline and lets the hosted service cache, retry and rate-limit politely.
3. **Every plugin has a golden test** over a real saved copy of its source: `testdata/<file>` in, `testdata/golden/<api>.json` out. `testdata/README.md` lists each file with its URL, the date you fetched it, its license, and a short quote of the part your plugin reads. Keep openly licensed files whole; trim vendor documentation pages to what the test needs.
3. **Every plugin has a golden test** over a real saved copy of its source: `testdata/<file>` in, `testdata/golden/<provider>-<source>.json` out. `testdata/README.md` lists each file with its URL, the date you fetched it, its license, and a short quote of the part your plugin reads. Keep openly licensed files whole; trim vendor documentation pages to what the test needs.
4. **Precision over recall.** A missed change is a bug we can fix later. A wrong change becomes a wrong pull request in someone's repo. When in doubt, don't emit.
5. **One pinned drop test per plugin**: a realistic input your plugin must *not* turn into an event (an unrelated section, a renamed heading, a non-breaking addition).
6. **Document the provider** in `providers/<api>.md`, copied from [`providers/_template.md`](providers/_template.md): what it watches, **why you chose that source** (and the alternatives you didn't), how the provider announces changes, what the plugin reads and ignores, and known gaps.
6. **Document the provider** in `providers/<provider>-<source>.md`, copied from [`providers/_template.md`](providers/_template.md): what it watches, **why you chose that source** (and the alternatives you didn't), how the provider announces changes, what the plugin reads and ignores, and known gaps.
7. **Nothing private.** This repo is public. No credentials, tokens, internal hostnames or links to private repos or systems, in code, tests, commit messages or PR descriptions. CI scans every PR for secrets and for known internal names.

## Sign your commits (DCO)
Expand Down Expand Up @@ -46,14 +46,14 @@ That adds `Signed-off-by: Your Name <[email protected]>`, matching the commit's au

Releases are tags on `main` (`v0.x.y`), made only by the **Release** workflow. Nothing is tagged on push, and agents never tag or release.

**Automatic, twice a week:** every Monday and Thursday at 14:00 UTC the workflow makes a **patch** release, but only if something was merged since the last tag; otherwise it does nothing. If the merges add a **new format plugin** (a new `plugins/<format>/` directory), the scheduled run skips with a warning instead: run it by hand with **minor**. So a merged plugin reaches patch-it's hosted ingestion service within a few days.
**Automatic, twice a week:** every Monday and Thursday at 14:00 UTC the workflow makes a **patch** release, but only if something was merged since the last tag; otherwise it does nothing. If the merges add a **new kind** (a new `plugins/<kind>/` directory), the scheduled run skips with a warning instead: run it by hand with **minor**. So a merged plugin reaches patch-it's hosted ingestion service within a few days.

**By hand, any time** (an urgent fix, or a **minor** release):

1. Merge the PRs that should go out.
2. **Actions → Release → Run workflow** (branch `main`), pick the bump:
- **patch** (`v0.4.1` → `v0.4.2`): fixes, a new provider on an existing format plugin, docs;
- **minor** (`v0.4.1` → `v0.5.0`): a new format plugin, new exported API, anything users should read about.
- **patch** (`v0.4.1` → `v0.4.2`): fixes, a new source on an existing kind, docs;
- **minor** (`v0.4.1` → `v0.5.0`): a new kind, new exported API, anything users should read about.
3. The workflow refuses if `main`'s HEAD is already tagged, then runs, on that exact commit, the internal-names check over every file, gitleaks over the whole history, `go build`, `go vet`, `gofmt` and `go test -race`. Any failure stops it before a tag exists.
4. When everything passes it pushes an annotated tag and publishes a GitHub release with generated notes (grouped by the PR's label: `plugin`/`enhancement`, `bug`, `documentation`; see [`.github/release.yml`](.github/release.yml)). The release carries one asset, `plugins.json`: the list of providers and sources printed from the registry at that tag (`go run ./cmd/list-plugins`). It is the only published list; nothing in the repo keeps one, and docs.patch-it.dev reads `https://github.com/patch-it-dev/patch-it-plugins/releases/latest/download/plugins.json`.

Expand Down
11 changes: 9 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
# make check everything below, in CI's order
# make check PATCH_IT=../patch-it also compare contract/fixtures/ with patch-it's
# make plugins print the providers and sources from the registry (JSON)
# make new-plugin ARGS='--provider acme --source api --kind openapi'
# scaffold a source (cmd/new-plugin; asks for what's missing)
#
# The `::error` lines are GitHub annotations; locally they're plain messages.

Expand All @@ -17,7 +19,7 @@ PYTHON ?= python3
MODULE := github.com/patch-it-dev/patch-it-plugins
ALLOWED := ^($(MODULE)(/|$$)|gopkg\.in/yaml\.v3$$)

.PHONY: check tidy build fmt vet imports nethttp test todos leak-guard contract plugins
.PHONY: check tidy build fmt vet imports nethttp test todos leak-guard contract plugins new-plugin

check: tidy build fmt vet imports nethttp test todos leak-guard contract
@echo "make check: all passed"
Expand Down Expand Up @@ -63,7 +65,7 @@ test:

# A provider started with new-plugin is done when nothing it wrote still says
# TODO(new-plugin).
TODO_DIRS := plugins core/migrate providers
TODO_DIRS := plugins providers
todos:
@left=$$(grep -rl 'TODO(new-plugin)' $(TODO_DIRS) || true); \
if [ -n "$$left" ]; then \
Expand All @@ -90,3 +92,8 @@ contract:
# Release workflow attaches it to each release as plugins.json.
plugins:
@$(GO) run ./cmd/list-plugins

# Scaffold a provider's source on a kind: cmd/new-plugin with ARGS. In a
# terminal it asks for whatever ARGS leaves out.
new-plugin:
$(GO) run ./cmd/new-plugin $(ARGS)
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,13 @@

You need Go (the version in `go.mod`), `make` and Python 3.

1. **Scaffold it**, on the kind that reads its source ([which kind](docs/writing-a-plugin.md#3-pick-the-kind)):
1. **Scaffold it**, on the kind that reads its source ([which kind?](docs/which-kind.md)); in a terminal, `make new-plugin` asks for what's missing:
```sh
go run ./cmd/new-plugin --api acme --provider Acme --format feed --source https://acme.example/changelog.xml
make new-plugin ARGS='--provider acme --name Acme --source api --kind notices --reader feed --url https://acme.example/changelog.xml'
```
2. **Fill in every `TODO(new-plugin)`**: the reader, the provider and source metadata in its `plugin.RegisterProvider` block, a saved copy of the source in `testdata/`, and "Why this source" in `providers/acme.md`. Then write the golden file and read it like code:
2. **Fill in every `TODO(new-plugin)`**: the reader, the provider and source metadata in its `plugin.RegisterProvider` block, a saved copy of the source in `testdata/`, and "Why this source" in `providers/acme-api.md`. Then write the golden file and read it like code:
```sh
go test ./plugins/feed -run TestAcme -update
go test ./plugins/notices -run TestAcmeApi -update
```
3. **Check it the way CI will, and open the PR** (`git commit -s`, one provider per PR):
```sh
Expand All @@ -30,7 +30,7 @@ You need Go (the version in `go.mod`), `make` and Python 3.

## More

- [How to write a plugin](docs/writing-a-plugin.md): providers, sources, kinds, versions, the test kit, step by step.
- [Which kind fits my provider?](docs/which-kind.md) and [how to write a plugin](docs/writing-a-plugin.md): providers, sources, kinds, versions, the test kit, step by step.
- [How it works](docs/how-it-works.md): what a plugin does, the layout, the checks, releases.
- What's covered: `make plugins` prints every provider and source from the code; each [release](https://github.com/patch-it-dev/patch-it-plugins/releases) carries it as `plugins.json`, and [docs.patch-it.dev](https://docs.patch-it.dev/supported-apis/plugins/) shows it.
- [CONTRIBUTING.md](CONTRIBUTING.md) (the rules), [AGENTS.md](AGENTS.md) (for AI coding agents), [Discussions](https://github.com/patch-it-dev/patch-it-plugins/discussions), [good first issues](https://github.com/patch-it-dev/patch-it-plugins/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
Expand Down
2 changes: 1 addition & 1 deletion cmd/list-plugins/main.go
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
// Command list-plugins prints what patch-it watches, as JSON: every registered
// provider with its sources (id, old slug, kind, layer, what it watches, the
// provider with its sources (id, kind, layer, what it watches, the
// URLs it reads, its page), built from the plugin registry at run time. The
// Release workflow attaches its output to each release as plugins.json; it is
// never committed.
Expand Down
Loading
Loading