Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Refer to the service that runs these plugins as "patch-it's hosted ingestion ser

## Rules for plugin code

- **Start a new provider with `go run ./cmd/new-plugin --api <id> --provider <Name> --format <format>`**, then fill in every `TODO(new-plugin)`; `make todos` (part of `make check` and CI_go) fails while one is left.
- **Start a new source with `make new-plugin ARGS='--provider <id> --name <Name> --source <id> --kind <kind>'`** (`go run ./cmd/new-plugin`; [which kind](docs/which-kind.md)), then fill in every `TODO(new-plugin)`; `make todos` (part of `make check` and CI_go) fails while one is left.
- **Fetch only through the environment** (`env.Get` or `core/kit/`). No `net/http` outside the fetch layer, no filesystem outside `testdata/`, no database.
- **Golden test + dated source quote**: a saved copy of the real source in `testdata/`, a row in `testdata/README.md` (URL, date fetched, license, quote), and `testdata/golden/<api>.json` with the exact events.
- **Precision over recall.** When unsure whether something is a change, don't emit an event.
Expand All @@ -40,6 +40,7 @@ Refer to the service that runs these plugins as "patch-it's hosted ingestion ser
make check # tidy, build, gofmt, vet, the import allowlist, the net/http guard, go test -race, no TODO(new-plugin) left, leak guard
make check PATCH_IT=../patch-it # also CI_contract: contract/fixtures/ against a patch-it checkout
make plugins # print the list of providers and sources from the registry (what each release attaches as plugins.json)
make new-plugin ARGS='--provider acme --source api --kind openapi' # scaffold a source (asks for what's missing in a terminal)
```

One target at a time: `make tidy`, `make build`, `make fmt`, `make vet`, `make imports`, `make nethttp`, `make test`, `make todos`, `make leak-guard`, `make contract`. Use `make GO=/path/to/go check` for another Go.
Expand All @@ -66,7 +67,7 @@ plugins/removedmembers/ apichangelog/ retirements/ feed/ eol/ deprecated: merg
plugin/ event/ fetch/ migrate/ kit/ retire/ sdkguide/ deprecated old import paths: they only
forward to the packages above; don't add to them
cmd/list-plugins/ prints the list from the registry (make plugins); internal/pluginlist/ is its logic
cmd/new-plugin/ scaffolds a provider on a format plugin (go run ./cmd/new-plugin --help)
cmd/new-plugin/ scaffolds a provider's source on a kind (make new-plugin ARGS=...; go run ./cmd/new-plugin --help)
contract/fixtures/ a copy of patch-it's change-event fixtures (CI_contract checks it)
Makefile make check (what CI_go runs, plus the leak guard), make plugins
docs/ contributor docs, including writing-a-plugin.md
Expand Down
8 changes: 4 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Thanks for helping patch-it watch more APIs. This file is the rulebook; [docs/writing-a-plugin.md](docs/writing-a-plugin.md) is the how-to.

**Start a new provider with the starter:** `go run ./cmd/new-plugin --api <id> --provider <Name> --format <format>` writes the definition, its registration, a test over placeholder sources, the testdata rows, the provider page, and the source's metadata in its `RegisterProvider` block; you fill in the `TODO(new-plugin)`s ([how](docs/writing-a-plugin.md#1-scaffold-it-with-new-plugin)).
**Start a new provider with the starter:** `make new-plugin ARGS='--provider <id> --source <id> --kind <kind>'` ([which kind?](docs/which-kind.md); in a terminal it asks for the rest) writes the definition, its registration, a test over placeholder sources, the testdata rows, the provider page, and the source's metadata in its `RegisterProvider` block; you fill in the `TODO(new-plugin)`s ([how](docs/writing-a-plugin.md#1-scaffold-it-with-new-plugin)).

## The rules

Expand Down Expand Up @@ -46,14 +46,14 @@ That adds `Signed-off-by: Your Name <[email protected]>`, matching the commit's au

Releases are tags on `main` (`v0.x.y`), made only by the **Release** workflow. Nothing is tagged on push, and agents never tag or release.

**Automatic, twice a week:** every Monday and Thursday at 14:00 UTC the workflow makes a **patch** release, but only if something was merged since the last tag; otherwise it does nothing. If the merges add a **new format plugin** (a new `plugins/<format>/` directory), the scheduled run skips with a warning instead: run it by hand with **minor**. So a merged plugin reaches patch-it's hosted ingestion service within a few days.
**Automatic, twice a week:** every Monday and Thursday at 14:00 UTC the workflow makes a **patch** release, but only if something was merged since the last tag; otherwise it does nothing. If the merges add a **new kind** (a new `plugins/<kind>/` directory), the scheduled run skips with a warning instead: run it by hand with **minor**. So a merged plugin reaches patch-it's hosted ingestion service within a few days.

**By hand, any time** (an urgent fix, or a **minor** release):

1. Merge the PRs that should go out.
2. **Actions → Release → Run workflow** (branch `main`), pick the bump:
- **patch** (`v0.4.1` → `v0.4.2`): fixes, a new provider on an existing format plugin, docs;
- **minor** (`v0.4.1` → `v0.5.0`): a new format plugin, new exported API, anything users should read about.
- **patch** (`v0.4.1` → `v0.4.2`): fixes, a new source on an existing kind, docs;
- **minor** (`v0.4.1` → `v0.5.0`): a new kind, new exported API, anything users should read about.
3. The workflow refuses if `main`'s HEAD is already tagged, then runs, on that exact commit, the internal-names check over every file, gitleaks over the whole history, `go build`, `go vet`, `gofmt` and `go test -race`. Any failure stops it before a tag exists.
4. When everything passes it pushes an annotated tag and publishes a GitHub release with generated notes (grouped by the PR's label: `plugin`/`enhancement`, `bug`, `documentation`; see [`.github/release.yml`](.github/release.yml)). The release carries one asset, `plugins.json`: the list of providers and sources printed from the registry at that tag (`go run ./cmd/list-plugins`). It is the only published list; nothing in the repo keeps one, and docs.patch-it.dev reads `https://github.com/patch-it-dev/patch-it-plugins/releases/latest/download/plugins.json`.

Expand Down
9 changes: 8 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
# make check everything below, in CI's order
# make check PATCH_IT=../patch-it also compare contract/fixtures/ with patch-it's
# make plugins print the providers and sources from the registry (JSON)
# make new-plugin ARGS='--provider acme --source api --kind openapi'
# scaffold a source (cmd/new-plugin; asks for what's missing)
#
# The `::error` lines are GitHub annotations; locally they're plain messages.

Expand All @@ -17,7 +19,7 @@ PYTHON ?= python3
MODULE := github.com/patch-it-dev/patch-it-plugins
ALLOWED := ^($(MODULE)(/|$$)|gopkg\.in/yaml\.v3$$)

.PHONY: check tidy build fmt vet imports nethttp test todos leak-guard contract plugins
.PHONY: check tidy build fmt vet imports nethttp test todos leak-guard contract plugins new-plugin

check: tidy build fmt vet imports nethttp test todos leak-guard contract
@echo "make check: all passed"
Expand Down Expand Up @@ -90,3 +92,8 @@ contract:
# Release workflow attaches it to each release as plugins.json.
plugins:
@$(GO) run ./cmd/list-plugins

# Scaffold a provider's source on a kind: cmd/new-plugin with ARGS. In a
# terminal it asks for whatever ARGS leaves out.
new-plugin:
$(GO) run ./cmd/new-plugin $(ARGS)
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,13 @@

You need Go (the version in `go.mod`), `make` and Python 3.

1. **Scaffold it**, on the kind that reads its source ([which kind](docs/writing-a-plugin.md#3-pick-the-kind)):
1. **Scaffold it**, on the kind that reads its source ([which kind?](docs/which-kind.md)); in a terminal, `make new-plugin` asks for what's missing:
```sh
go run ./cmd/new-plugin --api acme --provider Acme --format feed --source https://acme.example/changelog.xml
make new-plugin ARGS='--provider acme --name Acme --source api --kind notices --reader feed --url https://acme.example/changelog.xml'
```
2. **Fill in every `TODO(new-plugin)`**: the reader, the provider and source metadata in its `plugin.RegisterProvider` block, a saved copy of the source in `testdata/`, and "Why this source" in `providers/acme.md`. Then write the golden file and read it like code:
2. **Fill in every `TODO(new-plugin)`**: the reader, the provider and source metadata in its `plugin.RegisterProvider` block, a saved copy of the source in `testdata/`, and "Why this source" in `providers/acme-api.md`. Then write the golden file and read it like code:
```sh
go test ./plugins/notices -run TestAcme -update
go test ./plugins/notices -run TestAcmeApi -update
```
3. **Check it the way CI will, and open the PR** (`git commit -s`, one provider per PR):
```sh
Expand All @@ -30,7 +30,7 @@ You need Go (the version in `go.mod`), `make` and Python 3.

## More

- [How to write a plugin](docs/writing-a-plugin.md): providers, sources, kinds, versions, the test kit, step by step.
- [Which kind fits my provider?](docs/which-kind.md) and [how to write a plugin](docs/writing-a-plugin.md): providers, sources, kinds, versions, the test kit, step by step.
- [How it works](docs/how-it-works.md): what a plugin does, the layout, the checks, releases.
- What's covered: `make plugins` prints every provider and source from the code; each [release](https://github.com/patch-it-dev/patch-it-plugins/releases) carries it as `plugins.json`, and [docs.patch-it.dev](https://docs.patch-it.dev/supported-apis/plugins/) shows it.
- [CONTRIBUTING.md](CONTRIBUTING.md) (the rules), [AGENTS.md](AGENTS.md) (for AI coding agents), [Discussions](https://github.com/patch-it-dev/patch-it-plugins/discussions), [good first issues](https://github.com/patch-it-dev/patch-it-plugins/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
Expand Down
125 changes: 118 additions & 7 deletions cmd/new-plugin/formats.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,27 @@ type mapEntry struct {

const todo = "TODO(new-plugin)"

// legacyFormats are the old --format names, each the template it is now.
var legacyFormats = map[string]string{
"feed": "notices/feed",
"retirements": "notices/deprecations",
"migrationguide": "migrationguide/sdk",
"removedmembers": "migrationguide/members",
"changelog": "changelog/markdown",
"apichangelog": "changelog/json",
"catalog": "catalog",
"openapi": "openapi",
}

// defaultReaders are the reader --kind picks when --reader is left out.
var defaultReaders = map[string]string{
"notices": "deprecations",
"migrationguide": "sdk",
"changelog": "markdown",
}

var formats = map[string]formatSpec{
"feed": {
"notices/feed": {
kind: "notices",
layer: "api",
defDir: "plugins/notices/feed",
Expand Down Expand Up @@ -81,7 +100,7 @@ func {{.Lower}}Ingest(t *testing.T, file string) []event.EventV2 {
imports: []string{"github.com/patch-it-dev/patch-it-plugins/core/event", "github.com/patch-it-dev/patch-it-plugins/plugins/notices/feed"},
},

"retirements": {
"notices/deprecations": {
kind: "notices",
layer: "ai-models",
defDir: "plugins/notices/retire",
Expand Down Expand Up @@ -155,7 +174,7 @@ func {{.Lower}}Ingest(t *testing.T, file string) []event.EventV2 {
imports: []string{"encoding/json", "strings", "github.com/patch-it-dev/patch-it-plugins/core/event", "github.com/patch-it-dev/patch-it-plugins/plugins/notices/retire"},
},

"migrationguide": {
"migrationguide/sdk": {
kind: "migrationguide",
layer: "sdk",
defDir: "plugins/migrationguide/sdkguide",
Expand Down Expand Up @@ -222,7 +241,7 @@ func {{.Lower}}Ingest(t *testing.T, file string) []event.EventV2 {
imports: []string{"encoding/json", "strings", "github.com/patch-it-dev/patch-it-plugins/core/event", "github.com/patch-it-dev/patch-it-plugins/plugins/migrationguide/sdkguide"},
},

"removedmembers": {
"migrationguide/members": {
kind: "migrationguide",
layer: "package",
defDir: "plugins/migrationguide/members",
Expand Down Expand Up @@ -298,7 +317,7 @@ func {{.Lower}}Ingest(t *testing.T, file string) []event.EventV2 {
imports: []string{"slices", "github.com/patch-it-dev/patch-it-plugins/core/event", "github.com/patch-it-dev/patch-it-plugins/plugins/migrationguide/members"},
},

"changelog": {
"changelog/markdown": {
kind: "changelog",
layer: "package",
defDir: "plugins/changelog",
Expand Down Expand Up @@ -340,7 +359,7 @@ func {{.Lower}}Ingest(t *testing.T, file string) []event.EventV2 {
imports: []string{"github.com/patch-it-dev/patch-it-plugins/core/event"},
},

"apichangelog": {
"changelog/json": {
kind: "changelog",
layer: "api",
defDir: "plugins/changelog",
Expand Down Expand Up @@ -437,6 +456,8 @@ func {{.Lower}}Ingest(t *testing.T, file string) []event.EventV2 {
import (
"context"
"fmt"

"github.com/patch-it-dev/patch-it-plugins/core/plugin"
)

// {{.Ident}} adapts {{.Provider}}'s OpenAPI spec for the openapi plugin.
Expand All @@ -460,6 +481,14 @@ func ({{.Ident}}) LatestTag(ctx context.Context) (string, error) {
func ({{.Ident}}) Fetch(ctx context.Context, tag string) (Spec, error) {
return Spec{}, fmt.Errorf("{{.ID}}: Fetch isn't written yet (plugins/openapi/adapter/{{.File}}.go)")
}

// Versions are every release of the spec, oldest first, each with the Ref
// Fetch takes: what lets every pair of releases be diffed, not only the
// newest. ` + todo + `: list them with core/versions (GitHubReleases,
// GitHubCommits, GitHubTags), as versions.go does for the other adapters.
func ({{.Ident}}) Versions(ctx context.Context) ([]plugin.Version, error) {
return nil, fmt.Errorf("{{.ID}}: Versions isn't written yet (plugins/openapi/adapter/{{.File}}.go)")
}
`,
register: `plugin.Register(Plugin{Adapter: adapter.{{.Ident}}{}})`,
pluginOf: `Plugin{Adapter: adapter.{{.Ident}}{}}`,
Expand Down Expand Up @@ -501,7 +530,89 @@ func {{.Lower}}Ingest(t *testing.T, file string) []event.EventV2 {
}
`,
imports: []string{"context", "fmt", "os", "github.com/patch-it-dev/patch-it-plugins/plugins/openapi/adapter", "github.com/patch-it-dev/patch-it-plugins/core/event"},
next: "Write LatestTag and Fetch in plugins/openapi/adapter/{{.File}}.go (fetch.Get only); add Diff/Keep to the Register line if the spec needs them (see vercel.go).",
next: "Write LatestTag, Fetch and Versions in plugins/openapi/adapter/{{.File}}.go (fetch.Get only); add Diff/Keep to the Register line if the spec needs them (see vercel.go).",
},
"apisurface": {
kind: "apisurface",
layer: "sdk",
defDir: "plugins/apisurface",
def: `package apisurface

// {{.Ident}} is {{.Provider}}'s package on PyPI, its public API read from each
// release's wheel.
//
// ` + todo + `: check the name on PyPI and the top-level module code
// imports (from --package, or guessed). Say
// what you checked, with the date: that the wheel holds the package's .py
// (or .pyi) files.
var {{.Ident}} = Package{Name: "{{.PyPI}}", Ecosystem: "pypi", Module: "{{.Module}}"}
`,
register: `plugin.Register(Plugin{Package: {{.Ident}}})`,
pluginOf: `Plugin{Package: {{.Ident}}}`,
fixtures: map[string]string{
"{{.File}}-old.py": "# ` + todo + `: a placeholder for the older release's module. Save the real release's wheel instead (see apisurface_test.go).\ndef placeholder(a, b=1):\n pass\n",
"{{.File}}-new.py": "# ` + todo + `: a placeholder for the newer release's module: it drops b, one event.\ndef placeholder(a):\n pass\n",
},
// The drop test's newer release: the older one again, nothing gone.
dropFile: "# ` + todo + `: a placeholder: a release that only adds or changes private names.\ndef placeholder(a, b=1):\n pass\n\ndef _private(x):\n pass\n",
source: "{{.File}}-new.py",
ingest: `// {{.Lower}}Ingest compares two releases: 1.0.0, a wheel holding
// testdata/{{.File}}-old.py as {{.Module}}/__init__.py, and 1.1.0, one
// holding file. The wheels are built here so the placeholders are text.
// ` + todo + `: save the two real wheels (trimmed if they're large, as
// apisurface_test.go's are) and serve them instead.
func {{.Lower}}Ingest(t *testing.T, file string) []event.EventV2 {
t.Helper()
wheel := func(src string) []byte {
var buf bytes.Buffer
zw := zip.NewWriter(&buf)
w, err := zw.Create("{{.Module}}/__init__.py")
if err != nil {
t.Fatal(err)
}
w.Write(kittest.Read(t, src))
if err := zw.Close(); err != nil {
t.Fatal(err)
}
return buf.Bytes()
}
files := map[string][]byte{
"https://pypi.org/pypi/{{.PyPI}}/json": []byte(` + "`" + `{"releases": {"1.0.0": [{"upload_time_iso_8601": "2026-01-01T00:00:00Z"}], "1.1.0": [{"upload_time_iso_8601": "2026-02-01T00:00:00Z"}]}}` + "`" + `),
"https://pypi.org/pypi/{{.PyPI}}/1.0.0/json": []byte(` + "`" + `{"urls": [{"packagetype": "bdist_wheel", "filename": "{{.Module}}-1.0.0-py3-none-any.whl", "url": "https://example.com/{{.Module}}-1.0.0-py3-none-any.whl"}]}` + "`" + `),
"https://pypi.org/pypi/{{.PyPI}}/1.1.0/json": []byte(` + "`" + `{"urls": [{"packagetype": "bdist_wheel", "filename": "{{.Module}}-1.1.0-py3-none-any.whl", "url": "https://example.com/{{.Module}}-1.1.0-py3-none-any.whl"}]}` + "`" + `),
"https://example.com/{{.Module}}-1.0.0-py3-none-any.whl": wheel("testdata/{{.File}}-old.py"),
"https://example.com/{{.Module}}-1.1.0-py3-none-any.whl": wheel(file),
}
env := kittest.Env(t, nil)
env.Get = func(_ context.Context, url string) ([]byte, error) {
if b, ok := files[url]; ok {
return b, nil
}
t.Errorf("fetched %s, which the test didn't save", url)
return nil, fmt.Errorf("no saved copy of %s", url)
}
ctx, p := context.Background(), Plugin{Package: {{.Ident}}}
vs, err := p.ListVersions(ctx, env)
if err != nil || len(vs) != 2 {
t.Fatalf("versions %v: %v", vs, err)
}
from, err := p.ReadAt(ctx, env, vs[0])
if err != nil {
t.Fatal(err)
}
to, err := p.ReadAt(ctx, env, vs[1])
if err != nil {
t.Fatal(err)
}
res, err := p.Extract(ctx, env, from, to)
if err != nil || len(res.Rejected) > 0 {
t.Fatalf("%v %v", err, res.Rejected)
}
return res.Events
}
`,
imports: []string{"archive/zip", "bytes", "context", "fmt", "github.com/patch-it-dev/patch-it-plugins/core/event"},
next: "Save two real wheels of the package and serve them in {{.Lower}}Ingest; the kind reads their .py files, never runs them.",
},
}

Expand Down
Loading
Loading