Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ core/event/ the change event of patch-it's contract: types, fingerpri
core/diff/ core/normalize/ core/hint/ the OpenAPI diff engine and its helpers
plugins/<kind>/ one package per kind, with its providers: openapi (+ adapter/, one spec adapter
per provider), notices (+ retire/, feed/, eol/), migrationguide (+ sdkguide/,
members/), changelog (Markdown, RST or JSON), catalog, apisurface (+ pysurface/);
members/), changelog (Markdown, RST, GitHub release notes or JSON), catalog, apisurface (+ pysurface/);
plugins/all imports every one
cmd/list-plugins/ prints the list from the registry (make plugins); internal/pluginlist/ is its logic
cmd/new-plugin/ scaffolds a provider's source on a kind (make new-plugin ARGS=...; go run ./cmd/new-plugin --help)
Expand Down
2 changes: 1 addition & 1 deletion docs/which-kind.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Every kind is one of two shapes ([writing-a-plugin.md, step 4](writing-a-plugin.
|---|---|---|---|
| Does the provider publish an **OpenAPI spec** of its API, per release or per commit? | `openapi` | Two specs, diffed: removed and renamed operations, parameters, fields; deprecations | A spec repo with release tags; a spec file a repo commits on every change |
| Is there a **guide that lists what a release removed**, one per major or per release, in a table? | `migrationguide` | The guide at the release's tag: removed request parameters (reader `sdk`) or removed members of a module (reader `members`, RST or Markdown tables) | `MIGRATION.md` per major; "removed members" tables in a 2.0 migration guide; an API version removal guide |
| Is there a **changelog**: a file per release (`CHANGELOG.md`, `CHANGES.rst`, `HISTORY.rst`), or a machine-readable JSON changelog? | `changelog` | The release's section at its tag (reader `markdown`), or the life-cycle entries a JSON changelog adds commit by commit (reader `json`) | Keep a Changelog files; reStructuredText release notes; an API changelog JSON in a repo; a Terraform provider's changelog |
| Is there a **changelog**: a file per release (`CHANGELOG.md`, `CHANGES.rst`, `HISTORY.rst`), release notes on each GitHub release, or a machine-readable JSON changelog? | `changelog` | The release's section at its tag, or the tag's GitHub release notes (`Notes: true`) (reader `markdown`), or the life-cycle entries a JSON changelog adds commit by commit (reader `json`) | Keep a Changelog files; release-please and changesets changelogs; reStructuredText release notes; an SDK with only GitHub release notes; an API changelog JSON in a repo; a Terraform provider's changelog |
| Is there a **live page of dated notices**: a deprecations or shutdown table, an end-of-life table, or an **RSS/Atom feed** of announcements? | `notices` | The page as it is now, cited through a Web Archive snapshot, against the snapshot kept from the read before: a living page, with no versions of its own (reader `deprecations`, `feed`, or an end-of-life table) | A model deprecations page; an API end-of-life policy table; a changelog feed with a "deprecation" label; a Kubernetes API removals page |
| Is there a **flat catalog**, one JSON entry per name (a command, an endpoint), with no release tags? | `catalog` | Two snapshots, compared for names that disappeared entirely | A list of commands a server accepts, kept in a repo |
| Is it a **published package** with no changelog worth reading? | `apisurface` | Each release's files from the registry (PyPI first), compared: removed or renamed classes, functions, methods, and parameters removed or newly required | An SDK generated from a spec, released often, whose changelog says "chore: update types" |
Expand Down
2 changes: 1 addition & 1 deletion docs/writing-a-plugin.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ Write the numbers on the provider page, as `providers/vercel.md` does for its sp
| `plugins/openapi` | An OpenAPI spec per release, diffed with `core/diff/` and turned into events by `core/normalize/` | An `adapter.Adapter` in `plugins/openapi/adapter/` (latest tag, fetch a tag's spec; `Versions` to list every release) and a `plugin.RegisterProvider` block |
| `plugins/notices` | A live page of dated notices: a model provider's deprecations page (a table of date, model, replacement), a vendor's RSS or Atom feed, a vendor's table of whole API version ranges and their support status | A `retire.Page` in `plugins/notices/retire/providers.go`, a `feed.Feed` with a reader for that vendor's entries in `plugins/notices/feed/`, or an `eol.Page` in `plugins/notices/eol/` (`new-plugin` doesn't scaffold this reader yet) |
| `plugins/migrationguide` | A guide that lists what a release removed, at the release's tag: an SDK's migration guide per new major (removed request parameters), or a package's table of removed members (an RST simple table or a Markdown pipe table) | A `sdkguide.Guide` in `plugins/migrationguide/sdkguide/`, or a `members.Guide` in `plugins/migrationguide/members/` |
| `plugins/changelog` | A changelog: a `CHANGELOG.md` / `CHANGES.rst` / `HISTORY.rst`, each release's section at its tag (releases from PyPI or npm), or a vendor's machine-readable API changelog in JSON (per day, endpoint and API version, oasdiff change codes), commit by commit | A `changelog.File` with its `TagPrefix`, or a `changelog.Log` |
| `plugins/changelog` | A changelog: a `CHANGELOG.md` / `CHANGES.rst` / `HISTORY.rst`, each release's section at its tag (releases from PyPI or npm, or for any other ecosystem the repo's `TagPrefix` + X.Y.Z tags), or each GitHub release's notes (`Notes`), or a vendor's machine-readable API changelog in JSON (per day, endpoint and API version, oasdiff change codes), commit by commit | A `changelog.File` with its `Ecosystem` and `TagPrefix`, or a `changelog.Log` |
| `plugins/catalog` | A flat JSON catalog, one entry per name, with no release tags to pin — a commit SHA stands in | A `catalog.Catalog`, diffed between two snapshots for names that disappeared entirely |
| `plugins/apisurface` | A published package's public API, compared between two releases: the classes, functions, methods and parameters in each release's files (Python first: the wheel or sdist from PyPI) | A `apisurface.Package` (registry name, ecosystem, top-level module) and its `plugin.RegisterProvider` block |

Expand Down
176 changes: 152 additions & 24 deletions plugins/changelog/changelog.go
Original file line number Diff line number Diff line change
@@ -1,17 +1,22 @@
// Package changelog is the kind for a changelog, with a parser for each
// shape: a package's changelog file in Markdown or reStructuredText (File;
// this file, parse.go, classify.go), or a vendor's machine-readable API
// changelog in JSON (Log; json.go). One Plugin reads either.
// shape: a package's changelog file in Markdown or reStructuredText, or its
// release notes on GitHub (File; this file, parse.go, classify.go), or a
// vendor's machine-readable API changelog in JSON (Log; json.go). One Plugin
// reads either.
//
// A changelog file (CHANGELOG.md, CHANGES.rst, HISTORY.rst) is read at a release's tag: one
// change event per bullet that plainly removes, renames or deprecates names
// code uses. The tag never changes, so its URL is the event's provenance (no
// A changelog file (CHANGELOG.md, CHANGES.rst, HISTORY.rst) is read at a
// release's tag, and a release's notes at its tag's GitHub release: one change
// event per bullet that plainly removes, renames or deprecates names code
// uses. The tag never changes, so its URL is the event's provenance (no
// snapshot) and a rerun stores nothing new: the fingerprints dedupe.
//
// Which bullets count: those under a Keep a Changelog "Removed" or
// "Deprecated" heading, and the common variants ("Breaking changes",
// "Backwards incompatible changes", "Deprecations", reStructuredText titles,
// "**Deprecations**:"), plus a "Breaking change:" bullet under any heading.
// "⚠ BREAKING CHANGES", "Backwards incompatible changes", "Deprecations",
// changesets' "Major Changes", reStructuredText titles, "**Deprecations**:"),
// plus a bullet marked breaking under any heading ("Breaking change:", "⚠️",
// "**[BREAKING]**", "feat!:"). An area, issue or commit the bullet starts
// with ("**client:**", "[#12](...)", "abc1234:") is read past.
// A bullet must say which names in one plain sentence ("Removed `X`",
// "`X` class removed", "Deprecated `X` in favor of `Y`"); anything else
// (dropped Python versions, behavior changes, parameters of an unnamed call,
Expand All @@ -20,8 +25,12 @@ package changelog

import (
"context"
"encoding/json"
"fmt"
"log"
"net/url"
"regexp"
"slices"
"strings"

"github.com/patch-it-dev/patch-it-plugins/core/event"
Expand Down Expand Up @@ -62,16 +71,21 @@ func init() {
}

// File is one package's changelog. Unpinned (Tag and Release empty), it is
// read release by release: every version on the package registry, the file at
// that release's tag (TagPrefix + version). Pinned (At), it is one release.
// read release by release: every version on the package registry (PyPI, npm)
// or, for an ecosystem patch-it doesn't list (Go, Maven, NuGet, RubyGems,
// Packagist, Terraform), every TagPrefix + X.Y.Z tag in the repo; the file at
// that release's tag. With Notes, the changelog is each release's notes on
// GitHub instead of a file, and the versions are the repo's releases. Pinned
// (At), it is one release.
type File struct {
API string // the source id the events carry: "urllib3/urllib3"
Package string // the installed name: "urllib3"
Ecosystem string // where its releases are listed: "pypi"
Package string // the installed name: "urllib3"; a Go module's path without its /vN
Ecosystem string // where it's installed from: "pypi", "npm", "go", "maven", "nuget", "rubygems", "packagist", "terraform"
Module string // the import name, stripped from a symbol's front: "urllib3"
Repo string // owner/name on GitHub
TagPrefix string // a release's git tag is TagPrefix + its version: "v" for v0.86.0
Path string // the changelog's path in the repo
Notes bool // the changelog is each GitHub release's notes, not a file
Tag string // pinned: the release's tag: "2.0.0", "v0.86.0"
Release string // pinned: the release as the changelog heads it: "2.0.0"
}
Expand Down Expand Up @@ -104,11 +118,41 @@ var (
Urllib3200 = Urllib3.At("2.0.0")
)

// URL is the changelog as it was at the (pinned) tag: its own provenance.
// URL is what ReadAt fetches: the changelog as it was at the (pinned) tag,
// or the tag's release on GitHub's API.
func (f File) URL() string {
if f.Notes {
return fmt.Sprintf("%s/repos/%s/releases/tags/%s", versions.GitHubAPI, f.Repo, url.PathEscape(f.Tag))
}
return fmt.Sprintf("https://raw.githubusercontent.com/%s/%s/%s", f.Repo, f.Tag, f.Path)
}

// Cite is the events' provenance: the file at the tag, or the release's
// page, where a person reads the same notes.
func (f File) Cite() string {
if f.Notes {
return fmt.Sprintf("https://github.com/%s/releases/tag/%s", f.Repo, url.PathEscape(f.Tag))
}
return f.URL()
}

// what the changelog is, in a sentence.
func (f File) what() string {
if f.Notes {
return "release notes"
}
return "changelog"
}

// installed is the package as code installs it at the release: a Go
// module's path gains /vN from major 2 on (Go's semantic import versioning).
func (f File) installed() string {
if m := versions.Major(f.Release); f.Ecosystem == "go" && m >= 2 {
return fmt.Sprintf("%s/v%d", f.Package, m)
}
return f.Package
}

// Finding is one bullet that became a change, or one that was dropped.
type Finding struct {
Change Change
Expand All @@ -119,7 +163,12 @@ type Finding struct {

// Read finds the release's breaking and deprecation bullets.
func (f File) Read(page []byte) (heading string, out []Finding, err error) {
heading, sections, err := ReleaseNotes(page, f.Release)
var sections []Section
if f.Notes {
heading, sections, err = f.notes(page)
} else {
heading, sections, err = ReleaseNotes(page, f.Release)
}
if err != nil {
return "", nil, err
}
Expand All @@ -141,7 +190,70 @@ func (f File) Read(page []byte) (heading string, out []Finding, err error) {
out = append(out, Finding{Change: c, Section: s.Title, Bullet: b, Dropped: why})
}
}
return heading, out, nil
return heading, chained(out), nil
}

// chained follows each rename through the release's other renames and
// removals, so a replacement is the name the release ends with: "A renamed
// to B" and "B renamed to C" make A's replacement C; a chain that ends in a
// removal (or loops) leaves A removed, with no replacement to move to.
func chained(fs []Finding) []Finding {
renamed, removed := map[string]string{}, map[string]bool{}
for _, f := range fs {
if f.Dropped != "" || f.Change.ChangeType != event.ChangeTypeBreaking {
continue
}
for _, sym := range f.Change.Symbols {
if f.Change.Replacement != "" {
renamed[sym] = f.Change.Replacement
} else {
removed[sym] = true
}
}
}
for i, f := range fs {
r := f.Change.Replacement
// Only a breaking rename is followed: a deprecation in favor of a
// name stays a deprecation, whatever happens to that name.
if f.Dropped != "" || r == "" || f.Change.ChangeType != event.ChangeTypeBreaking {
continue
}
seen := map[string]bool{}
for next, ok := renamed[r]; ok && !seen[r]; next, ok = renamed[r] {
seen[r] = true
r = next
}
if removed[r] || seen[r] {
fs[i].Change.Replacement, fs[i].Change.Change = "", event.PackageRemovedSymbol
continue
}
fs[i].Change.Replacement = r
}
return fs
}

// notes are the sections of a GitHub release's body (the API's JSON for one
// release), headed by its name. A release with no notes has nothing to read.
func (f File) notes(page []byte) (string, []Section, error) {
var r struct {
Tag string `json:"tag_name"`
Name string `json:"name"`
Body string `json:"body"`
}
if err := json.Unmarshal(page, &r); err != nil {
return "", nil, fmt.Errorf("parsing the release %s: %w", f.Tag, err)
}
if r.Tag != f.Tag {
return "", nil, fmt.Errorf("the release read is %q, not %q", r.Tag, f.Tag)
}
if strings.TrimSpace(r.Body) == "" {
return "", nil, fmt.Errorf("release %s has no notes: %w", f.Tag, plugin.ErrNothingToRead)
}
heading := r.Name
if heading == "" {
heading = r.Tag
}
return heading, NotesSections(r.Body), nil
}

// added are the names the release's "Added" bullets name, as symbols.
Expand Down Expand Up @@ -194,13 +306,24 @@ func (p Plugin) ListVersions(ctx context.Context, env plugin.Env) ([]plugin.Vers
}
var vs []plugin.Version
var err error
switch f.Ecosystem {
case "pypi":
switch {
case f.Notes:
vs, err = versions.GitHubReleases(ctx, env.Get, f.Repo, func(tag string) bool {
v, ok := strings.CutPrefix(tag, f.TagPrefix)
return ok && versions.Final(v) && !strings.HasPrefix(v, "v")
})
for i := range vs {
vs[i].ID = strings.TrimPrefix(vs[i].Ref, f.TagPrefix)
}
slices.SortStableFunc(vs, func(a, b plugin.Version) int { return versions.Compare(a.ID, b.ID) })
return vs, err
case f.Ecosystem == "pypi":
vs, err = versions.PyPIReleases(ctx, env.Get, f.Package)
case "npm":
case f.Ecosystem == "npm":
vs, err = versions.NPMReleases(ctx, env.Get, f.Package)
default:
return nil, fmt.Errorf("%s: no registry to list %q releases from", f.Package, f.Ecosystem)
// No registry patch-it lists from: the repo's release tags.
return versions.GitHubTags(ctx, env.Get, f.Repo, regexp.MustCompile(`^`+regexp.QuoteMeta(f.TagPrefix)+`(\d+(?:\.\d+)+)$`))
}
if err != nil {
return nil, err
Expand All @@ -222,7 +345,7 @@ func (p Plugin) ReadAt(ctx context.Context, env plugin.Env, v plugin.Version) (p
if err != nil {
return plugin.Snapshot{}, fmt.Errorf("fetching %s: %w", f.URL(), err)
}
return plugin.Snapshot{Version: v, Content: page, SourceURL: f.URL()}, nil
return plugin.Snapshot{Version: v, Content: page, SourceURL: f.Cite()}, nil
}

// Extract is the to release's section of the changelog: an event per bullet
Expand All @@ -237,19 +360,20 @@ func (p Plugin) Extract(_ context.Context, env plugin.Env, from, to plugin.Snaps
f.Release, f.Tag = to.Version.ID, to.Version.Ref
heading, findings, err := f.Read(to.Content)
if err != nil {
return plugin.Result{}, fmt.Errorf("reading %s: %w", f.URL(), err)
return plugin.Result{}, fmt.Errorf("reading %s: %w", f.Cite(), err)
}
now := env.Now().UTC()
var events []event.Event
var rejected []error
seen := map[string]bool{} // a changelog that lists a bullet twice is one change
for _, fd := range findings {
if fd.Dropped != "" {
log.Printf("%s %s: dropped %q: %s", f.Package, f.Release, fd.Bullet.Text, fd.Dropped)
continue
}
c := fd.Change
subject := event.Subject{
Type: event.SubjectPackageRelease, Ecosystem: f.Ecosystem, Package: f.Package,
Type: event.SubjectPackageRelease, Ecosystem: f.Ecosystem, Package: f.installed(),
Change: c.Change, Symbols: c.Symbols, Replacement: c.Replacement,
}
e := event.Event{
Expand All @@ -260,13 +384,17 @@ func (p Plugin) Extract(_ context.Context, env plugin.Env, from, to plugin.Snaps
Breaking: c.ChangeType == event.ChangeTypeBreaking,
Subject: subject,
Description: describe(f, c),
MigrationHint: cap2000(fmt.Sprintf("%s (%s's changelog for %s)", fd.Bullet.Text, f.Package, f.Release)),
SourceURL: f.URL(),
MigrationHint: cap2000(fmt.Sprintf("%s (%s's %s for %s)", fd.Bullet.Text, f.Package, f.what(), f.Release)),
SourceURL: f.Cite(),
SourceExcerpt: cap2000(heading + "\n" + fd.Section + "\n" + fd.Bullet.Raw),
Confidence: 1.0,
DetectedAt: now,
}
e.Fingerprint = event.Fingerprint(e.API, e.Version, e.ChangeType, subject, "")
if seen[e.Fingerprint] {
continue
}
seen[e.Fingerprint] = true
if err := e.Validate(); err != nil {
rejected = append(rejected, fmt.Errorf("%s %s %v: %w", f.Package, f.Release, c.Symbols, err))
continue
Expand Down
Loading
Loading