Skip to content

add the apisurface kind: a Python package's public API between releases (I19.10) - #218

Merged
adssib merged 3 commits into
mainfrom
pl2-apisurface
Nov 21, 2026
Merged

adssib merged 3 commits into
mainfrom
pl2-apisurface

Conversation

@adssib

@adssib adssib commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #217

Closes #181

What this changes

The apisurface kind (I19.10), Python first: for a package on PyPI, ListVersions lists every final release, ReadAt downloads the release's wheel (or sdist) and reads its public API, Extract compares two releases into events.

  • Static, no execution, no new dependency. The archive is opened in memory (archive/zip, archive/tar; nothing written to disk, links never followed, MaxFileBytes / MaxSourceBytes caps). plugins/apisurface/pysurface reads the .py / .pyi files as text: logical lines (strings kept, comments dropped, brackets joined), indentation, and the headers of def / async def / class, __all__ and from … import. Bodies are skipped. A .pyi stub wins over its .py.
  • Why not a parser library or Python's ast: reading what someone else publishes must never run it, and a Python interpreter would be a second runtime next to the plugins; the pure-Go Python parsers available follow old grammars (no f-strings, /, match…) and would be the module's second dependency. The import allowlist is unchanged (gopkg.in/yaml.v3 only).
  • Public: not underscored, not in a private module (_client.py), in __all__ when there is one; an __init__'s re-exports (from ._client import OpenAI) count, followed to the definition.
  • Events (package_release): removed_symbol (a removed class's members aren't repeated), renamed_symbol (one removed, one added, same scope and kind, identical non-empty parameters), removed_parameter (not when the function now takes **kwargs), changed_default (a parameter lost its default: callers must pass it). Overloaded functions' parameters aren't compared. A new required parameter is logged, not an event: the contract has no subject for it. The source URL is the newer release's file; the excerpt quotes the definition's file and line in both releases.
  • Registered openai/python-sdk (pypi:openai, first name openai), providers/openai-python-sdk.md. Golden: openai 1.1.1 → 1.1.2, which renamed the misspelled AsssitantDeleted (2 events: its re-export and its module). The two wheels are real, trimmed to openai/types/beta/ and the __init__s, byte for byte (checked: the same surface under openai.types.beta as the whole wheels).
  • npm (.d.ts) is the follow-up, I19.12 · apisurface for npm: a package's exported API from its .d.ts files #212 (Backlog, unassigned).

Acceptance criteria

  • A golden test per ecosystem over two saved releases of a real package: Python (npm is I19.12 · apisurface for npm: a package's exported API from its .d.ts files #212)
  • Private names never produce events (underscored names, private modules, names outside __all__; TestRead)
  • No package code is executed
  • One source per SDK that already has an API source: OpenAI's (the others follow the same registration)

Calls I made (please check)

  • ToolRetreival → ToolRetrieval-style renames of a class without its own __init__ read as removals (no parameters to tell it's the same class): precision over recall.
  • Removed symbols are reported under every public path they had (a re-export and its module): code can import either.
  • Size caps are package variables (4 MiB per file, 128 MiB of source per release): safety limits on untrusted input, not tunables.

Checks

  • make check passes (tidy, build, gofmt, vet, import allowlist, net/http guard, go test -race, no TODO(new-plugin), leak guard); make contract against patch-it's main fixtures passes (11 files).
  • Goldens: one new golden (plugins/apisurface/testdata/golden/openai-1.1.2.json, 2 events); no existing golden changed

Related PRs

🤖 Generated with Claude Code

Python first: every release on PyPI, its wheel (or sdist) read in memory
as text by pysurface (no import, no execution, no new dependency), and
consecutive surfaces compared into removed and renamed symbols, removed
parameters and parameters that lost their default. Registers
openai/python-sdk, with a golden over two trimmed real wheels.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: adssib <[email protected]>
@claude

claude Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @adssib's task in 2m 25s —— View job


⚠️ Merge after fixes. The rules are all met and every check passes, but a few paths can emit wrong events or crash the run on untrusted input (inline comments below).

Rules

Rule Status Evidence
Metadata in the registration (Rule 1) ✅ apisurface.go:37-46: the provider's id, name, category and homepage match the existing openai provider in notices (merged, no conflict). The source python-sdk has layer sdk, Watches, two URLs, Packages: pypi:openai and its page. No hand-kept list was added. I read the words and they're accurate.
Provider page ✅ providers/openai-python-sdk.md has Why this source with the link, the date checked (2026-11-20), why it's the best source, the alternatives not used (CHANGELOG, release notes, OpenAPI) and a quote. Known gaps are stated.
Golden test + dated quote ✅ testdata/golden/openai-1.1.2.json (2 events) over real wheels trimmed byte for byte. testdata/README.md has a row per file with URL, date, license and quote.
Pinned drop test ✅ TestAnAddedNameIsNoEvent (adding AssistantDeleted produces no event)
Source URL + quote on every event ✅ source_url is the newer release's wheel. The excerpt quotes the file and line in both releases.
Fetch only via env ✅ Fetches only through env.Get. No net/http, filesystem or database. Archives are read in memory, links are skipped, sizes are capped.
Nothing private (Rule 2) ✅ Nothing private in the code, docs, test data, commit or PR body. The leak guard is clean.
contract/fixtures/ untouched ✅

Bugs (inline comments)

  1. self-only methods and constructors read as renames (apisurface.go:290). len(old.Params) > 0 is always true for methods, because self counts as a parameter. So close(self) → aclose(self), or one openai.resources class replacing another (both __init__(self, client)), becomes a confident "renamed, use Y with the same parameters" event.
  2. A star-import cycle overflows the stack (pysurface.go:157). public() follows from … import * across __init__s with no visited set. pkg/__init__ → from .sub import * plus pkg/sub/__init__ → from .. import * recurses until Go's fatal stack overflow, which kills the whole run, not just this plugin.
  3. Skipped files read as removals (apisurface.go:168). Skipped is stored but Extract ignores it, so a file that grows past MaxFileBytes turns every symbol in it into a removed_symbol event.
  4. Positional-only parameter renames (apisurface.go:322). def f(a, /) → def f(b, /) emits removed_parameter a, but callers can't pass a by name, so nothing breaks.
  5. PEP 695 generics (pysurface.go:59-60). def f[T](…) and class C[T](…) don't match, so moving to 3.12 syntax reads as removals. This doesn't affect openai today (it supports 3.8+), but the kind is generic.

Bugs 1–3 should be fixed before merge. Each fix is small, and a drop test per case would pin it down.

Verified

  • go build ./...: OK
  • go vet ./...: OK
  • gofmt -l .: printed nothing
  • go test -race ./...: all packages pass, including plugins/apisurface and plugins/apisurface/pysurface
  • python3 scripts/leak_guard.py --all: "no banned terms (8 hashed terms checked)"
  • I didn't run make check (it needed an approval I didn't have here). Its steps above are what I checked.
  • I found bugs 1–5 by reading the code. I couldn't run a scratch test here (no file writes), so a quick test per case is worth adding to confirm them.

Non-blocking

  • Inherited methods: the page lists them as a known gap, but the effect is a false event, not just a missed one. When a generated SDK moves a method from Foo into a base class or mixin, Foo.method gets a removed_symbol even though Foo().method still works. The same happens when a class or function becomes an alias assignment (Old = New), which is a common deprecation pattern. Consider dropping removals of methods when the class's bases changed, and following module-level Name = OtherName aliases as re-exports.
  • Baseline key: the baseline key and the events' api are the bare "openai". Nothing collides today (the notices source uses openai-models), but a later changelog source for the same package would key on "openai" too. Something like openai-python-sdk would be safer.
  • Yanked files: pickArtifact decodes yanked but doesn't use it. It's harmless, since whole yanked releases are already filtered out, but you could prefer a non-yanked file within a release.

Thanks for the careful write-up. The "Calls I made" section and the trimmed, byte-for-byte test data made this much easier to review.

Comment thread plugins/apisurface/apisurface.go Outdated
for _, g := range byParent {
if len(g[0]) == 1 && len(g[1]) == 1 {
old, now := a[g[0][0]], b[g[1][0]]
if old.Params != nil && !old.Overloaded && !now.Overloaded && slices.Equal(old.Params, now.Params) && len(old.Params) > 0 {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrong renamed_symbol when the only parameter is self. len(old.Params) > 0 is meant to require real parameters, but methods (and a class's __init__) always start with self, so it holds trivially. Two examples of what this would report as renames:

  • a class removes def close(self) and adds def aclose(self) in the same release
  • openai.resources removes one resource class and adds another; both have __init__(self, client: OpenAI), so they compare equal

Either case becomes "X was renamed to Y … Use Y, with the same parameters", which turns into a wrong PR in someone's repo. Suggestion: compare the parameters without a leading self/cls and require at least one left. Requiring the same signature apart from the name (header with the name swapped) would be safer still.

Comment thread plugins/apisurface/apisurface.go Outdated
now := env.Now().UTC()
var events []event.EventV2
var rejected []error
for _, c := range Compare(a.Surface, b.Surface) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skipped files show up as removals. ReadAt stores Skipped (files over MaxFileBytes), but Extract never looks at it. If a module is under 4 MiB in release N and over it in N+1, or the cap changes, every symbol in that file gets a removed_symbol event, even though nothing was removed. Suggestion: drop any change whose Before.File is in b.Skipped (or in a.Skipped, for symbols that would show up as new), and log it as a gap. A drop test for this case would pin it down.

add(n)
}
for _, s := range m.stars {
if t, ok := mods[s]; ok && t != m {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A star-import cycle recurses forever and crashes the process. public follows from X import * in __init__s with no hop limit, and the only guard is t != m, which catches direct self-imports only. Example:

  • pkg/__init__.py: from .sub import *
  • pkg/sub/__init__.py: from .. import *

Unless one of them has __all__, this loops pkg → pkg.sub → pkg until the Go stack overflows. A stack overflow is a fatal runtime error, not a recoverable panic, so it takes down every plugin in the run. The input is untrusted, so public needs a visited set or a hop limit, as resolve already has.

Comment on lines +59 to +60
defLine = regexp.MustCompile(`^(?:async\s+)?def\s+([A-Za-z_]\w*)\s*\(`)
classLine = regexp.MustCompile(`^class\s+([A-Za-z_]\w*)\s*[(:]`)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PEP 695 generics aren't matched, so they read as removals. def first[T](xs: list[T]) -> T: fails defLine because [ comes before (. class Box[T](Base): fails classLine for the same reason. Both lines end in :, so they fall into the other block. The function disappears from the surface, and the class's methods get recorded as module-level functions. When a package moves to the 3.12 syntax, that produces removed_symbol events for names that are still there. OpenAI's SDK supports 3.8, so this doesn't affect it today, but the kind is generic. Suggestion: allow an optional \[…\] after the name. At minimum, treat an unmatched def/class line as unreadable and skip that module, rather than reading it as gone.

switch {
case i < 0 && kwargs:
out = append(out, Change{Symbol: k, Note: fmt.Sprintf("%s: parameter %s gone, but **kwargs still takes it", k, op.Name)})
case i < 0:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renaming a positional-only parameter isn't breaking. With def f(a, /) → def f(b, /), callers can't pass a by name, so nothing breaks, yet this emits removed_parameter a. Suggestion: skip op.Kind == "positional" here. Comparing the positional count would be the right check, if anything.

adssib and others added 2 commits November 20, 2026 18:26
Base automatically changed from pl2-packages to main November 21, 2026 00:57
@adssib
adssib merged commit 11ea233 into main Nov 21, 2026
4 checks passed
@adssib
adssib deleted the pl2-apisurface branch November 21, 2026 00:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

I19.10 · apisurface kind: a package's public API between versions

1 participant