docs(spec): plugin publish (tarball) + categories design
Design for two coupled changes: drop git as the publish transport in favour of tar.zst uploads, and add a first-class plugin kind plus a configurable, validated category list. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Alex Dunmow
parent
32c6528162
commit
1d9ca44f55
@ -0,0 +1,253 @@
|
|||||||
|
# Plugin publish (tarball) + categories — design
|
||||||
|
|
||||||
|
Date: 2026-06-03
|
||||||
|
Status: Draft
|
||||||
|
|
||||||
|
## Motivation
|
||||||
|
|
||||||
|
Two related changes to the plugin registry:
|
||||||
|
|
||||||
|
1. **Drop git as the publish transport.** The current flow has `ninja plugin
|
||||||
|
publish` push a git tag to a bare repo served over smart-HTTP by the
|
||||||
|
orchestrator. This couples the client to git state (the `ninja` remote
|
||||||
|
stored in `.git/config` just bit us with a URL-shape drift bug), forces
|
||||||
|
the orchestrator to maintain bare repos plus an archive store, and gives
|
||||||
|
the developer a remote they didn't ask for. Move to a single artifact:
|
||||||
|
the CLI builds a `tar.zst` from the source tree and posts it.
|
||||||
|
|
||||||
|
2. **First-class plugin kind, configurable category list.** Today
|
||||||
|
`registry_plugins.categories` (`text[]`) and `Plugin.categories` (proto)
|
||||||
|
exist but nothing writes to them, there is no canonical list, and themes
|
||||||
|
are not distinguished from plugins. Add a `kind` field
|
||||||
|
(`plugin` | `theme`), validate categories against a server-side canonical
|
||||||
|
list, and surface both in the CLI.
|
||||||
|
|
||||||
|
Greenfield — no consumers depend on the existing model. We can change shapes
|
||||||
|
and wipe existing rows.
|
||||||
|
|
||||||
|
## Plugin kind & categories
|
||||||
|
|
||||||
|
### Model
|
||||||
|
|
||||||
|
`registry_plugins` gains:
|
||||||
|
- `kind text NOT NULL DEFAULT 'plugin' CHECK (kind IN ('plugin', 'theme'))`
|
||||||
|
|
||||||
|
`registry_categories` is a new table:
|
||||||
|
- `slug text PRIMARY KEY` — e.g. `analytics`, `social`, `import-export`
|
||||||
|
- `display_name text NOT NULL`
|
||||||
|
- `description text NOT NULL DEFAULT ''`
|
||||||
|
- `sort_order int NOT NULL DEFAULT 0`
|
||||||
|
|
||||||
|
Seeded at migration time with a starter list (configurable later via an
|
||||||
|
admin endpoint; out of scope for this spec). `registry_plugins.categories`
|
||||||
|
values must be subsets of `registry_categories.slug`. Validation is
|
||||||
|
enforced by the API layer, not by FK — categories may be removed later
|
||||||
|
without orphaning plugins.
|
||||||
|
|
||||||
|
Themes do not carry categories in v1. Categories on theme rows are rejected
|
||||||
|
with `CodeInvalidArgument`.
|
||||||
|
|
||||||
|
### plugin.mod
|
||||||
|
|
||||||
|
`[plugin]` gains two fields:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[plugin]
|
||||||
|
name = "lcars"
|
||||||
|
scope = "@themes"
|
||||||
|
version = "0.2.0"
|
||||||
|
kind = "theme" # plugin | theme — defaults to "plugin" if absent
|
||||||
|
categories = ["analytics"] # only allowed when kind = "plugin"
|
||||||
|
```
|
||||||
|
|
||||||
|
The CLI keeps `plugin.mod` in sync with the registry. On `init` it writes
|
||||||
|
both fields. On `publish` it sends them through and the orchestrator
|
||||||
|
verifies the tarball's `plugin.mod` matches the request.
|
||||||
|
|
||||||
|
### Proto
|
||||||
|
|
||||||
|
`Plugin` gains:
|
||||||
|
- `string kind = 10;`
|
||||||
|
|
||||||
|
`Plugin.categories` (existing) stays. `CreatePluginRequest` gains `kind`
|
||||||
|
and `categories`. Listing endpoints gain a `kind` filter:
|
||||||
|
|
||||||
|
```proto
|
||||||
|
message ListPluginsRequest {
|
||||||
|
int32 limit = 1;
|
||||||
|
int32 offset = 2;
|
||||||
|
string query = 3;
|
||||||
|
string kind = 4; // "" = any, "plugin", "theme"
|
||||||
|
repeated string categories = 5;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
A new `ListCategoriesRequest/Response` exposes the canonical list to the
|
||||||
|
CLI for interactive picking.
|
||||||
|
|
||||||
|
## Tarball publish flow
|
||||||
|
|
||||||
|
### Format
|
||||||
|
|
||||||
|
`tar.zst` — `archive/tar` from stdlib for tarring, `github.com/klauspost/compress/zstd`
|
||||||
|
for compression. Same dep on both ends. Default zstd level 3 (fast,
|
||||||
|
~gzip-9 ratio).
|
||||||
|
|
||||||
|
A hard size cap of 25 MiB on the *compressed* (posted) bytes, enforced
|
||||||
|
both sides. Configurable via `REGISTRY_PUBLISH_MAX_BYTES`. The orchestrator
|
||||||
|
additionally caps decompressed size at 4× the configured value to limit
|
||||||
|
zip-bomb exposure during validation.
|
||||||
|
|
||||||
|
### Archive contents
|
||||||
|
|
||||||
|
`git archive --format=tar HEAD` piped through zstd. This gives:
|
||||||
|
- Only tracked files. Anything matching `.gitignore` is excluded so long
|
||||||
|
as it isn't already tracked — i.e. `.gitignore` is honoured by virtue of
|
||||||
|
those files never having been added. The one edge case is a file that
|
||||||
|
was committed *before* being added to `.gitignore`: `git archive` will
|
||||||
|
still ship it. We do not paper over this — the developer must
|
||||||
|
`git rm --cached <file>` to drop it from the index. Documented in the
|
||||||
|
CLI publish output if we detect tracked files matching the current
|
||||||
|
`.gitignore`.
|
||||||
|
- A clean snapshot at HEAD.
|
||||||
|
- Deterministic enough that re-running publish with no changes produces
|
||||||
|
byte-identical archives modulo timestamps.
|
||||||
|
|
||||||
|
The CLI shells out to `git archive HEAD` (no in-process git library
|
||||||
|
dependency). It does not produce a tag, push, or modify `.git/config`.
|
||||||
|
|
||||||
|
### Proto change
|
||||||
|
|
||||||
|
`PublishVersionRequest` becomes:
|
||||||
|
|
||||||
|
```proto
|
||||||
|
message PublishVersionRequest {
|
||||||
|
string plugin_id = 1;
|
||||||
|
string version = 2; // was: git_ref; now plain "0.2.0"
|
||||||
|
string channel = 3;
|
||||||
|
bytes archive = 4; // tar.zst, max 25 MiB
|
||||||
|
string readme_md = 5;
|
||||||
|
string changelog_md = 6;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`git_ref` is removed.
|
||||||
|
|
||||||
|
### Server-side validation
|
||||||
|
|
||||||
|
The orchestrator:
|
||||||
|
1. Verifies the caller is a scope member.
|
||||||
|
2. Decompresses the archive in-memory, walks the tar, finds `plugin.mod`.
|
||||||
|
3. Parses `plugin.mod` and checks `scope`, `name`, `version`, `kind`, and
|
||||||
|
`categories` match the request and the plugin row.
|
||||||
|
4. Re-validates categories against `registry_categories.slug`.
|
||||||
|
5. Computes sha256 and size of the *compressed* archive bytes (what gets
|
||||||
|
stored).
|
||||||
|
6. Stores via `regstorage.Put` with key
|
||||||
|
`<scope>/<name>/<version>/source.tar.zst` and content-type
|
||||||
|
`application/zstd`.
|
||||||
|
7. Records the `registry_versions` row.
|
||||||
|
8. Returns the existing `PublishVersionResponse` shape (a signed
|
||||||
|
`archive_url` for immediate download verification).
|
||||||
|
|
||||||
|
`git_commit` / `git_tag` columns on `registry_versions` are dropped.
|
||||||
|
|
||||||
|
### Install flow
|
||||||
|
|
||||||
|
`ResolveInstall` still returns a signed URL. The only change is the object
|
||||||
|
extension (`.tar.zst` instead of `.tar.gz`) and content-type.
|
||||||
|
|
||||||
|
## Init flow
|
||||||
|
|
||||||
|
`ninja plugin init`:
|
||||||
|
|
||||||
|
1. Prompt scope (existing behavior).
|
||||||
|
2. Prompt plugin name (existing).
|
||||||
|
3. Prompt kind (`plugin` | `theme`). Default: `plugin`.
|
||||||
|
4. If `kind == plugin`: fetch the canonical category list via
|
||||||
|
`ListCategories` and let the user multi-select (zero or more).
|
||||||
|
5. Write `plugin.mod` with all fields.
|
||||||
|
6. Call `CreatePlugin` with kind + categories.
|
||||||
|
7. If in a git repo and `plugin.mod` is new or modified: `git add plugin.mod`,
|
||||||
|
`git commit -m "Add plugin.mod"`. If the file is unchanged from HEAD,
|
||||||
|
skip the commit.
|
||||||
|
8. **Does not** add a `ninja` git remote. Does not touch `.git/config`.
|
||||||
|
|
||||||
|
If not in a git repo, init still writes `plugin.mod` and registers the
|
||||||
|
plugin; it prints a warning that the user will need to `git init` before
|
||||||
|
they can `publish` (`git archive` requires a repo).
|
||||||
|
|
||||||
|
## Publish flow
|
||||||
|
|
||||||
|
`ninja plugin publish`:
|
||||||
|
|
||||||
|
1. Read `plugin.mod`.
|
||||||
|
2. Working-tree-clean check unless `--allow-dirty`.
|
||||||
|
3. Lookup `Plugin` via `GetPlugin(scope, name)` to verify it exists and to
|
||||||
|
get its `id`.
|
||||||
|
4. `git archive --format=tar HEAD` → zstd compress → bytes in memory.
|
||||||
|
5. Optional README/CHANGELOG bytes.
|
||||||
|
6. Call `PublishVersion(plugin_id, version, channel, archive, readme,
|
||||||
|
changelog)`.
|
||||||
|
7. Print version + warnings.
|
||||||
|
|
||||||
|
No tag is created. No git push. No `.git/config` writes anywhere in the
|
||||||
|
flow.
|
||||||
|
|
||||||
|
## What gets deleted
|
||||||
|
|
||||||
|
Orchestrator:
|
||||||
|
- `backend/internal/registry/git/` — the entire package (server, repo
|
||||||
|
manager, smart-HTTP, Provision, ResolveTag, ReadFileAtCommit, tests).
|
||||||
|
- `r.Route("/git", …)` in `registry_wiring.go`.
|
||||||
|
- `RegistryGitPath` config.
|
||||||
|
- `git_commit`, `git_tag` columns on `registry_versions`.
|
||||||
|
- `git_remote_url` field from `CreatePluginResponse` (the URL no longer
|
||||||
|
exists — there is nothing for the developer to push to manually).
|
||||||
|
|
||||||
|
CLI:
|
||||||
|
- `git remote remove ninja` / `git remote add ninja …` block in
|
||||||
|
`cmd/ninja/cmd/plugin.go`.
|
||||||
|
- `git tag -a vX.Y.Z` / `git push ninja vX.Y.Z` block.
|
||||||
|
|
||||||
|
Database:
|
||||||
|
- `git_commit`, `git_tag` columns dropped.
|
||||||
|
- `registry_plugins` rows can be truncated for dev environments since the
|
||||||
|
schema is changing; production is greenfield so this is moot.
|
||||||
|
|
||||||
|
## What gets added
|
||||||
|
|
||||||
|
Orchestrator:
|
||||||
|
- `registry_categories` table + a migration that seeds it with a starter
|
||||||
|
list. Slugs only; copy can be refined later.
|
||||||
|
- A `kind` column on `registry_plugins`.
|
||||||
|
- `ListCategories` RPC added to the existing `PluginRegistryService`
|
||||||
|
(one service is fine at this size).
|
||||||
|
- Archive validation helper (open zstd, walk tar, extract `plugin.mod`).
|
||||||
|
- `REGISTRY_PUBLISH_MAX_BYTES` config (default `25 * 1024 * 1024`).
|
||||||
|
|
||||||
|
CLI:
|
||||||
|
- `bump`, `version` (just shipped, no change).
|
||||||
|
- Init prompts for kind + categories; calls `ListCategories`.
|
||||||
|
- Publish builds `tar.zst` from `git archive HEAD`.
|
||||||
|
|
||||||
|
Proto:
|
||||||
|
- `Plugin.kind`, `CreatePluginRequest.kind/categories`,
|
||||||
|
`ListPluginsRequest.kind/categories`, new `ListCategories*` messages,
|
||||||
|
`PublishVersionRequest.archive` (bytes), removal of `git_ref` and the
|
||||||
|
top-level `git_remote_url` from `CreatePluginResponse`.
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
|
||||||
|
- What is the starter category list? Seed something reasonable; admins
|
||||||
|
can curate it later. Suggested starting set: `analytics`, `seo`,
|
||||||
|
`social`, `commerce`, `forms`, `import-export`, `media`, `developer`.
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- Admin UI for managing categories (use a migration for now).
|
||||||
|
- Category translations / i18n.
|
||||||
|
- Plugin search by category in the install UI (data is there once the
|
||||||
|
field is populated; consumers can use it when ready).
|
||||||
|
- Auth for the publish endpoint — already in place via existing bearer
|
||||||
|
token interceptor.
|
||||||
Loading…
x
Reference in New Issue
Block a user