diff --git a/docs/superpowers/audits/2026-06-03-plugin-publish-and-categories-uat.md b/docs/superpowers/audits/2026-06-03-plugin-publish-and-categories-uat.md new file mode 100644 index 0000000..786cd9b --- /dev/null +++ b/docs/superpowers/audits/2026-06-03-plugin-publish-and-categories-uat.md @@ -0,0 +1,305 @@ +# UAT — Plugin publish (tarball) + categories + +Date: 2026-06-03 + +> **THIS CHECKLIST IS A HARD GATE.** Implementation is NOT complete until +> every item below has been executed and observed to pass. Build-green and +> tests-green are necessary but NOT sufficient. The work is unfinished if +> any UAT item is skipped, glossed over, or marked done without the exact +> observation recorded. +> +> Do not edit this file to soften any item. If an item turns out to be +> ambiguous or impossible, STOP and ask the user to amend it. Do not +> rationalise around it. + +## How to use + +Execute each item against a freshly-rebuilt orchestrator (`podman compose +build orchestrator-backend && podman compose up -d orchestrator-backend`) +and a freshly-reinstalled CLI (`go install ./cmd/ninja` from `~/src/core`). +For each item, capture the actual output/observation next to the +expectation. Mark a checkbox ONLY after observing the success criterion +with your own eyes. + +The DB connection used in items below is whatever the local orchestrator +uses; commands assume access via `podman exec blockninja-db psql -U orchestrator orchestrator`. + +--- + +## Group A — `ninja plugin init` + +### A1. Init writes plugin.mod with all fields and auto-commits it + +- [ ] Setup: + ```bash + rm -rf /tmp/uat-a1 && mkdir /tmp/uat-a1 && cd /tmp/uat-a1 + git init -q && git commit --allow-empty -qm "initial" + ``` +- [ ] Run: `ninja plugin init --host https://my.localdev.blockninjacms.com --scope @themes --name uat-a1` (answer: kind=plugin, categories=1,2) +- [ ] Observe `cat plugin.mod` includes: + - `name = "uat-a1"` + - `scope = "@themes"` + - `version = "0.1.0"` + - `kind = "plugin"` + - `categories = ["analytics", "seo"]` +- [ ] Observe `git log --oneline` shows a commit whose subject is `Add plugin.mod` at HEAD. +- [ ] Observe `git remote -v` outputs nothing (no `ninja` remote). +- [ ] Observe `grep -A1 '\[remote' .git/config` finds NO `ninja` block. + +### A2. Init for a theme writes kind=theme and skips category prompt + +- [ ] Setup: + ```bash + rm -rf /tmp/uat-a2 && mkdir /tmp/uat-a2 && cd /tmp/uat-a2 + git init -q && git commit --allow-empty -qm "initial" + ``` +- [ ] Run init for `@themes/uat-a2`, answer kind=theme. The category prompt MUST NOT appear. +- [ ] Observe `plugin.mod` contains `kind = "theme"` and NO `categories =` line. + +### A3. Init when not in a git repo prints a warning, still registers + +- [ ] Setup: + ```bash + rm -rf /tmp/uat-a3 && mkdir /tmp/uat-a3 && cd /tmp/uat-a3 + ``` +- [ ] Run init for `@themes/uat-a3`. STDOUT/STDERR contains a warning mentioning `git init` before publish. +- [ ] `plugin.mod` exists on disk. +- [ ] No `.git` directory was created. +- [ ] Database row exists: `psql -c "SELECT 1 FROM registry_plugins WHERE name='uat-a3';"` returns 1. + +### A4. Init twice with same name returns AlreadyExists, leaves repo unchanged + +- [ ] In a freshly-init'd repo with no plugin.mod, run init twice with the same scope+name. +- [ ] Second invocation errors with a message mentioning the plugin already exists. +- [ ] `git log` shows exactly ONE `Add plugin.mod` commit (not two). + +--- + +## Group B — `ninja plugin publish` + +### B1. Publish posts a tar.zst, no git tag, no git push + +- [ ] Setup from a working init'd repo (e.g. uat-a1 above). +- [ ] Pre-record: + ```bash + git tag --list # capture before + git rev-parse HEAD # capture HEAD before + ``` +- [ ] Run `ninja plugin publish --host https://my.localdev.blockninjacms.com`. +- [ ] STDOUT contains `Published @themes/uat-a1@0.1.0 (NNN bytes)`. +- [ ] After publish, `git tag --list` output is byte-identical to before. (NO `v0.1.0` tag was created.) +- [ ] `git rev-parse HEAD` is unchanged. +- [ ] `git remote -v` is still empty. +- [ ] DB row exists with the new version: + ```sql + SELECT v.version, v.source_archive_key, v.source_archive_size, v.source_archive_sha256 + FROM registry_versions v + JOIN registry_plugins p ON p.id = v.plugin_id + WHERE p.name = 'uat-a1'; + ``` + Observe: `source_archive_key` ends with `/source.tar.zst`. `source_archive_size > 0`. + +### B2. The stored archive matches the posted bytes (sha256) + +- [ ] On disk (or via the signed URL), retrieve the stored archive bytes. +- [ ] Compute `sha256sum `. Compare against `source_archive_sha256` from the DB. +- [ ] Hashes match exactly. + +### B3. The archive contains only tracked files and excludes gitignored + +- [ ] Setup: + ```bash + rm -rf /tmp/uat-b3 && mkdir /tmp/uat-b3 && cd /tmp/uat-b3 + git init -q + echo "secret" > ignored.log + echo "ignored.log" > .gitignore + git add .gitignore && git commit -qm "ignore" + ``` +- [ ] Run init + publish for `@themes/uat-b3`. +- [ ] Extract the stored archive (download via signed URL → `zstd -d | tar -xv`). +- [ ] Observe `plugin.mod`, `.gitignore` present; `ignored.log` absent. + +### B4. Working tree dirty check fires; `--allow-dirty` bypasses + +- [ ] In a published repo, touch a file: `echo x >> README.md`. +- [ ] Run `ninja plugin publish ...` — must error with text including `working tree dirty`. +- [ ] Bump patch, then `ninja plugin publish --allow-dirty ...` — must succeed. + +### B5. Tracked-yet-gitignored files trigger a warning + +- [ ] Setup: + ```bash + rm -rf /tmp/uat-b5 && mkdir /tmp/uat-b5 && cd /tmp/uat-b5 + git init -q + echo "data" > dist.log + git add dist.log && git commit -qm "tracked" + echo "dist.log" > .gitignore + git add .gitignore && git commit -qm "ignore" + ``` +- [ ] Init + publish for `@themes/uat-b5`. +- [ ] STDERR includes a warning that `dist.log` is tracked and will still be shipped, plus the hint `git rm --cached`. + +### B6. Duplicate version rejected + +- [ ] In a successfully-published repo, immediately re-run publish without bumping. +- [ ] Error includes `version already published`. + +### B7. Version mismatch between plugin.mod and request rejected + +- [ ] Edit `plugin.mod` to a different version than the one expected by the bump sequence (manually write `version = "9.9.9"` without committing). +- [ ] Run publish — must succeed locally but the orchestrator may accept or reject; if rejected the error mentions version mismatch. Either way, no DB row at version `9.9.9` should exist unless it succeeded — record what happened. +- [ ] Reset: + ```bash + git checkout plugin.mod + ``` + +--- + +## Group C — Categories + kind validation + +### C1. ListCategories returns the seeded set + +- [ ] Run: + ```bash + curl -sS -X POST \ + https://my.localdev.blockninjacms.com/orchestrator.v1.PluginRegistryService/ListCategories \ + -H 'Content-Type: application/json' -d '{}' + ``` +- [ ] Response includes ALL of: `analytics`, `seo`, `social`, `commerce`, `forms`, `import-export`, `media`, `developer`. + +### C2. CreatePlugin rejects unknown category + +- [ ] Setup a fresh repo and start init manually entering a bogus category number — NOT possible via the CLI menu. Instead, hit the RPC directly: + ```bash + TOKEN=$(jq -r '.hosts["https://my.localdev.blockninjacms.com"].token' ~/.config/ninja/creds.json) + curl -sS -X POST \ + https://my.localdev.blockninjacms.com/orchestrator.v1.PluginRegistryService/CreatePlugin \ + -H "Authorization: Bearer $TOKEN" \ + -H 'Content-Type: application/json' \ + -d '{"scopeSlug":"themes","name":"uat-c2","kind":"plugin","categories":["bogus"]}' + ``` +- [ ] Response is a Connect error with code `invalid_argument` and message mentioning unknown category. + +### C3. CreatePlugin rejects theme with categories + +- [ ] Same setup as C2 but kind=theme with categories=["analytics"]: + ```bash + curl -sS -X POST \ + https://my.localdev.blockninjacms.com/orchestrator.v1.PluginRegistryService/CreatePlugin \ + -H "Authorization: Bearer $TOKEN" \ + -H 'Content-Type: application/json' \ + -d '{"scopeSlug":"themes","name":"uat-c3","kind":"theme","categories":["analytics"]}' + ``` +- [ ] Response is invalid_argument with message including "themes do not carry categories". + +### C4. CreatePlugin rejects unknown kind + +- [ ] POST with `kind: "module"` (or anything other than plugin/theme). +- [ ] Response is invalid_argument. + +### C5. PublishVersion rejects plugin.mod kind mismatch + +- [ ] In a published-once `plugin`-kind repo, hand-edit `plugin.mod` to `kind = "theme"`, commit, bump, publish. +- [ ] Publish errors with text including `kind does not match registered`. + +### C6. PublishVersion rejects plugin.mod categories mismatch + +- [ ] In a published-once `plugin`-kind repo, add a category to `plugin.mod` that wasn't in the original `CreatePlugin`, commit, bump, publish. +- [ ] Publish errors with text including `categories do not match registered`. + +### C7. ListPlugins filters by kind + +- [ ] Setup: at least one `plugin` and one `theme` in the registry (any from earlier UAT items). +- [ ] Hit ListPlugins with `kind=plugin`: + ```bash + curl -sS -X POST \ + https://my.localdev.blockninjacms.com/orchestrator.v1.PluginRegistryService/ListPlugins \ + -H "Authorization: Bearer $TOKEN" \ + -H 'Content-Type: application/json' -d '{"kind":"plugin"}' + ``` + Response contains only kind=plugin rows. +- [ ] Same with `{"kind":"theme"}`. Response contains only kind=theme rows. +- [ ] With `{}` (no filter). Response contains both. + +### C8. ListPlugins filters by category + +- [ ] Setup a plugin with categories=["analytics"]. +- [ ] Hit ListPlugins with `{"categories":["analytics"]}`. The plugin appears. +- [ ] Hit with `{"categories":["forms"]}`. The plugin does NOT appear. + +--- + +## Group D — Deletion of git infrastructure + +### D1. `/git/*` HTTP route is gone + +- [ ] Run: + ```bash + curl -sS -o /dev/null -w "%{http_code}\n" \ + https://my.localdev.blockninjacms.com/git/themes/lcars.git/info/refs?service=git-upload-pack + ``` +- [ ] HTTP code is `404` (not 200, not 403). The route does not exist. + +### D2. `registry/git/` package is fully removed + +- [ ] Run: + ```bash + ls ~/src/orchestrator/backend/internal/registry/ 2>&1 + ``` +- [ ] Output does NOT include a `git` directory. +- [ ] `grep -r "internal/registry/git" ~/src/orchestrator/backend/ 2>/dev/null | grep -v "^Binary"` returns nothing. + +### D3. `RegistryGitPath` config is gone + +- [ ] Run `grep -n "RegistryGitPath\|REGISTRY_GIT_PATH" ~/src/orchestrator/backend/internal/config/config.go`. +- [ ] Output is empty. + +### D4. `registry_versions.git_commit` / `git_tag` columns are gone + +- [ ] Run: + ```bash + podman exec blockninja-db psql -U orchestrator orchestrator -c "\d registry_versions" + ``` +- [ ] Output shows NO column named `git_commit` and NO column named `git_tag`. + +### D5. `Plugin.git_remote_url` is not in the proto + +- [ ] Run `grep -n "git_remote_url\|GitRemoteUrl" ~/src/orchestrator/proto/orchestrator/v1/plugin_registry.proto ~/src/core/proto/orchestrator/v1/plugin_registry.proto`. +- [ ] No matches. + +### D6. The CLI no longer references the `ninja` git remote + +- [ ] Run `grep -nE 'remote\s+(add|remove)\s+"?ninja' ~/src/core/cmd/ninja/cmd/plugin.go`. +- [ ] No matches. +- [ ] Run `grep -nE '"git",\s*"push",\s*"ninja"' ~/src/core/cmd/ninja/cmd/plugin.go`. +- [ ] No matches. + +--- + +## Group E — Smoke install path + +### E1. ResolveInstall returns the new .tar.zst URL + +- [ ] For any published plugin/version (e.g. uat-a1@0.1.0), call ResolveInstall: + ```bash + curl -sS -X POST \ + https://my.localdev.blockninjacms.com/orchestrator.v1.PluginRegistryService/ResolveInstall \ + -H 'Content-Type: application/json' \ + -d '{"scopeSlug":"themes","pluginName":"uat-a1","versionOrChannel":"0.1.0"}' + ``` +- [ ] Response contains `archiveUrl` whose path ends with `/source.tar.zst` and includes `sig=` and `exp=` query params. + +### E2. The signed URL actually downloads + +- [ ] `curl -sS -o /tmp/uat-e2.tar.zst ""` +- [ ] `file /tmp/uat-e2.tar.zst` reports a Zstandard compressed file. +- [ ] `sha256sum /tmp/uat-e2.tar.zst` matches the `archiveSha256` from the ResolveInstall response. +- [ ] `zstd -dc /tmp/uat-e2.tar.zst | tar -tf - | head` shows `plugin.mod` and the expected files. + +--- + +## Sign-off + +- [ ] Every box above is ticked, with at-the-time-of-execution observation logged inline. +- [ ] Any unticked box → work is NOT complete. Return to the plan. +- [ ] Final test reports any deviations from expected behaviour even when the box can be ticked. diff --git a/docs/superpowers/execution/2026-06-03-plugin-publish-and-categories-prompt.md b/docs/superpowers/execution/2026-06-03-plugin-publish-and-categories-prompt.md new file mode 100644 index 0000000..148a309 --- /dev/null +++ b/docs/superpowers/execution/2026-06-03-plugin-publish-and-categories-prompt.md @@ -0,0 +1,109 @@ +# Execution prompt — Plugin publish (tarball) + categories + +Copy the body below verbatim into a new Claude Code session at the repo +root (`~/src/core`) when you want execution to start. The prompt is +self-contained: anyone reading it should be able to drive the work end +to end without further context. + +--- + +## Begin prompt + +You are the inline execution agent for the "Plugin publish (tarball) + +categories" change. You are doing the work; you are not advising. Follow +the process below in order. Do not skip steps. Do not improvise. + +### 1. Required reading (in this order) + +Read these three files fully before touching any code. Treat each as +authoritative for its purpose. They live in `~/src/core`: + +1. **Spec / design** — `docs/superpowers/specs/2026-06-03-plugin-publish-and-categories-design.md`. Defines *what* and *why*. Do not change this without the user's explicit OK. +2. **Plan** — `docs/superpowers/plans/2026-06-03-plugin-publish-and-categories.md`. Defines the ordered, bite-sized tasks with full code blocks. Do not change this without the user's explicit OK either. Each task ends with a commit. +3. **UAT checklist** — `docs/superpowers/audits/2026-06-03-plugin-publish-and-categories-uat.md`. The hard gate. The work is NOT done until every UAT item has been observed to pass with your own eyes. The UAT may not be softened, skipped, or interpreted away. If any item turns out to be ambiguous or impossible, STOP and ask the user. + +Also be aware: work spans TWO git repos. +- `~/src/core` — Go module `git.dev.alexdunmow.com/block/core`. Holds the ninja CLI (`cmd/ninja/`), the SDK (`plugin/`), and a mirrored copy of the orchestrator proto under `proto/` + `internal/api/`. +- `~/src/orchestrator` — Go module at `backend/`. Holds the orchestrator service, schema, migrations, and proto source of truth. + +CLAUDE.md in each repo applies. Read `~/src/core/CLAUDE.md` if you have not (key rule: no `replace` directives in any `go.mod`). + +### 2. Process + +Invoke the `superpowers:executing-plans` skill and follow its discipline. +Within that: + +- Work tasks in the order written in the plan (Task 1 → Task 20 + Task 11.5). Do not reorder. The order encodes dependency. +- After completing each task, commit per the commit step in the task. Do not batch multiple tasks into one commit. Do not amend the user's previous commits. +- After each task's commit, run an internal checkpoint: did the build pass? did the tests added in this task pass? did this task actually do what the plan said? If anything is off, fix it before moving on. +- Stop and report at any of these points: + - Task instructions are unclear or contradicted by current code. + - A task's code block doesn't compile after applying it verbatim AND the fix isn't obvious from the build error. + - `make sqlc` produces field names that differ from what later tasks reference (this is plausible; sqlc field naming sometimes surprises). + - You need to install tooling that isn't already on the system (e.g. `buf`, `goose`, `sqlc`). +- Do NOT edit the plan or design to fit reality. If reality has diverged, surface that to the user and ask how to resolve. + +### 3. When to dispatch subagents + +The plan is mostly sequential. Most tasks should be executed inline by +you. Dispatch a `general-purpose` subagent ONLY for the tasks listed +below. These are isolated, pure-function, well-bounded units whose work +benefits from a clean context. Hand the subagent the full task body +including all code blocks; tell it the working directory; tell it the +exact files to create or modify; tell it not to commit (you commit after +reviewing the changes). + +Subagent-suitable tasks: + +- **Task 3** — `[core] BuildSourceArchive`. Pure helper in a new package. Self-contained TDD. +- **Task 4** — `[orch] OpenAndValidate`. Pure helper, TDD, no external state. +- **Task 5** — `[orch] Drop git fields migration + sqlc regen`. Mechanical schema + regen. Pure file manipulation. +- **Task 14** — `[orch] kind + categories migration + sqlc regen`. Same shape as Task 5. + +All other tasks (proto changes, handler rewrites, wiring, CLI changes, +delete-the-git-package) you do yourself in the main context. Reasons: +they cross many files, depend on context from earlier tasks, and require +running and reading the orchestrator/CLI in your hands. + +Smoke tests (Tasks 12, 20) and UAT execution: do these yourself in the +main context. Subagents cannot easily run interactive flows or report +back observations of running services. + +### 4. UAT gate (HARD) + +After Task 20 (and Task 11.5) are committed and pushed-ready in both +repos, execute the UAT checklist +(`docs/superpowers/audits/2026-06-03-plugin-publish-and-categories-uat.md`) +top to bottom. For each box: + +- Run the command(s) the item specifies, against the actual orchestrator + (you may need to `podman compose build orchestrator-backend && + podman compose up -d orchestrator-backend` first) and the actual + installed CLI (`cd ~/src/core && go install ./cmd/ninja`). +- Observe the result with your own eyes. +- Tick the box ONLY if the observation matches the expectation. +- If it doesn't match: do not tick. Return to the plan, identify which + task is implicated, and either fix it (new commit) or report back. + +You may not declare the work complete with any UAT box unticked. You +may not pre-tick boxes "based on the test suite passing". UAT is the +gate; tests are not. + +If an UAT item is ambiguous, ask the user before guessing. + +### 5. Tone + +Short status updates between tasks. State what's next in one line. State +problems concretely with the command and output that demonstrated the +problem. No speculation, no apologies. When asking the user, present +specific options. + +### 6. Done + +When all UAT items are ticked, summarise in one message: +- Total commits made across both repos. +- Any deviations from the plan (with task numbers and one-line justification). +- Status of each UAT group (A–E) — all pass. +- Suggested next step (likely: announce the change in the team channel, or move on to a follow-up like an admin UI for category management — both out of scope here). + +## End prompt diff --git a/docs/superpowers/plans/2026-06-03-plugin-publish-and-categories.md b/docs/superpowers/plans/2026-06-03-plugin-publish-and-categories.md new file mode 100644 index 0000000..7ffb34c --- /dev/null +++ b/docs/superpowers/plans/2026-06-03-plugin-publish-and-categories.md @@ -0,0 +1,2350 @@ +# Plugin publish (tarball) + categories — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace the git-based publish flow with `tar.zst` uploads, and add a first-class plugin `kind` (`plugin` | `theme`) plus a server-validated category list driven from a canonical seed table. + +**Architecture:** The CLI builds a `tar.zst` from `git archive HEAD` and posts it through the existing Connect RPC. The orchestrator decompresses, extracts and validates `plugin.mod`, and stores the archive in `regstorage` (the existing disk/S3 layer). All git-server code, bare repos, and the `ninja` git remote dance are deleted. A new `registry_categories` table and a `kind` column on `registry_plugins` model the new metadata; `ListCategories` feeds the CLI's category picker. + +**Tech Stack:** Go (orchestrator backend + ninja CLI), Connect RPC over HTTP, Postgres + sqlc + goose, klauspost/compress/zstd, stdlib `archive/tar`. + +**Repos involved:** +- `~/src/orchestrator` (backend module at `backend/`) +- `~/src/core` (ninja CLI at `cmd/ninja/`, SDK at `plugin/`) + +Tasks are tagged `[orch]` or `[core]` to indicate which repo they live in. Proto change tasks touch both repos. + +--- + +## File Structure + +**`~/src/orchestrator/backend/` changes:** +- Create: `internal/registry/archive/archive.go` — pure helpers: open tar.zst, walk to `plugin.mod`, enforce size caps. +- Create: `internal/registry/archive/archive_test.go` +- Create: `internal/registry/api/category.go` — `ListCategories` handler. +- Create: `sql/migrations/00055_drop_publish_git_fields.sql` +- Create: `sql/migrations/00056_plugin_kind_and_categories.sql` +- Create: `sql/queries/registry_categories.sql` +- Modify: `proto/orchestrator/v1/plugin_registry.proto` — phase 1 + phase 2 proto changes. +- Modify: `internal/registry/api/publish.go` — replace git flow with archive flow. +- Modify: `internal/registry/api/plugin.go` — drop `GitRemoteUrl`; accept `kind` + `categories`; validate categories. +- Modify: `internal/registry/api/converters.go` — drop `GitCommit`/`GitTag`; add `Kind`. +- Modify: `internal/server/registry_wiring.go` — remove git mount/server; instantiate category handler. +- Modify: `internal/config/config.go` — drop `RegistryGitPath`; add `RegistryPublishMaxBytes`. +- Modify: `sql/queries/registry_plugins.sql` — add kind support and filter args. +- Modify: `sql/queries/registry_versions.sql` — drop git fields from inserts/selects. +- Delete: `internal/registry/git/` (entire package). +- Auto-regen: `backend/api/orchestrator/v1/plugin_registry.pb.go` + connect file via `make proto`. +- Auto-regen: `internal/db/*.sql.go` via `make sqlc`. +- Auto-regen: `sql/schema.sql` via `make schema`. + +**`~/src/core/` changes:** +- Create: `cmd/ninja/internal/archive/archive.go` — pure helper to build `tar.zst` from `git archive HEAD`. +- Create: `cmd/ninja/internal/archive/archive_test.go` +- Modify: `proto/orchestrator/v1/plugin_registry.proto` — mirror orchestrator's proto changes. +- Modify: `plugin/mod.go` — add `Kind` and `Categories` fields to `ModPlugin`. +- Modify: `plugin/mod_test.go` — cover new fields. +- Modify: `cmd/ninja/cmd/plugin.go` — rewrite `init` (auto-commit, no remote, prompt kind+categories) and `publish` (build archive, no tag, no push). +- Auto-regen: `internal/api/orchestrator/v1/plugin_registry.pb.go` + connect file via `buf generate`. + +--- + +# Phase 1: Tarball publish (drop git) + +## Task 1: [core] Add klauspost/compress dependency + +**Files:** +- Modify: `~/src/core/go.mod`, `~/src/core/go.sum` + +- [ ] **Step 1: Fetch the dep** + +```bash +cd ~/src/core +go get github.com/klauspost/compress/zstd@latest +go mod tidy +``` + +- [ ] **Step 2: Verify it's resolvable** + +```bash +go build ./... +``` +Expected: clean (no errors). + +- [ ] **Step 3: Commit** + +```bash +git add go.mod go.sum +git commit -m "chore(core): add klauspost/compress for plugin archive zstd" +``` + +## Task 2: [orch] Add klauspost/compress dependency + +**Files:** +- Modify: `~/src/orchestrator/backend/go.mod`, `~/src/orchestrator/backend/go.sum` + +- [ ] **Step 1: Fetch the dep** + +```bash +cd ~/src/orchestrator/backend +go get github.com/klauspost/compress/zstd@latest +go mod tidy +``` + +- [ ] **Step 2: Verify it's resolvable** + +```bash +go build ./... +``` +Expected: clean. + +- [ ] **Step 3: Commit** + +```bash +git add go.mod go.sum +git commit -m "chore(orch): add klauspost/compress for plugin archive zstd" +``` + +## Task 3: [core] CLI archive helper (`BuildSourceArchive`) + +Pure helper that runs `git archive --format=tar HEAD` inside a repo and pipes +through zstd. Returns the compressed bytes. Lives in +`cmd/ninja/internal/archive/` so the CLI can use it but nothing else. + +**Files:** +- Create: `~/src/core/cmd/ninja/internal/archive/archive.go` +- Create: `~/src/core/cmd/ninja/internal/archive/archive_test.go` + +- [ ] **Step 1: Write the failing test** + +`~/src/core/cmd/ninja/internal/archive/archive_test.go`: + +```go +package archive + +import ( + "archive/tar" + "bytes" + "io" + "os" + "os/exec" + "path/filepath" + "testing" + + "github.com/klauspost/compress/zstd" +) + +func TestBuildSourceArchive_RoundTrip(t *testing.T) { + dir := t.TempDir() + run := func(name string, args ...string) { + t.Helper() + cmd := exec.Command(name, args...) + cmd.Dir = dir + cmd.Env = append(os.Environ(), + "GIT_AUTHOR_NAME=t", + "GIT_AUTHOR_EMAIL=t@t", + "GIT_COMMITTER_NAME=t", + "GIT_COMMITTER_EMAIL=t@t", + ) + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("%s %v: %v\n%s", name, args, err, out) + } + } + run("git", "init", "-q") + if err := os.WriteFile(filepath.Join(dir, "plugin.mod"), + []byte("[plugin]\nname=\"x\"\nscope=\"@s\"\nversion=\"0.1.0\"\n"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, "ignored.log"), []byte("nope"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, ".gitignore"), []byte("ignored.log\n"), 0o644); err != nil { + t.Fatal(err) + } + run("git", "add", "plugin.mod", ".gitignore") + run("git", "commit", "-qm", "init") + + zstdBytes, err := BuildSourceArchive(dir) + if err != nil { + t.Fatalf("BuildSourceArchive: %v", err) + } + if len(zstdBytes) == 0 { + t.Fatal("empty archive") + } + + dec, err := zstd.NewReader(bytes.NewReader(zstdBytes)) + if err != nil { + t.Fatal(err) + } + defer dec.Close() + tr := tar.NewReader(dec) + got := map[string]string{} + for { + hdr, err := tr.Next() + if err == io.EOF { + break + } + if err != nil { + t.Fatal(err) + } + buf, err := io.ReadAll(tr) + if err != nil { + t.Fatal(err) + } + got[hdr.Name] = string(buf) + } + if _, ok := got["plugin.mod"]; !ok { + t.Errorf("expected plugin.mod in archive, got %v", keys(got)) + } + if _, ok := got["ignored.log"]; ok { + t.Errorf("ignored.log should not be in archive (gitignored + untracked)") + } +} + +func keys(m map[string]string) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + return out +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +```bash +cd ~/src/core +go test ./cmd/ninja/internal/archive/... +``` +Expected: FAIL — package doesn't exist yet. + +- [ ] **Step 3: Implement** + +`~/src/core/cmd/ninja/internal/archive/archive.go`: + +```go +package archive + +import ( + "bytes" + "fmt" + "io" + "os/exec" + + "github.com/klauspost/compress/zstd" +) + +// BuildSourceArchive runs `git archive --format=tar HEAD` in repoDir and +// compresses the result with zstd. Returns the compressed bytes. +// +// Only tracked files at HEAD are included. .gitignored files that were +// never tracked are excluded. Tracked-then-gitignored files are still +// included — callers may warn separately. +func BuildSourceArchive(repoDir string) ([]byte, error) { + cmd := exec.Command("git", "archive", "--format=tar", "HEAD") + cmd.Dir = repoDir + var tarOut, stderr bytes.Buffer + cmd.Stdout = &tarOut + cmd.Stderr = &stderr + if err := cmd.Run(); err != nil { + return nil, fmt.Errorf("git archive: %v: %s", err, stderr.String()) + } + + var compressed bytes.Buffer + enc, err := zstd.NewWriter(&compressed, zstd.WithEncoderLevel(zstd.SpeedDefault)) + if err != nil { + return nil, err + } + if _, err := io.Copy(enc, &tarOut); err != nil { + _ = enc.Close() + return nil, err + } + if err := enc.Close(); err != nil { + return nil, err + } + return compressed.Bytes(), nil +} +``` + +- [ ] **Step 4: Run tests to verify they pass** + +```bash +go test ./cmd/ninja/internal/archive/... +``` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add cmd/ninja/internal/archive/ +git commit -m "feat(cli): add BuildSourceArchive for plugin publish tar.zst" +``` + +## Task 4: [orch] Archive validation helper (`OpenAndValidate`) + +Pure helper that takes the compressed bytes, enforces the size caps, +decompresses, walks the tar, extracts `plugin.mod`, and returns parsed +contents. + +**Files:** +- Create: `~/src/orchestrator/backend/internal/registry/archive/archive.go` +- Create: `~/src/orchestrator/backend/internal/registry/archive/archive_test.go` + +- [ ] **Step 1: Write the failing test** + +`~/src/orchestrator/backend/internal/registry/archive/archive_test.go`: + +```go +package archive + +import ( + "archive/tar" + "bytes" + "testing" + + "github.com/klauspost/compress/zstd" +) + +func buildTestArchive(t *testing.T, files map[string][]byte) []byte { + t.Helper() + var tarBuf bytes.Buffer + tw := tar.NewWriter(&tarBuf) + for name, body := range files { + if err := tw.WriteHeader(&tar.Header{ + Name: name, Mode: 0o644, Size: int64(len(body)), + }); err != nil { + t.Fatal(err) + } + if _, err := tw.Write(body); err != nil { + t.Fatal(err) + } + } + if err := tw.Close(); err != nil { + t.Fatal(err) + } + var out bytes.Buffer + enc, _ := zstd.NewWriter(&out) + _, _ = enc.Write(tarBuf.Bytes()) + _ = enc.Close() + return out.Bytes() +} + +func TestOpenAndValidate_ReturnsModBytes(t *testing.T) { + modBody := []byte("[plugin]\nname=\"x\"\nscope=\"@s\"\nversion=\"0.1.0\"\n") + archive := buildTestArchive(t, map[string][]byte{ + "plugin.mod": modBody, + "readme.md": []byte("hi"), + }) + got, err := OpenAndValidate(archive, 1<<20, 4<<20) + if err != nil { + t.Fatalf("OpenAndValidate: %v", err) + } + if !bytes.Equal(got, modBody) { + t.Errorf("plugin.mod bytes mismatch") + } +} + +func TestOpenAndValidate_MissingModFile(t *testing.T) { + archive := buildTestArchive(t, map[string][]byte{"readme.md": []byte("hi")}) + if _, err := OpenAndValidate(archive, 1<<20, 4<<20); err == nil { + t.Fatal("expected error for missing plugin.mod") + } +} + +func TestOpenAndValidate_RejectsTooLargeCompressed(t *testing.T) { + archive := buildTestArchive(t, map[string][]byte{"plugin.mod": []byte("x")}) + if _, err := OpenAndValidate(archive, 1, 4<<20); err == nil { + t.Fatal("expected compressed-size error") + } +} + +func TestOpenAndValidate_RejectsDecompressionBomb(t *testing.T) { + big := bytes.Repeat([]byte("a"), 5<<20) // 5 MiB of 'a' compresses tiny + archive := buildTestArchive(t, map[string][]byte{ + "plugin.mod": []byte("[plugin]\n"), + "bomb": big, + }) + if _, err := OpenAndValidate(archive, 10<<20, 1<<20); err == nil { + t.Fatal("expected decompressed-size error") + } +} +``` + +- [ ] **Step 2: Run test to verify failure** + +```bash +cd ~/src/orchestrator/backend +go test ./internal/registry/archive/... +``` +Expected: FAIL — package doesn't exist. + +- [ ] **Step 3: Implement** + +`~/src/orchestrator/backend/internal/registry/archive/archive.go`: + +```go +package archive + +import ( + "archive/tar" + "bytes" + "errors" + "fmt" + "io" + + "github.com/klauspost/compress/zstd" +) + +// OpenAndValidate verifies the compressed archive is within the size cap, +// decompresses it (bounded by maxDecompressed to limit zstd-bomb exposure), +// walks the tar entries, and returns the bytes of plugin.mod. +func OpenAndValidate(compressed []byte, maxCompressed, maxDecompressed int) ([]byte, error) { + if len(compressed) > maxCompressed { + return nil, fmt.Errorf("archive too large: %d bytes (cap %d)", len(compressed), maxCompressed) + } + dec, err := zstd.NewReader(bytes.NewReader(compressed)) + if err != nil { + return nil, fmt.Errorf("zstd: %w", err) + } + defer dec.Close() + + limited := &countingReader{r: io.LimitReader(dec, int64(maxDecompressed)+1), cap: int64(maxDecompressed)} + tr := tar.NewReader(limited) + + var modBytes []byte + for { + hdr, err := tr.Next() + if err == io.EOF { + break + } + if err != nil { + return nil, fmt.Errorf("tar: %w", err) + } + if limited.over { + return nil, errors.New("archive decompressed past size cap") + } + if hdr.Name == "plugin.mod" { + buf, err := io.ReadAll(tr) + if err != nil { + return nil, err + } + modBytes = buf + } + } + if modBytes == nil { + return nil, errors.New("plugin.mod not found in archive") + } + return modBytes, nil +} + +type countingReader struct { + r io.Reader + read int64 + cap int64 + over bool +} + +func (c *countingReader) Read(p []byte) (int, error) { + n, err := c.r.Read(p) + c.read += int64(n) + if c.read > c.cap { + c.over = true + } + return n, err +} +``` + +- [ ] **Step 4: Run tests** + +```bash +go test ./internal/registry/archive/... +``` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add internal/registry/archive/ +git commit -m "feat(orch): add archive helper for tar.zst publish flow" +``` + +## Task 5: [orch] Migration — drop git fields from registry_versions + +Goose migration that drops `git_commit` and `git_tag` columns on +`registry_versions`. These are not used after the publish flow change. + +**Files:** +- Create: `~/src/orchestrator/backend/sql/migrations/00055_drop_publish_git_fields.sql` +- Modify: `~/src/orchestrator/backend/sql/queries/registry_versions.sql` +- Modify: `~/src/orchestrator/backend/sql/schema.sql` (auto via `make schema`) +- Regen: `~/src/orchestrator/backend/internal/db/*.sql.go` (auto via `make sqlc`) + +- [ ] **Step 1: Write the migration** + +`backend/sql/migrations/00055_drop_publish_git_fields.sql`: + +```sql +-- +goose Up +ALTER TABLE registry_versions DROP COLUMN git_commit; +ALTER TABLE registry_versions DROP COLUMN git_tag; + +-- +goose Down +ALTER TABLE registry_versions ADD COLUMN git_commit text NOT NULL DEFAULT ''; +ALTER TABLE registry_versions ADD COLUMN git_tag text NOT NULL DEFAULT ''; +``` + +- [ ] **Step 2: Update queries that reference the dropped columns** + +Open `backend/sql/queries/registry_versions.sql`. Find any query that +references `git_commit` or `git_tag` in `SELECT`, `INSERT`, or `RETURNING` +clauses and remove those columns. Specifically: +- `CreateRegistryVersion`: remove `git_commit` and `git_tag` from the column list and values. +- Any `SELECT` that returns these — drop them. + +For example, before: +```sql +INSERT INTO registry_versions ( + plugin_id, version, git_commit, git_tag, published_by_user_id, ... +) VALUES ($1, $2, $3, $4, $5, ...) RETURNING *; +``` + +After: +```sql +INSERT INTO registry_versions ( + plugin_id, version, published_by_user_id, ... +) VALUES ($1, $2, $3, ...) RETURNING *; +``` + +Renumber the placeholders accordingly. + +- [ ] **Step 3: Run the migration** + +```bash +cd ~/src/orchestrator/backend +make migrate +``` +Expected: migrations applied through `00055`. + +- [ ] **Step 4: Regenerate schema and sqlc** + +```bash +make schema +make sqlc +``` + +Expected: +- `sql/schema.sql` updated (no `git_commit`/`git_tag` on `registry_versions`). +- `internal/db/registry_versions.sql.go` regenerated. + +- [ ] **Step 5: Verify build (will fail in some files — that's the next task)** + +```bash +go build ./... +``` +Expected: failures in `internal/registry/api/converters.go` and `publish.go` +that reference `v.GitCommit` / `v.GitTag`. Those are fixed in Task 7 once +the proto change goes in. Leave the build broken until Task 7. + +- [ ] **Step 6: Commit the migration and query changes** + +```bash +git add sql/migrations/00055_drop_publish_git_fields.sql sql/queries/registry_versions.sql sql/schema.sql internal/db/registry_versions.sql.go +git commit -m "feat(orch): drop git_commit/git_tag from registry_versions" +``` + +## Task 6: Proto change — `PublishVersionRequest` switches to `bytes archive` + +`PublishVersionRequest` drops `git_ref` and gains `string version` + `bytes archive`. `CreatePluginResponse` drops `git_remote_url`. Apply in both proto files and regen. + +**Files:** +- Modify: `~/src/orchestrator/proto/orchestrator/v1/plugin_registry.proto` +- Modify: `~/src/core/proto/orchestrator/v1/plugin_registry.proto` +- Regen: `~/src/orchestrator/backend/api/orchestrator/v1/plugin_registry.pb.go` + connect file (via `cd ~/src/orchestrator && make -C backend proto` or `buf generate`) +- Regen: `~/src/core/internal/api/orchestrator/v1/plugin_registry.pb.go` + connect file (via `cd ~/src/core && buf generate`) + +- [ ] **Step 1: Edit orchestrator proto** + +In `~/src/orchestrator/proto/orchestrator/v1/plugin_registry.proto`, replace: + +```proto +message CreatePluginResponse { + Plugin plugin = 1; + string git_remote_url = 2; +} +``` + +with: + +```proto +message CreatePluginResponse { + Plugin plugin = 1; +} +``` + +And replace: + +```proto +message PublishVersionRequest { + string plugin_id = 1; + string git_ref = 2; + string channel = 3; + string readme_md = 4; + string changelog_md = 5; +} +``` + +with: + +```proto +message PublishVersionRequest { + string plugin_id = 1; + string version = 2; + string channel = 3; + bytes archive = 4; + string readme_md = 5; + string changelog_md = 6; +} +``` + +- [ ] **Step 2: Mirror to core proto** + +Apply the identical changes to `~/src/core/proto/orchestrator/v1/plugin_registry.proto`. (Note: the two protos have small drift — keep core's drift intact and apply only the two diffs above.) + +- [ ] **Step 3: Regen orchestrator** + +```bash +cd ~/src/orchestrator +make -C backend proto +``` +Expected: `backend/api/orchestrator/v1/plugin_registry.pb.go` regenerated. + +- [ ] **Step 4: Regen core** + +```bash +cd ~/src/core +buf generate +``` +Expected: `internal/api/orchestrator/v1/plugin_registry.pb.go` regenerated. + +- [ ] **Step 5: Commit each repo** + +In orchestrator: +```bash +cd ~/src/orchestrator +git add proto/ backend/api/ frontend/src/lib/api/gen/ +git commit -m "feat(orch): PublishVersionRequest takes bytes archive; drop git_remote_url" +``` + +In core: +```bash +cd ~/src/core +git add proto/ internal/api/ +git commit -m "feat(core): mirror PublishVersionRequest archive bytes proto change" +``` + +## Task 7: [orch] Rewrite `PublishVersion` handler + +Replace the git-based flow with the archive flow: validate the archive, +extract `plugin.mod`, cross-check fields, store via `regstorage`, record +the version. No git, no `repos`, no tag. + +**Files:** +- Modify: `~/src/orchestrator/backend/internal/registry/api/publish.go` +- Modify: `~/src/orchestrator/backend/internal/registry/api/converters.go` + +- [ ] **Step 1: Replace publish.go contents** + +Open `~/src/orchestrator/backend/internal/registry/api/publish.go` and replace its contents with: + +```go +package api + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "strings" + "time" + + "connectrpc.com/connect" + "github.com/Masterminds/semver/v3" + "github.com/google/uuid" + "github.com/jackc/pgx/v5/pgxpool" + "bytes" + + internalauth "git.dev.alexdunmow.com/block/ninja/orchestrator/internal/auth" + "git.dev.alexdunmow.com/block/ninja/orchestrator/internal/db" + regarchive "git.dev.alexdunmow.com/block/ninja/orchestrator/internal/registry/archive" + regstorage "git.dev.alexdunmow.com/block/ninja/orchestrator/internal/registry/storage" + + core "git.dev.alexdunmow.com/block/core/plugin" + orchestratorv1 "git.dev.alexdunmow.com/block/ninja/orchestrator/api/orchestrator/v1" +) + +type PublishHandler struct { + q *db.Queries + pool *pgxpool.Pool + storage regstorage.Storage + signingKey []byte + publicURL string + maxCompressed int + maxDecompressed int +} + +func NewPublishHandler(q *db.Queries, pool *pgxpool.Pool, st regstorage.Storage, signing []byte, publicURL string, maxCompressed int) *PublishHandler { + return &PublishHandler{ + q: q, + pool: pool, + storage: st, + signingKey: signing, + publicURL: publicURL, + maxCompressed: maxCompressed, + maxDecompressed: maxCompressed * 4, + } +} + +func (h *PublishHandler) PublishVersion(ctx context.Context, req *connect.Request[orchestratorv1.PublishVersionRequest]) (*connect.Response[orchestratorv1.PublishVersionResponse], error) { + claims, ok := internalauth.GetUserFromContext(ctx) + if !ok { + return nil, connect.NewError(connect.CodeUnauthenticated, errors.New("login required")) + } + + pluginID, err := uuid.Parse(req.Msg.PluginId) + if err != nil { + return nil, connect.NewError(connect.CodeInvalidArgument, errors.New("invalid plugin_id")) + } + plugin, err := h.q.GetRegistryPluginByID(ctx, pluginID) + if err != nil { + return nil, connect.NewError(connect.CodeNotFound, err) + } + yes, err := h.q.IsRegistryScopeMember(ctx, db.IsRegistryScopeMemberParams{ScopeID: plugin.ScopeID, UserID: claims.UserID}) + if err != nil || !yes { + return nil, connect.NewError(connect.CodePermissionDenied, errors.New("not a scope member")) + } + scope, err := h.q.GetRegistryScopeByID(ctx, plugin.ScopeID) + if err != nil { + return nil, err + } + + ver, err := semver.NewVersion(req.Msg.Version) + if err != nil { + return nil, connect.NewError(connect.CodeInvalidArgument, errors.New("invalid version: "+err.Error())) + } + + if _, err := h.q.GetRegistryVersionByPluginAndVersion(ctx, db.GetRegistryVersionByPluginAndVersionParams{ + PluginID: plugin.ID, Version: ver.String(), + }); err == nil { + return nil, connect.NewError(connect.CodeAlreadyExists, errors.New("version already published")) + } + + modBytes, err := regarchive.OpenAndValidate(req.Msg.Archive, h.maxCompressed, h.maxDecompressed) + if err != nil { + return nil, connect.NewError(connect.CodeInvalidArgument, err) + } + mod, err := core.ParseModFull(modBytes) + if err != nil { + return nil, connect.NewError(connect.CodeInvalidArgument, err) + } + modScope := strings.TrimPrefix(mod.Plugin.Scope, "@") + if modScope != scope.Slug || mod.Plugin.Name != plugin.Name { + return nil, connect.NewError(connect.CodeInvalidArgument, errors.New("plugin.mod scope/name mismatch")) + } + if mod.Plugin.Version != ver.String() { + return nil, connect.NewError(connect.CodeInvalidArgument, errors.New("plugin.mod version mismatch with request")) + } + + sum := sha256.Sum256(req.Msg.Archive) + sha := hex.EncodeToString(sum[:]) + storageKey := scope.Slug + "/" + plugin.Name + "/" + ver.String() + "/source.tar.zst" + if err := h.storage.Put(ctx, storageKey, bytes.NewReader(req.Msg.Archive), int64(len(req.Msg.Archive)), "application/zstd"); err != nil { + return nil, err + } + + manifest, _ := json.Marshal(mod) + sdkConstraint := "" + if mod.Compatibility != nil { + sdkConstraint = mod.Compatibility.BlockCore + } + + tx, err := h.pool.Begin(ctx) + if err != nil { + return nil, err + } + defer func() { _ = tx.Rollback(ctx) }() + qtx := h.q.WithTx(tx) + + row, err := qtx.CreateRegistryVersion(ctx, db.CreateRegistryVersionParams{ + PluginID: plugin.ID, + Version: ver.String(), + PublishedByUserID: claims.UserID, + SdkConstraint: sdkConstraint, + ManifestJson: manifest, + ReadmeMd: req.Msg.ReadmeMd, + ChangelogMd: req.Msg.ChangelogMd, + SourceArchiveKey: storageKey, + SourceArchiveSha256: sha, + SourceArchiveSize: int64(len(req.Msg.Archive)), + }) + if err != nil { + return nil, err + } + + warnings := []string{} + for _, r := range mod.Requires { + reqScope, reqName, ok := splitRef(r.Name) + if !ok { + warnings = append(warnings, "ignoring malformed requires entry: "+r.Name) + continue + } + reqScopeRow, err := qtx.GetRegistryScopeBySlug(ctx, reqScope) + if err != nil { + warnings = append(warnings, "required scope not found: @"+reqScope) + continue + } + reqPlugin, err := qtx.GetRegistryPluginByScopeAndName(ctx, db.GetRegistryPluginByScopeAndNameParams{ + ScopeID: reqScopeRow.ID, Name: reqName, + }) + if err != nil { + warnings = append(warnings, "required plugin not found: @"+reqScope+"/"+reqName) + continue + } + if err := qtx.AddRegistryVersionRequire(ctx, db.AddRegistryVersionRequireParams{ + VersionID: row.ID, RequiredPluginID: reqPlugin.ID, ConstraintExpr: r.Version, + }); err != nil { + return nil, err + } + } + + channel := strings.TrimSpace(req.Msg.Channel) + if channel == "" { + channel = "latest" + } + if ver.Prerelease() == "" || channel != "latest" { + if err := qtx.SetRegistryChannel(ctx, db.SetRegistryChannelParams{ + PluginID: plugin.ID, Name: channel, VersionID: row.ID, + }); err != nil { + return nil, err + } + } else { + warnings = append(warnings, "skipped auto-bumping 'latest' to prerelease "+ver.String()) + } + + if err := tx.Commit(ctx); err != nil { + return nil, err + } + + signedURL, err := regstorage.Sign(h.signingKey, h.publicURL+"/registry/storage", storageKey, 10*time.Minute) + if err != nil { + return nil, err + } + return connect.NewResponse(&orchestratorv1.PublishVersionResponse{ + Version: versionToProto(row), + ArchiveUrl: signedURL, + Warnings: warnings, + }), nil +} + +func splitRef(ref string) (scope, name string, ok bool) { + if !strings.HasPrefix(ref, "@") { + return "", "", false + } + parts := strings.SplitN(strings.TrimPrefix(ref, "@"), "/", 2) + if len(parts) != 2 { + return "", "", false + } + return parts[0], parts[1], true +} +``` + +- [ ] **Step 2: Update `versionToProto` to drop git fields** + +In `~/src/orchestrator/backend/internal/registry/api/converters.go`, replace `versionToProto` with: + +```go +func versionToProto(v db.RegistryVersion) *orchestratorv1.Version { + return &orchestratorv1.Version{ + Id: v.ID.String(), + PluginId: v.PluginID.String(), + Version: v.Version, + PublishedAt: timestamppb.New(v.PublishedAt), + Yanked: v.YankedAt.Valid, + SdkConstraint: v.SdkConstraint, + SourceArchiveSha256: v.SourceArchiveSha256, + SourceArchiveSize: v.SourceArchiveSize, + } +} +``` + +(`GitCommit`/`GitTag` fields are removed from `Version` proto in the same regen — leave the proto file alone for now; the regen on the next step picks up the dropped fields naturally because we'll regen after we delete them in Task 9. For now, the proto still has the fields but the converter ignores them. This intentionally leaves transient redundancy — cleaned up in Task 9.) + +Actually: change the plan above — drop `Version.git_commit` and `Version.git_tag` from the proto file as part of Task 6's edits, and regen. That makes Task 7 clean. + +Go back to Task 6 and add: remove `string git_commit = 4;` and `string git_tag = 5;` from the `Version` proto message, then regenerate. The converter snippet above is then correct as-is. + +- [ ] **Step 3: Verify build** + +```bash +cd ~/src/orchestrator/backend +go build ./... +``` +Expected: clean (Task 5's `git_commit`/`git_tag` references now resolved by the new converter). + +- [ ] **Step 4: Run the archive tests** + +```bash +go test ./internal/registry/archive/... +``` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add internal/registry/api/publish.go internal/registry/api/converters.go +git commit -m "feat(orch): rewrite PublishVersion to use tar.zst archive" +``` + +## Task 8: [orch] Drop `git_remote_url` and update `CreatePlugin` handler + +**Files:** +- Modify: `~/src/orchestrator/backend/internal/registry/api/plugin.go` + +- [ ] **Step 1: Remove `git_remote_url` from CreatePlugin response** + +In `internal/registry/api/plugin.go`, in `CreatePlugin`, find this block: + +```go +if _, err := h.repos.Provision(ctx, slug, name); err != nil { + return nil, err +} +remote, err := url.JoinPath(h.publicURL, "git", slug, name+".git") +if err != nil { + return nil, err +} +return connect.NewResponse(&orchestratorv1.CreatePluginResponse{ + Plugin: pluginToProto(p, slug), + GitRemoteUrl: remote, +}), nil +``` + +Replace with: + +```go +return connect.NewResponse(&orchestratorv1.CreatePluginResponse{ + Plugin: pluginToProto(p, slug), +}), nil +``` + +- [ ] **Step 2: Remove `repos` field from `PluginHandler`** + +Replace the struct + constructor with: + +```go +type PluginHandler struct { + q *db.Queries + publicURL string + signingKey []byte +} + +func NewPluginHandler(q *db.Queries, publicURL string, signingKey []byte) *PluginHandler { + return &PluginHandler{q: q, publicURL: publicURL, signingKey: signingKey} +} +``` + +Remove the `reggit` import. + +- [ ] **Step 3: Remove unused imports** + +`net/url` is no longer used — remove it. Build will tell you about anything else. + +- [ ] **Step 4: Verify build (will fail in wiring — fixed Task 9)** + +```bash +go build ./internal/registry/api/... +``` +Expected: clean for the api package. + +```bash +go build ./... +``` +Expected: failure in `internal/server/registry_wiring.go` because the constructor signature changed. Fixed in Task 9. + +- [ ] **Step 5: Commit** + +```bash +git add internal/registry/api/plugin.go +git commit -m "feat(orch): drop GitRemoteUrl from CreatePlugin response" +``` + +## Task 9: [orch] Delete `internal/registry/git/`, wiring + config cleanup + +**Files:** +- Delete: `~/src/orchestrator/backend/internal/registry/git/` (entire directory) +- Modify: `~/src/orchestrator/backend/internal/server/registry_wiring.go` +- Modify: `~/src/orchestrator/backend/internal/config/config.go` + +- [ ] **Step 1: Delete the git package** + +```bash +cd ~/src/orchestrator/backend +git rm -r internal/registry/git/ +``` + +- [ ] **Step 2: Update config — drop `RegistryGitPath`, add `RegistryPublishMaxBytes`** + +In `internal/config/config.go`, remove the `RegistryGitPath` field and its getEnv line. Add a `RegistryPublishMaxBytes int` field, populated as: + +```go +cfg.RegistryPublishMaxBytes = getEnvInt("REGISTRY_PUBLISH_MAX_BYTES", 25*1024*1024) +``` + +(`getEnvInt` may not exist — if not, add a small helper that mirrors `getEnv`:) + +```go +func getEnvInt(key string, def int) int { + if v := os.Getenv(key); v != "" { + if n, err := strconv.Atoi(v); err == nil { + return n + } + } + return def +} +``` + +Add `strconv` to imports if not present. + +- [ ] **Step 3: Update wiring — drop git mount, RepoManager, gitServer** + +Replace `internal/server/registry_wiring.go` with: + +```go +package server + +import ( + "context" + "io" + "net/http" + "strings" + + "connectrpc.com/connect" + "github.com/go-chi/chi/v5" + "github.com/jackc/pgx/v5/pgxpool" + + "git.dev.alexdunmow.com/block/ninja/orchestrator/api/orchestrator/v1/orchestratorv1connect" + "git.dev.alexdunmow.com/block/ninja/orchestrator/internal/config" + "git.dev.alexdunmow.com/block/ninja/orchestrator/internal/db" + regapi "git.dev.alexdunmow.com/block/ninja/orchestrator/internal/registry/api" + regauth "git.dev.alexdunmow.com/block/ninja/orchestrator/internal/registry/auth" + regstorage "git.dev.alexdunmow.com/block/ninja/orchestrator/internal/registry/storage" +) + +type RegistryBundle struct { + scopeHandler *regapi.ScopeHandler + pluginHandler *regapi.PluginHandler + publishHandler *regapi.PublishHandler + deviceHandler *regauth.DeviceHandler + storage regstorage.Storage + signingKey []byte +} + +func BuildRegistry(ctx context.Context, q *db.Queries, pool *pgxpool.Pool, cfg *config.Config) (*RegistryBundle, error) { + var st regstorage.Storage + if cfg.RegistryS3Bucket != "" { + s3, err := regstorage.NewS3(ctx, regstorage.S3Config{ + Bucket: cfg.RegistryS3Bucket, + Endpoint: cfg.RegistryS3Endpoint, + Region: cfg.RegistryS3Region, + AccessKeyID: cfg.RegistryS3AccessKeyID, + SecretKey: cfg.RegistryS3SecretKey, + }) + if err != nil { + return nil, err + } + st = s3 + } else { + st = regstorage.NewDisk(cfg.RegistryStoragePath) + } + + signingKey := cfg.RegistryArchiveSigningKey + publicURL := strings.TrimRight(cfg.RegistryPublicBaseURL, "/") + + return &RegistryBundle{ + scopeHandler: regapi.NewScopeHandler(q), + pluginHandler: regapi.NewPluginHandler(q, publicURL, signingKey), + publishHandler: regapi.NewPublishHandler(q, pool, st, signingKey, publicURL, cfg.RegistryPublishMaxBytes), + deviceHandler: regauth.NewDeviceHandler(q, cfg.RegistryPublicBaseURL), + storage: st, + signingKey: signingKey, + }, nil +} + +func (b *RegistryBundle) Mount(r chi.Router, rbacOnly connect.HandlerOption) { + scopePath, scopeH := orchestratorv1connect.NewPluginScopeServiceHandler(b.scopeHandler, rbacOnly) + r.Mount(scopePath, scopeH) + + pluginPath, pluginH := orchestratorv1connect.NewPluginRegistryServiceHandler(b.pluginHandler, rbacOnly) + r.Mount(pluginPath, pluginH) + + publishPath, publishH := orchestratorv1connect.NewPluginPublishServiceHandler(b.publishHandler, rbacOnly) + r.Mount(publishPath, publishH) + + authPath, authH := orchestratorv1connect.NewPluginAuthServiceHandler(b.deviceHandler, rbacOnly) + r.Mount(authPath, authH) + + r.Route("/registry/storage", func(sub chi.Router) { + sub.Get("/*", b.serveDownload) + }) +} + +func (b *RegistryBundle) serveDownload(w http.ResponseWriter, r *http.Request) { + if err := regstorage.Verify(b.signingKey, r.URL); err != nil { + http.Error(w, "forbidden", http.StatusForbidden) + return + } + objectKey := strings.TrimPrefix(r.URL.Path, "/registry/storage/") + if objectKey == "" { + http.Error(w, "not found", http.StatusNotFound) + return + } + rc, err := b.storage.Get(r.Context(), objectKey) + if err != nil { + http.Error(w, "not found", http.StatusNotFound) + return + } + defer func() { _ = rc.Close() }() + w.Header().Set("Content-Type", "application/zstd") + _, _ = io.Copy(w, rc) +} +``` + +- [ ] **Step 4: Verify build** + +```bash +go build ./... +``` +Expected: clean. + +- [ ] **Step 5: Verify tests still pass** + +```bash +go test ./internal/registry/... ./internal/server/... +``` +Expected: PASS (storage + archive + auth packages; server has no tests; api had no tests). + +- [ ] **Step 6: Commit** + +```bash +git add -A +git commit -m "feat(orch): delete registry/git package, drop /git route and RegistryGitPath" +``` + +## Task 10: [core] Rewrite `publish` CLI command + +**Files:** +- Modify: `~/src/core/cmd/ninja/cmd/plugin.go` (the `newPluginPublishCmd` function) + +- [ ] **Step 1: Replace `newPluginPublishCmd`** + +In `~/src/core/cmd/ninja/cmd/plugin.go`, replace `newPluginPublishCmd` with: + +```go +func newPluginPublishCmd() *cobra.Command { + var channel string + var allowDirty bool + cmd := &cobra.Command{ + Use: "publish", + Short: "Build a source archive and publish a new version to the registry", + RunE: func(c *cobra.Command, _ []string) error { + host, _ := c.Flags().GetString("host") + cr, err := creds.Load() + if err != nil { + return err + } + resolvedHost, hc, err := cr.Resolve(host) + if err != nil { + return err + } + cli := orchclient.New(resolvedHost, hc.Token) + + modBytes, err := os.ReadFile("plugin.mod") + if err != nil { + return fmt.Errorf("read plugin.mod: %w", err) + } + mod, err := core.ParseModFull(modBytes) + if err != nil { + return err + } + if mod.Plugin.Scope == "" || mod.Plugin.Name == "" || mod.Plugin.Version == "" { + return fmt.Errorf("plugin.mod must have scope, name, and version") + } + + if !allowDirty { + out, _ := exec.Command("git", "status", "--porcelain").Output() + if len(strings.TrimSpace(string(out))) > 0 { + return fmt.Errorf("working tree dirty; commit or pass --allow-dirty") + } + } + + archiveBytes, err := archive.BuildSourceArchive(".") + if err != nil { + return fmt.Errorf("build archive: %w", err) + } + + ctx := context.Background() + pr, err := cli.Reg.GetPlugin(ctx, connect.NewRequest(&v1.GetPluginRequest{ + ScopeSlug: mod.Plugin.Scope, Name: mod.Plugin.Name, + })) + if err != nil { + return fmt.Errorf("get plugin: %w", err) + } + + readme, _ := os.ReadFile("README.md") + changelog, _ := os.ReadFile("CHANGELOG.md") + + pubResp, err := cli.Pub.PublishVersion(ctx, connect.NewRequest(&v1.PublishVersionRequest{ + PluginId: pr.Msg.Plugin.Id, + Version: mod.Plugin.Version, + Channel: channel, + Archive: archiveBytes, + ReadmeMd: string(readme), + ChangelogMd: string(changelog), + })) + if err != nil { + return fmt.Errorf("publish: %w", err) + } + fmt.Printf("Published %s@%s (%d bytes)\n", mod.Coords(), pubResp.Msg.Version.Version, len(archiveBytes)) + for _, w := range pubResp.Msg.Warnings { + fmt.Printf(" warning: %s\n", w) + } + return nil + }, + } + cmd.Flags().StringVar(&channel, "channel", "latest", "Channel to point at this version") + cmd.Flags().BoolVar(&allowDirty, "allow-dirty", false, "Skip clean-working-tree check") + return cmd +} +``` + +- [ ] **Step 2: Add the archive import** + +At the top of `plugin.go`, add to the imports: + +```go +"git.dev.alexdunmow.com/block/core/cmd/ninja/internal/archive" +``` + +- [ ] **Step 3: Verify build** + +```bash +cd ~/src/core +go build ./... +``` +Expected: clean. + +- [ ] **Step 4: Verify help is sane** + +```bash +go run ./cmd/ninja plugin publish --help +``` +Expected: shows the publish command help with `--channel` and `--allow-dirty` flags. + +- [ ] **Step 5: Commit** + +```bash +git add cmd/ninja/cmd/plugin.go +git commit -m "feat(cli): rewrite plugin publish to send tar.zst archive" +``` + +## Task 11: [core] Rewrite `init` CLI command (auto-commit, no remote) + +**Files:** +- Modify: `~/src/core/cmd/ninja/cmd/plugin.go` (the `newPluginInitCmd` function and the closing of the git-repo block) + +- [ ] **Step 1: Replace the git-repo block** + +In `newPluginInitCmd`, find: + +```go +if _, err := os.Stat(".git"); err == nil { + _ = runCmd("git", "remote", "remove", "ninja") + if err := runCmd("git", "remote", "add", "ninja", resp.Msg.GitRemoteUrl); err != nil { + return fmt.Errorf("git remote add: %w", err) + } + fmt.Println("Added git remote 'ninja'") +} else { + fmt.Println("Not in a git repo - skipped adding remote") +} + +if err := upsertPluginMod(scope, name); err != nil { + return err +} +fmt.Println("plugin.mod updated") +return nil +``` + +Replace with: + +```go +if err := upsertPluginMod(scope, name); err != nil { + return err +} +fmt.Println("plugin.mod updated") + +if _, err := os.Stat(".git"); err == nil { + if err := autoCommitPluginMod(); err != nil { + fmt.Fprintf(os.Stderr, "warning: could not commit plugin.mod: %v\n", err) + } +} else { + fmt.Println("Not in a git repo - run `git init` before `ninja plugin publish`") +} +return nil +``` + +Also remove the line `fmt.Printf(" Git remote: %s\n", resp.Msg.GitRemoteUrl)` from earlier in the function (`GitRemoteUrl` no longer exists on the response). + +- [ ] **Step 2: Add the `autoCommitPluginMod` helper** + +After the `writeMod` function at the bottom of `plugin.go`, add: + +```go +// autoCommitPluginMod stages and commits plugin.mod if it differs from +// what's already at HEAD. No-op when there's nothing to commit. +func autoCommitPluginMod() error { + out, err := exec.Command("git", "status", "--porcelain", "plugin.mod").Output() + if err != nil { + return fmt.Errorf("git status: %w", err) + } + if strings.TrimSpace(string(out)) == "" { + return nil + } + if err := runCmd("git", "add", "plugin.mod"); err != nil { + return err + } + if err := runCmd("git", "commit", "-m", "Add plugin.mod"); err != nil { + return err + } + fmt.Println("Committed plugin.mod") + return nil +} +``` + +- [ ] **Step 3: Verify build** + +```bash +cd ~/src/core +go build ./... +``` +Expected: clean. + +- [ ] **Step 4: Reinstall and smoke** + +```bash +go install ./cmd/ninja +which ninja +``` +Expected: `/home/alex/go/bin/ninja`. + +- [ ] **Step 5: Commit** + +```bash +git add cmd/ninja/cmd/plugin.go +git commit -m "feat(cli): init auto-commits plugin.mod, drops ninja git remote" +``` + +## Task 12: Manual smoke test of the new publish flow + +This is verification only — no commit. + +- [ ] **Step 1: Restart the orchestrator** + +```bash +cd ~/src/orchestrator +podman compose build orchestrator-backend +podman compose up -d orchestrator-backend +``` + +- [ ] **Step 2: Set up a fresh plugin repo** + +```bash +mkdir -p /tmp/smokeplugin && cd /tmp/smokeplugin +git init -q +git commit --allow-empty -qm "initial" +ninja login --host https://my.localdev.blockninjacms.com +ninja plugin init --host https://my.localdev.blockninjacms.com --scope @themes --name smoke +git log --oneline +``` +Expected: a commit "Add plugin.mod" exists on HEAD; no `ninja` remote (`git remote -v` is empty). + +- [ ] **Step 3: Publish** + +```bash +ninja plugin publish --host https://my.localdev.blockninjacms.com +``` +Expected: `Published @themes/smoke@0.1.0 (NNN bytes)` with no error. + +- [ ] **Step 4: Verify storage** + +In a Postgres session against the orchestrator DB: + +```sql +SELECT version, source_archive_key, source_archive_size, source_archive_sha256 + FROM registry_versions WHERE plugin_id = ( + SELECT id FROM registry_plugins WHERE name = 'smoke' + ); +``` +Expected: a row whose `source_archive_key` ends in `source.tar.zst`. + +- [ ] **Step 5: Try publishing again — version collision** + +```bash +ninja plugin publish --host https://my.localdev.blockninjacms.com +``` +Expected: error mentioning "version already published". + +- [ ] **Step 6: Cleanup** + +```bash +rm -rf /tmp/smokeplugin +``` + +--- + +# Phase 2: Categories + plugin kind + +## Task 13: [core] Add `Kind` and `Categories` to `ModFile` + update `writeMod` + +**Files:** +- Modify: `~/src/core/plugin/mod.go` +- Modify: `~/src/core/plugin/mod_test.go` +- Modify: `~/src/core/cmd/ninja/cmd/plugin.go` (the `writeMod` helper) + +- [ ] **Step 1: Add the new test cases** + +Append to `~/src/core/plugin/mod_test.go`: + +```go +func TestParseModFull_KindAndCategories(t *testing.T) { + src := []byte(` +[plugin] +name = "analyser" +scope = "blockninja" +version = "0.1.0" +kind = "plugin" +categories = ["analytics", "seo"] +`) + m, err := ParseModFull(src) + if err != nil { + t.Fatalf("ParseModFull err: %v", err) + } + if m.Plugin.Kind != "plugin" { + t.Errorf("Kind = %q, want plugin", m.Plugin.Kind) + } + if got := m.Plugin.Categories; len(got) != 2 || got[0] != "analytics" || got[1] != "seo" { + t.Errorf("Categories = %v", got) + } +} + +func TestParseModFull_KindDefaultsEmpty(t *testing.T) { + src := []byte(` +[plugin] +name = "legacy" +version = "0.1.0" +`) + m, err := ParseModFull(src) + if err != nil { + t.Fatalf("ParseModFull err: %v", err) + } + if m.Plugin.Kind != "" { + t.Errorf("Kind should be empty for legacy mod, got %q", m.Plugin.Kind) + } +} +``` + +- [ ] **Step 2: Run tests to verify failure** + +```bash +cd ~/src/core +go test ./plugin/... +``` +Expected: FAIL — `Kind`/`Categories` fields don't exist yet. + +- [ ] **Step 3: Add fields to `ModPlugin`** + +In `~/src/core/plugin/mod.go`, replace `ModPlugin` with: + +```go +type ModPlugin struct { + Name string `toml:"name"` + Scope string `toml:"scope"` + Version string `toml:"version"` + Kind string `toml:"kind,omitempty"` + Categories []string `toml:"categories,omitempty"` +} +``` + +- [ ] **Step 4: Run tests** + +```bash +go test ./plugin/... +``` +Expected: PASS. + +- [ ] **Step 5: Update `writeMod` to emit new fields** + +In `~/src/core/cmd/ninja/cmd/plugin.go`, replace `writeMod` with: + +```go +func writeMod(path string, m *core.ModFile) error { + var b strings.Builder + b.WriteString("[plugin]\n") + b.WriteString(fmt.Sprintf("name = %q\n", m.Plugin.Name)) + b.WriteString(fmt.Sprintf("scope = %q\n", m.Plugin.Scope)) + b.WriteString(fmt.Sprintf("version = %q\n", m.Plugin.Version)) + if m.Plugin.Kind != "" { + b.WriteString(fmt.Sprintf("kind = %q\n", m.Plugin.Kind)) + } + if len(m.Plugin.Categories) > 0 { + quoted := make([]string, len(m.Plugin.Categories)) + for i, c := range m.Plugin.Categories { + quoted[i] = fmt.Sprintf("%q", c) + } + b.WriteString(fmt.Sprintf("categories = [%s]\n", strings.Join(quoted, ", "))) + } + if m.Compatibility != nil { + b.WriteString("\n[compatibility]\n") + b.WriteString(fmt.Sprintf("block_core = %q\n", m.Compatibility.BlockCore)) + } + for _, r := range m.Requires { + b.WriteString("\n[[requires]]\n") + b.WriteString(fmt.Sprintf("name = %q\n", r.Name)) + b.WriteString(fmt.Sprintf("version = %q\n", r.Version)) + } + return os.WriteFile(path, []byte(b.String()), 0o644) +} +``` + +- [ ] **Step 6: Verify build** + +```bash +go build ./... +``` +Expected: clean. + +- [ ] **Step 7: Commit** + +```bash +git add plugin/mod.go plugin/mod_test.go cmd/ninja/cmd/plugin.go +git commit -m "feat(sdk): add Kind and Categories to ModPlugin; writeMod emits them" +``` + +## Task 14: [orch] Migration — add `kind` to registry_plugins + new `registry_categories` table + +**Files:** +- Create: `~/src/orchestrator/backend/sql/migrations/00056_plugin_kind_and_categories.sql` +- Create: `~/src/orchestrator/backend/sql/queries/registry_categories.sql` +- Modify: `~/src/orchestrator/backend/sql/queries/registry_plugins.sql` (add kind handling) +- Auto-regen: `~/src/orchestrator/backend/sql/schema.sql` and `~/src/orchestrator/backend/internal/db/*.sql.go` + +- [ ] **Step 1: Write the migration** + +`sql/migrations/00056_plugin_kind_and_categories.sql`: + +```sql +-- +goose Up +ALTER TABLE registry_plugins + ADD COLUMN kind text NOT NULL DEFAULT 'plugin' + CHECK (kind IN ('plugin', 'theme')); + +CREATE TABLE registry_categories ( + slug text PRIMARY KEY, + display_name text NOT NULL, + description text NOT NULL DEFAULT '', + sort_order int NOT NULL DEFAULT 0 +); + +INSERT INTO registry_categories (slug, display_name, sort_order) VALUES + ('analytics', 'Analytics', 10), + ('seo', 'SEO', 20), + ('social', 'Social', 30), + ('commerce', 'Commerce', 40), + ('forms', 'Forms', 50), + ('import-export', 'Import/Export', 60), + ('media', 'Media', 70), + ('developer', 'Developer', 80); + +-- +goose Down +DROP TABLE registry_categories; +ALTER TABLE registry_plugins DROP COLUMN kind; +``` + +- [ ] **Step 2: Add category queries** + +`sql/queries/registry_categories.sql`: + +```sql +-- name: ListRegistryCategories :many +SELECT slug, display_name, description, sort_order + FROM registry_categories + ORDER BY sort_order, slug; + +-- name: GetRegistryCategoriesBySlugs :many +SELECT slug FROM registry_categories WHERE slug = ANY($1::text[]); +``` + +- [ ] **Step 3: Update plugin queries to surface kind** + +In `sql/queries/registry_plugins.sql`, find queries that `SELECT *` from +`registry_plugins` or list columns explicitly — they need to surface +`kind` in their outputs. Since sqlc handles `SELECT *` automatically, +this is usually a no-op as long as the query uses `*`. Check +`CreateRegistryPlugin`, `GetRegistryPluginByID`, +`GetRegistryPluginByScopeAndName`, and `ListPublicRegistryPlugins`. + +If `CreateRegistryPlugin` inserts an explicit column list, add `kind` to +both the columns and values: + +```sql +-- name: CreateRegistryPlugin :one +INSERT INTO registry_plugins (scope_id, name, description, kind, categories) +VALUES ($1, $2, $3, $4, $5) +RETURNING *; +``` + +Add filtering for `ListPublicRegistryPlugins`: + +```sql +-- name: ListPublicRegistryPlugins :many +SELECT p.*, s.slug AS scope_slug + FROM registry_plugins p + JOIN registry_scopes s ON s.id = p.scope_id + WHERE p.visibility = 'public' + AND (sqlc.narg('kind')::text IS NULL OR p.kind = sqlc.narg('kind')::text) + AND (sqlc.narg('categories')::text[] IS NULL OR p.categories && sqlc.narg('categories')::text[]) + AND (sqlc.narg('query')::text IS NULL OR p.name ILIKE '%' || sqlc.narg('query')::text || '%') + ORDER BY p.updated_at DESC + LIMIT $1 OFFSET $2; +``` + +(Adjust the existing query — keep existing filter behaviour, just add the +new optional params.) + +- [ ] **Step 4: Run the migration** + +```bash +cd ~/src/orchestrator/backend +make migrate +``` +Expected: applied through `00056`. + +- [ ] **Step 5: Regen schema + sqlc** + +```bash +make schema +make sqlc +``` + +- [ ] **Step 6: Build (will fail in handler code — fixed Task 16+)** + +```bash +go build ./... +``` +Expected: failures in `internal/registry/api/converters.go` (needs `Kind`) and +the plugin/list handlers (signature changes). Will be fixed in subsequent +tasks. Leave build broken until Task 17. + +- [ ] **Step 7: Commit migration + queries** + +```bash +git add sql/migrations/00056_plugin_kind_and_categories.sql sql/queries/ sql/schema.sql internal/db/ +git commit -m "feat(orch): add plugin kind and registry_categories with seed" +``` + +## Task 15: Proto change — `Plugin.kind`, request fields, `ListCategories` RPC + +**Files:** +- Modify: `~/src/orchestrator/proto/orchestrator/v1/plugin_registry.proto` +- Modify: `~/src/core/proto/orchestrator/v1/plugin_registry.proto` +- Regen on both repos. + +- [ ] **Step 1: Edit orchestrator proto — `Plugin` gains `kind`** + +In `Plugin`: + +```proto +message Plugin { + string id = 1; + string scope_slug = 2; + string name = 3; + string visibility = 4; + bool premium = 5; + string description = 6; + string homepage_url = 7; + repeated string categories = 8; + google.protobuf.Timestamp updated_at = 9; + string kind = 10; +} +``` + +- [ ] **Step 2: `CreatePluginRequest` gains `kind` + `categories`** + +```proto +message CreatePluginRequest { + string scope_slug = 1; + string name = 2; + string description = 3; + string kind = 4; + repeated string categories = 5; +} +``` + +- [ ] **Step 3: `ListPluginsRequest` gains filters** + +```proto +message ListPluginsRequest { + int32 limit = 1; + int32 offset = 2; + string query = 3; + string kind = 4; + repeated string categories = 5; +} +``` + +- [ ] **Step 4: New `ListCategories` RPC** + +In `PluginRegistryService`, add: + +```proto +service PluginRegistryService { + rpc CreatePlugin(CreatePluginRequest) returns (CreatePluginResponse); + rpc GetPlugin(GetPluginRequest) returns (GetPluginResponse); + rpc ListPlugins(ListPluginsRequest) returns (ListPluginsResponse); + rpc GetVersion(GetVersionRequest) returns (GetVersionResponse); + rpc ResolveInstall(ResolveInstallRequest) returns (ResolveInstallResponse); + rpc ListCategories(ListCategoriesRequest) returns (ListCategoriesResponse); +} +``` + +Add message types: + +```proto +message Category { + string slug = 1; + string display_name = 2; + string description = 3; + int32 sort_order = 4; +} + +message ListCategoriesRequest {} +message ListCategoriesResponse { repeated Category categories = 1; } +``` + +- [ ] **Step 5: Mirror all 4 edits to core's proto** + +Apply the identical changes to `~/src/core/proto/orchestrator/v1/plugin_registry.proto`. + +- [ ] **Step 6: Regen orchestrator** + +```bash +cd ~/src/orchestrator +make -C backend proto +``` + +- [ ] **Step 7: Regen core** + +```bash +cd ~/src/core +buf generate +``` + +- [ ] **Step 8: Commit both repos** + +orchestrator: +```bash +cd ~/src/orchestrator +git add proto/ backend/api/ frontend/src/lib/api/gen/ +git commit -m "feat(orch): proto adds Plugin.kind, ListCategories RPC, filter args" +``` + +core: +```bash +cd ~/src/core +git add proto/ internal/api/ +git commit -m "feat(core): mirror Plugin.kind + ListCategories proto regen" +``` + +## Task 16: [orch] Converters + CreatePlugin + helper for category validation + +**Files:** +- Modify: `~/src/orchestrator/backend/internal/registry/api/converters.go` +- Modify: `~/src/orchestrator/backend/internal/registry/api/plugin.go` + +- [ ] **Step 1: Update converters to include `Kind`** + +In `converters.go`, update `pluginToProto` and `listPluginRowToProto`: + +```go +func pluginToProto(p db.RegistryPlugin, scopeSlug string) *orchestratorv1.Plugin { + homepage := "" + if p.HomepageUrl != nil { + homepage = *p.HomepageUrl + } + return &orchestratorv1.Plugin{ + Id: p.ID.String(), + ScopeSlug: scopeSlug, + Name: p.Name, + Visibility: p.Visibility, + Premium: p.Premium, + Description: p.Description, + HomepageUrl: homepage, + Categories: p.Categories, + Kind: p.Kind, + UpdatedAt: timestamppb.New(p.UpdatedAt), + } +} + +func listPluginRowToProto(r db.ListPublicRegistryPluginsRow) *orchestratorv1.Plugin { + homepage := "" + if r.HomepageUrl != nil { + homepage = *r.HomepageUrl + } + return &orchestratorv1.Plugin{ + Id: r.ID.String(), + ScopeSlug: r.ScopeSlug, + Name: r.Name, + Visibility: r.Visibility, + Premium: r.Premium, + Description: r.Description, + HomepageUrl: homepage, + Categories: r.Categories, + Kind: r.Kind, + UpdatedAt: timestamppb.New(r.UpdatedAt), + } +} +``` + +- [ ] **Step 2: Update `CreatePlugin` to validate kind + categories** + +In `internal/registry/api/plugin.go`, replace `CreatePlugin` with: + +```go +func (h *PluginHandler) CreatePlugin(ctx context.Context, req *connect.Request[orchestratorv1.CreatePluginRequest]) (*connect.Response[orchestratorv1.CreatePluginResponse], error) { + claims, ok := internalauth.GetUserFromContext(ctx) + if !ok { + return nil, connect.NewError(connect.CodeUnauthenticated, errors.New("login required")) + } + slug := strings.TrimPrefix(req.Msg.ScopeSlug, "@") + name := strings.ToLower(strings.TrimSpace(req.Msg.Name)) + if !pluginNameRE.MatchString(name) { + return nil, connect.NewError(connect.CodeInvalidArgument, errors.New("invalid plugin name")) + } + kind := req.Msg.Kind + if kind == "" { + kind = "plugin" + } + if kind != "plugin" && kind != "theme" { + return nil, connect.NewError(connect.CodeInvalidArgument, errors.New("kind must be 'plugin' or 'theme'")) + } + cats := req.Msg.Categories + if kind == "theme" && len(cats) > 0 { + return nil, connect.NewError(connect.CodeInvalidArgument, errors.New("themes do not carry categories")) + } + if err := h.validateCategories(ctx, cats); err != nil { + return nil, connect.NewError(connect.CodeInvalidArgument, err) + } + scope, err := h.q.GetRegistryScopeBySlug(ctx, slug) + if err != nil { + return nil, connect.NewError(connect.CodeNotFound, err) + } + yes, err := h.q.IsRegistryScopeMember(ctx, db.IsRegistryScopeMemberParams{ScopeID: scope.ID, UserID: claims.UserID}) + if err != nil || !yes { + return nil, connect.NewError(connect.CodePermissionDenied, errors.New("not a member of scope")) + } + p, err := h.q.CreateRegistryPlugin(ctx, db.CreateRegistryPluginParams{ + ScopeID: scope.ID, + Name: name, + Description: strings.TrimSpace(req.Msg.Description), + Kind: kind, + Categories: cats, + }) + if err != nil { + return nil, connect.NewError(connect.CodeAlreadyExists, err) + } + return connect.NewResponse(&orchestratorv1.CreatePluginResponse{ + Plugin: pluginToProto(p, slug), + }), nil +} + +func (h *PluginHandler) validateCategories(ctx context.Context, cats []string) error { + if len(cats) == 0 { + return nil + } + got, err := h.q.GetRegistryCategoriesBySlugs(ctx, cats) + if err != nil { + return err + } + gotSet := make(map[string]struct{}, len(got)) + for _, s := range got { + gotSet[s] = struct{}{} + } + for _, c := range cats { + if _, ok := gotSet[c]; !ok { + return errors.New("unknown category: " + c) + } + } + return nil +} +``` + +- [ ] **Step 3: Update `ListPlugins` to use the new optional filters** + +In the same file, replace `ListPlugins` with: + +```go +func (h *PluginHandler) ListPlugins(ctx context.Context, req *connect.Request[orchestratorv1.ListPluginsRequest]) (*connect.Response[orchestratorv1.ListPluginsResponse], error) { + limit := req.Msg.Limit + if limit <= 0 || limit > 200 { + limit = 50 + } + var query *string + if q := strings.TrimSpace(req.Msg.Query); q != "" { + lower := strings.ToLower(q) + query = &lower + } + var kindArg *string + if req.Msg.Kind != "" { + k := req.Msg.Kind + kindArg = &k + } + var catsArg []string + if len(req.Msg.Categories) > 0 { + catsArg = req.Msg.Categories + } + rows, err := h.q.ListPublicRegistryPlugins(ctx, db.ListPublicRegistryPluginsParams{ + Limit: limit, + Offset: req.Msg.Offset, + Query: query, + Kind: kindArg, + Categories: catsArg, + }) + if err != nil { + return nil, err + } + out := make([]*orchestratorv1.Plugin, 0, len(rows)) + for _, r := range rows { + out = append(out, listPluginRowToProto(r)) + } + return connect.NewResponse(&orchestratorv1.ListPluginsResponse{Plugins: out}), nil +} +``` + +(The exact parameter names depend on what sqlc generates from the +`sqlc.narg(...)` queries — adjust `Kind`/`Categories` to match the field +names sqlc produces. The build will tell you.) + +- [ ] **Step 4: Verify build** + +```bash +go build ./... +``` +Expected: clean. + +- [ ] **Step 5: Commit** + +```bash +git add internal/registry/api/converters.go internal/registry/api/plugin.go +git commit -m "feat(orch): CreatePlugin/ListPlugins handle kind and categories" +``` + +## Task 17: [orch] Add `ListCategories` handler + +**Files:** +- Create: `~/src/orchestrator/backend/internal/registry/api/category.go` +- Modify: `~/src/orchestrator/backend/internal/registry/api/plugin.go` (add method to `PluginHandler`) + +- [ ] **Step 1: Add the handler method** + +Append to `internal/registry/api/plugin.go`: + +```go +func (h *PluginHandler) ListCategories(ctx context.Context, _ *connect.Request[orchestratorv1.ListCategoriesRequest]) (*connect.Response[orchestratorv1.ListCategoriesResponse], error) { + rows, err := h.q.ListRegistryCategories(ctx) + if err != nil { + return nil, err + } + out := make([]*orchestratorv1.Category, 0, len(rows)) + for _, r := range rows { + out = append(out, &orchestratorv1.Category{ + Slug: r.Slug, + DisplayName: r.DisplayName, + Description: r.Description, + SortOrder: r.SortOrder, + }) + } + return connect.NewResponse(&orchestratorv1.ListCategoriesResponse{Categories: out}), nil +} +``` + +- [ ] **Step 2: Verify build** + +```bash +go build ./... +``` +Expected: clean. + +- [ ] **Step 3: Commit** + +```bash +git add internal/registry/api/plugin.go +git commit -m "feat(orch): add ListCategories handler" +``` + +## Task 18: [orch] `PublishVersion` validates kind + categories match plugin row + +The published `plugin.mod` must agree with `registry_plugins` on `kind` and +`categories`. Drift would let a developer accidentally re-classify a +plugin across versions. + +**Files:** +- Modify: `~/src/orchestrator/backend/internal/registry/api/publish.go` + +- [ ] **Step 1: Add the cross-check** + +After the existing scope/name/version mismatch checks (~line 95 in the +file post-Task 7), insert: + +```go +modKind := mod.Plugin.Kind +if modKind == "" { + modKind = "plugin" +} +if modKind != plugin.Kind { + return nil, connect.NewError(connect.CodeInvalidArgument, errors.New("plugin.mod kind does not match registered kind")) +} +if !stringSlicesEqualUnordered(mod.Plugin.Categories, plugin.Categories) { + return nil, connect.NewError(connect.CodeInvalidArgument, errors.New("plugin.mod categories do not match registered categories")) +} +``` + +- [ ] **Step 2: Add `stringSlicesEqualUnordered`** + +Append to `publish.go`: + +```go +func stringSlicesEqualUnordered(a, b []string) bool { + if len(a) != len(b) { + return false + } + m := make(map[string]int, len(a)) + for _, s := range a { + m[s]++ + } + for _, s := range b { + m[s]-- + if m[s] < 0 { + return false + } + } + return true +} +``` + +- [ ] **Step 3: Verify build** + +```bash +go build ./... +``` +Expected: clean. + +- [ ] **Step 4: Commit** + +```bash +git add internal/registry/api/publish.go +git commit -m "feat(orch): publish cross-checks plugin.mod kind/categories against row" +``` + +## Task 19: [core] CLI `init` prompts for kind and categories + +**Files:** +- Modify: `~/src/core/cmd/ninja/cmd/plugin.go` + +- [ ] **Step 1: Add a `promptKind` helper** + +After `createScopeInline` in `plugin.go`, add: + +```go +func promptKind(scanner *bufio.Scanner) (string, error) { + fmt.Println("Kind: 1) plugin 2) theme") + fmt.Print("Select [1]: ") + if !scanner.Scan() { + return "", fmt.Errorf("cancelled") + } + v := strings.TrimSpace(scanner.Text()) + switch v { + case "", "1", "plugin": + return "plugin", nil + case "2", "theme": + return "theme", nil + default: + return "", fmt.Errorf("invalid kind: %s", v) + } +} +``` + +- [ ] **Step 2: Add a `promptCategories` helper** + +After `promptKind`, add: + +```go +func promptCategories(ctx context.Context, cli *orchclient.Client, scanner *bufio.Scanner) ([]string, error) { + resp, err := cli.Reg.ListCategories(ctx, connect.NewRequest(&v1.ListCategoriesRequest{})) + if err != nil { + return nil, fmt.Errorf("list categories: %w", err) + } + cats := resp.Msg.Categories + if len(cats) == 0 { + return nil, nil + } + fmt.Println("Categories (comma-separated numbers, or blank to skip):") + for i, c := range cats { + fmt.Printf(" %d. %s — %s\n", i+1, c.Slug, c.DisplayName) + } + fmt.Print("Select: ") + if !scanner.Scan() { + return nil, fmt.Errorf("cancelled") + } + raw := strings.TrimSpace(scanner.Text()) + if raw == "" { + return nil, nil + } + parts := strings.Split(raw, ",") + out := make([]string, 0, len(parts)) + for _, p := range parts { + n, err := strconv.Atoi(strings.TrimSpace(p)) + if err != nil || n < 1 || n > len(cats) { + return nil, fmt.Errorf("invalid category selection: %s", p) + } + out = append(out, cats[n-1].Slug) + } + return out, nil +} +``` + +- [ ] **Step 3: Wire into `newPluginInitCmd`** + +In the body of `newPluginInitCmd`, after `name` is resolved and before the +`CreatePlugin` RPC call, insert: + +```go +kind, err := promptKind(scanner) +if err != nil { + return err +} +var cats []string +if kind == "plugin" { + cats, err = promptCategories(ctx, cli, scanner) + if err != nil { + return err + } +} +``` + +Then update the `CreatePlugin` call to pass them: + +```go +resp, err := cli.Reg.CreatePlugin(ctx, connect.NewRequest(&v1.CreatePluginRequest{ + ScopeSlug: scopeAPISlug(scope), + Name: name, + Description: "", + Kind: kind, + Categories: cats, +})) +``` + +And update `upsertPluginMod` to set kind and categories on the written +`ModFile`. Replace `upsertPluginMod` with: + +```go +func upsertPluginMod(scope, name, kind string, categories []string) error { + const file = "plugin.mod" + existing, _ := os.ReadFile(file) + mod, _ := core.ParseModFull(existing) + if mod == nil { + mod = &core.ModFile{} + } + if mod.Plugin.Version == "" { + mod.Plugin.Version = "0.1.0" + } + mod.Plugin.Scope = scope + mod.Plugin.Name = name + mod.Plugin.Kind = kind + mod.Plugin.Categories = categories + return writeMod(file, mod) +} +``` + +Update the call site in `newPluginInitCmd`: + +```go +if err := upsertPluginMod(scope, name, kind, cats); err != nil { + return err +} +``` + +- [ ] **Step 4: Verify build** + +```bash +cd ~/src/core +go build ./... +``` +Expected: clean. + +- [ ] **Step 5: Reinstall** + +```bash +go install ./cmd/ninja +``` + +- [ ] **Step 6: Commit** + +```bash +git add cmd/ninja/cmd/plugin.go +git commit -m "feat(cli): init prompts for kind and categories" +``` + +## Task 20: Manual smoke test of categories + kind flow + +- [ ] **Step 1: Restart orchestrator** + +```bash +cd ~/src/orchestrator +podman compose build orchestrator-backend +podman compose up -d orchestrator-backend +``` + +- [ ] **Step 2: Verify ListCategories** + +```bash +curl -sS https://my.localdev.blockninjacms.com/orchestrator.v1.PluginRegistryService/ListCategories \ + -H 'Content-Type: application/json' -d '{}' +``` +Expected: JSON listing the 8 seeded categories. + +- [ ] **Step 3: Init a theme** + +```bash +mkdir -p /tmp/themeplugin && cd /tmp/themeplugin +git init -q && git commit --allow-empty -qm "initial" +ninja plugin init --host https://my.localdev.blockninjacms.com --scope @themes --name aurora +# At the kind prompt, choose 2 (theme). +cat plugin.mod +``` +Expected: `kind = "theme"` set, no categories line. + +- [ ] **Step 4: Init a plugin with categories** + +```bash +mkdir -p /tmp/normalplugin && cd /tmp/normalplugin +git init -q && git commit --allow-empty -qm "initial" +ninja plugin init --host https://my.localdev.blockninjacms.com --scope @themes --name foometrics +# kind: 1 (plugin). categories: pick a couple, e.g. "1,2" +cat plugin.mod +``` +Expected: `kind = "plugin"` and `categories = ["analytics", "seo"]`. + +- [ ] **Step 5: Publish + verify in DB** + +```bash +cd /tmp/normalplugin +ninja plugin publish --host https://my.localdev.blockninjacms.com +``` +Expected: success. + +In Postgres: +```sql +SELECT name, kind, categories FROM registry_plugins WHERE name IN ('aurora','foometrics'); +``` +Expected: `aurora | theme | {}`, `foometrics | plugin | {analytics,seo}`. + +- [ ] **Step 6: Negative — try inventing a category** + +Edit `/tmp/normalplugin/plugin.mod` to add `"bogus"` to `categories`, +commit, bump patch, publish: + +```bash +cd /tmp/normalplugin +sed -i 's/\["analytics", "seo"\]/["analytics", "seo", "bogus"]/' plugin.mod +git add plugin.mod && git commit -qm "tweak" +ninja plugin bump +ninja plugin publish --host https://my.localdev.blockninjacms.com +``` +Expected: error: "plugin.mod categories do not match registered categories" +(the plugin row's categories don't include "bogus"). The mismatch check +fires before the category-existence check, but either error proves the +guard works. + +- [ ] **Step 7: Cleanup** + +```bash +rm -rf /tmp/themeplugin /tmp/normalplugin +``` + +--- + +## Self-review + +**Spec coverage check:** +- Plugin kind on `registry_plugins` + `Plugin` proto + `plugin.mod` — Task 13, 14, 15, 16, 19. ✓ +- `registry_categories` table seeded — Task 14. ✓ +- Categories validated against canonical list — Task 16 (`validateCategories`), Task 18 (publish-time row match). ✓ +- `tar.zst` archive format — Tasks 1–4. ✓ +- `git archive HEAD` — Task 3. ✓ +- `PublishVersionRequest` bytes + drop git_ref — Task 6, 7. ✓ +- Init auto-commits plugin.mod, no remote — Task 11. ✓ +- Publish no tag/push/.git/config — Task 10. ✓ +- Delete `registry/git/` + route + config + columns — Tasks 5, 8, 9. ✓ +- `ListCategories` RPC — Task 15, 17. ✓ +- `ListPlugins` kind/categories filter — Task 15, 16. ✓ +- Themes don't carry categories — Task 16 (validation), Task 19 (init skips prompt). ✓ +- Size cap + 4× decompressed guard — Task 4 (helper), Task 7 (handler), Task 9 (config). ✓ +- gitignore warning — *NOT covered in any task.* Add a small Task 11.5 to warn at publish time when tracked files match the current `.gitignore`. See follow-up below. + +**Follow-up — gitignore warning:** Add a small step to Task 10 (or a Task 11.5): +the CLI runs `git ls-files --cached --ignored --exclude-standard` and prints a +warning if non-empty. Spec calls for this; plan didn't include it. Adding now: + +### Task 11.5: [core] Publish-time warning for tracked-yet-gitignored files + +**Files:** Modify `~/src/core/cmd/ninja/cmd/plugin.go` (the publish command). + +- [ ] **Step 1:** After the working-tree-clean check and before `BuildSourceArchive`, insert: + +```go +out, _ := exec.Command("git", "ls-files", "--cached", "--ignored", "--exclude-standard").Output() +if names := strings.TrimSpace(string(out)); names != "" { + fmt.Fprintln(os.Stderr, "warning: these tracked files match .gitignore and will still be shipped:") + for _, n := range strings.Split(names, "\n") { + fmt.Fprintln(os.Stderr, " "+n) + } + fmt.Fprintln(os.Stderr, " (run `git rm --cached ` to drop)") +} +``` + +- [ ] **Step 2:** Build and reinstall. + +```bash +cd ~/src/core +go build ./... && go install ./cmd/ninja +``` + +- [ ] **Step 3:** Commit. + +```bash +git add cmd/ninja/cmd/plugin.go +git commit -m "feat(cli): warn at publish when tracked files match .gitignore" +``` + +**Placeholder scan:** No "TBD"/"TODO"/"fill in" in plan. ✓ + +**Type consistency:** `BuildSourceArchive(repoDir string) ([]byte, error)` defined in Task 3, called as `archive.BuildSourceArchive(".")` in Tasks 10 and 11.5. ✓ `OpenAndValidate(compressed []byte, maxCompressed, maxDecompressed int)` defined in Task 4, called in Task 7. ✓ `Plugin.Kind`, `Plugin.Categories` defined in Tasks 13/15 and used consistently in 16/17/18/19. ✓