A self-hosted, OCI-compliant container registry

Problem

If you want a container registry with real features — vulnerability scanning, image signing, RBAC, SSO, a tamper-evident audit trail — but you also want to own the infrastructure, your options are bad. distribution/distribution (the reference registry) does push/pull and almost nothing else, so you end up gluing on a handful of side services and maintaining the seams. Harbor gives you the features but is a heavy, multi-component deployment. ECR scopes to a single AWS account and bills per use. Docker Hub and Quay are SaaS — someone else’s rate limits, someone else’s outage.

I built one to find out what a self-hosted registry with the feature scope of Harbor or ECR actually costs to build and run in 2026 — and whether the same codebase could serve both a one-team internal deploy and a SaaS operator who genuinely needs tenant isolation.

Constraints

1. OCI Distribution Spec v1.1, end-to-end. docker, podman, buildkit, skopeo, crane — all of them must Just Work. No bespoke surface area; no client-side shimming.
2. Self-hosted first; single-tenant by default. The common case is one team running it on their own infrastructure. DEPLOYMENT_MODE=single ships that out of the box — one tenant, no SaaS chrome. Multi-tenant (DEPLOYMENT_MODE=multi) is the same binary and schema, opted into by operators who need it.
3. Isolation that fails closed. When multi-tenant is on, Host header → tenant, and isolation is two-layer: application code passes tenant_id on every query, and PostgreSQL row-level security enforces it at the DB, so a forgotten filter fails closed instead of leaking.
4. Pluggable storage. Operators pick MinIO, S3, GCS, Azure Blob, or filesystem at deploy time. The registry doesn’t care.
5. Assume any one service is compromised. Audit logs survive a malicious app. mTLS between every internal hop, not just at the edge. Distroless containers, no shell, nothing to drop into.
6. Local dev is docker compose up. Production is one Helm chart. Nothing else.

Architecture

Thirteen Go services as a small mesh — TLS terminating at a Traefik gateway, sync paths over gRPC + mTLS, async fan-out over a RabbitMQ topic exchange (registry.events):

request path

docker · crane

client

↓ /v2/*

gateway

traefik · tls · host routing

↓

core

oci /v2/* push · pull

gRPC + mTLS mesh

auth

jwt · api keys · mtls

metadata

pg source of truth

storage

grpc stream · pluggable

signer

cosign · notary v2

tenant

custom domains · dns

proxy

pull-through cache

async · rabbitmq topic exchange

webhook

hmac · retry ladder

audit

append-only · force rls

gc

mark-sweep · advisory

scanner

trivy · jsonrpc

data plane · postgres (rls) · redis · minio / s3 / gcs / azure

• Edge. gateway — Traefik v3 with TLS, host-based routing, rate-limiting.
• Data plane. core (OCI /v2/* push / pull / delete / list); storage (gRPC-streaming blob driver over the pluggable backends); proxy (pull-through cache for upstream registries).
• Control plane — gRPC + mTLS. auth (JWT RS256 with hot key-rotation + JWKS, Argon2id API keys, OAuth 2.0 + PKCE and SAML 2.0 SSO, RBAC); metadata (PostgreSQL source of truth for repos / tags / manifests); signer (Cosign + Notary v2, Vault-backed keys); tenant (tenant CRUD + custom-domain verification); management (REST BFF for the React dashboard, CLI, and Terraform).
• Async — RabbitMQ topic exchange. webhook (HMAC-signed retries on a 5s / 30s / 5m / 30m / 2h ladder, SSRF block-list); audit (append-only, FORCE ROW LEVEL SECURITY, per-tenant SHA-256 hash chain, dedicated INSERT-only Postgres role); gc (mark-sweep with per-tenant advisory locks); scanner (external-process JSON-RPC into Trivy by default, Grype / Clair adapters).
• Data plane. PostgreSQL 16 with RLS, Redis 7, RabbitMQ quorum queues with DLX, MinIO / S3 / GCS / Azure / filesystem.
• Observability. OpenTelemetry → Jaeger / Tempo / Datadog; Prometheus on a dedicated port; structured slog JSON with trace_id / tenant_id on every line.

A read-only MCP server also ships, exposing registry state as Model Context Protocol tools for Claude Desktop / Cursor.

The two design choices I’d defend hardest:

External-process scanner plugins, not Go .so plugins. exec.CommandContext, SHA-256 binary checksum, explicit env allowlist, 10 MB stdout limit. You give up in-process plugin convenience; you keep a sane security model. The class of supply-chain risks that come with dynamic loading of compiled code at runtime just isn’t worth the ergonomics.

One codebase, two deployment modes — not two products. Single- and multi-tenant share the identical schema and wire format; DEPLOYMENT_MODE picks the posture. In single mode a SingleTenantInjector interceptor stamps every request with the bootstrap tenant id and a second CreateTenant returns FAILED_PRECONDITION. Multi-tenancy rides on PostgreSQL RLS + advisory locks rather than multi-schema or multi-database, so a bug that forgets a filter fails closed at the DB instead of leaking — and there’s no N-times migration cost or painful cross-tenant analytics.

Result

• 75 / 75 OCI Distribution Spec v1.1 conformance (5 optional features skipped and advertised as such). Architecture was designed against the spec from commit one — the project’s CLAUDE.md drove the implementation, not the other way around — so there was no spec-discovery phase.
• 13 services, distroless and non-root. Go 1.25 builder → gcr.io/distroless/static-debian12:nonroot. No shell. No package manager. Nothing to escalate from.
• mTLS between every internal gRPC call, hot-reloading on cert-manager rotation, with a per-server peer-CN allowlist for defence-in-depth — not just TLS at the edge.
• 80% per-service test coverage floor, CI-enforced, with the OCI conformance suite running on every PR to main.
• 5 pluggable storage backends (MinIO / S3 / GCS / Azure Blob / filesystem) behind one driver interface; operators choose with STORAGE_DRIVER=….
• Supply-chain and admission controls that actually gate traffic: Cosign + Notary v2 signing against Vault-backed keys, repo-wide require_signature with a per-repo trusted-key allowlist, and CVSS-gated admission that blocks push/pull of images whose scan exceeds a configured severity.
• Tamper-evident audit — FORCE RLS + a per-tenant SHA-256 hash chain on an INSERT-only role — streamable to SIEM over syslog RFC 5424 / CEF / HTTPS.
• Machine-identity story for CI: federated workload identity lets GitHub Actions / GitLab / Buildkite runners exchange their OIDC token for a short-lived registry JWT — no stored secret.

What I’d do differently

Two real ones, both surfaced during hardening — and both are the kind of thing you find by chaos-testing before you’ve shipped, not after.

Detect connection-pool exhaustion explicitly, opt out of retry. The gRPC retry interceptor amplified load whenever the Postgres pool saturated — Acquire blocked, surfaced as codes.Internal, the client retried, and the pool got hammered harder. The fix is one line of logic: catch saturation, return codes.ResourceExhausted, and configure the interceptor to not retry that code. The lesson isn’t that I missed it — it’s that I designed retry policy before chaos-testing it.

Don’t keep state in memory in a distributed system. The signer service originally cached signatures in a sync.RWMutex map. A process restart erased them; VerifyManifest then 404’d until they were re-signed. The fix is obvious in hindsight: persist day-one — to Postgres, or follow Cosign’s model and push signatures as OCI artifacts. The lesson is that volatile distributed state is a footgun even in projects where every other surface is contract-first.