The cluster

Overview

Every node runs the same binary and serves web, DNS and mail. One master (signing authority) and one primary (single-writer workload) at a time. Both move by a signed succession order, decided in advance.

Paths are under /api/v1; super_admin token unless marked recovery route.

TOKEN=$(ozpanel admin token --email admin@example.com --dangerous | tail -1)
curl -s -H "Authorization: Bearer $TOKEN" http://127.0.0.1:9600/api/v1/cluster/state

Identity and transport

Per-node settings

Adding a node

• MachineBare Debian 13 or 12, root SSH by password. A machine with its own MariaDB datadir is refused: install-node.sh then join does not work. Join bare machines.
• WhereOn the master; other nodes answer 409. Mesh set up on the master.
• NetworkMaster reaches the new machine by SSH; it reaches the master on 656.
• ConsoleCluster → Operations: generic form for POST /cluster/join. No join wizard.

curl -s -X POST -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"name":"node2","real_ip":"203.0.113.11","fqdn":"node2.example.com","ssh_password":"…"}' \
  http://127.0.0.1:9600/api/v1/cluster/join

201 with job_id; poll GET /cluster/join/{job}. Dry check: POST /cluster/preflight.

Steps

Master side: first_login · preflight · coordinate (UUID, registry, mesh address) · provision (app key, binary, config, CA cert) · enroll (key stays on the node) · mesh_up (ping both ways) · bootstrap_node (30 min) · verify (Galera size ≥ 2).

The new node's own plan: 46 steps, 20 before its panel starts, 26 inside it.

A failed join

• Re-issueSame POST /cluster/join, same name. UUID kept, mesh not re-invited, done steps skipped.
• Not resumable/cluster/bootstrap/{job}/resume answers 400 "not a bootstrap job" for join_*.
• Master panel restartedJob failed (SSH passwords in memory). Re-issue.

After a join

• AutomaticAppended to succession (join order) · probes mirrored to its config · boot gate installed · tinc hooks · all gates re-derived · arbitrator converged.
• You decidePin a rank ({"rank":2}) to set who takes over as primary · region, label, public IP · give singleton apps more than one node.
• VerifyGET /cluster/membership-status, /cluster/planes, /cluster/interventions.

Roles

The declaration

One signed record: seq (higher wins) · primary_node · succession · max_lag_bytes · failback · kind (root, or succession signed by a successor's node key) · handover (outgoing primary's signed ack). No vote. Unverified = refused. The primary never leaves a live node without its ack.

GET  /cluster/workload
POST /cluster/workload/declare  {"primary":"node1","succession":["node2","node3"],"max_lag_bytes":0}

Role-bound planes

MariaDB (Galera) is multi-master. One role executor per node: one apply at a time, never older, measured after. GET /cluster/planes, /cluster/roles (180 s stale = unknown); role_transfer_stalled alert after 180 s.

Succession

Order: pinned rank ascending · then join order · then name. Down nodes keep their place and are skipped.

Master stops answering on every address → the effective successor holding the key claims it. Automatic.

Quorum

Data nodes weight 2, arbitrator (garbd) weight 1. Arbitrator only at an even data-node count, on the highest-ranked live successor, never the primary.

Failover

PUT /cluster/internet-probes {"probes":[{"host":"198.51.100.5","port":443,"proto":"tcp","note":"isp"}]}
POST /cluster/internet-probes/test     GET /cluster/connectivity     GET /cluster/survive-status

Unset targets: two public resolvers on TCP 443.

Fences

• Self-fence3+ nodes: primary without a majority stops PostgreSQL. Backstop: 90 s unconfirmed, writes stop.
• GaleraSide without quorum refuses writes.
• YieldPostgreSQL primary seeing a verified, reachable other primary stops.
• Survive aloneAll peers silent: 30 s settle (≥ 2× evs.inactive_timeout), own internet proven, majority with witnesses on 3+.

When the primary dies

1. Verdict settles to down, corroborated.
2. Master rechecks after 20 s (0 s if the mesh reported loss and its internet is proven).
3. Master signs a declaration naming the effective successor. Dead master: master succession first.
4. Every node applies it; GET /cluster/planes shows complete.

Timings: gate pass 2 min · idle liveness 60 s · mesh burst 2 s ×4 · observation age 90 s · transition expiry 10 min · failback retry 1/min. Drill, three nodes, primary cut: Redis 10 s, PostgreSQL promoted 72 s.

Handover and failback

POST /cluster/handover-to {"node":"node2"}

Any node. Refused before moving: 404 not a member · 409 down, already primary, planes disagree, transition running.

Sequence: seat + rank 1 → preflight → ready → old primary demotes with signed ack → target declares → promote → old primary rejoins as standby. Outcome: complete · partial · refused · failed. POST /cluster/handover (recovery route) makes the called node primary.

Automatic handback needs: seat with automatic policy · storage serving · no transition · planes agree · caught-up standby. Each refusal logged.

Boot gate

ozpanel boot-gate, PostgreSQL's ExecStartPre. Cached declaration plus any peer; only writes standby.signal; always exits 0. Only node in registry: not gated. Peer names this node: primary. Anything else: standby. Diverged Galera data archived to /var/backups/ozpanel/galera-diverged/.

Removing a node

POST   /cluster/nodes/node3/evict {"reason":"decommissioned"}
DELETE /cluster/nodes/node3/evict          # readmit

Refused: 403 not master · 409 itself, primary, master, or the only copy of a file on it (no override; unreadable storage needs "acknowledge_data_loss": true).

Phases: fence (tombstone, revoke certs, registry, DB accounts) → data (storage, repmgr, Sentinel, Galera) → records (DNS, apps, allowlists) → network (mesh, peers, firewall) → prove. Rerun resumes. GET /cluster/evictions. Readmitted nodes re-enrol; old certs stay revoked.

Console evict button sends no reason. Readmit: generic form only.

Addresses

• Publishedfloating, else public, else interface. Mesh never in DNS.
• Behind NATnat_egress_ip admitted by peers; self-measured when all peers agree. No port forward: the node dials out.
• Reach checkEvery 15 min peers dial each candidate address; unmeasured never reported reachable.
• Floating IPPOST /cluster/floating-ip {"floating_ip","mode":"automatic|manual","vps_ip","vps_password"|"tinc_invite","vps_ssh_port":22,"tinc_port":659,"tinc_network"}. Separate tinc network. Peers reached via the real gateway. Tunnel loss blackholes the default route. See Install.

Convergence

Config stored once, rendered on each node. Journal: gen, kind, scope, op, origin_node, made_by, made_at; no payloads, no secrets.

GET /cluster/changes · /cluster/convergence · /cluster/gates · /cluster/updates/verify · /cluster/sync-replication.

Recovery

What needs a person

GET /cluster/interventions (recovery route). Unmeasured never counts as pass.

Recovery token works without quorum: ozpanel admin recovery-token; GET /cluster/recovery-token, POST /cluster/recovery-token/rotate. Recovery screen: /recovery, recovery grants.

Every node off, then back

1. Wait for all nodes if you can.
2. Every node up, MariaDB stopped: POST /galera/recover-position.
3. Highest position only: POST /galera/bootstrap ({"force":true} if not marked safe).
4. Others join alone. Check interventions and planes.

Never bootstrap two nodes. Never force before recover-position everywhere.

Two of three really dead

1. Confirm both off from the hosting console. Merely unreachable: stop.
2. Survivor: POST /cluster/promote-survivor {} with the recovery token. 409 = a peer is still connected. Refused without proven internet.
3. Read steps: galera, redis, gluster (reports retired), postgres, dns, master.
4. Not master after ~5 min: POST /cluster/master/claim. Primary still dead: POST /cluster/workload/declare.

Files need two of three storage nodes; they return with the second node. Old-primary writes after the survivor's position are discarded.

Tools

• ozpanel recoverDiagnose only; --apply, --settle (75 s), --force, --config PATH.
• GET /galera/survivorWould this node bootstrap alone, and why.
• Not working in the consolePostgreSQL promote, rejoin, switchover buttons (501). Use handover.

Drills

Three nodes, current release. Passed: A1 primary off and on (handback) · B1 primary cut from mesh · B2 master cut · B3 successor cut · B4 one link cut, nothing moves. Not run on this release: other single-node, two-node, whole-cluster and handover-fault drills. Tools: scripts/drill-cut.sh --cut A:B [--what mesh|all] [--minutes N] / --heal, scripts/drill-record.sh.

Cluster CLI

More endpoints: GET /cluster/mesh, /cluster/network, /cluster/ca, /cluster/identity, /cluster/certificates, /cluster/secrets · POST /cluster/secrets/pull, /cluster/gates/reconcile, /cluster/connectivity/reconcile, /repmgrd/rejoin · GET /galera/status, /galera/recovery, /galera/garbd/status. Full list: API.

Previous: Installing ozpanel Next: Web hosting and Caddy