Aether Operator Guide

Deploy, scale, and monitor Aether clusters.

Cluster Bootstrap

An Aether cluster starts with a TOML configuration file. The aether cluster bootstrap command provisions nodes, forms the cluster, and brings it to a ready state.

# cluster.toml
[cluster]
name = "production"
size = 5

[cloud]
provider = "hetzner"
region = "fsn1"
instance_type = "cx31"

[security]
cluster_secret = "${secrets:cluster/secret}"

aether cluster bootstrap --config cluster.toml

The 12-step bootstrap orchestrator handles: cloud provisioning, node startup, peer discovery, consensus formation, quorum establishment, leader election, KV-Store initialization, artifact repository setup, and readiness verification.

Deploying Applications

# Upload artifact to cluster
aether artifact upload target/my-service-1.0.0.jar

# Deploy blueprint
aether deploy target/blueprint.toml

# Or use a strategy
aether deploy target/blueprint.toml --strategy canary

# Deployment lifecycle
aether deploy status          # Show deployment status
aether deploy promote         # Advance to next stage (canary/rolling)
aether deploy rollback        # Rollback deployment
aether deploy complete        # Finalize deployment

Deployment Strategies

Strategy	Behavior	Rollback
Immediate	All instances at once	Manual redeploy previous version
Rolling	Weighted traffic shift with health thresholds	`aether rollback`
Canary	Progressive traffic shift through configurable stages (e.g., 5% → 25% → 50% → 100%) with auto-evaluation	Automatic on health degradation
Blue-green	Atomic switchover via consensus (~5ms routing change, plus drain for in-flight requests)	Instant switch back
A/B testing	Deterministic split by request context	Remove variant

Scaling

Three Dimensions

Slice scaling — more instances on existing nodes. Millisecond startup (classes already loaded)
Node scaling — add machines. Node connects, restores state, begins serving
Worker group scaling — scale to thousands via SWIM gossip. Zero consensus overhead

Auto-Scaling

Tier 1 — Decision Tree (1-second intervals): reactive scaling on CPU, latency, queue depth, error rate. Always active.

Tier 2 — TTM Predictor (60-second intervals): ONNX ML model with 2-hour sliding window. Predicts load spikes before they happen.

If TTM fails, Decision Tree continues with default thresholds. No scaling disruption.

Manual Scaling

# Scale a specific slice
aether scale my-service --instances 5

# Scale the cluster
aether cluster scale --nodes 7

Monitoring

Built-in Observability

Prometheus metrics — scrape endpoint at /metrics (Micrometer)
Per-method tracking — P50/P95/P99 latency, call counts, error rates
Dynamic tracing — toggle per method at runtime: aether observability enable TRACING my-service.process
Dynamic log levels — aether logging set org.example DEBUG
Cluster events — 11 event types, REST API, WebSocket feed
Dashboard — Forge dashboard with live metrics (production dashboard in development)

Health & Alerts

# Check cluster health
aether health

# View active alerts
aether alerts

# Configure thresholds
aether thresholds set cpu_warning 0.75
aether thresholds set cpu_critical 0.90

Cloud Integration

Aether integrates with four cloud providers without vendor SDKs. All API clients use SigV4/JWT/OAuth2 implemented from scratch.

Provider	Compute	Secrets	Load Balancer	Discovery	Certificates
Hetzner	Yes	Yes (env vars)	Yes	Yes (labels)	Self-signed (HKDF)
AWS	EC2	Secrets Manager	ELBv2	Yes	ACM
GCP	Compute Engine	Secret Manager	NEGs	Yes	Certificate Manager
Azure	VMs	Key Vault	Load Balancers	Resource Graph	Key Vault

Cloud provider is selected via aether.toml. Adding a new provider is implementing an interface — no SDK to adopt.

Node Lifecycle

State	Description
JOINING	Node connecting to cluster, restoring state
ON_DUTY	Serving traffic, participating in consensus
DRAINING	Gracefully migrating workload, respecting disruption budgets
DECOMMISSIONED	No workload, awaiting shutdown
SHUTTING_DOWN	Final cleanup and exit

# Graceful drain (respects disruption budget)
aether node drain node-3

# Force shutdown
aether node shutdown node-3

Security

Transport security — mutual TLS for all TCP, AES-256-GCM for SWIM gossip. HKDF-derived from cluster secret, no external PKI
QUIC — mandatory TLS 1.3, 0-RTT reconnection, connection migration
RBAC — ADMIN / OPERATOR / VIEWER roles with per-route enforcement
API keys — SHA-256 hashed, configurable per role
Audit trail — 7 event types, cluster-wide aggregation
Key rotation — zero-downtime rotation for cluster secret and gossip keys

Backup & Recovery

Cluster metadata is periodically serialized to a local git repository — triggered on state changes and at configurable intervals. Git provides versioning, history, diffs, and optional remote push.

# Trigger manual backup
aether backup trigger

# List backups (shows git commit history)
aether backup list

# Restore from backup (by commit ID)
aether backup restore abc1234