Operations Reference

Deployment First

This page covers post-deployment operations. For initial deployment, start with the CLI Reference. Use Manual Deployment as an advanced reference for what the provisioner configures.

This page is an operator reference for those who have deployed FrameWorks and need to understand what’s running, debug issues, or customize their setup.

Directory Structure

CLI-managed Linux hosts use per-service directories rather than a single shared bin/ directory:

/opt/frameworks/
├── quartermaster/       # Native Go service binary and install sentinel
├── bridge/              # Native Go service binary and install sentinel
├── chatwoot/            # Docker Compose stack directory
├── edge/                # Edge Docker Compose stack directory
└── ...                  # One directory per native service or compose stack

/etc/frameworks/
├── quartermaster.env
├── commodore.env
├── purser.env
├── pki/              # Internal gRPC TLS material when enabled
├── certs/            # Edge/runtime certificates when applicable
└── ...               # Per-service environment files

Native systemd units are named frameworks-<service>.service and run /opt/frameworks/<service>/<service> with /etc/frameworks/<service>.env. Docker-mode services keep their docker-compose.yml and .env files under /opt/frameworks/<stack>/.

For manual native installs, create the base paths first:

sudo mkdir -p /etc/frameworks
sudo mkdir -p /opt/frameworks/<service>
sudo useradd -r -s /sbin/nologin frameworks 2>/dev/null || true
sudo chown -R frameworks:frameworks /opt/frameworks /etc/frameworks

Service Startup Order

The CLI owns startup sequencing. At a high level, infrastructure starts first, then mesh/control-plane services, then application services, interfaces, and edge workloads. Do not maintain a separate hand-authored service order in docs: the dependency graph lives in the provisioner and service registry.

For break-glass work, inspect the generated unit/compose files on the host and use Manual Deployment to find the current source of truth in code.

Health Check Reference

Start with the CLI so checks use the current manifest, service registry, and context:

frameworks cluster doctor --deep
frameworks cluster status
frameworks cluster diagnose ports
frameworks cluster logs <service>

For direct host inspection, use the generated service env files and the port diagnostics output from the same CLI version operating the cluster. Avoid copying port loops into runbooks; they drift as services move or split health, metrics, and gRPC listeners.

Firewall Configuration

Firewall policy should follow the manifest and service registry:

• Public hosts expose only public ingress: HTTP/HTTPS and the media protocols enabled for that edge or cluster.
• Backend nodes expose SSH/bootstrap access and the manifest’s WireGuard listen port to other backend nodes.
• Internal service ports stay private to the mesh or private network.
• Edge nodes do not join the backend mesh.

Run frameworks cluster preflight before provisioning and frameworks edge preflight for BYO edge hosts. For exact ports, use frameworks cluster diagnose ports and the generated edge configuration rather than a Markdown table.

Performance Tuning

The CLI applies these automatically with frameworks edge tune --write. For manual application:

Network tuning (/etc/sysctl.d/frameworks-edge.conf):

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.somaxconn = 8192
net.ipv4.ip_local_port_range = 16384 65535

File descriptors (/etc/security/limits.d/frameworks-edge.conf):

* soft nofile 1048576
* hard nofile 1048576

Apply changes:

sudo sysctl --system
# Reboot or re-login for limits changes

Troubleshooting

Service won’t start:

# Check status and logs
sudo systemctl status frameworks-quartermaster
sudo journalctl -u frameworks-quartermaster -f --no-pager -n 100

# Check if port is in use
sudo ss -tlnp | grep 18002

Database connection issues:

# Probe Postgres/YugabyteDB, ClickHouse, Kafka, and Redis reachability
frameworks cluster doctor

# Also run capability probes as each service's runtime database role
frameworks cluster doctor --deep

Schema migrations

For ClickHouse schema upgrades during rolling deploys, see ClickHouse Migrations.

Inter-service communication:

# Check Quartermaster's HTTP health endpoint
curl -fsS http://localhost:18002/health

# Test gRPC to Quartermaster over the internal-CA TLS listener
grpcurl -cacert /etc/frameworks/pki/ca.crt -servername quartermaster.internal \
  <quartermaster-host>:19002 grpc.health.v1.Health/Check

# Check if WireGuard is up
wg show
ping <peer-mesh-ip>  # mesh addresses are in 10.88.0.0/16 by default

Kafka issues:

# Check consumer group lag
kafka-consumer-groups.sh --bootstrap-server <broker>:9092 --describe --group periscope-ingest

# List topics
kafka-topics.sh --bootstrap-server <broker>:9092 --list

Manual Recovery Boundary

Manual dependency installation, schema bootstrapping, Kafka topic creation, and systemd unit generation are not documented as public procedures. They are generated by the provisioner and tested through the CLI. When you need to recover by hand, use the source-of-truth links in Manual Deployment, inspect the generated files on the affected host, and back-port fixes into the manifest/provisioner rather than into this page.