omv-ha

Check HTTP security headers for cloudless.online and/or manage.cloudless.online. Validates: HSTS, X-Content-Type-Options, X-Frame-Options, CSP, Referrer-Policy, Permissions-Policy. Also checks TLS grade and redirect behaviour (HTTP→HTTPS). Run after any Traefik/Cloudflare config change or before a p…

Check HTTP performance indicators for cloudless.online and/or manage.cloudless.online. Checks: gzip/brotli compression, cache-control directives, Cloudflare cache status (HIT/MISS/BYPASS), ETag presence, and TTFB (time-to-first-byte via curl timing). Run after CDN config changes or to baseline respo…

Compare package.json versions for cloudless.gr and cloudless-manager against npm latest. Flags dependencies that are more than one major version behind (breaking changes likely). Run before a quarterly maintenance pass or when planning upgrades.

Run all app scans (security headers, performance, CSRF guard, WebSocket auth) for cloudless.online and manage.cloudless.online and return a prioritised improvement list. This is the entry point for a full app audit — use it first, then drill into specifics.

List all IAM users in the AWS account with their access key status and last-used date. Runs via the omv-main-cli IAM user configured on the Pi. Returns a markdown table of users, their access keys, key status, and last-used info.

Simulate whether an IAM user is allowed to perform a given action on a resource. Uses aws iam simulate-principal-policy on the Pi. Useful to check if omv-main-cli (or another user) has a specific permission before running a task.

List all ACM (AWS Certificate Manager) certificates in us-east-1. Returns domain name, SANs, status, and expiry for each certificate. Useful to audit SSL/TLS certificates managed in AWS.

Add an inline IAM policy named "route53-delete-hc" to the omv-main-cli IAM user. The policy grants route53:DeleteHealthCheck on all resources. This is an additive IAM change — it does not remove existing permissions. Run aws_check_iam_permissions first to verify the user currently lacks this permiss…

Remove the inline IAM policy "route53-delete-hc" from the omv-main-cli user. Use this after deleting health checks to restore least-privilege access. Runs aws iam delete-user-policy on the Pi.

Delete an ACM certificate by ARN. Safety check: refuses to delete certificates with status ISSUED (still in use by CloudFront/ALB/etc.). Only deletes certificates in PENDING_VALIDATION, EXPIRED, FAILED, or INACTIVE status. Use aws_list_acm_certs to find the ARN first.

Fetch recent logs from the cloudless.gr Lambda function in CloudWatch. Lists available log groups first if no log_group_name is given. Logs are returned newest-first for the specified time window. Filter with a pattern (e.g. "ERROR", "timeout", "[warn]") to narrow results.

Check the status of Route 53 health checks for cloudless.gr. PRIMARY: CloudFront distribution (main path) SECONDARY: API Gateway → Pi 5 (failover path) Returns per-region health check observations and overall HEALTHY/UNHEALTHY status.

List or retrieve SSM parameters for cloudless.gr (prefix: /cloudless/production). SecureString values are masked in list mode. Use parameter_name + decrypt=true to reveal a specific value. Useful for auditing which secrets are configured, checking last-modified dates, or verifying a parameter exists…

Check if the cloudless.gr CloudFront distributions are responding. Tests HTTP response from the apex and www distributions. Returns HTTP status codes and basic response headers to confirm CDN is serving correctly.

Return a static summary of the cloudless.gr AWS infrastructure topology. Includes: Lambda, CloudFront, Route 53, API Gateway, SSM, and Pi failover path. Use as a quick reference when diagnosing issues or explaining the architecture.

Full CI health dashboard for a repo: per-workflow last run grouped by FAIL / RUNNING / OK. Detects the GitHub billing-lock pattern (job fails in <15s with 0 steps). Returns a verdict: HEALTHY | DEGRADED | CRITICAL plus the runner fleet status. Use this as the first call in any CI investigation. Equ…

Get all GitHub Actions check runs for a specific pull request. Returns each check name, status, conclusion, and a direct link to the run. Use to diagnose why a PR is blocked — identifies exactly which checks are failing, pending, or skipped.

Fetch the failed-step logs for a specific GitHub Actions workflow run. Equivalent to 'gh run view <id> --log-failed'. Returns the first `max_lines` lines of failure output so you can diagnose the root cause without opening the browser. Use after gh_ci_summary or gh_pr_checks identifies a failing run…

Analyse the last N runs of a workflow to detect intermittent (flaky) failures. Reports: total runs, success rate, longest failure streak, longest success streak, and a STABLE / FLAKY / CONSISTENTLY_FAILING verdict. FLAKY: success rate between 15% and 85% with at least 2 failures. CONSISTENTLY_FAILI…

Show what is currently deployed to production for cloudless.gr. Reports the most recent successful deploy-pi.yml run: commit SHA, branch, title, timestamp, and how long ago. Also shows the last failed deploy if the most recent deploy attempt was not successful. Use to quickly confirm whether a PR's…

List all DNS records for cloudless.online. Returns record ID, type, name, content, TTL, and proxied status. Always run this before adding or deleting records to check current state.

Add a new DNS record to cloudless.online. Supports A, AAAA, CNAME, TXT. For tunnel CNAMEs use content="<tunnel-id>.cfargotunnel.com".

Update an existing DNS record by ID. Only provided fields are changed. Use cloudflare_list_dns_records first to get the record ID.

Delete a DNS record by ID. This is irreversible — list records first.

Purge Cloudflare edge cache for cloudless.online. Use purge_everything=true to wipe all cached assets (use after major deploys). Or provide specific URLs to purge individual files. Note: purge_everything counts against rate limit — don't run in loops.

Get traffic analytics for cloudless.online via Cloudflare GraphQL API. Returns: requests, bandwidth, cached %, threats, unique visitors. Default: last 24 hours. Requires Zone Analytics:Read on the token.

Get key security and performance settings for cloudless.online zone. Shows: SSL mode, security level, min TLS version, HTTP/2, HTTP/3, HSTS, brotli, rocket loader. Useful for diagnosing TLS issues, checking caching mode, or verifying security posture.

List all API tokens for the Cloudflare account. Shows token ID, name, status, created date, and expiry. Requires the CLOUDFLARE_API_TOKEN to have "User API Tokens:Read" permission. Use this to audit tokens, find IDs for deletion, or check expiry.

List all available Cloudflare API token permission groups with their IDs. Use this to find the correct permission_group IDs before calling cloudflare_create_token. Filter by scope: "zone", "account", "user", or omit for all.

Create a new Cloudflare API token with specified permissions. Requires CLOUDFLARE_API_TOKEN to have "User API Tokens:Edit" permission. WORKFLOW: 1. Run cloudflare_list_permission_groups to find permission group IDs 2. Build policies array with resources + permission_groups 3. Call this tool EXAMPL…

Delete a Cloudflare API token by ID. This is irreversible. Use cloudflare_list_tokens to find the token ID first. Requires "User API Tokens:Edit" permission.

List all Worker routes for cloudless.online. Shows which URL patterns are handled by which Worker scripts. Useful for verifying cloudless-edge Worker is wired to the correct paths.

Check cloudflared tunnel service on omv-main. Tunnel: cloudless-tunnel (ID: a82f24a8-f767-4a59-bc77-1d59ad132be2) Returns systemd status + recent log lines + active connections.

Check cert-manager Certificate resources in K3s. Shows all certs across all namespaces: Ready state, expiry, issuer. Current certs: cloudless-online-tls, auth-cloudless-online-tls.

Restart cloudflared on omv-main. Use when tunnel is stuck or connections drop.

Check the health of one or both Raspberry Pi nodes. Returns: hostname, uptime, disk usage, free memory, load average, and running service count. Use this as a first step to assess overall cluster state.

Execute an arbitrary shell command on a specified Pi node via SSH. Use for ad-hoc diagnostics. Returns stdout + stderr + exit code. CAUTION: avoid commands that modify system state unless you know what you are doing.

List all running systemd services on a Pi node. Useful to verify which services are active and spot unexpected stopped services. Also shows failed units if any.

Check OpenMediaVault NAS services on a Pi node. Checks Samba (file sharing), openmediavault-engined, and lists active Samba connections.

Query active (non-resolved) alerts from the Alert API. The Alert API (FastAPI, alert-manager namespace, NodePort 30800) receives TCP/HTTP probes from the ESP32 watchdog device (192.168.1.201) every 30 seconds. Returns all currently active alerts with code, severity, trigger count, and last seen time…

Manually resolve an active alert in the Alert API. Use after confirming the underlying issue is fixed or auto-remediation has run. The alert remains in history as resolved and will re-fire if the ESP32 detects the problem again. Common codes: OMV_MAIN_K3S_API_DOWN, OMV_HA_K3S_AGENT_DOWN, CLOUDLESS_O…

Retrieve recent alert event history from the Alert API. Returns the last N events across all alert codes (FIRING, RESOLVED, ONGOING, FLAP_SUPPRESSED). FLAP_SUPPRESSED = ESP32 fired an alert but the Alert API's cluster-side check confirmed the service was up (false positive suppressed). Useful for po…

Get the current hardware state of the ESP32 watchdog (192.168.1.201) and combined cluster health. ESP32 hardware data: IP, firmware version, RSSI signal strength, free RAM, uptime, boot count, last heartbeat. System status: combined Pi node + ESP32 + alert summary from the Alert API. RSSI guide: ≥ −…

Check if both Pi nodes are ready to handle a failover scenario. Runs a comprehensive health check on OMV-HA (192.168.1.130) and OMV main (192.168.1.128): - SSH reachability - Disk space on / and /srv - Memory availability - Running services count - Samba share status - Network interface summary Use …

Compare Samba share configuration between OMV-HA and OMV main. Lists configured shares, active connections, and checks if share paths exist on each node. Useful for verifying the failover share setup is mirrored correctly.

Trigger an rsync from OMV main (192.168.1.128) to OMV-HA (192.168.1.130) to keep shares in sync. The rsync runs ON OMV main and pushes to OMV-HA via SSH. Default source path: /srv/dev-disk-by-uuid-*/ (OMV data disk). Always run with dry_run=true first to preview what would be transferred. CAUTION: T…

Verify the full HA failover chain for cloudless.gr (2026-05-23 architecture): 1. Tailscale Funnel active on omv.tail8eb71.ts.net → localhost:18443 2. k3s Traefik VIP (192.168.1.200:18443) health for Host: cloudless.gr and Host: omv.tail8eb71.ts.net 3. CloudFront secondary origin reachability check 4…

Verify the CloudFront origin group failover configuration for cloudless.gr. Checks: - Distribution status (Deployed vs InProgress) - Origins present (default + k3s-ha) - OriginGroup primary-with-ha exists with correct failover codes - DefaultCacheBehavior target is the origin group - /api/* CacheBeh…

End-to-end test of the k3s failover origin as CloudFront would reach it. Tests both host headers that k3s must accept: - Host: cloudless.gr (normal traffic, Cloudflare Tunnel path) - Host: omv.tail8eb71.ts.net (CloudFront secondary origin host) Also tests a GET and a redirect (/) to simulate real pa…

Remove remaining cloudless.online references from the k3s cluster. Actions taken: - Deletes the cloudless-online-tls Certificate resource in the cloudless namespace - Deletes the cloudless-online-tls Secret in the cloudless namespace - Reports any remaining cloudless.online references in manifests D…

Check network connectivity between both Pi nodes and to the internet. Tests: ping between nodes, DNS resolution, internet connectivity, and IPv6 status. Useful for diagnosing failover issues caused by network problems.

Inspect Navbar.tsx on omv-main and verify all features introduced in the navbar refactor: - Scroll shadow (scrolled state + useEffect) - Free Audit CTA (freeAudit i18n key + /contact link) - ARIA accessibility (aria-expanded on hamburger + user menu) - Resize listener (closes mobile menu on viewport…

Validate the PWA manifest configuration for cloudless.gr/cloudless.online. Checks brand-critical fields: theme_color (#0a7785), background_color (#fcfcfd), display_override array, start_url tracking param, id/scope/lang, icon purposes, and shortcuts. source options: - "static" — reads public/ma…

Trigger the deploy-pi.yml GitHub Actions workflow for cloudless.gr (Themis128/cloudless.gr) and watch it to completion. Combines gh_workflow_trigger + gh_workflow_watch in one call. The CI pipeline: lint/typecheck → Next.js build → Docker push to GHCR → k3s rollout. Only dispatches on branches that…

List self-hosted runners registered to a GitHub repo. Returns runner name, status (online/offline), OS, labels, and busy state. Use to verify a runner is online before triggering a workflow.

Register the omv-main self-hosted runner to a GitHub repo. Generates a registration token via GitHub API, SSHs to omv-main, extracts the runner tarball into ~/actions-runner-<repo-short-name>/, runs config.sh, installs and starts a systemd service. Idempotent — skips if already registered.

Trigger a workflow_dispatch on a GitHub repo and return the new run ID. Use to manually kick off a CI/CD pipeline (e.g., deploy-pi.yml, deploy.yml).

Poll a GitHub Actions workflow run until it completes (or times out). Returns job-by-job step status and final conclusion. Poll interval: 20s. Timeout: configurable, default 15 min. Use after gh_workflow_trigger to follow a CI/CD pipeline to completion.

List the most recent workflow runs for a repo, with status and conclusion. Useful for a quick CI health check without needing a specific run ID.

Full health check of a repo's self-hosted runner fleet. Lists each runner (online/offline/busy/labels), counts queued vs in-progress workflow runs, and flags two specific failure modes: • Zombie runner — reported "busy" while zero jobs are actually in_progress AND work is queued (fix: re-regis…

Add custom labels to a self-hosted runner, or replace its full custom label set, via the GitHub API — no re-registration or restart needed. runs-on matches by AND: a job runs on a runner only if that runner carries EVERY label in the job's runs-on array. Use this to partition a fleet into pools — e…

Restart the GitHub Actions self-hosted runner systemd service on omv-main via SSH. Use after applying a systemd override or when the runner appears stuck or offline. Service name is derived as: actions.runner.Themis128-{repo}.{runner}.service

Write an idempotent systemd drop-in override for the runner service on omv-main. Sets: Restart=on-failure, RestartSec=10s, StartLimitIntervalSec=0, network-online.target dependency, RUNNER_RETRY_RENEW_SECONDS=300, DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=1. Also writes /etc/systemd/resolved.conf…

Fetch recent journalctl logs from the GitHub Actions runner service on omv-main. Useful for diagnosing connection drops, DNS failures, job hangs, or stuck runs.

Cancel all queued or in-progress workflow runs for a repo. Use when the self-hosted runner crashed mid-job and GitHub still shows runs as queued/in-progress. Fetches both queued and in-progress runs, optionally filters by workflow name, and cancels each. Set dry_run=true to preview without cancellin…

Check Grafana's health endpoint (/api/health). Returns the running version, database state, and git commit. Access is via SSH to omv-main (Grafana is not exposed externally).

List all dashboards in Grafana. Returns a markdown list with folder, title, and UID for each dashboard. Use the UID to construct a Grafana URL for direct access.

List all configured datasources in Grafana. Returns name, type, URL, and whether each is the default datasource. Useful for verifying Prometheus, Loki, or other backends are connected.

Restart the Grafana deployment in the monitoring namespace to pick up ConfigMap or secret changes. Runs: kubectl rollout restart deployment/kube-prom-grafana -n monitoring Then waits for the rollout to complete (up to 3 minutes). Use grafana_check_health after to confirm the new pod is ready.

List active Grafana-managed alerts via the Alertmanager API. Returns alerts with their labels, severity, and when they started firing. This covers Grafana-managed alert rules (not Prometheus alerting rules — use prometheus_check_alerts for those).

Install or upgrade a Helm chart on the k3s cluster. Uploads a local values file via SFTP to omv-main, then runs helm upgrade --install. Returns helm output including status and deployed resources.

List all Helm releases across all namespaces on the k3s cluster.

Get detailed status of a specific Helm release including deployed resources.

Uninstall a Helm release from the k3s cluster.

Get a full status snapshot of the K3s Kubernetes cluster on OMV main (Pi 5). Returns: cluster nodes, all pods across namespaces, all services, and K3s systemd service status. Use first to assess overall cluster health.

List pods in the K3s cluster, optionally filtered by namespace. Returns pod names, namespace, status, restart count, and age.

Fetch logs from a K3s pod by namespace and label selector or pod name. Examples: - Home Assistant: namespace="home-assistant", selector="app=home-assistant" - cloudless.gr: namespace="cloudless", selector="app=cloudless" Returns the last N lines of logs.

Perform a rolling restart of a K3s deployment. Common deployments: - Home Assistant: namespace="home-assistant", deployment="home-assistant" - cloudless.gr app: namespace="cloudless", deployment="cloudless" Returns the rollout status after restart.

Check Home Assistant pod, logs, and service in the K3s cluster. Returns: pod status, last 50 log lines, service port, and recent events. Shortcut — no need to know the exact pod name or namespace.

Check the cloudless.online Next.js app running as a K3s deployment in the cloudless namespace. Returns: pod status, service, ingress, health check via Traefik VIP, and current IPv6 address.

Pre-pull a container image into the k3s containerd store (k8s.io namespace) on omv-main before a rollout, so pod startup is near-instant instead of waiting 4+ minutes for ECR pull. Uses: sudo ctr -n k8s.io images pull (the only tool that correctly targets the k8s.io namespace and honours ECR creden…

Get detailed description of a K3s resource (pod, deployment, service, ingress, etc.). Equivalent to: kubectl describe <kind> <name> -n <namespace> Useful for debugging pod scheduling issues, CrashLoopBackOff, image pull errors, etc.

Check Metabase pod status and optionally verify login. Returns pod state, restart count, readiness, and API health endpoint response. Note: Metabase takes ~4.5 minutes to become ready after startup (initialDelaySeconds=240).

Run a SQL query against the Metabase H2 embedded database. IMPORTANT: Metabase MUST be scaled to 0 before running — the H2 database cannot be accessed by two JVM processes simultaneously. This tool scales Metabase down, runs the query, then scales it back up. H2 facts: - DB path: /metabase-data/met…

Reset the Metabase admin password via H2 direct database update. Procedure: scale to 0 → apply H2 reset job → verify → scale back to 1. The hash must be a bcrypt hash using $2a$ prefix (NOT $2b$) — jBCrypt in Metabase v0.55 rejects $2b$. Generate via Python: import bcrypt, uuid salt = str(uuid.…

Fix the DuckDB file lock conflict that occurs when Metabase restarts. Symptom: "IO Error: Conflicting lock is held in PID 0" on analytics.duckdb. Fix: restart duckdb-api first (clears the stale lock), then restart Metabase. The lock occurs because the previous Metabase pod held the lock; when killed…

Get recent logs from the Metabase pod. Note: metabase.middleware.log DEBUG spam (health-check lines) cannot be suppressed in v0.55 OSS — Metabase programmatically resets logger levels. This is cosmetic only and does not affect function.

Check status of all ML pipeline CronJobs in the analytics namespace. Shows: last schedule time, last successful run, active jobs, recent pod states. Also reports recent Job runs (last 3 per CronJob) with completion status and duration. Use this to diagnose pipeline failures or verify a successful we…

Create an immediate Job from an ML CronJob (equivalent to kubectl create job --from=cronjob). Use this to run a pipeline step on-demand without waiting for the next schedule. Normal trigger order: ml-feature-engineer → ml-train-rfm → ml-train-churn → ml-train-collab → ml-train-anomaly. ⚠️ concurren…

Stream logs from the most recent pod of an ML CronJob. Use after ml_trigger_job or ml_pipeline_status to diagnose failures or confirm success. Fetches the last 200 lines by default.

Query the ml_runs table in DuckDB for training run history. Returns: model name, training timestamp, metrics (AUC/inertia/etc), training row count, champion flag. Requires at least one successful ml-feature-engineer + training run to have data.

Query ML score tables from DuckDB. Available tables: scores_rfm (user segments), scores_churn (churn probability), scores_recs (recommendations), anomaly_flags (API anomalies), scores_decay (content decay). Returns the first N rows with all columns.

List ML model files stored in S3 under the ml-models/ prefix. Shows: model name, version (latest + dated backups), file size, last modified. Use this to confirm a training job successfully saved its model artifact.

Query the most recent anomaly_flags from DuckDB. The anomaly-detect CronJob runs every 15 minutes scoring API request windows. Returns: window_start, is_anomaly, anomaly_score, request_count, error_rate, p95_ms. High anomaly_score (closer to 0 or positive) = more anomalous; threshold is typically ar…

Query ml_features table to see what feature sets have been computed and when. Shows: feature_name, computed_at, row_count, s3_parquet_path. Use to verify the ml-feature-engineer job ran successfully before triggering training jobs.

Clear a stale PID-0 DuckDB file lock that causes HTTP 503 from duckdb-api. This happens when a process dies without releasing the lock embedded in the .duckdb file header. Recovery procedure (automated): 1. Scale duckdb-api and metabase to 0 replicas (releases open file handles) 2. Run a one-shot d…

Check the k3s-agent service on omv-ha (Pi 4, 192.168.1.130). omv-ha was demoted from control-plane+etcd to agent-only on 2026-05-24 to fix 2-node etcd instability. Returns: k3s-agent service state, keepalived state, recent agent logs, memory, load average, and swap. Also verifies omv-ha appears in k…

Restart the k3s-agent systemd service on omv-ha (Pi 4, 192.168.1.130). Use when omv-ha shows NotReady in kubectl nodes, or when k3s-agent is stuck/crashed. Waits 5 seconds and reports the new service state. This does NOT re-register the node — use omv_ha_agent_rejoin for a full reinstall.

Full reinstall of k3s-agent on omv-ha after a hard disconnect. Use ONLY when omv-ha is completely missing from kubectl nodes (not just NotReady). Procedure (from Runbook — omv-ha Demotion: server → agent, 2026-05-24): Step 1: Delete node object from cluster (on omv-main) Step 2: Uninstall k3s-ag…

Run a PromQL instant query against Prometheus in the monitoring namespace. Access is via SSH to omv-main (Prometheus is not exposed externally). Returns the result values formatted as a readable table. Examples: "up", "kube_pod_status_phase{namespace='default'}", "node_memory_MemAvailable_bytes"