Skip to content

Troubleshooting

Current troubleshooting procedures for the single-node Oracle Cloud K3s cluster.

1. Establish Cluster Health

Run the read-only health checks first:

kubectl get nodes -o wide
kubectl get applications -n argocd
kubectl get pods -A

The expected control-plane node is Ready, ArgoCD applications are Synced/Healthy, and pods are Running or an expected Completed job.

2. Storage and Longhorn

Longhorn runs on the single K3s node with replication factor one. A healthy volume does not provide node-level high availability.

kubectl get pvc -A
kubectl get volumes.longhorn.io -n longhorn-system
kubectl get pods -n longhorn-system

Check the Longhorn UI through the internal HTTPS route before restarting storage components. Do not delete PVCs, snapshots, or finalizers during routine troubleshooting.

3. Database Connectivity

PostgreSQL is provided by pg in namespace db and is reached through pg-svc.db:5432.

kubectl get pod -n db pg-58f8cff6c-n44f6 -o wide
kubectl get svc -n db pg-svc
kubectl get networkpolicy -n db

Confirmed consumers are n8n, NocoDB, and pgAdmin. Check NetworkPolicy selectors before changing application configuration.

4. Monitoring

Prometheus, Grafana, and Alertmanager use node-bound local-path PVCs. Current persistent allocations are Prometheus 5 GiB, Grafana 1 GiB, and Alertmanager 500 MiB. Prometheus retention is one day.

kubectl get pvc -n monitoring
kubectl get pods -n monitoring
kubectl get networkpolicy -n monitoring

Avoid deleting StatefulSets or PVCs as a first response. Use GitOps changes for storage configuration and preserve the existing data path.

5. NetworkPolicy and Pod Security

NetworkPolicy enforcement is enabled. Application policies use namespace and pod selectors; direct NodePort access uses the Tailnet range.

kubectl get networkpolicy -A
kubectl describe networkpolicy <policy> -n <namespace>
kubectl get pods -A --show-labels
kubectl get ns -o custom-columns=NAME:.metadata.name,WARN:.metadata.labels.pod-security\\.kubernetes\\.io/warn,AUDIT:.metadata.labels.pod-security\\.kubernetes\\.io/audit,ENFORCE:.metadata.labels.pod-security\\.kubernetes\\.io/enforce

Application namespaces use PSA warn=baseline and audit=baseline. No enforce labels are set. Infrastructure namespaces remain separately reviewed.

6. Internal HTTPS

Traefik terminates internal HTTPS using certificates issued by cert-manager from a private CA.

kubectl get certificate -A
kubectl get ingress -A
curl -I https://<internal-domain>/

Install the private CA on trusted administrative clients. Do not bypass certificate verification in normal browser use.

7. GitOps Recovery

K3s desired state is stored in k3s-repo and reconciled by ArgoCD.

kubectl get application -n argocd
kubectl describe application <application> -n argocd

Pull the repository before editing. Validate and push changes through Git; avoid applying production manifests directly with kubectl.

Changelog

Date Change
Sep 15, 2026 Refreshed procedures for the single-node K3s host, enforced NetworkPolicy, PSA audit mode, and internal HTTPS.
Feb 13, 2026 Added monitoring stack migration report.
Feb 09, 2026 Added PostgreSQL connection checks and Longhorn UI details.
Jan 20, 2026 Initial placeholder created.