Skip to content

Phase 7: Production Application Fleet Deployment & Storage Tiering

Milestone Attribute Engineering Specification
Phase Scope Workload Fleet Onboarding, Stateful HA Database & Slack ChatOps Routing
Target Infrastructure K3s Kubernetes Cluster, CloudNativePG, 1TB SATA HDD, OCI Mumbai Uptime Kuma
Primary Code Paths kubernetes/apps/, kubernetes/platform/, infrastructure/oci/
Relevant Decisions ADR-009, ADR-010, ADR-012, ADR-013
Operational Status Production Verified (Platform v3.0.0)

1. Executive Summary & Objective

Phase 7 represents the culminating implementation phase of the platform modernization. With the physical hypervisor consolidated, cloud ingress optimized, GitOps bootstrapped, and storage partitioned, the full production fleet of 9 containerized workloads and the platform documentation portal are deployed onto k3s-prod.

Key achievements in Phase 7: 1. Core Stateful Resilience: CloudNativePG (CNPG) delivers declarative PostgreSQL high availability with continuous Write-Ahead Log (WAL) streaming. 2. Security Hardening: n8n workflow engine is hardened to run as a non-root container with dropped Linux capabilities and zero host SSH access. 3. Public Documentation Presence: The Homelab Documentation Portal (https://docs.vijaysingh.cloud) is live with sub-15ms edge latency. 4. Physical Storage Tiering: Heavy document and media workloads (Paperless-ngx, BookOrbit, Audiobookshelf) are mounted directly to the physical 1TB mechanical SATA HDD, preventing flash wear on the NVMe SSD. 5. Out-of-Band Observability: In-cluster Prometheus/Grafana and external Uptime Kuma on OCI Mumbai deliver automated health alerts to 5 dedicated Slack channels.

2. Engineering Rationale: Security & Storage Optimization

A. Neutralizing the Legacy n8n Vulnerability (ADR-009)

The historical n8n deployment ran with hostNetwork: true, ran as root in initContainers, and mounted private SSH keys to execute commands on the Proxmox host.

  • The Remediation: n8n is strictly isolated in its own Kubernetes namespace (automation) with non-root UID 1000, dropped Linux capabilities (drop: ["ALL"]), pure cluster-internal networking, and all host SSH keys purged. Any infrastructure automation must communicate via authenticated REST APIs or event webhooks.

B. Enterprise Database High Availability (ADR-010)

Replacing standalone PostgreSQL containers with CloudNativePG provides:

  • Continuous streaming WAL archiving for Point-in-Time Recovery (PITR).
  • Automated failover, self-healing replicas, and zero-downtime rolling upgrades.
  • RPO reduced from 24 hours to <15 minutes.

C. Physical Dual-Tier Storage Partitioning (ADR-013)

  • Hot Tier (256GB NVMe): High-IOPS transactional storage for etcd state and PostgreSQL active tables.
  • Cold Tier (1TB SATA HDD): Unstructured bulk files (scanned document PDFs, eBooks, audiobooks, and backup snapshots) mounted to /mnt/hdd/ via PersistentVolumes. This guarantees large bulk files never consume high-speed SSD space.

3. Application Fleet Architecture Matrix

Workload Namespace Public Ingress / Endpoint Storage Tier Operational Function
CloudNativePG database ClusterInternal NVMe (Hot) + HDD (WAL) Declarative HA PostgreSQL cluster for n8n, Miniflux, and Paperless
n8n (Hardened) automation hooks.vijaysingh.cloud NVMe Central automation hub piping RSS digests and incident alerts to Slack
Homelab Docs platform docs.vijaysingh.cloud NVMe (MkDocs) Architecture handbook and operational showcase portal
Homepage platform dash.vijaysingh.cloud (SSO) NVMe Central service directory with live K8s, Proxmox, and resource widgets
BookOrbit media books.vijaysingh.cloud (SSO) 1TB HDD (/mnt/hdd/books) Multi-user eBook, PDF, and comic library with reading progress sync
Paperless-ngx documents docs-ocr.vijaysingh.cloud (SSO) 1TB HDD (/mnt/hdd/paperless) OCR document scanning, tagging, and searchable PDF archive
Audiobookshelf media audio.vijaysingh.cloud (SSO) 1TB HDD (/mnt/hdd/media) Streaming server for audiobooks and podcasts
Miniflux productivity rss.vijaysingh.cloud (SSO) NVMe Lightweight Go RSS reader with automated daily digests
Linkding productivity links.vijaysingh.cloud (SSO) NVMe Minimalist, searchable bookmark manager
Wger / Ryot health health.vijaysingh.cloud (SSO) NVMe Workout, calorie, and meal planning tracker
Uptime Kuma (OCI Mumbai) status.vijaysingh.cloud OCI Compute Out-of-band health probes dispatching alerts to Slack #homelab-alerts

4. Technical Execution Details

1. Deploying CloudNativePG HA Operator & Cluster

The declarative PostgreSQL cluster was deployed via kubernetes/platform/postgres-operator/cluster.yaml:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgres-ha
  namespace: database
spec:
  instances: 2
  primaryUpdateStrategy: unsupervised
  storage:
    size: 20Gi
    storageClass: local-path # NVMe Hot Tier
  backup:
    barmanObjectStore:
      destinationPath: s3://postgres-backups/
      endpointURL: http://minio.database.svc.cluster.local:9000
      s3Credentials:
        accessKeyId:
          name: minio-creds
          key: ACCESS_KEY
        secretAccessKey:
          name: minio-creds
          key: SECRET_KEY
      wal:
        compression: gzip

2. Hardened n8n Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: n8n
  namespace: automation
spec:
  replicas: 1
  template:
    spec:
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
      containers:
      - name: n8n
        image: n8nio/n8n:latest
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        env:
        - name: DB_TYPE
          value: postgresdb
        - name: DB_POSTGRESDB_HOST
          value: postgres-ha-rw.database.svc.cluster.local

3. PersistentVolumes for Cold Storage Tier

PersistentVolumes mapping /mnt/hdd/ were created for document and media storage:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-hdd-books
spec:
  capacity:
    storage: 400Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /mnt/hdd/books
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-hdd-paperless
spec:
  capacity:
    storage: 200Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /mnt/hdd/paperless

4. Out-of-Band Uptime Kuma Deployment (OCI Mumbai)

Deployed Uptime Kuma on the OCI Mumbai instance via Ansible:

cd configuration
ansible-playbook -i inventory/hosts.yml playbooks/deploy_uptime_kuma.yml

Configured active HTTP probes monitoring:

  • https://docs.vijaysingh.cloud
  • https://hooks.vijaysingh.cloud
  • https://dash.vijaysingh.cloud
  • https://docs-ocr.vijaysingh.cloud
  • https://rss.vijaysingh.cloud

5. Multi-Channel Slack ChatOps Routing

Configured incident and event routing across dedicated Slack channels:

Slack Workspace Integrations:
├── #daily-briefing    <── Miniflux AI curated news digest (n8n daily at 08:00)
├── #documents         <── Paperless-ngx OCR completion notifications
├── #homelab-alerts    <── Uptime Kuma out-of-band availability alerts
├── #infra-monitoring  <── Prometheus node memory and disk threshold alerts
└── #deployments       <── Flux CD v2 GitOps reconciliation events

5. Verification & Quality Assertions

1. Workload Pod Health

kubectl get pods -A
# Output: All pods in automation, database, platform, media, documents, and productivity are Running.

2. Edge Latency & SSL Verification

curl -I https://docs.vijaysingh.cloud
curl -I https://hooks.vijaysingh.cloud
# Output: HTTP/2 200 OK, valid Cloudflare Edge SSL, TTFB <50ms.

3. Out-of-Band Alerting Simulation

Tested failure alerting by scaling down the ingress connector: 

kubectl -n platform scale deployment cloudflared --replicas=0
# Output: Within 60 seconds, Uptime Kuma on OCI Mumbai dispatches an outage alert to Slack #homelab-alerts.
kubectl -n platform scale deployment cloudflared --replicas=1
# Output: Uptime Kuma dispatches a "Service Restored" recovery alert.

6. Exit Gate & Modernization Conclusion

Phase 7 concludes the infrastructure modernization program. The platform is now fully deployed, operating as an autonomous, self-healing, single-node sovereign cloud with zero recurring cloud dependencies and comprehensive observability.