How do I build storage for VMs and Kubernetes that can survive restarts?

To make storage survive restarts for VMs and Kubernetes, you want persistent storage that lives outside the lifecycle of the guest OS/container/node. The core idea is:

• Don’t store important data on ephemeral local disk
• Put data on persistent volumes / datastores
• Reattach those volumes to VMs, pods, or containers after restarts

Here’s a practical way to think about it.

---

1) The main building blocks

For VMs

A VM usually needs:

• Boot disk: OS image, maybe persistent
• Data disks: application data, databases, logs, etc.

To survive restarts, the disks must be backed by something persistent:

• Virtual disk files on a durable datastore
• Block storage from SAN / iSCSI / Fibre Channel / NFS / Ceph / cloud disks
• Shared storage accessible to the hypervisor cluster

If the VM restarts, its virtual disks are reattached, so data remains.

---

For Kubernetes

Kubernetes pods are ephemeral, so local container filesystem is not persistent.

Use:

• PersistentVolume (PV): actual storage resource
• PersistentVolumeClaim (PVC): request for storage by a pod
• StorageClass: how storage is dynamically provisioned

When the pod restarts or is rescheduled, the PVC is re-mounted, and the data remains.

---

2) Typical storage options

Option A: Network-attached block storage

Examples:

• iSCSI
• Fibre Channel
• Cloud disks (EBS, PD, Managed Disks)
• Ceph RBD

Best for:

• Databases
• VM disks
• Stateful apps needing low-latency block storage

Pros:

• Good performance
• Supports filesystem or raw block use
• Works well for single-writer workloads

Cons:

• Usually attached to one node at a time
• Requires proper attach/detach handling

---

Option B: Shared file storage

Examples:

• NFS
• SMB/CIFS
• CephFS
• GlusterFS

Best for:

• Shared app data
• Multiple pods needing same files
• Home directories, content storage

Pros:

• Multiple clients can mount simultaneously
• Easy to use

Cons:

• Not ideal for heavy database workloads
• Performance depends on backend

---

Option C: Distributed storage platform

Examples:

• Ceph
• Longhorn
• OpenEBS
• Portworx
• vSAN

Best for:

• Kubernetes-native persistent storage
• VM + container converged environments
• High availability setups

Pros:

• Replication, failover, dynamic provisioning
• Can survive node loss
• Integrates with orchestration

Cons:

• More operational complexity
• Needs careful sizing and monitoring

---

Option D: Local disk with replication

Examples:

• RAID, ZFS, LVM + replication, DRBD, rsync-based backups

Best for:

• Small setups
• Edge appliances
• Special-purpose systems

Pros:

• Cheap, fast, simple on a single host

Cons:

• Usually not enough for true node failure survivability unless replicated elsewhere

---

3) How to design it so restarts don’t lose data

For VMs

Use:

1. Persistent datastore for virtual disks
2. Separate system disk and data disks
3. Snapshots/backups for recovery
4. Redundant storage backend if uptime matters

Typical layouts:

• VM OS disk on shared storage
• Database data on separate persistent volume
• Logs and cache possibly separate, depending on need

If a host reboots:

• VM restarts
• Virtual disks reattach
• Data remains intact

If a host dies:

• HA cluster or orchestration migrates VM to another host
• Storage must still be reachable

---

For Kubernetes

Use:

1. PVCs for all stateful data
2. StatefulSet for ordered, stable identity workloads
3. StorageClass with dynamic provisioning
4. ReadWriteOnce for single-writer volumes, or ReadWriteMany for shared volumes
5. Backups separate from storage replication

Example:

• PostgreSQL in Kubernetes:
  • Pod uses PVC
  • PVC backed by block storage
  • Pod restarts on another node
  • Volume is reattached and data persists

---

4) Common patterns

Pattern 1: VMs on a shared datastore

• Hypervisors connect to SAN/NAS/Ceph
• VM disks live there
• Any node can restart the VM

Good for:

• VMware, Proxmox, Hyper-V, KVM clusters

---

Pattern 2: Kubernetes with a CSI driver

• Storage backend exposes volumes through CSI
• Pods request PVCs
• Storage is created and attached automatically

Examples:

• AWS EBS CSI driver
• GCE PD CSI driver
• Ceph CSI
• Longhorn CSI

This is the standard modern approach.

---

Pattern 3: Kubernetes + shared file storage

• NFS export mounted into pods
• Good for shared content and simple persistence

Use carefully for databases.

---

5) Important reliability principles

Separate compute from storage

If a node dies, storage should not die with it.

Replicate storage

Use at least 2–3 copies or RAID plus backups, depending on requirements.

Don’t rely only on restart

A restart surviving does not equal data safe:

• reboot
• node failure
• disk failure
• cluster failure
• corruption
• operator mistakes

You need backups too.

Pick the right access mode

• RWO: one writer, common for databases and VM disks
• RWX: multiple writers, shared content
• ROX: read-only replicas

Use a filesystem or block device appropriately

• Block device for databases and VM disks
• Filesystem for documents, logs, shared app files

---

6) A simple reference architecture

If you want a robust setup for both VMs and Kubernetes:

• Storage backend: Ceph or SAN/NAS with redundancy
• VM platform: shared datastores for virtual disks
• Kubernetes: CSI driver for dynamic PVs
• Backups: separate backup system to object storage or backup server
• Monitoring: storage health, latency, capacity, and replication status

---

7) Practical recommendation by environment

Small home lab

• Proxmox + ZFS or Ceph
• NFS for simple shared storage
• Backups to another disk or machine

Small/medium production

• KVM/Proxmox/VMware with shared storage
• Kubernetes with CSI-backed PVCs
• Separate backup target

Larger production

• Distributed storage like Ceph, Portworx, vSAN, or enterprise SAN
• Proper monitoring, redundancy, and backup/DR plan

---

8) The shortest answer

To survive restarts:

• VMs: put VM disks on persistent shared storage or replicated block storage
• Kubernetes: use PersistentVolumes/PersistentVolumeClaims backed by a real storage system through a CSI driver
• Always back up data separately

If you want, I can also give you:

1. a diagram of a good VM + Kubernetes storage design, or
2. a step-by-step setup for Proxmox, KVM, Kubernetes, Ceph, or NFS.