Fix lost+found directory in longhorn Kubernetes storage

Longhorn is a 100% open source and simplified cloud-native persistent block storage for your Kubernetes environment. We deployed Longhorn in a Kubernetes cluster as can be seen here:

$ kubectl get storageclasses
NAME                     PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
longhorn (default)       driver.longhorn.io   Delete          Immediate           true                   20h

While deploying PostgreSQL container in Kubernetes with persistent storage request, the pod couldn’t initialize and it was giving the following error:

$ kubectl logs postgres-0
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

initdb: error: directory "/var/lib/postgresql/data/pgdata" exists but is not empty
initdb: detail: It contains a lost+found directory, perhaps due to it being a mount point.
initdb: hint: Using a mount point directly as the data directory is not recommended.
Create a subdirectory under the mount point.

If you describe the storageclass you will notice it uses ext4 filesystem which automatically adds a folder lost+found when a PV is created.

$ kubectl get sc longhorn -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    longhorn.io/last-applied-configmap: |
      kind: StorageClass
      apiVersion: storage.k8s.io/v1
      metadata:
        name: longhorn
        annotations:
          storageclass.kubernetes.io/is-default-class: "true"
      provisioner: driver.longhorn.io
      allowVolumeExpansion: true
      reclaimPolicy: "Delete"
      volumeBindingMode: Immediate
      parameters:
        numberOfReplicas: "3"
        staleReplicaTimeout: "30"
        fromBackup: ""
        fsType: "ext4"
        dataLocality: "disabled"
        unmapMarkSnapChainRemoved: "ignored"
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2024-08-12T14:33:38Z"
  name: longhorn
  resourceVersion: "4578"
  uid: 4ac796a7-4bda-44ae-8336-f56335bcb499
parameters:
  dataLocality: disabled
  fromBackup: ""
  fsType: ext4
  numberOfReplicas: "3"
  staleReplicaTimeout: "30"
  unmapMarkSnapChainRemoved: ignored
provisioner: driver.longhorn.io
reclaimPolicy: Delete
volumeBindingMode: Immediate

To fix this issue, we will need to recreate the storage class which uses XFS filesystem.

Let’s create a file longhorn-xfs-sc.yaml with the following content:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-xfs
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "30"
  fromBackup: ""
  fsType: "xfs"
  dataLocality: "disabled"
  unmapMarkSnapChainRemoved: "ignored"

Create a new StorageClass using the kubectl command:

$ kubectl apply -f longhorn-xfs-sc.yaml
storageclass.storage.k8s.io/longhorn-xfs created

After some minutes the PostgreSQL could initialize and start properly as seen in the loogs.

$ kubectl logs postgres-0
...
server started
CREATE DATABASE

/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

waiting for server to shut down...2024-08-13 21:53:06.653 UTC [49] LOG:  received fast shutdown request
.2024-08-13 21:53:06.657 UTC [49] LOG:  aborting any active transactions
2024-08-13 21:53:06.658 UTC [49] LOG:  background worker "logical replication launcher" (PID 55) exited with exit code 1
2024-08-13 21:53:06.659 UTC [50] LOG:  shutting down
2024-08-13 21:53:06.661 UTC [50] LOG:  checkpoint starting: shutdown immediate
2024-08-13 21:53:06.745 UTC [50] LOG:  checkpoint complete: wrote 922 buffers (5.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.074 s, sync=0.007 s, total=0.086 s; sync files=301, longest=0.004 s, average=0.001 s; distance=4255 kB, estimate=4255 kB; lsn=0/1912108, redo lsn=0/1912108
2024-08-13 21:53:06.750 UTC [49] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2024-08-13 21:53:06.778 UTC [1] LOG:  starting PostgreSQL 16.4 (Debian 16.4-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2024-08-13 21:53:06.778 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-08-13 21:53:06.778 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2024-08-13 21:53:06.782 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-08-13 21:53:06.787 UTC [65] LOG:  database system was shut down at 2024-08-13 21:53:06 UTC
2024-08-13 21:53:06.792 UTC [1] LOG:  database system is ready to accept connections

You have the option of deleting the storageclass that uses ext4 to create filesystem and maintain the one for the XFS, or you can choose to have both.

Your IT Journey Starts Here!

Ready to level up your IT skills? Our new eLearning platform is coming soon to help you master the latest technologies.

Be the first to know when we launch! Join our waitlist now.

Join our Linux and open source community. Subscribe to our newsletter for tips, tricks, and collaboration opportunities!

Recent Post

Leave a Comment

Your email address will not be published. Required fields are marked *

Related Post

Let’s clarify the differences between merge requests (commonly called pull requests in GitHub), releases, release candidates (RCs), tags, and branches […]

Kind (which in full means “Kubernetes IN Docker”), is a command line tool that enables you to run Kubernetes clusters […]

Are you looking for an easy way to migrate packages from one cPanel server to a new cPanel server? In […]

Let's Connect

Unleash the full potential of your business with CloudSpinx. Our expert solutions specialists are standing by to answer your questions and tailor a plan that perfectly aligns with your unique needs.
You will get a response from our solutions specialist within 12 hours
We understand emergencies can be stressful. For immediate assistance, chat with us now

Contact CloudSpinx today!

Download CloudSpinx Profile

Discover the full spectrum of our expertise and services by downloading our detailed Company Profile. Simply enter your first name, last name, and email address.