Rook is a robust and open source solution used to orchestrate distributed storage systems, particularly in cloud native environments. The core of Rook is Ceph, which is an open source distributed storage system. It has support for a variety of multiple storage options, such block storage, object storage, as well as POSIX-compliant shared file system. When there is a need to deploy an application that needs to write to an bucket within the cluster rook ceph can be used for this purpose.

In this blog, we look at how you can:

1. Create a Ceph Bucket when using Rook
2. Create User Credentials used to access the bucket
3. Connecting to the bucket endpoint using Rados GW
4. Uploading and downloading data from the bucket

The requirements for this setup are:

• Functional rook ceph deployment (assumption here is that it’s deployed in rook-ceph namespace)
• Access to Kubernetes cluster as admin user

1. Confirm rook ceph is working

Use kubectl command to check for the health status of your Rook cluster.

ROOK_NAMESPACE=rook-ceph
kubectl -n $ROOK_NAMESPACE get pods

All pods should be running in the Rook namespace. If not running the state should be Completed.

...
NAME                                                        READY   STATUS      RESTARTS         AGE
csi-cephfsplugin-czp6l                                      2/2     Running     5 (4h6m ago)     55d
csi-cephfsplugin-d7mnc                                      2/2     Running     0                24d
csi-cephfsplugin-g94km                                      2/2     Running     2 (24d ago)      25d
csi-cephfsplugin-j9789                                      2/2     Running     1 (24d ago)      24d
csi-cephfsplugin-provisioner-6d95d7699-5ks9k                5/5     Running     0                83d
csi-cephfsplugin-provisioner-6d95d7699-wdt6b                5/5     Running     0                24d
csi-cephfsplugin-t86th                                      2/2     Running     0                83d
csi-rbdplugin-6k26h                                         2/2     Running     1 (24d ago)      24d
csi-rbdplugin-8knl9                                         2/2     Running     5 (4h6m ago)     55d
csi-rbdplugin-provisioner-5f6987779d-jd75f                  5/5     Running     0                4h15m
csi-rbdplugin-provisioner-5f6987779d-zq2cm                  5/5     Running     0                56d
....

After you’ve confirmed pods are running, use Rook toolbox to get the actual status of rook cluster.

kubectl apply -f https://raw.githubusercontent.com/rook/rook/refs/heads/master/deploy/examples/toolbox.yaml

After the rook-ceph-tools pod is started, you can connect to it with:

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

Confirm if the overall cluster status is HEALTH_OK and there are no warning or error status messages displayed.

$ ceph -s
  cluster:
    id:     5fd1e75f-e7ed-42e9-8814-dd30c10983d6
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum a,c,d (age 4h)
    mgr: b(active, since 4h), standbys: a
    mds: 1/1 daemons up, 1 hot standby
    osd: 14 osds: 14 up (since 4h), 14 in (since 3w)
    rgw: 1 daemon active (1 hosts, 1 zones)

  data:
    volumes: 1/1 healthy
    pools:   12 pools, 265 pgs
    objects: 211.43k objects, 322 GiB
    usage:   882 GiB used, 3.6 TiB / 4.5 TiB avail
    pgs:     265 active+clean

  io:
    client:   4.4 MiB/s rd, 5.8 MiB/s wr, 3.66k op/s rd, 5.06k op/s wr

2. Create Ceph Bucket Storage Class

Ideally, this is available by default in rook-ceph namespace. List configured storage classes using the command below:

$ kubectl get sc
NAME                   PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
ceph-block (default)   rook-ceph.rbd.csi.ceph.com      Delete          Immediate           true                   29d
ceph-bucket            rook-ceph.ceph.rook.io/bucket   Delete          Immediate           false                  29d
ceph-filesystem        rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   29d

The following manifest contents can be used to create a storage class named ceph-bucket with rook-ceph.ceph.rook.io/bucket provisioner.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: ceph-bucket
# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
provisioner: rook-ceph.ceph.rook.io/bucket
reclaimPolicy: Delete
parameters:
  objectStoreName: ceph-objectstore
  objectStoreNamespace: rook-ceph
  region: "us-east-1"

The storageclass should have the provisioner set to “<provisioner-namespace>.ceph.rook.io/bucket“.

And the CephObjectStore resource should have ceph-objectstore name as referenced in storageclass object.

$ kubectl get -n rook-ceph CephObjectStore
NAME               PHASE   ENDPOINT                                                 SECUREENDPOINT   AGE
ceph-objectstore   Ready   http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:80                    29d

To output as yaml, run the following command:

kubectl get -n rook-ceph CephObjectStore -o yaml

Sample CephObjectStore creation manifest:

apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: ceph-objectstore
  namespace: rook-ceph
spec:
  metadataPool:
    failureDomain: host
    replicated:
      size: 3
  dataPool:
    failureDomain: host
    #replicated:
    #  size: 3
    erasureCoded:
      dataChunks: 2
      codingChunks: 1
  gateway:
    port: 80
    # securePort: 443
    # sslCertificateRef:
    instances: 1
    priorityClassName: system-cluster-critical

3. Request for an s3 bucket claim

The easiest way to request for S3 compatible bucket backend for your workloads in kubernetes is using Object Bucket Claim custom resource. When a bucket is provisioned, an Object Bucket (OB) is a custom resource automatically generated, and it contains information specific to the bucket.

We can create a test bucket OBC:

vim testobc.yaml

Adjust the settings in the manifest below to match your Rook setup. Just keep in mind – the ObjectBucketClaim CR is namespace scoped.

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: test-bucket
  namespace: default
spec:
  bucketName: test-bucket
  storageClassName: ceph-bucket
  additionalConfig:
    maxSize: "2Gi"

Where:

• metadata.name sets the name of the ObjectBucketClaim. The same name is used for the the Secret and ConfigMap created.
• metadata.namespace: The namespace where the OBC lives, and it’s also the namespace of the ConfigMap and Secret generated.
• spec.bucketName: Sets the name of the bucket. Just a heads-up – this has to be unique across the whole object store.
• spec.storageClassName: The name of the StorageClass defining which provisioner to use, the object store to connect to, and how the bucket should be retained.
• spec.additionalConfig: Optional list of key-value pairs that lets you set extra settings specific to the bucket created by this OBC. Options supported are:

• maxSize: Limits the total bucket size. Minimum recommended value is 4K.
• maxObjects: Sets a limit on how many objects can go into the bucket.
• bucketMaxObjects: (Disabled by default) Sets a per-bucket object limit, useful when multiple users share the same bucket.
• bucketMaxSize: (Disabled by default) Sets a per-bucket size limit, helpful for managing shared bucket storage.
• bucketPolicy: (Disabled by default) A raw JSON string defining an AWS S3 bucket policy. If set, it overrides any existing or default bucket policy automatically generated by the provisioner.
• bucketLifecycle: (Disabled by default) A raw JSON string defining an AWS S3 bucket lifecycle configuration. The rules must be sorted by ID to ensure idempotency.
• bucketOwner: (Disabled by default) Specifies an existing Ceph RGW user to own the bucket. If the bucket is owned by another user, it will be re-linked.

Once the resources are created, you will get object service endpoint, access key, and secret access key saved in a configuration map and secret.

kubectl apply -f testobc.yaml

Confirm creation of the OBC:

$ kubectl get obc -n default
NAME          AGE
test-bucket   9s

To output object bucket claim resource in yaml, run:

kubectl -n default get obc test-bucket -o yaml

Sample output:

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"objectbucket.io/v1alpha1","kind":"ObjectBucketClaim","metadata":{"annotations":{},"name":"test-bucket","namespace":"default"},"spec":{"additionalConfig":{"maxSize":"2Gi"},"bucketName":"test-bucket","storageClassName":"ceph-bucket"}}
  creationTimestamp: "2025-04-11T21:30:37Z"
  finalizers:
  - objectbucket.io/finalizer
  generation: 2
  labels:
    bucket-provisioner: rook-ceph.ceph.rook.io-bucket
  name: test-bucket
  namespace: default
  resourceVersion: "5983897"
  uid: 5f345606-f43d-4f14-a28c-fd02c29fbe88
spec:
  additionalConfig:
    maxSize: 2Gi
  bucketName: test-bucket
  objectBucketName: obc-default-test-bucket
  storageClassName: ceph-bucket
status:
  phase: Bound

In the status, if successfully provisioned you will see phase: Bound.

4. Obtain s3 endpoint and access credentials

On a successful creation of ObjectBucketClaim (OBC), a Ceph object store user and a bucket are automatically provisioned. A secret containing access and secrets keys for accessing the bucket is generated, and the endpoint details stored in a ConfigMap.

Export namespace and bucket name as env variables.

export NAMESPACE=default
export BUCKET=test-bucket

Viewing Secret contents

List secret with name of the bucket created:

$ kubectl -n $NAMESPACE get secret $BUCKET
NAME           TYPE                DATA   AGE
test-bucket    Opaque              2      2m17s

Output in YAML:

$ kubectl -n $NAMESPACE get secret $BUCKET -o yaml
apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: MkFCTEJOTExIWlNQR1gyMVpSR04=
  AWS_SECRET_ACCESS_KEY: VzlBUXVrNFhhZDN1VVpTYWQ1VDAzRGF4S0U1S3hEVWhPV1Fwam5Vcw==
kind: Secret
metadata:
  creationTimestamp: "2025-04-11T21:30:38Z"
  finalizers:
  - objectbucket.io/finalizer
  labels:
    bucket-provisioner: rook-ceph.ceph.rook.io-bucket
  name: test-bucket
  namespace: default
  ownerReferences:
  - apiVersion: objectbucket.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ObjectBucketClaim
    name: test-bucket
    uid: 5f345606-f43d-4f14-a28c-fd02c29fbe88
  resourceVersion: "5983892"
  uid: 4ce19e9d-e9b3-4af7-ad0a-06055aacca0c
type: Opaque

If you want to output the access key and secret values in plaintext, decode them using base64 -d.

• Print access key ID:

kubectl -n $NAMESPACE get secret $BUCKET -o jsonpath={.data.AWS_ACCESS_KEY_ID} | base64 -d;echo

• Print secret access key:

kubectl -n $NAMESPACE get secret $BUCKET -o jsonpath={.data.AWS_SECRET_ACCESS_KEY} | base64 -d;echo

Saving them as variables:

export AWS_ACCESS_KEY_ID=$(kubectl -n $NAMESPACE get secret $BUCKET_NAME -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 --decode)
export AWS_SECRET_ACCESS_KEY=$(kubectl -n $NAMESPACE get secret $BUCKET_NAME -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 --decode)

View ConfigMap contents

List ConfigMap with the name of bucket.

$ kubectl -n $NAMESPACE get cm $BUCKET
NAME          DATA   AGE
test-bucket   5      23m

Output in yaml:

kubectl -n $NAMESPACE get cm $BUCKET -o yaml

Sample output:

apiVersion: v1
data:
  BUCKET_HOST: rook-ceph-rgw-ceph-objectstore.rook-ceph.svc
  BUCKET_NAME: test-bucket
  BUCKET_PORT: "80"
  BUCKET_REGION: ""
  BUCKET_SUBREGION: ""
kind: ConfigMap
metadata:
  creationTimestamp: "2025-04-11T21:30:38Z"
  finalizers:
  - objectbucket.io/finalizer
  labels:
    bucket-provisioner: rook-ceph.ceph.rook.io-bucket
  name: test-bucket
  namespace: default
  ownerReferences:
  - apiVersion: objectbucket.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ObjectBucketClaim
    name: test-bucket
    uid: 5f345606-f43d-4f14-a28c-fd02c29fbe88
  resourceVersion: "5983893"
  uid: 5508f932-2992-454c-bd58-0a29d292be15

The ConfigMap includes the S3 endpoint host, bucket name, and port details.

BUCKET_HOST: rook-ceph-rgw-ceph-objectstore.rook-ceph.svc
BUCKET_NAME: test-bucket
BUCKET_PORT: "80"

To extract them dynamically and assign to variables, use the following commands:

export AWS_HOST=$(kubectl -n $NAMESPACE get cm $BUCKET_NAME -o jsonpath='{.data.BUCKET_HOST}')
export PORT=$(kubectl -n $NAMESPACE get cm $BUCKET_NAME -o jsonpath='{.data.BUCKET_PORT}')
export BUCKET_NAME=$(kubectl -n $NAMESPACE get cm $BUCKET_NAME -o jsonpath='{.data.BUCKET_NAME}')

5. Using Secret and ConfigMap in Pod

After creation of an ObjectBucketClaim, a Secret and a ConfigMap is the same namespace. These provide the necessary credentials and connection info to access the S3-compatible bucket.

To use the Secret and ConfigMap in a Pod:

1. Mount the Secret and ConfigMap as environment variables or files.
2. Use these values in your app to interact with the bucket.

Example Pod manifest snippet:

• Using envFrom: To expose all keys from the Secret and ConfigMap as environment variables in the pod, preserving their original names.

apiVersion: v1
kind: Pod
metadata:
  name: app-pod
  namespace: dev-user
spec:
  containers:
  - name: mycontainer
    image: redis
    envFrom:
    - configMapRef:
        name: ceph-bucket
    - secretRef:
        name: ceph-bucket

• Using env: when you need to map specific keys to the environment variable names expected by your application.

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  namespace: myapp
spec:
  containers:
  - name: mycontainer
    image: redis
    env:
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: my-bucket
          key: AWS_ACCESS_KEY_ID
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: my-bucket
          key: AWS_SECRET_ACCESS_KEY
    - name: BUCKET_HOST
      valueFrom:
        configMapKeyRef:
          name: my-bucket
          key: BUCKET_HOST
    - name: BUCKET_NAME
      valueFrom:
        configMapKeyRef:
          name: my-bucket
          key: BUCKET_NAME
    - name: BUCKET_PORT
      valueFrom:
        configMapKeyRef:
          name: my-bucket
          key: BUCKET_PORT

6. Mount bucket using s5cmd

Check out our detailed guide on how to mount an S3 bucket using s5cmd for fast and efficient access.

• How To Upload files to Rook Ceph or S3 Bucket using s5cmd