beyondwatts

Behind The Scenes
Other Posts
Search

Migrating mariadb-galera storage to ceph



I have mariadb-galera installed with the bitnami helm chart. After making sure there is a backup, the first step is to delete the helm release. This removes the services and pods but leaves the PVCs behind.

> helm delete mariadb-galera -n mariadb-galera

PVCs:

> k get pvc -n mariadb-galera
NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          VOLUMEATTRIBUTESCLASS   AGE
data-mariadb-galera-0   Bound    pvc-70577ec4-890e-446f-9bef-a30cbe5b0021   15Gi       RWO            synology-csi-retain   <unset>                 148d
data-mariadb-galera-1   Bound    pvc-23ec4e6c-f478-43ce-b33f-cc136fa6144c   15Gi       RWO            synology-csi-retain   <unset>                 147d
data-mariadb-galera-2   Bound    pvc-7326d28a-66f3-4625-af01-94becfb02e51   15Gi       RWO            synology-csi-retain   <unset>                 147d

Delete the first PVC - we’ll keep the other two for now as extra backups if needed!

> k delete pvc/data-mariadb-galera-0 -n mariadb-galera

Change the storage class in the helm values:

storageClass: "ceph-block"

And install a single replica mariadb instance - I have a special values file, called mariadb-galera-values-bootstrap.yml that will bootstrap the cluster

> helm install mariadb-galera -n mariadb-galera bitnami/mariadb-galera --version 13.0.0 --values mariadb-galera-values-bootstrap.yml

This will recreate the data-mariadb-galera-0 PVC although it won’t have any data yet! Scale the cluster down:

> k scale --replicas 0 statefulset/mariadb-galera -n mariadb-galera

Check the PVCs:

> k get pvc -n mariadb-galera
NAME                             STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          VOLUMEATTRIBUTESCLASS   AGE
data-mariadb-galera-0            Bound     pvc-09698cc7-1936-4a99-b16a-005421bc9d2d   15Gi       RWO   ceph-block            <unset>                 14m
data-mariadb-galera-1            Bound     pvc-23ec4e6c-f478-43ce-b33f-cc136fa6144c   15Gi       RWO   synology-csi-retain   <unset>                 147d
data-mariadb-galera-2            Bound     pvc-7326d28a-66f3-4625-af01-94becfb02e51   15Gi       RWO   synology-csi-retain   <unset>                 147d

Mount the old PV so we can copy data across

Delete the claim:

> k patch pv/pvc-70577ec4-890e-446f-9bef-a30cbe5b0021 -p '{"spec":{"claimRef": null}}'
persistentvolume/pvc-70577ec4-890e-446f-9bef-a30cbe5b0021 patched

Mount the old PV with a new claim:

  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: data-mariadb-galera-0-synology
    namespace: mariadb-galera
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 15Gi
    storageClassName: synology-csi-retain
    volumeMode: Filesystem
    volumeName: pvc-70577ec4-890e-446f-9bef-a30cbe5b0021

It’s there:

NAME                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          VOLUMEATTRIBUTESCLASS   AGE
data-mariadb-galera-0            Bound    pvc-09698cc7-1936-4a99-b16a-005421bc9d2d   15Gi       RWO            ceph-block            <unset>                 21m
data-mariadb-galera-0-synology   Bound    pvc-70577ec4-890e-446f-9bef-a30cbe5b0021   15Gi       RWO            synology-csi-retain   <unset>                 16m
data-mariadb-galera-1            Bound    pvc-23ec4e6c-f478-43ce-b33f-cc136fa6144c   15Gi       RWO            synology-csi-retain   <unset>                 147d
data-mariadb-galera-2            Bound    pvc-7326d28a-66f3-4625-af01-94becfb02e51   15Gi       RWO            synology-csi-retain   <unset>                 147d

Copy data to the new Ceph PVC with a Job:

apiVersion: batch/v1
kind: Job
metadata:
  namespace: mariadb-galera  # namespace where the pvc's exist
  name: volume-migration
spec:
  completions: 1
  parallelism: 1
  backoffLimit: 3
  template:
    metadata:
      name: volume-migration
      labels:
        name: volume-migration
    spec:
      restartPolicy: Never
      containers:
        - name: volume-migration
          image: ubuntu:xenial
          tty: true
          command: [ "/bin/sh" ]
          args: [ "-c", "cp -r -v /mnt/old/. /mnt/new" ]
          volumeMounts:
            - name: old-vol
              mountPath: /mnt/old
            - name: new-vol
              mountPath: /mnt/new
      volumes:
        - name: old-vol
          persistentVolumeClaim:
            claimName: data-mariadb-galera-0-synology # change to data source pvc
        - name: new-vol
          persistentVolumeClaim:
            claimName: data-mariadb-galera-0 # change to data target pvc

Monitor progress:

> k logs pod/volume-migration-l7nz5 -n mariadb-galera -f
'/mnt/old/./.bootstrap/done' -> '/mnt/new/./.bootstrap/done'
'/mnt/old/./data/GRA_6_362448_v2.log' -> '/mnt/new/./data/GRA_6_362448_v2.log'
'/mnt/old/./data/GRA_8_6513862_v2.log' -> '/mnt/new/./data/GRA_8_6513862_v2.log'
'/mnt/old/./data/ib_logfile0' -> '/mnt/new/./data/ib_logfile0'
'/mnt/old/./data/GRA_7_2406475_v2.log' -> '/mnt/new/./data/GRA_7_2406475_v2.log'
...

Once the job has completed, tidy up:

> k delete job/volume-migration -n mariadb-galera

Start up MariaDB Galera:

> k scale --replicas 1 statefulset/mariadb-galera -n mariadb-galera

At this point we have a single replica mariadb galera cluster up and running. We can have a good poke around and check everything has loaded and start up essential services.

The final step is to delete the old PVCs and then increase the number of replicas: