CephFS: a beginner’s guide

Ceph is the open source Software Defined Storage (SDS) king. By default Ceph stores objects, but did you know that you can use it to store files in a filesystem too? Or even replace Hadoop’s HDFS? Let’s take a look at CephFS.

I don’t have a Ceph cluster

Any sufficiently advanced technology is indistinguishable from magic.Arthur C. Clarke

Ceph is pretty complicated as it is, I highly suggest you to follow this guide with a Ceph test cluster beneath your keyboard.

• How to install Ceph with ceph-ansible

Ceph pools and CephFS

CephFS is a way to store files within a POSIX-compliant filesystem. CephFS lives on top of a RADOS cluster and can be used to support legacy applications. Another common use for CephFS is to replace Hadoop’s HDFS.

To get started you will need a Ceph Metadata Server (Ceph MDS). This server is responsible for maintaining metadata for CephFS files and enable common operations (ls, find etc.) without weighing down the whole cluster. There should be at least one to get started with CephFS, starting with Ceph Luminous there can be more than one Ceph MDS.

Each CephFS instance will need two pools dedicated to it, much like RBD. You will need two pools instead of one because one will keep the data and the other will keep metadata. Know that if you used ceph-ansible and installed an MDS server you may already have an instance of CephFS running.

Now that you know what CephFS is and what it needs to work: let’s get started. You will need to have a Ceph keyring on the host you’re using. The fastest way to get started is to ssh into a monitor node.

Initializing CephFS

The first thing in order to create a CephFS is to create the two pools:

$ ceph osd pool create cephfs_data <pg_num>
$ ceph osd pool create cephfs_metadata <pg_num>

Once you have them you can initialize the filesystem:

$ ceph fs new cephfs cephfs_metadata cephfs_data

CephFS status

You can verify your CephFS instance by doing:

$ ceph fs ls
name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]

To get more details you can use:

$ ceph fs status <cephfs>
cephfs - 1 clients
======
+------+--------+-----------+---------------+-------+-------+
| Rank | State  |    MDS    |    Activity   |  dns  |  inos |
+------+--------+-----------+---------------+-------+-------+
|  0   | active |    mon1   | Reqs:    0 /s |   10  |   12  |
+------+--------+-----------+---------------+-------+-------+
+-----------------+----------+-------+-------+
|       Pool      |   type   |  used | avail |
+-----------------+----------+-------+-------+
| cephfs_metadata | metadata | 2643  | 93.9G |
|   cephfs_data   |   data   |    0  | 93.9G |
+-----------------+----------+-------+-------+

+-------------+
| Standby MDS |
+-------------+
+-------------+
MDS version: ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)

Mounting CephFS (Kernel driver)

The kernel driver is a convenient way to mount CephFS. In order to use it you must have installed it, refer to your distribution for more information. Once you have it installed simply do:

$ mount -t ceph -o name=<name>,secret=<secret> <mon>:/ <target>

The name parameter is a Ceph User. mon is the monitor‘s hostname or IP address. The <target> is the local directory where you want to mount the filesystem to.

The secret parameter depends on the user, and it is quite insecure (it will be stored in your shell’s history and/or fstab). To avoid this you can simply use the secretfile parameter and specify a file containing the secret. The secret is not the keyring. In order to extract the secret from the keyring you can do:

$ ceph-authtool -p <keyring>

Although you can use the admin user and its secret, it is highly suggested to create a username/secret dedicated for each clientand to assign it the least privilege.

Mounting permanently (Kernel driver)

Using fstab, directly from the official docs:

{ipaddress}:{port}:/ {mount}/{mountpoint} {filesystem-name}     [name=username,secret=secretkey|secretfile=/path/to/secretfile],[{mount.options}]

Mounting CephFS (FUSE)

The FUSE is another way to mount a CephFS. Where you can’t use the kernel driver you can use this method. You will need to have the ceph.conf and ceph.keyring files containing the cluster configuration and required secrets in /etc/ceph on the client node. Once you have them you can do:

$ sudo ceph-fuse -m <monitor>:<port> <target>

The monitor is monitor’s hostname or IP address, the default port is 6789.

Mounting permanently (FUSE)

Using fstab, directly from the official docs:

none    {mountpoint}  fuse.ceph ceph.id={user-ID}[,ceph.conf={path/to/conf.conf}],_netdev,defaults  0 0

• Author
• Recent Posts

mark

The IT guy with a slight look of boredom in his eyes. Freelancer. Current interests: Kubernetes, Tensorflow, shiny new things.

Latest posts by mark (see all)

• 2020 A year in review for Marksei.com - 30 December 2020
• Red Hat pulls the kill switch on CentOS - 16 December 2020
• OpenZFS 2.0 released: unified ZFS for Linux and BSD - 9 December 2020