Last time, we published a blog post regarding CEPH RBD device mounting into OpenSVC service with pre_start and post_stop directives.
With awareness of the dream state :) developpers of OpenSVC (Christophe VAROQUI & Arnaud VERON) integrated natively CEPH RBD devices managing into OpenSVC (map, lock and mount)
Prerequisites
Have up and running CEPH cluster with at least 1 pool rbdpartigsanmdev01 (for example) for RBD images.
On our side, CEPH cluster is running on 2 GANDI 512MB/1VCORE nodes with 6 OSDs.
Each OSD is mapped on two GANDI disks : 1 for OSD data (5GB) and one for OSD journal (1GB). Thanks to OpenSVC for GANDI cloud integration :)
We're managing OSD, MON & ADMIN with OpenSVC so 9 services available for this cluster :
- 2 for MON (falcon/raven)
- 6 for OSD (falcon/raven)
- 1 for ADMIN (falcon)
root@falcon:~# ceph osd tree # id weight type name up/down reweight -1 6 root default -2 3 host falcon 0 1 osd.0 up 1 1 1 osd.1 up 1 2 1 osd.2 up 1 -3 3 host raven 3 1 osd.3 up 1 4 1 osd.4 up 1 5 1 osd.5 up 1
On "customer" node (condor), please deploy :
- 1 key /etc/ceph/ceph.client.condor.keyring with below right (rwx on pool rbdpartigsanmdev01)
- 1 /etc/ceph/ceph.conf file with at least monitor nodes (yes I have 2 monitor nodes, not the best practice I know)
Access list for condor key :
ceph auth caps client.condor mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=rbdpartigsanmdev01'
CEPH configuration file for "client" nodes :
root@condor:/etc/ceph# cat ceph.conf [global] mon_initial_members = falcon,raven auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx [mon.falcon] host = falcon mon_addr = x.x.x.x:6789 [mon.raven] host = raven mon_addr = x.x.x.x:6789
Create OpenSVC service to mount RBD disk
Have a look now on flaprdsvc01 service that mounting and locking 2 RBD images with EXT4 filesystem :
root@condor:/opt/opensvc/etc# cat flaprdsvc01.flox-arts.net.env [DEFAULT] app = FLA comment = FLA SVC PRD service mode = hosted cluster_type = failover service_type = PRD nodes = condor.flox-arts.net autostart_node = condor.flox-arts.net [vg#0] client_id@condor.flox-arts.net = condor # ACL key to use (/etc/ceph/ceph.client.condor.keyring) lock = exclusive lock_shared_tag = flaprdsvc01 images = rbdpartigsanmdev01/flaprdsvc01_lun001 rbdpartigsanmdev01/flaprdsvc01_lun002 type = rados [fs#0] type = ext4 mnt = /srv/flaprdsvc01/lun001 dev = /dev/rbd/rbdpartigsanmdev01/flaprdsvc01_lun001 [fs#1] type = ext4 mnt = /srv/flaprdsvc01/lun002 dev = /dev/rbd/rbdpartigsanmdev01/flaprdsvc01_lun002
It's time to start service, I'm so excited :
root@condor:~# /opt/opensvc/etc/flaprdsvc01.flox-arts.net start 16:46:57 INFO FLAPRDSVC01.FLOX-ARTS.NET.VG#0LOCK rbd -n client.condor --keyring /etc/ceph/ceph.client.condor.keyring lock add rbdpartigsanmdev01/flaprdsvc01_lun001 condor.flox-arts.net --shared flaprdsvc01 16:46:57 INFO FLAPRDSVC01.FLOX-ARTS.NET.VG#0LOCK rbd -n client.condor --keyring /etc/ceph/ceph.client.condor.keyring lock add rbdpartigsanmdev01/flaprdsvc01_lun002 condor.flox-arts.net --shared flaprdsvc01 16:46:57 INFO FLAPRDSVC01.FLOX-ARTS.NET.VG#0 rbd -n client.condor --keyring /etc/ceph/ceph.client.condor.keyring map rbdpartigsanmdev01/flaprdsvc01_lun001 16:46:57 INFO FLAPRDSVC01.FLOX-ARTS.NET.VG#0 rbd -n client.condor --keyring /etc/ceph/ceph.client.condor.keyring map rbdpartigsanmdev01/flaprdsvc01_lun002 16:46:57 INFO FLAPRDSVC01.FLOX-ARTS.NET.FS#0 e2fsck -p /dev/rbd/rbdpartigsanmdev01/flaprdsvc01_lun001 16:46:57 INFO FLAPRDSVC01.FLOX-ARTS.NET.FS#0 output: /dev/rbd/rbdpartigsanmdev01/flaprdsvc01_lun001: clean, 13/655360 files, 79665/2621440 blocks 16:46:57 INFO FLAPRDSVC01.FLOX-ARTS.NET.FS#0 mount -t ext4 /dev/rbd/rbdpartigsanmdev01/flaprdsvc01_lun001 /srv/flaprdsvc01/lun001 16:46:58 INFO FLAPRDSVC01.FLOX-ARTS.NET.FS#1 e2fsck -p /dev/rbd/rbdpartigsanmdev01/flaprdsvc01_lun002 16:46:58 INFO FLAPRDSVC01.FLOX-ARTS.NET.FS#1 output:/dev/rbd/rbdpartigsanmdev01/flaprdsvc01_lun002: clean, 13/12824 files, 31289/51200 blocks 16:46:58 INFO FLAPRDSVC01.FLOX-ARTS.NET.FS#1 mount -t ext4 /dev/rbd/rbdpartigsanmdev01/flaprdsvc01_lun002 /srv/flaprdsvc01/lun002
Now let’s check the behavior of service & CEPH devices (map/lock) :
root@condor:/opt/opensvc/lib# df -h | grep flaprdsvc01 /dev/rbd1 9.8G 23M 9.2G 1% /srv/flaprdsvc01/lun001 /dev/rbd2 45M 25M 16M 62% /srv/flaprdsvc01/lun00 root@condor:/opt/opensvc/lib# rbd showmapped id pool image snap device 1 rbdpartigsanmdev01 flaprdsvc01_lun001 - /dev/rbd1 2 rbdpartigsanmdev01 flaprdsvc01_lun002 - /dev/rbd2 root@falcon:~# rbd lock list rbdpartigsanmdev01/flaprdsvc01_lun001 There are 1 shared lock on this image. Lock tag: flaprdsvc01 Locker ID Address client.61800 condor.flox-arts.net 217.70.189.101:0/1016130 root@falcon:~# rbd lock list rbdpartigsanmdev01/flaprdsvc01_lun002 There are 1 shared lock on this image. Lock tag: flaprdsvc01 Locker ID Address client.61562 condor.flox-arts.net 217.70.189.101:0/1014196
Over... Next topic : CEPH snapshots & clones management in OpenSVC service ;)
- OpenSVC services http://www.opensvc.com
- CEPH software-defined storage ceph.com
- Gandi VPS Cloud Hosting http://www.gandi.net/hebergement