Twitter

  • RT @BF_TechServices: #International : Pourquoi choisir le #Canada 🇨🇦 ? Si vous avez les yeux rivés sur l’#AmériqueDuNord comme nouvelle cib…

CEPH RBD devices management with OpenSVC service

jeudi 26 mars 2015 | Blog, Veille | Aucun commentaire

Last time, we published a blog post regarding CEPH RBD device mounting into OpenSVC service with pre_start and post_stop directives.

ceph opensvc

With awareness of the dream state :) developpers of OpenSVC (Christophe VAROQUI & Arnaud VERON) integrated natively CEPH RBD devices managing into OpenSVC (map, lock and mount)

Prerequisites

Have up and running CEPH cluster with at least 1 pool rbdpartigsanmdev01 (for example) for RBD images.

On our side, CEPH cluster is running on 2 GANDI 512MB/1VCORE nodes with 6 OSDs.

Each OSD is mapped on two GANDI disks : 1 for OSD data (5GB) and one for OSD journal (1GB). Thanks to OpenSVC for GANDI cloud integration :)

ceph

We're managing OSD, MON & ADMIN with OpenSVC so 9 services available for this cluster :

  • 2 for MON (falcon/raven)
  • 6 for OSD (falcon/raven)
  • 1 for ADMIN (falcon)
root@falcon:~# ceph osd tree
# id	weight	type name	up/down	reweight
-1	6	root default
-2	3		host falcon
0	1			osd.0	up	1	
1	1			osd.1	up	1	
2	1			osd.2	up	1	
-3	3		host raven
3	1			osd.3	up	1	
4	1			osd.4	up	1	
5	1			osd.5	up	1

On "customer" node (condor), please deploy :

  • 1 key /etc/ceph/ceph.client.condor.keyring with below right (rwx on pool rbdpartigsanmdev01)
  • 1 /etc/ceph/ceph.conf file with at least monitor nodes (yes I have 2 monitor nodes, not the best practice I know)

Access list for condor key :

ceph auth caps client.condor mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=rbdpartigsanmdev01'

CEPH configuration file for "client" nodes :

root@condor:/etc/ceph# cat ceph.conf 
[global]
mon_initial_members = falcon,raven
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

[mon.falcon]
host = falcon
mon_addr = x.x.x.x:6789

[mon.raven]
host = raven
mon_addr = x.x.x.x:6789

Create OpenSVC service to mount RBD disk

Have a look now on flaprdsvc01 service that mounting and locking 2 RBD images with EXT4 filesystem :

root@condor:/opt/opensvc/etc# cat flaprdsvc01.flox-arts.net.env
[DEFAULT]
app = FLA
comment = FLA SVC PRD service
mode = hosted
cluster_type = failover
service_type = PRD
nodes = condor.flox-arts.net
autostart_node = condor.flox-arts.net

[vg#0]
client_id@condor.flox-arts.net = condor # ACL key to use (/etc/ceph/ceph.client.condor.keyring)
lock = exclusive
lock_shared_tag = flaprdsvc01
images = rbdpartigsanmdev01/flaprdsvc01_lun001 rbdpartigsanmdev01/flaprdsvc01_lun002
type = rados

[fs#0]
type = ext4
mnt = /srv/flaprdsvc01/lun001
dev = /dev/rbd/rbdpartigsanmdev01/flaprdsvc01_lun001

[fs#1]
type = ext4
mnt = /srv/flaprdsvc01/lun002
dev = /dev/rbd/rbdpartigsanmdev01/flaprdsvc01_lun002

It's time to start service, I'm so excited :

root@condor:~# /opt/opensvc/etc/flaprdsvc01.flox-arts.net start
16:46:57 INFO    FLAPRDSVC01.FLOX-ARTS.NET.VG#0LOCK rbd -n client.condor --keyring /etc/ceph/ceph.client.condor.keyring lock add rbdpartigsanmdev01/flaprdsvc01_lun001 condor.flox-arts.net --shared flaprdsvc01
16:46:57 INFO    FLAPRDSVC01.FLOX-ARTS.NET.VG#0LOCK rbd -n client.condor --keyring /etc/ceph/ceph.client.condor.keyring lock add rbdpartigsanmdev01/flaprdsvc01_lun002 condor.flox-arts.net --shared flaprdsvc01
16:46:57 INFO    FLAPRDSVC01.FLOX-ARTS.NET.VG#0     rbd -n client.condor --keyring /etc/ceph/ceph.client.condor.keyring map rbdpartigsanmdev01/flaprdsvc01_lun001
16:46:57 INFO    FLAPRDSVC01.FLOX-ARTS.NET.VG#0     rbd -n client.condor --keyring /etc/ceph/ceph.client.condor.keyring map rbdpartigsanmdev01/flaprdsvc01_lun002
16:46:57 INFO    FLAPRDSVC01.FLOX-ARTS.NET.FS#0     e2fsck -p /dev/rbd/rbdpartigsanmdev01/flaprdsvc01_lun001
16:46:57 INFO    FLAPRDSVC01.FLOX-ARTS.NET.FS#0     output: /dev/rbd/rbdpartigsanmdev01/flaprdsvc01_lun001: clean, 13/655360 files, 79665/2621440 blocks
16:46:57 INFO    FLAPRDSVC01.FLOX-ARTS.NET.FS#0     mount -t ext4 /dev/rbd/rbdpartigsanmdev01/flaprdsvc01_lun001 /srv/flaprdsvc01/lun001
16:46:58 INFO    FLAPRDSVC01.FLOX-ARTS.NET.FS#1     e2fsck -p /dev/rbd/rbdpartigsanmdev01/flaprdsvc01_lun002
16:46:58 INFO    FLAPRDSVC01.FLOX-ARTS.NET.FS#1     output:/dev/rbd/rbdpartigsanmdev01/flaprdsvc01_lun002: clean, 13/12824 files, 31289/51200 blocks
16:46:58 INFO    FLAPRDSVC01.FLOX-ARTS.NET.FS#1     mount -t ext4 /dev/rbd/rbdpartigsanmdev01/flaprdsvc01_lun002 /srv/flaprdsvc01/lun002

Now let’s check the behavior of service & CEPH devices (map/lock) :

root@condor:/opt/opensvc/lib# df -h | grep flaprdsvc01
/dev/rbd1       9.8G   23M  9.2G   1% /srv/flaprdsvc01/lun001
/dev/rbd2        45M   25M   16M  62% /srv/flaprdsvc01/lun00

root@condor:/opt/opensvc/lib# rbd showmapped
id pool               image              snap device    
1  rbdpartigsanmdev01 flaprdsvc01_lun001 -    /dev/rbd1 
2  rbdpartigsanmdev01 flaprdsvc01_lun002 -    /dev/rbd2

root@falcon:~# rbd lock list rbdpartigsanmdev01/flaprdsvc01_lun001              
There are 1 shared lock on this image.
Lock tag: flaprdsvc01
Locker       ID                   Address                  
client.61800 condor.flox-arts.net 217.70.189.101:0/1016130

root@falcon:~# rbd lock list rbdpartigsanmdev01/flaprdsvc01_lun002 
There are 1 shared lock on this image.
Lock tag: flaprdsvc01
Locker       ID                   Address                  
client.61562 condor.flox-arts.net 217.70.189.101:0/1014196

Over... Next topic : CEPH snapshots & clones management in OpenSVC service ;)

Partager


Commenter