HowTo: CMS XRootD configuration

XRootD CMS

Requirements

XRootD Local Redirector

• xrootd-cmst1.pic.es is an example of local CMS XRootD redirector at PIC.
• Relevant services are xrootd and cmsd which should be up and running:

[root@xrootd01-cmst1 ~]# service xrootd status
[default] xrootd (pid  26682) is running...

[root@xrootd01-cmst1 ~]# service cmsd status
[default] cmsd (pid  26583) is running...

• XRootD configuration files is locate at: /etc/xrootd/xrootd-clustered.cfg and should be defined as follows:

[root@xrootd01-cmst1 ~]# cat /etc/xrootd/xrootd-clustered.cfg | grep -v '#' | grep . 
xrd.port 1095
all.role server
all.sitename T1_ES_PIC
all.manager any xrootd-cms.infn.it+ 1213
xrootd.redirect xrootd-cmst1-door.pic.es:1094 /
all.export / nostage
cms.allow host *
xrootd.trace emsg login stall redirect
ofs.trace none
xrd.trace conn
cms.trace all
oss.namelib /usr/lib64/libXrdCmsTfc.so file:/etc/xrootd/storage-disk.xml?protocol=direct
xrootd.seclib /usr/lib64/libXrdSec.so
xrootd.fslib /usr/lib64/libXrdOfs.so
all.adminpath /var/run/xrootd
all.pidpath /var/run/xrootd
cms.delay startup 10
cms.fxhold 60s
xrd.report xrootd.t2.ucsd.edu:9931 every 60s all sync
xrootd.monitor all fstat 60s lfn ops ssq xfr 5 ident 5m dest fstat info user CMS-AAA-EU-COLLECTOR.cern.ch:9330

• Relevant packages that need to be installed are the following:

[root@xrootd01-cmst1 ~]# rpm -qa | grep xrootd | sort
cms-xrootd-dcache-1.2-7.osg.el6.noarch
xrootd-4.5.0-1.el6.x86_64
xrootd-client-4.5.0-1.el6.x86_64
xrootd-client-devel-4.5.0-1.el6.x86_64
xrootd-client-libs-4.5.0-1.el6.x86_64
xrootd-cmstfc-1.5.1-6.osg.el6.x86_64
xrootd-compat-3.3.6-1.el6.x86_64
xrootd-compat-client-libs-3.3.6-1.el6.x86_64
xrootd-compat-libs-3.3.6-1.el6.x86_64
xrootd-compat-server-libs-3.3.6-1.el6.x86_64
xrootd-devel-4.5.0-1.el6.x86_64
xrootd-libs-4.5.0-1.el6.x86_64
xrootd-selinux-4.5.0-1.el6.noarch
xrootd-server-4.5.0-1.el6.x86_64
xrootd-server-devel-4.5.0-1.el6.x86_64
xrootd-server-libs-4.5.0-1.el6.x86_64

• The above packages can be installed from the official XRootD repositories:

[root@xrootd02-cmst1 yum.repos.d]# cat /etc/yum.repos.d/xrootd-stable.repo 
[xrootd-stable]
name=XrootD official repository
baseurl=http://xrootd.org/binaries/stable/slc/6/$basearch http://xrootd.cern.ch/sw/repos/stable/slc/6/$basearch
enabled=1
gpgcheck=0
enablegroups=1
metadata_expire=43200

• PNFS mount is a required mountpoint. Read-only mode should be enough, and the sync option is recommended:

[root@xrootd01-cmst1 ~]# mount | grep pnfs
nfsv41.pic.es:/pnfs/pic.es/data/cms on /pnfs/pic.es/data/cms type nfs4 

(ro,sync,noatime,nodiratime,minorversion=1,nolock,noac,intr,bg,hard,addr=2001:67c:1148:201::11,clientaddr=2001:67c:1148:201::17)

dCache door configuration

• xrootd-cmst1-door.pic.es (usually door06.pic.es) is the alias for the CMS XRootD door at PIC.
• dCache CMS XRootD layout file example, where:

• xrootd.loginbroker.tags=srm to disable glue publishing
• xrootd.mover.queue=xrootd-wan to dedicated a specific mover queue (we want split WAN vs LAN movers)

############################################
# Domain: xrootd-cmst1-door06Domain
[xrootd-cmst1-${host.name}Domain]
dcache.java.memory.heap=2048m
xrootd.loginbroker.tags=srm
xrootd.mover.queue=xrootd-wan
# Cell: xrootd
[xrootd-cmst1-${host.name}Domain/xrootd]
xrootd.cms.tfc.path=/etc/dcache/storage-disk.xml
xrootd.cms.tfc.protocol=direct
xrootd.net.port=1094
xrootd.plugins=gplazma:gsi,authz:cms-tfc
## End xrootd-cmst1-door06Domain
############################################

• Relevant packages that need to be installed are the following:

[root@door06 ~]# rpm -qa | grep ^dcache- | sort
dcache-2.16.38-1.noarch
dcache-srmclient-3.0.9-1.noarch

• TFC plugin must be installed.

• Can be downloaded from https://www.dcache.org/downloads/xrootd4j/index.shtml (we use the .tar.gz file)

[root@door06 ~]# ls -ltrha /usr/local/share/dcache/plugins/
total 12K
drwxr-xr-x. 3 root root 4.0K Jun 20 10:09 ..
drwxr-xr-x. 3 root root 4.0K Jun 20 10:16 .
drwxr-xr-x. 2 root root 4.0K Jun 20 10:16 xrootd4j-cms-plugin-1.3.7

dCache pool configuration

• Pool configuration is needed for the XRootD monitoring service.
• An example of a dCache CMS pool layout file for dc040.pic.es is shown below:

############################################
# Domain: dc040_1Domain
[${host.name}_1Domain]
dcache.java.options.extra=-Djava.net.preferIPv6Addresses=true -Dcom.sun.management.jmxremote.port=7001 -Dcom.sun.management.jmxremote.ssl=false -
Dcom.sun.management.jmxremote.authenticate=false
dcache.java.memory.heap=6000m
pool.mover.xrootd.port.max=${dcache.net.wan.port.max}
pool.mover.xrootd.port.min=${dcache.net.wan.port.min}
pool.plugins.meta=org.dcache.pool.repository.meta.db.BerkeleyDBMetaDataRepository
pool.queues=ftp,dcap,nfs4,xrootd,xrootd-wan,webdav
# dc040_1Domain - xrootd configuration for cms pools
pool.mover.xrootd.plugins=edu.uchicago.monitor
xrootd.monitor.detailed=cms-aaa-eu-collector.cern.ch:9330:60
xrootd.monitor.site=T1_ES_PIC
xrootd.monitor.summary=xrootd.t2.ucsd.edu:9931:60
xrootd.monitor.vo=CMS
# Cell: pool
[${host.name}_1Domain/pool]
pool.name=${host.name}_1
pool.path=/dcpool/vpool1
pool.wait-for-files=${pool.path}/data
## End pool_1Domain
############################################

• Relevant packages that need to be installed are the following:

[root@dc040 ~]# rpm -qa | grep ^dcache- | sort
dcache-2.16.38-1.noarch
dcache-plugin-xrootd-monitor-7.0.0-0.noarch
dcache-srmclient-3.0.9-1.noarch