HowTo: CMS XRootD configuration

XRootD CMS

Requirements

storage-disk.xml file

• Every CMS site has his own storage-disk.xml file.
• This file should be the same for all servers running a CMS service.
• Below, an example for the PIC site storage-disk.xml file:

<storage-mapping>
  <lfn-to-pfn protocol="direct"
    path-match="/+store/test/xrootd/T1_ES_PIC/store/(.*)"
    result="/pnfs/pic.es/data/cms/disk/store/$1"/>
  <lfn-to-pfn protocol="direct"
    path-match=".*/LoadTest07_Debug_T1_ES_PIC_MSS/T1_ES_PIC_Disk/(.*)"
    result="/pnfs/pic.es/data/cms/disk/store/PhEDEx_LoadTest07/LoadTest07_Debug_T1_ES_PIC_MSS/T1_ES_PIC_Disk/$1"/>
  <lfn-to-pfn protocol="direct"
    path-match=".*/LoadTest07_ES_PIC_Disk_(.*)_.*_.*"
    result="/pnfs/pic.es/data/cms/disk/store/PhEDEx_LoadTest07/Source_Disk/LoadTest07_PIC_$1"/>
  <lfn-to-pfn protocol="direct"
    path-match="/+store/(.*)"
    result="/pnfs/pic.es/data/cms/disk/store/$1"/>

  <lfn-to-pfn protocol="xrootdlocal" chain="direct" 
    path-match=".*/HCtest/GenericTTbar/(.*)" 
    result="root://srm-pps.pic.es:1094/pnfs/pic.es/data/cms/store/mc/HCtest/GenericTTbar/$1"/>
  <lfn-to-pfn protocol="xrootdlocal" chain="direct" 
    path-match="(.*)" result="root://xrootd.pic.es:1094$1"/>

  <lfn-to-pfn protocol="xrootd" path-match="/+store/(.*)" result="root://xrootd-cms.infn.it//store/$1"/>

  <lfn-to-pfn protocol="srmv2" chain="direct"
    path-match="(.*)" result="srm://srmcms.pic.es:8443/srm/managerv2?SFN=$1"/>
  <lfn-to-pfn protocol="dcap" chain="direct"
    path-match="(.*)" result="dcap://dcap.pic.es$1"/>
  <lfn-to-pfn protocol="dccp" chain="direct"
    path-match="(.*)" result="dcap://dcap.pic.es$1"/>
  <lfn-to-pfn protocol="gsidcap" chain="direct"
    path-match="(.*)" result="gsidcap://gsidcap.pic.es:22128$1"/>
  <lfn-to-pfn protocol="root" chain="direct"
    path-match="(.*)" result="root://xrootd-cmst1-door.pic.es:1094$1"/>

  <pfn-to-lfn protocol="direct"
    path-match="/+pnfs/pic\.es/data/cms/disk/store/(.*)" result="/store/$1"/>
  <pfn-to-lfn protocol="srmv2" chain="direct"
    path-match=".*\?SFN=(.*)" result="$1"/>
  <pfn-to-lfn protocol="dcap" chain="direct"
    path-match="dcap://dcap\.pic\.es(.*)" result="$1"/>
  <pfn-to-lfn protocol="dccp" chain="direct"
    path-match="dcap://dcap\.pic\.es(.*)" result="$1"/>
  <pfn-to-lfn protocol="gsidcap" chain="direct"
    path-match="gsidcap://gsidcap\.pic\.es:22128(.*)" result="$1"/>
</storage-mapping>

XRootD Local Redirector

• xrootd-cmst1.pic.es is an example of local CMS XRootD redirector at PIC.
• Relevant services are xrootd and cmsd which should be up and running:

[root@xrootd01-cmst1 ~]# service xrootd status
[default] xrootd (pid  26682) is running...

[root@xrootd01-cmst1 ~]# service cmsd status
[default] cmsd (pid  26583) is running...

• XRootD configuration files is locate at: /etc/xrootd/xrootd-clustered.cfg and should be defined as follows:

[root@xrootd01-cmst1 ~]# cat /etc/xrootd/xrootd-clustered.cfg | grep -v '#' | grep . 
xrd.port 1095
all.role server
all.sitename T1_ES_PIC
all.manager any xrootd-cms.infn.it+ 1213
xrootd.redirect xrootd-cmst1-door.pic.es:1094 /
all.export / nostage
cms.allow host *
xrootd.trace emsg login stall redirect
ofs.trace none
xrd.trace conn
cms.trace all
oss.namelib /usr/lib64/libXrdCmsTfc.so file:/etc/xrootd/storage-disk.xml?protocol=direct
xrootd.seclib /usr/lib64/libXrdSec.so
xrootd.fslib /usr/lib64/libXrdOfs.so
all.adminpath /var/run/xrootd
all.pidpath /var/run/xrootd
cms.delay startup 10
cms.fxhold 60s
xrd.report xrootd.t2.ucsd.edu:9931 every 60s all sync
xrootd.monitor all fstat 60s lfn ops ssq xfr 5 ident 5m dest fstat info user CMS-AAA-EU-COLLECTOR.cern.ch:9330

• Relevant packages that need to be installed are the following:

[root@xrootd01-cmst1 ~]# rpm -qa | grep xrootd | sort
cms-xrootd-dcache-1.2-7.osg.el6.noarch
xrootd-4.5.0-1.el6.x86_64
xrootd-client-4.5.0-1.el6.x86_64
xrootd-client-devel-4.5.0-1.el6.x86_64
xrootd-client-libs-4.5.0-1.el6.x86_64
xrootd-cmstfc-1.5.1-6.osg.el6.x86_64
xrootd-compat-3.3.6-1.el6.x86_64
xrootd-compat-client-libs-3.3.6-1.el6.x86_64
xrootd-compat-libs-3.3.6-1.el6.x86_64
xrootd-compat-server-libs-3.3.6-1.el6.x86_64
xrootd-devel-4.5.0-1.el6.x86_64
xrootd-libs-4.5.0-1.el6.x86_64
xrootd-selinux-4.5.0-1.el6.noarch
xrootd-server-4.5.0-1.el6.x86_64
xrootd-server-devel-4.5.0-1.el6.x86_64
xrootd-server-libs-4.5.0-1.el6.x86_64

• The above packages can be installed from the official XRootD repositories:

[root@xrootd02-cmst1 yum.repos.d]# cat /etc/yum.repos.d/xrootd-stable.repo 
[xrootd-stable]
name=XrootD official repository
baseurl=http://xrootd.org/binaries/stable/slc/6/$basearch http://xrootd.cern.ch/sw/repos/stable/slc/6/$basearch
enabled=1
gpgcheck=0
enablegroups=1
metadata_expire=43200

• PNFS mount is a required mountpoint. Read-only mode should be enough, and the sync option is recommended:

[root@xrootd01-cmst1 ~]# mount | grep pnfs
nfsv41.pic.es:/pnfs/pic.es/data/cms on /pnfs/pic.es/data/cms type nfs4 

(ro,sync,noatime,nodiratime,minorversion=1,nolock,noac,intr,bg,hard,addr=2001:67c:1148:201::11,clientaddr=2001:67c:1148:201::17)

dCache door configuration

• xrootd-cmst1-door.pic.es (usually door06.pic.es) is the alias for the CMS XRootD door at PIC.
• dCache CMS XRootD layout file example, where:

• xrootd.loginbroker.tags=srm to disable glue publishing
• xrootd.mover.queue=xrootd-wan to dedicated a specific mover queue (we want split WAN vs LAN movers)

############################################
# Domain: xrootd-cmst1-door06Domain
[xrootd-cmst1-${host.name}Domain]
dcache.java.memory.heap=2048m
xrootd.loginbroker.tags=srm
xrootd.mover.queue=xrootd-wan
# Cell: xrootd
[xrootd-cmst1-${host.name}Domain/xrootd]
xrootd.cms.tfc.path=/etc/dcache/storage-disk.xml
xrootd.cms.tfc.protocol=direct
xrootd.net.port=1094
xrootd.plugins=gplazma:gsi,authz:cms-tfc
## End xrootd-cmst1-door06Domain
############################################

• Relevant packages that need to be installed are the following:

[root@door06 ~]# rpm -qa | grep ^dcache- | sort
dcache-2.16.38-1.noarch
dcache-srmclient-3.0.9-1.noarch

• TFC plugin must be installed.

• Can be downloaded from https://www.dcache.org/downloads/xrootd4j/index.shtml (we use the .tar.gz file)

[root@door06 ~]# ls -ltrha /usr/local/share/dcache/plugins/
total 12K
drwxr-xr-x. 3 root root 4.0K Jun 20 10:09 ..
drwxr-xr-x. 3 root root 4.0K Jun 20 10:16 .
drwxr-xr-x. 2 root root 4.0K Jun 20 10:16 xrootd4j-cms-plugin-1.3.7

dCache pool configuration

• Pool configuration is needed for the XRootD monitoring service.
• An example of a dCache CMS pool layout file for dc040.pic.es is shown below:

############################################
# Domain: dc040_1Domain
[${host.name}_1Domain]
dcache.java.options.extra=-Djava.net.preferIPv6Addresses=true -Dcom.sun.management.jmxremote.port=7001 -Dcom.sun.management.jmxremote.ssl=false -
Dcom.sun.management.jmxremote.authenticate=false
dcache.java.memory.heap=6000m
pool.mover.xrootd.port.max=${dcache.net.wan.port.max}
pool.mover.xrootd.port.min=${dcache.net.wan.port.min}
pool.plugins.meta=org.dcache.pool.repository.meta.db.BerkeleyDBMetaDataRepository
pool.queues=ftp,dcap,nfs4,xrootd,xrootd-wan,webdav
# dc040_1Domain - xrootd configuration for cms pools
pool.mover.xrootd.plugins=edu.uchicago.monitor
xrootd.monitor.detailed=cms-aaa-eu-collector.cern.ch:9330:60
xrootd.monitor.site=T1_ES_PIC
xrootd.monitor.summary=xrootd.t2.ucsd.edu:9931:60
xrootd.monitor.vo=CMS
# Cell: pool
[${host.name}_1Domain/pool]
pool.name=${host.name}_1
pool.path=/dcpool/vpool1
pool.wait-for-files=${pool.path}/data
## End pool_1Domain
############################################

• Relevant packages that need to be installed are the following:

[root@dc040 ~]# rpm -qa | grep ^dcache- | sort
dcache-2.16.38-1.noarch
dcache-plugin-xrootd-monitor-7.0.0-0.noarch
dcache-srmclient-3.0.9-1.noarch