HowTo: Permanent Migration to dCache Read Only Pools - Revision history

Mcaubet: /* CMS Example */

2017-05-26T11:11:26Z

CMS Example

Mcaubet: /* ATLAS Example */

2017-05-26T11:11:07Z

ATLAS Example

Mcaubet: /* ATLAS Example */

2017-05-26T10:41:45Z

ATLAS Example

Mcaubet: /* CMS Example */

2017-05-26T10:41:33Z

CMS Example

Mcaubet: /* ATLAS Example */

2017-05-26T10:40:30Z

ATLAS Example

Mcaubet: /* CMS Example */

2017-05-26T10:23:08Z

CMS Example

Mcaubet: /* CMS Example */

2017-05-25T15:23:59Z

CMS Example

Mcaubet: /* Source pools */

2017-05-25T14:22:29Z

Source pools

Mcaubet: /* Source pools */

2017-05-25T13:36:48Z

Source pools

Mcaubet: /* Source pools */

2017-05-25T11:26:07Z

Source pools

@@ Line 34: / Line 34: @@
 :::: You can setup higher values if you expect a very good performance, otherwise migration can become a bottleneck.
-== CMS Example ==
+== CMS T1 Example ==
 * Considering a CMS configuration like the following:
   # '''pgroup-cms'''

@@ Line 74: / Line 74: @@
     -exclude=dc100*,dc101* -id=caching_recently_accessed_files -verify -size=1070000.. -- pgroup-cms-ro
-== ATLAS Example ==
+== ATLAS T1 Example ==
 * Considering a ATLAS configuration like the following:
   # '''pgroup-atlas'''
@@ Line 117: / Line 117: @@
   migration cache -permanent -pins=keep -concurrency=1 -atime -accessed=0..600 -select=random -include=dc054_1 -target=pgroup \
     -exclude=dc091*,dc092*,dc093* -id=caching_recently_accessed_files -verify -size=1070000.. -- pgroup-atlas-ro

@@ Line 114: / Line 114: @@
 * If we want to do the same for the '''pgroup-atlas-recall''' pool group:
+:* Run on every pool in '''pgroup-atlas-recall'''
   migration cache -permanent -pins=keep -concurrency=1 -atime -accessed=0..600 -select=random -include=dc054_1 -target=pgroup \
     -exclude=dc091*,dc092*,dc093* -id=caching_recently_accessed_files -verify -size=1070000.. -- pgroup-atlas-ro

@@ Line 67: / Line 67: @@
   save
-* In short, the above ''migration'' command permanently runs and migrates data bigger or equal than 1070000 and accessed during the last 10', from any pool in '''pgroup-cms''' to '''pgroup-cms'''. This migration is performed with concurrency 1, checksum for each file is performed, atime is maintained in order to do not modify access time for the file and preserve the real access of a file. '''Random''' target location is selected in order to do not have in account the free space.
+* In short, the above ''migration'' command permanently runs and migrates data bigger or equal than 1070000 and accessed during the last 10', from any pool in '''pgroup-cms''' to '''pgroup-cms-ro'''. This migration is performed with concurrency 1, checksum for each file is performed, atime is maintained in order to do not modify access time for the file and preserve the real access of a file. '''Random''' target location is selected in order to do not have in account the free space.
 == ATLAS Example ==

@@ Line 70: / Line 70: @@
 == ATLAS Example ==
-* Considering a CMS configuration like the following:
+* Considering a ATLAS configuration like the following:
-  # '''pgroup-cms'''
+  # '''pgroup-atlas'''
-       |- dc021_1, dc021_2, dc021_3, dc021_4, dc021_5, dc021_6, dc021_7
+       |- dc020_1, dc020_2, dc020_3, dc020_4
-       |- dc022_1, dc022_2, dc022_3, dc022_4, dc022_5, dc022_6, dc022_7
+      |- dc028_1, dc028_2, dc028_3, dc028_4
-       |- dc029_1, dc029_2, dc029_3, dc029_4, dc029_5, dc029_6, dc029_7
+       |- dc032_1, dc032_2, dc032_3, dc032_4
-       |- dc031_1, dc031_2, dc031_3, dc031_4, dc031_5, dc031_6, dc031_7
+      |- dc033_1, dc033_2, dc033_3, dc033_4
-       |- dc037_1
+       |- dc036_1
-       |- dc040_1
+      |- dc041_1
-       |- dc044_1, dc044_2, dc044_3
+       |- dc042_1
-  # '''pgroup-cms-ro'''
+  # '''pgroup-atlas-ro'''
-       |- dc052_1
+       |- dc054_1
-  # '''pgroup-cms-recall'''
+  # '''pgroup-atlas-recall'''
-       |- dc100_1
+       |- dc091_1
-       |- dc101_1
+       |- dc092_1
-* We want make a permanent cache migration from '''pgroup-cms''' to '''pgroup-cms-ro''' according to the following rules:
+      |- dc093_1
 :* Copy only replicas accessed last 10' (0..600)
 :* Copy only files with size equal or bigger than 1070000 bytes
 :* Select target pools randomly (do not take account of the free space)
 :* Concurrency of 1 per migration process (want to avoid to stress the pools, so we consider the lowest value)
-:* Set target as '''pgroup''' (we want to migrate the data to any pool in '''pgroup-cms-ro''')
+:* Set target as '''pgroup''' (we want to migrate the data to any pool in '''pgroup-atlas-ro''')
 :* For security and in order to avoid updates in the poolmanager.conf that are not applied to the migration process, add the '''include''' and '''exclude''' options with the pools that will be affected.
-* Run on every pool in '''pgroup-cms'''
+* Run on every pool in '''pgroup-atlas'''
-  migration cache -permanent -pins=keep -concurrency=1 -atime -accessed=0..600 -select=random -include=dc052_1,dc053_1 -target=pgroup \
+  migration cache -permanent -pins=keep -concurrency=1 -atime -accessed=0..600 -select=random -include=dc054_1 -target=pgroup \
-    -exclude=dc021*,dc022*,dc029*,dc031*,dc037*,dc040*,dc044* -id=caching_recently_accessed_files -verify -size=1070000.. -- pgroup-cms-ro
+    -exclude=dc020*,dc028*,dc032*,dc033*,dc036*,dc041*,dc042*,dc046*,dc047*,dc048* -id=caching_recently_accessed_files -verify -size=1070000.. -- pgroup-atlas-ro
 * Optionally, '''save''' configuration in order to re-run it once pools gets restarted.
   save
-* In short, the above ''migration'' command permanently runs and migrates data bigger or equal than 1070000 and accessed during the last 10', from any pool in '''pgroup-cms''' to '''pgroup-cms'''. This migration is performed with concurrency 1, checksum for each file is performed, atime is maintained in order to do not modify access time for the file and preserve the real access of a file. '''Random''' target location is selected in order to do not have in account the free space.
+* In short, the above ''migration'' command permanently runs and migrates data bigger or equal than 1070000 and accessed during the last 10', from any pool in '''pgroup-atlas''' to '''pgroup-atlas-ro'''. This migration is performed with concurrency 1, checksum for each file is performed, atime is maintained in order to do not modify access time for the file and preserve the real access of a file. '''Random''' target location is selected in order to do not have in account the free space.

@@ Line 4: / Line 4: @@
 :* If you just add the line in the '''${pool.path}/setup''', please remember to execute '''reload -yes''' on the pool admin console, or just the command from there.
-  migration cache -permanent -pins=keep -concurrency=<concurrency> -atime -accessed=<string> -select=proportional|random -include=<glob>[,<glob>]<...> -target=pool|pgroup|link \
+  migration cache -permanent -pins=keep -concurrency=<concurrency> -atime -accessed=<string> -select=proportional|random \
-   -exclude=<glob>[,<glob>]<...> -id=<string> -verify -size=<string> -- <pool|pgroup|link>
+    -include=<glob>[,<glob>]<...> -target=pool|pgroup|link -exclude=<glob>[,<glob>]<...> -id=<string> -verify -size=<string> -- <pool|pgroup|link>
 * '''Options''':

@@ Line 7: / Line 7: @@
     -exclude=dc021*,dc022*,dc029*,dc031*,dc037*,dc040*,dc044* -id=caching_recently_accessed_files -verify -size=1070000.. -- pgroup-cms-ro
-* Options:
+* '''Options''':
-:* Filter options:
+:* ''Filter options'':
 ::* '''-accessed=<string>''': Only copy replicas accessed n seconds ago, or accessed within the given, possibly open-ended, interval. E.g. -accessed=0..60 matches files accessed within the last minute; -accesed=60.. matches files accessed one minute or more ago.
 ::* '''-select=proportional|random''': Determines how a pool is selected from the set of target pools. Defaults to ''proportional''.
 :::* '''proportional''': selects a pool with a probability proportional to the free space.
 :::* '''random''': selects a pool randomly.
-* Lifetime options:
+:* ''Lifetime options'':
 :::* '''-permanent''': Mark job as permanent.
 ::::: However, you need to ensure that the command is set in the '''${pool.path}/setup''' file. Otherwise on a pool restart it will not run it.
-:* Target options:
+:* ''Target options'':
 ::* '''-exclude=<glob>[,<glob>]<...>''': Exclude target pools matching any of the patterns. Single character (?) and multi character (*) wildcards may be used.
 :::: This option is set to avoid RW pools being included in case that something is miss-configured from the ''poolmanager'' side.
@@ Line 23: / Line 23: @@
 ::* '''-target=pool|pgroup|link''': Determines the interpretation of the target names. Defaults to pool.
 :::: We should setup ''pgroup'' because generally we will not want to migrate to a specific pool.
-:* Transfer options:
+:* ''Transfer options'':
 ::* '''-verify''': Force checksum computation when an existing target is updated.
 ::* '''-atime''': Maintain last access time.