DCACHE CHECK POOL METADATA

From Public PIC Wiki
Revision as of 13:29, 6 October 2017 by Mcaubet (Talk | contribs) (Special thanks)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

For the MoD

  • Open a ticket to incidents@support.pic.es with the alarm.
  • This alarm only affects the duration of the pool start up, nothing has to be done until the next working day where this can be fixed by dCache Administrators.
  • Same information as DCACHE CHECK POOL BTREE MEMORY as both alarms are strictly related.

For the Admin

  • Berkely DB should be tunned according to the number of meta files that a pool contains. If this not correctly adapted, pool start up will may need several hours to be fully available. Gerd explained how to tune up a pool according to this number. Below I show a modified explanation of what he said, adapting the explanation to the ATLAS pools.
  • The default je.log.fileCacheSize is 100. This could be increased by creating a file je.properties and adding the 'je.log.fileCacheSize=100'.
  • Also, other relevant settings are:
je.maxMemory=
je.maxMemoryPercent=20
  • These are equivalent. The former sets the number of bytes used for the btree cache, while the latter defines it as a percentage of the max heap size. E.g. if dcache.java.memory.heap is set to 2048m, then 410 MB is used for the btree cache.
  • The question is then how to determine a good size for the cache. There are some hidden utilities you can use to do this.
  • Below I show an example for dc042_1:
[root@dc042 meta]# java -cp  /usr/share/dcache/classes/je-*.jar com.sleepycat.je.util.DbPrintLog -h . -S
<DbPrintLog>
Log statistics:
                type               total         provisional               total                 min                 max                 avg             entries
                                   count               count               bytes               bytes               bytes               bytes         as % of log
               MapLN               1,324                   0          12,617,573                  76              11,985               9,529                 0.3
              NameLN                   4                   0                 141                  31                  38                  35                   0
       FileSummaryLN             231,140                   0          55,644,337                  24              49,701                 240                 1.4
                  IN              84,083               1,080         296,942,059                  45               5,709               3,531                 7.7
                 BIN             114,865             114,865         377,569,148                  44               6,675               3,287                 9.8
              DbTree                 318                   0           1,034,997               2,680               4,136               3,254                   0
              Commit           3,376,012                   0         114,784,408                  34                  34                  34                   3
           CkptStart                 158                   0               4,899                  30                  33                  31                   0
             CkptEnd                 226                   0              22,560                  67                 186                  99                   0
               Trace               2,506                   0             525,597                  48                 402                 209                   0
          FileHeader                 385                   0              14,630                  38                  38                  38                   0
           DEL_LN_TX             748,055                   0          51,409,527                  57                  69                  68                 1.3
           INS_LN_TX             619,747                   0         192,298,301                  96               1,443                 310                   5
           UPD_LN_TX           2,008,226                   0         398,854,718                  32               1,919                 198                10.4
              UPD_LN           6,636,173                   0       1,997,042,458                  18               1,819                 300                51.9
         NewBINDelta             720,540             720,540         348,480,254                  55               1,782                 483                 9.1
           key bytes          10,012,201                             387,254,450                   1                  61                  38              (10.1)
          data bytes           9,264,146                           2,036,632,128                   1               1,848                 219              (52.9)

Total bytes in portion of log read: 3,847,245,607
Total number of entries: 14,543,762

Per checkpoint interval info:
               lnTxn                  ln            mapLNTxn               mapLN          end to end        end to start        start to end         maxLNReplay             ckptEnd
                 273              63,082                   0                  10          30,455,407       3,857,456,063                   0              63,355   0x1f2f/0x6f2ef
               5,168             130,892                   0                  24          39,782,369       3,827,000,656                   0             136,060   0x226b/0x3a0d0
              27,098             132,905                   0                  15          50,120,541          16,509,395          33,611,146             160,003   0x252c/0x577ad
                 128              32,773                   0                   3           9,904,160       3,737,097,746                   0             134,154   0x254c/0x4014d
                  82              31,341                   0                  12          10,576,583       3,727,193,586                   0              31,423   0x2567/0xccd94
                   0                  72                   0                   4             711,899              51,780             660,119                  72   0x2567/0x17aa6f
              36,445              61,509                   0                   0          28,499,838       3,715,905,104                   0              97,954   0x25e5/0xc66d
                   2                 108                   0                   6           1,564,919              56,871           1,508,048                 110   0x25e5/0x18a764
              10,732              50,299                   0                   5          18,877,564       3,685,840,347                   0              61,055   0x2678/0x786e0
                  33                 190                   0                   6           1,627,301              94,279           1,533,022                 223   0x2678/0x205b85
               8,277              84,136                   0                   3          28,217,133          17,808,699          10,408,434              92,434   0x26be/0x52732
               5,401              25,394                   0                   7           9,824,151       3,637,118,349                   0              61,029   0x26cf/0x27849
              49,026              92,572                   0                   3          41,592,807       3,627,294,198                   0             141,598   0x2747/0x1ac630
                   2                  46                   0                   6             357,170              45,107             312,063                  48   0x2747/0x203962
              20,496              39,154                   0                   3          18,591,447       3,585,344,221                   0              59,654   0x27db/0xabb39
                   0                 154                   0                   6           1,187,463              83,676           1,103,787                 154   0x27db/0x1cd9c0
              17,107              49,468                   0                   3          18,905,722       3,565,565,311                   0              66,583   0x285c/0xc273a
                  58                   0                   0                   6             392,501              62,920             329,581                  58   0x285c/0x12246f
             113,890             201,969                   0                  17          79,488,418       3,546,267,088                   0             315,874   0x28ed/0xa5611
              22,147              98,579                   0                   8          40,657,515          36,319,235           4,338,280             120,726   0x2952/0x145e7c
                   0                 122                   0                   6           1,296,043              70,361           1,225,682              28,367   0x2952/0x282527
              19,570              65,944                   0                   5          27,570,660          25,251,492           2,319,168              85,520   0x2962/0x3138b
              37,714              28,889                   0                   3          21,505,514          19,337,884           2,167,630              66,613   0x297d/0x1a0c75
                   3                 332                   0                   6           2,437,155             124,630           2,312,525                 479   0x297d/0x3f3c98
              41,458              62,406                   0                   3          26,385,897       3,373,311,783                   0             103,904   0x29c7/0x81701
                   0                  46                   0                   6           1,404,642              46,045           1,358,597                  46   0x29c7/0x1d85e3
              41,443              87,747                   0                   3          39,909,271          14,624,816          25,284,455             129,198   0x29fb/0x1c237a
                   0                 123                   0                   5           1,185,318              68,652           1,116,666              79,470   0x29fb/0x2e39a0
               9,199              12,698                   0                   4           9,927,451       3,304,426,655                   0              21,899   0x2a00/0x2d1e3b
                   8                   0                   0                   5             118,374              35,364              83,010                   8   0x2a00/0x2eeca1
             122,697              96,787                   0                   3          66,966,017       3,294,380,830                   0             219,488   0x2a44/0xa122
              53,436              12,002                   0                   3          21,830,901       3,227,414,813                   0              65,438   0x2a53/0x1c9117
                  20                 167                   0                   6           1,308,679              85,964           1,222,715                 187   0x2a53/0x30891e
              18,552              38,805                   0                   6          17,501,977       3,204,275,233                   0              57,381   0x2a58/0xa6b37
                   9                 105                   0                   4             678,297              87,871             590,426                 114   0x2a58/0x14c4d0
              25,434              36,231                   0                  10          19,064,711       3,186,094,959                   0              61,675   0x2a5e/0x67f57
                  15                 159                   0                   6           1,467,746              83,228           1,384,518                 174   0x2a5e/0x1ce4b9
              12,236              14,099                   0                   3           8,375,785       3,165,562,502                   0              26,358   0x2a63/0x41c22
                   6                 184                   0                   6           1,058,988              89,948             969,040                 190   0x2a63/0x1444ce
              71,844              24,018                   0                   3          29,188,061          27,937,328           1,250,733              95,870   0x2aac/0x7e12b
               9,407              25,608                   0                   7          11,016,096       3,126,939,668                   0              35,029   0x2aba/0x17624b
                  20                 110                   0                   6             748,510              82,108             666,402                 130   0x2aba/0x22ce29
              64,997              49,072                   0                  15          27,822,548          27,438,951             383,597             114,142   0x2af3/0x1947d
              10,534              23,197                   0                   6          12,103,837       3,087,352,514                   0              33,737   0x2b01/0x21ae9a
                 122                   0                   0                  32             879,716             489,740             389,976                 122   0x2b01/0x2f1afe
              60,029             128,431                   0                  24          58,556,679          56,338,979           2,217,700             188,485   0x2b3c/0x191505
                  15                  52                   0                   6             530,374              53,795             476,579                 231   0x2b3c/0x212ccb
              48,799              11,045                   0                   4          18,178,427          17,837,860             340,567              59,855   0x2b41/0x56146
              45,998              29,564                   0                   3          20,154,236       2,997,103,481                   0              75,573   0x2b52/0x7bbc2
              39,570              60,757                   0                  15          30,273,541       2,976,949,245                   0             100,327   0x2b8e/0xbe847
              31,631              33,780                   0                  13          22,348,448          16,815,306           5,533,142              65,411   0x2bad/0x2fbde7
              16,831              40,178                   0                   5          20,571,223       2,924,327,256                   0              57,047   0x2bc6/0x38753e
               4,292              18,826                   0                   8           8,363,076       2,903,756,033                   0              23,118   0x2be9/0x1f7b02
                   0                 254                   0                   6           2,043,974             119,900           1,924,074                 254   0x2be9/0x3eab48
                  30              19,902                   0                   4           7,727,051       2,893,348,983                   0              19,954   0x2c00/0x1bfc93
              27,984              75,433                   0                  32          29,896,927       2,885,621,932                   0             103,417   0x2c13/0x1a69f2
              40,477              36,001                   0                  10          19,737,838          17,591,303           2,146,535              76,478   0x2c2a/0x1669e0
                   2                   2                   0                   4             194,668              36,808             157,860                  10   0x2c2a/0x19624c
              29,246              68,367                   0                   3          31,325,183       2,835,792,499                   0              97,615   0x2c4c/0x2d9acb
                   0                  26                   0                   5           1,118,302              27,604           1,090,698                  26   0x2c4c/0x3eab29
                   4              19,763                   0                   3           7,562,665       2,803,349,014                   0              19,775   0x2c53/0x197a52
                  26                 194                   0                   6           1,436,963             103,803           1,333,160                 220   0x2c53/0x2f6775
               6,255              14,988                   0                   8           9,972,102       2,794,349,386                   0              21,292   0x2c5f/0x2efa7b
              39,212              18,305                   0                   5          18,591,243       2,784,377,284                   0              57,517   0x2c8f/0x197b86
                   0                  41                   0                   6           1,495,215             121,720           1,373,495                  41   0x2c8f/0x304c35
                 347              23,340                   0                   3           7,036,643       2,764,290,826                   0              23,697   0x2c95/0x31498
              23,408               9,116                   0                   3          10,098,528       2,757,254,183                   0              32,524   0x2ca4/0x49578
              45,631              27,183                   0                   3          19,972,142       2,747,155,655                   0              72,814   0x2cad/0x428a6
                   0                   0                   0                   1             963,802                  69             963,733                   0   0x2cad/0x12dd80
                 276              19,066                   0                   4          10,705,056           5,926,693           4,778,363              19,342   0x2cc4/0x1d9fa0
                   0              25,292                   0                   2          10,568,825           7,122,257           3,446,568              25,292   0x2ccf/0x264d99
                 722              25,106                   0                   3           9,376,076       2,704,945,830                   0              26,568   0x2cd3/0x1cc865
                  15                  96                   0                   6             601,987              65,907             536,080                 111   0x2cd3/0x25f7e8
              50,662              51,942                   0                   4          29,283,798       2,694,967,767                   0             102,611   0x2d20/0x1b0a3e
              18,486              16,140                   0                   4          12,291,893       2,665,683,969                   0              34,626   0x2d3a/0x3e02f3
              67,481             104,443                   0                   5          48,233,597          15,261,303          32,972,294             171,924   0x2d79/0x230ef0
              19,276              34,062                   0                   3          18,103,592       2,605,158,479                   0             171,476   0x2deb/0x61f18
                   0                 225                   0                   6             607,824              37,538             570,286                 225   0x2deb/0xf6568
              14,038              14,130                   0                   3          13,699,081       2,586,447,063                   0              28,385   0x2e21/0x47d6f1
               4,147              13,212                   0                   3           5,363,836       2,572,747,982                   0              17,359   0x2e40/0x118ed
                   0                   0                   0                   4             133,826              37,921              95,905                   0   0x2e40/0x323af
              31,744              33,396                   0                   3          22,247,625       2,567,250,320                   0              65,140   0x2e45/0x256f78
                   9                   0                   0                   6             290,601              55,855             234,746                   9   0x2e45/0x29dea1
                  16              25,020                   0                   4          10,475,420       2,544,712,094                   0              25,039   0x2e5b/0x311fbd
               1,682              19,008                   0                   3           8,507,134       2,534,236,674                   0              20,690   0x2e5f/0x1a583b
                  12                   0                   0                   6             354,107              41,047             313,060                  12   0x2e5f/0x1fbf76
               6,849              54,863                   0                   3          20,176,980       2,525,375,433                   0              61,719   0x2e7d/0x2272ca
              30,012              67,588                   0                   5          28,465,075          27,606,214             858,861              97,600   0x2eb6/0xb06fd
              14,048              16,912                   0                   2           9,587,676       2,476,733,378                   0              66,881   0x2ee1/0x4bc59
              15,183              12,704                   0                   6          11,270,153           6,834,792           4,435,361              27,887   0x2ef4/0x181de2
              23,012              39,873                   0                   7          18,610,316       2,455,875,549                   0              62,948   0x2f1a/0x2e96e
              25,472              49,151                   0                   6          22,681,935       2,437,265,233                   0              74,623   0x2f5a/0x2bd5bd
               1,031              60,319                   0                   6          20,425,359       2,414,583,298                   0              61,350   0x2f66/0x32534c
              48,185              83,225                   0                   4          42,656,335          28,813,330          13,843,005             131,410   0x2f9d/0x5adb9b
              16,859              32,139                   0                   3          18,804,637          12,310,056           6,494,581              74,600   0x2fa3/0x489e38
              29,050              79,391                   0                   8          47,784,742          37,879,192           9,905,550             108,511   0x300a/0x26d0de
                  32              38,306                   0                   6          19,991,849          14,244,764           5,747,085             117,588   0x3026/0x26b107
                   0                  37                   0                   6             345,851              39,476             306,375              25,114   0x3026/0x2bf802
              17,725              34,199                   0                   3          19,155,357       2,264,574,525                   0              51,955   0x303f/0x1f149f
               1,379              23,909                   0                   6          10,787,679       2,245,419,168                   0              25,288   0x304f/0x2b197e
                   3                  12                   0                   5             297,677              34,135             263,542                  15   0x304f/0x2fa44b
              17,968              47,908                   0                   2          17,014,500       2,234,333,812                   0              65,891   0x306c/0x2162f
              45,151              19,691                   0                   3          20,592,703          18,998,392           1,594,311              64,842   0x3071/0xb216e
              18,246              13,219                   0                   3          14,900,440       2,196,726,609                   0              31,514   0x308d/0x55e7c6
                  26              12,008                   0                   5           7,554,365           4,234,132           3,320,233              12,034   0x30a1/0x309683
              35,556              69,762                   0                  20          31,161,221       2,174,271,804                   0             105,338   0x30c2/0x424e88
              33,203             114,692                   0                   5          49,168,609          43,804,782           5,363,827             147,895   0x30fd/0x359ee9
                 182              19,563                   0                   7          10,363,229       2,093,941,974                   0              87,398   0x310c/0x3b29c6
                  14              19,817                   0                   4           9,355,037           4,743,642           4,611,395              19,831   0x3110/0x315263
                  46              37,708                   0                   8          20,678,814          10,884,622           9,794,192              41,674   0x315c/0x3bae01
                  88                 279                   0                   5             761,100              44,429             716,671               3,390   0x315c/0x474b0d
              40,709              61,572                   0                   4          26,837,961       2,052,783,794                   0             102,607   0x3189/0x170b56
                  42                   0                   0                   6             814,957              77,642             737,315                  42   0x3189/0x237ac3
              69,126             107,504                   0                   5          52,094,116          47,372,898           4,721,218             176,645   0x31b8/0x436ee7
                  46                   0                   0                   5             393,197              51,476             341,721              82,129   0x31b8/0x496ed4
              48,348             117,823                   0                   3          48,879,763          41,965,088           6,914,675             166,196   0x3201/0x3856e7
              16,349              39,063                   0                   7          17,875,388          14,606,658           3,268,730              62,982   0x3208/0x17eba3
                 105                   0                   0                   6           3,394,912              60,919           3,333,993               2,571   0x3208/0x4bb903
                 217              17,310                   0                   7           5,776,110       1,902,493,500                   0              17,617   0x3210/0xb4571
              36,657              27,268                   0                   7          22,160,771           4,617,088          17,543,683              63,925   0x3224/0x2c3df4
              24,153              42,022                   0                   1          22,019,603       1,874,556,619                   0             114,113   0x3235/0x4b0f07
                  16              18,174                   0                   7           7,706,321       1,852,537,016                   0              18,190   0x324e/0x280f58
                 120              24,379                   0                   7          10,458,169       1,844,830,695                   0              24,499   0x3266/0x2f0d11
              14,236              34,916                   0                   3          17,998,326          12,723,415           5,274,911              49,152   0x327b/0x108207
                  79                   0                   0                   5             673,666              37,699             635,967               2,808   0x327b/0x1ac989
               6,624              22,958                   0                   3          11,999,224       1,815,700,534                   0              29,650   0x3290/0x394b01
              19,965              40,810                   0                   8          17,269,398       1,803,701,310                   0              60,775   0x32a7/0xfa097
               7,517              58,175                   0                   5          21,772,528       1,786,431,912                   0              65,692   0x32de/0x2aac87
               9,415              18,600                   0                   6          10,786,946       1,764,659,384                   0              28,015   0x32e1/0x36ae89
                  51                   0                   0                   6             622,591              52,177             570,414                  51   0x32e1/0x402e88
              30,139              66,899                   0                   4          30,497,693       1,753,249,847                   0              97,065   0x3315/0x47c6a5
                   6              41,284                   0                   1          16,342,999          12,394,744           3,948,255              41,290   0x3320/0xff97c
              21,091              17,059                   0                   6          11,525,646       1,706,409,155                   0              38,984   0x332b/0x27410a
                   0                  94                   0                   4             892,946              34,897             858,049                  94   0x332b/0x34e11c
                   3              52,035                   0                   3          18,160,444       1,693,990,563                   0              52,132   0x336d/0x18cf58
                  58              27,035                   0                   6           8,947,956       1,675,830,119                   0              27,093   0x3376/0x8c1cc
                   3                  66                   0                   4             501,044              49,007             452,037                  69   0x3376/0x106700
              27,134              44,130                   0                   4          22,994,532       1,666,381,119                   0              71,330   0x339f/0x3e1864
                   0                 108                   0                   4             439,894              35,044             404,850                 108   0x339f/0x44ceba
                  17              51,506                   0                   3          20,284,195       1,642,946,693                   0              51,631   0x33a6/0x4924dd
                  28              16,826                   0                   2           9,954,053       1,622,662,498                   0              16,854   0x33b6/0x487162
               2,079              43,231                   0                   2          20,850,856          13,682,453           7,168,403              45,310   0x33d2/0x556d0a
                   2              30,027                   0                   3          14,418,091           8,924,719           5,493,372              30,743   0x33d9/0x40b5
                   0               1,157                   0                   6           5,510,215             155,394           5,354,821               3,103   0x33d9/0x5454fc
              31,517              31,808                   0                   3          18,110,480       1,571,929,283                   0              64,085   0x33ee/0x37800c
              30,598              27,193                   0                   4          18,121,300          12,201,650           5,919,650              57,791   0x3402/0x1ad560
              38,905              34,440                   0                   4          18,443,706          15,100,893           3,342,813              78,249   0x340c/0x3161a
              51,887              95,209                   0                   4          40,262,214          29,791,799          10,470,415             152,283   0x343b/0x71660
                   1              28,686                   0                   7          10,081,465       1,476,991,583                   0             142,625   0x344c/0x85499
              28,401              13,707                   0                   3          10,169,877       1,466,910,118                   0              42,108   0x3457/0xaec2e
                   5              16,531                   0                   6          14,562,193           4,402,715          10,159,478              16,536   0x345b/0x50893f
                   0              15,920                   0                   7           9,514,322       1,442,178,048                   0              19,604   0x3460/0x492011
                   4              16,264                   0                   6          10,131,895       1,432,663,726                   0              16,268   0x3465/0x4b2348
              30,231              28,986                   0                   4          18,938,103          14,093,138           4,844,965              59,217   0x347c/0x3aef3f
                  14                   0                   0                   5             512,006              34,432             477,574                 120   0x347c/0x42bf45
              20,678              38,294                   0                   3          17,101,301       1,403,081,722                   0              58,975   0x3497/0x16843a
               9,869              25,099                   0                   5          12,548,707       1,385,980,421                   0              34,968   0x34b3/0x3d681d
              36,465              60,605                   0                   5          26,361,794          24,558,695           1,803,099              97,070   0x34d6/0x5e45f
                  36              23,345                   0                   3           9,621,974           6,699,792           2,922,182              23,911   0x34d9/0x1fb5
              25,436              49,579                   0                   7          20,880,498          18,608,524           2,271,974              75,850   0x34df/0xd8f27
                  32               1,485                   0                   6           4,197,555             413,166           3,784,389               1,535   0x34df/0x4d9bda
              31,535              59,968                   0                   5          25,248,912          24,728,797             520,115              91,849   0x3512/0x51cea
               1,601              24,378                   0                   6          10,268,411           8,200,696           2,067,715              26,160   0x351b/0x93565
              14,147              23,162                   0                   8          11,300,331       1,276,852,570                   0              37,400   0x3525/0x1d0cd0
              14,204              20,662                   0                   3          13,299,117       1,265,552,239                   0              34,866   0x3532/0x4f63fd
              17,514              24,728                   0                   3          16,797,576          12,513,187           4,284,389              42,242   0x3553/0x1e8685
               7,303              46,343                   0                   7          23,094,370          14,762,461           8,331,909              54,344   0x3580/0x4dbde7
              45,620              41,812                   0                   7          27,514,220          21,511,990           6,002,230              90,879   0x359c/0x27cfd3
              16,861              12,552                   0                   3           9,619,073       1,184,846,956                   0             104,424   0x35a0/0x21ffd4
                   0                 271                   0                   6           3,731,928              93,043           3,638,885                 271   0x35a0/0x5af1ac
              30,860              48,187                   0                   7          26,338,978          24,109,627           2,229,351              79,097   0x35ab/0x2314ce
              39,031              24,255                   0                   4          20,202,284       1,145,156,977                   0              63,301   0x35c4/0x262afa
               2,576              46,002                   0                   6          19,184,048           5,580,647          13,603,401              48,578   0x35dc/0x19b7aa
                   2                   0                   0                   6             174,028              33,089             140,939              31,536   0x35dc/0x1c5f76
             111,528              77,161                   0                   7          62,341,581          50,549,489          11,792,092             188,690   0x360c/0x401a43
              28,697              25,910                   0                   6          18,246,350          13,855,848           4,390,502              56,636   0x3618/0x255811
               4,335              12,127                   0                   6           9,004,926           5,130,460           3,874,466              18,985   0x3621/0x16290f
              18,413               8,607                   0                   5           8,878,516           7,701,363           1,177,153              27,027   0x3626/0x50c43
                   0               2,910                   0                   6           6,228,723             848,601           5,380,122               2,910   0x3626/0x641736
              28,784              22,689                   0                   3          21,119,867          14,626,185           6,493,682              52,178   0x362a/0x752db1
              41,676              62,528                   0                   4          34,112,988          29,303,765           4,809,223             105,238   0x363e/0x1b598d
                  17                 735                   0                   5          12,521,216              31,105          12,490,111               1,300   0x3643/0x41d20d
               7,802              34,494                   0                   7          33,127,169           7,514,191          25,612,978              43,003   0x3659/0x71898e
                   0               8,734                   0                   5           3,377,408         900,015,281                   0              25,130   0x365c/0xc7c0e
                 114               9,858                   0                   7          15,782,161             650,578          15,131,583               9,972   0x365f/0x64b69f
                   1              11,516                   0                   7           5,378,418         880,855,712                   0              19,548   0x3664/0x1e3191
                   7              22,005                   0                   7           8,922,863           6,691,187           2,231,676              22,012   0x3668/0xdc200
                  13              13,409                   0                   6          15,952,421             360,887          15,591,534              14,391   0x3671/0x6895a5
               1,749               8,568                   0                   3           3,660,457         850,602,010                   0              22,698   0x3676/0x7d9ce
               5,390              10,146                   0                   6          14,882,555           1,855,609          13,026,946              15,536   0x3688/0x525a49
              10,230               9,513                   0                   5           4,816,061         832,058,998                   0              28,465   0x3691/0x34086
              14,698              31,907                   0                   7          41,080,361          36,925,183           4,155,178              46,605   0x36aa/0x13bcaf
                   2               1,755                   0                   6           4,666,159             345,619           4,320,540              47,630   0x36aa/0x5aefde
              25,935              30,385                   0                   5          38,398,767         781,496,417                   0              57,119   0x36c5/0x42810d
               9,478              14,696                   0                   7          21,102,013         743,097,650                   0              24,174   0x36e1/0x5351ca
                  65              12,746                   0                   6           5,610,587           3,858,013           1,752,574              12,811   0x36ed/0x1057a5
               5,806               9,341                   0                   6          10,353,968           2,690,013           7,663,955              15,343   0x36fc/0x15be55
                   5                   0                   0                   6           4,894,393              33,744           4,860,649               5,931   0x36fc/0x606d0e
              28,754              58,230                   0                  18          28,401,618         701,136,689                   0              86,988   0x3708/0x480960
                   0              15,897                   0                   6          10,890,931           3,098,266           7,792,665              15,897   0x370a/0x55a193
                 126              10,927                   0                   6          26,068,972           2,214,107          23,854,865              16,100   0x372d/0x19a5ff
               9,837              16,155                   0                   6          11,096,409           6,346,816           4,749,593              29,696   0x3732/0x2a60d8
                 128              15,250                   0                   6           8,591,606           3,747,801           4,843,805              16,046   0x3734/0x14e34e
                 123              28,336                   0                   6          16,047,724           5,161,792          10,885,932              31,737   0x3736/0x712b3a
                  24               8,534                   0                   5           3,423,605           1,552,888           1,870,717              19,982   0x3740/0xcd22f
                 533                   0                   0                   6          20,325,930             275,776          20,050,154               3,884   0x3742/0x11cb59
                   0                   0                   0                   4              72,242              36,230              36,012                   0   0x3742/0x12e58b
              26,449              16,768                   0                   3          20,777,536         576,217,652                   0              43,217   0x3756/0x1ec2cb
              11,493              23,349                   0                   3          21,836,979          13,435,005           8,401,974              34,842   0x375c/0x3aca7e
                 309              32,239                   0                   6          17,558,766           9,588,731           7,970,035              56,790   0x3761/0x158a6c
                  12              26,562                   0                   7          12,407,639           6,425,279           5,982,360              27,257   0x3779/0x3a4743
               6,845              41,908                   0                   7          33,072,537          22,775,381          10,297,156              51,339   0x379b/0x69295c
               6,778              29,990                   0                   6          25,664,453          17,110,934           8,553,519              41,294   0x37a8/0x2701a1
                   5              16,509                   0                   9          19,648,397           8,979,547          10,668,850              44,387   0x37ae/0x21a42e
             116,569             137,423                   0                  12         129,250,694          72,386,145          56,864,549             267,033   0x37e2/0x163534
                  13              17,546                   0                   6          28,042,727              34,685          28,008,042             187,464   0x37e7/0x90ee1b
              33,326              30,872                   0                   6          43,125,925           9,805,827          33,320,098              81,756   0x37ec/0x280a40
                   3               5,332                   0                   5          19,873,538              37,653          19,835,885              30,103   0x37ef/0x261c42
                  17               2,306                   0                   6          19,166,423             244,987          18,921,436               7,657   0x37f1/0x196419
              10,753              20,277                   0                   6          40,648,779          11,079,498          29,569,281              32,618   0x37f6/0x234a64
                  10              22,876                   0                   6          18,391,375             164,634          18,226,741              49,034   0x37f8/0xabeb3
                  30              34,377                   0                   7          32,006,385           1,775,081          30,231,304              56,545   0x37fb/0x295c24
                   4              23,090                   0                   7          38,624,657           2,288,459          36,336,198              52,101   0x37ff/0x145fb5
                   6               8,689                   0                   6          27,108,848             949,959          26,158,889              24,449   0x3801/0x80d8a5
                   6               5,117                   0                   7          18,644,436              55,620          18,588,816              11,221   0x3803/0x6c2979
                 837                   0                   0                   6           6,661,670             288,422           6,373,248               5,956   0x3804/0x39391f
               5,863               6,917                   0                   0           3,705,888           3,705,888                   0              12,780   0x3804/0x71c53f
</DbPrintLog>
  • This prints some statistics about the database. In particular the two rows called key bytes and data bytes are relevant. For the following step you need the values from the avg column - i.e. 38 and 219 in this case. You also need the value of “Total number of entries” (14543762 in this case). Now we take those values and put them into this command:
[root@dc042 meta]# java -cp  /usr/share/dcache/classes/je-*.jar com.sleepycat.je.util.DbCacheSize -records 14543762 -key 38 -data 219

=== Environment Cache Overhead ===

3,157,213 minimum bytes

To account for JE daemon operation and record locks,
a significantly larger amount is needed in practice.

=== Database Cache Size ===

   Number of Bytes  Description
   ---------------  -----------
     1,195,759,832  Internal nodes only
     5,124,232,392  Internal nodes and leaf nodes
  • For further information see the DbCacheSize javadoc.
  • For this pool it tells us that we need a bit above 1195 MB to keep all the internal btree nodes cached. Oracle’s recommendation is that if the database is updated often, the cache is at least big enough to contain the internal nodes. Obviously you want to make it a bit bigger to leave room for it to grow. One could take this and configure the cache using the je.maxMemory setting. If the size is lower than the 20% of the max heap size you already use, then you don’t need to do anything (I do not suggest lowering it further).
  • Since this pool has 6000m of JAVA heap max space, it means that 20% = 1200MB, which is exactly cache for Internal btree nodes. So we will not touch this.
  • There is one caveat though: If you increase the cache size, less free space is left on the heap. If the pool is pushed to the limit, this may actually slow down the pool as garbage collection overhead increases. You need to ensure that enough space is left in addition to the cache (possibly by increasing the max heap size). An alternative to adjusting je.maxMemory is of course to adjust the max heap size. If you make it large enough so that 20% is enough to cache the internal btree nodes, then all is well. This may however mean you assign significantly more memory to the pool than it really needs. Yeah, there are lots of things to consider.
  • A little bonus: There is also this command:
[root@dc042 meta]#  java -cp  /usr/share/dcache/classes/je-*.jar com.sleepycat.je.util.DbSpace -h .
  File    Size (KB)  % Used
--------  ---------  ------
00001857       9765      41
00001f2a       9765      43
00001f2f       9765      39
00001f72       9765      44
0000216d       9765      40
000021cb       9765      40
0000226b       9765      45
000022e8       9763      38
000022fa       9765      39
00002390       9765      38
0000241d       9765      49
0000252c       9765      51
0000254c       9765      47
00002567       9765      39
000025b7       9765      42
000025db       9765      42
000025e5       9765      56
0000265f       9765      39
00002678       9765      61
0000268d       9762      48
00002692       9765      74
000026be       9765      48
000026cf       9765      40
00002700       9765      41
00002706       9765      38
00002718       9765      56
00002747       9765      44
0000274d       9765      48
000027db       9765      62
00002855       9765      39
0000285c       9765      39
00002882       9765      41
0000289d       9765      54
000028ba       9765      52
000028c1       9765      53
000028d0       9765      41
000028d7       9765      41
000028e7       9765      39
000028ed       9765      39
00002926       9765      39
0000292b       9765      43
00002945       9765      41
00002952       9765      41
00002958       9765      62
00002959       9764      38
00002962       9765      53
00002963       9765      47
0000297d       9765      51
0000297e       9765      41
000029b3       9765      45
000029c7       9765      56
000029ce       9765      42
000029d3       9765      43
000029d9       9765      54
000029fb       9765      42
00002a00       9765      54
00002a01       9765      39
00002a06       9765      39
00002a20       9765      45
00002a26       9765      61
00002a2b       9765      69
00002a30       9765      65
00002a44       9765      46
00002a4f       9765      50
00002a53       9765      64
00002a54       9765      47
00002a58       9765      66
00002a59       9765      43
00002a5e       9765      56
00002a63       9765      44
00002a69       9765      47
00002a8f       9765      39
00002aac       9765      54
00002aba       9765      42
00002abb       9765      44
00002ae0       9763      38
00002af3       9765      45
00002b01       9765      41
00002b08       9765      65
00002b0e       9765      41
00002b24       9765      45
00002b28       9765      61
00002b29       9765      41
00002b3c       9765      40
00002b3d       9765      40
00002b41       9765      47
00002b4c       9765      46
00002b52       9765      57
00002b64       9765      38
00002b87       9765      38
00002b8e       9765      47
00002b8f       9760      38
00002bad       9765      42
00002bbf       9765      44
00002bc6       9765      38
00002be9       9765      47
00002c00       9765      42
00002c06       9765      69
00002c0c       9765      55
00002c13       9765      38
00002c14       9765      38
00002c2a       9765      41
00002c2b       9765      41
00002c30       9765      54
00002c4c       9765      39
00002c53       9765      45
00002c5f       9765      44
00002c7c       9765      38
00002c8f       9765      49
00002c95       9765      38
00002ca4       9764      50
00002ca9       9765      42
00002cad       9763      41
00002cc4       9765      39
00002ccf       9765      43
00002cd3       9765      45
00002cf3       9765      48
00002d19       9765      38
00002d20       9765      38
00002d3a       9765      40
00002d3b       9765      41
00002d41       9765      43
00002d51       9765      39
00002d59       9765      40
00002d79       9765      62
00002d8f       9765      39
00002deb       9765      40
00002e21       9765      39
00002e40       9765      41
00002e41       9765      45
00002e45       9765      40
00002e5b       9765      58
00002e5f       9765      49
00002e60       9765      67
00002e7d       9765      48
00002e95       9765      38
00002eb1       9762      57
00002eb6       9765      53
00002ee1       9765      38
00002ef4       9763      43
00002ef8       9765      41
00002f1a       9765      53
00002f25       9765      65
00002f5a       9765      52
00002f5f       9765      63
00002f66       9765      41
00002f71       9765      42
00002f79       9765      61
00002f87       9765      49
00002f9d       9765      41
00002f9e       9765      42
00002fa3       9765      38
00002fa4       9765      46
00002fb1       9762      55
00002fc1       9765      41
00002fde       9765      38
0000300a       9765      43
0000300e       9762      47
00003026       9765      55
00003027       9765      57
0000303f       9765      63
0000304f       9765      48
00003056       9765      44
0000306c       9765      48
0000306d       9765      42
00003071       9765      45
0000308d       9765      39
000030a1       9765      54
000030a5       9765      51
000030ab       9765      60
000030c2       9765      46
000030d5       9765      43
000030dd       9765      38
000030e3       9765      57
000030e4       9765      55
000030fd       9765      48
0000310c       9764      42
00003110       9765      46
00003119       9765      47
0000315c       9765      48
0000315d       9765      54
00003185       9765      56
00003189       9765      52
0000318a       9765      58
0000319a       9765      46
0000319f       9764      39
000031b5       9764      53
000031b8       9765      40
000031b9       9765      74
000031e2       9765      65
000031fa       9765      58
000031fb       9765      52
00003201       9765      50
00003202       9764      44
00003208       9765      42
00003210       9763      40
00003218       9765      40
00003224       9765      51
00003229       9765      48
00003235       9765      40
0000324e       9765      46
00003266       9765      41
0000326c       9764      47
0000327b       9765      55
00003290       9765      48
00003294       9765      56
000032a7       9765      69
000032ad       9765      62
000032de       9765      47
000032e1       9765      46
000032e2       9765      71
000032e9       9765      45
00003315       9765      42
00003316       9764      54
00003320       9765      43
0000332b       9765      52
0000332c       9765      76
0000336d       9765      69
00003376       9765      72
00003377       9765      46
0000339f       9765      43
000033a0       9765      75
000033a6       9765      46
000033b6       9765      40
000033c0       9764      72
000033d2       9765      38
000033d3       9764      45
000033d9       9765      42
000033da       9765      44
000033ee       9765      41
000033fb       9764      38
00003402       9765      42
00003403       9765      53
0000340c       9765      76
00003428       9765      67
00003431       9765      51
00003436       9765      88
0000343b       9765      85
0000344c       9765      48
00003457       9765      46
0000345b       9765      46
00003460       9765      50
00003465       9765      45
00003466       9762      40
0000347c       9765      54
0000347d       9765      61
00003497       9765      63
000034b3       9765      64
000034bd       9765      54
000034c9       9764      71
000034d6       9765      65
000034d9       9765      86
000034da       9765      46
000034df       9765      51
000034e0       9764      60
00003504       9762      80
00003512       9761      76
0000351b       9765      65
00003525       9765      59
00003532       9765      52
00003540       9764      43
00003553       9765      80
00003558       9765      54
00003580       9762      49
00003586       9765      41
00003587       9763      50
0000359c       9765      40
000035a0       9765      46
000035a1       9765      37
000035a6       9765      96
000035ab       9765      52
000035ac       9765      39
000035c4       9765      61
000035d0       9765      89
000035dc       9765      74
000035dd       9765      43
000035f1       9765      43
000035fd       9765      37
00003604       9765      47
00003609       9764      40
0000360c       9765      59
0000360d       9765      37
00003618       9763      42
00003621       9765      42
00003626       9765      42
00003627       9765      41
0000362a       9765      38
0000362b       9765      73
00003630       9765      68
00003631       9764      42
0000363e       9763      39
00003643       9765      43
00003646       9765      38
0000364c       9762      40
00003659       9765      47
0000365c       9764      44
0000365f       9765      45
00003664       9765      69
00003668       9764      37
00003671       9765      43
00003676       9765      39
00003688       9765      37
00003691       9765      48
00003692       9764      37
00003695       9762      38
000036a3       9764      50
000036aa       9765      47
000036ab       9765      49
000036c2       9765      64
000036c3       9764      57
000036c5       9765      40
000036da       9764      40
000036e1       9765      43
000036ed       9762      39
000036fc       9765      41
000036fd       9765      59
00003701       9764      85
00003708       9765      43
0000370a       9763      49
00003728       9762      41
0000372a       9763      55
0000372d       9765      51
00003732       9763      48
00003734       9765      60
00003736       9765      42
00003740       9765      48
00003741       9765      39
00003742       9765      38
00003749       9764      40
00003756       9763      39
00003758       9762      46
0000375c       9765      67
0000375d       9765      37
00003761       9765      63
00003779       9765      55
00003785       9765      43
00003798       9763      60
0000379b       9765      41
0000379e       9765      37
000037a5       9764      39
000037a8       9764      48
000037ab       9765      38
000037ae       9765      54
000037b3       9764      42
000037b6       9761      36
000037be       9763      46
000037c1       9765      69
000037c3       9765      36
000037c4       9764      37
000037ce       9762      50
000037d1       9761      45
000037d3       9765      45
000037dd       9765      45
000037df       9765      50
000037e0       9763      39
000037e2       9765      13
000037e3       9763      18
000037e7       9765      49
000037e8       9763      41
000037e9       9765      42
000037ea       9763      40
000037eb       9765      39
000037ec       9761      39
000037ed       9763      41
000037ef       9763      36
000037f0       9764      36
000037f1       9764      39
000037f3       9765      41
000037f4       9761      53
000037f5       9765      36
000037f6       9765      54
000037f7       9763      53
000037f8       9765      63
000037f9       9765      60
000037fa       9765      70
000037fb       9765      76
000037fc       9765      72
000037fd       9765      73
000037fe       9765      83
000037ff       9765      87
00003800       9765      89
00003801       9762      86
00003802       9763      88
00003803       9764      90
00003804       8110      76
 TOTALS     3757891      49
  • It will tell you for each of the database files (the log segments) how big it is and what the utilization is. The Berkeley DB will try to keep the total utilization above 50%, but for a small pool like this one it cannot do it. I figure that you may find this interesting to know after reading about utilization above.
  • The log segment size and the utilization goal can be adjusted too, but I cannot give any sound advice on whether that’s a good idea and how to determine good values.
  • All of the above is only relevant if you use the Berkeley DB backend for pools.

Special thanks