View source for DCACHE CHECK POOL BTREE MEMORY

= For the MoD =
* Open a ticket to incidents@support.pic.es with the alarm. 
* This alarm only affects the duration of the pool start up, nothing has to be done until the next working day where this can be fixed by dCache Administrators.
* Same information as [[DCACHE CHECK POOL METADATA]] as both alarms are strictly related.

= For the Admin =

* Berkely DB should be tunned according to the number of meta files that a pool contains. If this not correctly adapted, pool start up will may need several hours to be fully available. Gerd explained how to tune up a pool according to this number. Below I show a modified explanation of what he said, adapting the explanation to the ATLAS pools.

* The default je.log.fileCacheSize is 100. This could be increased by creating a file je.properties and adding the 'je.log.fileCacheSize=100'.

* Also, other relevant settings are:

 je.maxMemory=
 je.maxMemoryPercent=20

* These are equivalent. The former sets the number of bytes used for the btree cache, while the latter defines it as a percentage of the max heap size. E.g. if dcache.java.memory.heap is set to 2048m, then 410 MB is used for the btree cache.

* The question is then how to determine a good size for the cache. There are some hidden utilities you can use to do this.

* Below I show an example for dc042_1:

 [root@dc042 meta]# java -cp  /usr/share/dcache/classes/je-6.3.8.jar com.sleepycat.je.util.DbPrintLog -h . -S
 <DbPrintLog>
 Log statistics:
                 type               total         provisional               total                 min                 max                 avg             entries
                                    count               count               bytes               bytes               bytes               bytes         as % of log
                MapLN               1,324                   0          12,617,573                  76              11,985               9,529                 0.3
               NameLN                   4                   0                 141                  31                  38                  35                   0
        FileSummaryLN             231,140                   0          55,644,337                  24              49,701                 240                 1.4
                   IN              84,083               1,080         296,942,059                  45               5,709               3,531                 7.7
                  BIN             114,865             114,865         377,569,148                  44               6,675               3,287                 9.8
               DbTree                 318                   0           1,034,997               2,680               4,136               3,254                   0
               Commit           3,376,012                   0         114,784,408                  34                  34                  34                   3
            CkptStart                 158                   0               4,899                  30                  33                  31                   0
              CkptEnd                 226                   0              22,560                  67                 186                  99                   0
                Trace               2,506                   0             525,597                  48                 402                 209                   0
           FileHeader                 385                   0              14,630                  38                  38                  38                   0
            DEL_LN_TX             748,055                   0          51,409,527                  57                  69                  68                 1.3
            INS_LN_TX             619,747                   0         192,298,301                  96               1,443                 310                   5
            UPD_LN_TX           2,008,226                   0         398,854,718                  32               1,919                 198                10.4
               UPD_LN           6,636,173                   0       1,997,042,458                  18               1,819                 300                51.9
          NewBINDelta             720,540             720,540         348,480,254                  55               1,782                 483                 9.1
            key bytes          10,012,201                             387,254,450                   1                  61                  38              (10.1)
           data bytes           9,264,146                           2,036,632,128                   1               1,848                 219              (52.9)
 
 Total bytes in portion of log read: 3,847,245,607
 Total number of entries: 14,543,762
 
 Per checkpoint interval info:
                lnTxn                  ln            mapLNTxn               mapLN          end to end        end to start        start to end         maxLNReplay             ckptEnd
                  273              63,082                   0                  10          30,455,407       3,857,456,063                   0              63,355   0x1f2f/0x6f2ef
                5,168             130,892                   0                  24          39,782,369       3,827,000,656                   0             136,060   0x226b/0x3a0d0
               27,098             132,905                   0                  15          50,120,541          16,509,395          33,611,146             160,003   0x252c/0x577ad
                  128              32,773                   0                   3           9,904,160       3,737,097,746                   0             134,154   0x254c/0x4014d
                   82              31,341                   0                  12          10,576,583       3,727,193,586                   0              31,423   0x2567/0xccd94
                    0                  72                   0                   4             711,899              51,780             660,119                  72   0x2567/0x17aa6f
               36,445              61,509                   0                   0          28,499,838       3,715,905,104                   0              97,954   0x25e5/0xc66d
                    2                 108                   0                   6           1,564,919              56,871           1,508,048                 110   0x25e5/0x18a764
               10,732              50,299                   0                   5          18,877,564       3,685,840,347                   0              61,055   0x2678/0x786e0
                   33                 190                   0                   6           1,627,301              94,279           1,533,022                 223   0x2678/0x205b85
                8,277              84,136                   0                   3          28,217,133          17,808,699          10,408,434              92,434   0x26be/0x52732
                5,401              25,394                   0                   7           9,824,151       3,637,118,349                   0              61,029   0x26cf/0x27849
               49,026              92,572                   0                   3          41,592,807       3,627,294,198                   0             141,598   0x2747/0x1ac630
                    2                  46                   0                   6             357,170              45,107             312,063                  48   0x2747/0x203962
               20,496              39,154                   0                   3          18,591,447       3,585,344,221                   0              59,654   0x27db/0xabb39
                    0                 154                   0                   6           1,187,463              83,676           1,103,787                 154   0x27db/0x1cd9c0
               17,107              49,468                   0                   3          18,905,722       3,565,565,311                   0              66,583   0x285c/0xc273a
                   58                   0                   0                   6             392,501              62,920             329,581                  58   0x285c/0x12246f
              113,890             201,969                   0                  17          79,488,418       3,546,267,088                   0             315,874   0x28ed/0xa5611
               22,147              98,579                   0                   8          40,657,515          36,319,235           4,338,280             120,726   0x2952/0x145e7c
                    0                 122                   0                   6           1,296,043              70,361           1,225,682              28,367   0x2952/0x282527
               19,570              65,944                   0                   5          27,570,660          25,251,492           2,319,168              85,520   0x2962/0x3138b
               37,714              28,889                   0                   3          21,505,514          19,337,884           2,167,630              66,613   0x297d/0x1a0c75
                    3                 332                   0                   6           2,437,155             124,630           2,312,525                 479   0x297d/0x3f3c98
               41,458              62,406                   0                   3          26,385,897       3,373,311,783                   0             103,904   0x29c7/0x81701
                    0                  46                   0                   6           1,404,642              46,045           1,358,597                  46   0x29c7/0x1d85e3
               41,443              87,747                   0                   3          39,909,271          14,624,816          25,284,455             129,198   0x29fb/0x1c237a
                    0                 123                   0                   5           1,185,318              68,652           1,116,666              79,470   0x29fb/0x2e39a0
                9,199              12,698                   0                   4           9,927,451       3,304,426,655                   0              21,899   0x2a00/0x2d1e3b
                    8                   0                   0                   5             118,374              35,364              83,010                   8   0x2a00/0x2eeca1
              122,697              96,787                   0                   3          66,966,017       3,294,380,830                   0             219,488   0x2a44/0xa122
               53,436              12,002                   0                   3          21,830,901       3,227,414,813                   0              65,438   0x2a53/0x1c9117
                   20                 167                   0                   6           1,308,679              85,964           1,222,715                 187   0x2a53/0x30891e
               18,552              38,805                   0                   6          17,501,977       3,204,275,233                   0              57,381   0x2a58/0xa6b37
                    9                 105                   0                   4             678,297              87,871             590,426                 114   0x2a58/0x14c4d0
               25,434              36,231                   0                  10          19,064,711       3,186,094,959                   0              61,675   0x2a5e/0x67f57
                   15                 159                   0                   6           1,467,746              83,228           1,384,518                 174   0x2a5e/0x1ce4b9
               12,236              14,099                   0                   3           8,375,785       3,165,562,502                   0              26,358   0x2a63/0x41c22
                    6                 184                   0                   6           1,058,988              89,948             969,040                 190   0x2a63/0x1444ce
               71,844              24,018                   0                   3          29,188,061          27,937,328           1,250,733              95,870   0x2aac/0x7e12b
                9,407              25,608                   0                   7          11,016,096       3,126,939,668                   0              35,029   0x2aba/0x17624b
                   20                 110                   0                   6             748,510              82,108             666,402                 130   0x2aba/0x22ce29
               64,997              49,072                   0                  15          27,822,548          27,438,951             383,597             114,142   0x2af3/0x1947d
               10,534              23,197                   0                   6          12,103,837       3,087,352,514                   0              33,737   0x2b01/0x21ae9a
                  122                   0                   0                  32             879,716             489,740             389,976                 122   0x2b01/0x2f1afe
               60,029             128,431                   0                  24          58,556,679          56,338,979           2,217,700             188,485   0x2b3c/0x191505
                   15                  52                   0                   6             530,374              53,795             476,579                 231   0x2b3c/0x212ccb
               48,799              11,045                   0                   4          18,178,427          17,837,860             340,567              59,855   0x2b41/0x56146
               45,998              29,564                   0                   3          20,154,236       2,997,103,481                   0              75,573   0x2b52/0x7bbc2
               39,570              60,757                   0                  15          30,273,541       2,976,949,245                   0             100,327   0x2b8e/0xbe847
               31,631              33,780                   0                  13          22,348,448          16,815,306           5,533,142              65,411   0x2bad/0x2fbde7
               16,831              40,178                   0                   5          20,571,223       2,924,327,256                   0              57,047   0x2bc6/0x38753e
                4,292              18,826                   0                   8           8,363,076       2,903,756,033                   0              23,118   0x2be9/0x1f7b02
                    0                 254                   0                   6           2,043,974             119,900           1,924,074                 254   0x2be9/0x3eab48
                   30              19,902                   0                   4           7,727,051       2,893,348,983                   0              19,954   0x2c00/0x1bfc93
               27,984              75,433                   0                  32          29,896,927       2,885,621,932                   0             103,417   0x2c13/0x1a69f2
               40,477              36,001                   0                  10          19,737,838          17,591,303           2,146,535              76,478   0x2c2a/0x1669e0
                    2                   2                   0                   4             194,668              36,808             157,860                  10   0x2c2a/0x19624c
               29,246              68,367                   0                   3          31,325,183       2,835,792,499                   0              97,615   0x2c4c/0x2d9acb
                    0                  26                   0                   5           1,118,302              27,604           1,090,698                  26   0x2c4c/0x3eab29
                    4              19,763                   0                   3           7,562,665       2,803,349,014                   0              19,775   0x2c53/0x197a52
                   26                 194                   0                   6           1,436,963             103,803           1,333,160                 220   0x2c53/0x2f6775
                6,255              14,988                   0                   8           9,972,102       2,794,349,386                   0              21,292   0x2c5f/0x2efa7b
               39,212              18,305                   0                   5          18,591,243       2,784,377,284                   0              57,517   0x2c8f/0x197b86
                    0                  41                   0                   6           1,495,215             121,720           1,373,495                  41   0x2c8f/0x304c35
                  347              23,340                   0                   3           7,036,643       2,764,290,826                   0              23,697   0x2c95/0x31498
               23,408               9,116                   0                   3          10,098,528       2,757,254,183                   0              32,524   0x2ca4/0x49578
               45,631              27,183                   0                   3          19,972,142       2,747,155,655                   0              72,814   0x2cad/0x428a6
                    0                   0                   0                   1             963,802                  69             963,733                   0   0x2cad/0x12dd80
                  276              19,066                   0                   4          10,705,056           5,926,693           4,778,363              19,342   0x2cc4/0x1d9fa0
                    0              25,292                   0                   2          10,568,825           7,122,257           3,446,568              25,292   0x2ccf/0x264d99
                  722              25,106                   0                   3           9,376,076       2,704,945,830                   0              26,568   0x2cd3/0x1cc865
                   15                  96                   0                   6             601,987              65,907             536,080                 111   0x2cd3/0x25f7e8
               50,662              51,942                   0                   4          29,283,798       2,694,967,767                   0             102,611   0x2d20/0x1b0a3e
               18,486              16,140                   0                   4          12,291,893       2,665,683,969                   0              34,626   0x2d3a/0x3e02f3
               67,481             104,443                   0                   5          48,233,597          15,261,303          32,972,294             171,924   0x2d79/0x230ef0
               19,276              34,062                   0                   3          18,103,592       2,605,158,479                   0             171,476   0x2deb/0x61f18
                    0                 225                   0                   6             607,824              37,538             570,286                 225   0x2deb/0xf6568
               14,038              14,130                   0                   3          13,699,081       2,586,447,063                   0              28,385   0x2e21/0x47d6f1
                4,147              13,212                   0                   3           5,363,836       2,572,747,982                   0              17,359   0x2e40/0x118ed
                    0                   0                   0                   4             133,826              37,921              95,905                   0   0x2e40/0x323af
               31,744              33,396                   0                   3          22,247,625       2,567,250,320                   0              65,140   0x2e45/0x256f78
                    9                   0                   0                   6             290,601              55,855             234,746                   9   0x2e45/0x29dea1
                   16              25,020                   0                   4          10,475,420       2,544,712,094                   0              25,039   0x2e5b/0x311fbd
                1,682              19,008                   0                   3           8,507,134       2,534,236,674                   0              20,690   0x2e5f/0x1a583b
                   12                   0                   0                   6             354,107              41,047             313,060                  12   0x2e5f/0x1fbf76
                6,849              54,863                   0                   3          20,176,980       2,525,375,433                   0              61,719   0x2e7d/0x2272ca
               30,012              67,588                   0                   5          28,465,075          27,606,214             858,861              97,600   0x2eb6/0xb06fd
               14,048              16,912                   0                   2           9,587,676       2,476,733,378                   0              66,881   0x2ee1/0x4bc59
               15,183              12,704                   0                   6          11,270,153           6,834,792           4,435,361              27,887   0x2ef4/0x181de2
               23,012              39,873                   0                   7          18,610,316       2,455,875,549                   0              62,948   0x2f1a/0x2e96e
               25,472              49,151                   0                   6          22,681,935       2,437,265,233                   0              74,623   0x2f5a/0x2bd5bd
                1,031              60,319                   0                   6          20,425,359       2,414,583,298                   0              61,350   0x2f66/0x32534c
               48,185              83,225                   0                   4          42,656,335          28,813,330          13,843,005             131,410   0x2f9d/0x5adb9b
               16,859              32,139                   0                   3          18,804,637          12,310,056           6,494,581              74,600   0x2fa3/0x489e38
               29,050              79,391                   0                   8          47,784,742          37,879,192           9,905,550             108,511   0x300a/0x26d0de
                   32              38,306                   0                   6          19,991,849          14,244,764           5,747,085             117,588   0x3026/0x26b107
                    0                  37                   0                   6             345,851              39,476             306,375              25,114   0x3026/0x2bf802
               17,725              34,199                   0                   3          19,155,357       2,264,574,525                   0              51,955   0x303f/0x1f149f
                1,379              23,909                   0                   6          10,787,679       2,245,419,168                   0              25,288   0x304f/0x2b197e
                    3                  12                   0                   5             297,677              34,135             263,542                  15   0x304f/0x2fa44b
               17,968              47,908                   0                   2          17,014,500       2,234,333,812                   0              65,891   0x306c/0x2162f
               45,151              19,691                   0                   3          20,592,703          18,998,392           1,594,311              64,842   0x3071/0xb216e
               18,246              13,219                   0                   3          14,900,440       2,196,726,609                   0              31,514   0x308d/0x55e7c6
                   26              12,008                   0                   5           7,554,365           4,234,132           3,320,233              12,034   0x30a1/0x309683
               35,556              69,762                   0                  20          31,161,221       2,174,271,804                   0             105,338   0x30c2/0x424e88
               33,203             114,692                   0                   5          49,168,609          43,804,782           5,363,827             147,895   0x30fd/0x359ee9
                  182              19,563                   0                   7          10,363,229       2,093,941,974                   0              87,398   0x310c/0x3b29c6
                   14              19,817                   0                   4           9,355,037           4,743,642           4,611,395              19,831   0x3110/0x315263
                   46              37,708                   0                   8          20,678,814          10,884,622           9,794,192              41,674   0x315c/0x3bae01
                   88                 279                   0                   5             761,100              44,429             716,671               3,390   0x315c/0x474b0d
               40,709              61,572                   0                   4          26,837,961       2,052,783,794                   0             102,607   0x3189/0x170b56
                   42                   0                   0                   6             814,957              77,642             737,315                  42   0x3189/0x237ac3
               69,126             107,504                   0                   5          52,094,116          47,372,898           4,721,218             176,645   0x31b8/0x436ee7
                   46                   0                   0                   5             393,197              51,476             341,721              82,129   0x31b8/0x496ed4
               48,348             117,823                   0                   3          48,879,763          41,965,088           6,914,675             166,196   0x3201/0x3856e7
               16,349              39,063                   0                   7          17,875,388          14,606,658           3,268,730              62,982   0x3208/0x17eba3
                  105                   0                   0                   6           3,394,912              60,919           3,333,993               2,571   0x3208/0x4bb903
                  217              17,310                   0                   7           5,776,110       1,902,493,500                   0              17,617   0x3210/0xb4571
               36,657              27,268                   0                   7          22,160,771           4,617,088          17,543,683              63,925   0x3224/0x2c3df4
               24,153              42,022                   0                   1          22,019,603       1,874,556,619                   0             114,113   0x3235/0x4b0f07
                   16              18,174                   0                   7           7,706,321       1,852,537,016                   0              18,190   0x324e/0x280f58
                  120              24,379                   0                   7          10,458,169       1,844,830,695                   0              24,499   0x3266/0x2f0d11
               14,236              34,916                   0                   3          17,998,326          12,723,415           5,274,911              49,152   0x327b/0x108207
                   79                   0                   0                   5             673,666              37,699             635,967               2,808   0x327b/0x1ac989
                6,624              22,958                   0                   3          11,999,224       1,815,700,534                   0              29,650   0x3290/0x394b01
               19,965              40,810                   0                   8          17,269,398       1,803,701,310                   0              60,775   0x32a7/0xfa097
                7,517              58,175                   0                   5          21,772,528       1,786,431,912                   0              65,692   0x32de/0x2aac87
                9,415              18,600                   0                   6          10,786,946       1,764,659,384                   0              28,015   0x32e1/0x36ae89
                   51                   0                   0                   6             622,591              52,177             570,414                  51   0x32e1/0x402e88
               30,139              66,899                   0                   4          30,497,693       1,753,249,847                   0              97,065   0x3315/0x47c6a5
                    6              41,284                   0                   1          16,342,999          12,394,744           3,948,255              41,290   0x3320/0xff97c
               21,091              17,059                   0                   6          11,525,646       1,706,409,155                   0              38,984   0x332b/0x27410a
                    0                  94                   0                   4             892,946              34,897             858,049                  94   0x332b/0x34e11c
                    3              52,035                   0                   3          18,160,444       1,693,990,563                   0              52,132   0x336d/0x18cf58
                   58              27,035                   0                   6           8,947,956       1,675,830,119                   0              27,093   0x3376/0x8c1cc
                    3                  66                   0                   4             501,044              49,007             452,037                  69   0x3376/0x106700
               27,134              44,130                   0                   4          22,994,532       1,666,381,119                   0              71,330   0x339f/0x3e1864
                    0                 108                   0                   4             439,894              35,044             404,850                 108   0x339f/0x44ceba
                   17              51,506                   0                   3          20,284,195       1,642,946,693                   0              51,631   0x33a6/0x4924dd
                   28              16,826                   0                   2           9,954,053       1,622,662,498                   0              16,854   0x33b6/0x487162
                2,079              43,231                   0                   2          20,850,856          13,682,453           7,168,403              45,310   0x33d2/0x556d0a
                    2              30,027                   0                   3          14,418,091           8,924,719           5,493,372              30,743   0x33d9/0x40b5
                    0               1,157                   0                   6           5,510,215             155,394           5,354,821               3,103   0x33d9/0x5454fc
               31,517              31,808                   0                   3          18,110,480       1,571,929,283                   0              64,085   0x33ee/0x37800c
               30,598              27,193                   0                   4          18,121,300          12,201,650           5,919,650              57,791   0x3402/0x1ad560
               38,905              34,440                   0                   4          18,443,706          15,100,893           3,342,813              78,249   0x340c/0x3161a
               51,887              95,209                   0                   4          40,262,214          29,791,799          10,470,415             152,283   0x343b/0x71660
                    1              28,686                   0                   7          10,081,465       1,476,991,583                   0             142,625   0x344c/0x85499
               28,401              13,707                   0                   3          10,169,877       1,466,910,118                   0              42,108   0x3457/0xaec2e
                    5              16,531                   0                   6          14,562,193           4,402,715          10,159,478              16,536   0x345b/0x50893f
                    0              15,920                   0                   7           9,514,322       1,442,178,048                   0              19,604   0x3460/0x492011
                    4              16,264                   0                   6          10,131,895       1,432,663,726                   0              16,268   0x3465/0x4b2348
               30,231              28,986                   0                   4          18,938,103          14,093,138           4,844,965              59,217   0x347c/0x3aef3f
                   14                   0                   0                   5             512,006              34,432             477,574                 120   0x347c/0x42bf45
               20,678              38,294                   0                   3          17,101,301       1,403,081,722                   0              58,975   0x3497/0x16843a
                9,869              25,099                   0                   5          12,548,707       1,385,980,421                   0              34,968   0x34b3/0x3d681d
               36,465              60,605                   0                   5          26,361,794          24,558,695           1,803,099              97,070   0x34d6/0x5e45f
                   36              23,345                   0                   3           9,621,974           6,699,792           2,922,182              23,911   0x34d9/0x1fb5
               25,436              49,579                   0                   7          20,880,498          18,608,524           2,271,974              75,850   0x34df/0xd8f27
                   32               1,485                   0                   6           4,197,555             413,166           3,784,389               1,535   0x34df/0x4d9bda
               31,535              59,968                   0                   5          25,248,912          24,728,797             520,115              91,849   0x3512/0x51cea
                1,601              24,378                   0                   6          10,268,411           8,200,696           2,067,715              26,160   0x351b/0x93565
               14,147              23,162                   0                   8          11,300,331       1,276,852,570                   0              37,400   0x3525/0x1d0cd0
               14,204              20,662                   0                   3          13,299,117       1,265,552,239                   0              34,866   0x3532/0x4f63fd
               17,514              24,728                   0                   3          16,797,576          12,513,187           4,284,389              42,242   0x3553/0x1e8685
                7,303              46,343                   0                   7          23,094,370          14,762,461           8,331,909              54,344   0x3580/0x4dbde7
               45,620              41,812                   0                   7          27,514,220          21,511,990           6,002,230              90,879   0x359c/0x27cfd3
               16,861              12,552                   0                   3           9,619,073       1,184,846,956                   0             104,424   0x35a0/0x21ffd4
                    0                 271                   0                   6           3,731,928              93,043           3,638,885                 271   0x35a0/0x5af1ac
               30,860              48,187                   0                   7          26,338,978          24,109,627           2,229,351              79,097   0x35ab/0x2314ce
               39,031              24,255                   0                   4          20,202,284       1,145,156,977                   0              63,301   0x35c4/0x262afa
                2,576              46,002                   0                   6          19,184,048           5,580,647          13,603,401              48,578   0x35dc/0x19b7aa
                    2                   0                   0                   6             174,028              33,089             140,939              31,536   0x35dc/0x1c5f76
              111,528              77,161                   0                   7          62,341,581          50,549,489          11,792,092             188,690   0x360c/0x401a43
               28,697              25,910                   0                   6          18,246,350          13,855,848           4,390,502              56,636   0x3618/0x255811
                4,335              12,127                   0                   6           9,004,926           5,130,460           3,874,466              18,985   0x3621/0x16290f
               18,413               8,607                   0                   5           8,878,516           7,701,363           1,177,153              27,027   0x3626/0x50c43
                    0               2,910                   0                   6           6,228,723             848,601           5,380,122               2,910   0x3626/0x641736
               28,784              22,689                   0                   3          21,119,867          14,626,185           6,493,682              52,178   0x362a/0x752db1
               41,676              62,528                   0                   4          34,112,988          29,303,765           4,809,223             105,238   0x363e/0x1b598d
                   17                 735                   0                   5          12,521,216              31,105          12,490,111               1,300   0x3643/0x41d20d
                7,802              34,494                   0                   7          33,127,169           7,514,191          25,612,978              43,003   0x3659/0x71898e
                    0               8,734                   0                   5           3,377,408         900,015,281                   0              25,130   0x365c/0xc7c0e
                  114               9,858                   0                   7          15,782,161             650,578          15,131,583               9,972   0x365f/0x64b69f
                    1              11,516                   0                   7           5,378,418         880,855,712                   0              19,548   0x3664/0x1e3191
                    7              22,005                   0                   7           8,922,863           6,691,187           2,231,676              22,012   0x3668/0xdc200
                   13              13,409                   0                   6          15,952,421             360,887          15,591,534              14,391   0x3671/0x6895a5
                1,749               8,568                   0                   3           3,660,457         850,602,010                   0              22,698   0x3676/0x7d9ce
                5,390              10,146                   0                   6          14,882,555           1,855,609          13,026,946              15,536   0x3688/0x525a49
               10,230               9,513                   0                   5           4,816,061         832,058,998                   0              28,465   0x3691/0x34086
               14,698              31,907                   0                   7          41,080,361          36,925,183           4,155,178              46,605   0x36aa/0x13bcaf
                    2               1,755                   0                   6           4,666,159             345,619           4,320,540              47,630   0x36aa/0x5aefde
               25,935              30,385                   0                   5          38,398,767         781,496,417                   0              57,119   0x36c5/0x42810d
                9,478              14,696                   0                   7          21,102,013         743,097,650                   0              24,174   0x36e1/0x5351ca
                   65              12,746                   0                   6           5,610,587           3,858,013           1,752,574              12,811   0x36ed/0x1057a5
                5,806               9,341                   0                   6          10,353,968           2,690,013           7,663,955              15,343   0x36fc/0x15be55
                    5                   0                   0                   6           4,894,393              33,744           4,860,649               5,931   0x36fc/0x606d0e
               28,754              58,230                   0                  18          28,401,618         701,136,689                   0              86,988   0x3708/0x480960
                    0              15,897                   0                   6          10,890,931           3,098,266           7,792,665              15,897   0x370a/0x55a193
                  126              10,927                   0                   6          26,068,972           2,214,107          23,854,865              16,100   0x372d/0x19a5ff
                9,837              16,155                   0                   6          11,096,409           6,346,816           4,749,593              29,696   0x3732/0x2a60d8
                  128              15,250                   0                   6           8,591,606           3,747,801           4,843,805              16,046   0x3734/0x14e34e
                  123              28,336                   0                   6          16,047,724           5,161,792          10,885,932              31,737   0x3736/0x712b3a
                   24               8,534                   0                   5           3,423,605           1,552,888           1,870,717              19,982   0x3740/0xcd22f
                  533                   0                   0                   6          20,325,930             275,776          20,050,154               3,884   0x3742/0x11cb59
                    0                   0                   0                   4              72,242              36,230              36,012                   0   0x3742/0x12e58b
               26,449              16,768                   0                   3          20,777,536         576,217,652                   0              43,217   0x3756/0x1ec2cb
               11,493              23,349                   0                   3          21,836,979          13,435,005           8,401,974              34,842   0x375c/0x3aca7e
                  309              32,239                   0                   6          17,558,766           9,588,731           7,970,035              56,790   0x3761/0x158a6c
                   12              26,562                   0                   7          12,407,639           6,425,279           5,982,360              27,257   0x3779/0x3a4743
                6,845              41,908                   0                   7          33,072,537          22,775,381          10,297,156              51,339   0x379b/0x69295c
                6,778              29,990                   0                   6          25,664,453          17,110,934           8,553,519              41,294   0x37a8/0x2701a1
                    5              16,509                   0                   9          19,648,397           8,979,547          10,668,850              44,387   0x37ae/0x21a42e
              116,569             137,423                   0                  12         129,250,694          72,386,145          56,864,549             267,033   0x37e2/0x163534
                   13              17,546                   0                   6          28,042,727              34,685          28,008,042             187,464   0x37e7/0x90ee1b
               33,326              30,872                   0                   6          43,125,925           9,805,827          33,320,098              81,756   0x37ec/0x280a40
                    3               5,332                   0                   5          19,873,538              37,653          19,835,885              30,103   0x37ef/0x261c42
                   17               2,306                   0                   6          19,166,423             244,987          18,921,436               7,657   0x37f1/0x196419
               10,753              20,277                   0                   6          40,648,779          11,079,498          29,569,281              32,618   0x37f6/0x234a64
                   10              22,876                   0                   6          18,391,375             164,634          18,226,741              49,034   0x37f8/0xabeb3
                   30              34,377                   0                   7          32,006,385           1,775,081          30,231,304              56,545   0x37fb/0x295c24
                    4              23,090                   0                   7          38,624,657           2,288,459          36,336,198              52,101   0x37ff/0x145fb5
                    6               8,689                   0                   6          27,108,848             949,959          26,158,889              24,449   0x3801/0x80d8a5
                    6               5,117                   0                   7          18,644,436              55,620          18,588,816              11,221   0x3803/0x6c2979
                  837                   0                   0                   6           6,661,670             288,422           6,373,248               5,956   0x3804/0x39391f
                5,863               6,917                   0                   0           3,705,888           3,705,888                   0              12,780   0x3804/0x71c53f
 </DbPrintLog>

* This prints some statistics about the database. In particular the two rows called key bytes and data bytes are relevant. For the following step you need the values from the avg column - i.e. 38 and 219 in this case. You also need the value of “Total number of entries” (14543762 in this case). Now we take those values and put them into this command:

 [root@dc042 meta]# java -cp  /usr/share/dcache/classes/je-*.jar com.sleepycat.je.util.DbCacheSize -records 14543762 -key 38 -data 219
 
 === Environment Cache Overhead ===
 
 3,157,213 minimum bytes
 
 To account for JE daemon operation and record locks,
 a significantly larger amount is needed in practice.
 
 === Database Cache Size ===
 
    Number of Bytes  Description
    ---------------  -----------
      1,195,759,832  Internal nodes only
      5,124,232,392  Internal nodes and leaf nodes

* For further information see the DbCacheSize javadoc.

* For this pool it tells us that we need a bit above 1195 MB to keep all the internal btree nodes cached. Oracle’s recommendation is that if the database is updated often, the cache is at least big enough to contain the internal nodes. Obviously you want to make it a bit bigger to leave room for it to grow. One could take this and configure the cache using the je.maxMemory setting. If the size is lower than the 20% of the max heap size you already use, then you don’t need to do anything (I do not suggest lowering it further).

* Since this pool has 6000m of JAVA heap max space, it means that 20% = 1200MB, which is exactly cache for Internal btree nodes. So we will not touch this.

* There is one caveat though: If you increase the cache size, less free space is left on the heap. If the pool is pushed to the limit, this may actually slow down the pool as garbage collection overhead increases. You need to ensure that enough space is left in addition to the cache (possibly by increasing the max heap size). An alternative to adjusting je.maxMemory is of course to adjust the max heap size. If you make it large enough so that 20% is enough to cache the internal btree nodes, then all is well. This may however mean you assign significantly more memory to the pool than it really needs. Yeah, there are lots of things to consider.

* A little bonus: There is also this command:

 [root@dc042 meta]#  java -cp  /usr/share/dcache/classes/je-*.jar com.sleepycat.je.util.DbSpace -h .
   File    Size (KB)  % Used
 --------  ---------  ------
 00001857       9765      41
 00001f2a       9765      43
 00001f2f       9765      39
 00001f72       9765      44
 0000216d       9765      40
 000021cb       9765      40
 0000226b       9765      45
 000022e8       9763      38
 000022fa       9765      39
 00002390       9765      38
 0000241d       9765      49
 0000252c       9765      51
 0000254c       9765      47
 00002567       9765      39
 000025b7       9765      42
 000025db       9765      42
 000025e5       9765      56
 0000265f       9765      39
 00002678       9765      61
 0000268d       9762      48
 00002692       9765      74
 000026be       9765      48
 000026cf       9765      40
 00002700       9765      41
 00002706       9765      38
 00002718       9765      56
 00002747       9765      44
 0000274d       9765      48
 000027db       9765      62
 00002855       9765      39
 0000285c       9765      39
 00002882       9765      41
 0000289d       9765      54
 000028ba       9765      52
 000028c1       9765      53
 000028d0       9765      41
 000028d7       9765      41
 000028e7       9765      39
 000028ed       9765      39
 00002926       9765      39
 0000292b       9765      43
 00002945       9765      41
 00002952       9765      41
 00002958       9765      62
 00002959       9764      38
 00002962       9765      53
 00002963       9765      47
 0000297d       9765      51
 0000297e       9765      41
 000029b3       9765      45
 000029c7       9765      56
 000029ce       9765      42
 000029d3       9765      43
 000029d9       9765      54
 000029fb       9765      42
 00002a00       9765      54
 00002a01       9765      39
 00002a06       9765      39
 00002a20       9765      45
 00002a26       9765      61
 00002a2b       9765      69
 00002a30       9765      65
 00002a44       9765      46
 00002a4f       9765      50
 00002a53       9765      64
 00002a54       9765      47
 00002a58       9765      66
 00002a59       9765      43
 00002a5e       9765      56
 00002a63       9765      44
 00002a69       9765      47
 00002a8f       9765      39
 00002aac       9765      54
 00002aba       9765      42
 00002abb       9765      44
 00002ae0       9763      38
 00002af3       9765      45
 00002b01       9765      41
 00002b08       9765      65
 00002b0e       9765      41
 00002b24       9765      45
 00002b28       9765      61
 00002b29       9765      41
 00002b3c       9765      40
 00002b3d       9765      40
 00002b41       9765      47
 00002b4c       9765      46
 00002b52       9765      57
 00002b64       9765      38
 00002b87       9765      38
 00002b8e       9765      47
 00002b8f       9760      38
 00002bad       9765      42
 00002bbf       9765      44
 00002bc6       9765      38
 00002be9       9765      47
 00002c00       9765      42
 00002c06       9765      69
 00002c0c       9765      55
 00002c13       9765      38
 00002c14       9765      38
 00002c2a       9765      41
 00002c2b       9765      41
 00002c30       9765      54
 00002c4c       9765      39
 00002c53       9765      45
 00002c5f       9765      44
 00002c7c       9765      38
 00002c8f       9765      49
 00002c95       9765      38
 00002ca4       9764      50
 00002ca9       9765      42
 00002cad       9763      41
 00002cc4       9765      39
 00002ccf       9765      43
 00002cd3       9765      45
 00002cf3       9765      48
 00002d19       9765      38
 00002d20       9765      38
 00002d3a       9765      40
 00002d3b       9765      41
 00002d41       9765      43
 00002d51       9765      39
 00002d59       9765      40
 00002d79       9765      62
 00002d8f       9765      39
 00002deb       9765      40
 00002e21       9765      39
 00002e40       9765      41
 00002e41       9765      45
 00002e45       9765      40
 00002e5b       9765      58
 00002e5f       9765      49
 00002e60       9765      67
 00002e7d       9765      48
 00002e95       9765      38
 00002eb1       9762      57
 00002eb6       9765      53
 00002ee1       9765      38
 00002ef4       9763      43
 00002ef8       9765      41
 00002f1a       9765      53
 00002f25       9765      65
 00002f5a       9765      52
 00002f5f       9765      63
 00002f66       9765      41
 00002f71       9765      42
 00002f79       9765      61
 00002f87       9765      49
 00002f9d       9765      41
 00002f9e       9765      42
 00002fa3       9765      38
 00002fa4       9765      46
 00002fb1       9762      55
 00002fc1       9765      41
 00002fde       9765      38
 0000300a       9765      43
 0000300e       9762      47
 00003026       9765      55
 00003027       9765      57
 0000303f       9765      63
 0000304f       9765      48
 00003056       9765      44
 0000306c       9765      48
 0000306d       9765      42
 00003071       9765      45
 0000308d       9765      39
 000030a1       9765      54
 000030a5       9765      51
 000030ab       9765      60
 000030c2       9765      46
 000030d5       9765      43
 000030dd       9765      38
 000030e3       9765      57
 000030e4       9765      55
 000030fd       9765      48
 0000310c       9764      42
 00003110       9765      46
 00003119       9765      47
 0000315c       9765      48
 0000315d       9765      54
 00003185       9765      56
 00003189       9765      52
 0000318a       9765      58
 0000319a       9765      46
 0000319f       9764      39
 000031b5       9764      53
 000031b8       9765      40
 000031b9       9765      74
 000031e2       9765      65
 000031fa       9765      58
 000031fb       9765      52
 00003201       9765      50
 00003202       9764      44
 00003208       9765      42
 00003210       9763      40
 00003218       9765      40
 00003224       9765      51
 00003229       9765      48
 00003235       9765      40
 0000324e       9765      46
 00003266       9765      41
 0000326c       9764      47
 0000327b       9765      55
 00003290       9765      48
 00003294       9765      56
 000032a7       9765      69
 000032ad       9765      62
 000032de       9765      47
 000032e1       9765      46
 000032e2       9765      71
 000032e9       9765      45
 00003315       9765      42
 00003316       9764      54
 00003320       9765      43
 0000332b       9765      52
 0000332c       9765      76
 0000336d       9765      69
 00003376       9765      72
 00003377       9765      46
 0000339f       9765      43
 000033a0       9765      75
 000033a6       9765      46
 000033b6       9765      40
 000033c0       9764      72
 000033d2       9765      38
 000033d3       9764      45
 000033d9       9765      42
 000033da       9765      44
 000033ee       9765      41
 000033fb       9764      38
 00003402       9765      42
 00003403       9765      53
 0000340c       9765      76
 00003428       9765      67
 00003431       9765      51
 00003436       9765      88
 0000343b       9765      85
 0000344c       9765      48
 00003457       9765      46
 0000345b       9765      46
 00003460       9765      50
 00003465       9765      45
 00003466       9762      40
 0000347c       9765      54
 0000347d       9765      61
 00003497       9765      63
 000034b3       9765      64
 000034bd       9765      54
 000034c9       9764      71
 000034d6       9765      65
 000034d9       9765      86
 000034da       9765      46
 000034df       9765      51
 000034e0       9764      60
 00003504       9762      80
 00003512       9761      76
 0000351b       9765      65
 00003525       9765      59
 00003532       9765      52
 00003540       9764      43
 00003553       9765      80
 00003558       9765      54
 00003580       9762      49
 00003586       9765      41
 00003587       9763      50
 0000359c       9765      40
 000035a0       9765      46
 000035a1       9765      37
 000035a6       9765      96
 000035ab       9765      52
 000035ac       9765      39
 000035c4       9765      61
 000035d0       9765      89
 000035dc       9765      74
 000035dd       9765      43
 000035f1       9765      43
 000035fd       9765      37
 00003604       9765      47
 00003609       9764      40
 0000360c       9765      59
 0000360d       9765      37
 00003618       9763      42
 00003621       9765      42
 00003626       9765      42
 00003627       9765      41
 0000362a       9765      38
 0000362b       9765      73
 00003630       9765      68
 00003631       9764      42
 0000363e       9763      39
 00003643       9765      43
 00003646       9765      38
 0000364c       9762      40
 00003659       9765      47
 0000365c       9764      44
 0000365f       9765      45
 00003664       9765      69
 00003668       9764      37
 00003671       9765      43
 00003676       9765      39
 00003688       9765      37
 00003691       9765      48
 00003692       9764      37
 00003695       9762      38
 000036a3       9764      50
 000036aa       9765      47
 000036ab       9765      49
 000036c2       9765      64
 000036c3       9764      57
 000036c5       9765      40
 000036da       9764      40
 000036e1       9765      43
 000036ed       9762      39
 000036fc       9765      41
 000036fd       9765      59
 00003701       9764      85
 00003708       9765      43
 0000370a       9763      49
 00003728       9762      41
 0000372a       9763      55
 0000372d       9765      51
 00003732       9763      48
 00003734       9765      60
 00003736       9765      42
 00003740       9765      48
 00003741       9765      39
 00003742       9765      38
 00003749       9764      40
 00003756       9763      39
 00003758       9762      46
 0000375c       9765      67
 0000375d       9765      37
 00003761       9765      63
 00003779       9765      55
 00003785       9765      43
 00003798       9763      60
 0000379b       9765      41
 0000379e       9765      37
 000037a5       9764      39
 000037a8       9764      48
 000037ab       9765      38
 000037ae       9765      54
 000037b3       9764      42
 000037b6       9761      36
 000037be       9763      46
 000037c1       9765      69
 000037c3       9765      36
 000037c4       9764      37
 000037ce       9762      50
 000037d1       9761      45
 000037d3       9765      45
 000037dd       9765      45
 000037df       9765      50
 000037e0       9763      39
 000037e2       9765      13
 000037e3       9763      18
 000037e7       9765      49
 000037e8       9763      41
 000037e9       9765      42
 000037ea       9763      40
 000037eb       9765      39
 000037ec       9761      39
 000037ed       9763      41
 000037ef       9763      36
 000037f0       9764      36
 000037f1       9764      39
 000037f3       9765      41
 000037f4       9761      53
 000037f5       9765      36
 000037f6       9765      54
 000037f7       9763      53
 000037f8       9765      63
 000037f9       9765      60
 000037fa       9765      70
 000037fb       9765      76
 000037fc       9765      72
 000037fd       9765      73
 000037fe       9765      83
 000037ff       9765      87
 00003800       9765      89
 00003801       9762      86
 00003802       9763      88
 00003803       9764      90
 00003804       8110      76
  TOTALS     3757891      49

* It will tell you for each of the database files (the log segments) how big it is and what the utilization is. The Berkeley DB will try to keep the total utilization above 50%, but for a small pool like this one it cannot do it. I figure that you may find this interesting to know after reading about utilization above.

* The log segment size and the utilization goal can be adjusted too, but I cannot give any sound advice on whether that’s a good idea and how to determine good values.

* All of the above is only relevant if you use the Berkeley DB backend for pools.