Difference between revisions of "DCACHE CHECK POOL BTREE MEMORY"
Jump to navigation
Jump to search
(Created page with "= For the MoD = * Open a ticket to incidents@support.pic.es with the alarm. * This alarm only affects the duration of the pool start up, nothing has to be done until the next...") |
|||
| (2 intermediate revisions by the same user not shown) | |||
| Line 702: | Line 702: | ||
* All of the above is only relevant if you use the Berkeley DB backend for pools. | * All of the above is only relevant if you use the Berkeley DB backend for pools. | ||
| + | |||
| + | == Special thanks == | ||
| + | * This documentation has been deployed thanks to Gerd Behrmann, you can find more in at: https://gist.github.com/gbehrmann/df1abbe45c03356eff2e | ||
| + | * Read more about this at: | ||
| + | :* https://backstage.forgerock.com/knowledge/kb/book/b74223337/a49979000 | ||
| + | :* http://docs.oracle.com/cd/E17277_02/html/java/com/sleepycat/je/EnvironmentConfig.html | ||
Latest revision as of 13:29, 6 October 2017
For the MoD
- Open a ticket to incidents@support.pic.es with the alarm.
- This alarm only affects the duration of the pool start up, nothing has to be done until the next working day where this can be fixed by dCache Administrators.
- Same information as DCACHE CHECK POOL METADATA as both alarms are strictly related.
For the Admin
- Berkely DB should be tunned according to the number of meta files that a pool contains. If this not correctly adapted, pool start up will may need several hours to be fully available. Gerd explained how to tune up a pool according to this number. Below I show a modified explanation of what he said, adapting the explanation to the ATLAS pools.
- The default je.log.fileCacheSize is 100. This could be increased by creating a file je.properties and adding the 'je.log.fileCacheSize=100'.
- Also, other relevant settings are:
je.maxMemory= je.maxMemoryPercent=20
- These are equivalent. The former sets the number of bytes used for the btree cache, while the latter defines it as a percentage of the max heap size. E.g. if dcache.java.memory.heap is set to 2048m, then 410 MB is used for the btree cache.
- The question is then how to determine a good size for the cache. There are some hidden utilities you can use to do this.
- Below I show an example for dc042_1:
[root@dc042 meta]# java -cp /usr/share/dcache/classes/je-6.3.8.jar com.sleepycat.je.util.DbPrintLog -h . -S
<DbPrintLog>
Log statistics:
type total provisional total min max avg entries
count count bytes bytes bytes bytes as % of log
MapLN 1,324 0 12,617,573 76 11,985 9,529 0.3
NameLN 4 0 141 31 38 35 0
FileSummaryLN 231,140 0 55,644,337 24 49,701 240 1.4
IN 84,083 1,080 296,942,059 45 5,709 3,531 7.7
BIN 114,865 114,865 377,569,148 44 6,675 3,287 9.8
DbTree 318 0 1,034,997 2,680 4,136 3,254 0
Commit 3,376,012 0 114,784,408 34 34 34 3
CkptStart 158 0 4,899 30 33 31 0
CkptEnd 226 0 22,560 67 186 99 0
Trace 2,506 0 525,597 48 402 209 0
FileHeader 385 0 14,630 38 38 38 0
DEL_LN_TX 748,055 0 51,409,527 57 69 68 1.3
INS_LN_TX 619,747 0 192,298,301 96 1,443 310 5
UPD_LN_TX 2,008,226 0 398,854,718 32 1,919 198 10.4
UPD_LN 6,636,173 0 1,997,042,458 18 1,819 300 51.9
NewBINDelta 720,540 720,540 348,480,254 55 1,782 483 9.1
key bytes 10,012,201 387,254,450 1 61 38 (10.1)
data bytes 9,264,146 2,036,632,128 1 1,848 219 (52.9)
Total bytes in portion of log read: 3,847,245,607
Total number of entries: 14,543,762
Per checkpoint interval info:
lnTxn ln mapLNTxn mapLN end to end end to start start to end maxLNReplay ckptEnd
273 63,082 0 10 30,455,407 3,857,456,063 0 63,355 0x1f2f/0x6f2ef
5,168 130,892 0 24 39,782,369 3,827,000,656 0 136,060 0x226b/0x3a0d0
27,098 132,905 0 15 50,120,541 16,509,395 33,611,146 160,003 0x252c/0x577ad
128 32,773 0 3 9,904,160 3,737,097,746 0 134,154 0x254c/0x4014d
82 31,341 0 12 10,576,583 3,727,193,586 0 31,423 0x2567/0xccd94
0 72 0 4 711,899 51,780 660,119 72 0x2567/0x17aa6f
36,445 61,509 0 0 28,499,838 3,715,905,104 0 97,954 0x25e5/0xc66d
2 108 0 6 1,564,919 56,871 1,508,048 110 0x25e5/0x18a764
10,732 50,299 0 5 18,877,564 3,685,840,347 0 61,055 0x2678/0x786e0
33 190 0 6 1,627,301 94,279 1,533,022 223 0x2678/0x205b85
8,277 84,136 0 3 28,217,133 17,808,699 10,408,434 92,434 0x26be/0x52732
5,401 25,394 0 7 9,824,151 3,637,118,349 0 61,029 0x26cf/0x27849
49,026 92,572 0 3 41,592,807 3,627,294,198 0 141,598 0x2747/0x1ac630
2 46 0 6 357,170 45,107 312,063 48 0x2747/0x203962
20,496 39,154 0 3 18,591,447 3,585,344,221 0 59,654 0x27db/0xabb39
0 154 0 6 1,187,463 83,676 1,103,787 154 0x27db/0x1cd9c0
17,107 49,468 0 3 18,905,722 3,565,565,311 0 66,583 0x285c/0xc273a
58 0 0 6 392,501 62,920 329,581 58 0x285c/0x12246f
113,890 201,969 0 17 79,488,418 3,546,267,088 0 315,874 0x28ed/0xa5611
22,147 98,579 0 8 40,657,515 36,319,235 4,338,280 120,726 0x2952/0x145e7c
0 122 0 6 1,296,043 70,361 1,225,682 28,367 0x2952/0x282527
19,570 65,944 0 5 27,570,660 25,251,492 2,319,168 85,520 0x2962/0x3138b
37,714 28,889 0 3 21,505,514 19,337,884 2,167,630 66,613 0x297d/0x1a0c75
3 332 0 6 2,437,155 124,630 2,312,525 479 0x297d/0x3f3c98
41,458 62,406 0 3 26,385,897 3,373,311,783 0 103,904 0x29c7/0x81701
0 46 0 6 1,404,642 46,045 1,358,597 46 0x29c7/0x1d85e3
41,443 87,747 0 3 39,909,271 14,624,816 25,284,455 129,198 0x29fb/0x1c237a
0 123 0 5 1,185,318 68,652 1,116,666 79,470 0x29fb/0x2e39a0
9,199 12,698 0 4 9,927,451 3,304,426,655 0 21,899 0x2a00/0x2d1e3b
8 0 0 5 118,374 35,364 83,010 8 0x2a00/0x2eeca1
122,697 96,787 0 3 66,966,017 3,294,380,830 0 219,488 0x2a44/0xa122
53,436 12,002 0 3 21,830,901 3,227,414,813 0 65,438 0x2a53/0x1c9117
20 167 0 6 1,308,679 85,964 1,222,715 187 0x2a53/0x30891e
18,552 38,805 0 6 17,501,977 3,204,275,233 0 57,381 0x2a58/0xa6b37
9 105 0 4 678,297 87,871 590,426 114 0x2a58/0x14c4d0
25,434 36,231 0 10 19,064,711 3,186,094,959 0 61,675 0x2a5e/0x67f57
15 159 0 6 1,467,746 83,228 1,384,518 174 0x2a5e/0x1ce4b9
12,236 14,099 0 3 8,375,785 3,165,562,502 0 26,358 0x2a63/0x41c22
6 184 0 6 1,058,988 89,948 969,040 190 0x2a63/0x1444ce
71,844 24,018 0 3 29,188,061 27,937,328 1,250,733 95,870 0x2aac/0x7e12b
9,407 25,608 0 7 11,016,096 3,126,939,668 0 35,029 0x2aba/0x17624b
20 110 0 6 748,510 82,108 666,402 130 0x2aba/0x22ce29
64,997 49,072 0 15 27,822,548 27,438,951 383,597 114,142 0x2af3/0x1947d
10,534 23,197 0 6 12,103,837 3,087,352,514 0 33,737 0x2b01/0x21ae9a
122 0 0 32 879,716 489,740 389,976 122 0x2b01/0x2f1afe
60,029 128,431 0 24 58,556,679 56,338,979 2,217,700 188,485 0x2b3c/0x191505
15 52 0 6 530,374 53,795 476,579 231 0x2b3c/0x212ccb
48,799 11,045 0 4 18,178,427 17,837,860 340,567 59,855 0x2b41/0x56146
45,998 29,564 0 3 20,154,236 2,997,103,481 0 75,573 0x2b52/0x7bbc2
39,570 60,757 0 15 30,273,541 2,976,949,245 0 100,327 0x2b8e/0xbe847
31,631 33,780 0 13 22,348,448 16,815,306 5,533,142 65,411 0x2bad/0x2fbde7
16,831 40,178 0 5 20,571,223 2,924,327,256 0 57,047 0x2bc6/0x38753e
4,292 18,826 0 8 8,363,076 2,903,756,033 0 23,118 0x2be9/0x1f7b02
0 254 0 6 2,043,974 119,900 1,924,074 254 0x2be9/0x3eab48
30 19,902 0 4 7,727,051 2,893,348,983 0 19,954 0x2c00/0x1bfc93
27,984 75,433 0 32 29,896,927 2,885,621,932 0 103,417 0x2c13/0x1a69f2
40,477 36,001 0 10 19,737,838 17,591,303 2,146,535 76,478 0x2c2a/0x1669e0
2 2 0 4 194,668 36,808 157,860 10 0x2c2a/0x19624c
29,246 68,367 0 3 31,325,183 2,835,792,499 0 97,615 0x2c4c/0x2d9acb
0 26 0 5 1,118,302 27,604 1,090,698 26 0x2c4c/0x3eab29
4 19,763 0 3 7,562,665 2,803,349,014 0 19,775 0x2c53/0x197a52
26 194 0 6 1,436,963 103,803 1,333,160 220 0x2c53/0x2f6775
6,255 14,988 0 8 9,972,102 2,794,349,386 0 21,292 0x2c5f/0x2efa7b
39,212 18,305 0 5 18,591,243 2,784,377,284 0 57,517 0x2c8f/0x197b86
0 41 0 6 1,495,215 121,720 1,373,495 41 0x2c8f/0x304c35
347 23,340 0 3 7,036,643 2,764,290,826 0 23,697 0x2c95/0x31498
23,408 9,116 0 3 10,098,528 2,757,254,183 0 32,524 0x2ca4/0x49578
45,631 27,183 0 3 19,972,142 2,747,155,655 0 72,814 0x2cad/0x428a6
0 0 0 1 963,802 69 963,733 0 0x2cad/0x12dd80
276 19,066 0 4 10,705,056 5,926,693 4,778,363 19,342 0x2cc4/0x1d9fa0
0 25,292 0 2 10,568,825 7,122,257 3,446,568 25,292 0x2ccf/0x264d99
722 25,106 0 3 9,376,076 2,704,945,830 0 26,568 0x2cd3/0x1cc865
15 96 0 6 601,987 65,907 536,080 111 0x2cd3/0x25f7e8
50,662 51,942 0 4 29,283,798 2,694,967,767 0 102,611 0x2d20/0x1b0a3e
18,486 16,140 0 4 12,291,893 2,665,683,969 0 34,626 0x2d3a/0x3e02f3
67,481 104,443 0 5 48,233,597 15,261,303 32,972,294 171,924 0x2d79/0x230ef0
19,276 34,062 0 3 18,103,592 2,605,158,479 0 171,476 0x2deb/0x61f18
0 225 0 6 607,824 37,538 570,286 225 0x2deb/0xf6568
14,038 14,130 0 3 13,699,081 2,586,447,063 0 28,385 0x2e21/0x47d6f1
4,147 13,212 0 3 5,363,836 2,572,747,982 0 17,359 0x2e40/0x118ed
0 0 0 4 133,826 37,921 95,905 0 0x2e40/0x323af
31,744 33,396 0 3 22,247,625 2,567,250,320 0 65,140 0x2e45/0x256f78
9 0 0 6 290,601 55,855 234,746 9 0x2e45/0x29dea1
16 25,020 0 4 10,475,420 2,544,712,094 0 25,039 0x2e5b/0x311fbd
1,682 19,008 0 3 8,507,134 2,534,236,674 0 20,690 0x2e5f/0x1a583b
12 0 0 6 354,107 41,047 313,060 12 0x2e5f/0x1fbf76
6,849 54,863 0 3 20,176,980 2,525,375,433 0 61,719 0x2e7d/0x2272ca
30,012 67,588 0 5 28,465,075 27,606,214 858,861 97,600 0x2eb6/0xb06fd
14,048 16,912 0 2 9,587,676 2,476,733,378 0 66,881 0x2ee1/0x4bc59
15,183 12,704 0 6 11,270,153 6,834,792 4,435,361 27,887 0x2ef4/0x181de2
23,012 39,873 0 7 18,610,316 2,455,875,549 0 62,948 0x2f1a/0x2e96e
25,472 49,151 0 6 22,681,935 2,437,265,233 0 74,623 0x2f5a/0x2bd5bd
1,031 60,319 0 6 20,425,359 2,414,583,298 0 61,350 0x2f66/0x32534c
48,185 83,225 0 4 42,656,335 28,813,330 13,843,005 131,410 0x2f9d/0x5adb9b
16,859 32,139 0 3 18,804,637 12,310,056 6,494,581 74,600 0x2fa3/0x489e38
29,050 79,391 0 8 47,784,742 37,879,192 9,905,550 108,511 0x300a/0x26d0de
32 38,306 0 6 19,991,849 14,244,764 5,747,085 117,588 0x3026/0x26b107
0 37 0 6 345,851 39,476 306,375 25,114 0x3026/0x2bf802
17,725 34,199 0 3 19,155,357 2,264,574,525 0 51,955 0x303f/0x1f149f
1,379 23,909 0 6 10,787,679 2,245,419,168 0 25,288 0x304f/0x2b197e
3 12 0 5 297,677 34,135 263,542 15 0x304f/0x2fa44b
17,968 47,908 0 2 17,014,500 2,234,333,812 0 65,891 0x306c/0x2162f
45,151 19,691 0 3 20,592,703 18,998,392 1,594,311 64,842 0x3071/0xb216e
18,246 13,219 0 3 14,900,440 2,196,726,609 0 31,514 0x308d/0x55e7c6
26 12,008 0 5 7,554,365 4,234,132 3,320,233 12,034 0x30a1/0x309683
35,556 69,762 0 20 31,161,221 2,174,271,804 0 105,338 0x30c2/0x424e88
33,203 114,692 0 5 49,168,609 43,804,782 5,363,827 147,895 0x30fd/0x359ee9
182 19,563 0 7 10,363,229 2,093,941,974 0 87,398 0x310c/0x3b29c6
14 19,817 0 4 9,355,037 4,743,642 4,611,395 19,831 0x3110/0x315263
46 37,708 0 8 20,678,814 10,884,622 9,794,192 41,674 0x315c/0x3bae01
88 279 0 5 761,100 44,429 716,671 3,390 0x315c/0x474b0d
40,709 61,572 0 4 26,837,961 2,052,783,794 0 102,607 0x3189/0x170b56
42 0 0 6 814,957 77,642 737,315 42 0x3189/0x237ac3
69,126 107,504 0 5 52,094,116 47,372,898 4,721,218 176,645 0x31b8/0x436ee7
46 0 0 5 393,197 51,476 341,721 82,129 0x31b8/0x496ed4
48,348 117,823 0 3 48,879,763 41,965,088 6,914,675 166,196 0x3201/0x3856e7
16,349 39,063 0 7 17,875,388 14,606,658 3,268,730 62,982 0x3208/0x17eba3
105 0 0 6 3,394,912 60,919 3,333,993 2,571 0x3208/0x4bb903
217 17,310 0 7 5,776,110 1,902,493,500 0 17,617 0x3210/0xb4571
36,657 27,268 0 7 22,160,771 4,617,088 17,543,683 63,925 0x3224/0x2c3df4
24,153 42,022 0 1 22,019,603 1,874,556,619 0 114,113 0x3235/0x4b0f07
16 18,174 0 7 7,706,321 1,852,537,016 0 18,190 0x324e/0x280f58
120 24,379 0 7 10,458,169 1,844,830,695 0 24,499 0x3266/0x2f0d11
14,236 34,916 0 3 17,998,326 12,723,415 5,274,911 49,152 0x327b/0x108207
79 0 0 5 673,666 37,699 635,967 2,808 0x327b/0x1ac989
6,624 22,958 0 3 11,999,224 1,815,700,534 0 29,650 0x3290/0x394b01
19,965 40,810 0 8 17,269,398 1,803,701,310 0 60,775 0x32a7/0xfa097
7,517 58,175 0 5 21,772,528 1,786,431,912 0 65,692 0x32de/0x2aac87
9,415 18,600 0 6 10,786,946 1,764,659,384 0 28,015 0x32e1/0x36ae89
51 0 0 6 622,591 52,177 570,414 51 0x32e1/0x402e88
30,139 66,899 0 4 30,497,693 1,753,249,847 0 97,065 0x3315/0x47c6a5
6 41,284 0 1 16,342,999 12,394,744 3,948,255 41,290 0x3320/0xff97c
21,091 17,059 0 6 11,525,646 1,706,409,155 0 38,984 0x332b/0x27410a
0 94 0 4 892,946 34,897 858,049 94 0x332b/0x34e11c
3 52,035 0 3 18,160,444 1,693,990,563 0 52,132 0x336d/0x18cf58
58 27,035 0 6 8,947,956 1,675,830,119 0 27,093 0x3376/0x8c1cc
3 66 0 4 501,044 49,007 452,037 69 0x3376/0x106700
27,134 44,130 0 4 22,994,532 1,666,381,119 0 71,330 0x339f/0x3e1864
0 108 0 4 439,894 35,044 404,850 108 0x339f/0x44ceba
17 51,506 0 3 20,284,195 1,642,946,693 0 51,631 0x33a6/0x4924dd
28 16,826 0 2 9,954,053 1,622,662,498 0 16,854 0x33b6/0x487162
2,079 43,231 0 2 20,850,856 13,682,453 7,168,403 45,310 0x33d2/0x556d0a
2 30,027 0 3 14,418,091 8,924,719 5,493,372 30,743 0x33d9/0x40b5
0 1,157 0 6 5,510,215 155,394 5,354,821 3,103 0x33d9/0x5454fc
31,517 31,808 0 3 18,110,480 1,571,929,283 0 64,085 0x33ee/0x37800c
30,598 27,193 0 4 18,121,300 12,201,650 5,919,650 57,791 0x3402/0x1ad560
38,905 34,440 0 4 18,443,706 15,100,893 3,342,813 78,249 0x340c/0x3161a
51,887 95,209 0 4 40,262,214 29,791,799 10,470,415 152,283 0x343b/0x71660
1 28,686 0 7 10,081,465 1,476,991,583 0 142,625 0x344c/0x85499
28,401 13,707 0 3 10,169,877 1,466,910,118 0 42,108 0x3457/0xaec2e
5 16,531 0 6 14,562,193 4,402,715 10,159,478 16,536 0x345b/0x50893f
0 15,920 0 7 9,514,322 1,442,178,048 0 19,604 0x3460/0x492011
4 16,264 0 6 10,131,895 1,432,663,726 0 16,268 0x3465/0x4b2348
30,231 28,986 0 4 18,938,103 14,093,138 4,844,965 59,217 0x347c/0x3aef3f
14 0 0 5 512,006 34,432 477,574 120 0x347c/0x42bf45
20,678 38,294 0 3 17,101,301 1,403,081,722 0 58,975 0x3497/0x16843a
9,869 25,099 0 5 12,548,707 1,385,980,421 0 34,968 0x34b3/0x3d681d
36,465 60,605 0 5 26,361,794 24,558,695 1,803,099 97,070 0x34d6/0x5e45f
36 23,345 0 3 9,621,974 6,699,792 2,922,182 23,911 0x34d9/0x1fb5
25,436 49,579 0 7 20,880,498 18,608,524 2,271,974 75,850 0x34df/0xd8f27
32 1,485 0 6 4,197,555 413,166 3,784,389 1,535 0x34df/0x4d9bda
31,535 59,968 0 5 25,248,912 24,728,797 520,115 91,849 0x3512/0x51cea
1,601 24,378 0 6 10,268,411 8,200,696 2,067,715 26,160 0x351b/0x93565
14,147 23,162 0 8 11,300,331 1,276,852,570 0 37,400 0x3525/0x1d0cd0
14,204 20,662 0 3 13,299,117 1,265,552,239 0 34,866 0x3532/0x4f63fd
17,514 24,728 0 3 16,797,576 12,513,187 4,284,389 42,242 0x3553/0x1e8685
7,303 46,343 0 7 23,094,370 14,762,461 8,331,909 54,344 0x3580/0x4dbde7
45,620 41,812 0 7 27,514,220 21,511,990 6,002,230 90,879 0x359c/0x27cfd3
16,861 12,552 0 3 9,619,073 1,184,846,956 0 104,424 0x35a0/0x21ffd4
0 271 0 6 3,731,928 93,043 3,638,885 271 0x35a0/0x5af1ac
30,860 48,187 0 7 26,338,978 24,109,627 2,229,351 79,097 0x35ab/0x2314ce
39,031 24,255 0 4 20,202,284 1,145,156,977 0 63,301 0x35c4/0x262afa
2,576 46,002 0 6 19,184,048 5,580,647 13,603,401 48,578 0x35dc/0x19b7aa
2 0 0 6 174,028 33,089 140,939 31,536 0x35dc/0x1c5f76
111,528 77,161 0 7 62,341,581 50,549,489 11,792,092 188,690 0x360c/0x401a43
28,697 25,910 0 6 18,246,350 13,855,848 4,390,502 56,636 0x3618/0x255811
4,335 12,127 0 6 9,004,926 5,130,460 3,874,466 18,985 0x3621/0x16290f
18,413 8,607 0 5 8,878,516 7,701,363 1,177,153 27,027 0x3626/0x50c43
0 2,910 0 6 6,228,723 848,601 5,380,122 2,910 0x3626/0x641736
28,784 22,689 0 3 21,119,867 14,626,185 6,493,682 52,178 0x362a/0x752db1
41,676 62,528 0 4 34,112,988 29,303,765 4,809,223 105,238 0x363e/0x1b598d
17 735 0 5 12,521,216 31,105 12,490,111 1,300 0x3643/0x41d20d
7,802 34,494 0 7 33,127,169 7,514,191 25,612,978 43,003 0x3659/0x71898e
0 8,734 0 5 3,377,408 900,015,281 0 25,130 0x365c/0xc7c0e
114 9,858 0 7 15,782,161 650,578 15,131,583 9,972 0x365f/0x64b69f
1 11,516 0 7 5,378,418 880,855,712 0 19,548 0x3664/0x1e3191
7 22,005 0 7 8,922,863 6,691,187 2,231,676 22,012 0x3668/0xdc200
13 13,409 0 6 15,952,421 360,887 15,591,534 14,391 0x3671/0x6895a5
1,749 8,568 0 3 3,660,457 850,602,010 0 22,698 0x3676/0x7d9ce
5,390 10,146 0 6 14,882,555 1,855,609 13,026,946 15,536 0x3688/0x525a49
10,230 9,513 0 5 4,816,061 832,058,998 0 28,465 0x3691/0x34086
14,698 31,907 0 7 41,080,361 36,925,183 4,155,178 46,605 0x36aa/0x13bcaf
2 1,755 0 6 4,666,159 345,619 4,320,540 47,630 0x36aa/0x5aefde
25,935 30,385 0 5 38,398,767 781,496,417 0 57,119 0x36c5/0x42810d
9,478 14,696 0 7 21,102,013 743,097,650 0 24,174 0x36e1/0x5351ca
65 12,746 0 6 5,610,587 3,858,013 1,752,574 12,811 0x36ed/0x1057a5
5,806 9,341 0 6 10,353,968 2,690,013 7,663,955 15,343 0x36fc/0x15be55
5 0 0 6 4,894,393 33,744 4,860,649 5,931 0x36fc/0x606d0e
28,754 58,230 0 18 28,401,618 701,136,689 0 86,988 0x3708/0x480960
0 15,897 0 6 10,890,931 3,098,266 7,792,665 15,897 0x370a/0x55a193
126 10,927 0 6 26,068,972 2,214,107 23,854,865 16,100 0x372d/0x19a5ff
9,837 16,155 0 6 11,096,409 6,346,816 4,749,593 29,696 0x3732/0x2a60d8
128 15,250 0 6 8,591,606 3,747,801 4,843,805 16,046 0x3734/0x14e34e
123 28,336 0 6 16,047,724 5,161,792 10,885,932 31,737 0x3736/0x712b3a
24 8,534 0 5 3,423,605 1,552,888 1,870,717 19,982 0x3740/0xcd22f
533 0 0 6 20,325,930 275,776 20,050,154 3,884 0x3742/0x11cb59
0 0 0 4 72,242 36,230 36,012 0 0x3742/0x12e58b
26,449 16,768 0 3 20,777,536 576,217,652 0 43,217 0x3756/0x1ec2cb
11,493 23,349 0 3 21,836,979 13,435,005 8,401,974 34,842 0x375c/0x3aca7e
309 32,239 0 6 17,558,766 9,588,731 7,970,035 56,790 0x3761/0x158a6c
12 26,562 0 7 12,407,639 6,425,279 5,982,360 27,257 0x3779/0x3a4743
6,845 41,908 0 7 33,072,537 22,775,381 10,297,156 51,339 0x379b/0x69295c
6,778 29,990 0 6 25,664,453 17,110,934 8,553,519 41,294 0x37a8/0x2701a1
5 16,509 0 9 19,648,397 8,979,547 10,668,850 44,387 0x37ae/0x21a42e
116,569 137,423 0 12 129,250,694 72,386,145 56,864,549 267,033 0x37e2/0x163534
13 17,546 0 6 28,042,727 34,685 28,008,042 187,464 0x37e7/0x90ee1b
33,326 30,872 0 6 43,125,925 9,805,827 33,320,098 81,756 0x37ec/0x280a40
3 5,332 0 5 19,873,538 37,653 19,835,885 30,103 0x37ef/0x261c42
17 2,306 0 6 19,166,423 244,987 18,921,436 7,657 0x37f1/0x196419
10,753 20,277 0 6 40,648,779 11,079,498 29,569,281 32,618 0x37f6/0x234a64
10 22,876 0 6 18,391,375 164,634 18,226,741 49,034 0x37f8/0xabeb3
30 34,377 0 7 32,006,385 1,775,081 30,231,304 56,545 0x37fb/0x295c24
4 23,090 0 7 38,624,657 2,288,459 36,336,198 52,101 0x37ff/0x145fb5
6 8,689 0 6 27,108,848 949,959 26,158,889 24,449 0x3801/0x80d8a5
6 5,117 0 7 18,644,436 55,620 18,588,816 11,221 0x3803/0x6c2979
837 0 0 6 6,661,670 288,422 6,373,248 5,956 0x3804/0x39391f
5,863 6,917 0 0 3,705,888 3,705,888 0 12,780 0x3804/0x71c53f
</DbPrintLog>
- This prints some statistics about the database. In particular the two rows called key bytes and data bytes are relevant. For the following step you need the values from the avg column - i.e. 38 and 219 in this case. You also need the value of “Total number of entries” (14543762 in this case). Now we take those values and put them into this command:
[root@dc042 meta]# java -cp /usr/share/dcache/classes/je-*.jar com.sleepycat.je.util.DbCacheSize -records 14543762 -key 38 -data 219
=== Environment Cache Overhead ===
3,157,213 minimum bytes
To account for JE daemon operation and record locks,
a significantly larger amount is needed in practice.
=== Database Cache Size ===
Number of Bytes Description
--------------- -----------
1,195,759,832 Internal nodes only
5,124,232,392 Internal nodes and leaf nodes
- For further information see the DbCacheSize javadoc.
- For this pool it tells us that we need a bit above 1195 MB to keep all the internal btree nodes cached. Oracle’s recommendation is that if the database is updated often, the cache is at least big enough to contain the internal nodes. Obviously you want to make it a bit bigger to leave room for it to grow. One could take this and configure the cache using the je.maxMemory setting. If the size is lower than the 20% of the max heap size you already use, then you don’t need to do anything (I do not suggest lowering it further).
- Since this pool has 6000m of JAVA heap max space, it means that 20% = 1200MB, which is exactly cache for Internal btree nodes. So we will not touch this.
- There is one caveat though: If you increase the cache size, less free space is left on the heap. If the pool is pushed to the limit, this may actually slow down the pool as garbage collection overhead increases. You need to ensure that enough space is left in addition to the cache (possibly by increasing the max heap size). An alternative to adjusting je.maxMemory is of course to adjust the max heap size. If you make it large enough so that 20% is enough to cache the internal btree nodes, then all is well. This may however mean you assign significantly more memory to the pool than it really needs. Yeah, there are lots of things to consider.
- A little bonus: There is also this command:
[root@dc042 meta]# java -cp /usr/share/dcache/classes/je-*.jar com.sleepycat.je.util.DbSpace -h . File Size (KB) % Used -------- --------- ------ 00001857 9765 41 00001f2a 9765 43 00001f2f 9765 39 00001f72 9765 44 0000216d 9765 40 000021cb 9765 40 0000226b 9765 45 000022e8 9763 38 000022fa 9765 39 00002390 9765 38 0000241d 9765 49 0000252c 9765 51 0000254c 9765 47 00002567 9765 39 000025b7 9765 42 000025db 9765 42 000025e5 9765 56 0000265f 9765 39 00002678 9765 61 0000268d 9762 48 00002692 9765 74 000026be 9765 48 000026cf 9765 40 00002700 9765 41 00002706 9765 38 00002718 9765 56 00002747 9765 44 0000274d 9765 48 000027db 9765 62 00002855 9765 39 0000285c 9765 39 00002882 9765 41 0000289d 9765 54 000028ba 9765 52 000028c1 9765 53 000028d0 9765 41 000028d7 9765 41 000028e7 9765 39 000028ed 9765 39 00002926 9765 39 0000292b 9765 43 00002945 9765 41 00002952 9765 41 00002958 9765 62 00002959 9764 38 00002962 9765 53 00002963 9765 47 0000297d 9765 51 0000297e 9765 41 000029b3 9765 45 000029c7 9765 56 000029ce 9765 42 000029d3 9765 43 000029d9 9765 54 000029fb 9765 42 00002a00 9765 54 00002a01 9765 39 00002a06 9765 39 00002a20 9765 45 00002a26 9765 61 00002a2b 9765 69 00002a30 9765 65 00002a44 9765 46 00002a4f 9765 50 00002a53 9765 64 00002a54 9765 47 00002a58 9765 66 00002a59 9765 43 00002a5e 9765 56 00002a63 9765 44 00002a69 9765 47 00002a8f 9765 39 00002aac 9765 54 00002aba 9765 42 00002abb 9765 44 00002ae0 9763 38 00002af3 9765 45 00002b01 9765 41 00002b08 9765 65 00002b0e 9765 41 00002b24 9765 45 00002b28 9765 61 00002b29 9765 41 00002b3c 9765 40 00002b3d 9765 40 00002b41 9765 47 00002b4c 9765 46 00002b52 9765 57 00002b64 9765 38 00002b87 9765 38 00002b8e 9765 47 00002b8f 9760 38 00002bad 9765 42 00002bbf 9765 44 00002bc6 9765 38 00002be9 9765 47 00002c00 9765 42 00002c06 9765 69 00002c0c 9765 55 00002c13 9765 38 00002c14 9765 38 00002c2a 9765 41 00002c2b 9765 41 00002c30 9765 54 00002c4c 9765 39 00002c53 9765 45 00002c5f 9765 44 00002c7c 9765 38 00002c8f 9765 49 00002c95 9765 38 00002ca4 9764 50 00002ca9 9765 42 00002cad 9763 41 00002cc4 9765 39 00002ccf 9765 43 00002cd3 9765 45 00002cf3 9765 48 00002d19 9765 38 00002d20 9765 38 00002d3a 9765 40 00002d3b 9765 41 00002d41 9765 43 00002d51 9765 39 00002d59 9765 40 00002d79 9765 62 00002d8f 9765 39 00002deb 9765 40 00002e21 9765 39 00002e40 9765 41 00002e41 9765 45 00002e45 9765 40 00002e5b 9765 58 00002e5f 9765 49 00002e60 9765 67 00002e7d 9765 48 00002e95 9765 38 00002eb1 9762 57 00002eb6 9765 53 00002ee1 9765 38 00002ef4 9763 43 00002ef8 9765 41 00002f1a 9765 53 00002f25 9765 65 00002f5a 9765 52 00002f5f 9765 63 00002f66 9765 41 00002f71 9765 42 00002f79 9765 61 00002f87 9765 49 00002f9d 9765 41 00002f9e 9765 42 00002fa3 9765 38 00002fa4 9765 46 00002fb1 9762 55 00002fc1 9765 41 00002fde 9765 38 0000300a 9765 43 0000300e 9762 47 00003026 9765 55 00003027 9765 57 0000303f 9765 63 0000304f 9765 48 00003056 9765 44 0000306c 9765 48 0000306d 9765 42 00003071 9765 45 0000308d 9765 39 000030a1 9765 54 000030a5 9765 51 000030ab 9765 60 000030c2 9765 46 000030d5 9765 43 000030dd 9765 38 000030e3 9765 57 000030e4 9765 55 000030fd 9765 48 0000310c 9764 42 00003110 9765 46 00003119 9765 47 0000315c 9765 48 0000315d 9765 54 00003185 9765 56 00003189 9765 52 0000318a 9765 58 0000319a 9765 46 0000319f 9764 39 000031b5 9764 53 000031b8 9765 40 000031b9 9765 74 000031e2 9765 65 000031fa 9765 58 000031fb 9765 52 00003201 9765 50 00003202 9764 44 00003208 9765 42 00003210 9763 40 00003218 9765 40 00003224 9765 51 00003229 9765 48 00003235 9765 40 0000324e 9765 46 00003266 9765 41 0000326c 9764 47 0000327b 9765 55 00003290 9765 48 00003294 9765 56 000032a7 9765 69 000032ad 9765 62 000032de 9765 47 000032e1 9765 46 000032e2 9765 71 000032e9 9765 45 00003315 9765 42 00003316 9764 54 00003320 9765 43 0000332b 9765 52 0000332c 9765 76 0000336d 9765 69 00003376 9765 72 00003377 9765 46 0000339f 9765 43 000033a0 9765 75 000033a6 9765 46 000033b6 9765 40 000033c0 9764 72 000033d2 9765 38 000033d3 9764 45 000033d9 9765 42 000033da 9765 44 000033ee 9765 41 000033fb 9764 38 00003402 9765 42 00003403 9765 53 0000340c 9765 76 00003428 9765 67 00003431 9765 51 00003436 9765 88 0000343b 9765 85 0000344c 9765 48 00003457 9765 46 0000345b 9765 46 00003460 9765 50 00003465 9765 45 00003466 9762 40 0000347c 9765 54 0000347d 9765 61 00003497 9765 63 000034b3 9765 64 000034bd 9765 54 000034c9 9764 71 000034d6 9765 65 000034d9 9765 86 000034da 9765 46 000034df 9765 51 000034e0 9764 60 00003504 9762 80 00003512 9761 76 0000351b 9765 65 00003525 9765 59 00003532 9765 52 00003540 9764 43 00003553 9765 80 00003558 9765 54 00003580 9762 49 00003586 9765 41 00003587 9763 50 0000359c 9765 40 000035a0 9765 46 000035a1 9765 37 000035a6 9765 96 000035ab 9765 52 000035ac 9765 39 000035c4 9765 61 000035d0 9765 89 000035dc 9765 74 000035dd 9765 43 000035f1 9765 43 000035fd 9765 37 00003604 9765 47 00003609 9764 40 0000360c 9765 59 0000360d 9765 37 00003618 9763 42 00003621 9765 42 00003626 9765 42 00003627 9765 41 0000362a 9765 38 0000362b 9765 73 00003630 9765 68 00003631 9764 42 0000363e 9763 39 00003643 9765 43 00003646 9765 38 0000364c 9762 40 00003659 9765 47 0000365c 9764 44 0000365f 9765 45 00003664 9765 69 00003668 9764 37 00003671 9765 43 00003676 9765 39 00003688 9765 37 00003691 9765 48 00003692 9764 37 00003695 9762 38 000036a3 9764 50 000036aa 9765 47 000036ab 9765 49 000036c2 9765 64 000036c3 9764 57 000036c5 9765 40 000036da 9764 40 000036e1 9765 43 000036ed 9762 39 000036fc 9765 41 000036fd 9765 59 00003701 9764 85 00003708 9765 43 0000370a 9763 49 00003728 9762 41 0000372a 9763 55 0000372d 9765 51 00003732 9763 48 00003734 9765 60 00003736 9765 42 00003740 9765 48 00003741 9765 39 00003742 9765 38 00003749 9764 40 00003756 9763 39 00003758 9762 46 0000375c 9765 67 0000375d 9765 37 00003761 9765 63 00003779 9765 55 00003785 9765 43 00003798 9763 60 0000379b 9765 41 0000379e 9765 37 000037a5 9764 39 000037a8 9764 48 000037ab 9765 38 000037ae 9765 54 000037b3 9764 42 000037b6 9761 36 000037be 9763 46 000037c1 9765 69 000037c3 9765 36 000037c4 9764 37 000037ce 9762 50 000037d1 9761 45 000037d3 9765 45 000037dd 9765 45 000037df 9765 50 000037e0 9763 39 000037e2 9765 13 000037e3 9763 18 000037e7 9765 49 000037e8 9763 41 000037e9 9765 42 000037ea 9763 40 000037eb 9765 39 000037ec 9761 39 000037ed 9763 41 000037ef 9763 36 000037f0 9764 36 000037f1 9764 39 000037f3 9765 41 000037f4 9761 53 000037f5 9765 36 000037f6 9765 54 000037f7 9763 53 000037f8 9765 63 000037f9 9765 60 000037fa 9765 70 000037fb 9765 76 000037fc 9765 72 000037fd 9765 73 000037fe 9765 83 000037ff 9765 87 00003800 9765 89 00003801 9762 86 00003802 9763 88 00003803 9764 90 00003804 8110 76 TOTALS 3757891 49
- It will tell you for each of the database files (the log segments) how big it is and what the utilization is. The Berkeley DB will try to keep the total utilization above 50%, but for a small pool like this one it cannot do it. I figure that you may find this interesting to know after reading about utilization above.
- The log segment size and the utilization goal can be adjusted too, but I cannot give any sound advice on whether that’s a good idea and how to determine good values.
- All of the above is only relevant if you use the Berkeley DB backend for pools.
Special thanks
- This documentation has been deployed thanks to Gerd Behrmann, you can find more in at: https://gist.github.com/gbehrmann/df1abbe45c03356eff2e
- Read more about this at: