https://pwiki.pic.es/index.php?action=history&feed=atom&title=Monitoring_dCache_with_ELK 2026-05-02T12:57:41Z Revision history for this page on the wiki MediaWiki 1.35.14 https://pwiki.pic.es/index.php?title=Monitoring_dCache_with_ELK&diff=980&oldid=prev 2022-06-13T15:16:23Z

<p></p> <table class="diff diff-contentalign-left diff-editfont-monospace" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 15:16, 13 June 2022</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1" >Line 1:</td> <td colspan="2" class="diff-lineno">Line 1:</td></tr> <tr><td class='diff-marker'>−</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>= PIC &amp; ELK <del class="diffchange diffchange-inline">= </del>'''(obsolete)'''</div></td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>= PIC &amp; ELK '''(obsolete)''' <ins class="diffchange diffchange-inline">=</ins></div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* PIC has an ELK (ElasticSearch + LogStash + Kibana) service in order to monitor the different ''batch'' &amp; ''storage'' services.</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* PIC has an ELK (ElasticSearch + LogStash + Kibana) service in order to monitor the different ''batch'' &amp; ''storage'' services.</div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* A Kibana4 Dashboard can be accessed in the following link: '''[https://kibana4.pic.es:8443/#/dashboard?_g=%28%29 Kibana4]'''</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* A Kibana4 Dashboard can be accessed in the following link: '''[https://kibana4.pic.es:8443/#/dashboard?_g=%28%29 Kibana4]'''</div></td></tr> </table>

Eplanas https://pwiki.pic.es/index.php?title=Monitoring_dCache_with_ELK&diff=979&oldid=prev 2022-06-13T15:16:00Z

<p></p> <table class="diff diff-contentalign-left diff-editfont-monospace" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 15:16, 13 June 2022</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1" >Line 1:</td> <td colspan="2" class="diff-lineno">Line 1:</td></tr> <tr><td class='diff-marker'>−</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>= PIC &amp; ELK =</div></td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>= PIC &amp; ELK = <ins class="diffchange diffchange-inline">'''(obsolete)'''</ins></div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* PIC has an ELK (ElasticSearch + LogStash + Kibana) service in order to monitor the different ''batch'' &amp; ''storage'' services.</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* PIC has an ELK (ElasticSearch + LogStash + Kibana) service in order to monitor the different ''batch'' &amp; ''storage'' services.</div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* A Kibana4 Dashboard can be accessed in the following link: '''[https://kibana4.pic.es:8443/#/dashboard?_g=%28%29 Kibana4]'''</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* A Kibana4 Dashboard can be accessed in the following link: '''[https://kibana4.pic.es:8443/#/dashboard?_g=%28%29 Kibana4]'''</div></td></tr> </table>

Eplanas https://pwiki.pic.es/index.php?title=Monitoring_dCache_with_ELK&diff=80&oldid=prev 2015-08-07T09:02:58Z

<p></p> <table class="diff diff-contentalign-left diff-editfont-monospace" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 09:02, 7 August 2015</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l6" >Line 6:</td> <td colspan="2" class="diff-lineno">Line 6:</td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Monitoring the number of transfers, stores, restores, removes &amp; hits in time per VO ==</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Monitoring the number of transfers, stores, restores, removes &amp; hits in time per VO ==</div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Link: '''[https://kibana4.pic.es:8443/#/dashboard/dCache-Statistics-in-Time-per-VO?_g=%28%29&amp;_a=%28filters:!%28%29,panels:!%28%28col:1,id:Count-%23hits-in-time-per-VO,row:6,size_x:6,size_y:5,type:visualization%29,%28col:7,id:Count-%23removes-in-time-per-VO,row:6,size_x:6,size_y:5,type:visualization%29,%28col:1,id:Count-%23restores-in-time-per-VO,row:11,size_x:6,size_y:5,type:visualization%29,%28col:1,id:Count-%23transfers-in-time-per-VO,row:1,size_x:12,size_y:5,type:visualization%29,%28col:7,id:Count-%23stores-in-time-per-VO,row:11,size_x:6,size_y:5,type:visualization%29%29,query:%28query_string:%28analyze_wildcard:!t,query:%27*%27%29%29,title:%27dCache%20Statistics%20in%20Time%20per%20VO%27%29 dCache Statistics in Time per VO]'''</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* Link: '''[https://kibana4.pic.es:8443/#/dashboard/dCache-Statistics-in-Time-per-VO?_g=%28%29&amp;_a=%28filters:!%28%29,panels:!%28%28col:1,id:Count-%23hits-in-time-per-VO,row:6,size_x:6,size_y:5,type:visualization%29,%28col:7,id:Count-%23removes-in-time-per-VO,row:6,size_x:6,size_y:5,type:visualization%29,%28col:1,id:Count-%23restores-in-time-per-VO,row:11,size_x:6,size_y:5,type:visualization%29,%28col:1,id:Count-%23transfers-in-time-per-VO,row:1,size_x:12,size_y:5,type:visualization%29,%28col:7,id:Count-%23stores-in-time-per-VO,row:11,size_x:6,size_y:5,type:visualization%29%29,query:%28query_string:%28analyze_wildcard:!t,query:%27*%27%29%29,title:%27dCache%20Statistics%20in%20Time%20per%20VO%27%29 dCache Statistics in Time per VO]'''</div></td></tr> <tr><td class='diff-marker'>−</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>* In this Dashboard we can find how many stores and restores have run in the last X hours (default value is <del class="diffchange diffchange-inline">1 week</del>), as well as the amount of transfers and hits. Number of removed files is also shown.</div></td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>* In this Dashboard we can find how many stores and restores have run in the last X hours (default value is <ins class="diffchange diffchange-inline">15 minutes</ins>), as well as the amount of transfers and hits. Number of removed files is also shown.</div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Monitoring transfers ==</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>== Monitoring transfers ==</div></td></tr> </table>

Mcaubet https://pwiki.pic.es/index.php?title=Monitoring_dCache_with_ELK&diff=79&oldid=prev 2015-08-07T09:02:26Z

<p></p> <table class="diff diff-contentalign-left diff-editfont-monospace" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 09:02, 7 August 2015</td> </tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1" >Line 1:</td> <td colspan="2" class="diff-lineno">Line 1:</td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= PIC &amp; ELK =</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= PIC &amp; ELK =</div></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* PIC has an ELK (ElasticSearch + LogStash + Kibana) service in order to monitor the different ''batch'' &amp; ''storage'' services.</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>* PIC has an ELK (ElasticSearch + LogStash + Kibana) service in order to monitor the different ''batch'' &amp; ''storage'' services.</div></td></tr> <tr><td class='diff-marker'>−</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div>* A Kibana4 Dashboard can be accessed in the following link:</div></td><td class='diff-marker'>+</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div>* A Kibana4 Dashboard can be accessed in the following link: <ins class="diffchange diffchange-inline">'''[</ins>https://kibana4.pic.es:8443/#/dashboard?_g=%28%29 <ins class="diffchange diffchange-inline">Kibana4]'''</ins></div></td></tr> <tr><td class='diff-marker'>−</td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div><del class="diffchange diffchange-inline">: </del>https://kibana4.pic.es:8443/#/dashboard?_g=%28%29</div></td><td colspan="2"> </td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"></td></tr> <tr><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= Monitoring Dashboards =</div></td><td class='diff-marker'> </td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>= Monitoring Dashboards =</div></td></tr> </table>

Mcaubet https://pwiki.pic.es/index.php?title=Monitoring_dCache_with_ELK&diff=78&oldid=prev 2015-08-07T09:01:34Z

<p>Created page with &quot;= PIC &amp; ELK = * PIC has an ELK (ElasticSearch + LogStash + Kibana) service in order to monitor the different &#039;&#039;batch&#039;&#039; &amp; &#039;&#039;storage&#039;&#039; services. * A Kibana4 Dashboard can be acc...&quot;</p> <p><b>New page</b></p><div>= PIC &amp; ELK =<br /> * PIC has an ELK (ElasticSearch + LogStash + Kibana) service in order to monitor the different ''batch'' &amp; ''storage'' services.<br /> * A Kibana4 Dashboard can be accessed in the following link:<br /> : https://kibana4.pic.es:8443/#/dashboard?_g=%28%29<br /> <br /> = Monitoring Dashboards =<br /> == Monitoring the number of transfers, stores, restores, removes &amp; hits in time per VO ==<br /> * Link: '''[https://kibana4.pic.es:8443/#/dashboard/dCache-Statistics-in-Time-per-VO?_g=%28%29&amp;_a=%28filters:!%28%29,panels:!%28%28col:1,id:Count-%23hits-in-time-per-VO,row:6,size_x:6,size_y:5,type:visualization%29,%28col:7,id:Count-%23removes-in-time-per-VO,row:6,size_x:6,size_y:5,type:visualization%29,%28col:1,id:Count-%23restores-in-time-per-VO,row:11,size_x:6,size_y:5,type:visualization%29,%28col:1,id:Count-%23transfers-in-time-per-VO,row:1,size_x:12,size_y:5,type:visualization%29,%28col:7,id:Count-%23stores-in-time-per-VO,row:11,size_x:6,size_y:5,type:visualization%29%29,query:%28query_string:%28analyze_wildcard:!t,query:%27*%27%29%29,title:%27dCache%20Statistics%20in%20Time%20per%20VO%27%29 dCache Statistics in Time per VO]'''<br /> * In this Dashboard we can find how many stores and restores have run in the last X hours (default value is 1 week), as well as the amount of transfers and hits. Number of removed files is also shown.<br /> <br /> == Monitoring transfers ==<br /> * Link: '''[https://kibana4.pic.es:8443/#/dashboard/dCache-Transfer-Dashboard?_g=%28%29&amp;_a=%28filters:!%28%29,panels:!%28%28col:1,id:dCache-transferred-bytes-per-DN,row:5,size_x:12,size_y:3,type:visualization%29,%28col:1,id:dCache-transferred-bytes-per-pool-and-protocol,row:1,size_x:6,size_y:4,type:visualization%29,%28col:7,id:%27dCache-transferred-bytes-per-pool-and-protocol-%28Top-20%29%27,row:1,size_x:6,size_y:4,type:visualization%29,%28col:1,id:dCache-transfer-percentiles,row:8,size_x:12,size_y:1,type:visualization%29%29,query:%28query_string:%28analyze_wildcard:!t,query:%27*%27%29%29,title:%27dCache%20Transfer%20Dashboard%27%29 dCache Transfer Dashboard]'''<br /> * Here we have different plots:<br /> :* A plot showing information about moved data in the last X hours for the top 20 dCache pools. Here also is specified how many data has been moved by protocol and which protocols have been used per pool.<br /> :* A plot showing information about moved data in the last X hours for all dCache pools. Here also is specified how many data has been moved by protocol and which protocols have been used per pool.<br /> :* A stack plot showing the amount of data moved by DN.<br /> :* A window with percentil statistics.<br /> <br /> == Transferred data by protocol ==<br /> <br /> * Link: '''[https://kibana4.pic.es:8443/#/dashboard/dCache-Transferred-Bytes-%28Read-ampersand-Writes%29-Dashboard?_g=%28%29&amp;_a=%28filters:!%28%29,panels:!%28%28col:1,id:dCache-Transferred-Bytes-with-DCap-3.0,row:6,size_x:6,size_y:3,type:visualization%29,%28col:1,id:dCache-Transferred-Bytes-with-GFtp-1.0,row:9,size_x:6,size_y:3,type:visualization%29,%28col:7,id:dCache-Transferred-Bytes-with-GFtp-2.0,row:6,size_x:6,size_y:3,type:visualization%29,%28col:7,id:dCache-Transferred-Bytes-with-Http-1.1,row:9,size_x:6,size_y:3,type:visualization%29,%28col:1,id:dCache-Transferred-Bytes-with-Xrootd-2.7,row:12,size_x:6,size_y:3,type:visualization%29,%28col:1,id:dCache-Transferred-Bytes-ALL-Protocols,row:1,size_x:12,size_y:5,type:visualization%29,%28col:7,id:dCache-Transferred-Bytes-with-NFS4-4.1,row:12,size_x:6,size_y:3,type:visualization%29,%28col:1,id:dCache-Transferred-Bytes-Enstore-Stores,row:15,size_x:6,size_y:3,type:visualization%29,%28col:7,id:dCache-Transferred-Bytes-Enstore-Restores,row:15,size_x:6,size_y:3,type:visualization%29%29,query:%28query_string:%28analyze_wildcard:!t,query:%27*%27%29%29,title:%27dCache%20Transferred%20Bytes%20%28Read%20%26%20Writes%29%20Dashboard%27%29 dCache Transferred Bytes (Read &amp; Writes) Dashboard]'''<br /> * In this Dashboard we have several plots, basically showing the amount of data moved by each protocol.<br /> * A big plot is showing the amount of data moved by all protocols<br /> * We have also 2 plots showing the amount of data moved due to stores and restores.</div>

Mcaubet