<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:inherit;
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1095132037;
mso-list-template-ids:1330182024;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style>
</head>
<body lang="EN-GB" link="#0563C1" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi Bill,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">The workqueue_ metrics relate to the processing of configuration updates. Config updates can be triggered by an update to an Ingress Resource, Endpoints in a monitored Service, and other events.
The purpose of these metrics is to observe the backlog to determine how quickly IC can respond to changes. They probably do not provide the data you want (“</span>I basically am interested in analyzing the time spent for a request in the ingress controller”)?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">When you use the Ingress Controller with NGINX open source or NGINX Plus, you can get this data from the log files: see
<a href="https://github.com/nginxinc/kubernetes-ingress/tree/v1.10.0/examples/custom-log-format">
https://github.com/nginxinc/kubernetes-ingress/tree/v1.10.0/examples/custom-log-format</a> and the NGINX variable $request_time
<a href="http://nginx.org/en/docs/http/ngx_http_log_module.html#var_request_time">
http://nginx.org/en/docs/http/ngx_http_log_module.html#var_request_time</a> When you use it with NGINX Plus, you additionally get aggregate request time data in the Prometheus data<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Feel free to raise additional questions about Ingress Controller on the github page:
<a href="https://github.com/nginxinc/kubernetes-ingress/issues">https://github.com/nginxinc/kubernetes-ingress/issues</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Owen<span style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">nginx <nginx-bounces@nginx.org> on behalf of William Shaya <wshaya@osisoft.com><br>
<b>Reply to: </b>"nginx@nginx.org" <nginx@nginx.org><br>
<b>Date: </b>Thursday, 28 January 2021 at 18:10<br>
<b>To: </b>"nginx@nginx.org" <nginx@nginx.org><br>
<b>Subject: </b>Nginx Ingress Controller with Prometheus Metrics Issue<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p style="margin-bottom:12.0pt"><strong><span lang="EN" style="font-family:"Calibri",sans-serif">EXTERNAL MAIL: nginx-bounces@nginx.org</span></strong><span lang="EN"><o:p></o:p></span></p>
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I have a Kubernetes cluster and am using the nginx ingress controller. I have followed the instructions at:<o:p></o:p></p>
<p class="MsoNormal"><a href="https://docs.nginx.com/nginx-ingress-controller/logging-and-monitoring/prometheus/">https://docs.nginx.com/nginx-ingress-controller/logging-and-monitoring/prometheus/</a><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I can see these metrics in Prometheus, however, I DO NOT see:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#404040;background:white">Workqueue metrics. <b><span style="border:none windowtext 1.0pt;padding:0cm">Note</span></b>: the workqueue is a queue used by the Ingress Controller
to process changes to the relevant resources in the cluster like Ingress resources. The Ingress Controller uses only one queue. The metrics for that queue will have the label </span><span style="font-size:11.5pt;font-family:inherit;color:#363534;border:none windowtext 1.0pt;padding:0cm;background:#F7F7F7">name="taskQueue"</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:54.0pt;text-indent:-18.0pt;line-height:15.0pt;mso-list:l0 level1 lfo2;background:white;vertical-align:baseline">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.5pt;font-family:inherit;color:#363534;border:none windowtext 1.0pt;padding:0cm;background:#F7F7F7">workqueue_depth</span><span style="font-size:11.5pt;font-family:inherit;color:#404040">. Current depth
of the workqueue.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:54.0pt;text-indent:-18.0pt;line-height:15.0pt;mso-list:l0 level1 lfo2;background:white;vertical-align:baseline">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.5pt;font-family:inherit;color:#363534;border:none windowtext 1.0pt;padding:0cm;background:#F7F7F7">workqueue_queue_duration_second</span><span style="font-size:11.5pt;font-family:inherit;color:#404040">.
How long in seconds an item stays in the workqueue before being requested.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:54.0pt;text-indent:-18.0pt;line-height:15.0pt;mso-list:l0 level1 lfo2;background:white;vertical-align:baseline">
<![if !supportLists]><span style="font-size:10.0pt;font-family:Symbol"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.5pt;font-family:inherit;color:#363534;border:none windowtext 1.0pt;padding:0cm;background:#F7F7F7">workqueue_work_duration_seconds</span><span style="font-size:11.5pt;font-family:inherit;color:#404040">.
How long in seconds processing an item from the workqueue takes.</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Any help is determining what the issue is here would be greatly appreciated. I basically am interested in analyzing the time spent for a request in the ingress controller<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Thanks<o:p></o:p></p>
<p class="MsoNormal">Bill<o:p></o:p></p>
</div>
</body>
</html>