<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-compose;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi Sangmin,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Your analysis looks correct. Client streaming RPCs can lead to unbound accumulation of ngx_chain_t objects (although, at very slow rate). Gzip module had a similar issue (<a href="https://trac.nginx.org/nginx/ticket/1046">https://trac.nginx.org/nginx/ticket/1046</a>).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Attaching somewhat simplified patch based on yours. I was able to reproduce the issue locally and the patch fixes it.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> nginx-devel <nginx-devel-bounces@nginx.org>
<b>On Behalf Of </b>Sangmin Lee<br>
<b>Sent:</b> Thursday, April 4, 2024 19:29<br>
<b>To:</b> nginx-devel@nginx.org<br>
<b>Subject:</b> I think I found a fix for the memory leak issue on gRPC module<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="line-height:12.0pt;background:#FFEB9C"><b><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#9C6500">CAUTION:</span></b><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:black"> This email has
been sent from an external source. Do not click links, open attachments, or provide sensitive business information unless you can verify the sender’s legitimacy.
<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">I am sending this mail again because I did a mistake while I was writing a mail. I didn't know, in gmail, "Ctrl - Enter" would send a mail immediately even without a title! <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I am sorry for that, so I am sending again.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">Hello, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I think I found the main cause of the memory leak issue when using gRPC stream so I made a patch for it. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Please find the test scenario and details here -- This is what I wrote.:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><a href="https://trac.nginx.org/nginx/ticket/2614" target="_blank">https://trac.nginx.org/nginx/ticket/2614</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">After I changed the memory pool totally on nginx and tested our workload -- long-lived gRPC streams with many connections -- using Valgrind and massif, I was able to find what brought up the memory leak issue.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">like the picture below. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<img border="0" width="446" height="218" style="width:4.6416in;height:2.275in" id="Picture_x0020_1" src="cid:image001.png@01DA9C29.C792CD90"><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">( I am not sure whether this picture will be sent properly )<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">After I patched one part, it seems okay now I have tested it for 1 week with out workload.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><img border="0" width="444" height="356" style="width:4.625in;height:3.7083in" id="Picture_x0020_2" src="cid:image002.png@01DA9C29.C792CD90"><o:p></o:p></p>
<div>
<p class="MsoNormal">( I am not sure whether this picture will be sent properly )<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">But because I am not familiar with Mercurial, I couldn't find a way to create PR like on github. I guess this mailing list is for this patch.<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">From my point of view, it is more like a workaround and I think the way of using ngx_chain_add_copy() or itself needs to be changed because it allocates a ngx_chain_t structure using ngx_alloc_chain_link() but inside of that, it just copies
pointer, like cl->buf = in->buf;<o:p></o:p></p>
</div>
<p class="MsoNormal">so this ngx_chain_t instance should be dealt with differently unlike other ngx_chain_t instances.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">But I am quite new to nginx codes so my view might be wrong.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Anyhow, please go over this patch and I would like to further talk here.<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">--------------------------------------------------------------------------------------------------------------------------------------------<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">diff --git a/src/http/modules/ngx_http_grpc_module.c b/src/http/modules/ngx_http_grpc_module.c<br>
index dfe49c586..1db67bd0a 100644<br>
--- a/src/http/modules/ngx_http_grpc_module.c<br>
+++ b/src/http/modules/ngx_http_grpc_module.c<br>
@@ -1462,6 +1462,12 @@ ngx_http_grpc_body_output_filter(void *data, ngx_chain_t *in)<br>
in = in->next;<br>
}<br>
<br>
+ ngx_chain_t *nl;<br>
+ for (ngx_chain_t *dl = ctx->in; dl != in; dl = nl ) {<br>
+ nl = dl->next;<br>
+ ngx_free_chain(r->pool, dl);<br>
+ }<br>
+<br>
ctx->in = in;<br>
<br>
if (last) { <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal">--------------------------------------------------------------------------------------------------------------------------------------------<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Best regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Sangmin<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>