nginx serving corrupt images
Dan Swaney
justdan23 at gmail.com
Fri Feb 24 02:42:29 UTC 2023
Ah-ah...I caught the NGINX failure in the SSL response:
2023/02/23 21:24:49 [debug] 4768#4528: *1 http filename:
> "./html/images/image001.jpg"
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 add cleanup: 000002DC83A7CBE8
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http static fd: 732
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http set discard body
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http map started
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 parse header: "Cookie: SessionId
> SessionId="
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http map started
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http map: "" ""
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http script var: ""
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http map: "" ""
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http script copy: "SessionId
> SessionId="
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http script var: ""
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http script copy: ";AuthId="
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http script var: "SessionId
> SessionId="
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http script copy: " SessionId="
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http script var:
> "ab3e389a16b819b5477309665cfcc4672f47d5657e1c154381c6eb8336963721"
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http script copy: "; Domain=
> win10-web-svr.dreamstone.com; Path=/; HttpOnly; Secure"
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 HTTP/1.1 200 OK
> Server: nginx/1.23.3
> Date: Fri, 24 Feb 2023 02:24:49 GMT
> Content-Type: application/octetstream
> *Content-Length: 123197*
> Last-Modified: Wed, 30 Nov 2022 19:41:21 GMT
> Connection: keep-alive
> ETag: "6387b1e1-1e13d"
> Strict-Transport-Security: max-age=15768000; preload
> X-Frame-Options: SAMEORIGIN
> X-XSS-Protection: 1
> X-Content-Type-Options: nosniff
> Referred-Policy: strict-origin
> Set-Cookie: SessionId SessionId=;AuthId=SessionId SessionId=
> SessionId=ab3e389a16b819b5477309665cfcc4672f47d5657e1c154381c6eb8336963721;
> Domain=win10-web-svr.dreamstone.com; Path=/; HttpOnly; Secure
> Accept-Ranges: bytes
>
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 write new buf t:1 f:0
> 000002DC83A7E2F0, pos 000002DC83A7E2F0, size: 626 file: 0, size: 0
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http write filter: l:0 f:0 s:626
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http output filter
> "/images/image001.jpg?"
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http copy filter:
> "/images/image001.jpg?"
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 malloc: 000002DC83A87340:32768
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http postpone filter
> "/images/image001.jpg?" 000002DC83A7E658
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 write old buf t:1 f:0
> 000002DC83A7E2F0, pos 000002DC83A7E2F0, size: 626 file: 0, size: 0
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 write new buf t:1 f:0
> 000002DC83A87340, pos 000002DC83A87340, size: 32768 file: 0, size: 0
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http write filter: l:0 f:1
> s:33394
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http write filter limit 2097152
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 malloc: 000002DC83A8F350:16384
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 SSL buf copy: 626
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 SSL buf copy: 15758
> *2023/02/23 21:24:49 [debug] 4768#4528: *1 SSL to write: 16384*
> 2023/02/23 21:24:49 [debug] 4768#4528: ssl remove session: B87DD7B9:32
> 2023/02/23 21:24:49 [debug] 4768#4528: shmtx lock
> 2023/02/23 21:24:49 [debug] 4768#4528: shmtx unlock
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 SSL_write: -1
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 SSL_get_error: 1
>
> *2023/02/23 21:24:49 [crit] 4768#4528: *1 SSL_write() failed (SSL:
> error:0A0C0103:SSL routines::internal error) while sending response to
> client, client: 192.168.80.130, server: win10-web-svr.dreamstone.com
> <http://win10-web-svr.dreamstone.com>, request: "GET /images/image001.jpg
> HTTP/1.1", host: "win10-web-svr.dreamstone.com
> <http://win10-web-svr.dreamstone.com>", referrer:
> "https://win10-web-svr.dreamstone.com/
> <https://win10-web-svr.dreamstone.com/>"*2023/02/23 21:24:49 [debug]
> 4768#4528: *1 http write filter FFFFFFFFFFFFFFFF
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http copy filter: -1
> "/images/image001.jpg?"
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http finalize request: -1,
> "/images/image001.jpg?" a:1, c:1
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http terminate request count:1
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http terminate cleanup count:1
> blk:0
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http posted request:
> "/images/image001.jpg?"
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http terminate handler count:1
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http request count:1 blk:0
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http close request
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 http log handler
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 run cleanup: 000002DC83A7CBE8
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 file cleanup: fd:732
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 free: 000002DC83A87340
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 free: 000002DC83A7BC00, unused: 0
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 free: 000002DC83A7DC20, unused:
> 1167
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 close http connection: 500
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 select del event fd:500 ev:768
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 reusable connection: 0
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 free: 000002DC83A647C0
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 free: 000002DC83A8F350
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 free: 000002DC8332EAB0, unused:
> 12
> 2023/02/23 21:24:49 [debug] 4768#4528: *1 free: 000002DC83A7DA10, unused:
> 384
>
On Thu, Feb 23, 2023 at 9:32 PM Dan Swaney <justdan23 at gmail.com> wrote:
> I tried to reproduce it with a simple web page and I'm seeing the issue
> which I believe you are having.
>
> By default, NGINX seems to default to some limit of 15k on responses of
> images returned directly.
>
> While the error.log in debug mode shows it constructs the response with
> the right content length, it fails to return a response if the file is over
> 15k.
>
> I'm using a build I did from NGINX 1.23.3. While I know I have returned
> files larger than this in proxy mode. This seems to be happening in normal
> web server page mode.
>
> [image: image.png]
>
>
> I even tried to increase it by adding:
>
> location / {
>> root html;
>> index index.html index.htm favicon.ico;
>>
>> types {
>> text/html html;
>> image/gif gif;
>> application/octetstream jpg;
>> application/octetstream png;
>> }
>>
>> set $max_image_size 50000;
>> client_max_body_size 50000;
>> }
>>
>
> On Thu, Feb 23, 2023 at 8:09 PM Dan Swaney <justdan23 at gmail.com> wrote:
>
>> Try this:
>>
>> 1. At the very top of your nginx.conf file, add a definition:
>>
>> error_log logs/error.log debug;
>>
>> 2. Within your nginx install directory (where nginx.exe exists),
>> create a 'logs' folder if one does not exist.
>> 3. Restart nginx.exe (if running as a Windows Service with NSSM, then
>> restart the service).
>> 4. If NGINX does not start...
>> - Check the 'error.log' as it will tell you if your nginx.conf has
>> something weird in it that it doesn't like.
>> 5. If NGINX started successfully...
>> - Run your test from your client browser.
>> - Go back to the Server and check your 'error.log' and look for
>> the URL request.
>> - It will show you everything it does with routing or looking for
>> files.
>>
>>
>> On Thu, Feb 23, 2023 at 7:22 PM Saint Michael <venefax at gmail.com> wrote:
>>
>>> Now it remains broken and I have no idea how to fix it. I guess is
>>> caching bad copies of the pictures.
>>> I re-uploaded the two images.
>>> Kindly let me know if this is "normal"
>>> [image: image.png]
>>>
>>> On Thu, Feb 23, 2023 at 7:08 PM Dan Swaney <justdan23 at gmail.com> wrote:
>>>
>>>> Now your image001.jpg is returning your home page:
>>>> [image: image.png]
>>>>
>>>> If I had to guess, your location routing is re-routing all sub-urls to
>>>> your base URL at '/' which is defaulting to your index.html (or whatever
>>>> you have set for your default).
>>>> The tip was when it returns the content type as 'text/html' instead of
>>>> 'image/jpg'.
>>>>
>>>> As Maxim cited, your 'location' directives are for routing URL paths
>>>> and not files. Think of the external viewed URL and the internal routed
>>>> URL location.
>>>>
>>>> Looking at your previous cited partial nginx.conf:
>>>>
>>>> - root /static/duc/;
>>>> - I usually define this in my 'location' base section only and
>>>> drop the initial '/' if it is relative to where NGINX is running.
>>>> - I then don't need it in any other 'location' sections for
>>>> sub-folders which have different security.
>>>> - You have it defined in your 'server' section.
>>>>
>>>>
>>>> - default_type 'text/html; charset=UTF-8';
>>>> - I usually define this at the top of my 'http' section.
>>>> - Normally I use 'default_type application/octet-stream;'
>>>> - You have it defined in your 'server' section.
>>>> - I see it returning your images as 'text/html'.
>>>>
>>>> - try_files $uri $uri/ /index.html;
>>>> - I usually define a default 'index' if at the root and nothing
>>>> else is added.
>>>> - For example: 'index index.html;'
>>>> - Remove the try_files like recommended earlier.
>>>> - If you need to restrict access to specific folder URL
>>>> mappings, then define a location of the URL mapping and add one line...
>>>> - 'deny all;'
>>>> - Otherwise leave it open to all sub URLs by doing nothing
>>>> more but use a single 'location /' rule.
>>>>
>>>>
>>>> On Thu, Feb 23, 2023 at 6:15 PM Dan Swaney <justdan23 at gmail.com> wrote:
>>>>
>>>>> I just ran your test and it works fine from Chrome and in Visual
>>>>> Studio Code:
>>>>>
>>>>> > wget https://x3x.us/index_files/image001.jpg
>>>>>> StatusCode : 200
>>>>>> StatusCode : 200
>>>>>> StatusDescription : OK
>>>>>> Content : {255, 216, 255, 224...}
>>>>>> RawContent : HTTP/1.1 200 OK
>>>>>> Connection: keep-alive
>>>>>> Strict-Transport-Security: max-age=31536000;
>>>>>> includeSubDomains
>>>>>> Accept-Ranges: bytes
>>>>>> Content-Length: 8834
>>>>>> Content-Type: image/jpeg
>>>>>> Date: Thu, 23 Feb 2023 23...
>>>>>> Headers : {[Connection, keep-alive],
>>>>>> [Strict-Transport-Security, max-age=31536000; includeSubDomains],
>>>>>> [Accept-Ranges, bytes],
>>>>>> [Content-Length, 8834]...}
>>>>>> RawContentLength : 8834
>>>>>
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Feb 22, 2023 at 7:36 PM Saint Michael <venefax at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> a) The error does not have a single line.
>>>>>> b) restarting does not fix it
>>>>>> c) my nginx is no acting as proxy
>>>>>> d) it happened twice and both times I fixed it by turning gzip off,
>>>>>> restarting, and back on.
>>>>>> e) I also noticed that I requested the image file with wget, get a
>>>>>> full HTML file for the whole document, but named as if it were the image
>>>>>> file.
>>>>>>
>>>>>> wget https://x3x.us/index_files/image001.jpg
>>>>>> but `stat image001.jpg' showed it was the entire text HTML file.
>>>>>>
>>>>>> http {
>>>>>> client_max_body_size 1500M;
>>>>>> include mime.types;
>>>>>> # default_type application/octet-stream;
>>>>>>
>>>>>> #log_format main '$remote_addr - $remote_user [$time_local]
>>>>>> "$request" '
>>>>>> # '$status $body_bytes_sent "$http_referer" '
>>>>>> # '"$http_user_agent" "$http_x_forwarded_for"';
>>>>>>
>>>>>> #access_log logs/access.log main;
>>>>>> sendfile on;
>>>>>> tcp_nopush on;
>>>>>> tcp_nodelay on;
>>>>>> gzip on;
>>>>>> gzip_vary on;
>>>>>> gzip_min_length 10240;
>>>>>> gzip_proxied expired no-cache no-store private auth;
>>>>>> gzip_types text/plain text/css text/xml text/javascript
>>>>>> application/x-javascript application/xml;
>>>>>> gzip_disable "MSIE [1-6]\.";
>>>>>> #keepalive_timeout 0;
>>>>>> keepalive_timeout 65;
>>>>>> types_hash_max_size 2048;
>>>>>> proxy_headers_hash_max_size 1024;
>>>>>> proxy_headers_hash_bucket_size 128;
>>>>>> #gzip on;
>>>>>> # error_log /var/log/nginx/error.log debug;
>>>>>>
>>>>>> error_log /var/log/nginx/error.log error;
>>>>>> error_log /var/log/nginx/error.log crit;
>>>>>>
>>>>>> access_log /var/log/nginx/access.log;
>>>>>>
>>>>>> server {
>>>>>> add_header Strict-Transport-Security "max-age=31536000;
>>>>>> includeSubDomains" always;
>>>>>> default_type 'text/html; charset=UTF-8';
>>>>>> listen 208.78.161.6:80;
>>>>>> server_name x3x.us;
>>>>>>
>>>>>> # Redirect all domains to https://x3x.us
>>>>>> if ($scheme != "https") {
>>>>>> return 301 https://x3x.us$request_uri;
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> server {
>>>>>> add_header Strict-Transport-Security "max-age=31536000;
>>>>>> includeSubDomains" always;
>>>>>> default_type 'text/html; charset=UTF-8';
>>>>>> listen 208.78.161.6:443 ssl;
>>>>>> server_name x3x.us;
>>>>>>
>>>>>> ssl_certificate /etc/letsencrypt/live/x3x.us/fullchain.pem;
>>>>>> ssl_certificate_key /etc/letsencrypt/live/x3x.us/privkey.pem;
>>>>>>
>>>>>> root /static/duc/;
>>>>>>
>>>>>> location / {
>>>>>>
>>>>>> try_files $uri $uri/ /index.html;
>>>>>>
>>>>>>
>>>>>> }
>>>>>>
>>>>>> } #server
>>>>>>
>>>>>> } #http
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 22, 2023 at 7:17 PM Maxim Dounin <mdounin at mdounin.ru>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello!
>>>>>>>
>>>>>>> On Wed, Feb 22, 2023 at 02:46:29PM -0500, Saint Michael wrote:
>>>>>>>
>>>>>>> > It's not a misconfiguration, is a huge bug.
>>>>>>> > A wasted two days of sleep for something that is 100% a bug.
>>>>>>> > Please read here:
>>>>>>> >
>>>>>>> https://laracasts.com/discuss/channels/general-discussion/homestead-nginx-serving-wrong-images-and-only-cut-in-the-middle
>>>>>>> > He mentions the same exact problem and also he points to
>>>>>>> >
>>>>>>> https://tech.blog.aknin.name/2011/11/04/nginxgzip-module-might-silently-corrupt-data-upon-backend-failure/
>>>>>>> > where the author says that Niginx will not fix it.
>>>>>>> > So he already tried he was rebuffed.
>>>>>>>
>>>>>>> The fun fact is that the referenced article doesn't state "will
>>>>>>> not fix", but rather "not a top priority". Further, proper error
>>>>>>> propagation is available in nginx for about 10 years now, since
>>>>>>> 2013 (http://hg.nginx.org/nginx/rev/d3eab5e2df5f, nginx 1.5.3).
>>>>>>> Quoting CHANGES:
>>>>>>>
>>>>>>> *) Change: now after receiving an incomplete response from a
>>>>>>> backend
>>>>>>> server nginx tries to send an available part of the response
>>>>>>> to a
>>>>>>> client, and then closes client connection.
>>>>>>>
>>>>>>> As long as nginx have an information about an error, it will
>>>>>>> preserve this information and propagate it to the client.
>>>>>>>
>>>>>>> Also note that it is only expected to make a difference if you are
>>>>>>> using nginx as a proxy, not to directly serve files. And only in
>>>>>>> case of errors. That is, if you are seeing the behaviour
>>>>>>> described, it might be a good idea to focus on the errors in the
>>>>>>> first place.
>>>>>>>
>>>>>>> I don't think it's anyhow related though, as switching gzip off
>>>>>>> and back on, as seems to be "the fix" described in the first link,
>>>>>>> is not going to help with anything. The important part is likely
>>>>>>> "restarted the server", so I would rather assume that "the server"
>>>>>>> (not sure if it refers to nginx or the whole server) was using an
>>>>>>> incorrect configuration and/or was out of some resources, and
>>>>>>> restart fixed it.
>>>>>>>
>>>>>>> Summing the above, if you want to find out what goes wrong in your
>>>>>>> case - you may want to provide more details. If you don't, nobody
>>>>>>> will be able to do it, unfortunately.
>>>>>>>
>>>>>>> The most basic thing I would recommend in the first place is to
>>>>>>> look into nginx error log, it is likely to contain important
>>>>>>> information if something goes wrong.
>>>>>>>
>>>>>>> --
>>>>>>> Maxim Dounin
>>>>>>> http://mdounin.ru/
>>>>>>> _______________________________________________
>>>>>>> nginx mailing list
>>>>>>> nginx at nginx.org
>>>>>>> https://mailman.nginx.org/mailman/listinfo/nginx
>>>>>>>
>>>>>> _______________________________________________
>>>>>> nginx mailing list
>>>>>> nginx at nginx.org
>>>>>> https://mailman.nginx.org/mailman/listinfo/nginx
>>>>>>
>>>>> _______________________________________________
>>>> nginx mailing list
>>>> nginx at nginx.org
>>>> https://mailman.nginx.org/mailman/listinfo/nginx
>>>>
>>> _______________________________________________
>>> nginx mailing list
>>> nginx at nginx.org
>>> https://mailman.nginx.org/mailman/listinfo/nginx
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20230223/5d0aef19/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 47810 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20230223/5d0aef19/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 282588 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20230223/5d0aef19/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 111869 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20230223/5d0aef19/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 193204 bytes
Desc: not available
URL: <http://mailman.nginx.org/pipermail/nginx/attachments/20230223/5d0aef19/attachment-0007.png>
More information about the nginx
mailing list