memcached problems...

[Marcus Clyne]

Igor Sysoev wrote:
> On Mon, Aug 31, 2009 at 05:34:07PM -0400, JG wrote:
>
>   
>> normally, i whould agree. but in this special case the system has trouble to handle the high io load due to really lots of files (about 8 million.... i know... crazy, not my content).
>>
>> to get a workaround for the hd io load, i just wanted to get nginx to load the swf and flv files into memcached which should be possible for my understanding. 
>>
>> i think i found one of my problems, the missing fallback server if the content isnt already in memcached. 
>>
>> but now i get 502 bad gateway error messages, and such strange error log entries
>>
>> 2009/08/31 23:21:17  30612#0: 8192 worker_connections is not enough while accepting new connection on 0.0.0.0:81
>>
>> ? no connections yet ... no production
>>
>> but anyway, i trimmed down my config to a minimum for testing
>>
>>
>>     server {
>>         listen       81;
>>         server_name  _;
>>
>>         #access_log  logs/host.access.log  main;
>>
>>         location / {
>>             set $memcached_key  $uri;
>>             memcached_pass      127.0.0.1:11211;
>>             error_page 404 502  = /fallback;
>> #           default_type        text/html;
>>
>>             root   /usr/share/nginx/html;
>>             index  index.html index.htm;
>>         }
>>
>>         location /fallback {
>>             proxy_pass http://127.0.0.1:81;
>>         }
>>
>>         location ~ \.php$ {
>>             set $memcached_key  $uri;
>>             memcached_pass      127.0.0.1:11211;
>>
>>             root           /home/www/htdocs;
>>             fastcgi_pass   127.0.0.1:9000;
>>             fastcgi_index  index.php;
>>             fastcgi_param  SCRIPT_FILENAME  /home/www/htdocs$fastcgi_script_name;
>>             include        fastcgi_params;
>>         }
>>
>>
>>
>> PHP works well, but it doesnt seem to get stored in memcached, why? and, for example if i try to download a zip file i get this in the error log
>>
>> 2009/08/31 23:30:16  30815#0: *12285 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 127.0.0.1, server: _, request: "GET /koch/_banner/paldauer/paldauer.zip HTTP/1.0", upstream: "http://127.0.0.1:81/yes/_this/doesnt/work.zip", host: "127.0.0.1:81", referrer: "http://ftp.nastyhost.de:81/koch/_banner/paldauer/index.php"
>>
>> please igor, anyhow if you got an idea how i can get nginx to use memcached to store content... 
>>     
>
> nginx can not store data in memcached, it can ony get data from it.
> As to "recv() failed (104: Connection reset by peer)" this is some connection
> problem between nginx and 127.0.0.1:81.
>
> However, as I already said, memcached will not resolve your problem.
> You just will waste CPU time and memory. Instead you should tune kernel
> to allow cache as much as possible. How much is the host physical memory
> and how much is whole size of flv/swf content ?
>
> I can not say about Linux, but in FreeBSD this can be done by increasing
> "sysctl kern.maxvnodes", which is 100,000 vnodes by default. A kernel
> stores file pages binding them to a vnode.
>   
On Linux you want to increase either

/proc/sys/fs/inode-max    (if you have it - I don't on my Ubunutu 9.04 
x64 - not sure why)

or

/proc/sys/fs/file-max

e.g.

You can view these by e.g.:

cat /proc/sys/fs/inode-max

or change them by e.g.:
 
echo 1000000 > /proc/sys/fs/file-max

If you use the inode-max option, a number 3-4 x as big as the file-max 
would be normal.

AFAIK this will reset upon reboot, so you'd probably want to add it to a 
startup script.

Marcus.