Image Hosting
Dennis Jacobfeuerborn
dennisml at conversis.de
Fri Oct 15 12:30:55 MSD 2010
But what is your problem exactly with the I/O?
The numbers you posted look fine. What are the symptoms you are seeing?
As Maxim pointed out iowait is fairly irrelevant as long as the requests
get reponded to in a timely manner.
Regards,
Dennis
On 10/15/2010 03:10 AM, iptablez at yahoo.com wrote:
> Hi Dennis,
>
> Sorry I think that word is already mentioned that I have a problem with disk i/o.
>
> That's why I'm asking wether I need onother software to be used together with nginx so I can give a best performance
>
>
> -----Original Message-----
> From: Dennis Jacobfeuerborn<dennisml at conversis.de>
> Date: Thu, 14 Oct 2010 19:43:17
> To:<nginx at nginx.org>
> Reply-To: nginx at nginx.org
> Subject: Re: Image Hosting
>
> This looks fairly benign to me. You get lots of random reads so you can
> expect the cpu to wait for i/o occasionally.
> One remedy is to spread the data across more disks and the other is to put
> more ram into the machine so more data can be cached.
>
> I have a 30TB system using 48 disks and two controllers here where the I/O
> wait is on average 85% yet it delivers the data with fairly low latency.
>
> You never actually mentioned what your real problem is (you just said "Now
> on we're having problem with disk i/o".)
>
> Regards,
> Dennis
>
> On 10/14/2010 05:05 PM, Indo Php wrote:
>> the output of iotop
>> Total DISK READ: 15.67 M/s | Total DISK WRITE: 0.00 B/s
>>
>> the output of vmstat 10 10
>> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
>> r b swpd free buff cache si so bi bo in cs us sy id wa
>> 0 1 31736 3518240 486048 2719488 0 1 646 321 70 39 1 4 88 7
>> 0 0 31736 3474616 489792 2763936 0 0 10909 96 12010 2568 1 5 76 17
>> 0 0 31736 3428900 492456 2797516 0 0 11062 152 12099 2546 1 6 78 15
>> 0 1 31736 3382620 494924 2833764 0 0 12728 136 12709 2580 1 7 76 16
>> 0 1 31696 3325956 497404 2872368 4 0 11678 244 12532 2571 2 6 77 15
>> 0 0 31696 3288048 500144 2906464 0 0 9947 178 12187 2539 1 6 80 13
>> 0 1 31684 3250064 503244 2947148 1 0 10628 166 12279 2613 1 7 78 15
>> 0 0 31680 3180800 507244 2995880 0 0 12200 186 12512 2680 1 7 73 19
>> 0 0 31680 3126368 510364 3047976 0 0 12083 149 12641 2738 2 6 70 21
>> 0 1 31680 3068704 513724 3093616 0 0 11916 176 12648 2799 1 7 74 18
>>
>>
>> ---------------------------------------------------------------------------
>> *From:* Dennis Jacobfeuerborn<dennisml at conversis.de>
>> *To:* nginx at nginx.org
>> *Sent:* Thu, October 14, 2010 9:57:26 PM
>> *Subject:* Re: Image Hosting
>>
>> Please re-run the vmstat as "vmstat 10 10". That should give a better
>> glimpse at what the system is doing.
>>
>> On 10/14/2010 04:26 PM, Indo Php wrote:
>> > Here's the output of "iostat -d 60 2"
>> > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
>> > sda 85.58 4947.94 2619.30 593632866 314252346
>> >
>> > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
>> > sda 114.41 8395.93 339.01 503840 20344
>> >
>> > here's the output of vmstat
>> > procs -----------memory---------- ---swap-- -----io---- -system--
>> ----cpu----
>> > r b swpd free buff cache si so bi bo in cs us sy id wa
>> > 0 1 332 63136 628344 6085040 0 0 617 324 17 27 1 4 88 6
>> >
>> > ---------------------------------------------------------------------------
>> > *From:* Dennis Jacobfeuerborn<dennisml at conversis.de
>> <mailto:dennisml at conversis.de>>
>> > *To:* nginx at nginx.org<mailto:nginx at nginx.org>
>> > *Sent:* Thu, October 14, 2010 7:54:26 PM
>> > *Subject:* Re: Image Hosting
>> >
>> > Ah, sorry I somehow didn't catch that last line with the actual HW setup.
>> >
>> > If my math skills don't fail me (and they very well might) the traffic and
>> > image size data mean that you serve about 300 images per second.
>> > Can you post a few lines of vmstat output and perhaps the output of "iostat
>> > -d 60 2"?
>> >
>> > Also how are the hits distributed across the whole pool of images? Are
>> > these hits truly random or are some images hit significantly more often
>> > than others?
>> >
>> > More ram would obviously take some pressure off the disks if they are
>> > really the problem.
>> >
>> > Regards,
>> > Dennis
>> >
>> > On 10/14/2010 02:26 PM, iptablez at yahoo.com<mailto:iptablez at yahoo.com>
>> <mailto:iptablez at yahoo.com<mailto:iptablez at yahoo.com>> wrote:
>> > > Actually the image size is arround 100KB each. The server is running in
>> > 250Mbps traffic.
>> > > I already described the disk I'm using is scsi 15K RPM in raid 0
>> > >
>> > >
>> > > Sent from my BlackBerry®
>> > > powered by Sinyal Kuat INDOSAT
>> > >
>> > > -----Original Message-----
>> > > From: Dennis Jacobfeuerborn<dennisml at conversis.de
>> <mailto:dennisml at conversis.de>
>> > <mailto:dennisml at conversis.de<mailto:dennisml at conversis.de>>>
>> > > Date: Thu, 14 Oct 2010 14:23:26
>> > > To:<nginx at nginx.org<mailto:nginx at nginx.org> <mailto:nginx at nginx.org
>> <mailto:nginx at nginx.org>>>
>> > > Reply-To: nginx at nginx.org<mailto:nginx at nginx.org>
>> <mailto:nginx at nginx.org<mailto:nginx at nginx.org>>
>> > > Subject: Re: Image Hosting
>> > >
>> > > But if those 5,000 req/s each hit a different 1mb image then you need to
>> > > read 5gb of data from you storage which if your storage is just a
>> plain old
>> > > sata disk is going to be a huge problem.
>> > >
>> > > Unfortunately Indo Php is not providing nearly enough information to give
>> > > any sort of advice. If he is lucky then putting lots of ram in the machine
>> > > for pagecache can help here if the cache hit ratio is good but if it
>> is not
>> > > then he probably has to distribute the I/O across more spindles and go for
>> > > a raid with lots of disks.
>> > >
>> > > Regards,
>> > > Dennis
>> > >
>> > > On 10/14/2010 10:33 AM, SplitIce wrote:
>> > >> Hmm well talking from experience nginx can serve upwards of 5,000
>> req/s for
>> > >> static files if configured right. Ill leave this for Igor or someone more
>> > >> expericed to reply to.
>> > >>
>> > >> On Thu, Oct 14, 2010 at 7:28 PM, Indo Php<iptablez at yahoo.com
>> <mailto:iptablez at yahoo.com>
>> > <mailto:iptablez at yahoo.com<mailto:iptablez at yahoo.com>>
>> > >> <mailto:iptablez at yahoo.com<mailto:iptablez at yahoo.com>
>> <mailto:iptablez at yahoo.com<mailto:iptablez at yahoo.com>>>> wrote:
>> > >>
>> > >> i tried to use AIO, but it seems my server is getting slower
>> > >>
>> > >>
>> ---------------------------------------------------------------------------
>> > >> *From:* SplitIce<mat999 at gmail.com<mailto:mat999 at gmail.com>
>> > <mailto:mat999 at gmail.com
>> <mailto:mat999 at gmail.com>><mailto:mat999 at gmail.com
>> <mailto:mat999 at gmail.com> <mailto:mat999 at gmail.com<mailto:mat999 at gmail.com>>>>
>> > >> *To:* nginx at nginx.org<mailto:nginx at nginx.org>
>> <mailto:nginx at nginx.org<mailto:nginx at nginx.org>><mailto:nginx at nginx.org
>> <mailto:nginx at nginx.org>
>> > <mailto:nginx at nginx.org<mailto:nginx at nginx.org>>>
>> > >> *Sent:* Thu, October 14, 2010 2:47:55 PM
>> > >> *Subject:* Re: Image Hosting
>> > >>
>> > >> Try any of these:
>> > >> 1) Compile nginx with AIO support
>> > >> 2) Try directio option
>> > >> 3) Try sendfile option
>> > >>
>> > >> On Thu, Oct 14, 2010 at 6:36 PM, Indo Php<iptablez at yahoo.com
>> <mailto:iptablez at yahoo.com>
>> > <mailto:iptablez at yahoo.com<mailto:iptablez at yahoo.com>>
>> > >> <mailto:iptablez at yahoo.com<mailto:iptablez at yahoo.com>
>> <mailto:iptablez at yahoo.com<mailto:iptablez at yahoo.com>>>> wrote:
>> > >>
>> > >> hi there,
>> > >>
>> > >> we have an image hosting, and of course we're using Nginx!
>> > >>
>> > >> Now on we're having problem with disk i/o. With 200GB of images.
>> > >> Is there any recommendation for this matter? Nginx configuration or
>> > >> Hardware/other software configuration?
>> > >>
>> > >> We already used Dual Quad Core 3Ghz, 8 GB RAM, 6x73GB 15k SCSI server
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>_______________________________________________
>> > >> nginx mailing list
>> > >> nginx at nginx.org<mailto:nginx at nginx.org> <mailto:nginx at nginx.org
>> <mailto:nginx at nginx.org>><mailto:nginx at nginx.org<mailto:nginx at nginx.org>
>> > <mailto:nginx at nginx.org<mailto:nginx at nginx.org>>>
>> >> > http://nginx.org/mailman/listinfo/nginx
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>_______________________________________________
>> > >> nginx mailing list
>> > >> nginx at nginx.org<mailto:nginx at nginx.org> <mailto:nginx at nginx.org
>> <mailto:nginx at nginx.org>><mailto:nginx at nginx.org<mailto:nginx at nginx.org>
>> > <mailto:nginx at nginx.org<mailto:nginx at nginx.org>>>
>> > >> http://nginx.org/mailman/listinfo/nginx
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> --
>> >> > Warez Scene<http://thewarezscene.org> Free Rapidshare Downloads
>> >> > <http://www.nexusddl.com>
>> > >>
>> > >>
>> > >>
>> > >>_______________________________________________
>> > >> nginx mailing list
>> > >> nginx at nginx.org<mailto:nginx at nginx.org> <mailto:nginx at nginx.org
>> <mailto:nginx at nginx.org>>
>> > >> http://nginx.org/mailman/listinfo/nginx
>> > >
>> > >
>> > >_______________________________________________
>> > > nginx mailing list
>> > > nginx at nginx.org<mailto:nginx at nginx.org> <mailto:nginx at nginx.org
>> <mailto:nginx at nginx.org>>
>> > > http://nginx.org/mailman/listinfo/nginx
>> > >_______________________________________________
>> > > nginx mailing list
>> > > nginx at nginx.org<mailto:nginx at nginx.org> <mailto:nginx at nginx.org
>> <mailto:nginx at nginx.org>>
>> > > http://nginx.org/mailman/listinfo/nginx
>> >
>> >
>> >_______________________________________________
>> > nginx mailing list
>> > nginx at nginx.org<mailto:nginx at nginx.org> <mailto:nginx at nginx.org
>> <mailto:nginx at nginx.org>>
>> > http://nginx.org/mailman/listinfo/nginx
>> >
>> >
>> >
>> >_______________________________________________
>> > nginx mailing list
>> > nginx at nginx.org<mailto:nginx at nginx.org>
>> > http://nginx.org/mailman/listinfo/nginx
>>
>>
>> _______________________________________________
>> nginx mailing list
>> nginx at nginx.org<mailto:nginx at nginx.org>
>> http://nginx.org/mailman/listinfo/nginx
>>
>>
>>
>> _______________________________________________
>> nginx mailing list
>> nginx at nginx.org
>> http://nginx.org/mailman/listinfo/nginx
>
>
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://nginx.org/mailman/listinfo/nginx
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://nginx.org/mailman/listinfo/nginx
More information about the nginx
mailing list