Debugging performance under high load

[agentzh]

Hello!

On Sun, Feb 17, 2013 at 3:52 AM, fluffypony wrote:
> How do I debug the poor
> performance so I at least know what to fix? Is there a way to step through
> exactly what is happening in a request under load to see where it's being
> delayed? I'd like to get it up to at least 1k RPS if not more, and I believe
> the server and the bandwidth are up to the task.
>

We've been using the Flame Graph tools to profile our online Nginx on
Linux in production.

It is a great tool to find out which part (be it a function or a code
path) is hot and slow (on various levels like the kernelspace, the C
level in userspace, or even high levels on scripting languages like
Lua).

See the ngx-sample-bt tool in my Nginx Systemtap Toolkit:

    https://github.com/agentzh/nginx-systemtap-toolkit#ngx-sample-bt

There's no need to recompile or restart your Nginx for the live
profiling. Just ensure that your Nginx executable is not stripped (the
DWARF debug symbols should be enabled by Nginx by default).

Another prerequisite to use tools in my Nginx Systemtap Toolkit is
that you have a working systemtap installation in your Linux system,
see the documentation for details:

    https://github.com/agentzh/nginx-systemtap-toolkit#prerequisites

Best regards,
-agentzh