sflow module and increasing useragent

[Neil Mckee]

On Aug 11, 2011, at 2:46 PM, Mark Moseley wrote:

> On Wed, Aug 10, 2011 at 12:42 PM, Neil Mckee <neil.mckee.ca at gmail.com> wrote:
>> 
>> On Aug 10, 2011, at 9:55 AM, Mark Moseley wrote:
>> 
>>> On Tue, Aug 9, 2011 at 10:37 PM, Neil Mckee <neil.mckee.ca at gmail.com> wrote:
>>>> It looks like it may be (erroneously!) ignoring that limit and sending the whole user-agent string.  Thanks for pointing it out.
>>>> 
>>>> If you are using sflowtool to print the output at the collector then it's probably chopping the field there,  so you would just need to tweak sflowtool/src/sflow.h file and recompile sflowtool.
>>>> 
>>>> The user-agent can be kilobytes long in some cases.   How many bytes is enough?   Please submit comments to the HTTP thread on http://groups.google.com/group/sflow.
>>>> 
>>>> Neil
>>>> 
>>>> 
>>>> On Aug 9, 2011, at 6:21 PM, Mark Moseley wrote:
>>>> 
>>>>> If I wanted to increase the amount of bytes logged by the sflow module
>>>>> for the user agent to, say, 128 bytes, what would I need to change?
>>>>> 
>>>>> I tried changing this:
>>>>> 
>>>>> (in ngx_http_sflow.h)
>>>>> #define SFLHTTP_MAX_USERAGENT_LEN 128
>>>>> 
>>>>> but I still get the user agent truncated at 64 bytes. Nor does it seem
>>>>> like SFLHTTP_MAX_USERAGENT_LEN is used anywhere in the code (or any
>>>>> SFLHTTP_MAX_* define).
>>>>> 
>>>>> I'm sure I could whack on it long enough to get it to work, but I ask
>>>>> mainly because I don't want to cause some buffer/format/etc overflow
>>>>> somewhere else down the line. Thanks!
>>> 
>>> Thanks for the info. That did the trick. I wrongly assumed the limit
>>> was being applied in the module itself.
>>> 
>>> BTW, is there anywhere that has info on APIs for host sflow? I've been
>>> googling to no avail. The perl Net::sFlow module chokes badly
>>> (presumably just expecting standard sflow), and I've not been able to
>>> track down a python module (which is what I'm really interested in).
>>> Just asking since we have this thread going, but I can repost to the
>>> google group if you think it'd be better asked there. Thanks!
>> 
>> Do you mean you are looking for a Python equivalent of what sflowtool.c does?
>> 
>> The data is all XDR-encoded,  so you might start with Python's  "xdrlib".   However it may be easier and more compact to just unpack the data manually.  The C implementation for Ganglia-gmond is an example of that.  It is much more compact than sflowtool.c,  and might be a better place to start if you just want the sFlow-HOST structures:
>> http://ganglia.svn.sourceforge.net/viewvc/ganglia/trunk/monitor-core/gmond/sflow.c?revision=2638&view=markup
>> (see the "process_sflow_datagram()" function near the bottom.)
>> 
>> sFlow XDR specs that have been finalized are published here:
>> http://www.sflow.org/developers/specifications.php
>> 
>> Neil
>> 
>> P.S. The Perl library should be able to skip over structures it doesn't recognize,  so you might want to point that out to it's author(s).
> 
> Cool, thanks, I'll take a look at that -- though working with XDR
> looks like it could be somewhat painful :)
> 
> On a related note, I get a trickle of errors like this in the nginx error log:
> 
> sFlow agent error: sfl_agent_error: receiver: flow sample too big for datagram
> 
> which must be hitting the in-code limit of 1400 bytes. Would it be
> quite horrible if I were to bump up SFL_DEFAULT_DATAGRAM_SIZE over
> 1460? I'm assuming under normal conditions, it'll just fragment, which
> I'm ok with at the rate they're occurring now. But again, I'm worried
> about some data structure in the code (that my casual reading of the
> code isn't turning up) will overflow.

IT looks like you'd have to bump up both SFL_MAX_DATAGRAM_SIZE and SFL_DEFAULT_DATAGRAM_SIZE.  For example:

#define SFL_MAX_DATAGRAM_SIZE 3000
#define SFL_MIN_DATAGRAM_SIZE 200
#define SFL_DEFAULT_DATAGRAM_SIZE 3000

If you are not using jumbo frames and packet-loss-in-transit ever hits 50% then you might have a problem getting data through to the collector (just when you really needed it) so in the end the right solution is to apply the length-limits as proposed on the sFlow mailing list.  This can be done in sfwb_sample_http() at the point where the string lengths are filled in for the sample.  I intend to put that fix in very soon,  and add the  missing "X-Forwarded-For" and "req_bytes" fields too.   If you think there are any other fields or counters missing then now is a good time to chime in.

Neil

> 
> Cool module, btw!

:)

> 
> Thanks!
> 
> _______________________________________________
> nginx mailing list
> nginx at nginx.org
> http://mailman.nginx.org/mailman/listinfo/nginx