sflow module and increasing useragent
neil.mckee.ca at gmail.com
Thu Aug 11 23:13:50 UTC 2011
On Aug 11, 2011, at 2:46 PM, Mark Moseley wrote:
> On Wed, Aug 10, 2011 at 12:42 PM, Neil Mckee <neil.mckee.ca at gmail.com> wrote:
>> On Aug 10, 2011, at 9:55 AM, Mark Moseley wrote:
>>> On Tue, Aug 9, 2011 at 10:37 PM, Neil Mckee <neil.mckee.ca at gmail.com> wrote:
>>>> It looks like it may be (erroneously!) ignoring that limit and sending the whole user-agent string. Thanks for pointing it out.
>>>> If you are using sflowtool to print the output at the collector then it's probably chopping the field there, so you would just need to tweak sflowtool/src/sflow.h file and recompile sflowtool.
>>>> The user-agent can be kilobytes long in some cases. How many bytes is enough? Please submit comments to the HTTP thread on http://groups.google.com/group/sflow.
>>>> On Aug 9, 2011, at 6:21 PM, Mark Moseley wrote:
>>>>> If I wanted to increase the amount of bytes logged by the sflow module
>>>>> for the user agent to, say, 128 bytes, what would I need to change?
>>>>> I tried changing this:
>>>>> (in ngx_http_sflow.h)
>>>>> #define SFLHTTP_MAX_USERAGENT_LEN 128
>>>>> but I still get the user agent truncated at 64 bytes. Nor does it seem
>>>>> like SFLHTTP_MAX_USERAGENT_LEN is used anywhere in the code (or any
>>>>> SFLHTTP_MAX_* define).
>>>>> I'm sure I could whack on it long enough to get it to work, but I ask
>>>>> mainly because I don't want to cause some buffer/format/etc overflow
>>>>> somewhere else down the line. Thanks!
>>> Thanks for the info. That did the trick. I wrongly assumed the limit
>>> was being applied in the module itself.
>>> BTW, is there anywhere that has info on APIs for host sflow? I've been
>>> googling to no avail. The perl Net::sFlow module chokes badly
>>> (presumably just expecting standard sflow), and I've not been able to
>>> track down a python module (which is what I'm really interested in).
>>> Just asking since we have this thread going, but I can repost to the
>>> google group if you think it'd be better asked there. Thanks!
>> Do you mean you are looking for a Python equivalent of what sflowtool.c does?
>> The data is all XDR-encoded, so you might start with Python's "xdrlib". However it may be easier and more compact to just unpack the data manually. The C implementation for Ganglia-gmond is an example of that. It is much more compact than sflowtool.c, and might be a better place to start if you just want the sFlow-HOST structures:
>> (see the "process_sflow_datagram()" function near the bottom.)
>> sFlow XDR specs that have been finalized are published here:
>> P.S. The Perl library should be able to skip over structures it doesn't recognize, so you might want to point that out to it's author(s).
> Cool, thanks, I'll take a look at that -- though working with XDR
> looks like it could be somewhat painful :)
> On a related note, I get a trickle of errors like this in the nginx error log:
> sFlow agent error: sfl_agent_error: receiver: flow sample too big for datagram
> which must be hitting the in-code limit of 1400 bytes. Would it be
> quite horrible if I were to bump up SFL_DEFAULT_DATAGRAM_SIZE over
> 1460? I'm assuming under normal conditions, it'll just fragment, which
> I'm ok with at the rate they're occurring now. But again, I'm worried
> about some data structure in the code (that my casual reading of the
> code isn't turning up) will overflow.
IT looks like you'd have to bump up both SFL_MAX_DATAGRAM_SIZE and SFL_DEFAULT_DATAGRAM_SIZE. For example:
#define SFL_MAX_DATAGRAM_SIZE 3000
#define SFL_MIN_DATAGRAM_SIZE 200
#define SFL_DEFAULT_DATAGRAM_SIZE 3000
If you are not using jumbo frames and packet-loss-in-transit ever hits 50% then you might have a problem getting data through to the collector (just when you really needed it) so in the end the right solution is to apply the length-limits as proposed on the sFlow mailing list. This can be done in sfwb_sample_http() at the point where the string lengths are filled in for the sample. I intend to put that fix in very soon, and add the missing "X-Forwarded-For" and "req_bytes" fields too. If you think there are any other fields or counters missing then now is a good time to chime in.
> Cool module, btw!
> nginx mailing list
> nginx at nginx.org
More information about the nginx