Mailing List Archive

Network Traffic Collection
Hello,

I am trying to collect traffic traffic from pcap file and store it in
a database but really confused how to organize it. Should I organize
it on connection basis/ flow basis or IP basis.

It might be an effort to write a customized traffic analysis tool like
wireshark with only required functionality. I would really appreciate
if someone can give me direction on write way of organizing the data
because right now I only see individual packets and no way of putting
them in some order.

Best,
Ali
Re: Network Traffic Collection [ In reply to ]
On 2012-02-23 21:11 , Maverick wrote:
> Hello,
>
> I am trying to collect traffic traffic from pcap file and store it in
> a database but really confused how to organize it. Should I organize
> it on connection basis/ flow basis or IP basis.
>
> It might be an effort to write a customized traffic analysis tool like
> wireshark with only required functionality. I would really appreciate
> if someone can give me direction on write way of organizing the data
> because right now I only see individual packets and no way of putting
> them in some order.

Does this all not completely depend on what you actually want to do with
it? You might want to start there instead of the other way around.

Greets,
Jeroen
Re: Network Traffic Collection [ In reply to ]
I want to be able to see information like how much traffic an ip send
over a period of time, what machines it talked to etc from this
perspective it should be IP based but I would really like to know how
other people do it.

Best,
Ali

On Thu, Feb 23, 2012 at 3:14 PM, Jeroen Massar <jeroen@unfix.org> wrote:
> On 2012-02-23 21:11 , Maverick wrote:
>> Hello,
>>
>> I am trying to collect traffic traffic from pcap file and store it in
>> a database but really confused how to organize it. Should I organize
>> it on connection basis/ flow basis or IP basis.
>>
>> It might be an effort to write a customized traffic analysis tool like
>> wireshark with only required functionality. I would really appreciate
>> if someone can give me direction on write way of organizing the data
>> because right now I only see individual packets and no way of putting
>> them in some order.
>
> Does this all not completely depend on what you actually want to do with
> it? You might want to start there instead of the other way around.
>
> Greets,
>  Jeroen
>
RE: Network Traffic Collection [ In reply to ]
Netflow + netflow collector.

Ken Matlock
Network Analyst
Systems and Technology Service Center
Sisters of Charity of Leavenworth Health System
12600 W. Colfax, Suite A-500
Lakewood, CO 80215

303-467-4671
matlockk@exempla.org

-----Original Message-----
From: Maverick [mailto:myeaddress@gmail.com]
Sent: Thursday, February 23, 2012 1:19 PM
To: Jeroen Massar
Cc: nanog@nanog.org
Subject: Re: Network Traffic Collection

I want to be able to see information like how much traffic an ip send over a period of time, what machines it talked to etc from this perspective it should be IP based but I would really like to know how other people do it.

Best,
Ali

On Thu, Feb 23, 2012 at 3:14 PM, Jeroen Massar <jeroen@unfix.org> wrote:
> On 2012-02-23 21:11 , Maverick wrote:
>> Hello,
>>
>> I am trying to collect traffic traffic from pcap file and store it in
>> a database but really confused how to organize it. Should I organize
>> it on connection basis/ flow basis or IP basis.
>>
>> It might be an effort to write a customized traffic analysis tool
>> like wireshark with only required functionality. I would really
>> appreciate if someone can give me direction on write way of
>> organizing the data because right now I only see individual packets
>> and no way of putting them in some order.
>
> Does this all not completely depend on what you actually want to do
> with it? You might want to start there instead of the other way around.
>
> Greets,
>  Jeroen
>

*** Exempla Confidentiality Notice *** The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any other dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify me immediately by replying to the message and deleting it from your computer. Thank you. *** Exempla Confidentiality Notice ***
Re: Network Traffic Collection [ In reply to ]
On Thu, Feb 23, 2012 at 12:19 PM, Maverick <myeaddress@gmail.com> wrote:
> I want to be able to see information like how much traffic an ip send
> over a period of time, what machines it talked to etc from this
> perspective it should be IP based but I would really like to know how
> other people do it.
>


Run argus on a span port.

-Suresh
Re: Network Traffic Collection [ In reply to ]
Random thought, anyone ever used Splunk for this kind of thing?

-mike

Sent from my iPhone

On Feb 23, 2012, at 10:30, Suresh Rajagopalan <sraja97@gmail.com> wrote:

> On Thu, Feb 23, 2012 at 12:19 PM, Maverick <myeaddress@gmail.com> wrote:
>> I want to be able to see information like how much traffic an ip send
>> over a period of time, what machines it talked to etc from this
>> perspective it should be IP based but I would really like to know how
>> other people do it.
>>
>
>
> Run argus on a span port.
>
> -Suresh
>
Re: Network Traffic Collection [ In reply to ]
Splunk is an amazing tool and did an awesome thing and introduced a free license in 4.3.

I'm using it at two sites now and I'm loving it!

On 2012-02-23, at 3:34 PM, Mike Lyon wrote:

> Random thought, anyone ever used Splunk for this kind of thing?
>
> -mike
>
> Sent from my iPhone
>
> On Feb 23, 2012, at 10:30, Suresh Rajagopalan <sraja97@gmail.com> wrote:
>
>> On Thu, Feb 23, 2012 at 12:19 PM, Maverick <myeaddress@gmail.com> wrote:
>>> I want to be able to see information like how much traffic an ip send
>>> over a period of time, what machines it talked to etc from this
>>> perspective it should be IP based but I would really like to know how
>>> other people do it.
>>>
>>
>>
>> Run argus on a span port.
>>
>> -Suresh
>>
>
Re: Network Traffic Collection [ In reply to ]
On 2012-02-23 21:34 , Mike Lyon wrote:
> Random thought, anyone ever used Splunk for this kind of thing?

Various folks have, the problem of course comes down to processing
power, thus you'll need to throw a lot of hardware against it to be able
to process traffic in a decent network.

Check http://www.raffy.ch/ and http://pixlcloud.com/ etc for more
details about this.

Greets,
Jeroen
Re: Network Traffic Collection [ In reply to ]
Run it with hadoop in EC2?

Sent from my iPhone

On Feb 23, 2012, at 10:52, Jeroen Massar <jeroen@unfix.org> wrote:

> On 2012-02-23 21:34 , Mike Lyon wrote:
>> Random thought, anyone ever used Splunk for this kind of thing?
>
> Various folks have, the problem of course comes down to processing
> power, thus you'll need to throw a lot of hardware against it to be able
> to process traffic in a decent network.
>
> Check http://www.raffy.ch/ and http://pixlcloud.com/ etc for more
> details about this.
>
> Greets,
> Jeroen
Re: Network Traffic Collection [ In reply to ]
On Thu, 23 Feb 2012, Maverick wrote:

> I want to be able to see information like how much traffic an ip send
> over a period of time, what machines it talked to etc from this
> perspective it should be IP based but I would really like to know how
> other people do it.

Truth is that most people probably don't do it, beyond temporary, ad-hoc
deployments, to solve a specific problem at a specific point in time.
Traffic capture and analysis doesn't scale too well into multi-Gb/s
service provider environments.

Netflow tools are an option if 'reasonably accurate' is good enough for
your needs.

jms
Re: Network Traffic Collection [ In reply to ]
----------- myeaddress@gmail.com wrote: ----------
From: Maverick <myeaddress@gmail.com>

>> It might be an effort to write a customized traffic analysis tool like
>> wireshark with only required functionality. I would really appreciate


I want to be able to see information like how much traffic an ip send
over a period of time, what machines it talked to etc from this
perspective it should be IP based but I would really like to know how
other people do it.
-------------------------------------------------


Wouldn't Wireshark provide this for you? In particular, the "Conversations"
tool under the "Statistics" drop down menu? It adds data to the tool in
real time. If you want a graphical output the I/O graphs also under the
"Statistics" menu can graph all, or slices of the data in the main
Wireshark output.

scott
Re: Network Traffic Collection [ In reply to ]
Netflow / Sflow with one of the fallowing software packages

http://www.plixer.com/products/netflow-sflow/scrutinizer-netflow-sflow.php
http://www.solarwinds.com/NetFlow

http://www.arbornetworks.com/
Or the hand full of other open source options out there.



Carlos Alcantar
Race Communications / Race Team Member
101 Haskins Way, So. San Francisco, CA. 94080
Phone: +1 415 376 3314 / carlos@race.com / http://www.race.com





-----Original Message-----
From: Maverick <myeaddress@gmail.com>
Date: Thu, 23 Feb 2012 15:19:24 -0500
To: Jeroen Massar <jeroen@unfix.org>
Cc: "nanog@nanog.org" <nanog@nanog.org>
Subject: Re: Network Traffic Collection

I want to be able to see information like how much traffic an ip send
over a period of time, what machines it talked to etc from this
perspective it should be IP based but I would really like to know how
other people do it.

Best,
Ali

On Thu, Feb 23, 2012 at 3:14 PM, Jeroen Massar <jeroen@unfix.org> wrote:
> On 2012-02-23 21:11 , Maverick wrote:
>> Hello,
>>
>> I am trying to collect traffic traffic from pcap file and store it in
>> a database but really confused how to organize it. Should I organize
>> it on connection basis/ flow basis or IP basis.
>>
>> It might be an effort to write a customized traffic analysis tool like
>> wireshark with only required functionality. I would really appreciate
>> if someone can give me direction on write way of organizing the data
>> because right now I only see individual packets and no way of putting
>> them in some order.
>
> Does this all not completely depend on what you actually want to do with
> it? You might want to start there instead of the other way around.
>
> Greets,
> Jeroen
>
Re: Network Traffic Collection [ In reply to ]
On Thu, Feb 23, 2012 at 1:59 PM, Justin M. Streiner
<streiner@cluebyfour.org> wrote:
> On Thu, 23 Feb 2012, Maverick wrote:
>
>> I want to be able to see information like how much traffic an ip send
>> over a period of time, what machines it talked to etc from this
>> perspective it should be IP based but I would really like to know how
>> other people do it.
>
>
> Truth is that most people probably don't do it, beyond temporary, ad-hoc
> deployments, to solve a specific problem at a specific point in time.
> Traffic capture and analysis doesn't scale too well into multi-Gb/s service
> provider environments.
>
> Netflow tools are an option if 'reasonably accurate' is good enough for your
> needs.
>
> jms
>

For high speed switched Ethernet environments, consider using sFlow.

You can treat sFlow as remote packet capture and use Wireshark/tcpdump
for troubleshooting network traffic:

http://blog.sflow.com/2011/11/wireshark.html

Or use sFlow reporting tools to find IP sources, protocols etc.:

http://sflow.org/products/collectors.php

Which tool to choose depends on your requirements.
Re: Network Traffic Collection [ In reply to ]
PCAP is not well suited to what you describe. Most people use Sflow/Cflow/...
instead.

Owen

On Feb 23, 2012, at 12:19 PM, Maverick wrote:

> I want to be able to see information like how much traffic an ip send
> over a period of time, what machines it talked to etc from this
> perspective it should be IP based but I would really like to know how
> other people do it.
>
> Best,
> Ali
>
> On Thu, Feb 23, 2012 at 3:14 PM, Jeroen Massar <jeroen@unfix.org> wrote:
>> On 2012-02-23 21:11 , Maverick wrote:
>>> Hello,
>>>
>>> I am trying to collect traffic traffic from pcap file and store it in
>>> a database but really confused how to organize it. Should I organize
>>> it on connection basis/ flow basis or IP basis.
>>>
>>> It might be an effort to write a customized traffic analysis tool like
>>> wireshark with only required functionality. I would really appreciate
>>> if someone can give me direction on write way of organizing the data
>>> because right now I only see individual packets and no way of putting
>>> them in some order.
>>
>> Does this all not completely depend on what you actually want to do with
>> it? You might want to start there instead of the other way around.
>>
>> Greets,
>> Jeroen
>>
Re: Network Traffic Collection [ In reply to ]
On Fri, Feb 24, 2012 at 12:20 AM, Matlock, Kenneth L
<MatlockK@exempla.org> wrote:
> Netflow + netflow collector.

+1 This guide should give you a good start.

http://techowto.files.wordpress.com/2008/09/ntop-guide.pdf

Regards

--
Mukom Akong Tamon
______________

"If we can't BREATH, we'll die. Yet, we don't LIVE in order to breath.
Ditto we SHOULDN'T WORK just to MAKE MONEY. Doing so puts us on a one
way street to IRRELEVANCE."


[In Search of Excellence & Perfection] - http://perfexcellence.org
[Moments of TechXcellence] - http://techexcellence.net
[ICT Business Integration] - http://ibiztech.wordpress.com
[About Me] - http://about.me/perfexcellence
Re: Network Traffic Collection [ In reply to ]
Thanks Mukom for the wonderful guide, this is really helpful. I have
few questions about ntop though.

How can I get access to the log files generated by ntop and do my own
parsing rather than looking for webbased results that are generated.
Are there any programs available that do parsing of ntops log files.
When I run ntop on pcap I don't get the throughput graphs as rrd
doesn't work on pcap is there any work around for that.

Thanks,
Ali

On Sat, Feb 25, 2012 at 2:27 AM, Mukom Akong T. <mukom.tamon@gmail.com> wrote:
> On Fri, Feb 24, 2012 at 12:20 AM, Matlock, Kenneth L
> <MatlockK@exempla.org> wrote:
>> Netflow + netflow collector.
>
> +1 This guide should give you a good start.
>
> http://techowto.files.wordpress.com/2008/09/ntop-guide.pdf
>
> Regards
>
> --
> Mukom Akong Tamon
> ______________
>
> "If we can't BREATH, we'll die. Yet, we don't LIVE in order to breath.
> Ditto we SHOULDN'T WORK just to MAKE MONEY. Doing so puts us on a one
> way street to IRRELEVANCE."
>
>
> [In Search of Excellence & Perfection] - http://perfexcellence.org
> [Moments of TechXcellence] - http://techexcellence.net
> [ICT Business Integration] - http://ibiztech.wordpress.com
> [About Me] - http://about.me/perfexcellence
Re: Network Traffic Collection [ In reply to ]
Hi Ali


On Sat, Feb 25, 2012 at 6:14 PM, Maverick <myeaddress@gmail.com> wrote:
> Thanks Mukom for the wonderful guide, this is really helpful. I have
> few questions about ntop though.
>
> How can I get access to the log files generated by ntop and do my own
> parsing rather than looking for webbased results that are generated.

It's been a while i looked under the hood of ntop. Remember that ntop
itself usually needs to be 'fed' traffic to analyse. I have never done
it myself but if I needed the raw data, I'd mirror a port and capture
it with tcpdump into a pcap file (watch disk space!!) the use whatever
analysis tool suits my needs to look at it.

> Are there any programs available that do parsing of ntops log files.
> When I run ntop on pcap I don't get the throughput graphs as rrd
> doesn't work on pcap is there any work around for that.

Not to my knowledge no. I think there's a switch (-f) for reading data
from a pcap file as opposed to a live feed. I have never played with
that as well.

There are other (possible more feature laden) commercial flow
collectors and analysers out there). I also started following trisul
earlier on in the project, you might want to check it out.



>
> Thanks,
> Ali
>
> On Sat, Feb 25, 2012 at 2:27 AM, Mukom Akong T. <mukom.tamon@gmail.com> wrote:
>> On Fri, Feb 24, 2012 at 12:20 AM, Matlock, Kenneth L
>> <MatlockK@exempla.org> wrote:
>>> Netflow + netflow collector.
>>
>> +1 This guide should give you a good start.
>>
>> http://techowto.files.wordpress.com/2008/09/ntop-guide.pdf
>>
>> Regards
>>
>> --
>> Mukom Akong Tamon
>> ______________
>>
>> "If we can't BREATH, we'll die. Yet, we don't LIVE in order to breath.
>> Ditto we SHOULDN'T WORK just to MAKE MONEY. Doing so puts us on a one
>> way street to IRRELEVANCE."
>>
>>
>> [In Search of Excellence & Perfection] - http://perfexcellence.org
>> [Moments of TechXcellence] - http://techexcellence.net
>> [ICT Business Integration] - http://ibiztech.wordpress.com
>> [About Me] - http://about.me/perfexcellence



--
Mukom Akong [Tamon]
______________

“We don't LIVE in order to BREATH. Similarly WORKING in order to make
MONEY puts us on a one way street to irrelevance.“


[In Search of Excellence & Perfection] - http://perfexcellence.org
[Moments of TechXcellence] - http://techexcellence.net
[ICT Business Integration] - http://ibiztech.wordpress.com
[About Me] - http://about.me/perfexcellence