Author Topic: Edd Noman's Guide to pfSense 03 - How-To Monitor Bandwidth Usage whit NtopNG  (Read 14692 times)

Offline Edd Noman

  • Administrator
  • Newbie
  • *****
  • Posts: 33
  • Karma: +2/-0
    • View Profile
Edd Noman's Guide to pfSense 03 - How-To Monitor Bandwidth Usage whit NtopNG

In this guide I will only focus on bandwidth monitoring whit use of the NtopNG package in pfSense, bandwidth monitoring is a complex topic on its own and I will try to provide the basic you need to understand and get started whit this task and how this can improve your network situation. pfSense have several option for monitoring bandwidth and you can read about them here:

A word on bandwidth monitoring:
While it used to be that monitoring your bandwidth meant solely focusing on internet traffic, bandwidth usage monitoring now encompasses a broader range of components. For example, you can monitor bandwidth speed or capacity, you can observe network traffic between devices or general web application traffic. Regardless of what traffic you are monitoring, though, its important to understand the bandwidth that is being utilized so you can ensure users are getting the best possible performance out of your network.

I would go as far as saying that monitoring your network bandwidth usage is the most critical function for any network administrator.

What does bandwidth really mean?
To sum it up in one word: data. Bandwidth is quantified as the amount of data transferred in time, typically measured in bits per second. Thirty years ago, data was sent through physical mediums like the postal service; now there are myriad ways to transmit and receive massive amounts of data with the push of a button.

Since most organizations rely on the internet to conduct business-critical operations, internet speed can make all the difference in their success. What people dont know, however, is that there are actually two different types of bandwidth speed: upload and download. Upload speed is the speed at which data is sent to its destination, while download speed refers to the rate at which data is received. It used to be normal for businesses to use low bandwidth services like 56.k modems to transmit information. Now, those who have the funds can install Gigabit speed to their infrastructure to support the growth in data consumption. Firms are still using DSL and cable connections to run their business, but service providers have been able to allocate more resources to support these lower-tier Telco options.
Bandwidth capacity is also an important consideration. Bandwidth capacity means the maximum data rate a link can transfer. Bandwidth capacity is an important factor to consider, because when you configure your infrastructure you need to make sure you can support the bandwidth that you require. For example, the service model of your cell phone plan is based on how much data you consume on the vendors network. When you're able to monitor your bandwidth usage, you're better able to determine what plan is right for your environment. As another example, consider an environment with hundreds of users. How do you determine what bandwidth to implement? Its important to have proper metrics and tools that can show you how much bandwidth you'll require to run your day-to-day operations. For business customers, it can become very difficult to forecast current and future consumption of bandwidth without network insight.

High-Speed Web-based Traffic Analysis and Flow Collection whit NtopNG
Ntopng is the next generation version of the original ntop, a network traffic probe that shows the network usage, similar to what the popular top Unix command does. ntopng is based on libpcap and it has been written in a portable way in order to virtually run on every Unix platform, MacOSX and on Windows as well.

Ntopng users can use a a web browser to navigate through ntop (that acts as a web server) traffic information and get a dump of the network status. In the latter case, ntopng can be seen as a simple RMON-like agent with an embedded web interface. The use of:
a web interface.
Limited configuration and administration via the web interface.
Reduced CPU and memory usage (they vary according to network size and traffic).

Features of Ntopng:
- Sort network traffic according to many criteria including IP address, port, L7 protocol, throughput, AS.
- Show network traffic and IPv4/v6 active hosts.
- Produce long-term reports about various network metrics such as throughput, application protocols
- Top X talkers/listeners, top ASs, top L7 applications.
- For each communication flow report network/application latency/RTT, TCP stats (retransmissions, packets OOO, packet lost), bytes/packets
- Store on disk persistent traffic statistics in RRD format.
- Geolocate hosts and display reports according to host location.
- Discover application protocols by leveraging on nDPI, ntop's DPI framework.
- Characterize HTTP traffic by leveraging on characterization services provided by Google and HTTP Blacklist.
- Show IP traffic distribution among the various protocols.
- Analyze IP traffic and sort it according to the source/destination.
- Display IP Traffic Subnet matrix (who's talking to who?)
- Report IP protocol usage sorted by protocol type.
- Produce HTML5/AJAX network traffic statistics.

Whit all that said, we can start by installing the ntopng package by navigating to System -> Package Manager -> Available Packages
Here you search for "ntop" and then click the install button

Now you would need to confirm that you would like to install selected package and its dependency

Now you will see the install process starting whit the progress bar and text output will run for some minute as the installer is about 250mb that would need to be downloaded and then installed so give it some time

When the package is done installing you need to navigate to Diagnostics -> ntopng Settings
The first thing you would want to do here is to set the password to use when login to the ntopng WebGUI

ntopng Admin Password: MysuperSecretandSEXYkeY
Confirm ntopng Admin Password: MysuperSecretandSEXYkeY
Save and Apply

Now after the access password is set we can configure the default settings for ntopng to use

General Options
Enable ntopng: Yes
Keep Data/Settings: Yes (this will keep all settings in case of updates and reinstallations)
Interface: LAN and WAN
DNS Mode: Decode DNS responses and resolve all numeric IPs
Local Networks: Consider all RFC1918 networks as local traffic (this depends on your layout of pfSense is behind NAT you want to use LAN Interface as local)
Disable Alerts: Yes
Save and Apply

When ntopng service is configured and have start running after its done loading when pressing the save button you want to update the GeoIP Data information so click on the green button for updating the information
At this point the configuration in pfSense is done, and you now need to connect to the ntopng WebGUI to finish its configuration so navigate to or IP-OF-pfSense:3000 or if you have followed my previous install guides

Once logged in, the first place to go is Settings and set your recording limits, this is done whit the gear icon on the top right corner

Out of the box, it will record RAW packets for 1 day in your File System, the Rolled up reports in MySQL for 30 days, and Total's for 1 year. You can adjust here to work with the available disk space and RAM you have for pfSense.

The Reports:
Everyone will be different and have their own needs for reporting, but I wanted to screenshot some of the cool reports you can generate and view in ntopng to share with you all.
You can also customize and work with anything which is captured going across the LAN.
You can view total traffic on your local network and sort by usage:

You can view Active Data Flows / Destination / Type in real time with ease on the Active Flows Report

You can even view specifics on a specific host on your Network like so, even with total usage, Activity Maps and more

Even break it down by Protocols

For those of you who are more interested in spying, yes you can see the top HTTP traffic destinations....

Now that is all there is to configure ntopng and where to find the different reports, and yes there is a large benefit to running this package even in your home environment as this will give you graphs and statistics of where all your data and bandwidth went and what device used it, and if you did not get the amount you charged for you have actual proof of your usage when you call your ISP to complain about their service and invoice.

If you follow this guide and it is not working for you and it broke your system, I am not responsible or liability for that as you should not take anything you read on the internet at face value and you should test settings like this in a lab environment and not on your production servers
« Last Edit: June 02, 2017, 02:46:36 PM by noman »