Mailing List Archive

stability of RRD and ntop (and nProbe)

I have some trouble making ntop (RRD in fact) stable. I have 8 probes (all some pfflowd daemon on 7 different pfsense (freebsd) firewalls) that send some NetFlow informations. Ntop works quite well for a few days but then some trouble arises...

I first noticed that there is a huge number (more than 100,000) of directories under /var/lib/ntop/rrd/<name of interface>/NetFlow. Curiously it happens only with some pfsenses.

I've tried 4 different installs (under FreeBSD, 2 different installs, with UFS then ZFS / DragonFlyBSD (Hammer) / Linux (tuned ext3)). Each time, there were issues with huge number of files (notably under UFS), either the kernel would crash (ZFS/Hammer) or ntop/RRD would crash (UFS/Ext3). Finally, under Linux/Debian, I have followed all recommandation here about filesystem:"]

That is:

Files System Level Optimization
  • Set the noatime and nodiratime option when mounting the ext3 file system. This will save updates to the directory blocks. (DavePlonka??)
  • Mount ext3 with data=writeback. This gives some performance boost at the expense of a possibility for old data to appear after a crash. Check man mount for details. (RyanKubica??)
  • Use a file system that indexes directories (ext3 with dir_index) and keep all rrd files in a single directory (DavePlonka??)

Yet, after a few days, my RRD graphs just disappeared (they do not update). When I relaunch ntop, they come back again.

I have checked in FAQ and forums and it seems that I am the only one with this problem.

I have two questions :
  • Have I done something wrong? Is it not some kind of well known problem with Ntop/NetFlow/RRD? (because I am nearly using it in a "out of the box" mode, just that I have 7 or 8 pfflowd probes, but I must not be the only one in this case).

  • Would nProbe, which has database backends, help me get rid of that huge number of files or just stabilize ntop in this case?

Many thanks !

PS: I use ntop version v4.0.3