Mailing List Archive

Idea for more timely virusdb updates
This thread on Trojan.JS.RunMe had me thinking: Hourly virus updates is
better than any of the commercial virus scanners, but obviously still has
issues, especially since a bunch of us obviously submitted updates that had
already been entered. I gather from these posts that the virusdb's actually
have some form of version number.

Suppose there was a DNS entry, say virusdb.clamav.net (or
version.virusdb.clamav.net, etc), that returned simply a text record with
the current DB version in it. Then, it would be possible to check the
version with a relatively cheap single UDP packet, rather than a full http
check, and people could check for DB updates more often than once an hour
without taxing the distribution system.

If nothing else, if this TXT record existing we could hack together some
shell script to check it and run freshclam as needed.

Just a thought.

==========================================================
Chris Candreva -- chris@westnet.com -- (914) 967-7816
WestNet Internet Services of Westchester
http://www.westnet.com/


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Mon, 9 Aug 2004, Christopher X. Candreva wrote:

> This thread on Trojan.JS.RunMe had me thinking: Hourly virus updates is
> better than any of the commercial virus scanners, but obviously still has
> issues, especially since a bunch of us obviously submitted updates that had
> already been entered. I gather from these posts that the virusdb's actually
> have some form of version number.
>
> Suppose there was a DNS entry, say virusdb.clamav.net (or
> version.virusdb.clamav.net, etc), that returned simply a text record with
> the current DB version in it. Then, it would be possible to check the
> version with a relatively cheap single UDP packet, rather than a full http
> check, and people could check for DB updates more often than once an hour
> without taxing the distribution system.

That's a very interesting idea, but I can imagine a few problems:
- we'd have to have a very short time-to-live or it would get stale
- the dns might know about the update before the mirrors all get it
- if everyone finds out about an update within 5 minutes of each other,
the mirrors might not handle the load

After seeing a Defcon talk on putting arbitrary data in DNS, though, I
wonder if we could put the daily updates (gpg signed) into DNS? That
would take a lot of load off the mirrors (occasional checks for main.cvd
updates are all that is required). And caching DNS servers would
distribute the load a bit.

Anyway, just another crazy idea for the developers to consider.

Damian Menscher
--
-=#| Physics Grad Student & SysAdmin @ U Illinois Urbana-Champaign |#=-
-=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc:(217)333-0038 |#=-
-=#| 4602 Beckman, VMIL/MS, Imaging Technology Group:(217)244-3074 |#=-
-=#| <menscher@uiuc.edu> www.uiuc.edu/~menscher/ Fax:(217)333-9819 |#=-
-=#| The above opinions are not necessarily those of my employers. |#=-


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Mon, 2004-08-09 at 16:55 -0400, Christopher X. Candreva wrote:
> This thread on Trojan.JS.RunMe had me thinking: Hourly virus updates is
> better than any of the commercial virus scanners, but obviously still has
> issues, especially since a bunch of us obviously submitted updates that had
> already been entered. I gather from these posts that the virusdb's actually
> have some form of version number.
>
> Suppose there was a DNS entry, say virusdb.clamav.net (or
> version.virusdb.clamav.net, etc), that returned simply a text record with
> the current DB version in it. Then, it would be possible to check the
> version with a relatively cheap single UDP packet, rather than a full http
> check, and people could check for DB updates more often than once an hour
> without taxing the distribution system.
>
> If nothing else, if this TXT record existing we could hack together some
> shell script to check it and run freshclam as needed.

Then all users would sworm to download the new sig, as soon as that
serial number incrimented, flooding the download server with update
requests.



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Mon, Aug 09, 2004 at 05:33:05PM -0400, Chris Meadors wrote:
> > Suppose there was a DNS entry, say virusdb.clamav.net (or
> > version.virusdb.clamav.net, etc), that returned simply a text record with
> > the current DB version in it. Then, it would be possible to check the
> > version with a relatively cheap single UDP packet, rather than a full http
> > check, and people could check for DB updates more often than once an hour
> > without taxing the distribution system.
>
> Then all users would sworm to download the new sig, as soon as that
> serial number incrimented, flooding the download server with update
> requests.

Only tracker.clamav.net (can be loadbalanced) should be able to handle
a fair number of connections, but daily.cvd.torrent is small enough
you could put it in a DNS TXT record :) (OK, DNS is far from secure,
so reliability will be at stake in that case... you might need to
cryptographically sign the file).

(1/2 :-)

--
#!perl -wpl # mmfppfmpmmpp mmpffm <pmmppfmfpppppfmmmf@fpffmm4mmmpmfpmf.ppppmf>
$p=3-2*/[^\W\dmpf_]/i;s.[a-z]{$p}.vec($f=join('',$p-1?chr(sub{$_[0]*9+$_[1]*3+
$_[2]}->(map{/p|f/i+/f/i}split//,$&)+97):qw(m p f)[map{((ord$&)%32-1)/$_%3}(9,
3,1)]),5,1)='`'lt$&;$f.eig; # Jan-Pieter Cornet


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: Idea for more timely virusdb updates [ In reply to ]
What about a deeper mirroring system? Perhaps one that supports
notification?

One of the things I like about BIND (not enough to use it, but still an
admired concept ;-) is the way zones can be distributed... notification
speeds things up if it works, polling creates a failsafe in which a missing
notify doesn't cause the world to end...

Hourly polls is a good thing - but if the system worked both ways, the
mirror could signal the end clients that it's time to download... those
notifies could be send only to clients that had registered to receive it (an
option in freshclam) and would not push the data, but trigger a freshclam
pull.

It could provide faster update response and smooth out the spikes in
download traffic, and could be used to maintain a larger set of mirrors...
without increasing polling frquency... a new "freshclam server" could allow
all larger users to easily run their own mirrors for internal
distribution...

Just a few ideas...

m/



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Damian Menscher wrote:

>On Mon, 9 Aug 2004, Christopher X. Candreva wrote:
>
>
>>Suppose there was a DNS entry, say virusdb.clamav.net (or
>>version.virusdb.clamav.net, etc), that returned simply a text record with
>>the current DB version in it.
>>

>After seeing a Defcon talk on putting arbitrary data in DNS, though, I
>wonder if we could put the daily updates (gpg signed) into DNS? That
>would take a lot of load off the mirrors (occasional checks for main.cvd
>updates are all that is required). And caching DNS servers would
>distribute the load a bit.
>
>
You know, this isn't so crazy after all. I put arbitrary data on my DNS
server so that exim
can get config data using dnsdb lookup. Its cheaper than mysql lookup
(Plus, you eliminate single point of failure),
and you can still update config from a central location instead of
updating each server config.

The only snag, is that TXT record is limited to a number of bytes ( I
tried putting 4096 bytes on it, it didn't work).
Now, the question is, can the daily (or hourly) updates fit in a single
TXT record?
If it must span multiple records than it will be somewhat complicated ....

Regards,

Fajar
--
http://justreadthis.com/


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Monday, August 09, 2004 11:18 PM [EST], Fajar A. Nugraha wrote:

>>
> You know, this isn't so crazy after all. I put arbitrary data on my
> DNS server so that exim
> can get config data using dnsdb lookup. Its cheaper than mysql
> lookup (Plus, you eliminate single point of failure),
> and you can still update config from a central location instead of
> updating each server config.
>
> The only snag, is that TXT record is limited to a number of bytes (
> I tried putting 4096 bytes on it, it didn't work).
> Now, the question is, can the daily (or hourly) updates fit in a
> single TXT record?
> If it must span multiple records than it will be somewhat
> complicated ....
>
> Regards,
>
> Fajar

I'd not recommend putting all the data in TXT records. TXT records
can be a max of 255 characters (anything more and you'll have problems
with other resolvers and such). But yeah, the version number in the
TXT records would be good, set the TTL to about 30-60 mins, and have
the freshclam client query and check the version.

I could assist with implementing the necessary server side scripts to
make the DNS management part really easy (I do something similar to
this all the time, rbldnsd makes stuff like this stupidly simple and
quick). Hell, I could even offer the DNSbl servers that the AHBL has
to host the zones if needbe.

--
Brian Bruns
The Summit Open Source Development Group
Open Solutions For A Closed World / Anti-Spam Resources
http://www.sosdg.org

The Abusive Hosts Blocking List
http://www.ahbl.org



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Mitch (WebCob) wrote:
> What about a deeper mirroring system? Perhaps one that supports
> notification?
>
> One of the things I like about BIND (not enough to use it, but still an
> admired concept ;-) is the way zones can be distributed... notification
> speeds things up if it works, polling creates a failsafe in which a missing
> notify doesn't cause the world to end...

right, but as discussed below, generally bind servers don't have 100k people
waiting for notifications and updates.

> Hourly polls is a good thing - but if the system worked both ways, the
> mirror could signal the end clients that it's time to download... those
> notifies could be send only to clients that had registered to receive it (an
> option in freshclam) and would not push the data, but trigger a freshclam
> pull.

with that option, the 'clients' would either have to remain connected the
entire time, which is completely not feasable, or somehow the database mirrors
would have to either 'remember' who to notify, or have some sort of registry
of people to notify (I can see how one might do this with a paid mirror
service), and then send out notifications (even a single UDP packet to 100k
servers could be quite bandwidth intensive. The architecture could work, yes,
but it doesn't scale well, and I don't think the clamav team has the resources
to do this sort of ass-kissing for free. They're already providing a
wonderful service to the internet community, we cannot bite the hand that
feeds us.

Another problem with this notification is there are still the spikes when the
notifications come out that EVERYONE AND THEIR BROTHER contacts the database
mirrors for updates. Your solution doesn't solve any problems imposed by
Christopher's idea, and actually introduces more.

In my opinion, the existing system is fine, and if you want better, you should
talk to the clamav folks about setting up some sort of 'priority' mirror, in
which you would pay a fee for having more enhanced services, like
notification, dns update polling, etc. And of course, proceeds (or at least a
major part of) would go to the clamav team for being the most kick ass
anti-virus product out there. I'm not sure how the official procedure would
be to roll something like this out, but now that I think of it, I may just go
about working on something like this. Gotta pay for my colocation somehow :)

Tomasz, et al.: Please expect to see an email from me by the end of the work
day tomorrow (or rather, today, but I haven't slept yet)

> It could provide faster update response and smooth out the spikes in
> download traffic, and could be used to maintain a larger set of mirrors...
> without increasing polling frquency... a new "freshclam server" could allow
> all larger users to easily run their own mirrors for internal
> distribution...

I would think that most 'larger users' (5+ node mail server cluster) would
already have an internal mirror. It's not difficult to do, and has been
discussed on this list, and in the clamav documentation many times.

> Just a few ideas...

hey, brainstorming is good, it's just the ideas aren't always ;)

-Jeremy

--
Jeremy Kitchen ++ Systems Administrator ++ Inter7 Internet Technologies, Inc.
jeremy@inter7.com ++ www.inter7.com ++ 866.528.3530 ++ 847.492.0470 int'l
kitchen @ #qmail #gentoo on EFnet ++ scriptkitchen.com/qmail



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tue, 10 Aug 2004, Fajar A. Nugraha wrote:

> The only snag, is that TXT record is limited to a number of bytes ( I tried
> putting 4096 bytes on it, it didn't work).
> Now, the question is, can the daily (or hourly) updates fit in a single TXT
> record?

I don't know that putting ALL of the records in DNS is necessary. The only
reason I was putting the version number there was to allow quick, more
frequent checks to see if you had the current version. It's possible to run
DNS with very short TTL times, even 0.

In terms of load on the servers: a smaller file would certainly help. I can
see the simplicity of just haveing two files to grab being attractive.
However, daily.cfg is now about 150k . I don't consider this big in the
scheme of things, but if we are talking about hundreds or thousands of
people trying to get the file, then the difference could be significant.

Suppose there was a numbered file for each version that would 'upgrade' you
from the previous version. IE, if I'm at 444, and current is 445, I grab
445.cvd. If I'm at 440, I grab 441.cvd, 442.cvd, through 445.cvd.

Downside -- obviously harder to maintain. Upside -- someone who is staying
constantly up to date is grabbing only a few bytes off the server at a time.

I think this is simpler than putting all the data into DNS.

I've also thought about rsync -- if putting the cvd files on an rsync server
would lighten the load at all.


==========================================================
Chris Candreva -- chris@westnet.com -- (914) 967-7816
WestNet Internet Services of Westchester
http://www.westnet.com/


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
At 16:03 10.08.2004, you wrote:

>...
>I've also thought about rsync -- if putting the cvd files on an rsync server
>would lighten the load at all.

Oh it would, rsync is quite effective. Another possibility might be to
patch the .cvd file(s)....

0.02

Erich


THINK
Püntenstrasse 39
8143 Stallikon
mailto:erich.titl@think.ch
PGP Fingerprint: BC9A 25BC 3954 3BC8 C024 8D8A B7D4 FF9D 05B8 0A16



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Erich Titl wanted us to know:

>>I've also thought about rsync -- if putting the cvd files on an rsync
>>server would lighten the load at all.
>Oh it would, rsync is quite effective. Another possibility might be to
>patch the .cvd file(s)....

Agree with rsync, depends how much changes in the file per download, but
I think it can be marvelously efficient.
Disagree with patching, binary files don't lend themselves to
diff/patch very well.

>0.02

Now we have 0.04 :-)

--
Regards... Todd
We should not be building surveillance technology into standards.
Law enforcement was not supposed to be easy. Where it is easy,
it's called a police state. -- Jeff Schiller on NANOG
Linux kernel 2.6.3-15mdkenterprise 2 users, load average: 0.09, 0.03, 0.01


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Erich Titl wrote the following on 08/10/2004 05:12 PM :

> At 16:03 10.08.2004, you wrote:
>
>> ...
>> I've also thought about rsync -- if putting the cvd files on an rsync
>> server
>> would lighten the load at all.
>
>
> Oh it would, rsync is quite effective.


Not much with compressed files like *.cvd.

> Another possibility might be to patch the .cvd file(s)....
>

That was one proposition I made last year. But in practice it seems
there isn't really a pressing need now.

Lionel.

--
Lionel Bouton - inet6
---------------------------------------------------------------------
o Siege social: 51, rue de Verdun - 92158 Suresnes
/ _ __ _ Acces Bureaux: 33 rue Benoit Malon - 92150 Suresnes
/ /\ /_ / /_ France
\/ \/_ / /_/ Tel. +33 (0) 1 41 44 85 36
Inetsys S.A. Fax +33 (0) 1 46 97 20 10




-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: Idea for more timely virusdb updates [ In reply to ]
> right, but as discussed below, generally bind servers don't have
> 100k people
> waiting for notifications and updates.
>

Nope, true... but like I suggested, the notification tree doesn't have to be
flat...

One server notifying 100000 servers is time consuming and sure - costs a lot
of bandwidth...

Lets assume that each notify takes 5 seconds... we have to have SOMETHING to
"measure"...

1 server notifying 100000 servers takes 500000 seconds. That's a little over
a day to push the notification - bad idea ;-)

1 server notifying 100 servers, which each in turn notify 100 servers and so
on...
1 to 100: 100 seconds
each of them notifying 100: 100 seconds (total notified 10100)
each of them notifying 100: 100 seconds (total notified 1010100!) in 5
minutes!

That's 10 times your value of 100000 servers. Each server would only have to
know about 100 others. Not a huge database - wouldn't even have to be
written to file. Each server could be responsible for polling it's master
once per hour.

> > Hourly polls is a good thing - but if the system worked both ways, the
> > mirror could signal the end clients that it's time to download... those
> > notifies could be send only to clients that had registered to
> receive it (an
> > option in freshclam) and would not push the data, but trigger a
> freshclam
> > pull.
>
> with that option, the 'clients' would either have to remain connected the
> entire time, which is completely not feasable, or somehow the
> database mirrors
> would have to either 'remember' who to notify, or have some sort
> of registry
> of people to notify (I can see how one might do this with a paid mirror
> service), and then send out notifications (even a single UDP
> packet to 100k
> servers could be quite bandwidth intensive. The architecture
> could work, yes,
> but it doesn't scale well, and I don't think the clamav team has
> the resources
> to do this sort of ass-kissing for free. They're already providing a
> wonderful service to the internet community, we cannot bite the hand that
> feeds us.

I wasn't proposing that it had to be done for free (not that it can't be
with the factor tree I explained above). It might even reduce the cost of
database distribution.

If each server is only pushing 100 updates @ 200KB per update (2MB total) we
can get 500 pushes per month for only a couple dollars.

> Another problem with this notification is there are still the
> spikes when the
> notifications come out that EVERYONE AND THEIR BROTHER contacts
> the database
> mirrors for updates. Your solution doesn't solve any problems imposed by
> Christopher's idea, and actually introduces more.

100 servers for 200KB (20MB is hardly a spike.) and as for clients remaining
connected, that is what a server is - connected. This isn't for end users,
or local workstations. It's an OPTION for people who process a lot of data,
are at high risk, and need immediate response. Then their own internal
freshclam clients can poll their local authoritative server as often as they
want, or use the same procedure to distribute to them (if they are full time
connected that is).

> In my opinion, the existing system is fine, and if you want
> better, you should
> talk to the clamav folks about setting up some sort of 'priority'

Yeah, we could, but I don't think it needs that. And setting up an internal
mirror doesn't address the response time of the updates, unless I start
hammering the main freshclam every few minutes... and I just don't think
that would be friendly.

With the sort of hierarchical distribution I'm talking about, you could even
use an ranking system to automatically organize the distribtion (while I'm
on a roll ;-)...

What I mean is that everyone would contact one of the "root" mirrors
initially. In the request to be notified, it would indicate the number of
clients it serves. If less than a certain number, then it could be referred
to a child of the root server. If that child becomes unavailable it could
contact the root again (at the next hourly polling time). How many servers
are there on the Internet? We could probably handle the whole lot of them
with no more than 4 or 5 levels. Push an update to the world in under 10
minutes. Think how many virus laden emails this could stop.

(visions of f5...) in fact, the root server could hand out the IP's of all
child servers not fully loaded. The client could register with the nearest
(by route time) one -

just ranting...

m/



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Aug 10, 2004, at 5:57 AM, Jeremy Kitchen wrote:

> Mitch (WebCob) wrote:
>> Just a few ideas...
>
> hey, brainstorming is good, it's just the ideas aren't always ;)

Another stupid idea...how about a mechanism where clam can have updates
"pushed" to it, so servers controlled by the clam team can distribute
mini updates to them. The admins would have to subscribe to it, like a
listserv, only instead of through email, it's done through this
theoretical mechanism. There wouldn't be traffic spikes (as big) for
times where there *aren't* updated db's available, only when there are
updates, and the updates are sent out as the clam servers are able to
handle the load.

Maybe like a modified GPG-signed listserv system only on it's own "clam
update daemon" port...take a little more configuration since the people
installing clam would have to subscribe and install a GPG key or
something like that in the process, but that shouldn't be something
back-breaking to figure out.

Maintenance would have to be done for the subscription mechanism, etc.,
like a listserv would, but it may be something that could be done. May
even be extendable so that a master server for a network could receive
the updates from the clam site (pushed from clamserv) then in turn be
told to push them out to machines on the internal network. (I know
this could already be set up, but it may be easier through this type of
model to set up and maintain...)

I'm probably overlooking something obvious, but again...just an idea,
right? :-)

-Bart



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tue, 10 Aug 2004, Bart Silverstrim wrote:
>
> Maybe like a modified GPG-signed listserv system only on it's own "clam
> update daemon" port...take a little more configuration since the people
> installing clam would have to subscribe and install a GPG key or
> something like that in the process, but that shouldn't be something
> back-breaking to figure out.

Ok, this is turning into a scary beast. But we already have several
mailing lists (clamav-users, for example) which can obviously handle a
bit of a load. Might be interesting to concoct a specially-formatted
message that the milter (or clamd itself) could recognize as a database
update, and automatically append to its list of signatures.

I'd imagine a format something like:

---gpg-cleartext-signed-message---
BEGIN clamd update 24.449
Worm.bagle.zz AAAABBBBCCCDDDDEEFEFKL..........
Worm.SkyNet.zz 111112222333344455666677........
...
END
---gpg-signature---
JDSLJGIREJIOJDGLSJLGHSLKJGLKSDJLKGJSLKJGIEJ*Y*G($Y*HHIO4k245j2jk
kdjaflkjkh325hjk35h2jkhkjhjkfdhjh42jkh345jk2h35jk2hkjhjkfhjskh32
fhjkhafdjhajk53h2jk5h3j2kh35jkhfay983489527938572035230398udfsfs
---end-signature---

When scanning stuff like this, clamd could automagically decode the gpg
signature and test that it is valid. If so, it looks at the sequence
number (24.449 in this case). If that's the next one in the series, it
appends the rules to its database. If not, it assumes it lost a message
somewhere and contacts a mirror via HTTP to get main 24 and daily 449.

Doing something like this would push a lot of the distribution load onto
sourceforge (which seems to get messages out to this list in about 1/2
hour). The gpg-signature prevents spoofing. And the sequence numbers
keep everyone current. The major problems I see are getting clamd to
recognize a message targeted for it, and the obvious problems of DoS
attacks (someone sending spoofed messages that would suck CPU time
decoding the gpg signature).

Anyway, just another wild-n-crazy idea to throw out there. I'm guessing
we're better off with the current method for now, but this might be an
interesting possibility for the future.

[.I haven't given up on DNS updates yet, but it's hard to come up with a
clean way to distribute >256 bytes of data that way, which means even
single rules don't always fit.]

Damian Menscher
--
-=#| Physics Grad Student & SysAdmin @ U Illinois Urbana-Champaign |#=-
-=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc:(217)333-0038 |#=-
-=#| 4602 Beckman, VMIL/MS, Imaging Technology Group:(217)244-3074 |#=-
-=#| <menscher@uiuc.edu> www.uiuc.edu/~menscher/ Fax:(217)333-9819 |#=-
-=#| The above opinions are not necessarily those of my employers. |#=-


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tue, 10 Aug 2004, Lionel Bouton wrote:

> > Another possibility might be to patch the .cvd file(s)....
> >
>
> That was one proposition I made last year. But in practice it seems there
> isn't really a pressing need now.

If people can't check for database updates more often than once an hour,
then there is a pressing need.

The mirror page talkes about the need for mirrors, about exponential growth,
and how at least a 10mbit pipe is needed to host a mirror. It puts March
2004 traffic at about 120gig/month

Some quick calculations:
daily.cvd is about 150k compressed, 334k uncompressed -- let's say 50%.
Greping the virses added for updated 447 gave me about 3k uncompressed -- so
let's say 2k compressed, on the outside.

For 2k of update, everyone downloaded 150k. That shows (at least for that
update) only 1.3% of what was downloaded was needed.

If only 1.3% of every update is actually needed, and people only downloaded
what they needed, the traffic on the mirrors would drop from 120gig/month to
1.6 gig/month.

If I am completely off by a factor of 10 -- say only 10% of every update
is actually needed, traffic on the mirrors drops from 120gig to 12gig.

There are a lot of assumptions here, but I would think ever reducing the
load on the virus servers by half would be significant.

==========================================================
Chris Candreva -- chris@westnet.com -- (914) 967-7816
WestNet Internet Services of Westchester
http://www.westnet.com/


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tuesday 10 August 2004 12:23 pm, Damian Menscher wrote:
> Ok, this is turning into a scary beast. But we already have several
> mailing lists (clamav-users, for example) which can obviously handle a
> bit of a load. Might be interesting to concoct a specially-formatted
> message that the milter (or clamd itself) could recognize as a database
> update, and automatically append to its list of signatures.

this is actually a pretty decent idea. I think it would be best to, rather
than have clamd try to detect it, have a special address on the machine that
processes the message via a program. Most MTAs I'm aware of (at least on the
unix side) can do this, I know qmail can for sure.

> I'd imagine a format something like:
[snip email message for the update]

> Doing something like this would push a lot of the distribution load onto
> sourceforge (which seems to get messages out to this list in about 1/2
> hour).

for something like this I wouldn't use sourceforge's mail servers :P They're
already bogged down as it is, us adding load to them like this would be bad,
and the notifications would eventually get slower, and slower, and slower...
having a dedicated list server for this purpose would be the best.


> The gpg-signature prevents spoofing. And the sequence numbers
> keep everyone current. The major problems I see are getting clamd to
> recognize a message targeted for it, and the obvious problems of DoS
> attacks (someone sending spoofed messages that would suck CPU time
> decoding the gpg signature).

yes, that's an unfortunate problem with this idea, however, if you used, as I
stated, a special address that uses program delivery, you'd have to hack the
listserver to get everyone's 'subscription' address to be able to do this.

> Anyway, just another wild-n-crazy idea to throw out there. I'm guessing
> we're better off with the current method for now, but this might be an
> interesting possibility for the future.

it definitely is interesting.

> [.I haven't given up on DNS updates yet, but it's hard to come up with a
> clean way to distribute >256 bytes of data that way, which means even
> single rules don't always fit.]

I wouldn't distribute the rule in DNS, however, a timestamp of sorts in dns
isn't a bad idea.

-Jeremy

--
Jeremy Kitchen ++ Systems Administrator ++ Inter7 Internet Technologies, Inc.
jeremy@inter7.com ++ www.inter7.com ++ 866.528.3530 ++ 847.492.0470 int'l
kitchen @ #qmail #gentoo on EFnet ++ scriptkitchen.com/qmail



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tuesday 10 August 2004 04:57 am, Jeremy Kitchen wrote:
> Tomasz, et al.: Please expect to see an email from me by the end of the
> work day tomorrow (or rather, today, but I haven't slept yet)

sigh, and after saying that I now have tons of work to do so I won't be able
to get this email to you guys until later.

I will send it though :)

-Jeremy

--
Jeremy Kitchen ++ Systems Administrator ++ Inter7 Internet Technologies, Inc.
jeremy@inter7.com ++ www.inter7.com ++ 866.528.3530 ++ 847.492.0470 int'l
kitchen @ #qmail #gentoo on EFnet ++ scriptkitchen.com/qmail



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tue, 10 Aug 2004, Jeremy Kitchen wrote:
> On Tuesday 10 August 2004 12:23 pm, Damian Menscher wrote:
> > Ok, this is turning into a scary beast. But we already have several
> > mailing lists (clamav-users, for example) which can obviously handle a
> > bit of a load. Might be interesting to concoct a specially-formatted
> > message that the milter (or clamd itself) could recognize as a database
> > update, and automatically append to its list of signatures.
>
> this is actually a pretty decent idea. I think it would be best to, rather
> than have clamd try to detect it, have a special address on the machine that
> processes the message via a program. Most MTAs I'm aware of (at least on the
> unix side) can do this, I know qmail can for sure.

Good idea. Taking it out of the milter allows for qmail/exim/postfix
compatibility, and sending to a dedicated address saves the effort of
processing every message (though presumably you're doing that anyway).

With sendmail, you could add to /etc/aliases something like:
clamav-updates | sigtool --add

Before people get too excited about this idea, though, there are some
issues that need to be fixed.

Anyone know if it's really feasible for us to obtain a mailserver that
can send out 2k emails to all (100,000?) users in a short (5-10 mins)
time? Assuming those numbers are reasonable, that means 200 meg of
data. Combined with SMTP overhead, it seems like it would be
troublesome. Additionally, there are potential bandwidth issues if you
consider we'd need to do that several times/day.

Updating the "main" database is one concern. Sending out a 2-meg email
to everyone seems like it might be too much load, but sending out the 1K
email telling everyone to get it means the mirrors will get swamped. I
can't think of a way around this, but hopefully someone else can?

Also, this doesn't give much provision for removing "bad" signatures
(that cause false positives) since it really just appends rules. We'd
need to figure out a way to delete signatures also. I could imagine
doing this by including a "null" signature, or using some other flag.

Finally, there's the whole issue of multiplying your points of failure.
If your current database is screwed, appending more to it will leave it
screwed. And if you add stuff to it a few times a day, chances are it
will get screwed up at some point. At least this issue has a simple
fix: include an MD5 sum with the update which must match your MD5 sum
after applying the update. If they don't match, you know something went
wrong, either with this update or a previous one. (This has the danger
that if the developers send an email with an incorrect MD5 hash,
everyone will thrash the mirrors.)

Note to the developers: please don't feel like you have to code up any
of our random ideas. I'm just having fun brainstorming about how to
optimize this process. I expect in another few days of discussion we'll
have converged on a fairly sane idea.

Damian Menscher
--
-=#| Physics Grad Student & SysAdmin @ U Illinois Urbana-Champaign |#=-
-=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc:(217)333-0038 |#=-
-=#| 4602 Beckman, VMIL/MS, Imaging Technology Group:(217)244-3074 |#=-
-=#| <menscher@uiuc.edu> www.uiuc.edu/~menscher/ Fax:(217)333-9819 |#=-
-=#| The above opinions are not necessarily those of my employers. |#=-


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tue, 10 Aug 2004, Damian Menscher wrote:
> Anyone know if it's really feasible for us to obtain a mailserver that
> can send out 2k emails to all (100,000?) users in a short (5-10 mins)
> time?

I haven't been following the whole discussion, but I thought this was
mostly to provide support to "power users". I think the average
small-time admin would be happy with the hourly updates.

Jeffrey Moskot
System Administrator
jef@math.miami.edu


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On 2004-08-10 14:41:28 -0500, Damian Menscher wrote:
> On Tue, 10 Aug 2004, Jeremy Kitchen wrote:
> > On Tuesday 10 August 2004 12:23 pm, Damian Menscher wrote:
> > > Ok, this is turning into a scary beast. But we already have several
> > > mailing lists (clamav-users, for example) which can obviously handle a
> > > bit of a load. Might be interesting to concoct a specially-formatted
> > > message that the milter (or clamd itself) could recognize as a database
> > > update, and automatically append to its list of signatures.
[...]
> Before people get too excited about this idea, though, there are some
> issues that need to be fixed.
>
> Anyone know if it's really feasible for us to obtain a mailserver that
> can send out 2k emails to all (100,000?) users in a short (5-10 mins)
> time?

How about using NNTP instead of SMTP? Then the clamav server doesn't
have to push out those messages to everybody but only to its neighbours
which will distribute it further.

hp


--
_ | Peter J. Holzer | Je höher der Norden, desto weniger wird
|_|_) | Sysadmin WSR | überhaupt gesprochen, also auch kein Dialekt.
| | | hjp@hjp.at | Hallig Gröde ist fast gänzlich dialektfrei.
__/ | http://www.hjp.at/ | -- Hannes Petersen in desd
Re: Idea for more timely virusdb updates [ In reply to ]
Christopher X. Candreva wrote:

>This thread on Trojan.JS.RunMe had me thinking: Hourly virus updates is
>better than any of the commercial virus scanners, but obviously still has
>issues, especially since a bunch of us obviously submitted updates that had
>already been entered. I gather from these posts that the virusdb's actually
>have some form of version number.
>
>
>
This could actualy be easily accomplished also by attaching a soa record
to a zone ... for example

dbversion.clamav.net

Incrementing the serial for that should be trivial enough.....writing a
mechanism to rapidly query against it and then to invoke a freshclam is
left as an exercise to the reader.

Presumably then the lists of Nameservers for that particular zone would
be expanded to about 10 or more. Notification from whatever master zone
server could be trivialy accomplished on that.

We should probably consider that the load balancing of all those end
users/isp's DNS resolvers may not be all it can be, particularly the
selection of which nameserver to talk to out of many for a particular zone.

Anyways I did a dig .... Arent CNAMEs that Point to CNAMEs contrary to RFC?
Might that be behind the infrequent dns resolution complaints?
Also... Is there any single name that covers ALL mirrors?
Also.... Any insight as to how the { presumably dynamic } selection to
alias the db-local to db.america is done?

c:\Documents and Settings\joe.JOE.000>dig database.clamav.net

; <<>> DiG 9.2.3rc3 <<>> database.clamav.net
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41
;; flags: qr rd ra; QUERY: 1, ANSWER: 15, AUTHORITY: 5, ADDITIONAL: 2

;; QUESTION SECTION:
;database.clamav.net. IN A

;; ANSWER SECTION:
database.clamav.net. 5 IN CNAME db.local.clamav.net.
db.local.clamav.net. 7200 IN CNAME db.america.clamav.net.
db.america.clamav.net. 5 IN A 128.121.60.235
db.america.clamav.net. 5 IN A 196.40.71.226
db.america.clamav.net. 5 IN A 199.239.233.95
db.america.clamav.net. 5 IN A 200.68.106.39
db.america.clamav.net. 5 IN A 24.244.193.21
db.america.clamav.net. 5 IN A 38.136.139.7
db.america.clamav.net. 5 IN A 64.18.103.6
db.america.clamav.net. 5 IN A 64.69.64.158
db.america.clamav.net. 5 IN A 65.75.154.69
db.america.clamav.net. 5 IN A 65.77.42.207
db.america.clamav.net. 5 IN A 66.139.75.171
db.america.clamav.net. 5 IN A 67.18.205.218
db.america.clamav.net. 5 IN A 69.93.108.98

;; AUTHORITY SECTION:
clamav.net. 7200 IN NS ns5.clamav.net.
clamav.net. 7200 IN NS ns1.oltrelinux.com.
clamav.net. 7200 IN NS ns2.clamav.net.
clamav.net. 7200 IN NS ns3.clamav.net.
clamav.net. 7200 IN NS ns4.clamav.net.

;; ADDITIONAL SECTION:
ns1.oltrelinux.com. 38516 IN A 194.242.226.43
ns5.clamav.net. 153717 IN A 80.69.66.9

;; Query time: 671 msec
;; SERVER: 64.95.32.37#53(64.95.32.37)
;; WHEN: Tue Aug 10 16:40:04 2004
;; MSG SIZE rcvd: 429




-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tue, 2004-08-10 at 12:40, Christopher X. Candreva wrote:

> If people can't check for database updates more often than once an hour,
> then there is a pressing need.
[...]
> If only 1.3% of every update is actually needed, and people only downloaded
> what they needed, the traffic on the mirrors would drop from 120gig/month to
> 1.6 gig/month.
>
> If I am completely off by a factor of 10 -- say only 10% of every update
> is actually needed, traffic on the mirrors drops from 120gig to 12gig.

That's one of the things that seems to be driving the size of daily.cvd
up - updating main.cvd entails a massive distribution of files to the
world.

Perhaps a tiered approach to the update files, with main.cvd,
monthly.cvd, weekly.cvd, daily.cvd, and hot.cvd

The advantage there is that the really big update could be distributed
very seldom - perhaps only with new code (the code generally has to be
upgraded every few months to deal with a new threat anyway).

If you had overlapping signatures between the files, you could add a
fuzzy-factor into freshclam that it might not bring down the latest
weekly/monthly if the other files overlap completely. That would
distribute the load on the freshclam servers for the larger updates, and
there would just be the very small daily.cvd (and perhaps hot.cvd)
downloads.

I like the idea of using DNS to signal the change - maybe just for
hot.cvd. so, whenever a major virus breakout occurs, the new sig would
be added to hot.cvd and the DNS TXT record changed. 10,000 users
pulling down a 2-3K file is not terribly hard for a server with decent
bandwidth
--
Daniel J McDonald, CCIE 2495, CNX
Austin Energy




-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tue, Aug 10, 2004 at 10:39:19PM +0200, Peter J. Holzer wrote:
> On 2004-08-10 14:41:28 -0500, Damian Menscher wrote:
[... about sending clamav updates quickly to all subscribers]
> > Anyone know if it's really feasible for us to obtain a mailserver that
> > can send out 2k emails to all (100,000?) users in a short (5-10 mins)
> > time?
>
> How about using NNTP instead of SMTP? Then the clamav server doesn't

Why use such an old protocol that isn't suited to binary transfers.

I've already mentioned this jokingly, but I was half serious: I think
setting up a bittorrent would solve a lot of the bandwidth problems.

You would need some place to get the daily.cvd.torrent file, which seems
to be about 170 bytes when I tried creating one yesterday (Small enough
to fit base64-encoded in a DNS TXT record, if you insist, but I doubt
that that is prudent to rely upon). Then you'd need a decent tracker,
or a bunch of trackers, and at least one seeder per tracker. I guess
that the current db.*.clamav.net hosts can easily host both a tracker
and a seeder.

If you then distribute a downloading clients that keeps seeding for just
1 hour (or until a preset share ratio was reached, say, 10x), you would
very quickly take a HUGE load off the download servers... and everyone
using clamav would automatically help the project by donating bandwidth
for the updates.

P2P - it's not just for downloading pirated Metallica mp3s.

HTH,

--
#!perl -wpl # mmfppfmpmmpp mmpffm <pmmppfmfpppppfmmmf@fpffmm4mmmpmfpmf.ppppmf>
$p=3-2*/[^\W\dmpf_]/i;s.[a-z]{$p}.vec($f=join('',$p-1?chr(sub{$_[0]*9+$_[1]*3+
$_[2]}->(map{/p|f/i+/f/i}split//,$&)+97):qw(m p f)[map{((ord$&)%32-1)/$_%3}(9,
3,1)]),5,1)='`'lt$&;$f.eig; # Jan-Pieter Cornet


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tuesday 10 August 2004 02:41 pm, Damian Menscher wrote:
[.snip: using a program delivery to process update mailing list mails]
> With sendmail, you could add to /etc/aliases something like:
> clamav-updates | sigtool --add

that's the ticket.

> Anyone know if it's really feasible for us to obtain a mailserver that
> can send out 2k emails to all (100,000?) users in a short (5-10 mins)
> time? Assuming those numbers are reasonable, that means 200 meg of
> data. Combined with SMTP overhead, it seems like it would be
> troublesome. Additionally, there are potential bandwidth issues if you
> consider we'd need to do that several times/day.

well, I would think this would be an 'optional' thing you could do, or maybe
part of a 'premium' service provided for a fee. As Jef mentioned, most small
time folks are perfectly happy with hourly updates in a pull configuration.

> Updating the "main" database is one concern. Sending out a 2-meg email
> to everyone seems like it might be too much load, but sending out the 1K
> email telling everyone to get it means the mirrors will get swamped. I
> can't think of a way around this, but hopefully someone else can?

well, I would hope that while also grabbing these daily.cvd updates via email,
that the admin is also running freshclam (perhaps less frequently now that
he/she only needs to check main.cvd once a day) to grab the main.cvd and
doesn't need notification for it. Forgive my ignorance if I'm not
interpreting the role of the main/daily.cvd files correctly:
main.cvd: updated daily with all of the updates done to daily.cvd throughout
the day
daily.cvd: 0sec updates to the database, get rolled into main.cvd nightly

> Also, this doesn't give much provision for removing "bad" signatures
> (that cause false positives) since it really just appends rules. We'd
> need to figure out a way to delete signatures also. I could imagine
> doing this by including a "null" signature, or using some other flag.

true. perhaps the first line of the email could be a command, and a simple
sh/perl/c program could parse it and then call the proper commands to add or
remove the signature that follows.

> Finally, there's the whole issue of multiplying your points of failure.
> If your current database is screwed, appending more to it will leave it
> screwed. And if you add stuff to it a few times a day, chances are it
> will get screwed up at some point. At least this issue has a simple
> fix: include an MD5 sum with the update which must match your MD5 sum
> after applying the update. If they don't match, you know something went
> wrong, either with this update or a previous one. (This has the danger
> that if the developers send an email with an incorrect MD5 hash,
> everyone will thrash the mirrors.)

eek.

> Note to the developers: please don't feel like you have to code up any
> of our random ideas. I'm just having fun brainstorming about how to
> optimize this process. I expect in another few days of discussion we'll
> have converged on a fairly sane idea.

or scrap the whole idea all together :)

-Jeremy

--
Jeremy Kitchen ++ Systems Administrator ++ Inter7 Internet Technologies, Inc.
jeremy@inter7.com ++ www.inter7.com ++ 866.528.3530 ++ 847.492.0470 int'l
kitchen @ #qmail #gentoo on EFnet ++ scriptkitchen.com/qmail



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Jeremy Kitchen wrote:
> or scrap the whole idea all together :)

Maybe the best thing written on the subject today! ;-) j/k

But really, what's the problem? Shouldn't "big time folks" complain to
the commercial companies to whom they pay for service and still they got
updates later than Clam? Instead hundreds of mails are written here with
one "solution" more far out than the other.

Please, I *think* you might have caught the attention of the developers
by now so please let them think about this for a moment. They still beat
everyone else so I just want to say thank you. Everything works great!
In combination with MailScanner which checks inside zip files and blocks
executables I stopped all the viruses even before Clam was updated. From
what I have seen from reading this list for some time many of you seem
to rely to heavily on too few layers of protections. Maybe that's why
you "must" have the updates immediately with no regard to server load or
maybe I missed the solution that took care of that one too in the flood
of mail. Premium servers for a fee is the best solution I have seen so far.

No offence meant to anyone in particular.

--
/Peter Bonivart

--Unix lovers do it in the Sun

Sun Fire V210, Solaris 9, Sendmail 8.12.10, MailScanner 4.32.5,
SpamAssassin 2.63 + DCC 1.2.50, ClamAV 0.75.1 + GMP 4.1.2, Vispan 1.4


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Jeremy Kitchen wrote:
> On Tuesday 10 August 2004 02:41 pm, Damian Menscher wrote:
> [.snip: using a program delivery to process update mailing list mails]
>
>>With sendmail, you could add to /etc/aliases something like:
>>clamav-updates | sigtool --add
>
>
> that's the ticket.
>
>

And a cool little DOS tool. Nothing like a well-known email address for a little
fun having. I imagine the blackhats will slam that rather quickly.

dp


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
OK, here's my pitch

I like the DNS idea as a way to push out just the version number of the
update. This "pattern serial number" would be the current version of the
CVD file.

A record like this in tinydns:

'dbversion.clamav.net:447:600

would create a DNS TXT record for "dbversion.clamav.net" with a value of
"447" with a TTL of 600 sec (10 minutes). I see no point in any more
information being recorded.

If freshclam were to initially do that DNS lookup, it could afford to look
every 10 minutes instead of hourly, and would dramatically cut down on the
amount of HTTP (or any other TCP) transactions required.

I think all the comments about using SMTP or NNTP suffer the same problem as
HTTP - they are no where near as fast or as natively "multicast" as DNS is -
oh yeah - and it's UDP too. DNS natively "shares the load", whereas all
other "load sharing" solutions would have to be created.

So I'd envisage freshclam doing the DNS lookup, and if the "pattern number"
TXT record returned is *different* (not smaller! DNS cache poisoning can
affect this solution, so just choose DIFFERENT) than the current "pattern
number", then it should check for an update. This has the advantage that it
could just be a new bit of code added in front of the existing freshclam
code.

The TTL > 0 allows you to even cut down the load on the primary DNS servers.
The ClamAV team should make a "policy" saying people aren't allowed to check
for updates more often than every "TTL" seconds and this within freshclam
would enforce it.

Just my 2c worth

--
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
In a perfect world, wouldn't this be the ultimate application for say,
multicast? Just keep casting the database over and over, when it
changes, you instantly have it! ;-)

--
Robert Blayzor, BOFH
INOC, LLC
rblayzor@inoc.net
PGP: http://www.inoc.net/~dev/
Key fingerprint = 1E02 DABE F989 BC03 3DF5 0E93 8D02 9D0B CB1A A7B0

Press Ctrl-Alt-Del now for IQ test.


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: Idea for more timely virusdb updates [ In reply to ]
> The mirror page talkes about the need for mirrors, about
> exponential growth,
> and how at least a 10mbit pipe is needed to host a mirror. It puts March
> 2004 traffic at about 120gig/month
>

I think I read it differently... I thought it was 120GB / month per mirror
(at that point in time there were 11 mirrors!)

QUOTE (http://www.clamav.net/doc/mirrors/clamav-mirror-howto.txt)
Without mirrors, the traffic on our main site was
100GB/month (May 2003).

On Feb 2004 the traffic on each mirror (11 in total)
reached 120GB/month.
END QUOTE

Not sure if I read it wrong, but that would put total consumption about 1320
GB - makes it more urgent doesn't it?

Unfortunately the round robin - no limits nature makes the "entry price" for
people who want to help too high for some. I wonder in the short term if
there is a way to create a lower % hit mirror which could say take 10% of
the normal average...

at 12GB / month there might be more takers

m/



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: Idea for more timely virusdb updates [ In reply to ]
> I've already mentioned this jokingly, but I was half serious: I think
> setting up a bittorrent would solve a lot of the bandwidth problems.
>

Been playing with that a bit recently - the more I think about it, the more
I like it... saw a website that has built a custom tracker to manage
leeches, and prevent people (regardless of client) from sponging without
contributing...

The old way could remain, for offline / intermittantly or heavily firewalled
users...

The addition of DNS version management could reduce overhead bandwidth that
occurs during useless polls...

The new way could provide higher frequency updates for those willing to
share and contribute some bytes.

m/



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Am Mittwoch, 11. August 2004 04:56 schrieb Robert Blayzor:

Hi,

> In a perfect world, wouldn't this be the ultimate application for say,
> multicast?

No, multicast is not the most efficient solution and does no caching. Honestly
imho the DNS approach is the most feasable and the most efficient solution as
it provides incoherent caching uses udp and reuses a widly available
infrastructure.

Regards,
-- martin

Dipl.-Phys. Martin Konold

e r f r a k o n
Erlewein, Frank, Konold & Partner - Beratende Ingenieure und Physiker
Nobelstrasse 15, 70569 Stuttgart, Germany
fon: 0711 67400963, fax: 0711 67400959
email: martin.konold@erfrakon.de


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: Idea for more timely virusdb updates [ In reply to ]
Daniel J McDonald wrote:

> That's one of the things that seems to be driving the size of
> daily.cvd up - updating main.cvd entails a massive
> distribution of files to the world.

Current main.cvd = 1103636 bytes, last updated on July 8
Current daily.cvd = 156470 bytes

A bit of mental arithmetic suggests thatdaily.cvd grows by about 5KB per
day.

A few sums in my head suggest that total download savings in a month if
main.cvd was updated fortnightly would be around 200KB (circa 3100KB
total download instead of 3300KB), a virtually insignificant difference.

> Perhaps a tiered approach to the update files, with main.cvd,
> monthly.cvd, weekly.cvd, daily.cvd, and hot.cvd

> The advantage there is that the really big update could be
> distributed very seldom - perhaps only with new code (the
> code generally has to be upgraded every few months to deal
> with a new threat anyway).

Big updates often remove false positives, improve detections of existing
viruses, so might still need monthly (or more frequent) updating.

> If you had overlapping signatures between the files, you
> could add a fuzzy-factor into freshclam that it might not
> bring down the latest weekly/monthly if the other files
> overlap completely. That would distribute the load on the
> freshclam servers for the larger updates, and there would
> just be the very small daily.cvd (and perhaps hot.cvd) downloads.

If we could use incremental (or, more correctly, differential) updates
which effectively create a new main.cvd then we could have a large
reduction in the load on the download servers. However, we then have
the problem of ensuring that main.cvd remains consistent.

> I like the idea of using DNS to signal the change - maybe
> just for hot.cvd. so, whenever a major virus breakout
> occurs, the new sig would be added to hot.cvd and the DNS
> TXT record changed. 10,000 users pulling down a 2-3K file is
> not terribly hard for a server with decent bandwidth

I've known DNS servers to completely ignore TTL figures and cache stuff
which should have expired, so this might not be reliable.

Cheers,

Phil
----
Phil Randal
Network Engineer
Herefordshire Council
Hereford, UK


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Aug 10, 2004, at 2:30 PM, Jeremy Kitchen wrote:

> On Tuesday 10 August 2004 12:23 pm, Damian Menscher wrote:
>> The gpg-signature prevents spoofing. And the sequence numbers
>> keep everyone current. The major problems I see are getting clamd to
>> recognize a message targeted for it, and the obvious problems of DoS
>> attacks (someone sending spoofed messages that would suck CPU time
>> decoding the gpg signature).
>
> yes, that's an unfortunate problem with this idea, however, if you
> used, as I
> stated, a special address that uses program delivery, you'd have to
> hack the
> listserver to get everyone's 'subscription' address to be able to do
> this.

Instead of having this piggyback on email, I was thinking more along
the line of a separate protocol just *modeled* after email. Separate
port, separate server daemon for it...maybe it would lessen the chances
of your updates getting filtered by spam filters and/or targeted for
probes and overflow attacks in the process.

That way it isn't hacking the MTAs out there to do work that isn't
meant or related to them...never liked the idea of bending programs
backwards to tack on added functionality. Seems to be another vector
for bug creep :-)


>> [.I haven't given up on DNS updates yet, but it's hard to come up with
>> a
>> clean way to distribute >256 bytes of data that way, which means even
>> single rules don't always fit.]
>
> I wouldn't distribute the rule in DNS, however, a timestamp of sorts
> in dns
> isn't a bad idea.

While DNS is an interesting idea, I'm worried more about what kind of
bugs and glitches this is going to uncover in the process (or what kind
of attacks would be crafted should this idea catches on.) Let's say
the idea does become popular, and clam and other programs out there
start taking advantage of it...I don't know about all of you, but I
didn't set up a DNS server on a system meant for constant hits from
other sources querying it; it's just a little system that can handle
the load of a small network and that's pretty much it :-) And what
about systems that restrict querying to certain IPs? If a service
starts getting abused, that tends to be when (clueful) admins start
taking steps to lock things down; many places with NTP servers, for
example, will host a "public" site until too many people start hitting
and if it starts to become a burden, the time server suddenly
disappears. :-(

-Bart



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Martin Konold wrote:

> No, multicast is not the most efficient solution and does no caching. Honestly
> imho the DNS approach is the most feasable and the most efficient solution as
> it provides incoherent caching uses udp and reuses a widly available
> infrastructure.

Why would you need caching? Especially when using PIM sparse.

--
Robert Blayzor, BOFH
INOC, LLC
rblayzor@inoc.net
PGP: http://www.inoc.net/~dev/
Key fingerprint = 1E02 DABE F989 BC03 3DF5 0E93 8D02 9D0B CB1A A7B0

There are two major products that come out of Berkeley: LSD and UNIX. We
don't believe this to be a coincidence. - Jeremy S. Anderson


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: Idea for more timely virusdb updates [ In reply to ]
> -----Original Message-----
> From: clamav-users-admin@lists.sourceforge.net
> [mailto:clamav-users-admin@lists.sourceforge.net]On Behalf Of Dennis
> Peterson
>
>
> Jeremy Kitchen wrote:
> > On Tuesday 10 August 2004 02:41 pm, Damian Menscher wrote:
> > [.snip: using a program delivery to process update mailing list mails]
> >
> >>With sendmail, you could add to /etc/aliases something like:
> >>clamav-updates | sigtool --add
> >
> >
> > that's the ticket.
> >
> >
>
> And a cool little DOS tool. Nothing like a well-known email
> address for a little
> fun having. I imagine the blackhats will slam that rather quickly.
>

The list can have a simple algorithm to filter any 'easy' or mnemotenic
email address thats trying to subscribe. That way we can reduce the problem.

In concern of cvd corruption because a malformed patches, I think that the
updater could backup some old states of the sig file in order to have the
posibility to downgrade without downloading sigs again (manually or
something).
In any case, the patch can always have a 'before MD5' and an 'after MD5',
just to be sure.

Another idea to save bandwidth (in a pull fashion) could be to download
instead of:
daily.cvd
something like:
daily.cvd?myver=XXX&MD5=XXXXXXXXXXXXXXXXXXXXXXXXXX

Were _myver_ and _MD5_ are my sigfile's version and it's MD5 respectively.
The server could make a patch to the last version in real time and send it,
instead of sending the 'big' file. The realtime patch could be done cat'ing
all patches needed together or already having created all the respective
daily_fromN_toCURRENT.cvd patches.
Could also be implemented directly into main.cvd.

Brainstorming is great... as long as I don't have to code resulting ideas ;)

Thanks to all for this great bits.

-Samuel



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Christopher X. Candreva wrote the following on 08/10/2004 07:40 PM :

> If people can't check for database updates more often than once an hour,
>
>then there is a pressing need.
>
>The mirror page talkes about the need for mirrors, about exponential growth,
>and how at least a 10mbit pipe is needed to host a mirror. It puts March
>2004 traffic at about 120gig/month
>
>

The bandwidth used on our mirrors (clamav.inet6.fr) as lowered since
then mainly thanks to the ability of new clients to abort the
transmission of a file when it is the same version as the one they
already have.

The ideal setup would be to push updates instead of clients polling
them. It would requires a separate architecture though (HTTP mirrors
can't push things).

Since some time I am thinking of a bittorrent approach too. Bittorrent
is quite efficient at distributing files and there are implementations
allowing multiple trackers to distribute the remaining server-side load.
If I'm not mistaken, there's only one thing to add :
querying for the latest available torrent which must be signed by the db
authors : DNS would be great for this indeed (hash sigs prevent cache
poisoning and thus distribution of invalid db, but not DoS).

Lionel.


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Am Mittwoch, 11. August 2004 13:53 schrieb Bart Silverstrim:

Hi Bart,

> the idea does become popular, and clam and other programs out there
> start taking advantage of it

DNS was developed exactly for this kind of purpose.

> ...I don't know about all of you, but I
> didn't set up a DNS server on a system meant for constant hits from
> other sources querying it

Sorry, I dont want to sound impolite but please try to become familiar how DNS
works in todays internet.

> ; it's just a little system that can handle
> the load of a small network and that's pretty much it :-)

Your little DNS server will only get queries from you very own clamav
installation but not from _anyone_ else.

> And what
> about systems that restrict querying to certain IPs?

?? This makes no sense. If you system is able to do http with using fqdn then
it is also able to use the DNS.

> If a service
> starts getting abused, that tends to be when (clueful) admins start
> taking steps to lock things down; many places with NTP servers, for
> example, will host a "public" site until too many people start hitting
> and if it starts to become a burden, the time server suddenly
> disappears. :-(

Typically the DNS load will be put on the ISPs. The ISPs prever _very_ much
people using the DNS which is an incoherent or loosly coherent caching and
distributed database to permanent polling, multicast or pushing.

Funny enough: The protocol ideas you are proposeing are putting _more_ load on
the DNS(*) than the direct DNS idea.

(*) it is save to assume that your protocol ideas don't use static ip numbers
but use DNS to do gethostbyname() resolving.

Regards,
-- martin

Dipl.-Phys. Martin Konold

e r f r a k o n
Erlewein, Frank, Konold & Partner - Beratende Ingenieure und Physiker
Nobelstrasse 15, 70569 Stuttgart, Germany
fon: 0711 67400963, fax: 0711 67400959
email: martin.konold@erfrakon.de


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Aug 11, 2004, at 10:32 AM, Martin Konold wrote:

> Am Mittwoch, 11. August 2004 13:53 schrieb Bart Silverstrim:
>
> Hi Bart,
>
>> the idea does become popular, and clam and other programs out there
>> start taking advantage of it
>
> DNS was developed exactly for this kind of purpose.

Storing non-DNS related information for retrieval? As I understand the
proposition (and the original lecture that this idea was based on),
it's was for hiding information in a very small records area of DNS for
propogating information...I don't think the designers for DNS had
spreading AV signatures (or files or other things that have been
proposed, non-Clam related) in mind at the time. Also I was worried
about the fact that DNS servers usually got traffic for updates from
peers and client/server lookups, not spreading files...that would boost
their hits and bandwidth.

>> ...I don't know about all of you, but I
>> didn't set up a DNS server on a system meant for constant hits from
>> other sources querying it
>
> Sorry, I dont want to sound impolite but please try to become familiar
> how DNS
> works in todays internet.

Prefacing it with not wanting to sound impolite still makes it sound
impolite but that's okay :-)

I'm not a DNS expert by any means. It's been five years or so since I
set up a bare linux system as an authoritative DNS server, other setups
I've made were all in-house caching servers. I'll make no claim to
knowing how things work exactly or the pitfalls of trying this out, or
the effect it would have on the servers.

But I *have* had enough experience to say that if I have these
questions, chances are someone else on the lists has them too,
vocalizing them or not...and they'll hopefully get an education from
the answers I get as the result of my chiming in :-)

>> ; it's just a little system that can handle
>> the load of a small network and that's pretty much it :-)
>
> Your little DNS server will only get queries from you very own clamav
> installation but not from _anyone_ else.

The proposal is just to have a few DNS servers, the authoritative ones,
seeded with the info then? Others would just cache it? Duh. Makes
sense.

>> And what
>> about systems that restrict querying to certain IPs?
>
> ?? This makes no sense. If you system is able to do http with using
> fqdn then
> it is also able to use the DNS.

I think at the time I was thinking about DNS servers updating from
peers, distributing the load of the records that are spread piecemeal.

> Funny enough: The protocol ideas you are proposeing are putting _more_
> load on
> the DNS(*) than the direct DNS idea.
>
> (*) it is save to assume that your protocol ideas don't use static ip
> numbers
> but use DNS to do gethostbyname() resolving.

You would be correct; it would be the load made by any listserv though.
any spreading of an "email like" server's load would increase lookups
(unless assisted by local caches heavily).

It's not necessarily the load on the server I think I was worrying
about, per se...it's trying to shoehorn the protocol to do more than it
was supposed to. It's probably a misunderstanding of the idea for
spreading the information through DNS, but I would think that the DNS
idea would hit a wall or block that would be imposed by the
restrictions of DNS itself and the way it works, so a new mechanism
would have to supplement it or some other clever hack. I would think
that it would be better to start a propagation idea from scratch rather
than a neat idea (I'm not trying to disregard it...heck, I'm probably
missing something obvious and am worrying about a non-issue) for
extending a protocol meant for task A being extended to also be able to
do task B.

-Bart



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Wed, 11 Aug 2004, Lionel Bouton wrote:

> Since some time I am thinking of a bittorrent approach too. Bittorrent
> is quite efficient at distributing files and there are implementations
> allowing multiple trackers to distribute the remaining server-side load.

Please take this as a question rather than a criticism of the approach:

My experience with bittorrent has been with downloading huge things,
like Fedora. It tends to start up really slowly (since it has to find
peers) and then speed up. But the speedup doesn't occur until several
megs have been downloaded. If we're only sending a 1-meg main.cvd, then
wouldn't bittorrent lose its advantage to all the overhead of finding
peers?


With regard to all the other ideas: Please remember to keep this
*simple*. Here's where I, IMHO, think we stand:

Opening a new port on a mailserver so updates can be pushed to it is a
BAD idea. As a sysadmin, I would not allow such a thing on my
production machines. It creates a huge security risk, since now you
have one more opening to a remote root vulnerability.

The idea of DNS sounds really good, but it doesn't appear we can fit all
the data there. And putting just a version number there appears to make
things worse, since it will just make everyone hit the mirrors at the
same time. If we can somehow distribute signatures that way it would be
nice, but it just doesn't seem practical.

I'm really starting to like the idea of a mailing list that can have
dedicated (and random for each site) subscription addresses and pipe the
list straight into "sigtool --add". It means we'd have to find someone
to host the list, but that's probably no more difficult than finding
someone to host a mirror. Presumably there could even be multiple
"mirrors" sending the list, to improve speed (taking an idea from
spammers who use open relays to do the hard part).

One thing to add to the mailing list approach: there needs to be some
sort of "heartbeat" or "dead man's switch" -- a way to know that the
mailing list is functional, but there are no needed updates, rather than
that the mailing list has broken. I suppose this might be a use for
that latest-db-version.clamav.net idea.

Damian Menscher
--
-=#| Physics Grad Student & SysAdmin @ U Illinois Urbana-Champaign |#=-
-=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc:(217)333-0038 |#=-
-=#| 4602 Beckman, VMIL/MS, Imaging Technology Group:(217)244-3074 |#=-
-=#| <menscher@uiuc.edu> www.uiuc.edu/~menscher/ Fax:(217)333-9819 |#=-
-=#| The above opinions are not necessarily those of my employers. |#=-


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Am Mittwoch, 11. August 2004 14:47 schrieb Robert Blayzor:

Hi,

> > No, multicast is not the most efficient solution and does no caching.
> > Honestly imho the DNS approach is the most feasable and the most
> > efficient solution as it provides incoherent caching uses udp and reuses
> > a widly available infrastructure.
>
> Why would you need caching?

PIM sparse helps in some situations but it put extra load on the routers. I
like the fact the PIM sparse is a "request-only service but still imho a pull
is better than a push approach.

For enterprise use and its security policies using http for the download is
most appropriate especially when using caching upstream http proxies.

> Especially when using PIM sparse.

Regards,
-- martin

Dipl.-Phys. Martin Konold

e r f r a k o n
Erlewein, Frank, Konold & Partner - Beratende Ingenieure und Physiker
Nobelstrasse 15, 70569 Stuttgart, Germany
fon: 0711 67400963, fax: 0711 67400959
email: martin.konold@erfrakon.de


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Damian Menscher wrote the following on 08/11/2004 04:40 PM :

>On Wed, 11 Aug 2004, Lionel Bouton wrote:
>
>
>
>>Since some time I am thinking of a bittorrent approach too. Bittorrent
>>is quite efficient at distributing files and there are implementations
>>allowing multiple trackers to distribute the remaining server-side load.
>>
>>
>
>Please take this as a question rather than a criticism of the approach:
>
>My experience with bittorrent has been with downloading huge things,
>like Fedora. It tends to start up really slowly (since it has to find
>peers) and then speed up. But the speedup doesn't occur until several
>megs have been downloaded. If we're only sending a 1-meg main.cvd, then
>wouldn't bittorrent lose its advantage to all the overhead of finding
>peers?
>
>
>

It depends on the torrents you download. Some starts really quickly.
From my experience it really depends on how much clients are really
available to upload files and the ratio of good/bad clients known by the
tracker : when numerous clients stop distributing content after
finishing a download, some (and sometimes most of the) clients the
tracker send you are now unavailable and you then start having timeouts
and asking other clients to the tracker. In a more controlled
environnement where the client behaviour will be hard-coded in a
hypothetical freshclam-torrent to serve the last two db files and
properly warn the trackers when they stop distributing contents, you
most probably won't experience such problems.
Remember that bittorrent was designed specifically to handle the very
same problem we are discussing, the quality of some popular
implementations may damage the efficiency of the protocol, but in the
clamav context we don't have to deal with this problem.

>With regard to all the other ideas: Please remember to keep this
>*simple*. Here's where I, IMHO, think we stand:
>
>Opening a new port on a mailserver so updates can be pushed to it is a
>BAD idea. As a sysadmin, I would not allow such a thing on my
>production machines. It creates a huge security risk, since now you
>have one more opening to a remote root vulnerability.
>
>

The freshclam process is less risky because tricking freshclam into
connecting to an arbitrary system requires injecting DNS entries
somewhere, controlling the routing process or even injecting content in
TCP conversations which all are more complex techniques than exploiting
a buffer overflow on a server for example. But the problem is not really
in opening ports or not but in controlling the quality of the code you
run. TCP and DNS are not secure so freshclam must add its security layer
above, theoritically it doesn't really matter if it is running as a TCP
client or server.

>The idea of DNS sounds really good, but it doesn't appear we can fit all
>the data there. And putting just a version number there appears to make
>things worse, since it will just make everyone hit the mirrors at the
>same time.
>

Using DNS doesn't mean the freshclam processes will all be waking up at
the same time to query the DNS entry. They may use shorter delays than
the current 1 or 2 hours, but even with one second delay, the queries
leaving the local DNS servers in the recursive process would be
distributed on a TTL-wide period. The TTL being controlled by the
authoritative servers.

If there's a problem of bandwidth/load on the mirrors, then move it away
from them by designing a new (parallel ?) distribution mechanism or
throw more mirrors on it. But as I said I'm not yet convinced there's a
real problem. Shorting the delay from db update to db install on each
mail server is good, but don't forget that 1 hour is already low
compared to the time between the first virus appearance and its
signature addition in the database. We will never eliminate that delay
and must do with it.



> If we can somehow distribute signatures that way it would be
>nice, but it just doesn't seem practical.
>
>I'm really starting to like the idea of a mailing list that can have
>dedicated (and random for each site) subscription addresses and pipe the
>list straight into "sigtool --add". It means we'd have to find someone
>to host the list, but that's probably no more difficult than finding
>someone to host a mirror.
>

Are you kidding ? Our mirrors have seen more than 120000 clamav clients
last month and we only see a *part* of the *european* clients. Mirrors
handled with round-robin DNS are fairly easy to setup, but sending
120000 mails for each db update can very well end up burying one mail
server under the load.

> Presumably there could even be multiple
>"mirrors" sending the list, to improve speed (taking an idea from
>spammers who use open relays to do the hard part).
>
>

Using spammers' techniques could help but at quite a cost in complexity
and in the end I don't think it would be practical for the end-users.


--
Lionel Bouton - inet6
---------------------------------------------------------------------
o Siege social: 51, rue de Verdun - 92158 Suresnes
/ _ __ _ Acces Bureaux: 33 rue Benoit Malon - 92150 Suresnes
/ /\ /_ / /_ France
\/ \/_ / /_/ Tel. +33 (0) 1 41 44 85 36
Inetsys S.A. Fax +33 (0) 1 46 97 20 10




-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Lionel Bouton wanted us to know:

>Since some time I am thinking of a bittorrent approach too. Bittorrent
>is quite efficient at distributing files and there are implementations
>allowing multiple trackers to distribute the remaining server-side load.

There's a libbt library written in C that could accomplish this without
having to spawn a scripting language interpreter (eg python).

>If I'm not mistaken, there's only one thing to add :
>querying for the latest available torrent which must be signed by the db
>authors : DNS would be great for this indeed (hash sigs prevent cache
>poisoning and thus distribution of invalid db, but not DoS).

I like this idea, but want to study it a little more as I'm afraid of
what will happen when a virus comes out that DOS's clamav nameservers
(thus preventing or making much more difficult the proliferation of
signatures to detect it). I think in such a case though, a fallback of
"poll for it anyway" would be sufficient but only if DNS A records are
still responsive.
--
Regards... Todd
They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety. --Benjamin Franklin
Linux kernel 2.6.3-15mdkenterprise 2 users, load average: 0.14, 0.06, 0.02


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: Idea for more timely virusdb updates [ In reply to ]
> Opening another port is simply no option for any serious
> enterprise use. There
> is simply no way to open another port in the firewall. In addition I am
> confident that IANA will not allow to reserve a fixed port number
> for this
> service. After all port numbers are a limited resource with todays IPv4
> networks.

bittorrent doesn't rely on a fixed port - it doesn't need one.

If I understand it right, seeders (people with full copies) and peers
(people with partial downloads) register their ports with the tracker, and
if shutdown properly, de-register themselves.

The problem with slow starts DEFINITELY has something to do with seeders and
peers not deregistering themselves (I see it in logs) and have seen FAQ's on
web sites hosting torrent files to this effect.

With a closed loop distribution system with a custom client that guaranteed
a 10X ratio and then cleanly deregistered itself, we would have a very
powerful distribution system.

Might even make a project on it's own.

m/



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Am Wednesday 11 August 2004 16:19 schrieb Bart Silverstrim:

Hi Bart,

> > DNS was developed exactly for this kind of purpose.
>
> Storing non-DNS related information for retrieval? As I understand the
> proposition (and the original lecture that this idea was based on),
> it's was for hiding information in a very small records area of DNS for
> propogating information...I don't think the designers for DNS had
> spreading AV signatures (or files or other things that have been
> proposed, non-Clam related) in mind at the time.

Please be aware of the fact that I don't think that DNS is the correct tool to
distribute files but for distributing something like a serial number it fits
perfectly.

The actual download of the data has to be done outside of the DNS system.

> Also I was worried
> about the fact that DNS servers usually got traffic for updates from
> peers and client/server lookups, not spreading files...that would boost
> their hits and bandwidth.

I am afraid this is a missunderstanding as I am not advocating to distribute
files via the dns but only a serial number.

> vocalizing them or not...and they'll hopefully get an education from
> the answers I get as the result of my chiming in :-)

I think this is totally fine :-)

> was supposed to. It's probably a misunderstanding of the idea for
> spreading the information through DNS, but I would think that the DNS
> idea would hit a wall or block that would be imposed by the
> restrictions of DNS itself and the way it works, so a new mechanism
> would have to supplement it or some other clever hack. I would think
> that it would be better to start a propagation idea from scratch rather
> than a neat idea (I'm not trying to disregard it...heck, I'm probably
> missing something obvious and am worrying about a non-issue) for
> extending a protocol meant for task A being extended to also be able to
> do task B.

In short: Using DNS for spreading _small_ pieces of information which
typically have some reasonable TTL (time to live) and which can deal with the
loose coherency of DNS is totally fine.

Abusing the DNS to directly transfer files etc is not appropriate as the DNS
infrastructure is not ready for such kind of "abuse".

Regards,
-- martin

Dipl.-Phys. Martin Konold

e r f r a k o n
Erlewein, Frank, Konold & Partner - Beratende Ingenieure und Physiker
Nobelstrasse 15, 70569 Stuttgart, Germany
fon: 0711 67400963, fax: 0711 67400959
email: martin.konold@erfrakon.de


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Am Wednesday 11 August 2004 16:40 schrieb Damian Menscher:

Hi,

> like Fedora. It tends to start up really slowly (since it has to find
> peers) and then speed up. But the speedup doesn't occur until several
> megs have been downloaded. If we're only sending a 1-meg main.cvd, then
> wouldn't bittorrent lose its advantage to all the overhead of finding
> peers?

IMHO this is a very valid concern.

> With regard to all the other ideas: Please remember to keep this
> *simple*. Here's where I, IMHO, think we stand:
>
> Opening a new port on a mailserver so updates can be pushed to it is a
> BAD idea. As a sysadmin, I would not allow such a thing on my
> production machines. It creates a huge security risk, since now you
> have one more opening to a remote root vulnerability.

Opening another port is simply no option for any serious enterprise use. There
is simply no way to open another port in the firewall. In addition I am
confident that IANA will not allow to reserve a fixed port number for this
service. After all port numbers are a limited resource with todays IPv4
networks.

> The idea of DNS sounds really good, but it doesn't appear we can fit all
> the data there.

Yes.

> And putting just a version number there appears to make
> things worse, since it will just make everyone hit the mirrors at the
> same time.

This is not really such a big problem as the DNS is still no push but a pull
service and the incoherency of the DNS leads to a smoothing effect.

> If we can somehow distribute signatures that way it would be
> nice, but it just doesn't seem practical.

I agree with you.

> I'm really starting to like the idea of a mailing list that can have

This is a very bad idea. As someone who is used to run _very_ big mailing
lists I can tell you that the resources to run a _big_ mailing list are 3 or
even 4 magnitudes bigger than a simple webserver offering the very same
single file to everyone interested via HTTP GET.

Offering this file e.g. 1MB via http get allows very easy to saturate any
backbone with useful data _without_ the need to handle DNS lookups,
generating an email, try delivery (multiple packages back and forth) and then
finally having about 3-5 percent of the connection be failures -->
retries,....

Using the very same hw resources (cpu, memory and bandwidth) with http get
allows for much more (think about a factor of 100 or 1000) information be
spread within a timeinterval.

Things to think about:
- Effort required to create a mail body
- overhead of 7bit email encoding
- effort required to do the email envelope
- Effort for queuing many emails (much copying on the server)
- No caching on the intermediate servers (only proxying)
- Handling bounces etc.
- Doing many DNS lookups

Regards,
-- martin

Dipl.-Phys. Martin Konold

e r f r a k o n
Erlewein, Frank, Konold & Partner - Beratende Ingenieure und Physiker
Nobelstrasse 15, 70569 Stuttgart, Germany
fon: 0711 67400963, fax: 0711 67400959
email: martin.konold@erfrakon.de


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Am Wednesday 11 August 2004 18:56 schrieb Mitch (WebCob):

Hi Mitch,

> > service. After all port numbers are a limited resource with todays IPv4
> > networks.
>
> bittorrent doesn't rely on a fixed port - it doesn't need one.

My above comment was an answer to the idea of letting an MTA run on an extra
fixed port. It was in no way related to bittorrent.

The problem with bittorent is that bittorent addresses a different problem
domain.

clamav pattern update:
- frequently changing small number of small files distributed from a single
point to many

bittorrent:
- slowly changing high number of potentially very big files distributed from
many sources to many destinations.

Regards,
-- martin

Dipl.-Phys. Martin Konold

e r f r a k o n
Erlewein, Frank, Konold & Partner - Beratende Ingenieure und Physiker
Nobelstrasse 15, 70569 Stuttgart, Germany
fon: 0711 67400963, fax: 0711 67400959
email: martin.konold@erfrakon.de


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
OK folks, chiming in again with my more or less educated opinion.

DNS was built exactly to distribute fast and reliably small chunks of
information (most or the time address related information to start with).
Is an MX record now exactly an address related information, one would guess
yes. Was DNS based blacklisting thought about when the BIND system was
developed? I doubt it, still I believe most of you admins around use it in
some form. The DNS system will survive much heavier load then we could
possibly ever burden it with.

I believe a pull method is preferred over a push method, because it is
easier to manage regionally. A system administrator could even decide to
use a specific group of servers. I doubt that any sensible sysadmin will
allow a push to be implemented anyway .... seee security risk.

To build smaller update chunks I beg you to reconsider the patch method,
even though the files are binary. They could easily be converted to a
textual multi line representation, then diffed with enough redundant
information to assert the patch is inserted at the right place, possibly
compressed for transport and applied to a text converted file, which of
course would need to be converted back, checksummed, compared, you name it.
All the necessary tools to do this are standard on *nix systems and there
is always cygwin for the Redmond addicts.

cheers
Erich

THINK
Püntenstrasse 39
8143 Stallikon
mailto:erich.titl@think.ch
PGP Fingerprint: BC9A 25BC 3954 3BC8 C024 8D8A B7D4 FF9D 05B8 0A16




-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Aug 11, 2004, at 10:40 AM, Damian Menscher wrote:

> On Wed, 11 Aug 2004, Lionel Bouton wrote:
>
>> Since some time I am thinking of a bittorrent approach too. Bittorrent
>> is quite efficient at distributing files and there are implementations
>> allowing multiple trackers to distribute the remaining server-side
>> load.
>
> Please take this as a question rather than a criticism of the approach:
>
> My experience with bittorrent has been with downloading huge things,
> like Fedora. <snip>

I've never used bittorrent so I'm afraid I can't comment there.

> With regard to all the other ideas: Please remember to keep this
> *simple*. Here's where I, IMHO, think we stand:
>
> Opening a new port on a mailserver so updates can be pushed to it is a
> BAD idea. As a sysadmin, I would not allow such a thing on my
> production machines. It creates a huge security risk, since now you
> have one more opening to a remote root vulnerability.

Just a clarification of what I accidentally proposed earlier: it
wouldn't be so much a mail server doing this, just a daemon and
application modeled after mail. I think it's pretty clear thanks to
SPAMmers all around the Internet that email protocols are broken, but
the basic simplicity of the model behind SMTP could be applied to send
out a subscription of encrypted and signed updates to people who sign
up for it...it would be more like a whitelisted email system in
*concept* only. I was proposing that only because the basics are
already out there in framework in the form of email...but we don't want
something that accepts random or additional info. Just updates for our
Clam programs.

Yes, it could be another root vulnerability, and it would be a bigger
target because AV kiddies are usually the kind that are more likely to
attempt DOS attacks against servers if there's a central target to hit.

> I'm really starting to like the idea of a mailing list that can have
> dedicated (and random for each site) subscription addresses and pipe
> the
> list straight into "sigtool --add". It means we'd have to find someone
> to host the list, but that's probably no more difficult than finding
> someone to host a mirror. Presumably there could even be multiple
> "mirrors" sending the list, to improve speed (taking an idea from
> spammers who use open relays to do the hard part).
>
> One thing to add to the mailing list approach: there needs to be some
> sort of "heartbeat" or "dead man's switch" -- a way to know that the
> mailing list is functional, but there are no needed updates, rather
> than
> that the mailing list has broken. I suppose this might be a use for
> that latest-db-version.clamav.net idea.

Here's a second idea to combine with the first...use a freenet model.
At least I think that's the name...

It's P2P and anonymous; and (my memory is foggy...can someone confirm
the details?) it is kind of like a mesh "network within a network". It
was originally meant as a totally free and distributed way for P2P
transfers of information. Everyone shares information and it gets
distributed on computers, and you as a client/server have no control or
idea what is in your allocated "sharing space". Could be illegal
material, could be shakespeare, you don't know (Freenet, that is).

If we had a meshed system of a "live network within a network" of
updates with this model, it may be an interesting infrastructure not
only for rapid updates, but impossible (improbable?) denial of service
attacks, and the possibility of even tagging exe's for analysis
later...they could just be swept up in the "grid" and analyzed when
they reach the appropriate team members. The sigs could be updates and
swept into the grid where they'd be distributed to sysadmins.

Again, probably impractical, but just enjoying the brainstorming that's
going on on the list recently. :-) It would be more complicated that
previous ideas, yes, but it may lay groundwork for future features or
ideas, like maybe a way to monitor virus activity and send out
statistics to users who wish to setup that ability for monitoring
outbreaks in certain regions of the Internet.

-Bart



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Aug 11, 2004, at 1:22 PM, Martin Konold wrote:

> Am Wednesday 11 August 2004 16:19 schrieb Bart Silverstrim:
>
> Hi Bart,
>
>>> DNS was developed exactly for this kind of purpose.
>>
>> Storing non-DNS related information for retrieval? As I understand
>> the
>> proposition (and the original lecture that this idea was based on),
>> it's was for hiding information in a very small records area of DNS
>> for
>> propogating information...I don't think the designers for DNS had
>> spreading AV signatures (or files or other things that have been
>> proposed, non-Clam related) in mind at the time.
>
> Please be aware of the fact that I don't think that DNS is the correct
> tool to
> distribute files but for distributing something like a serial number
> it fits
> perfectly.
>
> The actual download of the data has to be done outside of the DNS
> system.

Okay, this would be where my misunderstanding was entering the picture
:-) I was thinking DNS would be the distribution engine, not the
notification engine.

Sorry for the confusion!

> Abusing the DNS to directly transfer files etc is not appropriate as
> the DNS
> infrastructure is not ready for such kind of "abuse".

Give it time...someone's going to do it.

I'm surprised someone hasn't found a hole in DNS that would allow it to
act as a way to distribute viruses via DNS records yet...

-Bart



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Am Thursday 12 August 2004 15:16 schrieb Bart Silverstrim:

> > Abusing the DNS to directly transfer files etc is not appropriate as
> > the DNS
> > infrastructure is not ready for such kind of "abuse".
>
> Give it time...someone's going to do it.

Well, people have tried to abuse it before fortunately they can be rather
easily blocked because the DNS system is strictly hierarchical.

> I'm surprised someone hasn't found a hole in DNS that would allow it to
> act as a way to distribute viruses via DNS records yet...

Provided there is a bug in the resolver or DNS server code somewhere of course
you can attack systems using malicious DNS records. There have been several
examples in the past.

I will stop now in this thread because we are becoming seriously off topic.

Regards,
-- martin

Dipl.-Phys. Martin Konold

e r f r a k o n
Erlewein, Frank, Konold & Partner - Beratende Ingenieure und Physiker
Nobelstrasse 15, 70569 Stuttgart, Germany
fon: 0711 67400963, fax: 0711 67400959
email: martin.konold@erfrakon.de


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Wed, Aug 11, 2004 at 08:34:48PM +0200, Martin Konold wrote:
> The problem with bittorent is that bittorent addresses a different problem
> domain.
>
> clamav pattern update:
> - frequently changing small number of small files distributed from a single
> point to many
>
> bittorrent:
> - slowly changing high number of potentially very big files distributed from
> many sources to many destinations.

This isn't correct. You somehow confusingly assume all current
bittorrent downloads are related? They are not. Each individual .torrent
starts out as a one-to-many distribution.

The nice thing about bittorrent is that practically immediately after
a third client connects, it becomes a many-to-many transfer, utilising
the available upload capacity of all clients.

So each individual torrent you find on those popular websites that list
all torrents, started as a single-point-to-many distribution. And the
number of torrents available there isn't slowly changing, in fact, it's
often changing way faster than new virus definitions are released :)

The main difference is that most currently offered torrents comprise
many megabytes, while a virus definition file would only be a few
kilobytes. But that doesn't invalidate the protocol, certainly not with
a high number of downloaders.

If anyone has questions on how the bittorrent protocol works, there
is quite a bit of info on the official website:
http://www.bitconjurer.org/BitTorrent/
and there's a wiki FAQ: http://wiki.theory.org/index.php/BitTorrentFAQ

--
#!perl -wpl # mmfppfmpmmpp mmpffm <pmmppfmfpppppfmmmf@fpffmm4mmmpmfpmf.ppppmf>
$p=3-2*/[^\W\dmpf_]/i;s.[a-z]{$p}.vec($f=join('',$p-1?chr(sub{$_[0]*9+$_[1]*3+
$_[2]}->(map{/p|f/i+/f/i}split//,$&)+97):qw(m p f)[map{((ord$&)%32-1)/$_%3}(9,
3,1)]),5,1)='`'lt$&;$f.eig; # Jan-Pieter Cornet


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Wed, Aug 11, 2004 at 03:07:35PM +0200, Lionel Bouton wrote:
> The ideal setup would be to push updates instead of clients polling
> them. It would requires a separate architecture though (HTTP mirrors
> can't push things).
>
> Since some time I am thinking of a bittorrent approach too. Bittorrent
> i...

All this should fail for the *majority* of ClamAV sites!!

Push updates implies people have put "clam servers" out on the Internet so
that they are reachable - I don't think so! That's what firewalls were
invented for.

Similarly, BitTorrent *requires* "raw" Internet access in order to operate -
again - not a normal situation for an AV server.

DNS for serial numbers plus HTTP for actual data transfer still sounds best
to me... All outgoing connections only, all well established (nothing exotic)


--
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Sat, 14 Aug 2004 08:02:51 +1200
Jason Haar <Jason.Haar@trimble.co.nz> wrote:

> DNS for serial numbers plus HTTP for actual data transfer still sounds

New version of freshclam will work in this way. Big thanks to all for
the interesting thread !

--
oo ..... Tomasz Kojm <tkojm@clamav.net>
(\/)\......... http://www.ClamAV.net/gpg/tkojm.gpg
\..........._ 0DCA5A08407D5288279DB43454822DC8985A444B
//\ /\ Fri Aug 13 22:26:46 CEST 2004
RE: Idea for more timely virusdb updates [ In reply to ]
> Similarly, BitTorrent *requires* "raw" Internet access in order
> to operate -
> again - not a normal situation for an AV server.
>

Don't know what exactly you meant by "raw" as opposed to sauteed, broiled,
baked or toasted, but BitTorrent does NOT require unfirewalled access. It
does require a small port range to be forwarded to it, BUT that port range
is not required to be the same on any two hosts.

When the host contacts the tracker, it tells the tracker which ports it is
listening on so the tracker can distribute load to it.

m/



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: Idea for more timely virusdb updates [ In reply to ]
> > DNS for serial numbers plus HTTP for actual data transfer still sounds
>
> New version of freshclam will work in this way. Big thanks to all for
> the interesting thread !
>

Sounds cool Tomasz! Be interested to hear if this helps reduce the load on
the mirrors at all. Once this is tested, an update to recommended polling
times would be appreciated (for anyone not running freshclam as a daemon)

Thanks!

m/



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Fri, 13 Aug 2004, Tomasz Kojm wrote:

> New version of freshclam will work in this way. Big thanks to all for
> the interesting thread !

That's C-a-n-d-r-e-v-a .

For the CHANGES file.
:-)

-Chris

==========================================================
Chris Candreva -- chris@westnet.com -- (914) 967-7816
WestNet Internet Services of Westchester
http://www.westnet.com/


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Fri, Aug 13, 2004 at 02:22:55PM -0700, Mitch (WebCob) wrote:
> Don't know what exactly you meant by "raw" as opposed to sauteed, broiled,
> baked or toasted, but BitTorrent does NOT require unfirewalled access. It
> does require a small port range to be forwarded to it, BUT that port range
> is not required to be the same on any two hosts.

Well that means "raw" :-) It means the *incoming* ports involved need to be
open to the Internet. Creating outgoing SMTP/HTTP/FTP from within a
firewalled network doesn't mean you have to open up your firewall for any
incoming ports - BitTorrent does require that.

That falls into the "must have really good business case - can we put you in
a standalone DMZ?" case for most largish companies.


--
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Am Friday 13 August 2004 23:23 schrieb Mitch (WebCob):

Hi,

> > > DNS for serial numbers plus HTTP for actual data transfer still sounds

> > New version of freshclam will work in this way.

> the mirrors at all. Once this is tested, an update to recommended polling
> times would be appreciated (for anyone not running freshclam as a daemon)

Actually this is not required. A very nice feature of DNS is that it not only
transports the information (serial number) but also the metainformation when
to refetch the data.

This means that the clamav nameserver control the TTL (time to live) of the
data.

A nice side effect of this is that it allows for a flexible _server_ based
control of the DNS retry intervals of the clients.

E.g. the virusdb people get a new signature/virus. While investigating the
data they might consider to already lower the TTL so that the clients check
the DNS more frequently. As soon as the new signature/virusdb is available
the clients notice this rather fast.

In addition by managing the TTL the server can control how hard it is hit when
a new signature is available. Shorter TTL means less smoothing but better
latency.

IMHO using the DNS intelligently helps not only for the QoS but also the
internet as a whole because it avoids unnecessary and unproductive load on
all systems involved.

Side note: Lets hope that no 1$107 gets the idea to patent such a mechanism as
it just happend recently to us KDE people. (A KDE person had a UI idea and
presented it on a KDE ml. KDE implemented this idea in the following months
but MS took the idea, added the obvious and patented it :-(. MS even cites
the KDE ml as the source of the initial idea.....)

Regards,
-- martin

Dipl.-Phys. Martin Konold

e r f r a k o n
Erlewein, Frank, Konold & Partner - Beratende Ingenieure und Physiker
Nobelstrasse 15, 70569 Stuttgart, Germany
fon: 0711 67400963, fax: 0711 67400959
email: martin.konold@erfrakon.de


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Fri, 13 Aug 2004 22:34:43 +0200
Tomasz Kojm <tkojm@clamav.net> wrote:
>On Sat, 14 Aug 2004 08:02:51 +1200
>Jason Haar <Jason.Haar@trimble.co.nz> wrote:
>
>> DNS for serial numbers plus HTTP for actual data
>>transfer still sounds

>
>New version of freshclam will work in this way. Big
>thanks to all for
>the interesting thread !
>

How about incremental updates? Can it possibly make way to
next freshclam versions too?

Checking serial number (or version) via DNS will certainly
reduce some traffic on the mirrors, but I'm not sure how
much of the 91.53 GB (last month traffic here for *.cvd)
will be reduced. Version checking (assuming most clients
use the new freshclam) in my raw calculation uses 173840
(cvd hit) * 512 bytes = 85 MB. Not much.

If we can use incremental update, for each database update
we'll be distributing small ammount of data (say ... 1 - 5
kB) instead of the 162kB daily.cvd (or more) that we have
now.

If we distribute the incremental update via DNS (I still
have a thing for DNS), ISPs will cache this too so mirror
traffic will be much much much reduced.

I have a Linode with 25 GB monthly bandwidth on a fast
datacenter, and I'll be happy to contribute half of it for
clamav db mirror. But since mirror bandwidth requirement
is SOOO high, I can't do that can I? I'm sure that with
lower bandwidth requirement many people will be happy to
pinch in.

Regards,

Fajar


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Sat, Aug 14, 2004 at 01:07:06PM +0700, Fajar Nugraha wrote:
> How about incremental updates? Can it possibly make way to
> next freshclam versions too?

Too true. Some commercial AVs separate "patterns" into a library (large)
plus incremental files (small).

Typically the next "library" release contains the previous incremental
patterns, and the actual incremental files are deleted - stopping crustiness
from developing...

daily.cvd is 160K at the moment, and will be totally re-downloaded the next
time it's updated. However, this fiddling around trying to remember which
old one-off pattern files can now be deleted/ignored does add complexity -
something "daily.cvd" doesn't have to worry about...

I suspect the digital signing of those files might be screwing up
alternative options on this? After all, you don't want to be checking sigs
on 12 files before actually being able to start doing AV! Perhaps a
different way could be brought into play...

Can the digital sig be separated from the daily file, and have the current
CVD file changed into a better format that rsync could do a good job of
incrementing/appended it? That way an update would in effect be appending
some lines to the "daily.RAW" file, plus downloading the new (tiny) digital
signature of that file?


Just a thought

-
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Saturday 14 August 2004 07:11 am, Jason Haar wrote:
> On Sat, Aug 14, 2004 at 01:07:06PM +0700, Fajar Nugraha wrote:
> > How about incremental updates? Can it possibly make way to
> > next freshclam versions too?
>
> Too true. Some commercial AVs separate "patterns" into a library (large)
> plus incremental files (small).
>
> Typically the next "library" release contains the previous incremental
> patterns, and the actual incremental files are deleted - stopping
> crustiness from developing...

I still don't see why rsync can't be used here. It can easily do incremental
updates.

-Jeremy

--
Jeremy Kitchen ++ Systems Administrator ++ Inter7 Internet Technologies, Inc.
jeremy@inter7.com ++ www.inter7.com ++ 866.528.3530 ++ 847.492.0470 int'l
kitchen @ #qmail #gentoo on EFnet ++ scriptkitchen.com/qmail



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Sun, 15 Aug 2004 08:25:36 -0500
Jeremy Kitchen <kitchen@scriptkitchen.com> wrote:
>On Saturday 14 August 2004 07:11 am, Jason Haar wrote:
>> On Sat, Aug 14, 2004 at 01:07:06PM +0700, Fajar Nugraha
>>wrote:
>> > How about incremental updates?

>I still don't see why rsync can't be used here. It can
>easily do incremental
>updates.

True. However,
(1) many firewall admins allow outgoing HTTP and DNS
ports; I cannot say the same for rsync port.
(2) The uncompressed signature (viruses.db*) files is a
good candidate for rsync (or even a simple diff command).
I don't know how well rsync or diff performs on the
compressed-signed *.cvd.

Regards,

Fajar


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
At 18:40 15.08.2004, Fajar Nugraha wrote:
>On Sun, 15 Aug 2004 08:25:36 -0500
> Jeremy Kitchen <kitchen@scriptkitchen.com> wrote:
>>On Saturday 14 August 2004 07:11 am, Jason Haar wrote:
>>>On Sat, Aug 14, 2004 at 01:07:06PM +0700, Fajar Nugraha wrote:
>>> > How about incremental updates?
>
>>I still don't see why rsync can't be used here. It can easily do
>>incremental updates.
>
>True. However, (1) many firewall admins allow outgoing HTTP and DNS ports;
>I cannot say the same for rsync port.
>(2) The uncompressed signature (viruses.db*) files is a good candidate for
>rsync (or even a simple diff command). I don't know how well rsync or diff
>performs on the compressed-signed *.cvd.

badly....., unless you make those incremental too, e.g. daily.454.cvd and
provide means to join those.

cheers
Erich


THINK
Püntenstrasse 39
8143 Stallikon
mailto:erich.titl@think.ch
PGP Fingerprint: BC9A 25BC 3954 3BC8 C024 8D8A B7D4 FF9D 05B8 0A16




-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: Idea for more timely virusdb updates [ In reply to ]
> >I still don't see why rsync can't be used here. It can
> >easily do incremental
> >updates.
>
> True. However,
> (1) many firewall admins allow outgoing HTTP and DNS
> ports; I cannot say the same for rsync port.
> (2) The uncompressed signature (viruses.db*) files is a
> good candidate for rsync (or even a simple diff command).
> I don't know how well rsync or diff performs on the
> compressed-signed *.cvd.

Hmmm... interesting points... but what about this option?

Rsync and diff are generic "patching" mechanisms meant to accomodate data
without a known format - we don't have that problem here.

My understanding is that for the most part database updates are additions,
though sometimes there may be deletions or updates to preexisting keys....

Lets say on the SERVER side, those updates were kept in something of the
form:

version|status|signature|md5

Where version is the version number containing the change...
status is + (new sig), - (remove sig), or = (update sig) (the sematics are
important, the values of the enum are not of course)
and signature contains whatever the current fields of the database are...
md5 would be the checksum of a database if all patches applied to this point
are sucessful

Then, any freshclam could connect, something like:

http://somemirror.db?version=xxx

The server would then return all updates > xxx, which would allow the
freshclam to patch it's local database, and verify the last md5 is a match
for the md5 of the updated local db. If the update fails to produce a
matching checksum, freshclam could then pull a fresh copy in it's entirety.


This would mean the mirrors would have to support basic scripting (PHP?) but
we could trade a significant portion of the bandwidth for a few cpu
cycles...

m/



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Hi,

new freshclam is ready for testing:

Sun Aug 22 02:07:13 CEST 2004 (tk)
----------------------------------
* freshclam: Support version verification through DNS (DNSDatabaseInfo).
Based on idea by Christopher X. Candreva <chris*westnet.com>,
see http://www.gossamer-threads.com/lists/clamav/users/11102

Because most users are happy with hourly checks we didn't decide to use
a ttl-related instant checking (i.e. sleep(ttl); check; slep(ttl)...) in
daemon mode but removed the limit of maximal checks instead (only in
DNSDatabaseInfo mode).

We're not willing to use DNS for database updates. On September 1, the
databases will be restructured and the size of daily.cvd limited.

Thanks,

--
oo ..... Tomasz Kojm <tkojm@clamav.net>
(\/)\......... http://www.ClamAV.net/gpg/tkojm.gpg
\..........._ 0DCA5A08407D5288279DB43454822DC8985A444B
//\ /\ Sun Aug 22 02:28:48 CEST 2004
Re: Idea for more timely virusdb updates [ In reply to ]
libresolv don't exist on FreeBSD.

Regards

gmake[2]: Entering directory
`/usr/ports/security/clamav-devel/work/clamav-devel-20040822/clamdscan'
/bin/sh ../libtool --mode=link cc -O3 -pipe -funroll-loops -ffast-math
-march=pentiumpro -I/usr/local/include -L/usr/local/lib -lcipher -o
clamdscan output.o cfgparser.o memory.o clamdscan.o client.o
../clamscan/options.o ../clamscan/getopt.o ../libclamav/libclamav.la
-lresolv
cc -O3 -pipe -funroll-loops -ffast-math -march=pentiumpro
-I/usr/local/include -o .libs/clamdscan output.o cfgparser.o memory.o
clamdscan.o client.o ../clamscan/options.o ../clamscan/getopt.o
-L/usr/local/lib ../libclamav/.libs/libclamav.so -lcipher -lz -lgmp
-lc_r -lresolv -Wl,--rpath -Wl,/usr/local/lib
/usr/libexec/elf/ld: cannot find -lresolv

Tomasz Kojm wrote:
> Hi,
>
> new freshclam is ready for testing:
>
> Sun Aug 22 02:07:13 CEST 2004 (tk)
> ----------------------------------
> * freshclam: Support version verification through DNS (DNSDatabaseInfo).
> Based on idea by Christopher X. Candreva <chris*westnet.com>,
> see http://www.gossamer-threads.com/lists/clamav/users/11102
>
> Because most users are happy with hourly checks we didn't decide to use
> a ttl-related instant checking (i.e. sleep(ttl); check; slep(ttl)...) in
> daemon mode but removed the limit of maximal checks instead (only in
> DNSDatabaseInfo mode).
>
> We're not willing to use DNS for database updates. On September 1, the
> databases will be restructured and the size of daily.cvd limited.
>
> Thanks,
>

--
Marcus Grando
Grupos Internet S/A
marcus(at)corp.grupos.com.br



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Tomasz Kojm wrote:

>Hi,
>
>new freshclam is ready for testing:
>
>
>
[snip]

>We're not willing to use DNS for database updates.
>
Ouch ...

>On September 1, the
>databases will be restructured and the size of daily.cvd limited.
>
>
>
Well, at least there's that :)
I'm concerned about the fact that you only use one NS for cvd.clamav.net
though.
Is there a fallback scenario (automagically revert to HTTP GETs) if that
NS fails?

Regards,

Fajar



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tue, 24 Aug 2004 10:26:50 +0700
"Fajar A. Nugraha" <fajar@telkom.co.id> wrote:

> I'm concerned about the fact that you only use one NS for
> cvd.clamav.net though.

There will be at least 6 when the testing period is over.

> Is there a fallback scenario (automagically revert to HTTP GETs) if
> that NS fails?

Yes, it's already implemented.

--
oo ..... Tomasz Kojm <tkojm@clamav.net>
(\/)\......... http://www.ClamAV.net/gpg/tkojm.gpg
\..........._ 0DCA5A08407D5288279DB43454822DC8985A444B
//\ /\ Thu Aug 26 03:16:16 CEST 2004