Mailing List Archive

Idea for more timely virusdb updates
This thread on Trojan.JS.RunMe had me thinking: Hourly virus updates is
better than any of the commercial virus scanners, but obviously still has
issues, especially since a bunch of us obviously submitted updates that had
already been entered. I gather from these posts that the virusdb's actually
have some form of version number.

Suppose there was a DNS entry, say virusdb.clamav.net (or
version.virusdb.clamav.net, etc), that returned simply a text record with
the current DB version in it. Then, it would be possible to check the
version with a relatively cheap single UDP packet, rather than a full http
check, and people could check for DB updates more often than once an hour
without taxing the distribution system.

If nothing else, if this TXT record existing we could hack together some
shell script to check it and run freshclam as needed.

Just a thought.

==========================================================
Chris Candreva -- chris@westnet.com -- (914) 967-7816
WestNet Internet Services of Westchester
http://www.westnet.com/


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Mon, 9 Aug 2004, Christopher X. Candreva wrote:

> This thread on Trojan.JS.RunMe had me thinking: Hourly virus updates is
> better than any of the commercial virus scanners, but obviously still has
> issues, especially since a bunch of us obviously submitted updates that had
> already been entered. I gather from these posts that the virusdb's actually
> have some form of version number.
>
> Suppose there was a DNS entry, say virusdb.clamav.net (or
> version.virusdb.clamav.net, etc), that returned simply a text record with
> the current DB version in it. Then, it would be possible to check the
> version with a relatively cheap single UDP packet, rather than a full http
> check, and people could check for DB updates more often than once an hour
> without taxing the distribution system.

That's a very interesting idea, but I can imagine a few problems:
- we'd have to have a very short time-to-live or it would get stale
- the dns might know about the update before the mirrors all get it
- if everyone finds out about an update within 5 minutes of each other,
the mirrors might not handle the load

After seeing a Defcon talk on putting arbitrary data in DNS, though, I
wonder if we could put the daily updates (gpg signed) into DNS? That
would take a lot of load off the mirrors (occasional checks for main.cvd
updates are all that is required). And caching DNS servers would
distribute the load a bit.

Anyway, just another crazy idea for the developers to consider.

Damian Menscher
--
-=#| Physics Grad Student & SysAdmin @ U Illinois Urbana-Champaign |#=-
-=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc:(217)333-0038 |#=-
-=#| 4602 Beckman, VMIL/MS, Imaging Technology Group:(217)244-3074 |#=-
-=#| <menscher@uiuc.edu> www.uiuc.edu/~menscher/ Fax:(217)333-9819 |#=-
-=#| The above opinions are not necessarily those of my employers. |#=-


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Mon, 2004-08-09 at 16:55 -0400, Christopher X. Candreva wrote:
> This thread on Trojan.JS.RunMe had me thinking: Hourly virus updates is
> better than any of the commercial virus scanners, but obviously still has
> issues, especially since a bunch of us obviously submitted updates that had
> already been entered. I gather from these posts that the virusdb's actually
> have some form of version number.
>
> Suppose there was a DNS entry, say virusdb.clamav.net (or
> version.virusdb.clamav.net, etc), that returned simply a text record with
> the current DB version in it. Then, it would be possible to check the
> version with a relatively cheap single UDP packet, rather than a full http
> check, and people could check for DB updates more often than once an hour
> without taxing the distribution system.
>
> If nothing else, if this TXT record existing we could hack together some
> shell script to check it and run freshclam as needed.

Then all users would sworm to download the new sig, as soon as that
serial number incrimented, flooding the download server with update
requests.



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Mon, Aug 09, 2004 at 05:33:05PM -0400, Chris Meadors wrote:
> > Suppose there was a DNS entry, say virusdb.clamav.net (or
> > version.virusdb.clamav.net, etc), that returned simply a text record with
> > the current DB version in it. Then, it would be possible to check the
> > version with a relatively cheap single UDP packet, rather than a full http
> > check, and people could check for DB updates more often than once an hour
> > without taxing the distribution system.
>
> Then all users would sworm to download the new sig, as soon as that
> serial number incrimented, flooding the download server with update
> requests.

Only tracker.clamav.net (can be loadbalanced) should be able to handle
a fair number of connections, but daily.cvd.torrent is small enough
you could put it in a DNS TXT record :) (OK, DNS is far from secure,
so reliability will be at stake in that case... you might need to
cryptographically sign the file).

(1/2 :-)

--
#!perl -wpl # mmfppfmpmmpp mmpffm <pmmppfmfpppppfmmmf@fpffmm4mmmpmfpmf.ppppmf>
$p=3-2*/[^\W\dmpf_]/i;s.[a-z]{$p}.vec($f=join('',$p-1?chr(sub{$_[0]*9+$_[1]*3+
$_[2]}->(map{/p|f/i+/f/i}split//,$&)+97):qw(m p f)[map{((ord$&)%32-1)/$_%3}(9,
3,1)]),5,1)='`'lt$&;$f.eig; # Jan-Pieter Cornet


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: Idea for more timely virusdb updates [ In reply to ]
What about a deeper mirroring system? Perhaps one that supports
notification?

One of the things I like about BIND (not enough to use it, but still an
admired concept ;-) is the way zones can be distributed... notification
speeds things up if it works, polling creates a failsafe in which a missing
notify doesn't cause the world to end...

Hourly polls is a good thing - but if the system worked both ways, the
mirror could signal the end clients that it's time to download... those
notifies could be send only to clients that had registered to receive it (an
option in freshclam) and would not push the data, but trigger a freshclam
pull.

It could provide faster update response and smooth out the spikes in
download traffic, and could be used to maintain a larger set of mirrors...
without increasing polling frquency... a new "freshclam server" could allow
all larger users to easily run their own mirrors for internal
distribution...

Just a few ideas...

m/



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Damian Menscher wrote:

>On Mon, 9 Aug 2004, Christopher X. Candreva wrote:
>
>
>>Suppose there was a DNS entry, say virusdb.clamav.net (or
>>version.virusdb.clamav.net, etc), that returned simply a text record with
>>the current DB version in it.
>>

>After seeing a Defcon talk on putting arbitrary data in DNS, though, I
>wonder if we could put the daily updates (gpg signed) into DNS? That
>would take a lot of load off the mirrors (occasional checks for main.cvd
>updates are all that is required). And caching DNS servers would
>distribute the load a bit.
>
>
You know, this isn't so crazy after all. I put arbitrary data on my DNS
server so that exim
can get config data using dnsdb lookup. Its cheaper than mysql lookup
(Plus, you eliminate single point of failure),
and you can still update config from a central location instead of
updating each server config.

The only snag, is that TXT record is limited to a number of bytes ( I
tried putting 4096 bytes on it, it didn't work).
Now, the question is, can the daily (or hourly) updates fit in a single
TXT record?
If it must span multiple records than it will be somewhat complicated ....

Regards,

Fajar
--
http://justreadthis.com/


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Monday, August 09, 2004 11:18 PM [EST], Fajar A. Nugraha wrote:

>>
> You know, this isn't so crazy after all. I put arbitrary data on my
> DNS server so that exim
> can get config data using dnsdb lookup. Its cheaper than mysql
> lookup (Plus, you eliminate single point of failure),
> and you can still update config from a central location instead of
> updating each server config.
>
> The only snag, is that TXT record is limited to a number of bytes (
> I tried putting 4096 bytes on it, it didn't work).
> Now, the question is, can the daily (or hourly) updates fit in a
> single TXT record?
> If it must span multiple records than it will be somewhat
> complicated ....
>
> Regards,
>
> Fajar

I'd not recommend putting all the data in TXT records. TXT records
can be a max of 255 characters (anything more and you'll have problems
with other resolvers and such). But yeah, the version number in the
TXT records would be good, set the TTL to about 30-60 mins, and have
the freshclam client query and check the version.

I could assist with implementing the necessary server side scripts to
make the DNS management part really easy (I do something similar to
this all the time, rbldnsd makes stuff like this stupidly simple and
quick). Hell, I could even offer the DNSbl servers that the AHBL has
to host the zones if needbe.

--
Brian Bruns
The Summit Open Source Development Group
Open Solutions For A Closed World / Anti-Spam Resources
http://www.sosdg.org

The Abusive Hosts Blocking List
http://www.ahbl.org



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Mitch (WebCob) wrote:
> What about a deeper mirroring system? Perhaps one that supports
> notification?
>
> One of the things I like about BIND (not enough to use it, but still an
> admired concept ;-) is the way zones can be distributed... notification
> speeds things up if it works, polling creates a failsafe in which a missing
> notify doesn't cause the world to end...

right, but as discussed below, generally bind servers don't have 100k people
waiting for notifications and updates.

> Hourly polls is a good thing - but if the system worked both ways, the
> mirror could signal the end clients that it's time to download... those
> notifies could be send only to clients that had registered to receive it (an
> option in freshclam) and would not push the data, but trigger a freshclam
> pull.

with that option, the 'clients' would either have to remain connected the
entire time, which is completely not feasable, or somehow the database mirrors
would have to either 'remember' who to notify, or have some sort of registry
of people to notify (I can see how one might do this with a paid mirror
service), and then send out notifications (even a single UDP packet to 100k
servers could be quite bandwidth intensive. The architecture could work, yes,
but it doesn't scale well, and I don't think the clamav team has the resources
to do this sort of ass-kissing for free. They're already providing a
wonderful service to the internet community, we cannot bite the hand that
feeds us.

Another problem with this notification is there are still the spikes when the
notifications come out that EVERYONE AND THEIR BROTHER contacts the database
mirrors for updates. Your solution doesn't solve any problems imposed by
Christopher's idea, and actually introduces more.

In my opinion, the existing system is fine, and if you want better, you should
talk to the clamav folks about setting up some sort of 'priority' mirror, in
which you would pay a fee for having more enhanced services, like
notification, dns update polling, etc. And of course, proceeds (or at least a
major part of) would go to the clamav team for being the most kick ass
anti-virus product out there. I'm not sure how the official procedure would
be to roll something like this out, but now that I think of it, I may just go
about working on something like this. Gotta pay for my colocation somehow :)

Tomasz, et al.: Please expect to see an email from me by the end of the work
day tomorrow (or rather, today, but I haven't slept yet)

> It could provide faster update response and smooth out the spikes in
> download traffic, and could be used to maintain a larger set of mirrors...
> without increasing polling frquency... a new "freshclam server" could allow
> all larger users to easily run their own mirrors for internal
> distribution...

I would think that most 'larger users' (5+ node mail server cluster) would
already have an internal mirror. It's not difficult to do, and has been
discussed on this list, and in the clamav documentation many times.

> Just a few ideas...

hey, brainstorming is good, it's just the ideas aren't always ;)

-Jeremy

--
Jeremy Kitchen ++ Systems Administrator ++ Inter7 Internet Technologies, Inc.
jeremy@inter7.com ++ www.inter7.com ++ 866.528.3530 ++ 847.492.0470 int'l
kitchen @ #qmail #gentoo on EFnet ++ scriptkitchen.com/qmail



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tue, 10 Aug 2004, Fajar A. Nugraha wrote:

> The only snag, is that TXT record is limited to a number of bytes ( I tried
> putting 4096 bytes on it, it didn't work).
> Now, the question is, can the daily (or hourly) updates fit in a single TXT
> record?

I don't know that putting ALL of the records in DNS is necessary. The only
reason I was putting the version number there was to allow quick, more
frequent checks to see if you had the current version. It's possible to run
DNS with very short TTL times, even 0.

In terms of load on the servers: a smaller file would certainly help. I can
see the simplicity of just haveing two files to grab being attractive.
However, daily.cfg is now about 150k . I don't consider this big in the
scheme of things, but if we are talking about hundreds or thousands of
people trying to get the file, then the difference could be significant.

Suppose there was a numbered file for each version that would 'upgrade' you
from the previous version. IE, if I'm at 444, and current is 445, I grab
445.cvd. If I'm at 440, I grab 441.cvd, 442.cvd, through 445.cvd.

Downside -- obviously harder to maintain. Upside -- someone who is staying
constantly up to date is grabbing only a few bytes off the server at a time.

I think this is simpler than putting all the data into DNS.

I've also thought about rsync -- if putting the cvd files on an rsync server
would lighten the load at all.


==========================================================
Chris Candreva -- chris@westnet.com -- (914) 967-7816
WestNet Internet Services of Westchester
http://www.westnet.com/


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
At 16:03 10.08.2004, you wrote:

>...
>I've also thought about rsync -- if putting the cvd files on an rsync server
>would lighten the load at all.

Oh it would, rsync is quite effective. Another possibility might be to
patch the .cvd file(s)....

0.02

Erich


THINK
Püntenstrasse 39
8143 Stallikon
mailto:erich.titl@think.ch
PGP Fingerprint: BC9A 25BC 3954 3BC8 C024 8D8A B7D4 FF9D 05B8 0A16



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Erich Titl wanted us to know:

>>I've also thought about rsync -- if putting the cvd files on an rsync
>>server would lighten the load at all.
>Oh it would, rsync is quite effective. Another possibility might be to
>patch the .cvd file(s)....

Agree with rsync, depends how much changes in the file per download, but
I think it can be marvelously efficient.
Disagree with patching, binary files don't lend themselves to
diff/patch very well.

>0.02

Now we have 0.04 :-)

--
Regards... Todd
We should not be building surveillance technology into standards.
Law enforcement was not supposed to be easy. Where it is easy,
it's called a police state. -- Jeff Schiller on NANOG
Linux kernel 2.6.3-15mdkenterprise 2 users, load average: 0.09, 0.03, 0.01


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
Erich Titl wrote the following on 08/10/2004 05:12 PM :

> At 16:03 10.08.2004, you wrote:
>
>> ...
>> I've also thought about rsync -- if putting the cvd files on an rsync
>> server
>> would lighten the load at all.
>
>
> Oh it would, rsync is quite effective.


Not much with compressed files like *.cvd.

> Another possibility might be to patch the .cvd file(s)....
>

That was one proposition I made last year. But in practice it seems
there isn't really a pressing need now.

Lionel.

--
Lionel Bouton - inet6
---------------------------------------------------------------------
o Siege social: 51, rue de Verdun - 92158 Suresnes
/ _ __ _ Acces Bureaux: 33 rue Benoit Malon - 92150 Suresnes
/ /\ /_ / /_ France
\/ \/_ / /_/ Tel. +33 (0) 1 41 44 85 36
Inetsys S.A. Fax +33 (0) 1 46 97 20 10




-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: Idea for more timely virusdb updates [ In reply to ]
> right, but as discussed below, generally bind servers don't have
> 100k people
> waiting for notifications and updates.
>

Nope, true... but like I suggested, the notification tree doesn't have to be
flat...

One server notifying 100000 servers is time consuming and sure - costs a lot
of bandwidth...

Lets assume that each notify takes 5 seconds... we have to have SOMETHING to
"measure"...

1 server notifying 100000 servers takes 500000 seconds. That's a little over
a day to push the notification - bad idea ;-)

1 server notifying 100 servers, which each in turn notify 100 servers and so
on...
1 to 100: 100 seconds
each of them notifying 100: 100 seconds (total notified 10100)
each of them notifying 100: 100 seconds (total notified 1010100!) in 5
minutes!

That's 10 times your value of 100000 servers. Each server would only have to
know about 100 others. Not a huge database - wouldn't even have to be
written to file. Each server could be responsible for polling it's master
once per hour.

> > Hourly polls is a good thing - but if the system worked both ways, the
> > mirror could signal the end clients that it's time to download... those
> > notifies could be send only to clients that had registered to
> receive it (an
> > option in freshclam) and would not push the data, but trigger a
> freshclam
> > pull.
>
> with that option, the 'clients' would either have to remain connected the
> entire time, which is completely not feasable, or somehow the
> database mirrors
> would have to either 'remember' who to notify, or have some sort
> of registry
> of people to notify (I can see how one might do this with a paid mirror
> service), and then send out notifications (even a single UDP
> packet to 100k
> servers could be quite bandwidth intensive. The architecture
> could work, yes,
> but it doesn't scale well, and I don't think the clamav team has
> the resources
> to do this sort of ass-kissing for free. They're already providing a
> wonderful service to the internet community, we cannot bite the hand that
> feeds us.

I wasn't proposing that it had to be done for free (not that it can't be
with the factor tree I explained above). It might even reduce the cost of
database distribution.

If each server is only pushing 100 updates @ 200KB per update (2MB total) we
can get 500 pushes per month for only a couple dollars.

> Another problem with this notification is there are still the
> spikes when the
> notifications come out that EVERYONE AND THEIR BROTHER contacts
> the database
> mirrors for updates. Your solution doesn't solve any problems imposed by
> Christopher's idea, and actually introduces more.

100 servers for 200KB (20MB is hardly a spike.) and as for clients remaining
connected, that is what a server is - connected. This isn't for end users,
or local workstations. It's an OPTION for people who process a lot of data,
are at high risk, and need immediate response. Then their own internal
freshclam clients can poll their local authoritative server as often as they
want, or use the same procedure to distribute to them (if they are full time
connected that is).

> In my opinion, the existing system is fine, and if you want
> better, you should
> talk to the clamav folks about setting up some sort of 'priority'

Yeah, we could, but I don't think it needs that. And setting up an internal
mirror doesn't address the response time of the updates, unless I start
hammering the main freshclam every few minutes... and I just don't think
that would be friendly.

With the sort of hierarchical distribution I'm talking about, you could even
use an ranking system to automatically organize the distribtion (while I'm
on a roll ;-)...

What I mean is that everyone would contact one of the "root" mirrors
initially. In the request to be notified, it would indicate the number of
clients it serves. If less than a certain number, then it could be referred
to a child of the root server. If that child becomes unavailable it could
contact the root again (at the next hourly polling time). How many servers
are there on the Internet? We could probably handle the whole lot of them
with no more than 4 or 5 levels. Push an update to the world in under 10
minutes. Think how many virus laden emails this could stop.

(visions of f5...) in fact, the root server could hand out the IP's of all
child servers not fully loaded. The client could register with the nearest
(by route time) one -

just ranting...

m/



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Aug 10, 2004, at 5:57 AM, Jeremy Kitchen wrote:

> Mitch (WebCob) wrote:
>> Just a few ideas...
>
> hey, brainstorming is good, it's just the ideas aren't always ;)

Another stupid idea...how about a mechanism where clam can have updates
"pushed" to it, so servers controlled by the clam team can distribute
mini updates to them. The admins would have to subscribe to it, like a
listserv, only instead of through email, it's done through this
theoretical mechanism. There wouldn't be traffic spikes (as big) for
times where there *aren't* updated db's available, only when there are
updates, and the updates are sent out as the clam servers are able to
handle the load.

Maybe like a modified GPG-signed listserv system only on it's own "clam
update daemon" port...take a little more configuration since the people
installing clam would have to subscribe and install a GPG key or
something like that in the process, but that shouldn't be something
back-breaking to figure out.

Maintenance would have to be done for the subscription mechanism, etc.,
like a listserv would, but it may be something that could be done. May
even be extendable so that a master server for a network could receive
the updates from the clam site (pushed from clamserv) then in turn be
told to push them out to machines on the internal network. (I know
this could already be set up, but it may be easier through this type of
model to set up and maintain...)

I'm probably overlooking something obvious, but again...just an idea,
right? :-)

-Bart



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tue, 10 Aug 2004, Bart Silverstrim wrote:
>
> Maybe like a modified GPG-signed listserv system only on it's own "clam
> update daemon" port...take a little more configuration since the people
> installing clam would have to subscribe and install a GPG key or
> something like that in the process, but that shouldn't be something
> back-breaking to figure out.

Ok, this is turning into a scary beast. But we already have several
mailing lists (clamav-users, for example) which can obviously handle a
bit of a load. Might be interesting to concoct a specially-formatted
message that the milter (or clamd itself) could recognize as a database
update, and automatically append to its list of signatures.

I'd imagine a format something like:

---gpg-cleartext-signed-message---
BEGIN clamd update 24.449
Worm.bagle.zz AAAABBBBCCCDDDDEEFEFKL..........
Worm.SkyNet.zz 111112222333344455666677........
...
END
---gpg-signature---
JDSLJGIREJIOJDGLSJLGHSLKJGLKSDJLKGJSLKJGIEJ*Y*G($Y*HHIO4k245j2jk
kdjaflkjkh325hjk35h2jkhkjhjkfdhjh42jkh345jk2h35jk2hkjhjkfhjskh32
fhjkhafdjhajk53h2jk5h3j2kh35jkhfay983489527938572035230398udfsfs
---end-signature---

When scanning stuff like this, clamd could automagically decode the gpg
signature and test that it is valid. If so, it looks at the sequence
number (24.449 in this case). If that's the next one in the series, it
appends the rules to its database. If not, it assumes it lost a message
somewhere and contacts a mirror via HTTP to get main 24 and daily 449.

Doing something like this would push a lot of the distribution load onto
sourceforge (which seems to get messages out to this list in about 1/2
hour). The gpg-signature prevents spoofing. And the sequence numbers
keep everyone current. The major problems I see are getting clamd to
recognize a message targeted for it, and the obvious problems of DoS
attacks (someone sending spoofed messages that would suck CPU time
decoding the gpg signature).

Anyway, just another wild-n-crazy idea to throw out there. I'm guessing
we're better off with the current method for now, but this might be an
interesting possibility for the future.

[.I haven't given up on DNS updates yet, but it's hard to come up with a
clean way to distribute >256 bytes of data that way, which means even
single rules don't always fit.]

Damian Menscher
--
-=#| Physics Grad Student & SysAdmin @ U Illinois Urbana-Champaign |#=-
-=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc:(217)333-0038 |#=-
-=#| 4602 Beckman, VMIL/MS, Imaging Technology Group:(217)244-3074 |#=-
-=#| <menscher@uiuc.edu> www.uiuc.edu/~menscher/ Fax:(217)333-9819 |#=-
-=#| The above opinions are not necessarily those of my employers. |#=-


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tue, 10 Aug 2004, Lionel Bouton wrote:

> > Another possibility might be to patch the .cvd file(s)....
> >
>
> That was one proposition I made last year. But in practice it seems there
> isn't really a pressing need now.

If people can't check for database updates more often than once an hour,
then there is a pressing need.

The mirror page talkes about the need for mirrors, about exponential growth,
and how at least a 10mbit pipe is needed to host a mirror. It puts March
2004 traffic at about 120gig/month

Some quick calculations:
daily.cvd is about 150k compressed, 334k uncompressed -- let's say 50%.
Greping the virses added for updated 447 gave me about 3k uncompressed -- so
let's say 2k compressed, on the outside.

For 2k of update, everyone downloaded 150k. That shows (at least for that
update) only 1.3% of what was downloaded was needed.

If only 1.3% of every update is actually needed, and people only downloaded
what they needed, the traffic on the mirrors would drop from 120gig/month to
1.6 gig/month.

If I am completely off by a factor of 10 -- say only 10% of every update
is actually needed, traffic on the mirrors drops from 120gig to 12gig.

There are a lot of assumptions here, but I would think ever reducing the
load on the virus servers by half would be significant.

==========================================================
Chris Candreva -- chris@westnet.com -- (914) 967-7816
WestNet Internet Services of Westchester
http://www.westnet.com/


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tuesday 10 August 2004 12:23 pm, Damian Menscher wrote:
> Ok, this is turning into a scary beast. But we already have several
> mailing lists (clamav-users, for example) which can obviously handle a
> bit of a load. Might be interesting to concoct a specially-formatted
> message that the milter (or clamd itself) could recognize as a database
> update, and automatically append to its list of signatures.

this is actually a pretty decent idea. I think it would be best to, rather
than have clamd try to detect it, have a special address on the machine that
processes the message via a program. Most MTAs I'm aware of (at least on the
unix side) can do this, I know qmail can for sure.

> I'd imagine a format something like:
[snip email message for the update]

> Doing something like this would push a lot of the distribution load onto
> sourceforge (which seems to get messages out to this list in about 1/2
> hour).

for something like this I wouldn't use sourceforge's mail servers :P They're
already bogged down as it is, us adding load to them like this would be bad,
and the notifications would eventually get slower, and slower, and slower...
having a dedicated list server for this purpose would be the best.


> The gpg-signature prevents spoofing. And the sequence numbers
> keep everyone current. The major problems I see are getting clamd to
> recognize a message targeted for it, and the obvious problems of DoS
> attacks (someone sending spoofed messages that would suck CPU time
> decoding the gpg signature).

yes, that's an unfortunate problem with this idea, however, if you used, as I
stated, a special address that uses program delivery, you'd have to hack the
listserver to get everyone's 'subscription' address to be able to do this.

> Anyway, just another wild-n-crazy idea to throw out there. I'm guessing
> we're better off with the current method for now, but this might be an
> interesting possibility for the future.

it definitely is interesting.

> [.I haven't given up on DNS updates yet, but it's hard to come up with a
> clean way to distribute >256 bytes of data that way, which means even
> single rules don't always fit.]

I wouldn't distribute the rule in DNS, however, a timestamp of sorts in dns
isn't a bad idea.

-Jeremy

--
Jeremy Kitchen ++ Systems Administrator ++ Inter7 Internet Technologies, Inc.
jeremy@inter7.com ++ www.inter7.com ++ 866.528.3530 ++ 847.492.0470 int'l
kitchen @ #qmail #gentoo on EFnet ++ scriptkitchen.com/qmail



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tuesday 10 August 2004 04:57 am, Jeremy Kitchen wrote:
> Tomasz, et al.: Please expect to see an email from me by the end of the
> work day tomorrow (or rather, today, but I haven't slept yet)

sigh, and after saying that I now have tons of work to do so I won't be able
to get this email to you guys until later.

I will send it though :)

-Jeremy

--
Jeremy Kitchen ++ Systems Administrator ++ Inter7 Internet Technologies, Inc.
jeremy@inter7.com ++ www.inter7.com ++ 866.528.3530 ++ 847.492.0470 int'l
kitchen @ #qmail #gentoo on EFnet ++ scriptkitchen.com/qmail



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tue, 10 Aug 2004, Jeremy Kitchen wrote:
> On Tuesday 10 August 2004 12:23 pm, Damian Menscher wrote:
> > Ok, this is turning into a scary beast. But we already have several
> > mailing lists (clamav-users, for example) which can obviously handle a
> > bit of a load. Might be interesting to concoct a specially-formatted
> > message that the milter (or clamd itself) could recognize as a database
> > update, and automatically append to its list of signatures.
>
> this is actually a pretty decent idea. I think it would be best to, rather
> than have clamd try to detect it, have a special address on the machine that
> processes the message via a program. Most MTAs I'm aware of (at least on the
> unix side) can do this, I know qmail can for sure.

Good idea. Taking it out of the milter allows for qmail/exim/postfix
compatibility, and sending to a dedicated address saves the effort of
processing every message (though presumably you're doing that anyway).

With sendmail, you could add to /etc/aliases something like:
clamav-updates | sigtool --add

Before people get too excited about this idea, though, there are some
issues that need to be fixed.

Anyone know if it's really feasible for us to obtain a mailserver that
can send out 2k emails to all (100,000?) users in a short (5-10 mins)
time? Assuming those numbers are reasonable, that means 200 meg of
data. Combined with SMTP overhead, it seems like it would be
troublesome. Additionally, there are potential bandwidth issues if you
consider we'd need to do that several times/day.

Updating the "main" database is one concern. Sending out a 2-meg email
to everyone seems like it might be too much load, but sending out the 1K
email telling everyone to get it means the mirrors will get swamped. I
can't think of a way around this, but hopefully someone else can?

Also, this doesn't give much provision for removing "bad" signatures
(that cause false positives) since it really just appends rules. We'd
need to figure out a way to delete signatures also. I could imagine
doing this by including a "null" signature, or using some other flag.

Finally, there's the whole issue of multiplying your points of failure.
If your current database is screwed, appending more to it will leave it
screwed. And if you add stuff to it a few times a day, chances are it
will get screwed up at some point. At least this issue has a simple
fix: include an MD5 sum with the update which must match your MD5 sum
after applying the update. If they don't match, you know something went
wrong, either with this update or a previous one. (This has the danger
that if the developers send an email with an incorrect MD5 hash,
everyone will thrash the mirrors.)

Note to the developers: please don't feel like you have to code up any
of our random ideas. I'm just having fun brainstorming about how to
optimize this process. I expect in another few days of discussion we'll
have converged on a fairly sane idea.

Damian Menscher
--
-=#| Physics Grad Student & SysAdmin @ U Illinois Urbana-Champaign |#=-
-=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc:(217)333-0038 |#=-
-=#| 4602 Beckman, VMIL/MS, Imaging Technology Group:(217)244-3074 |#=-
-=#| <menscher@uiuc.edu> www.uiuc.edu/~menscher/ Fax:(217)333-9819 |#=-
-=#| The above opinions are not necessarily those of my employers. |#=-


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tue, 10 Aug 2004, Damian Menscher wrote:
> Anyone know if it's really feasible for us to obtain a mailserver that
> can send out 2k emails to all (100,000?) users in a short (5-10 mins)
> time?

I haven't been following the whole discussion, but I thought this was
mostly to provide support to "power users". I think the average
small-time admin would be happy with the hourly updates.

Jeffrey Moskot
System Administrator
jef@math.miami.edu


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On 2004-08-10 14:41:28 -0500, Damian Menscher wrote:
> On Tue, 10 Aug 2004, Jeremy Kitchen wrote:
> > On Tuesday 10 August 2004 12:23 pm, Damian Menscher wrote:
> > > Ok, this is turning into a scary beast. But we already have several
> > > mailing lists (clamav-users, for example) which can obviously handle a
> > > bit of a load. Might be interesting to concoct a specially-formatted
> > > message that the milter (or clamd itself) could recognize as a database
> > > update, and automatically append to its list of signatures.
[...]
> Before people get too excited about this idea, though, there are some
> issues that need to be fixed.
>
> Anyone know if it's really feasible for us to obtain a mailserver that
> can send out 2k emails to all (100,000?) users in a short (5-10 mins)
> time?

How about using NNTP instead of SMTP? Then the clamav server doesn't
have to push out those messages to everybody but only to its neighbours
which will distribute it further.

hp


--
_ | Peter J. Holzer | Je höher der Norden, desto weniger wird
|_|_) | Sysadmin WSR | überhaupt gesprochen, also auch kein Dialekt.
| | | hjp@hjp.at | Hallig Gröde ist fast gänzlich dialektfrei.
__/ | http://www.hjp.at/ | -- Hannes Petersen in desd
Re: Idea for more timely virusdb updates [ In reply to ]
Christopher X. Candreva wrote:

>This thread on Trojan.JS.RunMe had me thinking: Hourly virus updates is
>better than any of the commercial virus scanners, but obviously still has
>issues, especially since a bunch of us obviously submitted updates that had
>already been entered. I gather from these posts that the virusdb's actually
>have some form of version number.
>
>
>
This could actualy be easily accomplished also by attaching a soa record
to a zone ... for example

dbversion.clamav.net

Incrementing the serial for that should be trivial enough.....writing a
mechanism to rapidly query against it and then to invoke a freshclam is
left as an exercise to the reader.

Presumably then the lists of Nameservers for that particular zone would
be expanded to about 10 or more. Notification from whatever master zone
server could be trivialy accomplished on that.

We should probably consider that the load balancing of all those end
users/isp's DNS resolvers may not be all it can be, particularly the
selection of which nameserver to talk to out of many for a particular zone.

Anyways I did a dig .... Arent CNAMEs that Point to CNAMEs contrary to RFC?
Might that be behind the infrequent dns resolution complaints?
Also... Is there any single name that covers ALL mirrors?
Also.... Any insight as to how the { presumably dynamic } selection to
alias the db-local to db.america is done?

c:\Documents and Settings\joe.JOE.000>dig database.clamav.net

; <<>> DiG 9.2.3rc3 <<>> database.clamav.net
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41
;; flags: qr rd ra; QUERY: 1, ANSWER: 15, AUTHORITY: 5, ADDITIONAL: 2

;; QUESTION SECTION:
;database.clamav.net. IN A

;; ANSWER SECTION:
database.clamav.net. 5 IN CNAME db.local.clamav.net.
db.local.clamav.net. 7200 IN CNAME db.america.clamav.net.
db.america.clamav.net. 5 IN A 128.121.60.235
db.america.clamav.net. 5 IN A 196.40.71.226
db.america.clamav.net. 5 IN A 199.239.233.95
db.america.clamav.net. 5 IN A 200.68.106.39
db.america.clamav.net. 5 IN A 24.244.193.21
db.america.clamav.net. 5 IN A 38.136.139.7
db.america.clamav.net. 5 IN A 64.18.103.6
db.america.clamav.net. 5 IN A 64.69.64.158
db.america.clamav.net. 5 IN A 65.75.154.69
db.america.clamav.net. 5 IN A 65.77.42.207
db.america.clamav.net. 5 IN A 66.139.75.171
db.america.clamav.net. 5 IN A 67.18.205.218
db.america.clamav.net. 5 IN A 69.93.108.98

;; AUTHORITY SECTION:
clamav.net. 7200 IN NS ns5.clamav.net.
clamav.net. 7200 IN NS ns1.oltrelinux.com.
clamav.net. 7200 IN NS ns2.clamav.net.
clamav.net. 7200 IN NS ns3.clamav.net.
clamav.net. 7200 IN NS ns4.clamav.net.

;; ADDITIONAL SECTION:
ns1.oltrelinux.com. 38516 IN A 194.242.226.43
ns5.clamav.net. 153717 IN A 80.69.66.9

;; Query time: 671 msec
;; SERVER: 64.95.32.37#53(64.95.32.37)
;; WHEN: Tue Aug 10 16:40:04 2004
;; MSG SIZE rcvd: 429




-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tue, 2004-08-10 at 12:40, Christopher X. Candreva wrote:

> If people can't check for database updates more often than once an hour,
> then there is a pressing need.
[...]
> If only 1.3% of every update is actually needed, and people only downloaded
> what they needed, the traffic on the mirrors would drop from 120gig/month to
> 1.6 gig/month.
>
> If I am completely off by a factor of 10 -- say only 10% of every update
> is actually needed, traffic on the mirrors drops from 120gig to 12gig.

That's one of the things that seems to be driving the size of daily.cvd
up - updating main.cvd entails a massive distribution of files to the
world.

Perhaps a tiered approach to the update files, with main.cvd,
monthly.cvd, weekly.cvd, daily.cvd, and hot.cvd

The advantage there is that the really big update could be distributed
very seldom - perhaps only with new code (the code generally has to be
upgraded every few months to deal with a new threat anyway).

If you had overlapping signatures between the files, you could add a
fuzzy-factor into freshclam that it might not bring down the latest
weekly/monthly if the other files overlap completely. That would
distribute the load on the freshclam servers for the larger updates, and
there would just be the very small daily.cvd (and perhaps hot.cvd)
downloads.

I like the idea of using DNS to signal the change - maybe just for
hot.cvd. so, whenever a major virus breakout occurs, the new sig would
be added to hot.cvd and the DNS TXT record changed. 10,000 users
pulling down a 2-3K file is not terribly hard for a server with decent
bandwidth
--
Daniel J McDonald, CCIE 2495, CNX
Austin Energy




-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tue, Aug 10, 2004 at 10:39:19PM +0200, Peter J. Holzer wrote:
> On 2004-08-10 14:41:28 -0500, Damian Menscher wrote:
[... about sending clamav updates quickly to all subscribers]
> > Anyone know if it's really feasible for us to obtain a mailserver that
> > can send out 2k emails to all (100,000?) users in a short (5-10 mins)
> > time?
>
> How about using NNTP instead of SMTP? Then the clamav server doesn't

Why use such an old protocol that isn't suited to binary transfers.

I've already mentioned this jokingly, but I was half serious: I think
setting up a bittorrent would solve a lot of the bandwidth problems.

You would need some place to get the daily.cvd.torrent file, which seems
to be about 170 bytes when I tried creating one yesterday (Small enough
to fit base64-encoded in a DNS TXT record, if you insist, but I doubt
that that is prudent to rely upon). Then you'd need a decent tracker,
or a bunch of trackers, and at least one seeder per tracker. I guess
that the current db.*.clamav.net hosts can easily host both a tracker
and a seeder.

If you then distribute a downloading clients that keeps seeding for just
1 hour (or until a preset share ratio was reached, say, 10x), you would
very quickly take a HUGE load off the download servers... and everyone
using clamav would automatically help the project by donating bandwidth
for the updates.

P2P - it's not just for downloading pirated Metallica mp3s.

HTH,

--
#!perl -wpl # mmfppfmpmmpp mmpffm <pmmppfmfpppppfmmmf@fpffmm4mmmpmfpmf.ppppmf>
$p=3-2*/[^\W\dmpf_]/i;s.[a-z]{$p}.vec($f=join('',$p-1?chr(sub{$_[0]*9+$_[1]*3+
$_[2]}->(map{/p|f/i+/f/i}split//,$&)+97):qw(m p f)[map{((ord$&)%32-1)/$_%3}(9,
3,1)]),5,1)='`'lt$&;$f.eig; # Jan-Pieter Cornet


-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: Idea for more timely virusdb updates [ In reply to ]
On Tuesday 10 August 2004 02:41 pm, Damian Menscher wrote:
[.snip: using a program delivery to process update mailing list mails]
> With sendmail, you could add to /etc/aliases something like:
> clamav-updates | sigtool --add

that's the ticket.

> Anyone know if it's really feasible for us to obtain a mailserver that
> can send out 2k emails to all (100,000?) users in a short (5-10 mins)
> time? Assuming those numbers are reasonable, that means 200 meg of
> data. Combined with SMTP overhead, it seems like it would be
> troublesome. Additionally, there are potential bandwidth issues if you
> consider we'd need to do that several times/day.

well, I would think this would be an 'optional' thing you could do, or maybe
part of a 'premium' service provided for a fee. As Jef mentioned, most small
time folks are perfectly happy with hourly updates in a pull configuration.

> Updating the "main" database is one concern. Sending out a 2-meg email
> to everyone seems like it might be too much load, but sending out the 1K
> email telling everyone to get it means the mirrors will get swamped. I
> can't think of a way around this, but hopefully someone else can?

well, I would hope that while also grabbing these daily.cvd updates via email,
that the admin is also running freshclam (perhaps less frequently now that
he/she only needs to check main.cvd once a day) to grab the main.cvd and
doesn't need notification for it. Forgive my ignorance if I'm not
interpreting the role of the main/daily.cvd files correctly:
main.cvd: updated daily with all of the updates done to daily.cvd throughout
the day
daily.cvd: 0sec updates to the database, get rolled into main.cvd nightly

> Also, this doesn't give much provision for removing "bad" signatures
> (that cause false positives) since it really just appends rules. We'd
> need to figure out a way to delete signatures also. I could imagine
> doing this by including a "null" signature, or using some other flag.

true. perhaps the first line of the email could be a command, and a simple
sh/perl/c program could parse it and then call the proper commands to add or
remove the signature that follows.

> Finally, there's the whole issue of multiplying your points of failure.
> If your current database is screwed, appending more to it will leave it
> screwed. And if you add stuff to it a few times a day, chances are it
> will get screwed up at some point. At least this issue has a simple
> fix: include an MD5 sum with the update which must match your MD5 sum
> after applying the update. If they don't match, you know something went
> wrong, either with this update or a previous one. (This has the danger
> that if the developers send an email with an incorrect MD5 hash,
> everyone will thrash the mirrors.)

eek.

> Note to the developers: please don't feel like you have to code up any
> of our random ideas. I'm just having fun brainstorming about how to
> optimize this process. I expect in another few days of discussion we'll
> have converged on a fairly sane idea.

or scrap the whole idea all together :)

-Jeremy

--
Jeremy Kitchen ++ Systems Administrator ++ Inter7 Internet Technologies, Inc.
jeremy@inter7.com ++ www.inter7.com ++ 866.528.3530 ++ 847.492.0470 int'l
kitchen @ #qmail #gentoo on EFnet ++ scriptkitchen.com/qmail



-------------------------------------------------------
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
_______________________________________________
Clamav-users mailing list
Clamav-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/clamav-users

1 2 3  View All