Mailing List Archive

Scan for dummy file with /dev/zero takes longer
Hello,

I'm trying to get some stats on how long a scan takes by different
size, but I encountered an unexpected behavior when scanning a file
generated in a specific way.
A scan for a dummy file filled with /dev/zero takes much longer than
with /dev/urandom. I think the processing time should be the same or
less.
I'd like to know how to avoid this problem because that may cause stuck service.
I'm using ClamAV version 0.101.2/25504.

You can reproduce the problem by doing the following.

A 10MB file with /dev/zero: Takes almost 7 times as long as /dev/urandom.
```
$ dd if=/dev/urandom of=dummy-dd-10MB-with-urandom.iso bs=10MB count=1
$ dd if=/dev/zero of=dummy-dd-10MB-with-zero.iso bs=10MB count=1
$ ls -ltr dummy-dd-10MB-with-*
-rw-r--r-- 1 user user 10000000 Jul 9 03:41 dummy-dd-10MB-with-zero.iso
-rw-r--r-- 1 user user 10000000 Jul 9 03:41 dummy-dd-10MB-with-urandom.iso

$ time (echo "SCAN dummy-dd-10MB-with-zero.iso" | nc -U
/var/run/clamd.scan/clamd.sock)
dummy-dd-10MB-with-zero.iso: OK
real 0m4.056s
user 0m0.008s
sys 0m0.004s

$ time (echo "SCAN dummy-dd-10MB-with-urandom.iso" | nc -U
/var/run/clamd.scan/clamd.sock)
dummy-dd-10MB-with-urandom.iso: OK
real 0m0.569s
user 0m0.012s
sys 0m0.000s
```


A 250MB file with /dev/zero: Takes almost 8 times as long as /dev/urandom.
```
$ dd if=/dev/zero of=dummy-dd-250MB-with-zero.iso bs=25MB count=10
$ dd if=/dev/urandom of=dummy-dd-250MB-with-urandom.iso bs=25MB count=10
$ ls -ltr dummy-dd-250MB-with-*
-rw-r--r-- 1 user user 250000000 Jul 9 03:44 dummy-dd-250MB-with-urandom.iso
-rw-r--r-- 1 user user 250000000 Jul 9 03:44 dummy-dd-250MB-with-zero.iso

$ time (echo "SCAN dummy-dd-250MB-with-zero.iso" | nc -U
/var/run/clamd.scan/clamd.sock)
dummy-dd-250MB-with-zero.iso: OK
real 1m42.949s
user 0m0.009s
sys 0m0.003s

$time (echo "SCAN dummy-dd-250MB-with-urandom.iso" | nc -U
/var/run/clamd.scan/clamd.sock)
dummy-dd-250MB-with-urandom.iso: OK
real 0m12.905s
user 0m0.004s
sys 0m0.007s
```

Thanks.

--
Taizo Ito

_______________________________________________

clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users


Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml
Re: Scan for dummy file with /dev/zero takes longer [ In reply to ]
Taizo,

The reason for the difference is that there are a lot of subsignatures used
in the published set of logical signatures that begin with some number of
zeroes (more so than the majority of random byte sequences), so the ClamAV
pattern matcher has to do a lot more work on the all-zeroes file
determining that none of the signatures fully match. Also, there are
likely some short all-zero subsigs that get used based on how certain
subsignature features are implemented, and these can also affect
performance on large files consisting mainly of zeroes.

Hope that helps! Thanks for asking about this - your observation is a good
reminder to us that a large all-zero file makes a good test case for
catching signatures that might have egregious performance impacts. :)

-Andrew

Andrew Williams
Malware Research Team
Cisco Talos

On Tue, Jul 9, 2019 at 11:07 PM Taizo ITO <taizo.ito@hennge.com> wrote:

> Hello,
>
> I'm trying to get some stats on how long a scan takes by different
> size, but I encountered an unexpected behavior when scanning a file
> generated in a specific way.
> A scan for a dummy file filled with /dev/zero takes much longer than
> with /dev/urandom. I think the processing time should be the same or
> less.
> I'd like to know how to avoid this problem because that may cause stuck
> service.
> I'm using ClamAV version 0.101.2/25504.
>
> You can reproduce the problem by doing the following.
>
> A 10MB file with /dev/zero: Takes almost 7 times as long as /dev/urandom.
> ```
> $ dd if=/dev/urandom of=dummy-dd-10MB-with-urandom.iso bs=10MB count=1
> $ dd if=/dev/zero of=dummy-dd-10MB-with-zero.iso bs=10MB count=1
> $ ls -ltr dummy-dd-10MB-with-*
> -rw-r--r-- 1 user user 10000000 Jul 9 03:41 dummy-dd-10MB-with-zero.iso
> -rw-r--r-- 1 user user 10000000 Jul 9 03:41 dummy-dd-10MB-with-urandom.iso
>
> $ time (echo "SCAN dummy-dd-10MB-with-zero.iso" | nc -U
> /var/run/clamd.scan/clamd.sock)
> dummy-dd-10MB-with-zero.iso: OK
> real 0m4.056s
> user 0m0.008s
> sys 0m0.004s
>
> $ time (echo "SCAN dummy-dd-10MB-with-urandom.iso" | nc -U
> /var/run/clamd.scan/clamd.sock)
> dummy-dd-10MB-with-urandom.iso: OK
> real 0m0.569s
> user 0m0.012s
> sys 0m0.000s
> ```
>
>
> A 250MB file with /dev/zero: Takes almost 8 times as long as /dev/urandom.
> ```
> $ dd if=/dev/zero of=dummy-dd-250MB-with-zero.iso bs=25MB count=10
> $ dd if=/dev/urandom of=dummy-dd-250MB-with-urandom.iso bs=25MB count=10
> $ ls -ltr dummy-dd-250MB-with-*
> -rw-r--r-- 1 user user 250000000 Jul 9 03:44
> dummy-dd-250MB-with-urandom.iso
> -rw-r--r-- 1 user user 250000000 Jul 9 03:44 dummy-dd-250MB-with-zero.iso
>
> $ time (echo "SCAN dummy-dd-250MB-with-zero.iso" | nc -U
> /var/run/clamd.scan/clamd.sock)
> dummy-dd-250MB-with-zero.iso: OK
> real 1m42.949s
> user 0m0.009s
> sys 0m0.003s
>
> $time (echo "SCAN dummy-dd-250MB-with-urandom.iso" | nc -U
> /var/run/clamd.scan/clamd.sock)
> dummy-dd-250MB-with-urandom.iso: OK
> real 0m12.905s
> user 0m0.004s
> sys 0m0.007s
> ```
>
> Thanks.
>
> --
> Taizo Ito
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>