Mailing List Archive

Hi Matt

I have seen several issues with FAS270s and high cpu utilisation related to several bugs in pre-7.2.4 releases, especially with the filer sat in background processes, cpu spikes and nfs perf issues. Troubleshooting these bugs is a pain, and the solution is to upgrade anyway, so I would be inclined to upgrade to 7.2.6 first and see if the issue still persists.

cheers
Kenneth

> Subject: High CPU utilisation
> Date: Thu, 6 Nov 2008 09:09:41 +0000
> From: MDAVIES@generalatlantic.com
> To: toasters@mathworks.com
>
> Hi all,
>
> We are having a problem with a FAS270 that appears to be having high cpu utilisation issues.
>
> To be honest I have looked at sysstat and statit and I cannot see the problem. I would appreciate if someone could take a look at the below outputs and let me know what I'm missing.
>
> Thanks
>
> Matt

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE!
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Nils,

1, yes it is both a source and destination for snapmirror, there is no
snapvault

2, see below.

ukfiler*> wafl scan status
Volume CIFS_UK:
Scan id Type of scan progress
3 active bitmap rearrangement fbn 1605 of 2230 w/
max_chain_len 3
191730 container block reclamation block 2042 of 2230
Volume root:
Scan id Type of scan progress
5 active bitmap rearrangement fbn 63 of 107 w/
max_chain_len 3
191636 container block reclamation block 64 of 107
Volume UKEXCHDB:
Scan id Type of scan progress
8 active bitmap rearrangement fbn 138 of 713 w/
max_chain_len 3
Volume UKEXCHLOGS:
Scan id Type of scan progress
10 active bitmap rearrangement fbn 369 of 401 w/
max_chain_len 3
191608 container block reclamation block 391 of 401
Volume UKVMWARE:
Scan id Type of scan progress
12 active bitmap rearrangement fbn 1538 of 4461 w/
max_chain_len 3
Volume DUEXCHLOGS:
Scan id Type of scan progress
191713 container block reclamation block 879 of 892
Volume UKVIRTUALCENTRE:
Scan id Type of scan progress
16 active bitmap rearrangement fbn 197 of 1115 w/
max_chain_len 3
Volume UKVMWARE1:
Scan id Type of scan progress
18 active bitmap rearrangement fbn 4215 of 4461 w/
max_chain_len 3
Volume UKVMWARENFS:
Scan id Type of scan progress
21 active bitmap rearrangement fbn 1019 of 1784 w/
max_chain_len 3
Volume TESTUKEXCHLOGS:
Scan id Type of scan progress
191708 container block reclamation block 393 of 401
Volume UKEXCH3DB:
Scan id Type of scan progress
152073 active bitmap rearrangement fbn 548 of 571 w/
max_chain_len 3
191729 container block reclamation block 356 of 571
Volume vol1:
Scan id Type of scan progress
191630 container block reclamation block 0 of 0
Volume UKEXCH3LOGS:
Scan id Type of scan progress
152075 active bitmap rearrangement fbn 29 of 321 w/
max_chain_len 3
191728 container block reclamation block 195 of 321

Thanks

Matt

-----Original Message-----
From: Nils Vogels [mailto:bacardicoke@gmail.com]
Sent: 06 November 2008 09:29
To: Davies,Matt
Subject: Re: High CPU utilisation

Couple of questions ..

1) Do you have snapvault/snapmirror running?
2) What does "wafl scan status" tell you?

Greetings,

Nils

On Thu, Nov 6, 2008 at 10:09 AM, Davies,Matt
<MDAVIES@generalatlantic.com> wrote:
> Hi all,
>
> We are having a problem with a FAS270 that appears to be having high
cpu utilisation issues.
>
> To be honest I have looked at sysstat and statit and I cannot see the
problem. I would appreciate if someone could take a look at the below
outputs and let me know what I'm missing.
>
> Thanks
>
> Matt
>
> ukfiler*> sysstat -i 3
> CPU NFS CIFS iSCSI Net kB/s Disk kB/s iSCSI kB/s
Cache
> in out read write in out
age
> 85% 18 0 26 203 13 661 0 91 0
24
> 68% 9 1 50 283 23 692 0 221 9
24
> 92% 7 4 25 144 11 1643 2443 90 3
24
> 78% 12 1 74 345 21 618 0 198 0
24
> 67% 8 0 45 288 18 722 0 205 5
24
> 96% 13 1 32 236 14 1365 1708 151 1
24
> 81% 8 4 39 222 16 843 867 167 4
24
> 85% 8 0 33 443 10 1212 1456 118 0
24
> 99% 16 2 43 370 21 912 0 168 4
24
> 99% 14 16 23 219 26 1023 0 98 0
24
> 99% 12 0 36 585 39 2452 3781 465 18
24
> ukfiler*>
>
> Hostname: ukfiler ID: 0084264545 Memory: 1022 MB
> NetApp Release 7.2.3: Thu Jul 5 10:06:16 PDT 2007
> <6O>
> Start time: Wed Nov 5 22:45:03 GMT 2008
>
> CPU Statistics
> 64.822501 time (seconds) 100 %
> 52.221602 system time 81 %
> 0.511694 rupt time 1 % (161581 rupts x 3
usec/rupt)
> 51.709908 non-rupt system time 80 %
> 77.423400 idle time 119 %
>
> 4.011144 time in CP 6 % 100 %
> 0.052061 rupt time in CP 1 % (12144 rupts x 4
usec/rupt)
>
> Multiprocessor Statistics (per second)
> cpu0 cpu1 total
> sk switches 6581.74 7585.82 14167.57
> hard switches 1485.87 1560.68 3046.55
> domain switches 72.75 65.01 137.76
> CP rupts 125.45 61.89 187.34
> nonCP rupts 1350.03 955.30 2305.33
> IPI rupts 18.23 17.32 35.56
>
> CP rupt usec 565.04 238.10 803.13
> nonCP rupt usec 4342.69 2747.95 7090.64
> idle 632999.86 561391.13 1194390.83
> kahuna 334624.38 409797.24 744421.77
> network 6697.98 8506.20 15204.18
> storage 4193.90 5622.94 9816.84
> exempt 3941.24 4036.10 7977.34
> raid 7872.95 4627.33 12500.27
> target 4409.87 2761.70 7171.57
> netcache 0.00 0.00 0.00
> netcache2 0.00 0.00 0.00
> cifs 352.07 271.33 623.39
> wafl_exempt 0.00 0.00 0.00
>
> 49.342358 seconds with one or more CPUs active ( 76%)
>
> 46.750469 seconds with one CPU active ( 72%)
> 2.591888 seconds with both CPUs active ( 4%)
>
> Domain Utilization By Exempt (per second)
> 0.00 idle 0.00 kahuna
> 0.00 network 0.00 storage
> 0.00 exempt 0.00 raid
> 0.00 target 0.00 netcache
> 0.00 netcache2 0.00 cifs
> 0.00 wafl_exempt
>
>
> CSMP Domain Switches (per second)
> From\To idle kahuna network storage exempt
raid target netcache netcache2 cifs wafl_exem
> pt
> idle 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
> 0
> kahuna 0.00 0.00 27.09 2.65 0.00
5.32 1.87 0.00 0.00 18.79 0.0
> 0
> network 0.00 27.09 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.20 0.0
> 0
> storage 0.00 2.65 0.00 0.00 0.00
12.96 0.00 0.00 0.00 0.00 0.0
> 0
> exempt 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
> 0
> raid 0.00 5.32 0.00 12.96 0.00
0.00 0.00 0.00 0.00 0.00 0.0
> 0
> target 0.00 1.87 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
> 0
> netcache 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
> 0
> netcache2 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
> 0
> cifs 0.00 18.79 0.20 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
> 0
> wafl_exempt 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.
> 00
>
> Miscellaneous Statistics (per second)
> 3046.55 hard context switches 10.38 NFS operations
> 1.42 CIFS operations 0.00 HTTP operations
> 0.00 NetCache URLs 0.00 streaming packets
> 1240.42 network KB received 100.32 network KB
transmitted
> 655.39 disk KB read 1146.02 disk KB written
> 648.90 NVRAM KB written 0.00 nolog KB written
> 8.27 WAFL bufs given to clients 0.00 checksum cache hits
( 0%)
> 1.23 no checksum - partial buffer 0.00 FCP operations
> 74.48 iSCSI operations
>
> WAFL Statistics (per second)
> 15.94 name cache hits ( 95%) 0.86 name cache misses
( 5%)
> 529973.33 buf hash hits ( 100%) 1710.32 buf hash misses
( 0%)
> 516.44 inode cache hits ( 100%) 0.06 inode cache misses
( 0%)
> 262496.95 buf cache hits ( 100%) 19.65 buf cache misses
( 0%)
> 30.17 blocks read 67.00 blocks read-ahead
> 58.10 chains read-ahead 0.99 dummy reads
> 14.25 blocks speculative read-ahead 226.31 blocks written
> 2.27 stripes written 0.00 blocks over-written
> 0.09 wafl_timer generated CP 0.00 snapshot generated
CP
> 0.00 wafl_avail_bufs generated CP 0.00 dirty_blk_cnt
generated CP
> 0.00 full NV-log generated CP 0.00 back-to-back CP
> 0.00 flush generated CP 0.00 sync generated CP
> 0.00 wafl_avail_vbufs generated CP 0.00 deferred
back-to-back CP
> 0.00 low datavecs generated CP 304.72 non-restart messages
> 4.98 IOWAIT suspends 133872 buffers
>
> RAID Statistics (per second)
> 40.91 xors 0.00 long dispatches [0]
> 0.00 long consumed [0] 0.00 long consumed hipri
[0]
> 0.00 long low priority [0] 0.00 long high priority
[0]
> 0.00 long monitor tics [0] 0.00 long monitor clears
[0]
> 0.00 long dispatches [1] 0.00 long consumed [1]
> 0.00 long consumed hipri [1] 0.00 long low priority
[1]
> 0.00 long high priority [1] 0.00 long monitor tics
[1]
> 0.00 long monitor clears [1] 18 max batch
> 2.11 blocked mode xor 19.68 timed mode xor
> 0.05 fast adjustments 0.05 slow adjustments
> 0 avg batch start 0 avg stripe/msec
> 2.64 tetrises written 0.00 master tetrises
> 0.00 slave tetrises 29.70 stripes written
> 11.22 partial stripes 18.48 full stripes
> 226.68 blocks written 31.56 blocks read
> 1.13 1 blocks per stripe size 6 0.29 2 blocks per stripe
size 6
> 0.17 3 blocks per stripe size 6 0.35 4 blocks per stripe
size 6
> 1.80 5 blocks per stripe size 6 15.63 6 blocks per stripe
size 6
> 1.11 1 blocks per stripe size 16 0.31 2 blocks per stripe
size 16
> 0.32 3 blocks per stripe size 16 0.19 4 blocks per stripe
size 16
> 0.09 5 blocks per stripe size 16 0.20 6 blocks per stripe
size 16
> 0.05 7 blocks per stripe size 16 0.20 8 blocks per stripe
size 16
> 0.12 9 blocks per stripe size 16 0.23 10 blocks per stripe
size 16
> 0.23 11 blocks per stripe size 16 0.56 12 blocks per stripe
size 16
> 0.83 13 blocks per stripe size 16 1.26 14 blocks per stripe
size 16
> 1.76 15 blocks per stripe size 16 2.85 16 blocks per stripe
size 16
>
> Network Interface Statistics (per second)
> iface side bytes packets multicasts errors collisions
pkt drops
> e0a recv 114768.96 92.82 0.00 0.00
0.00
> xmit 193.85 1.43 0.05 0.00 0.00
> e0b recv 521347.10 383.35 0.00 0.00
0.00
> xmit 53015.11 280.33 0.14 0.00 0.00
> vh recv 0.00 0.00 0.00 0.00
0.00
> xmit 0.00 0.00 0.00 0.00 0.00
> mtrunk1 recv 636116.06 476.18 0.00 0.00
0.00
> xmit 53208.96 281.77 0.25 0.00 0.00
> mtrunk1-1 recv 588.41 5.26 2.10 0.00
0.00
> xmit 665.54 2.92 0.08 0.00 0.00
> mtrunk1-21 recv 261620.02 204.87 0.00 0.00
0.00
> xmit 22000.14 120.08 0.05 0.00 0.00
> mtrunk1-22 recv 0.00 0.00 0.00 0.00
0.00
> xmit 5.28 0.05 0.05 0.00 0.00
> mtrunk1-23 recv 371871.00 264.20 0.02 0.00
0.00
> xmit 26855.71 156.13 0.05 0.00 0.00
>
> Disk Statistics (per second)
> ut% is the percent of time the disk was busy.
> xfers is the number of data-transfer commands issued per
second.
> xfers = ureads + writes + cpreads + greads + gwrites
> chain is the average number of 4K blocks per command.
> usecs is the average disk round-trip time per 4K block.
>
> disk ut% xfers ureads--chain-usecs writes--chain-usecs
cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs
> /aggr2/plex0/rg0:
> 0b.45 1 1.97 0.19 1.00 12333 1.10 9.42 988
0.69 5.07 768 0.00 .... . 0.00 .... .
> 0b.44 2 2.16 0.19 1.00 38167 1.28 8.35 925
0.69 5.07 838 0.00 .... . 0.00 .... .
> 0b.43 5 6.03 4.29 1.04 14107 1.13 6.84 1489
0.62 2.85 2430 0.00 .... . 0.00 .... .
> 0b.42 4 5.20 3.72 1.05 15565 0.88 8.00 1197
0.60 5.03 939 0.00 .... . 0.00 .... .
> 0b.41 4 5.09 3.76 1.01 14919 0.88 8.98 1184
0.45 4.03 1325 0.00 .... . 0.00 .... .
> 0b.40 5 5.28 4.09 1.08 13686 0.74 9.42 1210
0.45 5.28 1477 0.00 .... . 0.00 .... .
> 0b.39 5 5.54 4.18 1.08 15908 0.83 9.63 1142
0.52 4.44 1788 0.00 .... . 0.00 .... .
> 0b.38 4 5.45 3.90 1.00 17528 0.91 8.51 1327
0.63 3.56 1349 0.00 .... . 0.00 .... .
> 0b.37 4 5.29 3.72 1.07 16442 0.97 7.95 1359
0.60 4.23 1509 0.00 .... . 0.00 .... .
> 0b.36 4 5.01 3.50 1.01 13113 0.89 8.66 1371
0.62 4.18 1413 0.00 .... . 0.00 .... .
> 0b.32 5 5.54 4.03 1.36 11093 0.91 8.22 1433
0.60 3.92 1255 0.00 .... . 0.00 .... .
> 0b.17 4 5.06 3.76 1.01 13752 0.83 9.11 1268
0.46 4.60 1254 0.00 .... . 0.00 .... .
> 0b.33 5 5.51 3.90 1.02 14456 0.88 7.89 1553
0.73 4.02 1280 0.00 .... . 0.00 .... .
> 0b.18 4 5.51 3.79 1.19 12478 0.97 7.76 1591
0.74 3.71 1472 0.00 .... . 0.00 .... .
> 0b.34 5 5.51 3.89 1.12 13188 0.89 8.95 1301
0.73 3.62 1653 0.00 .... . 0.00 .... .
> 0b.19 4 5.04 3.56 1.07 11947 0.83 8.94 1308
0.65 3.50 2238 0.00 .... . 0.00 .... .
> 0b.35 4 4.74 3.33 1.14 12020 0.80 9.21 1353
0.60 4.26 1416 0.00 .... . 0.00 .... .
> 0b.20 4 5.32 3.89 1.01 13341 0.76 9.27 1352
0.68 3.86 1729 0.00 .... . 0.00 .... .
> /aggr1/plex0/rg0:
> 0b.24 2 2.39 0.19 1.00 13667 1.73 11.25 655
0.48 5.06 1140 0.00 .... . 0.00 .... .
> 0b.29 5 7.30 4.97 1.26 17742 1.71 10.48 666
0.62 3.45 1543 0.00 .... . 0.00 .... .
> 0b.26 4 6.34 4.37 1.28 13500 1.42 12.46 595
0.56 3.97 1427 0.00 .... . 0.00 .... .
> 0b.28 5 7.05 4.95 1.14 19218 1.50 11.82 695
0.60 3.44 1642 0.00 .... . 0.00 .... .
> 0b.27 7 6.20 4.26 1.05 19422 1.45 12.28 659
0.49 4.56 1000 0.00 .... . 0.00 .... .
> 0b.22 5 6.90 4.81 1.21 18814 1.53 11.51 835
0.56 3.81 1591 0.00 .... . 0.00 .... .
> 0b.21 4 6.57 4.37 1.11 18990 1.51 11.73 843
0.69 3.80 1655 0.00 .... . 0.00 .... .
> 0b.23 2 2.36 0.19 1.00 9667 1.70 11.42 677
0.48 5.06 1178 0.00 .... . 0.00 .... .
>
> Aggregate statistics:
> Minimum 1 1.97 0.19 0.74
0.45 0.00 0.00
> Mean 4 5.15 3.44 1.11
0.59 0.00 0.00
> Maximum 7 7.30 4.97 1.73
0.74 0.00 0.00
>
> Spares and other disks:
> 0b.25 0 0.00 0.00 .... . 0.00 .... .
0.00 .... . 0.00 .... . 0.00 .... .
>
> Spares and other disks:
> 0b.16 0 0.00 0.00 .... . 0.00 .... .
0.00 .... . 0.00 .... . 0.00 .... .
>
> FCP Statistics (per second)
> 0.00 FCP Bytes recv 0.00 FCP Bytes sent
> 0.00 FCP ops
>
> iSCSI Statistics (per second)
> 516537.68 iSCSI Bytes recv 22866.13 iSCSI Bytes xmit
> 74.48 iSCSI ops
>
> Interrupt Statistics (per second)
> 1604.06 Clock (IRQ 5) 35.51 IPI (IRQ 8)
> 35.51 IPI (IRQ 9) 160.84 FCAL (IRQ 23)
> 259.37 SB MAC (IRQ 25) 41.22 SB MAC (IRQ 26)
> 35.56 IPI 2172.07 total
>
> Data Mover Statistics (per second)
> 627.05 total dma transfer KB 408.55 dma descriptors
> 82.58 dma desc wait count 753.86 total dma transfer
KB
> 523.52 dma descriptors 112.63 dma desc wait count
> 0.00 total dma transfer KB 0.00 dma descriptors
> 0.00 dma desc wait count 0.00 total dma transfer
KB
> 0.00 dma descriptors 0.00 dma desc wait count
>
> ____________________________
> Matt Davies
> Director of International IT Operations
> General Atlantic
> 83 Pall Mall
> London
> SW1Y 5ES
>
> Tel: +44 207 484 3203
> Fax: +44 207 484 2803
> Mobile: +44 777 559 4265
> ____________________________
>
>
>
>
> _____________________________________________________________
> This e-mail (including all attachments) is confidential and may be
privileged.
> It is for the exclusive use of the addressee only. If you are not the
addressee,
> you are hereby notified that any dissemination of this communication
is strictly
> prohibited. If you have received this communication in error, please
erase all
> copies of the message and its attachments and notify us immediately at
> help@generalatlantic.com <mailto:help@generalatlantic.com>. Thank You.
>
>



--
Simple guidelines to happiness:
Work like you don't need the money,
Love like your heart has never been broken and
Dance like no one can see you.

Hi,

if you have Snapmirror running - try "wafl scan status" (priv set advanced) - lock for deswizzling

https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb16746

http://now.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=263215

-----Ursprüngliche Nachricht-----
Von: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] Im Auftrag von Davies,Matt
Gesendet: Donnerstag, 6. November 2008 10:10
An: toasters@mathworks.com
Betreff: High CPU utilisation

Hi all,

We are having a problem with a FAS270 that appears to be having high cpu utilisation issues.

To be honest I have looked at sysstat and statit and I cannot see the problem. I would appreciate if someone could take a look at the below outputs and let me know what I'm missing.

Thanks

Matt

ukfiler*> sysstat -i 3
CPU NFS CIFS iSCSI Net kB/s Disk kB/s iSCSI kB/s Cache
in out read write in out age
85% 18 0 26 203 13 661 0 91 0 24
68% 9 1 50 283 23 692 0 221 9 24
92% 7 4 25 144 11 1643 2443 90 3 24
78% 12 1 74 345 21 618 0 198 0 24
67% 8 0 45 288 18 722 0 205 5 24
96% 13 1 32 236 14 1365 1708 151 1 24
81% 8 4 39 222 16 843 867 167 4 24
85% 8 0 33 443 10 1212 1456 118 0 24
99% 16 2 43 370 21 912 0 168 4 24
99% 14 16 23 219 26 1023 0 98 0 24
99% 12 0 36 585 39 2452 3781 465 18 24
ukfiler*>

Hostname: ukfiler ID: 0084264545 Memory: 1022 MB
NetApp Release 7.2.3: Thu Jul 5 10:06:16 PDT 2007
<6O>
Start time: Wed Nov 5 22:45:03 GMT 2008

CPU Statistics
64.822501 time (seconds) 100 %
52.221602 system time 81 %
0.511694 rupt time 1 % (161581 rupts x 3 usec/rupt)
51.709908 non-rupt system time 80 %
77.423400 idle time 119 %

4.011144 time in CP 6 % 100 %
0.052061 rupt time in CP 1 % (12144 rupts x 4 usec/rupt)

Multiprocessor Statistics (per second)
cpu0 cpu1 total
sk switches 6581.74 7585.82 14167.57
hard switches 1485.87 1560.68 3046.55
domain switches 72.75 65.01 137.76
CP rupts 125.45 61.89 187.34
nonCP rupts 1350.03 955.30 2305.33
IPI rupts 18.23 17.32 35.56

CP rupt usec 565.04 238.10 803.13
nonCP rupt usec 4342.69 2747.95 7090.64
idle 632999.86 561391.13 1194390.83
kahuna 334624.38 409797.24 744421.77
network 6697.98 8506.20 15204.18
storage 4193.90 5622.94 9816.84
exempt 3941.24 4036.10 7977.34
raid 7872.95 4627.33 12500.27
target 4409.87 2761.70 7171.57
netcache 0.00 0.00 0.00
netcache2 0.00 0.00 0.00
cifs 352.07 271.33 623.39
wafl_exempt 0.00 0.00 0.00

49.342358 seconds with one or more CPUs active ( 76%)

46.750469 seconds with one CPU active ( 72%)
2.591888 seconds with both CPUs active ( 4%)

Domain Utilization By Exempt (per second)
0.00 idle 0.00 kahuna
0.00 network 0.00 storage
0.00 exempt 0.00 raid
0.00 target 0.00 netcache
0.00 netcache2 0.00 cifs
0.00 wafl_exempt


CSMP Domain Switches (per second)
From\To idle kahuna network storage exempt raid target netcache netcache2 cifs wafl_exem
pt
idle 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0
0
kahuna 0.00 0.00 27.09 2.65 0.00 5.32 1.87 0.00 0.00 18.79 0.0
0
network 0.00 27.09 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.20 0.0
0
storage 0.00 2.65 0.00 0.00 0.00 12.96 0.00 0.00 0.00 0.00 0.0
0
exempt 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0
0
raid 0.00 5.32 0.00 12.96 0.00 0.00 0.00 0.00 0.00 0.00 0.0
0
target 0.00 1.87 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0
0
netcache 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0
0
netcache2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0
0
cifs 0.00 18.79 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0
0
wafl_exempt 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.
00

Miscellaneous Statistics (per second)
3046.55 hard context switches 10.38 NFS operations
1.42 CIFS operations 0.00 HTTP operations
0.00 NetCache URLs 0.00 streaming packets
1240.42 network KB received 100.32 network KB transmitted
655.39 disk KB read 1146.02 disk KB written
648.90 NVRAM KB written 0.00 nolog KB written
8.27 WAFL bufs given to clients 0.00 checksum cache hits ( 0%)
1.23 no checksum - partial buffer 0.00 FCP operations
74.48 iSCSI operations

WAFL Statistics (per second)
15.94 name cache hits ( 95%) 0.86 name cache misses ( 5%)
529973.33 buf hash hits ( 100%) 1710.32 buf hash misses ( 0%)
516.44 inode cache hits ( 100%) 0.06 inode cache misses ( 0%)
262496.95 buf cache hits ( 100%) 19.65 buf cache misses ( 0%)
30.17 blocks read 67.00 blocks read-ahead
58.10 chains read-ahead 0.99 dummy reads
14.25 blocks speculative read-ahead 226.31 blocks written
2.27 stripes written 0.00 blocks over-written
0.09 wafl_timer generated CP 0.00 snapshot generated CP
0.00 wafl_avail_bufs generated CP 0.00 dirty_blk_cnt generated CP
0.00 full NV-log generated CP 0.00 back-to-back CP
0.00 flush generated CP 0.00 sync generated CP
0.00 wafl_avail_vbufs generated CP 0.00 deferred back-to-back CP
0.00 low datavecs generated CP 304.72 non-restart messages
4.98 IOWAIT suspends 133872 buffers

RAID Statistics (per second)
40.91 xors 0.00 long dispatches [0]
0.00 long consumed [0] 0.00 long consumed hipri [0]
0.00 long low priority [0] 0.00 long high priority [0]
0.00 long monitor tics [0] 0.00 long monitor clears [0]
0.00 long dispatches [1] 0.00 long consumed [1]
0.00 long consumed hipri [1] 0.00 long low priority [1]
0.00 long high priority [1] 0.00 long monitor tics [1]
0.00 long monitor clears [1] 18 max batch
2.11 blocked mode xor 19.68 timed mode xor
0.05 fast adjustments 0.05 slow adjustments
0 avg batch start 0 avg stripe/msec
2.64 tetrises written 0.00 master tetrises
0.00 slave tetrises 29.70 stripes written
11.22 partial stripes 18.48 full stripes
226.68 blocks written 31.56 blocks read
1.13 1 blocks per stripe size 6 0.29 2 blocks per stripe size 6
0.17 3 blocks per stripe size 6 0.35 4 blocks per stripe size 6
1.80 5 blocks per stripe size 6 15.63 6 blocks per stripe size 6
1.11 1 blocks per stripe size 16 0.31 2 blocks per stripe size 16
0.32 3 blocks per stripe size 16 0.19 4 blocks per stripe size 16
0.09 5 blocks per stripe size 16 0.20 6 blocks per stripe size 16
0.05 7 blocks per stripe size 16 0.20 8 blocks per stripe size 16
0.12 9 blocks per stripe size 16 0.23 10 blocks per stripe size 16
0.23 11 blocks per stripe size 16 0.56 12 blocks per stripe size 16
0.83 13 blocks per stripe size 16 1.26 14 blocks per stripe size 16
1.76 15 blocks per stripe size 16 2.85 16 blocks per stripe size 16

Network Interface Statistics (per second)
iface side bytes packets multicasts errors collisions pkt drops
e0a recv 114768.96 92.82 0.00 0.00 0.00
xmit 193.85 1.43 0.05 0.00 0.00
e0b recv 521347.10 383.35 0.00 0.00 0.00
xmit 53015.11 280.33 0.14 0.00 0.00
vh recv 0.00 0.00 0.00 0.00 0.00
xmit 0.00 0.00 0.00 0.00 0.00
mtrunk1 recv 636116.06 476.18 0.00 0.00 0.00
xmit 53208.96 281.77 0.25 0.00 0.00
mtrunk1-1 recv 588.41 5.26 2.10 0.00 0.00
xmit 665.54 2.92 0.08 0.00 0.00
mtrunk1-21 recv 261620.02 204.87 0.00 0.00 0.00
xmit 22000.14 120.08 0.05 0.00 0.00
mtrunk1-22 recv 0.00 0.00 0.00 0.00 0.00
xmit 5.28 0.05 0.05 0.00 0.00
mtrunk1-23 recv 371871.00 264.20 0.02 0.00 0.00
xmit 26855.71 156.13 0.05 0.00 0.00

Disk Statistics (per second)
ut% is the percent of time the disk was busy.
xfers is the number of data-transfer commands issued per second.
xfers = ureads + writes + cpreads + greads + gwrites
chain is the average number of 4K blocks per command.
usecs is the average disk round-trip time per 4K block.

disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs
/aggr2/plex0/rg0:
0b.45 1 1.97 0.19 1.00 12333 1.10 9.42 988 0.69 5.07 768 0.00 .... . 0.00 .... .
0b.44 2 2.16 0.19 1.00 38167 1.28 8.35 925 0.69 5.07 838 0.00 .... . 0.00 .... .
0b.43 5 6.03 4.29 1.04 14107 1.13 6.84 1489 0.62 2.85 2430 0.00 .... . 0.00 .... .
0b.42 4 5.20 3.72 1.05 15565 0.88 8.00 1197 0.60 5.03 939 0.00 .... . 0.00 .... .
0b.41 4 5.09 3.76 1.01 14919 0.88 8.98 1184 0.45 4.03 1325 0.00 .... . 0.00 .... .
0b.40 5 5.28 4.09 1.08 13686 0.74 9.42 1210 0.45 5.28 1477 0.00 .... . 0.00 .... .
0b.39 5 5.54 4.18 1.08 15908 0.83 9.63 1142 0.52 4.44 1788 0.00 .... . 0.00 .... .
0b.38 4 5.45 3.90 1.00 17528 0.91 8.51 1327 0.63 3.56 1349 0.00 .... . 0.00 .... .
0b.37 4 5.29 3.72 1.07 16442 0.97 7.95 1359 0.60 4.23 1509 0.00 .... . 0.00 .... .
0b.36 4 5.01 3.50 1.01 13113 0.89 8.66 1371 0.62 4.18 1413 0.00 .... . 0.00 .... .
0b.32 5 5.54 4.03 1.36 11093 0.91 8.22 1433 0.60 3.92 1255 0.00 .... . 0.00 .... .
0b.17 4 5.06 3.76 1.01 13752 0.83 9.11 1268 0.46 4.60 1254 0.00 .... . 0.00 .... .
0b.33 5 5.51 3.90 1.02 14456 0.88 7.89 1553 0.73 4.02 1280 0.00 .... . 0.00 .... .
0b.18 4 5.51 3.79 1.19 12478 0.97 7.76 1591 0.74 3.71 1472 0.00 .... . 0.00 .... .
0b.34 5 5.51 3.89 1.12 13188 0.89 8.95 1301 0.73 3.62 1653 0.00 .... . 0.00 .... .
0b.19 4 5.04 3.56 1.07 11947 0.83 8.94 1308 0.65 3.50 2238 0.00 .... . 0.00 .... .
0b.35 4 4.74 3.33 1.14 12020 0.80 9.21 1353 0.60 4.26 1416 0.00 .... . 0.00 .... .
0b.20 4 5.32 3.89 1.01 13341 0.76 9.27 1352 0.68 3.86 1729 0.00 .... . 0.00 .... .
/aggr1/plex0/rg0:
0b.24 2 2.39 0.19 1.00 13667 1.73 11.25 655 0.48 5.06 1140 0.00 .... . 0.00 .... .
0b.29 5 7.30 4.97 1.26 17742 1.71 10.48 666 0.62 3.45 1543 0.00 .... . 0.00 .... .
0b.26 4 6.34 4.37 1.28 13500 1.42 12.46 595 0.56 3.97 1427 0.00 .... . 0.00 .... .
0b.28 5 7.05 4.95 1.14 19218 1.50 11.82 695 0.60 3.44 1642 0.00 .... . 0.00 .... .
0b.27 7 6.20 4.26 1.05 19422 1.45 12.28 659 0.49 4.56 1000 0.00 .... . 0.00 .... .
0b.22 5 6.90 4.81 1.21 18814 1.53 11.51 835 0.56 3.81 1591 0.00 .... . 0.00 .... .
0b.21 4 6.57 4.37 1.11 18990 1.51 11.73 843 0.69 3.80 1655 0.00 .... . 0.00 .... .
0b.23 2 2.36 0.19 1.00 9667 1.70 11.42 677 0.48 5.06 1178 0.00 .... . 0.00 .... .

Aggregate statistics:
Minimum 1 1.97 0.19 0.74 0.45 0.00 0.00
Mean 4 5.15 3.44 1.11 0.59 0.00 0.00
Maximum 7 7.30 4.97 1.73 0.74 0.00 0.00

Spares and other disks:
0b.25 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .

Spares and other disks:
0b.16 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .

FCP Statistics (per second)
0.00 FCP Bytes recv 0.00 FCP Bytes sent
0.00 FCP ops

iSCSI Statistics (per second)
516537.68 iSCSI Bytes recv 22866.13 iSCSI Bytes xmit
74.48 iSCSI ops

Interrupt Statistics (per second)
1604.06 Clock (IRQ 5) 35.51 IPI (IRQ 8)
35.51 IPI (IRQ 9) 160.84 FCAL (IRQ 23)
259.37 SB MAC (IRQ 25) 41.22 SB MAC (IRQ 26)
35.56 IPI 2172.07 total

Data Mover Statistics (per second)
627.05 total dma transfer KB 408.55 dma descriptors
82.58 dma desc wait count 753.86 total dma transfer KB
523.52 dma descriptors 112.63 dma desc wait count
0.00 total dma transfer KB 0.00 dma descriptors
0.00 dma desc wait count 0.00 total dma transfer KB
0.00 dma descriptors 0.00 dma desc wait count

____________________________
Matt Davies
Director of International IT Operations
General Atlantic
83 Pall Mall
London
SW1Y 5ES

Tel: +44 207 484 3203
Fax: +44 207 484 2803
Mobile: +44 777 559 4265
____________________________




_____________________________________________________________
This e-mail (including all attachments) is confidential and may be privileged.
It is for the exclusive use of the addressee only. If you are not the addressee, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please erase all copies of the message and its attachments and notify us immediately at help@generalatlantic.com <mailto:help@generalatlantic.com>. Thank You.

I'd agree; we had high CPU utilization on a pre 7.2.4 release on a 960
cluster due to background tasks not being given the correct priority and
thus affecting clients.


-----Original Message-----
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com]
On Behalf Of Kenneth Heal
Sent: 06 November 2008 09:42
To: mdavies@generalatlantic.com; toasters@mathworks.com
Subject: RE: High CPU utilisation


Hi Matt

I have seen several issues with FAS270s and high cpu utilisation related
to several bugs in pre-7.2.4 releases, especially with the filer sat in
background processes, cpu spikes and nfs perf issues. Troubleshooting
these bugs is a pain, and the solution is to upgrade anyway, so I would
be inclined to upgrade to 7.2.6 first and see if the issue still
persists.

cheers
Kenneth

> Subject: High CPU utilisation
> Date: Thu, 6 Nov 2008 09:09:41 +0000
> From: MDAVIES@generalatlantic.com
> To: toasters@mathworks.com
>
> Hi all,
>
> We are having a problem with a FAS270 that appears to be having high
cpu utilisation issues.
>
> To be honest I have looked at sysstat and statit and I cannot see the
problem. I would appreciate if someone could take a look at the below
outputs and let me know what I'm missing.
>
> Thanks
>
> Matt

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE!
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/


To report this email as spam click
https://www.mailcontrol.com/sr/CVCY6ygGE7DTndxI!oX7Us8RWzDHXrou56581NVR9
y+0LES29YJo8kWK7nyITuN9F4Cg!TqFUW1ICd5VsZuQpQ== .

Given that your FAS270 is on 7.2.3, one likely culprit is Bug# 250630
which is first fixed in 7.2.3P6. This issue is specific to the FAS200
series systems.

http://now.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=250630

Once you are on a release with the fix for this in place, the
workaround should be removed.

-jbl

On Nov 6, 2008, at 4:09 AM, Davies,Matt wrote:

> Hi all,
>
> We are having a problem with a FAS270 that appears to be having high
> cpu utilisation issues.
>
> To be honest I have looked at sysstat and statit and I cannot see
> the problem. I would appreciate if someone could take a look at the
> below outputs and let me know what I'm missing.
>
> Thanks
>
> Matt
>
> ukfiler*> sysstat -i 3
> CPU NFS CIFS iSCSI Net kB/s Disk kB/s iSCSI kB/
> s Cache
> in out read write in
> out age
> 85% 18 0 26 203 13 661 0 91
> 0 24
> 68% 9 1 50 283 23 692 0 221
> 9 24
> 92% 7 4 25 144 11 1643 2443 90
> 3 24
> 78% 12 1 74 345 21 618 0 198
> 0 24
> 67% 8 0 45 288 18 722 0 205
> 5 24
> 96% 13 1 32 236 14 1365 1708 151
> 1 24
> 81% 8 4 39 222 16 843 867 167
> 4 24
> 85% 8 0 33 443 10 1212 1456 118
> 0 24
> 99% 16 2 43 370 21 912 0 168
> 4 24
> 99% 14 16 23 219 26 1023 0 98
> 0 24
> 99% 12 0 36 585 39 2452 3781 465
> 18 24
> ukfiler*>
>
> Hostname: ukfiler ID: 0084264545 Memory: 1022 MB
> NetApp Release 7.2.3: Thu Jul 5 10:06:16 PDT 2007
> <6O>
> Start time: Wed Nov 5 22:45:03 GMT 2008
>
> CPU Statistics
> 64.822501 time (seconds) 100 %
> 52.221602 system time 81 %
> 0.511694 rupt time 1 % (161581 rupts x 3 usec/
> rupt)
> 51.709908 non-rupt system time 80 %
> 77.423400 idle time 119 %
>
> 4.011144 time in CP 6 % 100 %
> 0.052061 rupt time in CP 1 % (12144 rupts x
> 4 usec/rupt)
>
> Multiprocessor Statistics (per second)
> cpu0 cpu1 total
> sk switches 6581.74 7585.82 14167.57
> hard switches 1485.87 1560.68 3046.55
> domain switches 72.75 65.01 137.76
> CP rupts 125.45 61.89 187.34
> nonCP rupts 1350.03 955.30 2305.33
> IPI rupts 18.23 17.32 35.56
>
> CP rupt usec 565.04 238.10 803.13
> nonCP rupt usec 4342.69 2747.95 7090.64
> idle 632999.86 561391.13 1194390.83
> kahuna 334624.38 409797.24 744421.77
> network 6697.98 8506.20 15204.18
> storage 4193.90 5622.94 9816.84
> exempt 3941.24 4036.10 7977.34
> raid 7872.95 4627.33 12500.27
> target 4409.87 2761.70 7171.57
> netcache 0.00 0.00 0.00
> netcache2 0.00 0.00 0.00
> cifs 352.07 271.33 623.39
> wafl_exempt 0.00 0.00 0.00
>
> 49.342358 seconds with one or more CPUs active ( 76%)
>
> 46.750469 seconds with one CPU active ( 72%)
> 2.591888 seconds with both CPUs active ( 4%)
>
> Domain Utilization By Exempt (per second)
> 0.00 idle 0.00 kahuna
> 0.00 network 0.00 storage
> 0.00 exempt 0.00 raid
> 0.00 target 0.00 netcache
> 0.00 netcache2 0.00 cifs
> 0.00 wafl_exempt
>
>
> CSMP Domain Switches (per second)
> From\To idle kahuna network storage
> exempt raid target netcache netcache2 cifs
> wafl_exem
> pt
> idle 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00 0.0
> 0
> kahuna 0.00 0.00 27.09 2.65
> 0.00 5.32 1.87 0.00 0.00 18.79 0.0
> 0
> network 0.00 27.09 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.20 0.0
> 0
> storage 0.00 2.65 0.00 0.00
> 0.00 12.96 0.00 0.00 0.00 0.00 0.0
> 0
> exempt 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00 0.0
> 0
> raid 0.00 5.32 0.00 12.96
> 0.00 0.00 0.00 0.00 0.00 0.00 0.0
> 0
> target 0.00 1.87 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00 0.0
> 0
> netcache 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00 0.0
> 0
> netcache2 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00 0.0
> 0
> cifs 0.00 18.79 0.20 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00 0.0
> 0
> wafl_exempt 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00 0.
> 00
>
> Miscellaneous Statistics (per second)
> 3046.55 hard context switches 10.38 NFS operations
> 1.42 CIFS operations 0.00 HTTP operations
> 0.00 NetCache URLs 0.00 streaming packets
> 1240.42 network KB received 100.32 network KB
> transmitted
> 655.39 disk KB read 1146.02 disk KB written
> 648.90 NVRAM KB written 0.00 nolog KB written
> 8.27 WAFL bufs given to clients 0.00 checksum cache
> hits ( 0%)
> 1.23 no checksum - partial buffer 0.00 FCP operations
> 74.48 iSCSI operations
>
> WAFL Statistics (per second)
> 15.94 name cache hits ( 95%) 0.86 name cache
> misses ( 5%)
> 529973.33 buf hash hits ( 100%) 1710.32 buf hash
> misses ( 0%)
> 516.44 inode cache hits ( 100%) 0.06 inode cache
> misses ( 0%)
> 262496.95 buf cache hits ( 100%) 19.65 buf cache
> misses ( 0%)
> 30.17 blocks read 67.00 blocks read-ahead
> 58.10 chains read-ahead 0.99 dummy reads
> 14.25 blocks speculative read-ahead 226.31 blocks written
> 2.27 stripes written 0.00 blocks over-written
> 0.09 wafl_timer generated CP 0.00 snapshot generated
> CP
> 0.00 wafl_avail_bufs generated CP 0.00 dirty_blk_cnt
> generated CP
> 0.00 full NV-log generated CP 0.00 back-to-back CP
> 0.00 flush generated CP 0.00 sync generated CP
> 0.00 wafl_avail_vbufs generated CP 0.00 deferred back-to-
> back CP
> 0.00 low datavecs generated CP 304.72 non-restart messages
> 4.98 IOWAIT suspends 133872 buffers
>
> RAID Statistics (per second)
> 40.91 xors 0.00 long dispatches [0]
> 0.00 long consumed [0] 0.00 long consumed
> hipri [0]
> 0.00 long low priority [0] 0.00 long high priority
> [0]
> 0.00 long monitor tics [0] 0.00 long monitor
> clears [0]
> 0.00 long dispatches [1] 0.00 long consumed [1]
> 0.00 long consumed hipri [1] 0.00 long low priority
> [1]
> 0.00 long high priority [1] 0.00 long monitor tics
> [1]
> 0.00 long monitor clears [1] 18 max batch
> 2.11 blocked mode xor 19.68 timed mode xor
> 0.05 fast adjustments 0.05 slow adjustments
> 0 avg batch start 0 avg stripe/msec
> 2.64 tetrises written 0.00 master tetrises
> 0.00 slave tetrises 29.70 stripes written
> 11.22 partial stripes 18.48 full stripes
> 226.68 blocks written 31.56 blocks read
> 1.13 1 blocks per stripe size 6 0.29 2 blocks per
> stripe size 6
> 0.17 3 blocks per stripe size 6 0.35 4 blocks per
> stripe size 6
> 1.80 5 blocks per stripe size 6 15.63 6 blocks per
> stripe size 6
> 1.11 1 blocks per stripe size 16 0.31 2 blocks per
> stripe size 16
> 0.32 3 blocks per stripe size 16 0.19 4 blocks per
> stripe size 16
> 0.09 5 blocks per stripe size 16 0.20 6 blocks per
> stripe size 16
> 0.05 7 blocks per stripe size 16 0.20 8 blocks per
> stripe size 16
> 0.12 9 blocks per stripe size 16 0.23 10 blocks per
> stripe size 16
> 0.23 11 blocks per stripe size 16 0.56 12 blocks per
> stripe size 16
> 0.83 13 blocks per stripe size 16 1.26 14 blocks per
> stripe size 16
> 1.76 15 blocks per stripe size 16 2.85 16 blocks per
> stripe size 16
>
> Network Interface Statistics (per second)
> iface side bytes packets multicasts errors
> collisions pkt drops
> e0a recv 114768.96 92.82 0.00
> 0.00 0.00
> xmit 193.85 1.43 0.05 0.00 0.00
> e0b recv 521347.10 383.35 0.00
> 0.00 0.00
> xmit 53015.11 280.33 0.14 0.00 0.00
> vh recv 0.00 0.00 0.00
> 0.00 0.00
> xmit 0.00 0.00 0.00 0.00 0.00
> mtrunk1 recv 636116.06 476.18 0.00
> 0.00 0.00
> xmit 53208.96 281.77 0.25 0.00 0.00
> mtrunk1-1 recv 588.41 5.26 2.10
> 0.00 0.00
> xmit 665.54 2.92 0.08 0.00 0.00
> mtrunk1-21 recv 261620.02 204.87 0.00
> 0.00 0.00
> xmit 22000.14 120.08 0.05 0.00 0.00
> mtrunk1-22 recv 0.00 0.00 0.00
> 0.00 0.00
> xmit 5.28 0.05 0.05 0.00 0.00
> mtrunk1-23 recv 371871.00 264.20 0.02
> 0.00 0.00
> xmit 26855.71 156.13 0.05 0.00 0.00
>
> Disk Statistics (per second)
> ut% is the percent of time the disk was busy.
> xfers is the number of data-transfer commands issued per
> second.
> xfers = ureads + writes + cpreads + greads + gwrites
> chain is the average number of 4K blocks per command.
> usecs is the average disk round-trip time per 4K block.
>
> disk ut% xfers ureads--chain-usecs writes--chain-usecs
> cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs
> /aggr2/plex0/rg0:
> 0b.45 1 1.97 0.19 1.00 12333 1.10 9.42
> 988 0.69 5.07 768 0.00 .... . 0.00 .... .
> 0b.44 2 2.16 0.19 1.00 38167 1.28 8.35
> 925 0.69 5.07 838 0.00 .... . 0.00 .... .
> 0b.43 5 6.03 4.29 1.04 14107 1.13 6.84
> 1489 0.62 2.85 2430 0.00 .... . 0.00 .... .
> 0b.42 4 5.20 3.72 1.05 15565 0.88 8.00
> 1197 0.60 5.03 939 0.00 .... . 0.00 .... .
> 0b.41 4 5.09 3.76 1.01 14919 0.88 8.98
> 1184 0.45 4.03 1325 0.00 .... . 0.00 .... .
> 0b.40 5 5.28 4.09 1.08 13686 0.74 9.42
> 1210 0.45 5.28 1477 0.00 .... . 0.00 .... .
> 0b.39 5 5.54 4.18 1.08 15908 0.83 9.63
> 1142 0.52 4.44 1788 0.00 .... . 0.00 .... .
> 0b.38 4 5.45 3.90 1.00 17528 0.91 8.51
> 1327 0.63 3.56 1349 0.00 .... . 0.00 .... .
> 0b.37 4 5.29 3.72 1.07 16442 0.97 7.95
> 1359 0.60 4.23 1509 0.00 .... . 0.00 .... .
> 0b.36 4 5.01 3.50 1.01 13113 0.89 8.66
> 1371 0.62 4.18 1413 0.00 .... . 0.00 .... .
> 0b.32 5 5.54 4.03 1.36 11093 0.91 8.22
> 1433 0.60 3.92 1255 0.00 .... . 0.00 .... .
> 0b.17 4 5.06 3.76 1.01 13752 0.83 9.11
> 1268 0.46 4.60 1254 0.00 .... . 0.00 .... .
> 0b.33 5 5.51 3.90 1.02 14456 0.88 7.89
> 1553 0.73 4.02 1280 0.00 .... . 0.00 .... .
> 0b.18 4 5.51 3.79 1.19 12478 0.97 7.76
> 1591 0.74 3.71 1472 0.00 .... . 0.00 .... .
> 0b.34 5 5.51 3.89 1.12 13188 0.89 8.95
> 1301 0.73 3.62 1653 0.00 .... . 0.00 .... .
> 0b.19 4 5.04 3.56 1.07 11947 0.83 8.94
> 1308 0.65 3.50 2238 0.00 .... . 0.00 .... .
> 0b.35 4 4.74 3.33 1.14 12020 0.80 9.21
> 1353 0.60 4.26 1416 0.00 .... . 0.00 .... .
> 0b.20 4 5.32 3.89 1.01 13341 0.76 9.27
> 1352 0.68 3.86 1729 0.00 .... . 0.00 .... .
> /aggr1/plex0/rg0:
> 0b.24 2 2.39 0.19 1.00 13667 1.73 11.25
> 655 0.48 5.06 1140 0.00 .... . 0.00 .... .
> 0b.29 5 7.30 4.97 1.26 17742 1.71 10.48
> 666 0.62 3.45 1543 0.00 .... . 0.00 .... .
> 0b.26 4 6.34 4.37 1.28 13500 1.42 12.46
> 595 0.56 3.97 1427 0.00 .... . 0.00 .... .
> 0b.28 5 7.05 4.95 1.14 19218 1.50 11.82
> 695 0.60 3.44 1642 0.00 .... . 0.00 .... .
> 0b.27 7 6.20 4.26 1.05 19422 1.45 12.28
> 659 0.49 4.56 1000 0.00 .... . 0.00 .... .
> 0b.22 5 6.90 4.81 1.21 18814 1.53 11.51
> 835 0.56 3.81 1591 0.00 .... . 0.00 .... .
> 0b.21 4 6.57 4.37 1.11 18990 1.51 11.73
> 843 0.69 3.80 1655 0.00 .... . 0.00 .... .
> 0b.23 2 2.36 0.19 1.00 9667 1.70 11.42
> 677 0.48 5.06 1178 0.00 .... . 0.00 .... .
>
> Aggregate statistics:
> Minimum 1 1.97 0.19
> 0.74 0.45 0.00 0.00
> Mean 4 5.15 3.44
> 1.11 0.59 0.00 0.00
> Maximum 7 7.30 4.97
> 1.73 0.74 0.00 0.00
>
> Spares and other disks:
> 0b.25 0 0.00 0.00 .... .
> 0.00 .... . 0.00 .... . 0.00 .... .
> 0.00 .... .
>
> Spares and other disks:
> 0b.16 0 0.00 0.00 .... .
> 0.00 .... . 0.00 .... . 0.00 .... .
> 0.00 .... .
>
> FCP Statistics (per second)
> 0.00 FCP Bytes recv 0.00 FCP Bytes sent
> 0.00 FCP ops
>
> iSCSI Statistics (per second)
> 516537.68 iSCSI Bytes recv 22866.13 iSCSI Bytes xmit
> 74.48 iSCSI ops
>
> Interrupt Statistics (per second)
> 1604.06 Clock (IRQ 5) 35.51 IPI (IRQ 8)
> 35.51 IPI (IRQ 9) 160.84 FCAL (IRQ 23)
> 259.37 SB MAC (IRQ 25) 41.22 SB MAC (IRQ 26)
> 35.56 IPI 2172.07 total
>
> Data Mover Statistics (per second)
> 627.05 total dma transfer KB 408.55 dma descriptors
> 82.58 dma desc wait count 753.86 total dma transfer
> KB
> 523.52 dma descriptors 112.63 dma desc wait count
> 0.00 total dma transfer KB 0.00 dma descriptors
> 0.00 dma desc wait count 0.00 total dma transfer
> KB
> 0.00 dma descriptors 0.00 dma desc wait count
>
> ____________________________
> Matt Davies
> Director of International IT Operations
> General Atlantic
> 83 Pall Mall
> London
> SW1Y 5ES
>
> Tel: +44 207 484 3203
> Fax: +44 207 484 2803
> Mobile: +44 777 559 4265
> ____________________________
>
>
>
>
> _____________________________________________________________
> This e-mail (including all attachments) is confidential and may be
> privileged.
> It is for the exclusive use of the addressee only. If you are not
> the addressee,
> you are hereby notified that any dissemination of this communication
> is strictly
> prohibited. If you have received this communication in error, please
> erase all
> copies of the message and its attachments and notify us immediately at
> help@generalatlantic.com <mailto:help@generalatlantic.com>. Thank You.
>

Jeff,

The clients are experiencing some increase in latency, so I will
investigate further, and see the problems occur during times of high
utilisation.

Thanks

Matt
-----Original Message-----
From: Jeff Mohler [mailto:speedtoys.racing@gmail.com]
Sent: 06 November 2008 19:35
To: Davies,Matt
Cc: toasters@mathworks.com
Subject: Re: High CPU utilisation

Matt:

Are you experiencing client latency issues?

A filer with a "high cpu state" is just that..and many have nothing to
do with client protocol latency.

CPU's doing what it's doing. It's not a windows/unix box.

I dont see from this statit, that you have any actual performance
issue, just "high cpu" which could be innumerable low priority tasks
unique to your platform and OS.



_____________________________________________________________
This e-mail (including all attachments) is confidential and may be privileged.
It is for the exclusive use of the addressee only. If you are not the addressee,
you are hereby notified that any dissemination of this communication is strictly
prohibited. If you have received this communication in error, please erase all
copies of the message and its attachments and notify us immediately at
help@generalatlantic.com <mailto:help@generalatlantic.com>. Thank You.