Mailing List Archive

Frame discards on Brocade with 7-Mode FMC using 8G HBA and DS14
This weekend I did head swap from FAS3140 to FAS8040 in 7-Mode FMC. Storage is DS14mk4 and one DS2246 that was added later. Backend switches are Brocade 300 so we tried to change HBA ports from 4G to 8G hoping to boost overall performance. But almost immediately I observed intermittent

Sat Oct 1 23:00:08 YEKT [netapp2:cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of netapp1 disabled (status of backup mailbox is uncertain).
Sat Oct 1 23:00:14 YEKT [netapp2:cf.fsm.takeoverOfPartnerEnabled:notice]: Failover monitor: takeover of netapp1 enabled

On both heads. There was no load, all applications were still stopped at this point. There were no other errors visible in logs.

Looking on switches, the only errors reported for ports were c3 discards. They were most frequent on DS14 ports, but also happened for HBA and even ATTO ports (1 and 13 in this example). Statistic is collected several minutes after resetting counters.

netapp2_fcsw1:admin> porterrshow
frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout
tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx
=======================================================================================================================
0: 124.5k 90.9k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1: 327.5k 290.8k 0 0 0 0 0 0 0 3 0 0 0 0 0 0 3
2: 278.4k 244.0k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
3: 27.0k 29.1k 0 0 0 0 0 0 0 102 0 0 0 0 0 0 102
4: 426.5k 474.3k 0 0 0 0 0 0 0 324 0 0 0 0 0 324 0
5: 41.9k 48.9k 0 0 0 0 0 0 0 120 0 0 0 0 0 0 120
6: 98.8k 109.8k 0 0 0 0 0 0 0 6 0 0 0 0 0 0 6
7: 111.4k 133.1k 0 0 0 0 0 0 0 2 0 0 0 0 0 0 2
8: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
9: 56.2k 64.0k 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1
10: 13.4k 14.9k 0 0 0 0 0 0 0 52 0 0 0 0 0 0 52
11: 46.4k 50.4k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
12: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
13: 22.0k 23.8k 0 0 0 0 0 0 0 38 0 0 0 0 0 0 38
14: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
15: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
16: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
17: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
18: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
19: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
20: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
21: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
22: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
23: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

I could not find any best practice or recommendation regarding port speeds in such configuration (actually, documentation recommends setting port speed to maximum supported by connected device, which in this case means 8G). After forcing HBA back to 4G the above messages and c3 discards went away.

So I'm looking for practical experience in this area. May be there are some settings we missed. Switches are running 7.0.2b (minimal version matching IMT). Shelf firmware is the latest. Data ONTAP is 8.2.4P4.


---
With best regards

Andrei Borzenkov
Senior system engineer
FTS WEMEAI RUC RU SC TMS FOS

FUJITSU
Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation
Tel.: +7 495 730 62 20 ( reception)
Mob.: +7 916 678 7208
Fax: +7 495 730 62 14
E-mail: Andrei.Borzenkov@ts.fujitsu.com
Web: ru.fujitsu.com
Company details: ts.fujitsu.com/imprint
This communication contains information that is confidential, proprietary in nature and/or privileged.š It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation.
Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.


_______________________________________________
Toasters mailing list
Toasters@teaparty.net
http://www.teaparty.net/mailman/listinfo/toasters
Re: Frame discards on Brocade with 7-Mode FMC using 8G HBA and DS14 [ In reply to ]
What FC ports are you using on the FAS8040 (onboard/PCIe Card)?
how about the sysconfig -a for the FC ports involved (not the whole thing!)?
Do you have the correct SFP+s in the FAS8040?
Where did they come from? Did you take them from the FAS3140 (X6539 -> 4Gb)?
(if so, those are likely 4Gb)

The supported ones on the FAS8040 are X6596->16Gb

Of course, if you are using a PCIe crad, that would be different too.


--tmac

*Tim McCarthy, **Principal Consultant*

*Proud Member of the #NetAppATeam <https://twitter.com/NetAppATeam>*

*I Blog at TMACsRack <https://tmacsrack.wordpress.com/>*



[image: NetApp - In partnership with Alpine Testing Solutions][image:
NetApp Certified Data Administrator, ONTAP]
<https://www.certmetrics.com/netapp/public/badge.aspx?t=c&d=2012-11-05&i=35&ci=NETAPP00041276>[image:
NetApp Certified Implementation Engineer - SAN Specialist, ONTAP]
<https://www.certmetrics.com/netapp/public/badge.aspx?t=c&d=2012-11-08&i=36&ci=NETAPP00041276>[image:
NetApp Certified Storage Installation Engineer, ONTAP]
<https://www.certmetrics.com/netapp/public/badge.aspx?t=c&d=2015-10-13&i=38&ci=NETAPP00041276>[image:
NetApp Certified Implementation Engineer - Data Protection Specialist]
<https://www.certmetrics.com/netapp/public/badge.aspx?t=c&d=2015-10-15&i=11&ci=NETAPP00041276>


NetApp Candidate ID: NETAPP00041276
FlexPod Design: Oct 2015 - Jan 2018, S0N62WE1BMVEYF3M
FlexPod Implementation: Oct 2015 - Jan 2018, JH3QJT4KLEQ41HPH

RHCE6 110-107-141
<https://www.redhat.com/wapps/training/certification/verify.html?certNumber=110-107-141&isSearch=False&verify=Verify>


On Mon, Oct 3, 2016 at 5:32 AM, andrei.borzenkov@ts.fujitsu.com <
andrei.borzenkov@ts.fujitsu.com> wrote:

> This weekend I did head swap from FAS3140 to FAS8040 in 7-Mode FMC.
> Storage is DS14mk4 and one DS2246 that was added later. Backend switches
> are Brocade 300 so we tried to change HBA ports from 4G to 8G hoping to
> boost overall performance. But almost immediately I observed intermittent
>
> Sat Oct 1 23:00:08 YEKT [netapp2:cf.fsm.takeoverOfPartnerDisabled:error]:
> Failover monitor: takeover of netapp1 disabled (status of backup mailbox is
> uncertain).
> Sat Oct 1 23:00:14 YEKT [netapp2:cf.fsm.takeoverOfPartnerEnabled:notice]:
> Failover monitor: takeover of netapp1 enabled
>
> On both heads. There was no load, all applications were still stopped at
> this point. There were no other errors visible in logs.
>
> Looking on switches, the only errors reported for ports were c3 discards.
> They were most frequent on DS14 ports, but also happened for HBA and even
> ATTO ports (1 and 13 in this example). Statistic is collected several
> minutes after resetting counters.
>
> netapp2_fcsw1:admin> porterrshow
> frames enc crc crc too too bad enc
> disc link loss loss frjt fbsy c3timeout
> tx rx in err g_eof shrt long eof out c3
> fail sync sig tx rx
> ============================================================
> ===========================================================
> 0: 124.5k 90.9k 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 1: 327.5k 290.8k 0 0 0 0 0 0 0 3
> 0 0 0 0 0 0 3
> 2: 278.4k 244.0k 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 3: 27.0k 29.1k 0 0 0 0 0 0 0 102
> 0 0 0 0 0 0 102
> 4: 426.5k 474.3k 0 0 0 0 0 0 0 324
> 0 0 0 0 0 324 0
> 5: 41.9k 48.9k 0 0 0 0 0 0 0 120
> 0 0 0 0 0 0 120
> 6: 98.8k 109.8k 0 0 0 0 0 0 0 6
> 0 0 0 0 0 0 6
> 7: 111.4k 133.1k 0 0 0 0 0 0 0 2
> 0 0 0 0 0 0 2
> 8: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 9: 56.2k 64.0k 0 0 0 0 0 0 0 1
> 0 0 0 0 0 0 1
> 10: 13.4k 14.9k 0 0 0 0 0 0 0 52
> 0 0 0 0 0 0 52
> 11: 46.4k 50.4k 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 12: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 13: 22.0k 23.8k 0 0 0 0 0 0 0 38
> 0 0 0 0 0 0 38
> 14: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 15: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 16: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 17: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 18: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 19: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 20: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 21: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 22: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 23: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
>
> I could not find any best practice or recommendation regarding port speeds
> in such configuration (actually, documentation recommends setting port
> speed to maximum supported by connected device, which in this case means
> 8G). After forcing HBA back to 4G the above messages and c3 discards went
> away.
>
> So I'm looking for practical experience in this area. May be there are
> some settings we missed. Switches are running 7.0.2b (minimal version
> matching IMT). Shelf firmware is the latest. Data ONTAP is 8.2.4P4.
>
>
> ---
> With best regards
>
> Andrei Borzenkov
> Senior system engineer
> FTS WEMEAI RUC RU SC TMS FOS
>
> FUJITSU
> Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation
> Tel.: +7 495 730 62 20 ( reception)
> Mob.: +7 916 678 7208
> Fax: +7 495 730 62 14
> E-mail: Andrei.Borzenkov@ts.fujitsu.com
> Web: ru.fujitsu.com
> Company details: ts.fujitsu.com/imprint
> This communication contains information that is confidential, proprietary
> in nature and/or privileged. It is for the exclusive use of the intended
> recipient(s). If you are not the intended recipient(s) or the person
> responsible for delivering it to the intended recipient(s), please note
> that any form of dissemination, distribution or copying of this
> communication is strictly prohibited and may be unlawful. If you have
> received this communication in error, please immediately notify the sender
> and delete the original communication. Thank you for your cooperation.
> Please be advised that neither Fujitsu, its affiliates, its employees or
> agents accept liability for any errors, omissions or damages caused by
> delays of receipt or by any virus infection in this message or its
> attachments, or which may otherwise arise as a result of this e-mail
> transmission.
>
>
> _______________________________________________
> Toasters mailing list
> Toasters@teaparty.net
> http://www.teaparty.net/mailman/listinfo/toasters
>
Re: Frame discards on Brocade with 7-Mode FMC using 8G HBA and DS14 [ In reply to ]
Have you checked the fillword settings for the ports? 8gb with fillword
mode 0 can cause issues.

On Monday, 3 October 2016, andrei.borzenkov@ts.fujitsu.com <
andrei.borzenkov@ts.fujitsu.com> wrote:

> This weekend I did head swap from FAS3140 to FAS8040 in 7-Mode FMC.
> Storage is DS14mk4 and one DS2246 that was added later. Backend switches
> are Brocade 300 so we tried to change HBA ports from 4G to 8G hoping to
> boost overall performance. But almost immediately I observed intermittent
>
> Sat Oct 1 23:00:08 YEKT [netapp2:cf.fsm.takeoverOfPartnerDisabled:error]:
> Failover monitor: takeover of netapp1 disabled (status of backup mailbox is
> uncertain).
> Sat Oct 1 23:00:14 YEKT [netapp2:cf.fsm.takeoverOfPartnerEnabled:notice]:
> Failover monitor: takeover of netapp1 enabled
>
> On both heads. There was no load, all applications were still stopped at
> this point. There were no other errors visible in logs.
>
> Looking on switches, the only errors reported for ports were c3 discards.
> They were most frequent on DS14 ports, but also happened for HBA and even
> ATTO ports (1 and 13 in this example). Statistic is collected several
> minutes after resetting counters.
>
> netapp2_fcsw1:admin> porterrshow
> frames enc crc crc too too bad enc
> disc link loss loss frjt fbsy c3timeout
> tx rx in err g_eof shrt long eof out c3
> fail sync sig tx rx
> ============================================================
> ===========================================================
> 0: 124.5k 90.9k 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 1: 327.5k 290.8k 0 0 0 0 0 0 0 3
> 0 0 0 0 0 0 3
> 2: 278.4k 244.0k 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 3: 27.0k 29.1k 0 0 0 0 0 0 0 102
> 0 0 0 0 0 0 102
> 4: 426.5k 474.3k 0 0 0 0 0 0 0 324
> 0 0 0 0 0 324 0
> 5: 41.9k 48.9k 0 0 0 0 0 0 0 120
> 0 0 0 0 0 0 120
> 6: 98.8k 109.8k 0 0 0 0 0 0 0 6
> 0 0 0 0 0 0 6
> 7: 111.4k 133.1k 0 0 0 0 0 0 0 2
> 0 0 0 0 0 0 2
> 8: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 9: 56.2k 64.0k 0 0 0 0 0 0 0 1
> 0 0 0 0 0 0 1
> 10: 13.4k 14.9k 0 0 0 0 0 0 0 52
> 0 0 0 0 0 0 52
> 11: 46.4k 50.4k 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 12: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 13: 22.0k 23.8k 0 0 0 0 0 0 0 38
> 0 0 0 0 0 0 38
> 14: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 15: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 16: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 17: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 18: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 19: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 20: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 21: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 22: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
> 23: 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0
>
> I could not find any best practice or recommendation regarding port speeds
> in such configuration (actually, documentation recommends setting port
> speed to maximum supported by connected device, which in this case means
> 8G). After forcing HBA back to 4G the above messages and c3 discards went
> away.
>
> So I'm looking for practical experience in this area. May be there are
> some settings we missed. Switches are running 7.0.2b (minimal version
> matching IMT). Shelf firmware is the latest. Data ONTAP is 8.2.4P4.
>
>
> ---
> With best regards
>
> Andrei Borzenkov
> Senior system engineer
> FTS WEMEAI RUC RU SC TMS FOS
>
> FUJITSU
> Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation
> Tel.: +7 495 730 62 20 ( reception)
> Mob.: +7 916 678 7208
> Fax: +7 495 730 62 14
> E-mail: Andrei.Borzenkov@ts.fujitsu.com <javascript:;>
> Web: ru.fujitsu.com
> Company details: ts.fujitsu.com/imprint
> This communication contains information that is confidential, proprietary
> in nature and/or privileged. It is for the exclusive use of the intended
> recipient(s). If you are not the intended recipient(s) or the person
> responsible for delivering it to the intended recipient(s), please note
> that any form of dissemination, distribution or copying of this
> communication is strictly prohibited and may be unlawful. If you have
> received this communication in error, please immediately notify the sender
> and delete the original communication. Thank you for your cooperation.
> Please be advised that neither Fujitsu, its affiliates, its employees or
> agents accept liability for any errors, omissions or damages caused by
> delays of receipt or by any virus infection in this message or its
> attachments, or which may otherwise arise as a result of this e-mail
> transmission.
>
>
> _______________________________________________
> Toasters mailing list
> Toasters@teaparty.net <javascript:;>
> http://www.teaparty.net/mailman/listinfo/toasters
>
RE: Frame discards on Brocade with 7-Mode FMC using 8G HBA and DS14 [ In reply to ]
What FC ports are you using on the FAS8040 (onboard/PCIe Card)?
[Borzenkov, Andrei] onboard

how about the sysconfig -a for the FC ports involved (not the whole thing!)?
[Borzenkov, Andrei]
slot 0: FC Host Adapter 0e (QLogic 8324 rev. 2, N-port, <UP>)
Firmware rev: 7.5.0
Host Port Id: 0x10100
FC Node Name: 5:00a:098003:8826d0
FC Port Name: 5:00a:098003:8826d0
SFP Vendor: AVAGO
SFP Part Number: AFBR-57F5MZ-NA1
SFP Serial Number: AC1609J00D5
SFP Capabilities: 4, 8 or 16 Gbit
Others are the same

Do you have the correct SFP+s in the FAS8040?
[Borzenkov, Andrei] Whatever NetApp ships with FAS8040.

Where did they come from?
[Borzenkov, Andrei] NetApp

Did you take them from the FAS3140 (X6539 -> 4Gb)?
[Borzenkov, Andrei] Of course, not. You seriously expect link to come up at 8G with 4G SFPs?

(if so, those are likely 4Gb)

The supported ones on the FAS8040 are X6596->16Gb
[Borzenkov, Andrei] Yes, I know.

Of course, if you are using a PCIe crad, that would be different too.


--tmac

Tim McCarthy, Principal Consultant

Proud Member of the #NetAppATeam<https://twitter.com/NetAppATeam>

I Blog at TMACsRack<https://tmacsrack.wordpress.com/>



[NetApp - In partnership with Alpine Testing Solutions][NetApp Certified Data Administrator, ONTAP]<https://www.certmetrics.com/netapp/public/badge.aspx?t=c&d=2012-11-05&i=35&ci=NETAPP00041276>[NetApp Certified Implementation Engineer - SAN Specialist, ONTAP]<https://www.certmetrics.com/netapp/public/badge.aspx?t=c&d=2012-11-08&i=36&ci=NETAPP00041276>[NetApp Certified Storage Installation Engineer, ONTAP]<https://www.certmetrics.com/netapp/public/badge.aspx?t=c&d=2015-10-13&i=38&ci=NETAPP00041276>[NetApp Certified Implementation Engineer - Data Protection Specialist]<https://www.certmetrics.com/netapp/public/badge.aspx?t=c&d=2015-10-15&i=11&ci=NETAPP00041276>[https://dl.dropboxusercontent.com/u/6874230/FlexPod2.jpg][http://dl.dropbox.com/u/6874230/rhce.jpeg]


NetApp Candidate ID: NETAPP00041276
FlexPod Design: Oct 2015 - Jan 2018, S0N62WE1BMVEYF3M
FlexPod Implementation: Oct 2015 - Jan 2018, JH3QJT4KLEQ41HPH

RHCE6 110-107-141<https://www.redhat.com/wapps/training/certification/verify.html?certNumber=110-107-141&isSearch=False&verify=Verify>

On Mon, Oct 3, 2016 at 5:32 AM, andrei.borzenkov@ts.fujitsu.com<mailto:andrei.borzenkov@ts.fujitsu.com> <andrei.borzenkov@ts.fujitsu.com<mailto:andrei.borzenkov@ts.fujitsu.com>> wrote:
This weekend I did head swap from FAS3140 to FAS8040 in 7-Mode FMC. Storage is DS14mk4 and one DS2246 that was added later. Backend switches are Brocade 300 so we tried to change HBA ports from 4G to 8G hoping to boost overall performance. But almost immediately I observed intermittent

Sat Oct 1 23:00:08 YEKT [netapp2:cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of netapp1 disabled (status of backup mailbox is uncertain).
Sat Oct 1 23:00:14 YEKT [netapp2:cf.fsm.takeoverOfPartnerEnabled:notice]: Failover monitor: takeover of netapp1 enabled

On both heads. There was no load, all applications were still stopped at this point. There were no other errors visible in logs.

Looking on switches, the only errors reported for ports were c3 discards. They were most frequent on DS14 ports, but also happened for HBA and even ATTO ports (1 and 13 in this example). Statistic is collected several minutes after resetting counters.

netapp2_fcsw1:admin> porterrshow
frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout
tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx
=======================================================================================================================
0: 124.5k 90.9k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1: 327.5k 290.8k 0 0 0 0 0 0 0 3 0 0 0 0 0 0 3
2: 278.4k 244.0k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
3: 27.0k 29.1k 0 0 0 0 0 0 0 102 0 0 0 0 0 0 102
4: 426.5k 474.3k 0 0 0 0 0 0 0 324 0 0 0 0 0 324 0
5: 41.9k 48.9k 0 0 0 0 0 0 0 120 0 0 0 0 0 0 120
6: 98.8k 109.8k 0 0 0 0 0 0 0 6 0 0 0 0 0 0 6
7: 111.4k 133.1k 0 0 0 0 0 0 0 2 0 0 0 0 0 0 2
8: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
9: 56.2k 64.0k 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1
10: 13.4k 14.9k 0 0 0 0 0 0 0 52 0 0 0 0 0 0 52
11: 46.4k 50.4k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
12: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
13: 22.0k 23.8k 0 0 0 0 0 0 0 38 0 0 0 0 0 0 38
14: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
15: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
16: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
17: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
18: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
19: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
20: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
21: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
22: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
23: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

I could not find any best practice or recommendation regarding port speeds in such configuration (actually, documentation recommends setting port speed to maximum supported by connected device, which in this case means 8G). After forcing HBA back to 4G the above messages and c3 discards went away.

So I'm looking for practical experience in this area. May be there are some settings we missed. Switches are running 7.0.2b (minimal version matching IMT). Shelf firmware is the latest. Data ONTAP is 8.2.4P4.


---
With best regards

Andrei Borzenkov
Senior system engineer
FTS WEMEAI RUC RU SC TMS FOS

FUJITSU
Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation
Tel.: +7 495 730 62 20 ( reception)
Mob.: +7 916 678 7208
Fax: +7 495 730 62 14
E-mail: Andrei.Borzenkov@ts.fujitsu.com<mailto:Andrei.Borzenkov@ts.fujitsu.com>
Web: ru.fujitsu.com<http://ru.fujitsu.com>
Company details: ts.fujitsu.com/imprint<http://ts.fujitsu.com/imprint>
This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation.
Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.


_______________________________________________
Toasters mailing list
Toasters@teaparty.net<mailto:Toasters@teaparty.net>
http://www.teaparty.net/mailman/listinfo/toasters
RE: Frame discards on Brocade with 7-Mode FMC using 8G HBA and DS14 [ In reply to ]
Wrong fillword matters for connection of 4G hosts to 8G switch ports. 8G host with 8G switch should be OK (they are running with the same defaults). Also, fillword mismatch exhibits itself in large number of “enc out” errors. There are none in this case. But no, I did not, because there were no symptoms that usually prompt for this.

---
With best regards

Andrei Borzenkov
Senior system engineer
FTS WEMEAI RUC RU SC TMS FOS
[cid:image001.gif@01CBF835.B3FEDA90]
FUJITSU
Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation
Tel.: +7 495 730 62 20 ( reception)
Mob.: +7 916 678 7208
Fax: +7 495 730 62 14
E-mail: Andrei.Borzenkov@ts.fujitsu.com<mailto:Andrei.Borzenkov@ts.fujitsu.com>
Web: ru.fujitsu.com<http://ts.fujitsu.com/>
Company details: ts.fujitsu.com/imprint<http://ts.fujitsu.com/imprint.html>
This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation.
Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.

From: Basil [mailto:basilberntsen@gmail.com]
Sent: Monday, October 03, 2016 2:40 PM
To: Borzenkov, Andrei
Cc: toasters@teaparty.net
Subject: Re: Frame discards on Brocade with 7-Mode FMC using 8G HBA and DS14

Have you checked the fillword settings for the ports? 8gb with fillword mode 0 can cause issues.

On Monday, 3 October 2016, andrei.borzenkov@ts.fujitsu.com<mailto:andrei.borzenkov@ts.fujitsu.com> <andrei.borzenkov@ts.fujitsu.com<mailto:andrei.borzenkov@ts.fujitsu.com>> wrote:
This weekend I did head swap from FAS3140 to FAS8040 in 7-Mode FMC. Storage is DS14mk4 and one DS2246 that was added later. Backend switches are Brocade 300 so we tried to change HBA ports from 4G to 8G hoping to boost overall performance. But almost immediately I observed intermittent

Sat Oct 1 23:00:08 YEKT [netapp2:cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of netapp1 disabled (status of backup mailbox is uncertain).
Sat Oct 1 23:00:14 YEKT [netapp2:cf.fsm.takeoverOfPartnerEnabled:notice]: Failover monitor: takeover of netapp1 enabled

On both heads. There was no load, all applications were still stopped at this point. There were no other errors visible in logs.

Looking on switches, the only errors reported for ports were c3 discards. They were most frequent on DS14 ports, but also happened for HBA and even ATTO ports (1 and 13 in this example). Statistic is collected several minutes after resetting counters.

netapp2_fcsw1:admin> porterrshow
frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout
tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx
=======================================================================================================================
0: 124.5k 90.9k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1: 327.5k 290.8k 0 0 0 0 0 0 0 3 0 0 0 0 0 0 3
2: 278.4k 244.0k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
3: 27.0k 29.1k 0 0 0 0 0 0 0 102 0 0 0 0 0 0 102
4: 426.5k 474.3k 0 0 0 0 0 0 0 324 0 0 0 0 0 324 0
5: 41.9k 48.9k 0 0 0 0 0 0 0 120 0 0 0 0 0 0 120
6: 98.8k 109.8k 0 0 0 0 0 0 0 6 0 0 0 0 0 0 6
7: 111.4k 133.1k 0 0 0 0 0 0 0 2 0 0 0 0 0 0 2
8: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
9: 56.2k 64.0k 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1
10: 13.4k 14.9k 0 0 0 0 0 0 0 52 0 0 0 0 0 0 52
11: 46.4k 50.4k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
12: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
13: 22.0k 23.8k 0 0 0 0 0 0 0 38 0 0 0 0 0 0 38
14: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
15: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
16: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
17: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
18: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
19: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
20: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
21: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
22: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
23: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

I could not find any best practice or recommendation regarding port speeds in such configuration (actually, documentation recommends setting port speed to maximum supported by connected device, which in this case means 8G). After forcing HBA back to 4G the above messages and c3 discards went away.

So I'm looking for practical experience in this area. May be there are some settings we missed. Switches are running 7.0.2b (minimal version matching IMT). Shelf firmware is the latest. Data ONTAP is 8.2.4P4.


---
With best regards

Andrei Borzenkov
Senior system engineer
FTS WEMEAI RUC RU SC TMS FOS

FUJITSU
Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation
Tel.: +7 495 730 62 20 ( reception)
Mob.: +7 916 678 7208
Fax: +7 495 730 62 14
E-mail: Andrei.Borzenkov@ts.fujitsu.com<javascript:;>
Web: ru.fujitsu.com<http://ru.fujitsu.com>
Company details: ts.fujitsu.com/imprint<http://ts.fujitsu.com/imprint>
This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation.
Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.


_______________________________________________
Toasters mailing list
Toasters@teaparty.net<javascript:;>
http://www.teaparty.net/mailman/listinfo/toasters
Re: Frame discards on Brocade with 7-Mode FMC using 8G HBA and DS14 [ In reply to ]
You would not believe the silly stuff I have seen customers do.
 

Get Outlook for iOS




On Mon, Oct 3, 2016 at 7:40 AM -0400, "andrei.borzenkov@ts.fujitsu.com" <andrei.borzenkov@ts.fujitsu.com> wrote:





















 



What FC ports are you using on the FAS8040 (onboard/PCIe Card)?


[Borzenkov, Andrei] onboard


 



how about the sysconfig -a for the FC ports involved (not the whole thing!)?


[Borzenkov, Andrei]



       slot 0: FC Host Adapter 0e (QLogic 8324 rev. 2, N-port, <UP>)


              Firmware rev:      7.5.0


              Host Port Id:      0x10100


              FC Node Name:      5:00a:098003:8826d0


              FC Port Name:      5:00a:098003:8826d0


              SFP Vendor:        AVAGO


              SFP Part Number:   AFBR-57F5MZ-NA1


              SFP Serial Number: AC1609J00D5


              SFP Capabilities:  4, 8 or 16 Gbit


Others are the same


 




Do you have the correct SFP+s in the FAS8040?


[Borzenkov, Andrei] Whatever NetApp ships with FAS8040.


 




Where did they come from?


[Borzenkov, Andrei] NetApp


 


Did you take them from the FAS3140 (X6539 -> 4Gb)?


[Borzenkov, Andrei] Of course, not. You seriously expect link to come up at 8G with 4G SFPs?


 




(if so, those are likely 4Gb)




 




The supported ones on the FAS8040 are X6596->16Gb


[Borzenkov, Andrei] Yes, I know.




 




Of course, if you are using a PCIe crad, that would be different too.




 






















--tmac



 




Tim McCarthy, Principal Consultant




Proud Member of the #NetAppATeam


I Blog at TMACsRack


 











 


 


NetApp Candidate ID: NETAPP00041276


        FlexPod Design: Oct 2015 - Jan 2018, S0N62WE1BMVEYF3M


FlexPod Implementation: Oct 2015 - Jan 2018, JH3QJT4KLEQ41HPH


 



           RHCE6 110-107-141 


















 



On Mon, Oct 3, 2016 at 5:32 AM,
andrei.borzenkov@ts.fujitsu.com <andrei.borzenkov@ts.fujitsu.com> wrote:


This weekend I did head swap from FAS3140 to FAS8040 in 7-Mode FMC. Storage is DS14mk4 and one DS2246 that was added later. Backend switches are Brocade 300 so we tried to change HBA ports from 4G to 8G hoping to boost overall performance.
But almost immediately I observed intermittent



Sat Oct  1 23:00:08 YEKT [netapp2:cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of netapp1 disabled (status of backup mailbox is uncertain).

Sat Oct  1 23:00:14 YEKT [netapp2:cf.fsm.takeoverOfPartnerEnabled:notice]: Failover monitor: takeover of netapp1 enabled



On both heads. There was no load, all applications were still stopped at this point. There were no other errors visible in logs.



Looking on switches, the only errors reported for ports were c3 discards. They were most frequent on DS14 ports, but also happened for HBA and even ATTO ports (1 and 13 in this example). Statistic is collected several minutes after resetting counters.



netapp2_fcsw1:admin> porterrshow

          frames      enc    crc    crc    too    too    bad    enc   disc   link   loss   loss   frjt   fbsy  c3timeout

       tx     rx      in    err    g_eof  shrt   long   eof     out   c3    fail    sync   sig                 tx    rx

     =======================================================================================================================

  0:  124.5k  90.9k   0      0      0      0      0      0      0      0      0      0      0      0      0      0      0

  1:  327.5k 290.8k   0      0      0      0      0      0      0      3      0      0      0      0      0      0      3

  2:  278.4k 244.0k   0      0      0      0      0      0      0      0      0      0      0      0      0      0      0

  3:   27.0k  29.1k   0      0      0      0      0      0      0    102      0      0      0      0      0      0    102

  4:  426.5k 474.3k   0      0      0      0      0      0      0    324      0      0      0      0      0    324      0

  5:   41.9k  48.9k   0      0      0      0      0      0      0    120      0      0      0      0      0      0    120

  6:   98.8k 109.8k   0      0      0      0      0      0      0      6      0      0      0      0      0      0      6

  7:  111.4k 133.1k   0      0      0      0      0      0      0      2      0      0      0      0      0      0      2

  8:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0

  9:   56.2k  64.0k   0      0      0      0      0      0      0      1      0      0      0      0      0      0      1

 10:   13.4k  14.9k   0      0      0      0      0      0      0     52      0      0      0      0      0      0     52

 11:   46.4k  50.4k   0      0      0      0      0      0      0      0      0      0      0      0      0      0      0

 12:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0

 13:   22.0k  23.8k   0      0      0      0      0      0      0     38      0      0      0      0      0      0     38

 14:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0

 15:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0

 16:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0

 17:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0

 18:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0

 19:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0

 20:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0

 21:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0

 22:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0

 23:    0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0



I could not find any best practice or recommendation regarding port speeds in such configuration (actually, documentation recommends setting port speed to maximum supported by connected device, which in this case means 8G). After forcing HBA back to 4G the
above messages and c3 discards went away.



So I'm looking for practical experience in this area. May be there are some settings we missed. Switches are running 7.0.2b (minimal version matching IMT). Shelf firmware is the latest. Data ONTAP is 8.2.4P4.





---

With best regards



Andrei Borzenkov

Senior system engineer

FTS WEMEAI RUC RU SC TMS FOS



FUJITSU

Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation

Tel.: +7 495 730 62 20 ( reception)

Mob.: +7 916 678 7208

Fax: +7 495 730 62 14

E-mail: Andrei.Borzenkov@ts.fujitsu.com

Web: ru.fujitsu.com

Company details: ts.fujitsu.com/imprint

This communication contains information that is confidential, proprietary in nature and/or privileged.  It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended
recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original
communication. Thank you for your cooperation.

Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a
result of this e-mail transmission.





_______________________________________________

Toasters mailing list

Toasters@teaparty.net

http://www.teaparty.net/mailman/listinfo/toasters



 
RE: Frame discards on Brocade with 7-Mode FMC using 8G HBA and DS14 [ In reply to ]
Hi,

Yes, UTA2 do not support direct connection of DS14 because they do not support FC-AL. They are supported in FMC though for connection to backend switches and that is what we have here.

It is clear that setting HBA to 8G won’t make DS14 faster. Still customer has also ATTO 6500N and any future storage extension will go over ATTO as well so it is still beneficial to use full available speed.

---
With best regards

Andrei Borzenkov
Senior system engineer
FTS WEMEAI RUC RU SC TMS FOS
[cid:image001.gif@01CBF835.B3FEDA90]
FUJITSU
Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation
Tel.: +7 495 730 62 20 ( reception)
Mob.: +7 916 678 7208
Fax: +7 495 730 62 14
E-mail: Andrei.Borzenkov@ts.fujitsu.com<mailto:Andrei.Borzenkov@ts.fujitsu.com>
Web: ru.fujitsu.com<http://ts.fujitsu.com/>
Company details: ts.fujitsu.com/imprint<http://ts.fujitsu.com/imprint.html>
This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation.
Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.

From: gene [mailto:genec220@verizon.net]
Sent: Monday, October 03, 2016 8:51 PM
To: 'Tim McCarthy'; Borzenkov, Andrei
Subject: RE: Frame discards on Brocade with 7-Mode FMC using 8G HBA and DS14

Just making sure you got this since my post bounced from toasters…

<assuming you were talking about DS14 connections and not FC SAN targets>
The DS14 shelves are supported on the UTA2 ports. You would need FC HBAs such as the X1132, or X2054.

“The FAS80x0 onboard UTA2 ports and X1143A UTA2 ports are not supported for connectivity to DS14 shelves. To connect to DS14 shelves, use the X1132A-R6 or X2054A-R6 Fibre Channel adapters.”

Also, unless your complete path is 8Gb, you won’t get the 8Gb. The 8Gb links would auto-negotiate down to 4Gb. Shouldn’t error though.

You won’t get any better perf out of the DS14’s by setting the path links to 8Gb either since the fastest the DS14MK4 can run is 4Gb.

{good seeing you, although a mere glance, at Insight, Tim. I continue to learn from your posts. Hope you are well.}

Good luck,

gene


From: gene [mailto:genec220@verizon.net]
Sent: Monday, October 03, 2016 11:34 AM
To: 'Tim McCarthy'; NGC-andrei.borzenkov-ts.fujitsu.com; NGC-andrei.borzenkov-ts.fujitsu.com
Cc: toasters@teaparty.net<mailto:toasters@teaparty.net>
Subject: RE: Frame discards on Brocade with 7-Mode FMC using 8G HBA and DS14

Hi Andrei, I do not want to assume even though you say “backend switches” so are you trying to use the UTA2 ports to connect to the DS14’s or as FC SAN target ports?

gene


From: toasters-bounces@teaparty.net<mailto:toasters-bounces@teaparty.net> [mailto:toasters-bounces@teaparty.net] On Behalf Of Tim McCarthy
Sent: Monday, October 03, 2016 7:47 AM
To: NGC-andrei.borzenkov-ts.fujitsu.com; NGC-andrei.borzenkov-ts.fujitsu.com
Cc: toasters@teaparty.net<mailto:toasters@teaparty.net>
Subject: Re: Frame discards on Brocade with 7-Mode FMC using 8G HBA and DS14

You would not believe the silly stuff I have seen customers do.


Get Outlook for iOS<https://aka.ms/o0ukef>



On Mon, Oct 3, 2016 at 7:40 AM -0400, "andrei.borzenkov@ts.fujitsu.com<mailto:andrei.borzenkov@ts.fujitsu.com>" <andrei.borzenkov@ts.fujitsu.com<mailto:andrei.borzenkov@ts.fujitsu.com>> wrote:

What FC ports are you using on the FAS8040 (onboard/PCIe Card)?
[Borzenkov, Andrei] onboard

how about the sysconfig -a for the FC ports involved (not the whole thing!)?
[Borzenkov, Andrei]
slot 0: FC Host Adapter 0e (QLogic 8324 rev. 2, N-port, <UP>)
Firmware rev: 7.5.0
Host Port Id: 0x10100
FC Node Name: 5:00a:098003:8826d0
FC Port Name: 5:00a:098003:8826d0
SFP Vendor: AVAGO
SFP Part Number: AFBR-57F5MZ-NA1
SFP Serial Number: AC1609J00D5
SFP Capabilities: 4, 8 or 16 Gbit
Others are the same

Do you have the correct SFP+s in the FAS8040?
[Borzenkov, Andrei] Whatever NetApp ships with FAS8040.

Where did they come from?
[Borzenkov, Andrei] NetApp

Did you take them from the FAS3140 (X6539 -> 4Gb)?
[Borzenkov, Andrei] Of course, not. You seriously expect link to come up at 8G with 4G SFPs?

(if so, those are likely 4Gb)

The supported ones on the FAS8040 are X6596->16Gb
[Borzenkov, Andrei] Yes, I know.

Of course, if you are using a PCIe crad, that would be different too.


--tmac

Tim McCarthy, Principal Consultant

Proud Member of the #NetAppATeam<https://twitter.com/NetAppATeam>

I Blog at TMACsRack<https://tmacsrack.wordpress.com/>



[NetApp - In partnership with Alpine Testing Solutions][NetApp Certified Data Administrator, ONTAP]<https://www.certmetrics.com/netapp/public/badge.aspx?t=c&d=2012-11-05&i=35&ci=NETAPP00041276>[NetApp Certified Implementation Engineer - SAN Specialist, ONTAP]<https://www.certmetrics.com/netapp/public/badge.aspx?t=c&d=2012-11-08&i=36&ci=NETAPP00041276>[NetApp Certified Storage Installation Engineer, ONTAP]<https://www.certmetrics.com/netapp/public/badge.aspx?t=c&d=2015-10-13&i=38&ci=NETAPP00041276>[NetApp Certified Implementation Engineer - Data Protection Specialist]<https://www.certmetrics.com/netapp/public/badge.aspx?t=c&d=2015-10-15&i=11&ci=NETAPP00041276>[https://dl.dropboxusercontent.com/u/6874230/FlexPod2.jpg][http://dl.dropbox.com/u/6874230/rhce.jpeg]


NetApp Candidate ID: NETAPP00041276
FlexPod Design: Oct 2015 - Jan 2018, S0N62WE1BMVEYF3M
FlexPod Implementation: Oct 2015 - Jan 2018, JH3QJT4KLEQ41HPH

RHCE6 110-107-141<https://www.redhat.com/wapps/training/certification/verify.html?certNumber=110-107-141&isSearch=False&verify=Verify>

On Mon, Oct 3, 2016 at 5:32 AM, andrei.borzenkov@ts.fujitsu.com<mailto:andrei.borzenkov@ts.fujitsu.com> <andrei.borzenkov@ts.fujitsu.com<mailto:andrei.borzenkov@ts.fujitsu.com>> wrote:
This weekend I did head swap from FAS3140 to FAS8040 in 7-Mode FMC. Storage is DS14mk4 and one DS2246 that was added later. Backend switches are Brocade 300 so we tried to change HBA ports from 4G to 8G hoping to boost overall performance. But almost immediately I observed intermittent

Sat Oct 1 23:00:08 YEKT [netapp2:cf.fsm.takeoverOfPartnerDisabled:error]: Failover monitor: takeover of netapp1 disabled (status of backup mailbox is uncertain).
Sat Oct 1 23:00:14 YEKT [netapp2:cf.fsm.takeoverOfPartnerEnabled:notice]: Failover monitor: takeover of netapp1 enabled

On both heads. There was no load, all applications were still stopped at this point. There were no other errors visible in logs.

Looking on switches, the only errors reported for ports were c3 discards. They were most frequent on DS14 ports, but also happened for HBA and even ATTO ports (1 and 13 in this example). Statistic is collected several minutes after resetting counters.

netapp2_fcsw1:admin> porterrshow
frames enc crc crc too too bad enc disc link loss loss frjt fbsy c3timeout
tx rx in err g_eof shrt long eof out c3 fail sync sig tx rx
=======================================================================================================================
0: 124.5k 90.9k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
1: 327.5k 290.8k 0 0 0 0 0 0 0 3 0 0 0 0 0 0 3
2: 278.4k 244.0k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
3: 27.0k 29.1k 0 0 0 0 0 0 0 102 0 0 0 0 0 0 102
4: 426.5k 474.3k 0 0 0 0 0 0 0 324 0 0 0 0 0 324 0
5: 41.9k 48.9k 0 0 0 0 0 0 0 120 0 0 0 0 0 0 120
6: 98.8k 109.8k 0 0 0 0 0 0 0 6 0 0 0 0 0 0 6
7: 111.4k 133.1k 0 0 0 0 0 0 0 2 0 0 0 0 0 0 2
8: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
9: 56.2k 64.0k 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1
10: 13.4k 14.9k 0 0 0 0 0 0 0 52 0 0 0 0 0 0 52
11: 46.4k 50.4k 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
12: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
13: 22.0k 23.8k 0 0 0 0 0 0 0 38 0 0 0 0 0 0 38
14: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
15: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
16: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
17: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
18: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
19: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
20: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
21: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
22: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
23: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

I could not find any best practice or recommendation regarding port speeds in such configuration (actually, documentation recommends setting port speed to maximum supported by connected device, which in this case means 8G). After forcing HBA back to 4G the above messages and c3 discards went away.

So I'm looking for practical experience in this area. May be there are some settings we missed. Switches are running 7.0.2b (minimal version matching IMT). Shelf firmware is the latest. Data ONTAP is 8.2.4P4.


---
With best regards

Andrei Borzenkov
Senior system engineer
FTS WEMEAI RUC RU SC TMS FOS

FUJITSU
Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation
Tel.: +7 495 730 62 20 ( reception)
Mob.: +7 916 678 7208
Fax: +7 495 730 62 14
E-mail: Andrei.Borzenkov@ts.fujitsu.com<mailto:Andrei.Borzenkov@ts.fujitsu.com>
Web: ru.fujitsu.com<http://ru.fujitsu.com>
Company details: ts.fujitsu.com/imprint<http://ts.fujitsu.com/imprint>
This communication contains information that is confidential, proprietary in nature and/or privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation.
Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.


_______________________________________________
Toasters mailing list
Toasters@teaparty.net<mailto:Toasters@teaparty.net>
http://www.teaparty.net/mailman/listinfo/toasters