Mailing List Archive

Hot remove disk shelves?
Hello Toasters,

We have a FAS3170 HA pair with some SATA disk shelves (AT-FCX)
attached to the end of a chain where none of the disks are in use
(all are spares, all have ownership assigned). We are running
ONTAP 8.0.1.

The chain is singly attached to both controllers, i.e., the
shelf "A" modules are chained to a single FC adapter on one
controller and the shelf "B" modules are chained to a single
FC adapter on the other controller.

The shelves we want to remove are at the far end of the chain.
Essentially we want to undo the result of a hot shelf add without
downtime.

Is it possible to hot remove these shelves? Would it help to
"un-own" the disks first?

I suppose this could be done using CF takeover/giveback
but that might be too disruptive.

Steve Losen scl@virginia.edu phone: 434-924-0640

University of Virginia ITC Unix Support
RE: Hot remove disk shelves? [ In reply to ]
AFAIK there is no official way to do this without downtime.


-----Original Message-----
From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Steve Losen
Sent: 02 June 2011 14:03
To: toasters@mathworks.com
Subject: Hot remove disk shelves?


Hello Toasters,

We have a FAS3170 HA pair with some SATA disk shelves (AT-FCX) attached to the end of a chain where none of the disks are in use (all are spares, all have ownership assigned). We are running ONTAP 8.0.1.

The chain is singly attached to both controllers, i.e., the shelf "A" modules are chained to a single FC adapter on one controller and the shelf "B" modules are chained to a single FC adapter on the other controller.

The shelves we want to remove are at the far end of the chain.
Essentially we want to undo the result of a hot shelf add without downtime.

Is it possible to hot remove these shelves? Would it help to "un-own" the disks first?

I suppose this could be done using CF takeover/giveback but that might be too disruptive.

Steve Losen scl@virginia.edu phone: 434-924-0640

University of Virginia ITC Unix Support




To report this email as spam click https://www.mailcontrol.com/sr/wQw0zmjPoHdJTZGyOCrrhg== .


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom
More information can be found at www.csr.com. Follow CSR on Twitter at http://twitter.com/CSR_PLC and read our blog at www.csr.com/blog
Re: Hot remove disk shelves? [ In reply to ]
I've seen people do this without problems, but I've not had to do it
yet myself. Having the disks unassigned is a good idea since you'll
want them unowned when you add them to other heads. If you can afford
to do it while in takeover mode that probably couldn't hurt.

Let us know how it goes. I'm under the impression this works, but is
unsupported from netapp eng still.

-Blake

On Thu, Jun 2, 2011 at 6:03 AM, Steve Losen <scl@sasha.acc.virginia.edu> wrote:
>
> Hello Toasters,
>
> We have a FAS3170 HA pair with some SATA disk shelves (AT-FCX)
> attached to the end of a chain where none of the disks are in use
> (all are spares, all have ownership assigned).  We are running
> ONTAP 8.0.1.
>
> The chain is singly attached to both controllers, i.e., the
> shelf "A" modules are chained to a single FC adapter on one
> controller and the shelf "B" modules are chained to a single
> FC adapter on the other controller.
>
> The shelves we want to remove are at the far end of the chain.
> Essentially we want to undo the result of a hot shelf add without
> downtime.
>
> Is it possible to hot remove these shelves?  Would it help to
> "un-own" the disks first?
>
> I suppose this could be done using CF takeover/giveback
> but that might be too disruptive.
>
> Steve Losen   scl@virginia.edu    phone: 434-924-0640
>
> University of Virginia               ITC Unix Support
>
>
>
RE: Hot remove disk shelves? [ In reply to ]
It may appear to work at first, but you'll have a panic and crash eventually. It might last a week or two, but you'll eventually have problems.

If you move quick and you have your FC/NFS timeouts set right, you can shut down, decable the shelves, and turn it back on before hosts will notice. I've done that myself more than a few times.


>-----Original Message-----
>From: Blake Golliher [mailto:thelastman@gmail.com]
>Sent: Thursday, June 02, 2011 3:07 PM
>To: Steve Losen
>Cc: toasters@mathworks.com
>Subject: Re: Hot remove disk shelves?
>
>I've seen people do this without problems, but I've not had to do it
>yet myself. Having the disks unassigned is a good idea since you'll
>want them unowned when you add them to other heads. If you can afford
>to do it while in takeover mode that probably couldn't hurt.
>
>Let us know how it goes. I'm under the impression this works, but is
>unsupported from netapp eng still.
>
>-Blake
>
>On Thu, Jun 2, 2011 at 6:03 AM, Steve Losen <scl@sasha.acc.virginia.edu>
>wrote:
>>
>> Hello Toasters,
>>
>> We have a FAS3170 HA pair with some SATA disk shelves (AT-FCX)
>> attached to the end of a chain where none of the disks are in use
>> (all are spares, all have ownership assigned).  We are running
>> ONTAP 8.0.1.
>>
>> The chain is singly attached to both controllers, i.e., the
>> shelf "A" modules are chained to a single FC adapter on one
>> controller and the shelf "B" modules are chained to a single
>> FC adapter on the other controller.
>>
>> The shelves we want to remove are at the far end of the chain.
>> Essentially we want to undo the result of a hot shelf add without
>> downtime.
>>
>> Is it possible to hot remove these shelves?  Would it help to
>> "un-own" the disks first?
>>
>> I suppose this could be done using CF takeover/giveback
>> but that might be too disruptive.
>>
>> Steve Losen   scl@virginia.edu    phone: 434-924-0640
>>
>> University of Virginia               ITC Unix Support
>>
>>
>>
RE: Hot remove disk shelves? [ In reply to ]
I've heard that asked before, and the answer was no. The issue was something about internal device maps. The filers build maps of which devices are where on the loops, and that includes the shelf itself as well as the drives. Both controllers are aware of the shelves at all times, even if they aren't actually using disks inside them. The only safe way to remove the shelf is at a time when neither controller is aware of it, and that means when both are off.


>-----Original Message-----
>From: Borzenkov, Andrey [mailto:andrey.borzenkov@ts.fujitsu.com]
>Sent: Thursday, June 02, 2011 4:06 PM
>To: Steiner, Jeffrey; Blake Golliher; Steve Losen
>Cc: toasters@mathworks.com
>Subject: RE: Hot remove disk shelves?
>
>Would it be possible via takeover/giveback? I.e. - takeover node2,
>unplug shelf on node2 side then do the same with node1? Would it still
>lead to problem mentioned?
>
>
>---
>With best regards
>
>Andrey Borzenkov
>Senior system engineer
>Service operations
>
>-----Original Message-----
>From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com]
>On Behalf Of Steiner, Jeffrey
>Sent: Thursday, June 02, 2011 6:45 PM
>To: Blake Golliher; Steve Losen
>Cc: toasters@mathworks.com
>Subject: RE: Hot remove disk shelves?
>
>It may appear to work at first, but you'll have a panic and crash
>eventually. It might last a week or two, but you'll eventually have
>problems.
>
>If you move quick and you have your FC/NFS timeouts set right, you can
>shut down, decable the shelves, and turn it back on before hosts will
>notice. I've done that myself more than a few times.
>
>
>>-----Original Message-----
>>From: Blake Golliher [mailto:thelastman@gmail.com]
>>Sent: Thursday, June 02, 2011 3:07 PM
>>To: Steve Losen
>>Cc: toasters@mathworks.com
>>Subject: Re: Hot remove disk shelves?
>>
>>I've seen people do this without problems, but I've not had to do it
>>yet myself. Having the disks unassigned is a good idea since you'll
>>want them unowned when you add them to other heads. If you can afford
>>to do it while in takeover mode that probably couldn't hurt.
>>
>>Let us know how it goes. I'm under the impression this works, but is
>>unsupported from netapp eng still.
>>
>>-Blake
>>
>>On Thu, Jun 2, 2011 at 6:03 AM, Steve Losen
><scl@sasha.acc.virginia.edu>
>>wrote:
>>>
>>> Hello Toasters,
>>>
>>> We have a FAS3170 HA pair with some SATA disk shelves (AT-FCX)
>>> attached to the end of a chain where none of the disks are in use
>>> (all are spares, all have ownership assigned).  We are running
>>> ONTAP 8.0.1.
>>>
>>> The chain is singly attached to both controllers, i.e., the
>>> shelf "A" modules are chained to a single FC adapter on one
>>> controller and the shelf "B" modules are chained to a single
>>> FC adapter on the other controller.
>>>
>>> The shelves we want to remove are at the far end of the chain.
>>> Essentially we want to undo the result of a hot shelf add without
>>> downtime.
>>>
>>> Is it possible to hot remove these shelves?  Would it help to
>>> "un-own" the disks first?
>>>
>>> I suppose this could be done using CF takeover/giveback
>>> but that might be too disruptive.
>>>
>>> Steve Losen   scl@virginia.edu    phone: 434-924-0640
>>>
>>> University of Virginia               ITC Unix Support
>>>
>>>
>>>
>
Re: Hot remove disk shelves? [ In reply to ]
On 2011 Jun 2, at 16:45 , Steiner, Jeffrey wrote:
> It may appear to work at first, but you'll have a panic and crash eventually. It might last a week or two, but you'll eventually have problems.

I accidentally removed a shelf full of unowned disks once from a production machine, and that worked without a (technical) problem. It is however unsupported by netapp, so I got mildly grilled by my coworkers for putting a production filer in danger.

In this case, the shelf I removed was the only one on that loop. That might make things slightly "better", but I don't know for sure.

You definitely want to unown the disks first, regardless of whether you power down or not.

Note that takeover isn't going to help. At least not in the normal takeover situation. Both partners see the same disks, so the one that's still up still sees a shelf disappearing.

Powering down seems to me like your only (supported) option.

> If you move quick and you have your FC/NFS timeouts set right, you can shut down, decable the shelves, and turn it back on before hosts will notice. I've done that myself more than a few times.
>
>
>> -----Original Message-----
>> From: Blake Golliher [mailto:thelastman@gmail.com]
>> Sent: Thursday, June 02, 2011 3:07 PM
>> To: Steve Losen
>> Cc: toasters@mathworks.com
>> Subject: Re: Hot remove disk shelves?
>>
>> I've seen people do this without problems, but I've not had to do it
>> yet myself. Having the disks unassigned is a good idea since you'll
>> want them unowned when you add them to other heads. If you can afford
>> to do it while in takeover mode that probably couldn't hurt.
>>
>> Let us know how it goes. I'm under the impression this works, but is
>> unsupported from netapp eng still.
>>
>> -Blake
>>
>> On Thu, Jun 2, 2011 at 6:03 AM, Steve Losen <scl@sasha.acc.virginia.edu>
>> wrote:
>>>
>>> Hello Toasters,
>>>
>>> We have a FAS3170 HA pair with some SATA disk shelves (AT-FCX)
>>> attached to the end of a chain where none of the disks are in use
>>> (all are spares, all have ownership assigned). We are running
>>> ONTAP 8.0.1.
>>>
>>> The chain is singly attached to both controllers, i.e., the
>>> shelf "A" modules are chained to a single FC adapter on one
>>> controller and the shelf "B" modules are chained to a single
>>> FC adapter on the other controller.
>>>
>>> The shelves we want to remove are at the far end of the chain.
>>> Essentially we want to undo the result of a hot shelf add without
>>> downtime.
>>>
>>> Is it possible to hot remove these shelves? Would it help to
>>> "un-own" the disks first?
>>>
>>> I suppose this could be done using CF takeover/giveback
>>> but that might be too disruptive.
>>>
>>> Steve Losen scl@virginia.edu phone: 434-924-0640
>>>
>>> University of Virginia ITC Unix Support
>>>
>>>
>>>
>
>

--
Jan-Pieter Cornet <johnpc@xs4all.nl>
"People are continuously reinventing the flat tyre".
Re: Hot remove disk shelves? [ In reply to ]
Related-- I've had situations where a shelf is inadvertently offline to one of the filer heads in a cluster.  It didn't panic (it was like this for months...) but the cluster wouldn't fail over due to the shelf count mismatch.



________________________________
From: "Steiner, Jeffrey" <Jeffrey.Steiner@netapp.com>
To: "Borzenkov, Andrey" <andrey.borzenkov@ts.fujitsu.com>; Blake Golliher <thelastman@gmail.com>; Steve Losen <scl@sasha.acc.virginia.edu>
Cc: toasters@mathworks.com
Sent: Thursday, June 2, 2011 11:13 AM
Subject: RE: Hot remove disk shelves?

I've heard that asked before, and the answer was no. The issue was something about internal device maps. The filers build maps of which devices are where on the loops, and that includes the shelf itself as well as the drives. Both controllers are aware of the shelves at all times, even if they aren't actually using disks inside them. The only safe way to remove the shelf is at a time when neither controller is aware of it, and that means when both are off.


>-----Original Message-----
>From: Borzenkov, Andrey [mailto:andrey.borzenkov@ts.fujitsu.com]
>Sent: Thursday, June 02, 2011 4:06 PM
>To: Steiner, Jeffrey; Blake Golliher; Steve Losen
>Cc: toasters@mathworks.com
>Subject: RE: Hot remove disk shelves?
>
>Would it be possible via takeover/giveback? I.e. - takeover node2,
>unplug shelf on node2 side then do the same with node1? Would it still
>lead to problem mentioned?
>
>
>---
>With best regards
>
>Andrey Borzenkov
>Senior system engineer
>Service operations
>
>-----Original Message-----
>From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com]
>On Behalf Of Steiner, Jeffrey
>Sent: Thursday, June 02, 2011 6:45 PM
>To: Blake Golliher; Steve Losen
>Cc: toasters@mathworks.com
>Subject: RE: Hot remove disk shelves?
>
>It may appear to work at first, but you'll have a panic and crash
>eventually. It might last a week or two, but you'll eventually have
>problems.
>
>If you move quick and you have your FC/NFS timeouts set right, you can
>shut down, decable the shelves, and turn it back on before hosts will
>notice. I've done that myself more than a few times.
>
>
>>-----Original Message-----
>>From: Blake Golliher [mailto:thelastman@gmail.com]
>>Sent: Thursday, June 02, 2011 3:07 PM
>>To: Steve Losen
>>Cc: toasters@mathworks.com
>>Subject: Re: Hot remove disk shelves?
>>
>>I've seen people do this without problems, but I've not had to do it
>>yet myself.  Having the disks unassigned is a good idea since you'll
>>want them unowned when you add them to other heads.  If you can afford
>>to do it while in takeover mode that probably couldn't hurt.
>>
>>Let us know how it goes.  I'm under the impression this works, but is
>>unsupported from netapp eng still.
>>
>>-Blake
>>
>>On Thu, Jun 2, 2011 at 6:03 AM, Steve Losen
><scl@sasha.acc.virginia.edu>
>>wrote:
>>>
>>> Hello Toasters,
>>>
>>> We have a FAS3170 HA pair with some SATA disk shelves (AT-FCX)
>>> attached to the end of a chain where none of the disks are in use
>>> (all are spares, all have ownership assigned).  We are running
>>> ONTAP 8.0.1.
>>>
>>> The chain is singly attached to both controllers, i.e., the
>>> shelf "A" modules are chained to a single FC adapter on one
>>> controller and the shelf "B" modules are chained to a single
>>> FC adapter on the other controller.
>>>
>>> The shelves we want to remove are at the far end of the chain.
>>> Essentially we want to undo the result of a hot shelf add without
>>> downtime.
>>>
>>> Is it possible to hot remove these shelves?  Would it help to
>>> "un-own" the disks first?
>>>
>>> I suppose this could be done using CF takeover/giveback
>>> but that might be too disruptive.
>>>
>>> Steve Losen   scl@virginia.edu    phone: 434-924-0640
>>>
>>> University of Virginia               ITC Unix Support
>>>
>>>
>>>
>