Mailing List Archive

HOWTO: Build a high available iscsi Target using heartbeat, drbd and ietd for ESX Server 3.5
Hello,
I just setup a high available iscsi target for an ESX 3.5 server. I can
switch the target between my two hosts without any problems on the ESX
3.5 server side with the exception of a reconnect and a hang in the
virtual machine running on iscsi stoage for ca. 1 second (worst case)
sometimes I don't even notice.

I used the following components:

- Debian Etch
- heartbeat 2.1.3
- drbd 8.1.2 (primary/standby)
- iscsitarget 0.4.15 (ietd)

I attached the ha.cf, drbd.conf, iscsi.xml (CIB), the ressource agent I
used and of course the ietd.conf.

Attention: If you do that, get sure that you specify a Serial Number for
each LUN you export also see README.vmware in the iscsitarget tarball.

Don't forget to adopt the listening ip address in the ietd ressource
agent.

Thomas
Re: HOWTO: Build a high available iscsi Target using heartbeat, drbd and ietd for ESX Server 3.5 [ In reply to ]
Hi,

On Mon, Jan 21, 2008 at 01:43:26PM +0100, Thomas Glanzmann wrote:
> Hello,
> I just setup a high available iscsi target for an ESX 3.5 server. I can
> switch the target between my two hosts without any problems on the ESX
> 3.5 server side with the exception of a reconnect and a hang in the
> virtual machine running on iscsi stoage for ca. 1 second (worst case)
> sometimes I don't even notice.
>
> I used the following components:
>
> - Debian Etch
> - heartbeat 2.1.3
> - drbd 8.1.2 (primary/standby)
> - iscsitarget 0.4.15 (ietd)
>
> I attached the ha.cf, drbd.conf, iscsi.xml (CIB), the ressource agent I
> used and of course the ietd.conf.
>
> Attention: If you do that, get sure that you specify a Serial Number for
> each LUN you export also see README.vmware in the iscsitarget tarball.
>
> Don't forget to adopt the listening ip address in the ietd ressource
> agent.

Nice effort. Thanks for sharing it. Perhaps you'd like to put
this into the wiki.linux-ha.org. If you do, don't forget to
pepper the doc with "YMMV."

A few comments:

1. cib is a bit too lean. There are no attributes set for the
ietd resource.

2. ietd RA is Linux specific. If it has to be then you should
check if it runs on Linux and if not bail out with an appropriate
message.

3. I understand that fixing various memory parameters is
important for ietd performance, but that has no place in the RA.
Placing those settings in the XML info as comment should suffice.
The admins may choose different settings anyway.

4. There are various modprobe statements. Is that necessary? It
should be better to assume that ietd init script has been run and
then just add/remove new targets using the RA.

5. Is this RA just an example?

Cheers,

Dejan

> Thomas

> use_logd yes
> bcast eth1
> mcast eth0 239.0.0.2 694 1 0
> node ha-1 ha-2
> respawn hacluster /usr/lib/heartbeat/dopd
> apiauth dopd uid=hacluster gid=haclient
> crm on
> watchdog /dev/watchdog


> global {
> usage-count no;
> }
>
> common {
> syncer {
> rate 100M;
> }
>
> handlers {
> outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater";
> }
> }
>
> resource postgres {
> protocol C;
>
> startup {
> degr-wfc-timeout 120;
> }
>
> disk {
> on-io-error pass_on;
> fencing resource-only;
> }
>
> on ha-1 {
> device /dev/drbd0;
> disk /dev/sdb;
> address 157.163.248.191:7788;
> meta-disk internal;
> }
>
> on ha-2 {
> device /dev/drbd0;
> disk /dev/sdb;
> address 157.163.248.192:7788;
> meta-disk internal;
> }
> }

> #!/bin/bash
>
> . ${OCF_ROOT}/resource.d/heartbeat/.ocf-shellfuncs
>
> export PID=/var/run/iscsi_trgt.pid
> export EXECUTABLE=/usr/sbin/ietd
>
> # Don't touch this "memsize thingy" unless you are blessed # with knowledge
> # about it.
> MEM_SIZE=1048576
>
> configure_memsize()
> {
> if [ -e /proc/sys/net/core/wmem_max ]; then
> echo ${MEM_SIZE} > /proc/sys/net/core/wmem_max
> fi
>
> if [ -e /proc/sys/net/core/rmem_max ]; then
> echo ${MEM_SIZE} > /proc/sys/net/core/rmem_max
> fi
>
> if [ -e /proc/sys/net/core/wmem_default ]; then
> echo ${MEM_SIZE} > /proc/sys/net/core/wmem_default
> fi
>
> if [ -e /proc/sys/net/core/rmem_default ]; then
> echo ${MEM_SIZE} > /proc/sys/net/core/rmem_default
> fi
>
> if [ -e /proc/sys/net/ipv4/tcp_mem ]; then
> echo "${MEM_SIZE} ${MEM_SIZE} ${MEM_SIZE}" > /proc/sys/net/ipv4/tcp_mem
> fi
>
> if [ -e /proc/sys/net/ipv4/tcp_rmem ]; then
> echo "${MEM_SIZE} ${MEM_SIZE} ${MEM_SIZE}" > /proc/sys/net/ipv4/tcp_rmem
> fi
>
> if [ -e /proc/sys/net/ipv4/tcp_wmem ]; then
> echo "${MEM_SIZE} ${MEM_SIZE} ${MEM_SIZE}" > /proc/sys/net/ipv4/tcp_wmem
> fi
> }
>
> case "$1" in
> start)
> if [ -f "${PID}" ]; then
> kill -0 `cat ${PID}` &> /dev/null && exit 0;
>
> else
> rm -f "${PID}"
> fi
>
> configure_memsize
> modprobe -q crc32c
> modprobe iscsi_trgt
> ${EXECUTABLE} -a 10.0.0.193
> ;;
>
> stop)
> if [ -f "${PID}" ]; then
> /usr/sbin/ietadm --op delete
> kill `cat ${PID}` &> /dev/null
> fi
>
> rm -f "${PID}"
>
> exit 0;
> ;;
>
> status)
> if [ -f "${PID}" ]; then
> kill -0 `cat ${PID}` &> /dev/null && exit 0;
> fi
>
> exit 1;
> ;;
>
> monitor)
> if [ -f "${PID}" ]; then
> kill -0 `cat ${PID}` &> /dev/null && exit 0;
> fi
>
> exit 7;
> ;;
>
> meta-data)
> cat <<END
> <?xml version="1.0"?>
> <!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd">
> <resource-agent name="ietd">
> <version>1.0</version>
>
> <longdesc lang="en">
> OCF Ressource Agent for ISCSI Enterprise Target Daemon.
> </longdesc>
>
> <shortdesc lang="en">OCF Ressource Agent for ISCSI Enterprise Target Daemon.</shortdesc>
>
> <actions>
> <action name="start" timeout="90" />
> <action name="stop" timeout="100" />
> <action name="status" timeout="60" />
> <action name="monitor" depth="0" timeout="30s" interval="10s" start-delay="10s" />
> <action name="meta-data" timeout="5s" />
> <action name="validate-all" timeout="20s" />
> </actions>
> </resource-agent>
> END
> ;;
> esac

> Target iqn.2001-04.de.glanzmann:iscsi-test1
> Lun 0 Path=/dev/drbd0,Type=fileio,ScsiSN=AMHDSK-061031-03

> _______________________________________________
> Linux-HA mailing list
> Linux-HA@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
Re: HOWTO: Build a high available iscsi Target using heartbeat, drbd and ietd for ESX Server 3.5 [ In reply to ]
Hi Thomas,

I just did a very similar thing, except in my case I am using shared
storage (MD3000 - SAS) and theres a bit more fun to that part of it
(multipath, stonith, etc) - also I setup heartbeat in v1 mode not CRM
mode.

I plan to post a walkthrough at some point in the future (I also setup
SMB and NFS based storage) if anyone is interested.

Interesting about the ScsiSN thing - I didnt see that readme file and
what I did to solve that problem was use a different Lun number on
each target.. but this solution makes much more sense - thanks for the
tip.

Regards,
Trent

On 21/01/2008, at 9:43 PM, Thomas Glanzmann wrote:

> Hello,
> I just setup a high available iscsi target for an ESX 3.5 server. I
> can
> switch the target between my two hosts without any problems on the ESX
> 3.5 server side with the exception of a reconnect and a hang in the
> virtual machine running on iscsi stoage for ca. 1 second (worst case)
> sometimes I don't even notice.
>
> I used the following components:
>
> - Debian Etch
> - heartbeat 2.1.3
> - drbd 8.1.2 (primary/standby)
> - iscsitarget 0.4.15 (ietd)
>
> I attached the ha.cf, drbd.conf, iscsi.xml (CIB), the ressource
> agent I
> used and of course the ietd.conf.
>
> Attention: If you do that, get sure that you specify a Serial Number
> for
> each LUN you export also see README.vmware in the iscsitarget tarball.
>
> Don't forget to adopt the listening ip address in the ietd ressource
> agent.
>
> Thomas
> <
> ha
> .cf
> >
> <
> iscsi
> .xml
> >
> <
> drbd
> .conf
> ><ietd.txt><ietd.conf>_______________________________________________
> Linux-HA mailing list
> Linux-HA@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
Re: HOWTO: Build a high available iscsi Target using heartbeat, drbd and ietd for ESX Server 3.5 [ In reply to ]
Hello Dejan,

> Nice effort. Thanks for sharing it. Perhaps you'd like to put
> this into the wiki.linux-ha.org. If you do, don't forget to
> pepper the doc with "YMMV."

I am going to do that.

> 1. cib is a bit too lean. There are no attributes set for the ietd
> resource.

Well I have that default ressource agent, which I use for all kind of
scenarios and I just adopt it. The only time I used attributes is when I
had to. That was openvpn (I have two instances running and they use
different config files). But I am going to fix that.

> 2. ietd RA is Linux specific. If it has to be then you should check if
> it runs on Linux and if not bail out with an appropriate message.

It is Linux specific (at least to my knowldege). I will add that error
message.

> 3. I understand that fixing various memory parameters is important for
> ietd performance, but that has no place in the RA. Placing those
> settings in the XML info as comment should suffice. The admins may
> choose different settings anyway.

Actually I don't have a clue. I just looked at the example init script
that was provided by in the ietd distribution and adopted the
information given in there.

> 4. There are various modprobe statements. Is that necessary? It should
> be better to assume that ietd init script has been run and then just
> add/remove new targets using the RA.

I am unaware if that is possible. But most of the time (apache, nfs
server) I work by shutting the whole thing down and start it elsewhere
even if it would be possible to it on a per lun basis.

> 5. Is this RA just an example?

Yes, a working example. But to be honest I have four or three of this
RAs in production.

Thomas
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
Re: HOWTO: Build a high available iscsi Target using heartbeat, drbd and ietd for ESX Server 3.5 [ In reply to ]
Trent,

> I just did a very similar thing, except in my case I am using shared
> storage (MD3000 - SAS) and theres a bit more fun to that part of it
> (multipath, stonith, etc) - also I setup heartbeat in v1 mode not CRM
> mode.

nice, I neve had a MD3000 on my hands.

> I plan to post a walkthrough at some point in the future (I also setup
> SMB and NFS based storage) if anyone is interested.

Well, I am for sure interested. I plan to provide some education on
linux-ha myself, because for me it wasn't that easy to understand the
existing resources.

> Interesting about the ScsiSN thing - I didnt see that readme file and
> what I did to solve that problem was use a different Lun number on
> each target.. but this solution makes much more sense - thanks for
> the tip.

I would always use different LUN numbers, too. However in the future I
am going to use different LUN numbers and different ScsiSNs.

Thomas
_______________________________________________
Linux-HA mailing list
Linux-HA@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems