Mailing List Archive

drbd hanging on file write
Hi all,

I just set up a drbd replication pair and shipped it to the colo. It was
working great.

Then we upgraded the kernel to 2.6.15-1.1833_FC4smp. I rebuilt the
kernel modules on one machine, rebooted, connected and became primary.
Then I rebooted the other machine into 2.6.15 so I could build its new
modules. While I was waiting for that, I started transferring some data
onto the drbd partition (fs already there, had been working great) by
untarring. It had gone for a little bit when tar hung. I couldn't kill
it. I figured it would timeout. It didn't. Also no errors in
/var/log/messages. The other machine couldn't connect at this point, so
I figured I had hosed something and that I would just start with what
should be a pristine copy of the data on the other machine and make the
bad machine resync.

I power-cycled both machines, made the good machine primary, ran an fsck
on the fs, mounted it. The bad machine connected and started to resync.
It all looked as if it was fine. Yay! So then i realized that I had the
network throttling down way too low (it's a 20G partition) and I needed
to restart the drbd. So I disconnected the secondary. And unmounted the
filesystem on the primary. At least I tried to. The umount failed (I had
modified one file) And now the primary machine is hung again.

So I've rebooted the secondary system back into 2.6.11, and I'm going to
power cycle the primary again. Any ideas as to what's going on?

Thanks!
Monty Taylor
Senior Consultant, MySQL, Inc.
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: drbd hanging on file write [ In reply to ]
> I just set up a drbd replication pair and shipped it to the colo. It was working great.

so you did local tests first, and all was working as expected?

> Then we upgraded the kernel to 2.6.15-1.1833_FC4smp. I rebuilt the
> kernel modules on one machine, rebooted, connected and became primary.
> Then I rebooted the other machine into 2.6.15 so I could build its new
> modules. While I was waiting for that, I started transferring some
> data onto the drbd partition (fs already there, had been working
> great) by untarring. It had gone for a little bit when tar hung. I
> couldn't kill it. I figured it would timeout. It didn't. Also no
> errors in /var/log/messages. The other machine couldn't connect at
> this point, so I figured I had hosed something and that I would just
> start with what should be a pristine copy of the data on the other
> machine and make the bad machine resync.
>
> I power-cycled both machines, made the good machine primary, ran an
> fsck on the fs, mounted it. The bad machine connected and started to
> resync. It all looked as if it was fine. Yay! So then i realized that
> I had the network throttling down way too low (it's a 20G partition)
> and I needed to restart the drbd. So I disconnected the secondary. And
> unmounted the filesystem on the primary. At least I tried to. The
> umount failed (I had modified one file) And now the primary machine is
> hung again.

> So I've rebooted the secondary system back into 2.6.11, and I'm going
> to power cycle the primary again. Any ideas as to what's going on?

now. your report is somewhat unspecific.
anyways, when I read your last sentence "machine is hung",
this might point to a deadlock that could occur when stressing the box.

this possible deadlock is due to a bio_alloc(,GFP_KERNEL) in drbd where
is should have been GFP_NOIO, and has been recognized and fixed just
after we released 0.7.17.

may I ask you to try again with recent drbd svn?
svn co http://svn.drbd.org/drbd/branches/drbd-0.7
revision 2111 and greater should contain that fix.
there may be a 0.7.18 bugfix release because of that.

please report your findings.

thanks,

--
: Lars Ellenberg Tel +43-1-8178292-0 :
: LINBIT Information Technologies GmbH Fax +43-1-8178292-82 :
: Schoenbrunner Str. 244, A-1120 Vienna/Europe http://www.linbit.com :
__
please use the "List-Reply" function of your email client.
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: drbd hanging on file write [ In reply to ]
> I just set up a drbd replication pair and shipped it to the colo. It was working great.

so you did local tests first, and all was working as expected?

> Then we upgraded the kernel to 2.6.15-1.1833_FC4smp. I rebuilt the
> kernel modules on one machine, rebooted, connected and became primary.
> Then I rebooted the other machine into 2.6.15 so I could build its new
> modules. While I was waiting for that, I started transferring some
> data onto the drbd partition (fs already there, had been working
> great) by untarring. It had gone for a little bit when tar hung. I
> couldn't kill it. I figured it would timeout. It didn't. Also no
> errors in /var/log/messages. The other machine couldn't connect at
> this point, so I figured I had hosed something and that I would just
> start with what should be a pristine copy of the data on the other
> machine and make the bad machine resync.
>
> I power-cycled both machines, made the good machine primary, ran an
> fsck on the fs, mounted it. The bad machine connected and started to
> resync. It all looked as if it was fine. Yay! So then i realized that
> I had the network throttling down way too low (it's a 20G partition)
> and I needed to restart the drbd. So I disconnected the secondary. And
> unmounted the filesystem on the primary. At least I tried to. The
> umount failed (I had modified one file) And now the primary machine is
> hung again.

> So I've rebooted the secondary system back into 2.6.11, and I'm going
> to power cycle the primary again. Any ideas as to what's going on?

now. your report is somewhat unspecific.
anyways, when I read your last sentence "machine is hung",
this might point to a deadlock that could occur when stressing the box.

this possible deadlock is due to a bio_alloc(,GFP_KERNEL) in drbd where
is should have been GFP_NOIO, and has been recognized and fixed just
after we released 0.7.17.

may I ask you to try again with recent drbd svn?
svn co http://svn.drbd.org/drbd/branches/drbd-0.7
revision 2111 and greater should contain that fix.
there may be a 0.7.18 bugfix release because of that.

please report your findings.

thanks,

--
: Lars Ellenberg Tel +43-1-8178292-0 :
: LINBIT Information Technologies GmbH Fax +43-1-8178292-82 :
: Schoenbrunner Str. 244, A-1120 Vienna/Europe http://www.linbit.com :
__
please use the "List-Reply" function of your email client.
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user



_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: drbd hanging on file write [ In reply to ]
Lars Ellenberg wrote:
>> I just set up a drbd replication pair and shipped it to the colo. It was working great.
>
> so you did local tests first, and all was working as expected?

Yes.

> now. your report is somewhat unspecific.
> anyways, when I read your last sentence "machine is hung",
> this might point to a deadlock that could occur when stressing the box.

Fair enough. I think the hang was that block device was busy and
couldn't be unmounted, and so when I tried to shutdown the machine it
blocked waiting for the fs to unmount.

> this possible deadlock is due to a bio_alloc(,GFP_KERNEL) in drbd where
> is should have been GFP_NOIO, and has been recognized and fixed just
> after we released 0.7.17.
>
> may I ask you to try again with recent drbd svn?
> svn co http://svn.drbd.org/drbd/branches/drbd-0.7
> revision 2111 and greater should contain that fix.
> there may be a 0.7.18 bugfix release because of that.

I'll do that when I get the next set of test machines up. I downgraded
the kernel back to 2.6.11 for now on these boxes and everything works as
expected again.

> please report your findings.

I will when I get another test machine and 2.6.15 again, which should be
next week. If there's a possible known deadlock, I bet that's what I ran
into. On the other hand, is the default value for on-disconnect
reconnect or freeze_io? Because if it's freeze_io I would maybe see that
being what happened, too.

I'll keep trying to isolate for you. I think we're going to be using
drbd a bit more as part of our Professional Services offerings to some
clients, so it'll be nice to know where the problem actually sits.

Thanks!
Monty Taylor
Senior Consultant, MySQL, Inc.
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: drbd hanging on file write [ In reply to ]
A small comment on your upgrade process...

> Then I rebooted the other machine into 2.6.15 so I could build its new
> modules. While I was waiting for that, I started transferring some data

You do not need to reboot a machine to build the drbd modules for a new kernel. First install your
new kernel and then build drbd as shown below:

cd /usr/src
tar -xzf /path/to/drbd/source/drbd-0.7.17.tar.gz
cd drbd-0.7.17
make rpm KDIR=/usr/src/kernels/2.6.9-34.EL-x86_64
make rpm KDIR=/usr/src/kernels/2.6.9-34.EL-smp-x86_64

Adjust kernel path as needed.

You can then stop heartbeat and drbd, install the new rpms:

rpm -Fvh --force dist/RPMS/x86_64/drbd-0.7.17-1.x86_64.rpm
rpm -Fvh --force dist/RPMS/x86_64/drbd-debuginfo-0.7.14-1.x86_64.rpm

NOTE: --force only needed if using same drbd version but only upgrading kernel.

Now install the kernel modules

rpm -ivh dist/RPMS/x86_64/drbd-km*.rpm

Then reboot and your machine will come up ready.

Diego
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: drbd hanging on file write [ In reply to ]
/ 2006-03-16 14:04:41 -0500
\ Monty Taylor:
> Lars Ellenberg wrote:
> >> I just set up a drbd replication pair and shipped it to the colo.
> >> It was working great.
> > so you did local tests first, and all was working as expected?
>
> Yes.
>
> >now. your report is somewhat unspecific.
> >anyways, when I read your last sentence "machine is hung",
> >this might point to a deadlock that could occur when stressing the box.
>
> Fair enough. I think the hang was that block device was busy and
> couldn't be unmounted, and so when I tried to shutdown the machine it
> blocked waiting for the fs to unmount.

> >please report your findings.
>
> I will when I get another test machine and 2.6.15 again, which should
> be next week. If there's a possible known deadlock, I bet that's what
> I ran into. On the other hand, is the default value for on-disconnect
> reconnect or freeze_io? Because if it's freeze_io I would maybe see
> that being what happened, too.

freeze_io is mentioned, but not yet configurable.
the kernel side implementation is missing from drbd 0.7 ...
default would be reconnect.

> I'll keep trying to isolate for you. I think we're going to be using
> drbd a bit more as part of our Professional Services offerings to some
> clients, so it'll be nice to know where the problem actually sits.

sounds interessting!

cheers,

--
: Lars Ellenberg Tel +43-1-8178292-0 :
: LINBIT Information Technologies GmbH Fax +43-1-8178292-82 :
: Schoenbrunner Str. 244, A-1120 Vienna/Europe http://www.linbit.com :
__
please use the "List-Reply" function of your email client.
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: drbd hanging on file write [ In reply to ]
/ 2006-03-16 14:55:48 -0500
\ Diego Julian Remolina:
> A small comment on your upgrade process...
>
> >Then I rebooted the other machine into 2.6.15 so I could build its
> >new modules. While I was waiting for that, I started transferring
> >some data
>
> You do not need to reboot a machine to build the drbd modules for a
> new kernel. First install your new kernel and then build drbd as
> shown below:
>
> cd /usr/src
> tar -xzf /path/to/drbd/source/drbd-0.7.17.tar.gz
> cd drbd-0.7.17
> make rpm KDIR=/usr/src/kernels/2.6.9-34.EL-x86_64
> make rpm KDIR=/usr/src/kernels/2.6.9-34.EL-smp-x86_64
>
> Adjust kernel path as needed.
>
> You can then stop heartbeat and drbd, install the new rpms:
>
> rpm -Fvh --force dist/RPMS/x86_64/drbd-0.7.17-1.x86_64.rpm
> rpm -Fvh --force dist/RPMS/x86_64/drbd-debuginfo-0.7.14-1.x86_64.rpm
>
> NOTE: --force only needed if using same drbd version but only upgrading kernel.

since these are the userspace things, you do not need to re-install these.
you just need to put the new kernel specific module in place:

> Now install the kernel modules
>
> rpm -ivh dist/RPMS/x86_64/drbd-km*.rpm

--
: Lars Ellenberg Tel +43-1-8178292-0 :
: LINBIT Information Technologies GmbH Fax +43-1-8178292-82 :
: Schoenbrunner Str. 244, A-1120 Vienna/Europe http://www.linbit.com :
__
please use the "List-Reply" function of your email client.
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: drbd hanging on file write [ In reply to ]
/ 2006-03-16 14:04:41 -0500
\ Monty Taylor:
> Lars Ellenberg wrote:
> >> I just set up a drbd replication pair and shipped it to the colo.
> >> It was working great.
> > so you did local tests first, and all was working as expected?
>
> Yes.
>
> >now. your report is somewhat unspecific.
> >anyways, when I read your last sentence "machine is hung",
> >this might point to a deadlock that could occur when stressing the box.
>
> Fair enough. I think the hang was that block device was busy and
> couldn't be unmounted, and so when I tried to shutdown the machine it
> blocked waiting for the fs to unmount.

> >please report your findings.
>
> I will when I get another test machine and 2.6.15 again, which should
> be next week. If there's a possible known deadlock, I bet that's what
> I ran into. On the other hand, is the default value for on-disconnect
> reconnect or freeze_io? Because if it's freeze_io I would maybe see
> that being what happened, too.

freeze_io is mentioned, but not yet configurable.
the kernel side implementation is missing from drbd 0.7 ...
default would be reconnect.

> I'll keep trying to isolate for you. I think we're going to be using
> drbd a bit more as part of our Professional Services offerings to some
> clients, so it'll be nice to know where the problem actually sits.

sounds interessting!

cheers,

--
: Lars Ellenberg Tel +43-1-8178292-0 :
: LINBIT Information Technologies GmbH Fax +43-1-8178292-82 :
: Schoenbrunner Str. 244, A-1120 Vienna/Europe http://www.linbit.com :
__
please use the "List-Reply" function of your email client.
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user



_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: drbd hanging on file write [ In reply to ]
/ 2006-03-16 14:55:48 -0500
\ Diego Julian Remolina:
> A small comment on your upgrade process...
>
> >Then I rebooted the other machine into 2.6.15 so I could build its
> >new modules. While I was waiting for that, I started transferring
> >some data
>
> You do not need to reboot a machine to build the drbd modules for a
> new kernel. First install your new kernel and then build drbd as
> shown below:
>
> cd /usr/src
> tar -xzf /path/to/drbd/source/drbd-0.7.17.tar.gz
> cd drbd-0.7.17
> make rpm KDIR=/usr/src/kernels/2.6.9-34.EL-x86_64
> make rpm KDIR=/usr/src/kernels/2.6.9-34.EL-smp-x86_64
>
> Adjust kernel path as needed.
>
> You can then stop heartbeat and drbd, install the new rpms:
>
> rpm -Fvh --force dist/RPMS/x86_64/drbd-0.7.17-1.x86_64.rpm
> rpm -Fvh --force dist/RPMS/x86_64/drbd-debuginfo-0.7.14-1.x86_64.rpm
>
> NOTE: --force only needed if using same drbd version but only upgrading kernel.

since these are the userspace things, you do not need to re-install these.
you just need to put the new kernel specific module in place:

> Now install the kernel modules
>
> rpm -ivh dist/RPMS/x86_64/drbd-km*.rpm

--
: Lars Ellenberg Tel +43-1-8178292-0 :
: LINBIT Information Technologies GmbH Fax +43-1-8178292-82 :
: Schoenbrunner Str. 244, A-1120 Vienna/Europe http://www.linbit.com :
__
please use the "List-Reply" function of your email client.
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user


_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: drbd hanging on file write [ In reply to ]
<mailto:drbd-user@lists.linbit.com>in continue of .....
http://www.gossamer-threads.com/lists/drbd/users/9891#9896
<mailto:drbd-user@lists.linbit.com>

also fedora core 4 2.6.15-1.1833_FC4smp
drbd hangs when second node offline.

will get the same problem next way:
#drbdadm primary all
#drbdadm disconnect all
# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2111 build by xxx@fff.fff.fff, 2002-01-04 17:49:37
0: cs:StandAlone st:Primary/Unknown ld:Consistent
ns:107634 nr:0 dw:38003 dr:87491 al:35 bm:10 lo:0 pe:0 ua:0 ap:0


but if i'm try do anything (any writings) system hang (all processes
that try writing go in state D)

may try do next:
#mkfs /dev/drbd0
and it's hangs ...

# drbdadm dump
global {
minor-count 1;
}

resource HAData {
protocol C;
incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep
60 ; halt -f";
on s02.flora.lv {
device /dev/drbd0;
disk /dev/sdb;
address 10.0.2.2:7788;
meta-disk internal;
}
on s01.flora.lv {
device /dev/drbd0;
disk /dev/md0;
address 10.0.2.1:7788;
meta-disk internal;
}
net {
timeout 30;
ping-int 5;
max-buffers 2048;
max-epoch-size 2048;
ko-count 4;
on-disconnect reconnect;
}
disk {
on-io-error detach;
size 134G;
}
syncer {
rate 60M;
group 1;
al-extents 127;
}
startup {
wfc-timeout 15;
degr-wfc-timeout 60;
}
}


_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: drbd hanging on file write [ In reply to ]
in continue of .....
<mailto:drbd-user@lists.linbit.com>http://www.gossamer-threads.com/lists/drbd/users/9891#9896
<mailto:drbd-user@lists.linbit.com>

also fedora core 4 2.6.15-1.1833_FC4smp
drbd hangs when second node offline.

will get the same problem next way:
#drbdadm primary all
#drbdadm disconnect all
# cat /proc/drbd
version: 0.7.17 (api:77/proto:74)
SVN Revision: 2111 build by xxx@fff.fff.fff, 2002-01-04 17:49:37
0: cs:StandAlone st:Primary/Unknown ld:Consistent
ns:107634 nr:0 dw:38003 dr:87491 al:35 bm:10 lo:0 pe:0 ua:0 ap:0


but if i'm try do anything (any writings) system hang (all processes
that try writing go in state D)

may try do next:
#mkfs /dev/drbd0
and it's hangs ...

# drbdadm dump
global {
minor-count 1;
}

resource HAData {
protocol C;
incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep
60 ; halt -f";
on s02.flora.lv {
device /dev/drbd0;
disk /dev/sdb;
address 10.0.2.2:7788;
meta-disk internal;
}
on s01.flora.lv {
device /dev/drbd0;
disk /dev/md0;
address 10.0.2.1:7788;
meta-disk internal;
}
net {
timeout 30;
ping-int 5;
max-buffers 2048;
max-epoch-size 2048;
ko-count 4;
on-disconnect reconnect;
}
disk {
on-io-error detach;
size 134G;
}
syncer {
rate 60M;
group 1;
al-extents 127;
}
startup {
wfc-timeout 15;
degr-wfc-timeout 60;
}
}


_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: Re: drbd hanging on file write [ In reply to ]
Am Freitag, 24. März 2006 11:06 schrieb Vadim:
> <mailto:drbd-user@lists.linbit.com>in continue of .....
> http://www.gossamer-threads.com/lists/drbd/users/9891#9896
> <mailto:drbd-user@lists.linbit.com>
>
> also fedora core 4 2.6.15-1.1833_FC4smp
> drbd hangs when second node offline.

And when the other node is online, and DRBD is connected, then
everything is fine ?

[. Slowly I am developing an antipathy against these FC4 kernels ]

-Phil
--
: Dipl-Ing Philipp Reisner Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH Fax +43-1-8178292-82 :
: Schönbrunnerstr 244, 1120 Vienna, Austria http://www.linbit.com :
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: drbd hanging on file write [ In reply to ]
Philipp Reisner wrote:
> And when the other node is online, and DRBD is connected, then
> everything is fine ?
yes. works fine, but if second node go offline - first will stop normal
working after any writes to that and possible result broken filesystem ...


also situation:
one node restarting. if it's do simple restart then drbd recconects and
syncs and works normally, but if node will be off and write occurs on
first node and second will not be available - then state "D" of writing
processes is guaranteed, and also read processes will stop respond :(
/that situtation i'm got if 5 minutes make work in single mode on second
node ... without networking/

_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: Re: drbd hanging on file write [ In reply to ]
/ 2006-03-24 13:30:30 +0200
\ DaLiV:
> Philipp Reisner wrote:
> >And when the other node is online, and DRBD is connected, then
> >everything is fine ?
> yes. works fine, but if second node go offline - first will stop
> normal working after any writes to that and possible result broken
> filesystem ...

well.
We'll try to reproduce this here locally as soon as time permits.

until then, please use a kernel that works,
not some funny FC4 2.6.something-<smallnumber>.<bignumber>
:-/

in case you manage to reproduce this with a kernel.org kernel
and drbd 0.7.17 svn revison >= 2111, please scream out loud.

--
: Lars Ellenberg Tel +43-1-8178292-0 :
: LINBIT Information Technologies GmbH Fax +43-1-8178292-82 :
: Schoenbrunner Str. 244, A-1120 Vienna/Europe http://www.linbit.com :
__
please use the "List-Reply" function of your email client.
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: drbd hanging on file write [ In reply to ]
Am Freitag, 24. März 2006 11:06 schrieb Vadim:
> <mailto:drbd-user@lists.linbit.com>in continue of .....
> http://www.gossamer-threads.com/lists/drbd/users/9891#9896
> <mailto:drbd-user@lists.linbit.com>
>
> also fedora core 4 2.6.15-1.1833_FC4smp
> drbd hangs when second node offline.

And when the other node is online, and DRBD is connected, then
everything is fine ?

[. Slowly I am developing an antipathy against these FC4 kernels ]

-Phil
--
: Dipl-Ing Philipp Reisner Tel +43-1-8178292-50 :
: LINBIT Information Technologies GmbH Fax +43-1-8178292-82 :
: Schönbrunnerstr 244, 1120 Vienna, Austria http://www.linbit.com :
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
Re: drbd hanging on file write [ In reply to ]
/ 2006-03-24 13:30:30 +0200
\ DaLiV:
> Philipp Reisner wrote:
> >And when the other node is online, and DRBD is connected, then
> >everything is fine ?
> yes. works fine, but if second node go offline - first will stop
> normal working after any writes to that and possible result broken
> filesystem ...

well.
We'll try to reproduce this here locally as soon as time permits.

until then, please use a kernel that works,
not some funny FC4 2.6.something-<smallnumber>.<bignumber>
:-/

in case you manage to reproduce this with a kernel.org kernel
and drbd 0.7.17 svn revison >= 2111, please scream out loud.

--
: Lars Ellenberg Tel +43-1-8178292-0 :
: LINBIT Information Technologies GmbH Fax +43-1-8178292-82 :
: Schoenbrunner Str. 244, A-1120 Vienna/Europe http://www.linbit.com :
__
please use the "List-Reply" function of your email client.
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user



_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user