Mirroring over network
From FBSD_tips
Why I don't need iSCSI, or The GEOM gate class (and attendant utilities) is a wonderful thing.
Contents |
[edit] Rationale and goals
I have been gmirroring all the machines I run that I can't justify putting a "real" (scsi hardware raid) disk mirroring system into for a while now. I find myself wishing that I could do the same for my laptops and/or machines that for one reason or another I cannot fit more than one drive into. Enter Ggated. Ggated exports a device onto the network so that a Ggated client can see it as if it is,for all intents and purposes local. I intend to determine if this is a workable backup for failure of the hard drive in a single drive machine.
[edit] Test hardware
To test this I took 2 seagate PATA-100 drives and put one in an IBM Thinkcenter (system details here) and the other in a PowerSpec (system details here). I installed 6.2-Rel on the ThinkCenter (I had to update it to -p8 because of a bug in ggatec, then it worked fine). The switch was a Cisco 2950 24 port 10/100. The NIC in the ThinkCenter uses the fxp driver (this is good), the NIC in the PowerSpec uses rl (this is bad, usually). Both drives are the same make and model, PATA-100, both cabled singlely and jumpered master.
[edit] Convert the client
Now we will convert the installation to a gmirror after booting to single user mode, like this. First we make it so the kernel will allow us to write the Gmirror metadata to the mounted filesystem with the debugflags system oid. Then we create the new mirror, I name it 'thinkc' to avoid name collisions on the PowerSpec later when it sees the metadata written on the remotely synced device.
# sysctl kern.geom.debugflags=16 # gmirror label -v -b round-robin thinkc /dev/ad0
For the purposes of being neat, and even tho we are going to reboot shortly it is a good practice to put the 'shoot yourself in the foot' option we just set back to 'safe' like this :
# sysctl kern.geom.debugflags=0
We then add 'geom_mirror_load="YES"' to /etc/loader.conf and change all occurances of /dev/ad0 to /dev/mirror/thinkc in /etc/fstab and reboot. Now the ThinkCenter is running on a mirror with only one drive in it that should look like this.
# gmirror list Geom name: thinkc State: COMPLETE Components: 1 Balance: round-robin Slice: 4096 Flags: NONE GenID: 0 SyncID: 1 ID: 387519943 Providers: 1. Name: mirror/thinkc Mediasize: 100030242304 (93G) Sectorsize: 512 Mode: r4w4e5 Consumers: 1. Name: ad0 Mediasize: 100030242816 (93G) Sectorsize: 512 Mode: r1w1e1 State: ACTIVE Priority: 0 Flags: NONE GenID: 0 SyncID: 1 ID: 4018930450
[edit] Set up and connect ggated
Now on the PowerSpec we will set up Ggated. First, make a config file, /etc/ggated.conf with this in it, where 'thinkc' resolves to the IP of tyhe Geom gate client.
thinkc RW /dev/ad10
Then we run the ggated exe with that file as it's argument.
# ggated /etc/ggated.conf
Now on the ThinkCenter, we have to set up the ggated client and insert the remote device into the gmirror and write some starting and ending time files.
# ggatec create -o rw 10.200.1.127 /dev/ad10 ggate0
[edit] Mirror the client
# gmirror insert thinkc ggate0; date > start.txt # while gmirror status | grep DEGRADED > /dev/null ; do echo > /dev/null; sleep 1; done; date > ended.txt
Once the mirror id complete you will see this.
# gmirror status
Name Status Components
mirror/thinkc COMPLETE ad0
ggate0
And we will check the start and end times. The sync ended up taking about 2 1/2 hours.
# cat start.txt Sun Oct 14 17:49:01 EDT 2007 # cat ended.txt Sun Oct 14 20:18:21 EDT 2007
The time to complete the network backed mirroring job ended up being about 2 1/2 hours for the 100 GB drive. During the sync, bmon on the client showed something close to this the whole time.
.khome.utcorp.net. (local) │ Rate # % │ Rate # %
0 fxp0 │ 265.90KiB 4.01K │ 11.23MiB 7.73K
1 lo0 │ 0 B 0 │ 0 B 0
Total │ 265.90KiB 4.01K │ 11.23MiB 7.73K
RX KiB [Bytes] (sel)
359.03 ........................*...................................
299.19 ........................*...................................
239.35 ***********************.************************************
179.52 ************************************************************
119.68 ************************************************************
59.84 ************************************************************ [-0.15%]
1 5 10 15 20 25 30 35 40 45 50 55 60 s
TX MiB
15.05 ........................*...................................
12.54 ........................*...................................
10.03 **.********************.************************************
7.52 **.*********************************************************
5.02 **.*********************************************************
2.51 **.********************************************************* [-0.15%]
1 5 10 15 20 25 30 35 40 45 50 55 60 s
RX TX │ RX TX
Bytes 191.59MiB 4.02GiB │ Packets 2.87M 5.53M
Errors 0 0 │ Dropped N/A 0
Multicast 40 N/A │ Collisions N/A 0
And gstat on the ggated server show this.
dT: 0.501s w: 0.500s filter: ad10
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
0 86 0 0 0.0 86 10989 2.1 17.8| ad10
[edit] Swap drives
So now that we have a Gmirror split across 2 machines, we can swap the 2 drives and boot from the 'backup' in the Ggated server. You will the the IDs of the mirror consumers swap.
On the think center :
# gmirror list Geom name: thinkc State: DEGRADED Components: 2 Balance: round-robin Slice: 4096 Flags: NONE GenID: 0 SyncID: 2 ID: 387519943 Providers: 1. Name: mirror/thinkc Mediasize: 100030242304 (93G) Sectorsize: 512 Mode: r4w4e5 Consumers: 1. Name: ad0 Mediasize: 100030242816 (93G) Sectorsize: 512 Mode: r1w1e1 State: ACTIVE Priority: 0 Flags: DIRTY GenID: 0 SyncID: 2 ID: 3192624547
On the Powerspec :
# gmirror list Geom name: thinkc State: DEGRADED Components: 2 Balance: round-robin Slice: 4096 Flags: NONE GenID: 0 SyncID: 1 ID: 387519943 Providers: 1. Name: mirror/thinkc Mediasize: 100030242304 (93G) Sectorsize: 512 Mode: r0w0e0 Consumers: 1. Name: ad10 Mediasize: 100030242816 (93G) Sectorsize: 512 Mode: r1w1e1 State: ACTIVE Priority: 0 Flags: NONE GenID: 0 SyncID: 1 ID: 4018930450
[edit] Local performance
To compare the performance of the Ggated backed Gmirror sync to a local sync, I put both drives into the PowerSpec and loaded geom_mirror. Gstat looked like this during the sync.
> gstat
dT: 0.501s w: 0.500s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
2 313 0 0 0.0 313 39591 3.7 97.0| ad10
0 309 309 39589 2.7 0 0 0.0 81.4| mirror/thinkc
0 0 0 0 0.0 0 0 0.0 0.0| mirror/thinkcs1
0 309 309 39589 2.6 0 0 0.0 79.6| ad4
The local sync of the 100 GB took about 40 minutes, or roughly 4::1, which the gstat numbers seem in line with too.
# cat start.txt ended.txt Mon Oct 15 13:42:42 UTC 2007 Mon Oct 15 14:22:57 UTC 2007
[edit] Discussion
Now, these are not cutting edge machines, obviously. The issue I feel is the most critical for performance would be the disk system. The other critical issue is the network: high quality NICs and switch/other medium. Still the performance is quite within the boundaries for many purposes. This is nearly perfect for a laprop HD failure scenerio, e.g a nightly gmirror of the laptop to an identical drive in a server and a simple swap in the event of a failure. It also bears noting that gmirrors can be comprised of groups of drives greater than 2, you could have backup of already redundant systems in circumstance dictated the need (adding a ggated backed 3rd drive to a gmirrored pair).
I had some problems with sporatic hanging, I queried the GEOM list and got this response :
Date: Wed, 17 Oct 2007 22:15:31 +0200
From: Pawel Jakub Dawidek <pjd@FreeBSD.org>
To: kurtseel <kurtseel@primetime.com>
Cc: freebsd-geom@freebsd.org
Subject: Re: gmirror + ggated question
Message-ID: <20071017201531.GE50219@garage.freebsd.pl>
In-Reply-To: <471650AA.30903@primetime.com>
References: <471650AA.30903@primetime.com>
Next in thread | Previous in thread | Raw E-Mail | Index | Archive | Help
--C94crkcyjafcjHxo
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Wed, Oct 17, 2007 at 02:12:58PM -0400, kurtseel wrote:
> I built a mirror of a local drive and a ggated backed device. I ran
> iozone on it
> and it runs along fine until a certain point when it slows down to a
> near stand
> still. It doesn't break the mirror or crash the system, but it does slow
> the system
> down to a near stop.
You haven't said which FreeBSD version you use. If it's not HEAD nor
RELENG_7, try this patch:
http://www.freebsd.org/cgi/cvsweb.cgi/src/sbin/ggate/shared/ggate.c.diff?r=1=3D1.8;r2=3D1.9
Pawel Jakub Dawidek http://www.wheel.pl
pjd@FreeBSD.org http://www.FreeBSD.org
FreeBSD committer Am I Evil? Yes, I Am!
And here is the patch :
src/sbin/ggate/shared/ggate.c 2006/12/18 11:12:00 1.8
+++ src/sbin/ggate/shared/ggate.c 2007/04/06 11:19:48 1.9
@@ -23,7 +23,7 @@
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
- * $FreeBSD: /usr/local/www/cvsroot/FreeBSD/src/sbin/ggate/shared/ggate.c,v 1.8 2006/12/18 11:12:00 pjd Exp $
+ * $FreeBSD: /usr/local/www/cvsroot/FreeBSD/src/sbin/ggate/shared/ggate.c,v 1.9 2007/04/06 11:19:48 pjd Exp $
*/
#include <stdio.h>
@@ -222,6 +222,16 @@ g_gate_load_module(void)
}
}
+/*
+ * When we send from ggatec packets larger than 32kB, performance drops
+ * significantly (eg. to 256kB/s over 1Gbit/s link). This is not a problem
+ * when data is send from ggated. I don't know why, so for now I limit
+ * size of packets send from ggatec to 32kB by defining MAX_SEND_SIZE
+ * in ggatec Makefile.
+ */
+#ifndef MAX_SEND_SIZE
+#define MAX_SEND_SIZE MAXPHYS
+#endif
ssize_t
g_gate_send(int s, const void *buf, size_t len, int flags)
{
@@ -229,7 +239,7 @@ g_gate_send(int s, const void *buf, size
const unsigned char *p = buf;
while (len > 0) {
- done2 = send(s, p, len, flags);
+ done2 = send(s, p, MIN(len, MAX_SEND_SIZE), flags);
if (done2 == 0)
break;
else if (done2 == -1) {
I was using 6.2-REL so I applied the patch and it seemed to help.
Gongo 22:18, 15 October 2007 (UTC)
