You1 might have noticed the downtime with the server over the last couple of days.
We have been migrating to a new server over the last couple of days, and although it was supposed to be seamless, I fucked up royally.
The hosting company we use has a very funky vKVM2 web-interface set up. Basically it boots your actual hard drive from your dedicated server into a qEMU virtual machine (a bit like vmware) and then gives you an embedded VNC client to connect to it with. Not only can I boot my hard drive, but I can get it to boot off an .iso3 image. So I can boot a nice minimal live-cd4 distribution which has ssh and dd. Annoyingly not ftp5 but that’s by the by.
So, my big plan. Both servers have (it turns out, exactly) the same hard drives, and I’m a clever linux bonny. I know how to linux, and I especially know how to `dd`. So, using nothing but dd, the pipe tool and an ssh connection, I should be able to clone over my server -no bother at all, and job is done and done.
After working out that the 2008-minimal .iso k-panics, I set to on the second problem of this really simple exercise. Neither of the machines have a real-world6 IP address (in fact they both had the same 10.x.x.x), but they did have internet access. This is the part where I remembered the ftp-backup, and then noticed the lack of an executable.
But we have ssh -both in and out- and that means I have sftp.
Which doesn’t help either. I have no routable address to host a server, and there’s no chance in hell I’m going to transfer an image of a 250GB HDD thought my home internet connection, with its maximum upload of 512k/s. Actually, I can’t transfer squat through my internet connection, but then if people will leave unsecured access points lying around -and Windows7 can’t seem to stop itself automatically connecting to any Tom, Dick or Harry unsecured wireless network it can find, and I can’t seem to stop myself installing VNC and enabling ICS Wizard….
So the internet is a little sketchy.
Turns out I had an active account on a friends web server -thanks newb8 so after realising my auth details still worked (past tense as soon as he reads this) I started thinking about dumping data onto there.
copies everything from hda (your primary master hard drive) to hdb (the primary slave). if= and of= (for those of you that don’t catch on quick) is literally specifying the input and the output file. If its not specified, then dd assumes the data is coming in or out on the pipe.
So, we can be clever, and because linux treats everything as a file9, we can just as easily dump it as an image file, rather than as a straight copy onto another hard disk.
And now we have a file that is our hard drive, very nice.
Except, if my original hard drive is a 250gb hard drive, even if there’s only 30gb in use, the image is still going to be 250gb. Which is the problem in my case. I need compression.
gives me it compressed as a gzip (the -c tells it to output to stdout (the pipe) rather than a file) which is kinda shibby, and truncates all the white space. So instead of our image being 250gb, its now about 30gb. Far more manageable.
The issue now, is that I have nowhere to store, even temporarily, a 30gb file. I’m still not waiting for… this internet to download and then upload the file again, even if I could somehow find space on the server to create it in the first place.
But wait, there’s no bother, I use ssh. So instead of telling dd on the local machine to write the file out, I can just pipe the data through the ssh connection into whatever is running at the other end, which I can make be dd on the remote server, to write it back out again.
How do you like them apples? Pretty sweet, no? And the code snippet even highlights the quotes, which effectivly highlights whats going on on the remote machine.10
It should have worked without any bother. Except it didn’t. The virtual machine (apart from taking hours, and I do mean 14+ hours) kept crashing and the opperation had to be started again from scratch. Many times.
Unsure if its my internet causing the page to time-out and therefore resetting the VNC session, or what-the-fuck, I decide to use ssh and try and remove myself from this as much as possible, so I go for the old reverse-ssh trick, always a winner in those situations where you can get network access going out, but not back in.11
ssh -R 2222:localhost:22 me@newbsBox
## Step 2: SSH from my PC to newbs server, and start a screen session (so when I get disconnected, the session doesn’t drop).
## Step 3: Inside of my screen session, ssh to the localport 2222 and it gets forwarded to port 22 on the new server:
ssh -p 2222 root@localhost
So now, both my machine at home, and the old server connects to newbs server.
From my PC at home, I tell newb’s server to tell my old server to connect again to newbs server and dd the image across as we were doing before, but this time if my home internet goes down, its not going to affect the remote transfer. I think. I’m quite confused now, and I sure as hell was then.
It still took hours, but I finally managed to get a full backup onto his box. Just have to get it back off again now.
Except trying to do the reverse crashes the new server even more. Lots and lots of mad faffing about and it keeps keeling over half way through trying to restore the data. Thrashing my hosting centres network and qemu server is just not working, so onto the next step.
It was at this point, I discovered another setting in the hosting centres control panel called “rescue-mode”, that I had previously dismissed as not being of any help. After I bothered reading documentation, I found out this actually boots my real dedicated server off a netboot image, letting it boot up the box on its own, real, actual hardware, and without touching the actual hard drive. Real-world IP address, too, which is absolutely fab. As with putting it into vKVM mode, you change it as a boot option12 and they email you your user name and password for the temporary mode.
A couple of minutes later, we’ve got the auth details, and I’m logged into my new server, and I can ssh and dd and the disks are offline.
and I’m leaving it to run, no bother, everything is good. When its done, I reboot the box and find out that as I forgot to de-compress the image first, the disk is trashed.
It shouldn’t have been a problem -a quick reboot into rescue mode later and I should have been able to re-run the command (using gunzip -c) and everything would have been fine -except this time when I reboot into rescue-mode, I dont get any auth details emailed through. Still nothing when I go into vKVM mode either, and the previous details have stopped working. After much faffing about rebooting trying to get access to anything I use (once again) the web management page and re-install the OS. 30minutes later I have a mostly up to date linux box and some auth details to access it.
By this time (about two days after I started) I was in no mood to fuck about. I set the new box updating and with a copy of the old world file (on Gentoo all of the packages you have installed are recorded in your world), I told the new server to install all of the same programs onto the new box.
On the old server I remounted the /home partition as read only, so it wouldn’t change and then from the new server ran:
ssh root@OLDserver ‘gzip -c | dd if=/dev/sda7’ | gunzip -c | dd of=/dev/sda7
and waited. After transferring a copy of the SQL databases, the apache vhosts information and updating the DNS, here we are, working as new.
And may I say, what a pain in the fucking arse. I’m smoking a d00b and then getting into bed with my bf -its nearly 7am and we have to be up in 5 hours.
-I still maintain its better than coding css tho :S
- I say you, and I mean You. There’s got to be like 2 people in the entire world reading this, and I doubt both of them noticed. ↩
- Virtual Keyboard, Video, Mouse. KVM Switches are used usually to allow you to control multiple computers with a single monitor, keyboard, mou… you get the idea. This one is virtual. ↩
- I’m getting sick of trying to write footnotes and give links. If you’re bright enough to read this, then you’re bright enough to google (that’s two ‘o’s…) ↩
- They now call it the Install CD. Its <60mb, it gives you a nice, useful CLI -that means no X and no pretty graphics, well, framebuffer looks pretty, but you should be booting `gentoo-nofb` in vmware.
Did you notice I pointed you to the 2007 version? The 2008 kernel panics in qEMU. ↩
- I get ftp backup as part of the hosting. Firstly, what a stupid way of doing it, and secondly, what freaking liveCD doesn’t have a copy of `ftp` on it? I know I can use wget, but its no good for putting…. ↩
- as in a non-RFC 1918 ip addess. Yes people do actually read these things -And yes I cited this before I google’d for a link for you whores. ↩
- Yeah, I could have used linux. But its a box I was given along with the nasty and odd wireless USB mess and I like using my linux box. I don’t want it sat underneath the window in a headless state. ↩
- I’m not belittling him; its what he calls himself on his own IRC server ↩
- yes, I know, but please shut up. 99% of the freaking time…. ↩
- That strikes me as horrible, on on and all that. Suggestions please. ↩
- with thanks to brandonhutchinson.com who makes more sense than the man-page ↩
- It seems that all of the machines netboot grub and then use that to choose what they should actually be booting (my kernel on the HDD, one of their kernels via the network or into vKVM/rescue-mode), So I can change what the machine boots via a management webpage without changing my bootloader settings on the machine. I can even power-cycle the hardware via the webpage, which is handy ↩