Posts Tagged ‘dd’

Network cloning disks like a pro

October 20th, 2013 No comments

So you want to clone a hard drive over a network, it’s ok, we’ve all been there. We want quick and easy and preferably some level of integrity insurance.

Boot systemrescuecd on box boxes and make sure both have an IP address and can ping each other. My example IPs are listed below -it’s worth pointing out that you should only do this over a trusted network, there’s no transport level security so potentially someone could be intercepting your stream.


linux treats everything as files, so we can throw an iso image as easily as we can throw a hard drive, eg

nc -lp 1337 | dd of=/tmp/bigIsoFromSource.iso

dd if=./MyBigIso.iso | nc 1337

Nice (nice). Now, you should be running md5sum on the files at both ends to make sure it wasn’t corrupted in transport, but if it’s a big file (or a disk partition) that could take a while. Wouldn’t it be good if we could split the streams (or split the pipe) so we could send the same read data from dd to both netcat and md5sum on the source, and the same on the destination? tee is our friend.

nc -lp 1337 | tee >(md5sum) | dd of=/tmp/bigIsoFromSource.iso

dd if=./MyBigIso.iso | tee >(md5sum) | nc 1337

Cool beans, so now it gets checked on both sides of the network connection. It doesn’t let us see what’s going on though. you can send a `kill -USR1 <ddpid>` to dd’s process ID and it’ll print to stderror how far it’s gotten, but unless you stick it inside of wait it’s not automagically updating. For this we can use pipe viewer, which lets us see the state of data going through a pipe. If you’re feeling clever and know how big the file/partition/disk is, then you can pass the size to pv for even prettier output.

nc -lp 1337 | tee >(md5sum) | pv -s4.5G | dd of=/tmp/bigIsoFromSource.iso

dd if=./MyBigIso.iso | tee >(md5sum) | pv -s4.5G |  nc 1337

and then just to be silly we’re going to compress it and calculate the hashes on both sides for both the raw input/output as well as for the compressed file that gets shoved over the network (hey, cores are cheap, right?)

nc -lp 1337 | tee >(echo -e “compressed: $(md5sum)’\n) | pv -s4.5G | bunzip2 | tee >(echo -e “raw: $(md5sum)’\n) | dd of=/tmp/bigIsoFromSource.iso

dd if=./MyBigIso.iso | tee >(echo -e “raw: $(md5sum)’\n) | pv -s4.5G | bzip2 | tee >(echo -e “compressed: $(md5sum)’\n) |  nc 1337

*note nc under centos wants just the l and no p passing on the destination side, otherwise it errors out. so `nc -lp 13337` fails, but `nc -l 1337`works.

Categories: Linux Tags: , ,

Server Move

September 30th, 2008 No comments

You1 might have noticed the downtime with the server over the last couple of days.

We have been migrating to a new server over the last couple of days, and although it was supposed to be seamless, I fucked up royally.

Read more…


  1. I say you, and I mean You. There’s got to be like 2 people in the entire world reading this, and I doubt both of them noticed.