Archive

Posts Tagged ‘pv’

Network cloning disks like a pro

October 20th, 2013 No comments

So you want to clone a hard drive over a network, it’s ok, we’ve all been there. We want quick and easy and preferably some level of integrity insurance.

Boot systemrescuecd on box boxes and make sure both have an IP address and can ping each other. My example IPs are listed below -it’s worth pointing out that you should only do this over a trusted network, there’s no transport level security so potentially someone could be intercepting your stream.

Source: 192.168.0.5
Destination: 192.168.0.10

linux treats everything as files, so we can throw an iso image as easily as we can throw a hard drive, eg

#destination
nc -lp 1337 | dd of=/tmp/bigIsoFromSource.iso

#source
dd if=./MyBigIso.iso | nc 192.168.0.10 1337

Nice (nice). Now, you should be running md5sum on the files at both ends to make sure it wasn’t corrupted in transport, but if it’s a big file (or a disk partition) that could take a while. Wouldn’t it be good if we could split the streams (or split the pipe) so we could send the same read data from dd to both netcat and md5sum on the source, and the same on the destination? tee is our friend.

#destination
nc -lp 1337 | tee >(md5sum) | dd of=/tmp/bigIsoFromSource.iso

#source
dd if=./MyBigIso.iso | tee >(md5sum) | nc 192.168.0.10 1337

Cool beans, so now it gets checked on both sides of the network connection. It doesn’t let us see what’s going on though. you can send a `kill -USR1 <ddpid>` to dd’s process ID and it’ll print to stderror how far it’s gotten, but unless you stick it inside of wait it’s not automagically updating. For this we can use pipe viewer, which lets us see the state of data going through a pipe. If you’re feeling clever and know how big the file/partition/disk is, then you can pass the size to pv for even prettier output.

#destination
nc -lp 1337 | tee >(md5sum) | pv -s4.5G | dd of=/tmp/bigIsoFromSource.iso

#source
dd if=./MyBigIso.iso | tee >(md5sum) | pv -s4.5G |  nc 192.168.0.10 1337

and then just to be silly we’re going to compress it and calculate the hashes on both sides for both the raw input/output as well as for the compressed file that gets shoved over the network (hey, cores are cheap, right?)

#destination
nc -lp 1337 | tee >(echo -e “compressed: $(md5sum)’\n) | pv -s4.5G | bunzip2 | tee >(echo -e “raw: $(md5sum)’\n) | dd of=/tmp/bigIsoFromSource.iso

#source
dd if=./MyBigIso.iso | tee >(echo -e “raw: $(md5sum)’\n) | pv -s4.5G | bzip2 | tee >(echo -e “compressed: $(md5sum)’\n) |  nc 192.168.0.10 1337

*note nc under centos wants just the l and no p passing on the destination side, otherwise it errors out. so `nc -lp 13337` fails, but `nc -l 1337`works.

Categories: Linux Tags: , ,