Friday, November 25, 2011

Linux: Splitting a Large File into Small Files

Recently I was trying to transfer a large ISO file across a horribly unstable VPN. The transfer would fail at various amounts of transfer percentages. So, I thought I'd best split the file up into 10MB chunks, then rsync those over and stitch it back together.

That way if it failed 90% of the way through, I wouldn't have to resend all the data, just that last 10%. The way I managed to do this, was to:

split --bytes=10m file.iso file_part 

What happens now is, you have a bunch of 10MB files called

file_partaa
file_partab
flie_partac
file_partad
...

So, just rsync all those files to the destination:

rsync -e ssh -a --progress file_part* user@desination.host.com:

When that completes, login to the remote host and put them back together:

cat file_part* > orig_file.iso


Done and done.

1 comment:

  1. Not a bad solution but there is an easier way to achieve it without the use of split. rsync has a --partial flag which does what you need.

    ReplyDelete