to resume the download by using gsutil-CodePudding

I have been downloading the file by using gsutil, and the process has crushed.

The documentation on gsutil is located at :

https://cloud.google.com/storage/docs/gsutil_install#redhat

The file location is described on : https://genebass.org/downloads

How can I resume the file download instead of starting from scratch ?

I have been looking for answers to a similar question, although those have been provided to different questions. For example :

GSutil resume download using tracker files

CodePudding user response：

I'm not sure which command you're using (cp or rsync), but either way gsutil will fortunately take care of resuming downloads for you.

From the docs for gsutil cp:

gsutil automatically resumes interrupted downloads and interrupted resumable uploads, except when performing streaming transfers.

So, if you're using gsutil cp, it will automatically resume the partially downloaded files without starting them over. However, resuming with cp will also re-download the files that were already completed. To avoid this, use the -n flag so the files you've already downloaded are skipped, something like:

gsutil cp -n -r gs://ukbb-exome-public/300k/results/variant_results.mt .

If instead you're using gsutil rsync, then it will simply resume downloading.

CodePudding user response：

As mentioned in GCP docs, using the gsutil cp command:

gsutil automatically performs a resumable upload whenever you use the cp command to upload an object that is larger than 8 MiB. You do not need to specify any special command line options to make this happen. [. . .] Similarly, gsutil automatically performs resumable downloads (using standard HTTP Range GET operations) whenever you use the cp command, unless the destination is a stream. In this case, a partially downloaded temporary file will be visible in the destination directory. Upon completion, the original file is deleted and overwritten with the downloaded contents.

If you're also using gsutil in large production tasks, you may find useful information on Scripting Production Transfers.

Alternatively, you can achieve resumable download from Google Cloud Storage using the Range header (just take note of the HTTP specification threshold).