Saturday, March 15, 2008
Friday, January 11, 2008
How do download Osiris dumps fast
1. SSH to HPC and go to the output folder
2. Compress the folder using the command
tar -cvzpf archive.tgz /auto/rcf-19/kallos/new_beam/MS
where the path will be different for each case. This compresses the total file size by ~10 times.
3. Download the single archive file using an ftp client. The time is roughly 2.5 minutes per GB of original (uncompressed) data. The total time savings is about 20 times compared to uncompressed data.
4. Decompresses locally the files using a program like winrar.
5. You are set to go and analyze the data!
2. Compress the folder using the command
tar -cvzpf archive.tgz /auto/rcf-19/kallos/new_beam/MS
where the path will be different for each case. This compresses the total file size by ~10 times.
3. Download the single archive file using an ftp client. The time is roughly 2.5 minutes per GB of original (uncompressed) data. The total time savings is about 20 times compared to uncompressed data.
4. Decompresses locally the files using a program like winrar.
5. You are set to go and analyze the data!
Subscribe to:
Posts (Atom)