Yes, RSYNC (Remote Sync), a popular and widely used open-source utility on Unix and Linux operating systems, is fully capable of handling files larger than 2GB. This capability applies when transferring files, synchronizing directories, or performing other file operations.
RSYNC’s capacity to handle files larger than 2GB was implemented in versions subsequent to 2.6.0. According to RSYNC’s man page documentation, long before version 3.0.0, there were certain problems related to transferring files over 2GB, due to an internal variable format problem. This issue was resolved in later updates. The changes were accomplished through the use of a “rolling checksum” algorithm, which facilitates large files’ transfers by sending only the differences between the source and destination files (source: “The rsync algorithm”, Andrew Tridgell, Paul Mackerras).
RSYNC first divides the source file into chunks and checks if the same chunk exists in the destination file. If it does, RSYNC simply notes its location; if it doesn’t exist, RSYNC compresses the chunk and sends it to the destination. Even if the destination file is rearranged, RSYNC would only update the differences, resulting in a very efficient and fast operation.
Another feature that aids in the handling of large files is the ‘—partial’ option in RSYNC. If a transfer of a large file fails or is interrupted, instead of starting the transfer from the beginning, RSYNC simply continues from where it left off, thus saving both time and resources (source: RSYNC man page documentation).
RSYNC is not limited by file size from a technical perspective but it can be limited by the file system where the data is being stored or the available storage space. For example, FAT32 file system holds a maximum file size of 4GB. That said, most modern systems such as NTFS, ext4 and HFS+ support files far greater than 2GB (source: Linux File Systems: Ext2 vs Ext3 vs Ext4).
In practical application, the Dropbox backup team used RSYNC for its capability of transferring large files efficiently. The team leveraged its ability to resume interrupted transfers and to deal with packet loss during the transfer of large files over unreliable networks (source: “Moving petabytes in a day: Our journey to a new backup system”, Dropbox Tech Blog).
In conclusion, RSYNC is quite robust in handling large files, not just 2GB, but even much larger. Its intelligent design enables it only to sync the differences in files, rather than the whole file, which significantly improves speed and efficiency.
