Moving Large Files Between Two Servers Using rsync Over SSH: A Comprehensive Guide

The rsync command over SSH is one of the most practical methods in Linux for quickly, securely, and reliably transferring large files between two servers. Particularly when moving website files, backup archives, media folders, or application data that can reach hundreds of GB in size, rsync offers a significant speed advantage over traditional copying methods by only transferring the changed blocks. The basic usage principle is straightforward: files from the source server are encrypted and synchronized to the target server via an SSH tunnel; if the connection is interrupted, the process can resume from where it left off.

In server migrations, hosting upgrades, data center changes, or backup operations, both speed and data integrity are critical. Downloading a large archive in one piece and then re-uploading it can lead to wasted time and may require starting over due to connection issues. This is where rsync comes into play, preserving file timestamps, permissions, ownership information, and directory structure. As a result, it is frequently favored by Linux system administrators, DevOps teams, and web hosting experts in daily operations.

In this guide, we will walk you through the process of moving large files between two Linux servers using rsync over SSH, with step-by-step examples of actual commands. We will also share parameters for performance optimization, security checks, error scenarios, testing transfer methods, and key considerations before moving to the production environment. If you are planning to transition to a new VPS or hosting environment, ensure that you have sufficient disk space, SSH access, and the correct user permissions on both your source and target servers before following this guide. For relevant infrastructure options, check out Hostragons VPS Server Solutions and Linux Hosting Packages.

What is rsync and Why is it Preferred for Transferring Large Files?

rsync is a powerful command-line tool used for synchronizing files and directories either locally or between remote servers. Its most notable feature is the delta transfer algorithm, which transfers only the changed parts of a file instead of sending the entire file each time. For example, if there is only a 2 GB change in an 80 GB backup file, it is possible to transfer just the changed blocks under suitable conditions rather than moving the entire file again.

SSH encrypts the transfer, ensuring that the rsync command does not send file content as plain text over the network. This feature is especially important for customer data, e-commerce order records, media archives, database backups, and proprietary application files. In the world of hosting, data security is not limited to SSL certificates; the transfer channel between servers must also be secure. While using SSL Certificate for website traffic, SSH encryption should be preferred for server-to-server file transfers.

Compared to FTP or manual download-upload methods, rsync is more flexible. It can preserve permissions, transfer symbolic links, delete files at the target, limit bandwidth, and test what will be done prior to transfer. This level of control offers significant advantages during risky operations such as transferring large files.

When Should You Use the rsync Command Over SSH?

The rsync command over SSH is ideal for transferring large, numerous, or frequently changing files. For a simple visual folder, manual SFTP may suffice; however, when thousands of files with varying permissions and large archives are involved, rsync becomes a much more reliable option.

• If you are transferring a website from an old server to a new VPS server.
• If you are transferring directories containing thousands of media files, like the WordPress uploads folder.
• If you are copying daily, weekly, or monthly backups to a different server.
• If you are changing data centers or hosting providers.
• If you are synchronizing application files between production and backup environments.
• If you are planning long transfers with a chance of connection loss.

For example, downloading a 250 GB media folder to your desktop and then uploading it to the new server means you are performing two separate transfers. This not only extends the total time but also bottlenecks your home or office internet connection. By using rsync, you can eliminate this intermediary step and transfer directly from the source server to the target server.

Prerequisites Before You Start

Knowing the rsync command alone is not enough for a successful operation. Checking the conditions of both the source and target servers helps reduce the risk of data loss or incomplete transfers during the process. The following checks should be mandatory in production environments.

1. Verify SSH Access

You should be able to establish an SSH connection from the source server to the target server. The standard connection can be tested as follows: ssh user@target-server-ip. If SSH operates on a different port, the connection can be tested with: ssh -p 2222 user@target-server-ip. If this test fails, rsync will not work either. Check the firewall, Fail2ban, SSH port, user permissions, and password or key authentication.

2. Check if rsync is Installed

Most Linux distributions come with rsync pre-installed; however, it is necessary to verify this. On Debian and Ubuntu-based systems, you can use the command rsync --version. If it is not installed, you can install it with apt install rsync. For AlmaLinux, Rocky Linux, or CentOS derivatives, use dnf install rsync or yum install rsync. If you are using a Linux-based VPS from Hostragons, the installation usually takes just a few seconds.

3. Calculate Disk Space

If there is not enough free space on the target server, the transfer will fail. You can check the size of the source directory using du -sh /source/directory. The target's disk status should be checked with the df -h command. In practice, not only the file size but also temporary files and logs should be accounted for. For example, if you are transferring 500 GB of data, it is healthier to plan for at least 550-600 GB of available space on the target.

4. Plan for Critical Services and Changing Files

Files on live websites may change during transfer. Especially user uploads, cache folders, and log files are constantly updated. In large transfers, the primary data is transferred first, followed by putting the site into maintenance mode for a final rsync. This way, the downtime can be reduced to just minutes. You can complete your planning with support content like WordPress hosting migration guide.

The Basic rsync Command: Transferring Files Between Two Servers

The most commonly used basic structure is: rsync -avz /source/directory/ user@target-ip:/target/directory/. Here, -a is the archive mode which attempts to preserve permissions, timestamps, symbolic links, and similar information. -v provides detailed output. -z applies compression during transfer. The trailing slash at the end of the directory is an important detail. If you write /source/directory/, the contents of the directory will be copied to the target; if you write /source/directory, the directory itself will be created under the target.

Let's consider a sample scenario: You want to transfer 180 GB of web files from the /var/www/site/public_html/ folder on the old server to the /var/www/site/public_html/ folder on the new server. The command operates as follows: rsync -avz /var/www/site/public_html/ root@203.0.113.20:/var/www/site/public_html/. This command transfers files from the source server to the target server over SSH.

If you prefer not to use root, it is safer to create a special user with write permissions for the relevant directory on the target server. For example, you can use a user named deploy and set the necessary folder ownership accordingly. While working directly as root in production environments is practical, it is recommended to use a user with limited privileges for security reasons.

Recommended Secure and Resilient rsync Parameters

In large file transfers, it is not always correct to rely solely on the basic command. Additional parameters are necessary for connection interruptions, partial files, incorrect deletions, or slow networks. The following command serves as a solid starting point for many real transfer scenarios:

rsync -aHAXv --numeric-ids --partial --progress --append-verify -e ssh /source/directory/ user@target-ip:/target/directory/

In this command, -a helps preserve archive mode, -H maintains hard link information, -A preserves ACL permissions, and -X retains extended attributes. --numeric-ids uses ID values rather than user and group names, which can be helpful when user names differ across servers. --partial leaves partial files without deleting them. --progress shows the transfer status. --append-verify helps verify the added part when continuing from where it left off, especially for large files.

Not every parameter may be necessary on every system. For instance, while -aHAX may suffice for a simple website transfer, preserving ACL and extended attribute information may become critical in an application that uses specific file permissions. Therefore, it's essential to conduct a trial transfer first.

What if You are Using a Different SSH Port?

For security reasons, SSH may operate on a port other than 22. In this case, the SSH parameter must be specified within rsync. Example: rsync -avz -e ssh -p 2222 /source/directory/ user@target-ip:/target/directory/. However, the -e parameter must be used correctly in the command line. The correct usage principle is as follows: rsync -avz -e 'ssh -p 2222' /source/directory/ user@target-ip:/target/directory/.

Since single quotes may behave differently in JSON and some panels, ensure that the quotes are correct when typing directly in the terminal. If you are connecting with an SSH key, the -i parameter can also be added: rsync -avz -e 'ssh -p 2222 -i /root/.ssh/id_rsa' /source/directory/ user@target-ip:/target/directory/. This method is often used in automation and cron-based backup jobs.

Pre-Transfer Testing: Using --dry-run

One of the most professional steps before a large data transfer is to use --dry-run. This parameter does not actually copy the files; it only shows which files will be transferred. Thus, incorrect directories, faulty target paths, or unexpected deletions can be detected in advance.

Example test command: rsync -avz --dry-run /var/www/site/ user@target-ip:/var/www/site/. You will see the files to be transferred, directories to be created, and a summary of the process in the output. If you plan to use the --delete parameter, dry-run becomes even more critical. This is because --delete can remove files from the target that do not exist in the source. When used correctly, it mirrors the target to the source; however, when used in the wrong directory, it can lead to significant data loss.

The professional application sequence usually follows this order: first, an SSH test is conducted, then disk space is checked, after that, rsync --dry-run is executed, a real test is performed with a small directory, and finally, the main transfer is initiated. This approach adds a few minutes of extra time but prevents transfer errors that could take hours to resolve.

Performance Considerations: Compression, Bandwidth, and Encryption Settings

The performance of rsync is not solely dependent on file size. The CPU power of the servers, disk type, network latency, bandwidth, and number of files are also important. A large number of small files can be transferred slower than a single large file because metadata processes are required for each file. VPS environments with NVMe disks can show significantly better performance in file scanning and writing. Therefore, if you are planning a heavy transfer, it makes sense to consider NVMe VPS Server options.

The -z compression parameter is beneficial for text files, logs, and certain types of backups. However, it may consume CPU without providing much gain for files that are already compressed, like zip, tar.gz, jpg, mp4, or gzip. If you are transferring large media files between two powerful servers with a 1 Gbps connection, you may achieve better performance without -z. Conversely, in low bandwidth scenarios and text-heavy files, -z can provide significant savings.

To limit bandwidth, --bwlimit can be used. For example, --bwlimit=50000 sets a limit of approximately 50 MB/s. This is essential to avoid choking the network traffic of a live website. Performing full-speed transfers during peak hours on a production server can negatively affect visitor experience. Planning the transfer during nighttime, applying a bandwidth limit, and monitoring load average values during the process are good practices.

Comparing rsync, scp, SFTP, and FTP

There are multiple methods for file transfer, but each method has its strengths and weaknesses. The following table summarizes which tool is more suitable for large file and server transfer scenarios.

As seen in the table, rsync stands out, especially in repetitive synchronizations and large data transfers. While scp is simple and convenient, it is not as powerful as rsync in intelligently detecting changed files. FTP is not suitable for modern security standards when used without encryption. While FTP access can be practical in hosting panels, SSH-based methods should be preferred for critical transfers between servers.

Example Workflow for Live Website Migration

When migrating a website from an old server to a new one, the goal is not just to copy files. DNS, database, SSL, PHP version, file permissions, and email configurations should also be checked. Below is a low-downtime example workflow for a large file website.

• Prepare the web server, PHP, database, and necessary modules on the new server.
• Before the transfer, lower the DNS TTL value of the domain to a low value like 300 seconds. You can evaluate Domain Lookup and Domain Management for domain management.
• Start the initial rsync transfer while the site is live. This stage moves most of the large data.
• Take a backup of the database and transfer it to the new server.
• Put the site into short maintenance mode.
• Run the second rsync command to synchronize the last changed files.
• Update the database again or use an incremental strategy.
• Check file permissions, site access, and logs on the new server.
• Install or renew the SSL certificate. You may find SSL Certificate Installation content helpful for this.
• Redirect the DNS records to the new IP address.

In this method, the longest duration is spent on the main file transfer conducted while the site is operational. Downtime occurs only during the final synchronization and DNS transition. This approach can reduce downtime from hours to just minutes for WordPress sites with large media archives.

Checking Data Integrity

Once the transfer is complete, it is not enough for the command to have finished executing. File counts, sizes, and critical files should be checked. For a simple check, the directory sizes can be compared using the du -sh command on both the source and target. However, due to filesystem block differences, the values may not appear identical. For a more detailed check, rsync can be run again with the same parameters. If there are no new files to transfer, the output will be minimal.

For checksum verification, rsync -avc --dry-run /source/directory/ user@target-ip:/target/directory/ can be used. Here, the -c parameter compares files not only by timestamp and size but also by checksum. It should be used cautiously as it can create CPU and disk load with very large data. This extra verification provides assurance for critical archives, database dump files, or customer documents.

If you have migrated a website, make sure to conduct application-level checks. The homepage, admin panel, visual uploads, payment page, contact form, and error logs should be reviewed. In e-commerce sites, merely transferring files is not sufficient; payment integrations and cron jobs should also be tested on the new server.

Common Errors and Solutions

Permission Denied Error

This error typically occurs due to SSH user permissions, write permissions for the target directory, or key authentication issues. First, try establishing a normal SSH connection with the same user. If the connection is successful, perform a test to create a file in the target directory. If necessary, change the ownership of the target folder to the relevant user using chown, or plan a sudo-based strategy.

No Space Left on Device Error

The transfer stops when there is insufficient space left on the target disk. Check the disk partitions using df -h. Sometimes the main disk is not full, but /tmp or the target mount point is full. In large transfers, log files, old backups, and cache folders can be cleaned up. For a permanent solution, increasing the disk size or transitioning to a higher-capacity VPS plan may be necessary.

Connection Reset or Broken Pipe Error

Network interruptions may occur during prolonged transfers. If you are using --partial and --append-verify, you can rerun the command to continue from where it left off. Also, SSH keepalive settings can help. On the client side, settings like ServerAliveInterval 60 can reduce idle disconnections.

Very Slow Transfer

Slow transfers can result from the number of files, disk I/O limits, compression load, or network capacity. If there are many small files, creating a tar archive first may be faster in some cases; however, this reduces rsync's advantage in detecting file-based changes. Test removing the -z parameter for large and already compressed files. Additionally, use tools like iotop, htop, and iftop to identify whether the bottleneck is on the CPU, disk, or network side.

Best Practices for Security

Security should not be overlooked during file transfers between servers. Particularly, root access, SSH keys, and backup files should be treated as sensitive. It is recommended to use SSH keys instead of password-based logins. Key files should have strict permissions, private keys should not be shared, and ideally, they should be protected with a passphrase.

Creating a temporary user for the transfer and granting access only to necessary directories is a good method. After the process is complete, the user's permissions can be revoked or the account can be disabled. Limiting SSH access to specific IP addresses also reduces the attack surface. Firewall rules, Fail2ban, and an up-to-date OpenSSH version should be among the basic checks.

If transferring database dump files, customer documents, or archives containing personal data, unnecessary temporary files should be cleaned up on both the source and target after the transfer. Additionally, leaving backup files like .sql, .zip, or .tar under the web root directory poses a serious security risk. These files should be stored in a directory closed to web access or deleted after the process.

Using rsync for Automated Backups

rsync can be used not only for one-time transfers but also for regular backups. For example, a cron job can be defined to send a specific folder to a backup server every night. In this scenario, a passwordless but secure connection is established using SSH keys, rsync only transfers the changed files, and a bandwidth limit can be applied.

Example logic: rsync -a --delete --bwlimit=30000 /var/www/site/ backup@backup-ip:/backup/site/. This command can synchronize the target directory with the source. However, it should be used carefully due to the --delete parameter. If a file is accidentally deleted in the source, it will also be deleted in the target. Therefore, critical backup strategies should prefer solutions that support snapshots, daily folders, or versioning.

In hosting operations, the 3-2-1 backup rule is a good reference: at least 3 copies of the data should be kept in 2 different media, with 1 copy in a different location. rsync makes the server-to-server copy aspect of this strategy practical. For a more comprehensive infrastructure, you can strengthen your plan with Backup Solutions and Corporate Hosting Services content.

Final Checklist

Before initiating the transfer of large files, applying the following short checklist reduces the risk of errors. This list should especially become standard procedure in production environments.

• Is rsync installed on both the source and target servers?
• Was the SSH connection tested with the correct user and port?
• Is there sufficient disk space on the target server?
• Was the initial operation checked with --dry-run?
• Was maintenance mode and the final synchronization planned for the live site?
• If --delete is to be used, was the target path verified?
• Will file size, permissions, and application tests be conducted post-transfer?
• Will unnecessary temporary backup and dump files be securely cleaned?

While these checks may seem simple, the most common issues in real life often arise from incorrect target directories, missing permissions, insufficient disk space, and unplanned DNS transitions. A well-planned rsync process shortens transfer times and increases the likelihood of successful recovery.

Frequently Asked Questions

If the connection drops while using the rsync command over SSH, will the transfer start over?

No, if the correct parameters are used, it will not start over. The --partial and --append-verify options help preserve incomplete large files and allow the command to resume from where it left off when run again. However, for critical transfers, it is advisable to check the target disk space and the status of the partial files before rerunning the command.

Can a database be directly transferred with rsync?

rsync transfers files; directly copying active database files is not advisable. mysqldump or physical backup tools should be used for MySQL or MariaDB, and the resulting backup file should be transferred to the target server using rsync. In live systems, maintenance mode or replication strategies should be planned for data consistency.

Is root access required on both servers to use rsync?

Root access is not mandatory. A user with write permissions to the target directory is sufficient. However, if file ownership, system directories, ACLs, or special permissions need to be preserved, higher privileges may be required. For security reasons, it is more appropriate to operate with a user with limited privileges whenever possible.

Is the --delete parameter safe?

When used correctly, --delete is very useful for synchronizing the target directory with the source; however, if used in the wrong target directory, it can lead to file loss. Therefore, commands that include --delete should always be tested with --dry-run first, and the source and target paths should be carefully verified.

Which is better for transferring large files, rsync or scp?

For one-time small copies, scp may be sufficient; however, in large file transfers, numerous files, and repetitive synchronizations, rsync is generally better. This is because it can detect changed data, resume interrupted transfers, limit bandwidth, and provides a test mode.

Summary and Next Steps

Transferring large files between two servers using the rsync command over SSH is a fast, secure, and manageable method. With the right parameters, it becomes resilient to connection drops, saves time by only transferring changed files, and reduces downtime in live site migration processes. For the best results, first conduct SSH and disk checks, test with --dry-run, and then plan the main transfer methodically.

If you are considering transitioning to a new server, high-capacity storage, or a more performant hosting infrastructure, you can explore Hostragons solutions; with VPS, hosting, domain, and SSL options tailored to your needs, you can make your transfer process more secure. Start with small tests for a smooth transition, measure, and plan for the final synchronization.