Drupal files synchronization
Hi all, My company is using 3 Drupal servers with load balancer to distribute the load. The problem is the files in sites/default/files folder need to be synchronized among these 3 Drupal servers. I am thinking of using rsync for this purpose but I heard rsync may not be the best way to do it since it shouldn't delete the files so the number of files will grow quickly. Has anybody used rsync to synchronize the files? Is there a better method to synchronize files in sites/default/files? Thank you in advance! Cheers, Ivan _________________________________________________________________ New Windows 7: Find the right PC for you. Learn more. http://windows.microsoft.com/shop
You can try with a NFS mount point; Put your files on one server (which can be another one that web servers), and share the file folder with NFS ot the frontends. You can even do other things like altering URL with a reverse proxy to make them point directly to this server, put a really ligthweight HTTP server like lighthttpd on this box. Pierre. On Thu, 2009-10-22 at 19:16 +0800, Ivan Working wrote:
Hi all,
My company is using 3 Drupal servers with load balancer to distribute the load. The problem is the files in sites/default/files folder need to be synchronized among these 3 Drupal servers.
I am thinking of using rsync for this purpose but I heard rsync may not be the best way to do it since it shouldn't delete the files so the number of files will grow quickly.
Has anybody used rsync to synchronize the files? Is there a better method to synchronize files in sites/default/files?
Thank you in advance!
Cheers, Ivan
______________________________________________________________________ New Windows 7: Find the right PC for you. Learn more. -- [ Drupal support list | http://lists.drupal.org/ ]
Ivan, I would think that rsync would be a good way to do this with a good bit of testing. If a user deletes a file, rsync could delete it on the other servers. You would still rely on daily backups in case something goes wrong or a user deletes something they should not have deleted. I have used rysnc, but not for several prod servers behind a load balancer. I have used it to sync sites/site.com/files between prod, test and dev and it does work. rsync has many command switches so make sure to read up on it and set it up with the switches you need. Mark On Oct 22, 2009, at 7:16 AM, Ivan Working wrote:
Hi all,
My company is using 3 Drupal servers with load balancer to distribute the load. The problem is the files in sites/default/files folder need to be synchronized among these 3 Drupal servers.
I am thinking of using rsync for this purpose but I heard rsync may not be the best way to do it since it shouldn't delete the files so the number of files will grow quickly.
Has anybody used rsync to synchronize the files? Is there a better method to synchronize files in sites/default/files?
Thank you in advance!
Cheers, Ivan
New Windows 7: Find the right PC for you. Learn more. -- [ Drupal support list | http://lists.drupal.org/ ]
You have be careful with rsync, at least you don't update code files. But if you do, you'll have to put your site in maintenance mode to avoid users running drupal code whenever files are desync with database, this could lead to serious issues (may be it could have side effect with uploaded files too). Pierre. On Thu, 2009-10-22 at 19:16 +0800, Ivan Working wrote:
Hi all,
My company is using 3 Drupal servers with load balancer to distribute the load. The problem is the files in sites/default/files folder need to be synchronized among these 3 Drupal servers.
I am thinking of using rsync for this purpose but I heard rsync may not be the best way to do it since it shouldn't delete the files so the number of files will grow quickly.
Has anybody used rsync to synchronize the files? Is there a better method to synchronize files in sites/default/files?
Thank you in advance!
Cheers, Ivan
______________________________________________________________________ New Windows 7: Find the right PC for you. Learn more. -- [ Drupal support list | http://lists.drupal.org/ ]
RSYNC for files generally works fine. I been doing this on a pretty heavily trafficked site for a year now. Of course our setup is one server for Drupal then another for static content (with some core hacks to rewrite all static requests to a static. subdomain). To get around the RSYNC lag I have mod_rewrite on the static server. Basically if the file isn't there then it sends the request back to the main server. As I said - this works fine in a single php/single static server setup. Add in more than 1 PHP server and you got problems, unless you keep each visitor assigned to the same server at all times. The biggest problem comes with CSS or JS aggregation. If I make a request and it goes to server 1 and the CSS/JS are aggregated, then the request to load those files ends up on server 2 chances are great (very great) that RSYNC wouldn't have caught up yet and now I got a 404 on those files, resulting in a very borked site. Having said that, the best solution for multiple PHP servers is using a shared mount of some sort (NFS, SSHFS, etc.). That way if my initial request goes to server 1, the CSS and JS aggregate files are created and instantly available on all servers. Jamie Holly http://www.intoxination.net http://www.hollyit.net Pierre Rineau wrote:
You have be careful with rsync, at least you don't update code files. But if you do, you'll have to put your site in maintenance mode to avoid users running drupal code whenever files are desync with database, this could lead to serious issues (may be it could have side effect with uploaded files too).
Pierre.
On Thu, 2009-10-22 at 19:16 +0800, Ivan Working wrote:
Hi all,
My company is using 3 Drupal servers with load balancer to distribute the load. The problem is the files in sites/default/files folder need to be synchronized among these 3 Drupal servers.
I am thinking of using rsync for this purpose but I heard rsync may not be the best way to do it since it shouldn't delete the files so the number of files will grow quickly.
Has anybody used rsync to synchronize the files? Is there a better method to synchronize files in sites/default/files?
Thank you in advance!
Cheers, Ivan
______________________________________________________________________ New Windows 7: Find the right PC for you. Learn more. -- [ Drupal support list | http://lists.drupal.org/ ]
This really depends on the nature of the site. If the site is basically a content provider -- a limited group of users are logging in and adding files -- you can send all the logged in users to one server. In that scenario, rsync works fine. one way to do that is to require https for logins and direct all https requests to the key server. If you are running a site where everyone logs in and adds files, ala Facebook, you'll need more sophisticated file management. If you have only three servers in a single location, you might consider putting a single NFS file server behind them. Bigger setups use SANs. 2009/10/22 Ivan Working <workingivan@hotmail.com>:
Hi all,
My company is using 3 Drupal servers with load balancer to distribute the load. The problem is the files in sites/default/files folder need to be synchronized among these 3 Drupal servers.
I am thinking of using rsync for this purpose but I heard rsync may not be the best way to do it since it shouldn't delete the files so the number of files will grow quickly.
Has anybody used rsync to synchronize the files? Is there a better method to synchronize files in sites/default/files?
Thank you in advance!
Cheers, Ivan
________________________________ New Windows 7: Find the right PC for you. Learn more. -- [ Drupal support list | http://lists.drupal.org/ ]
-- Cary Gordon The Cherry Hill Company http://chillco.com
Hi all,
My company is using 3 Drupal servers with load balancer to distribute the load. The problem is the files in sites/default/files folder need to be synchronized among these 3 Drupal servers.
I am thinking of using rsync for this purpose but I heard rsync may not be the best way to do it since it shouldn't delete the files so the number of files will grow quickly.
Has anybody used rsync to synchronize the files? Is there a better method to synchronize files in sites/default/files?
Thank you in advance!
Cheers, Ivan
________________________________ New Windows 7: Find the right PC for you. Learn more. -- [ Drupal support list | http://lists.drupal.org/ ]
-- Cary Gordon The Cherry Hill Company http://chillco.com -- [ Drupal support list | http://lists.drupal.org/ ]
Maybe you can also consider to share only the directory sites/default/files from the principal server by NFS and mount only this directory on the other two servers. In this way the changes are available intermediately.
Take a look on capistrano: http://www.capify.org/index.php/Capistrano Att, Pablo Lacerda de Miranda Analista de Sistemas Pontífica Universidade Católica de São Paulo pablolmiranda@gmail.com
On Thu, 22 Oct 2009, Ivan Working wrote:
My company is using 3 Drupal servers with load balancer to distribute the load. The problem is the files in sites/default/files folder need to be synchronized among these 3 Drupal servers.
I am thinking of using rsync for this purpose but I heard rsync may not be the best way to do it since it shouldn't delete the files so the number of files will grow quickly.
Has anybody used rsync to synchronize the files? Is there a better method to synchronize files in sites/default/files?
I would be hesitant to use rsync to keep three sets of files universally synched like this, although I'm sure it could be done. The best method is likely to involve a central file repository. NFS share a file directory. If you don't like NFS, then SSHFS mount the files directory from each of the machines. If you're dealing with windows, there is (please don't anybody shoot me--this is not! how I would do it) smbmount. SSHFS is the most easily deployed solution, but you will of course not want to be using password based auth, and for the sake of initial simplicity, you will want to use keys without passphrases. If this is a really huge setup, you might consider one of the distributed filesystems, which store the files everywhere on the cluster, and export them to the files directories. That is the most effective for load balancing purposes, but you'd have to really need it to take the trouble to implement it, and for anything smaller than thousands of files, I wouldn't even consider that option. Luke
participants (8)
-
Cary Gordon -
Gastón Pablo Pérez -
Ivan Working -
Jamie Holly -
Luke -
Mark Shropshire -
Pablo L. de Miranda -
Pierre Rineau