Workflow: staging server, source control, and the files directory
Hi all, I have a question about people's practices using a staging server and source control. I'm keen for us to have a sensible workflow while developing sites, and currently we have the following setup. * Developers have local copies of the site where they develop new functionality, which they check in to source control (Subversion). * The staging server has a checked out version of the repository. * The repository has a post-commit hook that automatically updates the staging server. * There are some changes we make directly via the staging server (ie not in code) and for that there's a script that clears caches, dumps the database, and stores that in the repository. (We try to keep as much in code as possible.) Things have been working well so far, but just recently we started adding content to the site (on the staging server). The problem comes with uploading files via the wysiwyg editor, as this bypasses the normal mechanism of checking into source control and letting the post-commit hook update the staging server automatically. So the question is, what's the best way to keep the staging server's files directory synced with the repository? I'd like to keep things as automated as possible. Here's some ideas: * On some event (eg. post-commit, cron) I just do something like a simple svn add files/*. This won't prune nicely though. * Exclude the files directory from source control altogether, and change the local update process to just copy from the staging server's files directory to the local. This would work, just feels slightly wrong to me as the db contains content that refers to the files, and that *is* in source control. * Maybe I could treat the files directory as a vendor branch and use something like svn_load_dirs.pl to try and keep them in sync. I've not used this script before, and don't know if it would manage this simple situation automatically, or if it would need some manual help. I suppose I could also write a script to parse the output of svn st, but as always would prefer to keep the workload as low as possible :) * Nice super simple idea I've not considered?! I'd love to hear what other people in this situation do, or just any ideas at all. Many thanks, Andy . . . . . . . Andy Fowlston +44 (0)20 8747 5068 andy@pedalo.co.uk Skype: andy.pedalo www.pedalo.co.uk This email is intended only for the above named addressee/s. This email may be confidential or legally privileged. If you have received this email and you are not a named addressee, you must not use, copy, distribute or disclose the email or any part of its contents or take any action in reliance on it. If you have received this email in error, please email the sender by replying to this message and delete it from your system. All reasonable precautions have been taken to ensure no viruses are present in this email. pedalo limited cannot accept responsibility for loss or damage arising from the use of this email or attachments and recommends that you subject these to your virus checking procedures prior to use. Any views or opinions presented are solely those of the author and not necessarily those of Pedalo Limited Please consider the environment before printing this email
My opinion is that nothing in the files directory should ever be in source control - that will just cause pain. Those are transient, website-created files. They even have the wrong permissions typically to be managed successfully. They should be backed up, and on fancier website configurations need to be shared or synced, but should not be in source control. If you have files that *are* "assets", but not user-generated (like graphics files that don't go with the code) they might go in an "assets" directory at the webroot or above, and be under source control there. -Randy On Fri, Feb 18, 2011 at 3:14 AM, Andy Fowlston <andy@pedalo.co.uk> wrote:
Hi all,
I have a question about people's practices using a staging server and source control. I'm keen for us to have a sensible workflow while developing sites, and currently we have the following setup.
* Developers have local copies of the site where they develop new functionality, which they check in to source control (Subversion). * The staging server has a checked out version of the repository. * The repository has a post-commit hook that automatically updates the staging server. * There are some changes we make directly via the staging server (ie not in code) and for that there's a script that clears caches, dumps the database, and stores that in the repository. (We try to keep as much in code as possible.)
Things have been working well so far, but just recently we started adding content to the site (on the staging server). The problem comes with uploading files via the wysiwyg editor, as this bypasses the normal mechanism of checking into source control and letting the post-commit hook update the staging server automatically. So the question is, what's the best way to keep the staging server's files directory synced with the repository? I'd like to keep things as automated as possible. Here's some ideas:
* On some event (eg. post-commit, cron) I just do something like a simple svn add files/*. This won't prune nicely though. * Exclude the files directory from source control altogether, and change the local update process to just copy from the staging server's files directory to the local. This would work, just feels slightly wrong to me as the db contains content that refers to the files, and that *is* in source control. * Maybe I could treat the files directory as a vendor branch and use something like svn_load_dirs.pl to try and keep them in sync. I've not used this script before, and don't know if it would manage this simple situation automatically, or if it would need some manual help. I suppose I could also write a script to parse the output of svn st, but as always would prefer to keep the workload as low as possible :) * Nice super simple idea I've not considered?!
I'd love to hear what other people in this situation do, or just any ideas at all.
Many thanks,
Andy
. . . . . . . Andy Fowlston +44 (0)20 8747 5068 andy@pedalo.co.uk Skype: andy.pedalo www.pedalo.co.uk
This email is intended only for the above named addressee/s. This email may be confidential or legally privileged. If you have received this email and you are not a named addressee, you must not use, copy, distribute or disclose the email or any part of its contents or take any action in reliance on it. If you have received this email in error, please email the sender by replying to this message and delete it from your system. All reasonable precautions have been taken to ensure no viruses are present in this email.
pedalo limited cannot accept responsibility for loss or damage arising from the use of this email or attachments and recommends that you subject these to your virus checking procedures prior to use. Any views or opinions presented are solely those of the author and not necessarily those of Pedalo Limited
Please consider the environment before printing this email
-- Randy Fay Drupal Module and Site Development randy@randyfay.com +1 970.462.7450
Thanks for the input Randy. That certainly keeps life simpler, though it still feels a bit funny having the markup stored in the repository (via the db dump) but leaving out the assets they refer to. (Btw I'd never consider keeping the files directory under source control after going live, and I will always use source control for non-content assets: the question is only about setting up the initial content that the site will go live with.) Thanks again, I appreciate your insight. Andy . . . . . . . Andy Fowlston +44 (0)20 8747 5068 andy@pedalo.co.uk Skype: andy.pedalo www.pedalo.co.uk This email is intended only for the above named addressee/s. This email may be confidential or legally privileged. If you have received this email and you are not a named addressee, you must not use, copy, distribute or disclose the email or any part of its contents or take any action in reliance on it. If you have received this email in error, please email the sender by replying to this message and delete it from your system. All reasonable precautions have been taken to ensure no viruses are present in this email. pedalo limited cannot accept responsibility for loss or damage arising from the use of this email or attachments and recommends that you subject these to your virus checking procedures prior to use. Any views or opinions presented are solely those of the author and not necessarily those of Pedalo Limited Please consider the environment before printing this email From: development-bounces@drupal.org [mailto:development-bounces@drupal.org] On Behalf Of Randy Fay Sent: 18 February 2011 13:51 To: development@drupal.org Subject: Re: [development] Workflow: staging server, source control, and the files directory My opinion is that nothing in the files directory should ever be in source control - that will just cause pain. Those are transient, website-created files. They even have the wrong permissions typically to be managed successfully. They should be backed up, and on fancier website configurations need to be shared or synced, but should not be in source control. If you have files that *are* "assets", but not user-generated (like graphics files that don't go with the code) they might go in an "assets" directory at the webroot or above, and be under source control there. -Randy On Fri, Feb 18, 2011 at 3:14 AM, Andy Fowlston <andy@pedalo.co.uk> wrote: Hi all, I have a question about people's practices using a staging server and source control. I'm keen for us to have a sensible workflow while developing sites, and currently we have the following setup. * Developers have local copies of the site where they develop new functionality, which they check in to source control (Subversion). * The staging server has a checked out version of the repository. * The repository has a post-commit hook that automatically updates the staging server. * There are some changes we make directly via the staging server (ie not in code) and for that there's a script that clears caches, dumps the database, and stores that in the repository. (We try to keep as much in code as possible.) Things have been working well so far, but just recently we started adding content to the site (on the staging server). The problem comes with uploading files via the wysiwyg editor, as this bypasses the normal mechanism of checking into source control and letting the post-commit hook update the staging server automatically. So the question is, what's the best way to keep the staging server's files directory synced with the repository? I'd like to keep things as automated as possible. Here's some ideas: * On some event (eg. post-commit, cron) I just do something like a simple svn add files/*. This won't prune nicely though. * Exclude the files directory from source control altogether, and change the local update process to just copy from the staging server's files directory to the local. This would work, just feels slightly wrong to me as the db contains content that refers to the files, and that *is* in source control. * Maybe I could treat the files directory as a vendor branch and use something like svn_load_dirs.pl to try and keep them in sync. I've not used this script before, and don't know if it would manage this simple situation automatically, or if it would need some manual help. I suppose I could also write a script to parse the output of svn st, but as always would prefer to keep the workload as low as possible :) * Nice super simple idea I've not considered?! I'd love to hear what other people in this situation do, or just any ideas at all. Many thanks, Andy . . . . . . . Andy Fowlston +44 (0)20 8747 5068 andy@pedalo.co.uk Skype: andy.pedalo www.pedalo.co.uk This email is intended only for the above named addressee/s. This email may be confidential or legally privileged. If you have received this email and you are not a named addressee, you must not use, copy, distribute or disclose the email or any part of its contents or take any action in reliance on it. If you have received this email in error, please email the sender by replying to this message and delete it from your system. All reasonable precautions have been taken to ensure no viruses are present in this email. pedalo limited cannot accept responsibility for loss or damage arising from the use of this email or attachments and recommends that you subject these to your virus checking procedures prior to use. Any views or opinions presented are solely those of the author and not necessarily those of Pedalo Limited Please consider the environment before printing this email -- Randy Fay Drupal Module and Site Development randy@randyfay.com +1 970.462.7450
On a tangential note, I am not sure I would consider it 'best practice' to keep a database dump in the repository, at least once it gets larger than a few hundred mb or so, especially if you commit updated db dumps often: 1. DB dumps are not incrementally updated, so it can't be stored efficiently (ie, a delta between revisions usually isn't smaller than the entire dump). 2. Some source control systems (I'm looking at you, git...) choke with large files (>100mb). Specifically with Git, you start running into memory allocation errors with 'git merge' and 'git gc'; they can be worked around to a certain degree by using some arguments to limit the amount of memory available for packing the repository, but as your repository size continues to grow, packing with limited memory becomes less and less efficient, and the problem starts feeding on itself. Eventually, it gets to a point where normal git operations such as pulls or merges take a significant amount of time, but a call to 'git gc' to clean and optimize the repository sends the server into swap memory until you kill the command. What we currently do at PINGV is use 'Backup & Migrate' on all of our staging sites. We have regular backups of the staging server being made, while at the same time, those of us with local environments just download updated DB exports as necessary to grab the latest config. This keeps our git repository much smaller, faster, and memory efficient. On 11-02-18 09:35 AM, Andy Fowlston wrote:
Thanks for the input Randy. That certainly keeps life simpler, though it still feels a bit funny having the markup stored in the repository (via the db dump) but leaving out the assets they refer to. (Btw I'd never consider keeping the files directory under source control after going live, and I will always use source control for non-content assets: the question is only about setting up the initial content that the site will go live with.)
Thanks again, I appreciate your insight.
Andy
. . . . . . . Andy Fowlston +44 (0)20 8747 5068 andy@pedalo.co.uk Skype: andy.pedalo www.pedalo.co.uk
This email is intended only for the above named addressee/s. This email may be confidential or legally privileged. If you have received this email and you are not a named addressee, you must not use, copy, distribute or disclose the email or any part of its contents or take any action in reliance on it. If you have received this email in error, please email the sender by replying to this message and delete it from your system. All reasonable precautions have been taken to ensure no viruses are present in this email.
pedalo limited cannot accept responsibility for loss or damage arising from the use of this email or attachments and recommends that you subject these to your virus checking procedures prior to use. Any views or opinions presented are solely those of the author and not necessarily those of Pedalo Limited
Please consider the environment before printing this email
From: development-bounces@drupal.org [mailto:development-bounces@drupal.org] On Behalf Of Randy Fay Sent: 18 February 2011 13:51 To: development@drupal.org Subject: Re: [development] Workflow: staging server, source control, and the files directory
My opinion is that nothing in the files directory should ever be in source control - that will just cause pain. Those are transient, website-created files. They even have the wrong permissions typically to be managed successfully. They should be backed up, and on fancier website configurations need to be shared or synced, but should not be in source control.
If you have files that *are* "assets", but not user-generated (like graphics files that don't go with the code) they might go in an "assets" directory at the webroot or above, and be under source control there.
-Randy On Fri, Feb 18, 2011 at 3:14 AM, Andy Fowlston<andy@pedalo.co.uk> wrote: Hi all,
I have a question about people's practices using a staging server and source control. I'm keen for us to have a sensible workflow while developing sites, and currently we have the following setup.
* Developers have local copies of the site where they develop new functionality, which they check in to source control (Subversion). * The staging server has a checked out version of the repository. * The repository has a post-commit hook that automatically updates the staging server. * There are some changes we make directly via the staging server (ie not in code) and for that there's a script that clears caches, dumps the database, and stores that in the repository. (We try to keep as much in code as possible.)
Things have been working well so far, but just recently we started adding content to the site (on the staging server). The problem comes with uploading files via the wysiwyg editor, as this bypasses the normal mechanism of checking into source control and letting the post-commit hook update the staging server automatically. So the question is, what's the best way to keep the staging server's files directory synced with the repository? I'd like to keep things as automated as possible. Here's some ideas:
* On some event (eg. post-commit, cron) I just do something like a simple svn add files/*. This won't prune nicely though. * Exclude the files directory from source control altogether, and change the local update process to just copy from the staging server's files directory to the local. This would work, just feels slightly wrong to me as the db contains content that refers to the files, and that *is* in source control. * Maybe I could treat the files directory as a vendor branch and use something like svn_load_dirs.pl to try and keep them in sync. I've not used this script before, and don't know if it would manage this simple situation automatically, or if it would need some manual help. I suppose I could also write a script to parse the output of svn st, but as always would prefer to keep the workload as low as possible :) * Nice super simple idea I've not considered?!
I'd love to hear what other people in this situation do, or just any ideas at all.
Many thanks,
Andy
. . . . . . . Andy Fowlston +44 (0)20 8747 5068 andy@pedalo.co.uk Skype: andy.pedalo www.pedalo.co.uk
This email is intended only for the above named addressee/s. This email may be confidential or legally privileged. If you have received this email and you are not a named addressee, you must not use, copy, distribute or disclose the email or any part of its contents or take any action in reliance on it. If you have received this email in error, please email the sender by replying to this message and delete it from your system. All reasonable precautions have been taken to ensure no viruses are present in this email.
pedalo limited cannot accept responsibility for loss or damage arising from the use of this email or attachments and recommends that you subject these to your virus checking procedures prior to use. Any views or opinions presented are solely those of the author and not necessarily those of Pedalo Limited
Please consider the environment before printing this email
Thanks Brian, Wesley, Laura. Replies below. Brian wrote:
On a tangential note, I am not sure I would consider it 'best practice' to keep a database dump in the repository
Agreed, this is only until we're all capable of putting everything in code (it's a relatively new workflow process for us).
DB dumps are not incrementally updated, so it can't be stored efficiently (ie, a delta between revisions usually isn't smaller than the entire dump).
Using drush to give neat output and ignore cache tables actually means that a diff is (just about) understandable. Not fun though, and definitely not the way forwards.
We have regular backups of the staging server being made, while at the same time, those of us with local environments just download updated DB exports as necessary to grab the latest config.
From a db point of view this sounds very similar to what we do. Could you elaborate on how you keep the files in sync? Wesley wrote:
Why not use drush with site aliases? http://drupal.org/node/670460
I love it! Well I've only skimmed it, but it looks perfect thanks! Laura wrote:
you should exclude imagecache (aka "image" module in D7) renderings from source control
Thanks, that's already set up:)
having broken images in your local instance is not deadly, in my experience.
Yeah, maybe I'm overthinking this bit:) Thanks all, Andy . . . . . . . Andy Fowlston +44 (0)20 8747 5068 andy@pedalo.co.uk Skype: andy.pedalo www.pedalo.co.uk This email is intended only for the above named addressee/s. This email may be confidential or legally privileged. If you have received this email and you are not a named addressee, you must not use, copy, distribute or disclose the email or any part of its contents or take any action in reliance on it. If you have received this email in error, please email the sender by replying to this message and delete it from your system. All reasonable precautions have been taken to ensure no viruses are present in this email. pedalo limited cannot accept responsibility for loss or damage arising from the use of this email or attachments and recommends that you subject these to your virus checking procedures prior to use. Any views or opinions presented are solely those of the author and not necessarily those of Pedalo Limited Please consider the environment before printing this email
On 11-02-18 11:00 AM, Andy Fowlston wrote:
We have regular backups of the staging server being made, while at the same time, those of us with local environments just download updated DB exports as necessary to grab the latest config. From a db point of view this sounds very similar to what we do. Could you elaborate on how you keep the files in sync? Keep which files? The database exports?
We look at it this way - staging is the canonical database. All config is done on staging, never in the developer-specific local environments. In this scenario, there is no need to keep database dumps 'in sync' - as soon as you grab a new dump from staging, you are up to date. We keep full backups of the staging db in case something goes wrong, but we don't keep database files in any sort of version control for the purposes of having a synced dump. Brian
Brian wrote:
From a db point of view this sounds very similar to what we do. Could you elaborate on how you keep the files in sync? Keep which files? The database exports?
Sorry, no not the database at all: I'm wondering about any content related files in the files directory (I presume you use source control for everything outside the files directory?).
We look at it this way - staging is the canonical database. All config is done on staging, never in the developer-specific local environments.
Yeah, that's how we're doing things at the mo. I'd like to move more (ideally all) of the config to source files, but we're not there yet. Thanks, Andy . . . . . . . Andy Fowlston +44 (0)20 8747 5068 andy@pedalo.co.uk Skype: andy.pedalo www.pedalo.co.uk This email is intended only for the above named addressee/s. This email may be confidential or legally privileged. If you have received this email and you are not a named addressee, you must not use, copy, distribute or disclose the email or any part of its contents or take any action in reliance on it. If you have received this email in error, please email the sender by replying to this message and delete it from your system. All reasonable precautions have been taken to ensure no viruses are present in this email. pedalo limited cannot accept responsibility for loss or damage arising from the use of this email or attachments and recommends that you subject these to your virus checking procedures prior to use. Any views or opinions presented are solely those of the author and not necessarily those of Pedalo Limited Please consider the environment before printing this email
Ah, I misunderstood. We take it on a case-by-case basis. 1. The imagecache directory is always excluded, as those are automagically regenerated anyways if they don't exist. We also exclude the 'js' and 'css' folders, as putting the aggregated js and css files in git is pointless. 2. We generally exclude the files directory altogether, and just put up with broken images on our local dev environments. 3. If absolutely necessary, we may rsync the files directory down to a dev environment periodically. We have on occasion tried to keep small copies of the files directory in git, but we run into problems where nodes are deleted on the site, and Drupal deletes the corresponding file from the filesystem. However, having not been removed from git with 'git rm <filename>', someone has to push the filesystem changes with 'git add -u' to make the files directory sync properly. Also, we run into the odd issue where files uploaded by developers on thier local copies conflict with files uploaded on the staging server (test.txt, anyone?). This often needs sorting out... All-in-all, it's extra work with little extra gain, other than prevent some broken images or file links on the non-canonical copies of the site, which can be fixed as needed by rsyncing the files directory down to the non-canonical development copies. In short, I keep all the site code in git. For anything that's not code (or part of a code package, such as theme or module images), I try to leave out unless there is a good reason for inclusion. Brian On 11-02-18 11:18 AM, Andy Fowlston wrote:
Brian wrote:
From a db point of view this sounds very similar to what we do. Could you elaborate on how you keep the files in sync? Keep which files? The database exports? Sorry, no not the database at all: I'm wondering about any content related files in the files directory (I presume you use source control for everything outside the files directory?).
We look at it this way - staging is the canonical database. All config is done on staging, never in the developer-specific local environments. Yeah, that's how we're doing things at the mo. I'd like to move more (ideally all) of the config to source files, but we're not there yet.
Thanks,
Andy
. . . . . . . Andy Fowlston +44 (0)20 8747 5068 andy@pedalo.co.uk Skype: andy.pedalo www.pedalo.co.uk
This email is intended only for the above named addressee/s. This email may be confidential or legally privileged. If you have received this email and you are not a named addressee, you must not use, copy, distribute or disclose the email or any part of its contents or take any action in reliance on it. If you have received this email in error, please email the sender by replying to this message and delete it from your system. All reasonable precautions have been taken to ensure no viruses are present in this email.
pedalo limited cannot accept responsibility for loss or damage arising from the use of this email or attachments and recommends that you subject these to your virus checking procedures prior to use. Any views or opinions presented are solely those of the author and not necessarily those of Pedalo Limited
Please consider the environment before printing this email
Also, we run into the odd issue where files uploaded by developers on thier local copies conflict with files uploaded on the staging server (test.txt, anyone?).
To clarify on the above - developers don't add files to git on their local copies, but a test file will sit in the developer's files directory until manually deleted. If a file with the same name is uploaded to the canonical server, and then pushed to git, the developer will need to resolve the conflict before he can merge the latest updates with his local site. On 11-02-18 11:41 AM, Brian Vuyk wrote:
Ah, I misunderstood.
We take it on a case-by-case basis.
1. The imagecache directory is always excluded, as those are automagically regenerated anyways if they don't exist. We also exclude the 'js' and 'css' folders, as putting the aggregated js and css files in git is pointless. 2. We generally exclude the files directory altogether, and just put up with broken images on our local dev environments. 3. If absolutely necessary, we may rsync the files directory down to a dev environment periodically.
We have on occasion tried to keep small copies of the files directory in git, but we run into problems where nodes are deleted on the site, and Drupal deletes the corresponding file from the filesystem. However, having not been removed from git with 'git rm <filename>', someone has to push the filesystem changes with 'git add -u' to make the files directory sync properly. Also, we run into the odd issue where files uploaded by developers on thier local copies conflict with files uploaded on the staging server (test.txt, anyone?). This often needs sorting out...
All-in-all, it's extra work with little extra gain, other than prevent some broken images or file links on the non-canonical copies of the site, which can be fixed as needed by rsyncing the files directory down to the non-canonical development copies.
In short, I keep all the site code in git. For anything that's not code (or part of a code package, such as theme or module images), I try to leave out unless there is a good reason for inclusion.
Brian
On 11-02-18 11:18 AM, Andy Fowlston wrote:
Brian wrote:
From a db point of view this sounds very similar to what we do. Could you elaborate on how you keep the files in sync? Keep which files? The database exports? Sorry, no not the database at all: I'm wondering about any content related files in the files directory (I presume you use source control for everything outside the files directory?).
We look at it this way - staging is the canonical database. All config is done on staging, never in the developer-specific local environments. Yeah, that's how we're doing things at the mo. I'd like to move more (ideally all) of the config to source files, but we're not there yet.
Thanks,
Andy
. . . . . . . Andy Fowlston +44 (0)20 8747 5068 andy@pedalo.co.uk Skype: andy.pedalo www.pedalo.co.uk
This email is intended only for the above named addressee/s. This email may be confidential or legally privileged. If you have received this email and you are not a named addressee, you must not use, copy, distribute or disclose the email or any part of its contents or take any action in reliance on it. If you have received this email in error, please email the sender by replying to this message and delete it from your system. All reasonable precautions have been taken to ensure no viruses are present in this email.
pedalo limited cannot accept responsibility for loss or damage arising from the use of this email or attachments and recommends that you subject these to your virus checking procedures prior to use. Any views or opinions presented are solely those of the author and not necessarily those of Pedalo Limited
Please consider the environment before printing this email
Brian wrote:
we run into problems where nodes are deleted on the site, and Drupal deletes the corresponding file from the filesystem. However, having not been removed from git with 'git rm <filename>', someone has to push the filesystem changes with 'git add -u' to make the files directory sync properly. <snip> This often needs sorting out...
That's *exactly* the issue I'm running into!
All-in-all, it's extra work with little extra gain <snip> I try to leave out unless there is a good reason for inclusion.
That seems to be the consensus so far :) Thanks again for your time, Andy . . . . . . . Andy Fowlston +44 (0)20 8747 5068 andy@pedalo.co.uk Skype: andy.pedalo www.pedalo.co.uk This email is intended only for the above named addressee/s. This email may be confidential or legally privileged. If you have received this email and you are not a named addressee, you must not use, copy, distribute or disclose the email or any part of its contents or take any action in reliance on it. If you have received this email in error, please email the sender by replying to this message and delete it from your system. All reasonable precautions have been taken to ensure no viruses are present in this email. pedalo limited cannot accept responsibility for loss or damage arising from the use of this email or attachments and recommends that you subject these to your virus checking procedures prior to use. Any views or opinions presented are solely those of the author and not necessarily those of Pedalo Limited Please consider the environment before printing this email
Why not use drush with site aliases? http://drupal.org/node/670460 On Fri, 2011-02-18 at 14:35 +0000, Andy Fowlston wrote:
Thanks for the input Randy. That certainly keeps life simpler, though it still feels a bit funny having the markup stored in the repository (via the db dump) but leaving out the assets they refer to. (Btw I'd never consider keeping the files directory under source control after going live, and I will always use source control for non-content assets: the question is only about setting up the initial content that the site will go live with.)
Thanks again, I appreciate your insight.
Andy
. . . . . . . Andy Fowlston +44 (0)20 8747 5068 andy@pedalo.co.uk Skype: andy.pedalo www.pedalo.co.uk
This email is intended only for the above named addressee/s. This email may be confidential or legally privileged. If you have received this email and you are not a named addressee, you must not use, copy, distribute or disclose the email or any part of its contents or take any action in reliance on it. If you have received this email in error, please email the sender by replying to this message and delete it from your system. All reasonable precautions have been taken to ensure no viruses are present in this email.
pedalo limited cannot accept responsibility for loss or damage arising from the use of this email or attachments and recommends that you subject these to your virus checking procedures prior to use. Any views or opinions presented are solely those of the author and not necessarily those of Pedalo Limited
Please consider the environment before printing this email
From: development-bounces@drupal.org [mailto:development-bounces@drupal.org] On Behalf Of Randy Fay Sent: 18 February 2011 13:51 To: development@drupal.org Subject: Re: [development] Workflow: staging server, source control, and the files directory
My opinion is that nothing in the files directory should ever be in source control - that will just cause pain. Those are transient, website-created files. They even have the wrong permissions typically to be managed successfully. They should be backed up, and on fancier website configurations need to be shared or synced, but should not be in source control.
If you have files that *are* "assets", but not user-generated (like graphics files that don't go with the code) they might go in an "assets" directory at the webroot or above, and be under source control there.
-Randy On Fri, Feb 18, 2011 at 3:14 AM, Andy Fowlston <andy@pedalo.co.uk> wrote: Hi all,
I have a question about people's practices using a staging server and source control. I'm keen for us to have a sensible workflow while developing sites, and currently we have the following setup.
* Developers have local copies of the site where they develop new functionality, which they check in to source control (Subversion). * The staging server has a checked out version of the repository. * The repository has a post-commit hook that automatically updates the staging server. * There are some changes we make directly via the staging server (ie not in code) and for that there's a script that clears caches, dumps the database, and stores that in the repository. (We try to keep as much in code as possible.)
Things have been working well so far, but just recently we started adding content to the site (on the staging server). The problem comes with uploading files via the wysiwyg editor, as this bypasses the normal mechanism of checking into source control and letting the post-commit hook update the staging server automatically. So the question is, what's the best way to keep the staging server's files directory synced with the repository? I'd like to keep things as automated as possible. Here's some ideas:
* On some event (eg. post-commit, cron) I just do something like a simple svn add files/*. This won't prune nicely though. * Exclude the files directory from source control altogether, and change the local update process to just copy from the staging server's files directory to the local. This would work, just feels slightly wrong to me as the db contains content that refers to the files, and that *is* in source control. * Maybe I could treat the files directory as a vendor branch and use something like svn_load_dirs.pl to try and keep them in sync. I've not used this script before, and don't know if it would manage this simple situation automatically, or if it would need some manual help. I suppose I could also write a script to parse the output of svn st, but as always would prefer to keep the workload as low as possible :) * Nice super simple idea I've not considered?!
I'd love to hear what other people in this situation do, or just any ideas at all.
Many thanks,
Andy
. . . . . . . Andy Fowlston +44 (0)20 8747 5068 andy@pedalo.co.uk Skype: andy.pedalo www.pedalo.co.uk
This email is intended only for the above named addressee/s. This email may be confidential or legally privileged. If you have received this email and you are not a named addressee, you must not use, copy, distribute or disclose the email or any part of its contents or take any action in reliance on it. If you have received this email in error, please email the sender by replying to this message and delete it from your system. All reasonable precautions have been taken to ensure no viruses are present in this email.
pedalo limited cannot accept responsibility for loss or damage arising from the use of this email or attachments and recommends that you subject these to your virus checking procedures prior to use. Any views or opinions presented are solely those of the author and not necessarily those of Pedalo Limited
Please consider the environment before printing this email
On Feb 18, 2011, at 3:14 AM, Andy Fowlston wrote:
* Exclude the files directory from source control altogether, and change the local update process to just copy from the staging server's files directory to the local. This would work, just feels slightly wrong to me as the db contains content that refers to the files, and that *is* in source control.
It could become ungainly if you had a lot of uploaded files, but you in theory *could* keep the raw image files in source control. However, by all means, you should exclude imagecache (aka "image" module in D7) renderings from source control as your local environment will generate those anyway, and we've found that those can cause all kinds of conflicts between what Drupal and Git think are fresh. Nothing like a stupid conflict between copies of a non-critical image file to gum up what should be a quick merge. (I know you mention subversion, but the theory's the same.) In general, though, unless you're working on the theme, having broken images in your local instance is not deadly, in my experience. Laura Scott http://pingv.com
participants (5)
-
Andy Fowlston -
Brian Vuyk -
Laura Scott -
Randy Fay -
Wesley Nichols