[development] Workflow: staging server, source control, and the files directory

[Brian Vuyk]

On a tangential note, I am not sure I would consider it 'best practice' 
to keep a database dump in the repository, at least once it gets larger 
than a few hundred mb or so, especially if you commit updated db dumps 
often:

1. DB dumps are not incrementally updated, so it can't be stored 
efficiently (ie, a delta between revisions usually isn't smaller than 
the entire dump).
2. Some source control systems (I'm looking at you, git...) choke with 
large files (>100mb).

Specifically with Git, you start running into memory allocation errors 
with 'git merge' and 'git gc'; they can be worked around to a certain 
degree by using some arguments to limit the amount of memory available 
for packing the repository, but as your repository size continues to 
grow, packing with limited memory becomes less and less efficient, and 
the problem starts feeding on itself.

Eventually, it gets to a point where normal git operations such as pulls 
or merges take a significant amount of time, but a call to 'git gc' to 
clean and optimize the repository sends the server into swap memory 
until you kill the command.

What we currently do at PINGV is use 'Backup & Migrate' on all of our 
staging sites. We have regular backups of the staging server being made, 
while at the same time, those of us with local environments just 
download updated DB exports as necessary to grab the latest config. This 
keeps our git repository much smaller, faster, and memory efficient.


On 11-02-18 09:35 AM, Andy Fowlston wrote:
> Thanks for the input Randy. That certainly keeps life simpler, though it still feels a bit funny having the markup stored in the repository (via the db dump) but leaving out the assets they refer to. (Btw I'd never consider keeping the files directory under source control after going live, and I will always use source control for non-content assets: the question is only about setting up the initial content that the site will go live with.)
>
> Thanks again, I appreciate your insight.
>
> Andy
>
> . . . . . . .
> Andy Fowlston
> +44 (0)20 8747 5068
> andy at pedalo.co.uk
> Skype: andy.pedalo
> www.pedalo.co.uk
>
> This email is intended only for the above named addressee/s. This email may be confidential or legally privileged. If you have received this email and you are not a named addressee, you must not use, copy, distribute or disclose the email or any part of its contents or take any action in reliance on it. If you have received this email in error, please email the sender by replying to this message and delete it from your system. All reasonable precautions have been taken to ensure no viruses are present in this email.
>
> pedalo limited cannot accept responsibility for loss or damage arising from the use of this email or attachments and recommends that you subject these to your virus checking procedures prior to use. Any views or opinions presented are solely those of the author and not necessarily those of Pedalo Limited
>
> Please consider the environment before printing this email
>
> From: development-bounces at drupal.org [mailto:development-bounces at drupal.org] On Behalf Of Randy Fay
> Sent: 18 February 2011 13:51
> To: development at drupal.org
> Subject: Re: [development] Workflow: staging server, source control, and the files directory
>
> My opinion is that nothing in the files directory should ever be in source control - that will just cause pain. Those are transient, website-created files. They even have the wrong permissions typically to be managed successfully. They should be backed up, and on fancier website configurations need to be shared or synced, but should not be in source control.
>
> If you have files that *are* "assets", but not user-generated (like graphics files that don't go with the code) they might go in an "assets" directory at the webroot or above, and be under source control there.
>
> -Randy
> On Fri, Feb 18, 2011 at 3:14 AM, Andy Fowlston<andy at pedalo.co.uk>  wrote:
> Hi all,
>
> I have a question about people's practices using a staging server and source control. I'm keen for us to have a sensible workflow while developing sites, and currently we have the following setup.
>
> * Developers have local copies of the site where they develop new functionality, which they check in to source control (Subversion).
> * The staging server has a checked out version of the repository.
> * The repository has a post-commit hook that automatically updates the staging server.
> * There are some changes we make directly via the staging server (ie not in code) and for that there's a script that clears caches, dumps the database, and stores that in the repository. (We try to keep as much in code as possible.)
>
> Things have been working well so far, but just recently we started adding content to the site (on the staging server). The problem comes with uploading files via the wysiwyg editor, as this bypasses the normal mechanism of checking into source control and letting the post-commit hook update the staging server automatically. So the question is, what's the best way to keep the staging server's files directory synced with the repository? I'd like to keep things as automated as possible. Here's some ideas:
>
> * On some event (eg. post-commit, cron) I just do something like a simple svn add files/*. This won't prune nicely though.
> * Exclude the files directory from source control altogether, and change the local update process to just copy from the staging server's files directory to the local. This would work, just feels slightly wrong to me as the db contains content that refers to the files, and that *is* in source control.
> * Maybe I could treat the files directory as a vendor branch and use something like svn_load_dirs.pl to try and keep them in sync. I've not used this script before, and don't know if it would manage this simple situation automatically, or if it would need some manual help. I suppose I could also write a script to parse the output of svn st, but as always would prefer to keep the workload as low as possible :)
> * Nice super simple idea I've not considered?!
>
> I'd love to hear what other people in this situation do, or just any ideas at all.
>
> Many thanks,
>
> Andy
>
> . . . . . . .
> Andy Fowlston
> +44 (0)20 8747 5068
> andy at pedalo.co.uk
> Skype: andy.pedalo
> www.pedalo.co.uk
>
> This email is intended only for the above named addressee/s. This email may be confidential or legally privileged. If you have received this email and you are not a named addressee, you must not use, copy, distribute or disclose the email or any part of its contents or take any action in reliance on it. If you have received this email in error, please email the sender by replying to this message and delete it from your system. All reasonable precautions have been taken to ensure no viruses are present in this email.
>
> pedalo limited cannot accept responsibility for loss or damage arising from the use of this email or attachments and recommends that you subject these to your virus checking procedures prior to use. Any views or opinions presented are solely those of the author and not necessarily those of Pedalo Limited
>
> Please consider the environment before printing this email
>
>
>