Local repository best practices
I've been doing some research for a client on best practices surrounding repository management, and wanted to take this discussion to the developer list and see if we could get some general consensus, and document it as a best practice in the handbooks. I found a bunch of discussion on this topic: * http://drupal.org/node/118936 * http://www.workhabit.com/labs/svn-repository-structure-drupal-projects * http://ceardach.com/blog/2008/06/development-environment-drupal * http://pingv.com/blog/cyberswat/200706/project-management-drupal * http://www.svntalk.com/node/6 * http://agaricdesign.com/agarics-svn-repository-structure-for-site-maintenanc... * https://svn.bryght.com/dev/browser * http://www.advantagelabs.com/advantage_labs_svn_repository_structure * http://www.codegobbler.com/svn-repository-structure-drupal-projects Most of these discussions are around Drupal *shops* who setup SVN to manage the development of their clients' sites, or who host their clients' Drupal sites and manage updates via SVN. However, I'm curious what people generally use in terms of repository structure/deployment strategy *after* development is complete, and they hand the site over to the client so that they can be self-sustaining? It doesn't really seem like they need a vendor branch with various versions of Drupal and contributed modules. The repository is around just their own site. We typically do something like this: /branches: This is where "crazy" development happens that will break other stuff. - /drupal-6-port - /crazy-new-feature /tags: This is where deployments go. Whenever we move a deployment to prod, we tag it first so that we can see a historical record of things that were put 'live' - /2008-09-03-description-of-deployment - /2008-09-08-description-of-deployment /trunk: The latest development code in preparation of deployment. Should be kept stable at all times. Minor fixes can go in here, but major ones (particularly porting tasks) get a dedicated branch. Beneath each directory above is something like this: - /www: Drupal files, modules, etc. - /assets: Things like wireframes, TinyMCE .zip files, and anything else that shouldn't be web acessible. - /db: Database backup files What are other people doing for this use case? -Angie
I doubt you'll find consensus, but good luck on trying ;). I'm writing this from the perspective of a client. I don't develop modules for other people, but just for use in house. Here's what I do, but I confess to not using svn for version management of modules we don't develop. Rather we rely on release module and the drupal.org infrastructure for releases we don't care about but rather only use svn for the code that's specific to us. Some of the folks at our college are trying out using SVN to manage drupal deployment, they often complain that it's more trouble than its worth. I organize my SVN tree around deployable units, and for me that's designed and organized around the test plan. That makes it easier to understand what's been tested and ready for deployment. themes/trunk - all the themes for my site custom_module_a/trunk The version tree for that module Projects that have database changes have a db_api directory within them, so that we can tag/branch accordingly. Here's some real examples of project paths. In the example ilc refers to a large project that handles independent learning contracts at my college. projects/themes/trunk/evergreen_app projects/ilc/trunk/db_api projects/ilc/trunk/modules/myilc projects/ilc/trunk/xsl projects/ilc/1.0/db_api projects/ilc/1.0/modules/myilc et.c. I then write svn export scripts to deploy the bundles to test and production sites. This leaves me free to deploy verson_1 of module_x or version 2 of module_y, and have separate test plans/ release dates for each of these. Doing this cuts down on the work of creating a bunch of branches every time I have a separate development track going on. I tag within the themes trunk at major theme rev changes. I explain this only to suggest that best practices should really be built around your sites test and deployment strategies. For small organizations with small number of projects, this might be properly organized around the site, but as you grow, (we use drupal for our school's portal) there's a lot of need to design svn repositiories around the project rather than the site. We do use tags branches and trunks as you've indicated. This may n ot be what you're asking and if not, I apologize. ;) Dave On Sep 9, 2008, at 1:15 PM, Angela Byron wrote:
I've been doing some research for a client on best practices surrounding repository management, and wanted to take this discussion to the developer list and see if we could get some general consensus, and document it as a best practice in the handbooks.
I found a bunch of discussion on this topic: * http://drupal.org/node/118936 * http://www.workhabit.com/labs/svn-repository-structure-drupal- projects * http://ceardach.com/blog/2008/06/development-environment-drupal * http://pingv.com/blog/cyberswat/200706/project-management-drupal * http://www.svntalk.com/node/6 * http://agaricdesign.com/agarics-svn-repository-structure-for-site- maintenance-and-web-development * https://svn.bryght.com/dev/browser * http://www.advantagelabs.com/advantage_labs_svn_repository_structure * http://www.codegobbler.com/svn-repository-structure-drupal-projects
Most of these discussions are around Drupal *shops* who setup SVN to manage the development of their clients' sites, or who host their clients' Drupal sites and manage updates via SVN.
However, I'm curious what people generally use in terms of repository structure/deployment strategy *after* development is complete, and they hand the site over to the client so that they can be self-sustaining? It doesn't really seem like they need a vendor branch with various versions of Drupal and contributed modules. The repository is around just their own site.
We typically do something like this:
/branches: This is where "crazy" development happens that will break other stuff. - /drupal-6-port - /crazy-new-feature
/tags: This is where deployments go. Whenever we move a deployment to prod, we tag it first so that we can see a historical record of things that were put 'live' - /2008-09-03-description-of-deployment - /2008-09-08-description-of-deployment
/trunk: The latest development code in preparation of deployment. Should be kept stable at all times. Minor fixes can go in here, but major ones (particularly porting tasks) get a dedicated branch.
Beneath each directory above is something like this:
- /www: Drupal files, modules, etc. - /assets: Things like wireframes, TinyMCE .zip files, and anything else that shouldn't be web acessible. - /db: Database backup files
What are other people doing for this use case?
-Angie
Webchick, When I hand over source code to a client, I tell them to treat it like a vendor branch (as documented in the subversion manual). Then, they might customize if further for themselves. But most of my clients would ask me to make changes. I would give them updated files, they'd run the update.php script and be good to go. I've considered documenting my own source code repository, as I've put a lot of thought into it. But have not had time to write it up. I use subversion, and arrange it like this: drupal/5/ contains everything for drupal-5 based sites (drupal/6/ for 6.x and so on) drupal/5/vendor/ contains unmodified drupal core. For example drupal/5/vendor/5.6 contains Drupal-5.6. drupal/5/custom contains drupal with my patches. Yes, from time to time drupal must be patched. For example, drupal/5/custom/5.6 is modified Drupal-5.6 drupal/5/modules contains all the modules I use in all my sites. Sometimes modules need patching, too. I don't bother to keep a vendor branch of each one. I just keep the version I prefer. Usually I've checked a particular version of the module out of drupal cvs. Sometimes I keep multiple versions of a module around, for example drupal/5/modules/location is the latest location module while drupal/5/modules/location-1.x is an older version. I also have directories drupal/5/themes and drupal/5/profiles. drupal/5/sites contains the "sites/all" directory of the sites I'm working on. For example, drupal/5/sites/orphanage/modules contains modules used by theorphanage.com, while drupal/5/sites/drupalforfacebook/modules contains modules used by drupalforfacebook.org. These directories do not actually "contain" modules. Instead, I use svn:externals to refer to modules in drupal/5/modules/. By using svn:externals, I can patch one of my modules for one site, check in the change, then all my sites get the fix when I update them. This approach may not work for everyone, but it works for me. drupal/5/work contains the root of each project I'm working on. Again, I use svn:externals here. Each work directory contains htdocs/ (an svn:externals link to drupal/5/custom/X.y), and also svn:externals to the drupal/5/sites/something. (I use some well-placed symbolic links to keep the site-specific code completely separate from Drupal core). The work directory also contains documentation and other files specific to a particular site, which are not part of the website itself. So the key feature of my approach is the use of the svn:externals property. Using this, I can change a module (or version of core) for one site, and easily propagate the change to other sites. Another recommendation of mine is to not make the drupal root directory the root of the project. Instead, put drupal in a directory (I call it htdocs/) and keep documentation (or comps, or anything not part of the site) in source control, but not separate from the website files. I think this is a great question and look forward to reading other responses. -Dave On Tue, 09 Sep 2008 16:15:17 -0400, "Angela Byron" <drupal-devel@webchick.net> said:
However, I'm curious what people generally use in terms of repository structure/deployment strategy *after* development is complete, and they hand the site over to the client so that they can be self-sustaining? It doesn't really seem like they need a vendor branch with various versions of Drupal and contributed modules. The repository is around just their own site.
I have a client that really wants to use GIT to manage their code. So, if anyone would like to chime in with non-svn development workflows/organization that would be great too. I'm also curious if people would expand a bit more about how they manage their DB changes. It sounds like angie/lullabot store a DB dump in the repository and then upload that with their deployment. Is this standard practice for other people? -Ryan
On Tuesday 09 September 2008 23:53:24 Ryan Cross wrote:
I have a client that really wants to use GIT to manage their code. So, if anyone would like to chime in with non-svn development workflows/organization that would be great too.
There are really an _enormous_ number of different ways one can use git to manage a drupal project, but the right implementation with git would be highly dependent on your client's workflow. For example, I've made my local development environment into something of a git proof-of-concept: I regularly rsync off of cvs.drupal.org to provide a local mirror (for speed purposes), pull those into git repos using git- cvsimport, then push them into bare git repos that I also maintain locally (I've yet to find just the right mojo to get git-cvsimport to interact directly with bare git repos). I do this for both contrib and core. Then, I structure out individual drupal installs using git subprojects, wherein each drupal module retains the remote reference to the appropriate location within the contrib repo, so it can automatically be aware of and gobble up any upstream changes. I can also make small tweaks to a particular install on a new branch, or I can also pop off a new branch if I'm looking to set up a new site with a slightly different configuration. Most of that is utter and complete overkill for only one machine (as I said, proof of concept!), but the place where it gets nifty is the push/pulling between my laptop and desktop. It's getting to easy to keep the two synced up in all the right ways that I find myself forgetting which machine I made a given change on - then laughing as I `git pull` and remember that it doesn't matter :) However, that kind of a setup probably wouldn't be much use for a single site. If your client is really gung-ho about using git, and they're sufficiently familiar with using git on the command line, AND there are a few folks who'll be working on the site simultaneously, I'd personally recommend pulling down as much of the necessary code via drush as possible to your dev server, then setting up each developer with access to the dev server's git repo; clone off of that, then let people push/pull to each other and the dev server basically willy nilly (git is largely capable of taking care of the details, barring big sticky conflicts) until you're ready to push out to live. From there, it starts getting more complicated, and into questions about deployment. And on that...
I'm also curious if people would expand a bit more about how they manage their DB changes. It sounds like angie/lullabot store a DB dump in the repository and then upload that with their deployment. Is this standard practice for other people?
You should probably have a looksee at last months' discussion: http://lists.drupal.org/pipermail/development/2008-August/030745.html
-Ryan
cheers, Sam
Hi, I actually use git to manage all my clients, (Including the ones with SVN as git has a great SVN connector). Basically I have my base systems which I use to roll out to all clients and then I create a local system which pulls from the base repository and then loads on any additional modules. To do any db updates I push this through the hook_update_n() and then I use update.sh http://drupal.org/node/233091 to automatically run the updates as soon as the new version is committed to the repository and it will put the system into maintenance mode and do the updates and take it out of maintenance module if there are no errors. That is for the test system. Then I use branches to build a release version and once this has past muster I use tag and then do a fetch in the live system to bring in the code and then I check out the release branch. Works really good. I just wish more people would use git. Also this means that I don't have to give people commit access to systems, then can just email me the changes and I can sign them off and move them into live. If there is a problem I can see there patches and who signed off the patch and who committed it. Git has a great chain of evidence. Nothing comes even close to this. Also because of the extremely strong branch management in git moving fixes between the rc branches and the master is usually a single command with no messing around with fixing up broken patches. You will have 1 problem with git. which is what I have. every other SRM pails in comparison, esp once you start using branches that way they are meant be. An example of this is that I am doing a lot of changes to simple access and upgrading to D6, and most of the commits that I do in 5, with git I can move to 6 with generally no intervention from me. So keeping the 2 releases in sync is very easy. please excuse the rant. Gordon. On 10/09/2008, at 2:53 PM, Ryan Cross wrote:
I have a client that really wants to use GIT to manage their code. So, if anyone would like to chime in with non-svn development workflows/organization that would be great too.
I'm also curious if people would expand a bit more about how they manage their DB changes. It sounds like angie/lullabot store a DB dump in the repository and then upload that with their deployment. Is this standard practice for other people?
-Ryan !DSPAM:1000,48c757c671618992556831!
Apparently, the Drupal e-commerce project is essentially being developed under GIT and synced from time to time to Drupal CVS. ----- Original Message ----- From: Ryan Cross To: development@drupal.org Sent: Wednesday, September 10, 2008 6:53 AM Subject: Re: [development] Local repository best practices I have a client that really wants to use GIT to manage their code. So, if anyone would like to chime in with non-svn development workflows/organization that would be great too. [...]
I'm very curious about GIT... Anyone have any response to the criticism that decentralized repositories (GIT) result in more need for manual merging than centralized (SVN) ? -Matt Frédéric G. MARAND wrote:
Apparently, the Drupal e-commerce project is essentially being developed under GIT and synced from time to time to Drupal CVS.
----- Original Message ----- From: Ryan Cross To: development@drupal.org Sent: Wednesday, September 10, 2008 6:53 AM Subject: Re: [development] Local repository best practices
I have a client that really wants to use GIT to manage their code. So, if anyone would like to chime in with non-svn development workflows/organization that would be great too. [...]
Quoting Ryan Cross <drupal@ryancross.com>:
I have a client that really wants to use GIT to manage their code. So, if anyone would like to chime in with non-svn development workflows/organization that would be great too.
I'm also curious if people would expand a bit more about how they manage their DB changes. It sounds like angie/lullabot store a DB dump in the repository and then upload that with their deployment. Is this standard practice for other people?
Maybe http://drupal.org/node/288873 will help. Earnie -- http://for-my-kids.com/ -- http://give-me-an-offer.com/
I wrote up how I move database changes from development to deployment a while back: http://www.dave-cohen.com/node/1779 Since then, there was a thread on this list about using unique/random ids in place of serial ids in drupal's db. I think that would be a great change and eliminate some of the hoops I jump through. If you're looking for standard practice, I think most shops share a common install (perhaps a copy of the live site). One developer makes configuration changes to that install, then the others copy the full database to their development copies. This is burdensome and blows away changes made to local copies. Managing this is a headache that gets discussed often. As far as I can tell there is no ideal solution. -Dave On Tuesday 09 September 2008, Ryan Cross wrote:
I'm also curious if people would expand a bit more about how they manage their DB changes. It sounds like angie/lullabot store a DB dump in the repository and then upload that with their deployment. Is this standard practice for other people?
participants (10)
-
Angela Byron -
Dave Cohen -
David Cohen
-
David Metzler -
Earnie Boyd -
Frédéric G. MARAND -
Gordon Heydon -
Matt Chapman -
Ryan Cross -
Sam Boyer