[development] Solving the dev->staging->live problem

[Michael Haggerty]

Reading through this, I got the sense that not everyone is talking  
about the same thing. Something that may be useful is to break the  
problem down into the various areas of staging that need to be  
accounted for.

In Drupal, you really need to stage about 5 types of things (this list  
can grow a bit):

1) Themes - changes to a theme should not be made on a live server

2) Modules - esp. if you are developing a custom module, it should not  
be made on a production server.

3) Content - content CAN be staged on a production server with  
Drupal's publish and workflow features.

4) Dynamic Content - dynamic content cannot really be staged on a  
production server, from the standpoint that you are not going to see  
the different permutations of blocks while information is not published.

5) Users - if you are setting up a bunch of new users, staging them on  
a production server is not a good idea.

There are some strategies I have used in the past to accomplish all of  
these things. Each solution was specific to a client. Short  
description follows, and would love to hear everyone's feedback.

1) Staging themes / moving to production: we had a client who needed  
to alter the layout to specific pages in their site regularly for a 24  
hour period (then change everything back). We set up a staging server  
just for these takeovers. It had a physical reproduction of Drupal and  
the production database. The client would copy the theme under a  
different name, make adjustments to the CSS, and rsync it over to the  
production servers through a simple script.

Simply changing the theme was not a good idea for their purposes, they  
had a lot of blocks that were displaying on different pages in the  
application. We wrote a small module that duplicated records in the  
block table for the new theme, then implemented it. We also added some  
logic for doing this based on time of day.

2) Staging custom modules: the hardest part about staging custom  
modules is ensuring they work with live data. Selenium is a great tool  
for writing short functional tests, we did this extensively on a  
couple of sites and trained the client to construct and perform  
automated functional tests each time they pushed things into production.

3) Content Staging - One of the things people don't always understand  
about staging content is that there are ample workflow controls in  
Drupal to allow users to submit content for editing and review prior  
to publication. We have used workflow + views to create queues for  
content that needs to be reviewed, controlled access to those queues  
based on user role, and used views to track overall workflow  
statistics based on the time things were published. This has been  
effective for everything from small non-profits to large publishing  
sites.

4) Dynamic Content - Sometimes editors want to see the impact content  
will have on dynamically generated pages before publication. This  
comes up especially when we are using panels. Have not found a good  
overall solution and tend to work things out with clients on a case by  
case basis. Some strategies include writing small modules to export /  
import nodes after publication, using web services to grab nodes that  
have been published on a staging site, and cut and paste.

5) Use Staging - Occasionally, we have to import large numbers of  
users on a regular basis. Staging users is always a tricky issue, even  
when we are using open authentication standards like openid. The issue  
is typically getting the permissions right, it can become hard to use  
Drupal's user admin interface when we are talking about 5000 non- 
alphabetical user records. Typically, we write a script to do the  
import and modify user permissions.

This does not deal with moving code from dev into staging, but really  
is just about that QA that needs to happen before moving things into  
production.

M

On Aug 11, 2008, at 10:56 AM, Alex Barth wrote:

>
> On Aug 10, 2008, at 12:22 PM, Greg Dunlap wrote:
>> On Sun, Aug 10, 2008 at 6:06 AM, Victor Kane <victorkane at gmail.com>  
>> wrote:
>>
>> Alex Barth recently contact me about exactly this, having come to  
>> the exact same conclusion. He has opened an issue in deploy's issue  
>> queue:
>>
>> http://drupal.org/node/291921
>>
>> He himself wrote a module quite similar to Deploy called Ports  
>> which is pretty interesting and abstracts import/export a bit  
>> beyond what I did. Maybe he can chime in here about what he's done  
>> because I've only taken really a cursory glance at it.
>>
>
> I'm happy that this discussion is coming up.
>
> I started writing port module for generating installer profiles and  
> was then pleasantly surprised to see similarities with Greg's deploy  
> module.
>
> At port module's core there is a very simple idea: provide a hook to  
> let modules define a matching export and import function. Here is  
> the ports hook definition for imagecache for example:
>
> // Implementation of hook_ports()
> function imagecache_ports() {
>  $ports = array();
>  $ports['imagecache_presets']['name'] = t('Image cache presets');
>  $ports['imagecache_presets']['export callback'] =  
> 'imagecache_export_presets';
>  $ports['imagecache_presets']['import callback'] =  
> 'imagecache_import_presets';
>  $ports['imagecache_presets']['type'] = PORT_STRUCTURE; // not yet  
> implemented
>  $ports['imagecache_presets']['version'] = '1.68.2.3';  // not yet  
> implemented
>  return $ports;
> }
>
> This simple approach is pretty powerful, because it makes it really  
> easy to generate a single export array that contains all the  
> information for importing it on a target site. In its simplest  
> applications, you can copy / paste this array to your target site or  
> you can use it to generate install functions for an install profile   
> (both operations currently supported by port module).
>
> After doing a first proof of concept as a result of a client  
> project, I saw how similar certain approaches in deploy module are  
> and got in touch with Greg. I had another look at deploy module this  
> weekend: I'm actually thinking that a deploy like module and a port  
> like module could play very well together by port providing the  
> structure in which modules should export and import configuration  
> and deploy providing the XML-RPC integration, deployment  
> functionality and the UI.
>
> Another module that could be implemented on top of port module is  
> the install profile wizard (recent discussion on a related thread: http://drupal.org/node/230059#comment-957328) 
> .
>
> That said, port is a proof of concept, this is the current status  
> and limitations:
>
> # Basic hook_ports() system defined
> # UI for copy/paste deployment of export data
> # UI for generating callable PHP functions
> # on-behalf implementations for menus, user roles and permissions,  
> module status, content types, imagecache, nodeprofile and spaces.
> # while there is a flag for PORT_STRUCTURE and PORT_CONTENT I  
> haven't thought deeper about content export/import - I'm just  
> thinking that it's a very closely related problem
> # while there's a slot for version number, port isn't handling any  
> version comparisons atm
> # there is no 'update' handling, no notion of mapping
> # there is no way of exporting parts of a modules configuration yet.  
> While modules can define more than one export / import pair, there's  
> no way of exporting just a part of one export function. e. g. export  
> only certain imagecache definitions, not all of them. this one  
> should be easy.
>
> Ideas for update handling in port: I'm leaning towards not dealing  
> with it on this level. In the existing implementation modules would  
> deal with create new vs update by themselves. I haven't thought much  
> about this though and there might be a smart helper that port could  
> provide so that we know on this level what's an update and what's new.
>
> Plans for port: don't know yet. I'm in touch with Greg on deploy and  
> I plan to get in touch with Boris Mann on install profile wizards.  
> These are the two projects where I see overlap.
>
> I'd be very curious to get your thoughts on port module's approach.
>
> Check out the code here:
> http://cvs.drupal.org/viewvc.py/drupal/contributions/sandbox/alex_b/port/
>
> -
>
> Alex Barth
> http://www.developmentseed.org/blog
> tel (202) 250-3633
>
>
>
>