[development] SoC 2007 proposal - Drupal automated staging toolkit (was Summer of Code 2007 Idea - Drupal software appliance)

Thu Mar 22 18:59:05 UTC 2007

I've submitted my SoC 2007 Drupal proposal to Google. Thanks to
everybody for the feedback. A formatted PDF version is here:
http://www.abeharry.info/SoC2007_DrupalAST_FullProposal.pdf Here is
the abstract:

The Drupal automated staging toolkit is a proposed set of code
libraries, file schemas and parsers, and code generators, for
automatically creating a Drupal site with specific module code
versions, sample users and data, and a specific LAMP stack
configuration for hosting the Drupal site. The toolkit also has the
ability to stage this generated site on an existing physical server
location, and also as a self-contained virtual machine consisting of a
minimal Linux environment, required LAMP software, and the Drupal
site.

The automated staging toolkit is designed to be part of an automated
unit- and regression testing environment, by providing testers with a
simple, fast way to automatically generate a complete Drupal site
running specific code versions, and using specific LAMP server
configurations. It is also intended for use as part of a performance
and scalability testing environment by providing the ability to
rapidly build and then benchmark the effects of different application,
web and database server configurations on Drupal site performance and
scalability.

This toolkit will be used in the following way:
1. The tester creates or reuses an XML(or other structured) file using
a schema describing the Drupal site code-tree, including modules
installed/enabled/disabled, and the versions of each module to be
used.
2. The tester creates or reuses an XML file using a schema describing
         the LAMP stack web server, PHP/application server, and
database server configuration; e.g Apache vs. Lighttpd, mod_php
vs.FastCGI, choice of op-code cache, MySQL vs. PostgreSQL, and so on.
3. The tester creates or reuses an XML file using a schema describing
the sample users and content data the site will contain.
4. The parsers take each file and generate scripts in a lightweight
language (Python or Ruby or PHP-CLI.) These scripts, when executed,
use functions in the code libraries to download Drupal modules,
generate database scripts and datasets, and write server configuration
files.
5. Given a physical server target location, the Drupal modules,
database scripts and server configuration files are deployed to the
designated server location, to produce a new, ready-to-test Drupal
site.

Time and resource permitting, a builder in the toolkit will also use
the generated Drupal site and servers' configuration as input to build
a self-contained virtual machine image in Xen, VMWare, or potentially
the Amazon EC AMI format. This virtual image can also be be used in
testing environments, including advanced performance testing scenarios
such as evaluating clustering, distributed database topologies, and
alternative storage and computing models like Amazon S3 and Elastic
Cloud. The ability to rapidly generate self-contained Drupal virtual
machine images will also be extremely valuable to Drupal consultants
and solution providers for marketing, prototyping and demonstrating
Drupal solutions to potential clients, and large organizations and
ASPs like CivicSpace looking to take advantage of the massive benefits
of virtualization technology from VMware, Xensource, Amazon, and the
like.

Allister.

On 19/03/07, KevinImNotSpacey <kevin.amerson at gmail.com> wrote:
> Hey Allister,
>
> I can't speak to Google SoC requirements, but your idea sounds great.  I
> could see some code being written to automate setting up the environment,
> perhaps that could be the focus.  I know there is a set of repetitive tasks
> that we go through whenever evaluating a new module or upgrading an existing
> one:
>
> 1) Get the module from drupal.org
> 2) Stage our database and other contrib modules
> 3) Copy over the module code and enable it through the admin UI
> 4) Test expected behaviors
>
> A separate but somewhat related question/task/complicated activity is moving
> changes between environments, such as moving a discrete change from dev ->
> integration -> test -> production.  There is a module called importexportapi
> that does some of this, but no real complete solution.  Perhaps your project
> could include a methodology + utility to allow us to hot deploy drupal
> changes to a prod environment.
>
> Kevin
>
>
>
> On 3/19/07, Allister Beharry <allister.beharry at gmail.com> wrote:
> > Thanks for the heads-up Robert. I've been reading the SoC project
> > proposal for automating testing of patches, and also your posts on a
> > performance and scalability testing environment for Drupal. Virtual
> > machine images have been posited as part of an solution for both. So
> > I'm going to channel Laskey wrt my original idea and propose a better
> > one. (I'm just punching this out before I race to class so it's gonna
> > be brief/fragmented with typos. I'll elaborate further when I get
> > back.)
> >
> > A key requirement of automated testing of Drupal is the ability to
> > create a Drupal site with a specific configuration. In the case of
> > unit testing patches and code and also regression testing, the Drupal
> > site must be running specfic versions of core and module code. Now
> > automating tests is one thing but at the end-of-the day a human will
> > have to be able to eyeball a Drupal site with specific versions of
> > core and module code. There needs to be an automated way for a tester
> > to create a complete Drupal sit by simply specifiying what code
> > versions he/she wants running
> >
> > For performance and scalability testing,a human tester will want to
> > vary things like choice of web server - Apache, lightpd, IIS - choice
> > of db - MySQL, PostgreSQL - PHP configuration - mod_pho, FastCGI, PHP
> > enhancements like APC, Zend optimizer, reverse proxies, as well as
> > varying load parameters for the Drupal site as you mentioned - number
> > of core/contrib modules, number of databases, memcache, database
> > topography, so on.
> > You'll also need to populate the db with sample content and user
> > account. For testing on Amazon EC you'd need an AMI image
> >
> > What is needed is some kind of interface or configuration script that
> > a tester can use to quickly generate a Drupal site configured with
> > these parameters.  The script interpreter would take a configuration
> > script and a designated target, and spit out:
> > 1. A ready to run tarball with drupal settings.php + php.ini +
> > database script  thatcan be immediately deployed to an existing web +
> > database server setup
> > 2. A virtual machine image in either VMWare or Xen or AMI format.
> >
> > Well it sounds kind of ambitious, but even if done on a scaled back
> > way, could have applications well outside Drupal. What do you think?
> >
> > Allister.
> >
> > On 19/03/07, Robert Douglass <rob at robshouse.net> wrote:
> > > Allister,
> > >
> > > before I discuss your idea, it should be noted that you can't propose a
> > > GSoC project which doesn't produce code. As you describe it, your idea
> > > is mostly a task for a sysadmin and doesn't produce code, therefore
> > > wouldn't be eligible for the program.
> > >
> > > Your idea, however, is very good. I had envisioned a similar use of
> > > virtual images here: http://groups.drupal.org/node/2742
> The problem with
> > > doing my idea as GSoC is that it is more than one person could
> > > reasonably manage, I think, and is dependent on infrastructure that
> > > we're not in a position to provide.
> > >
> > > I think what is needed is some brainstorming that combines your software
> > > appliance with performance, scalability and regression testing. Please
> > > chew on these ideas and come up with an alternate proposal. We'd love to
> > > have you apply to GSoC with a Drupal project.
> > >
> > > cheers,
> > >
> > > Robert
> > >
> > > Allister Beharry wrote:
> > > > Hello all,
> > > > I'll be participating in SoC 07 and I'd appreciate it if I can get
> > > > some feedback on my idea for a project with Drupal. Here is the
> > > > synopsis:
> > > >
> > > > My project idea for SoC 2007 is a Drupal software appliance – a
> > > > self-contained virtual image consisting of a minimal Linux
> > > > environment, a LAMP stack, and Drupal 5.1  – which can be distributed
> > > > as a bootable LiveCD or virtual machine image which can be run on
> > > > virtualization software like VMware Player or MS Virtual PC.
> > > >
> > > > At ~50Mb or less, a Drupal appliance is a perfect way for potential
> > > > users with no system administration experience to evaluate Drupal.
> > > > Advocacy for Drupal can be extended by distributing the Live CD with
> > > > magazines and in trade shows. Drupal companies and consultants can
> > > > rapidly prototype and demonstrate a Drupal a solution to their clients
> > > > starting with the base appliance image. Drupal admins can easily test
> > > > module patches before deployment to live sites. The virtual machine
> > > > image can also be part of an automated testing framework by utilizing
> > > > the scripting capabilities of virtualization software like VMware
> > > > Server.
> > > >
> > > > This project can build on existing LiveDistro images from Knoppix or
> > > > Damn Small Linux, and virtual appliance images like those from Virtual
> > > > Appliance (
> http://virtualappliances.net/products/lamp/). The main
> > > > tasks will be:
> > > > 1.Collecting requirements from the Drupal community;
> > > > 2.Building the target environment;
> > > > 3.Installing and integrating Drupal and necessary modules;
> > > > 4.Comprehensive testing of the entire appliance;
> > > > 5.Coding hooks in the virtualization software for use of the appliance
> > > > by automated testing tools;
> > > > 6.Promotion and obtaining distribution channels for the release of the
> > > > appliance.
> > > >
> > > > Thanks a lot.
> > > > Allister
> > > >
> > >
> > >
> > > --
> > > * * * * *
> > > Lullabot's First Ever Advanced Workshops Are Here!
> > > Drupal API & Module Building - Advanced Drupal Themeing
> > > April 9th-13th - Providence, RI
> > > Early Bird Discounts Available Now
> > > http://www.lullabot.com/training
> > > * * * * *
> > >
> > >
> >
>
>