The right way to manage Drupal webiste development
Hi, We are 4 developpers using Drupal to realize Blogs and news portals. We use SVN to share drupal PHP code. Each developper has his own instance on his computer with his setting.php and mysql database. This is fine for a one shot development... BUT, (there is a but...) 1/ When we make some changes using the admin interface on a developer instance, we have to extract the DB and restore it into the integration instance... At this time each developper must import the integration DB in his instance each time there's an update. It's a huge work to determine what table is a content table, param table or system table... 2/ When our sites will be online, we will need to continue the development. The only way we found to manage this is to lock param tables on the live instance. And when we'll want to update the live website, we'll have to : - update the php code (thanks svn) - and (this is quite touchy) update the param tables using a sql import.... This is the only way we found today to manage our devs... We are looking for some best practice on this subject. Do you know http://drupal.org/project/DAST ? Do you think it can be a solution? -- Michel Lévy-Provençal +336 23 93 60 16 | mikiane.com
Michel Levy-Provençal wrote:
1/ When we make some changes using the admin interface on a developer instance, we have to extract the DB and restore it into the integration instance... At this time each developper must import the integration DB in his instance each time there's an update. It's a huge work to determine what table is a content table, param table or system table...
2/ When our sites will be online, we will need to continue the development. The only way we found to manage this is to lock param tables on the live instance. And when we'll want to update the live website, we'll have to : - update the php code (thanks svn) - and (this is quite touchy) update the param tables using a sql import....
This is an age old problem - how to merge database tables from multiple developers each working with their own instance of the database. Prior to deployment the problem is trivial. A staging server houses the 'master' database which has the canonical data/configuration. If developers want an up to date configuration in their development environment, they pull it from the staging server work on it and manually reproduce tested changes to the staging server. No attention needs to be paid to a live instance of a site, since it doesn't exit. This becomes complicated when a site goes live however. The staging server may still (from a development testing standpoint) contain the desired data/configuration - BUT - the truth is the live site (from a practical standpoint) *is canon*. How does one effectively push/merge changes to a live server that may now have dozens of administrators acting on and changing the configuration in real time. Or for that matter pull/merge the changes from the live site. Some configuration and content data can easily be separated, but there are significant numbers of tables that blur the line. CCK tables are a prime example of this. The tables for content types and their fields define the content type, its configuration and store the data. To add to the problem the table structures can change at a moments notice if a content type has a new field added to it. One partial solution to this sort of problem is to regularly dump the structures of the canonical database tables to files - and check them into the repository. The database structure check ins can be automated, and scripts can be run during developer *code* check in to diff the database structures. This will catch new/removed tables and changes to their structure but offers no means of automating data merges. Developer diligence is still required to merge data. A similar approach can be used in conjunction with the database structure check ins - where the actual data from *known* configuration tables is checked in to the repository. e.g. the system, variable or permission tables. In the case of variable or permission though you must have some sort of test in place to ensure that the data is consistent with the code (fixing a typo in a perm name in hook_perm has consequences in a live environment). Anyway - these are just some of my random thoughts on the topic. Perhaps this discussion can move to g.d.o. Maybe there are some new ideas on separation of configuration and data or automated test strategies to ensure code, data and configuration integrity that others want to share. andre
Michel Levy-Provençal wrote:
2/ When our sites will be online, we will need to continue the development. The only way we found to manage this is to lock param tables on the live instance. And when we'll want to update the live website, we'll have to : - update the php code (thanks svn) - and (this is quite touchy) update the param tables using a sql import....
At Advantage Labs, we find it quite handy to create a custom module for each project. Most Drupal developers probably do this anyway - for form_alter hooks, canned views, and other behavior overrides. We also log configuration changes in that module's .install file. There, we can put variable_set's, sql queries, node_type_save()'s, whatever we want. Deployment means: A. committing the SVN changes (and updating the instance, if necessary) B. running update.php This is certainly more time-intensive, but it pays off if you have concurrent development and/or any sort of staging/QA process. You can consistently QA and commit independent changes FWIW, we (and all other Advantage Labs Drupal hosting customers) also use a custom SVN process for Drupal management, where each commit by a developer results in 3 steps: 1. A pre-commit hook that "svn commit"'s the production site's data (for file uploads, database, etc. changes) 2. The actual commit by the developer, provided step #1 results in no merge errors. 3. An "svn update" on the production server instance to sync up to the developer's changes. At that point, for the process you're describing, we'd run update.php to bring in the config changes. Also, the developer can get a refreshed db dump that includes more of the production data and config changes. Allie Micka pajunas interactive, inc. http://www.pajunas.com scalable web hosting and open source strategies
Another idea we had today about this question... This is a chalkboard solution. An idea... Something that has to be simplified and tested. We gonna continue the reflexion and try to implement the solution we have found after... If somebody has any idea to simplify the process, you are welcome... _______________________________________________________________________ Imagine we have 3 instances: - Production site - Staging site - Integration site All the php code is commited by the different developpers on the Integration site. Every modification of the configuration (blocks, menus, views, ...) are made on the integration server using the Drupal GUI. For each version of the site, we use a special ID suffix for each record in the database. In the folowing example : - the Production website is in V0 and the Production DB is using 0000xxxxxx ids. - the Integration website is in V1 and the Integration DB is using 0001xxxxxx ids. while(0) { 1st Step: "Production to Staging synchro": The Staging DB is configured to be a slave of the Production DB to allow a synchronisation Production->Staging. 2nd step: "Integration to Staging synchro": We transfer the Integration php code to the Staging website. We change the configuration of the Staging DB: the Staging DB is now configured to be the slave of the Integration DB to allow a synchronisation Integration->Staging. Using suffixed ids allow us to limit conflicts between Integration and Production. At this step each new element "created" in the Integration cannot conflict with the elements from the Production. 3rd Step: "Staging period" The Staging DB is configured (another time) as slave of the Production DB. This allow us to tests the Staging as a real-time replication of the Production with the specific functions of the Integration. PROBLEM: At this step each update done on the Integration is lost.... The Production overides it. Only created elements are saved... (And this has to be fixed!!!!) 4rd Step: "Go Live!" When the website is correctly tested, we stop the Production website, dump the staging and transfer code and datas from Staging to Production. 5th Step and last step: "Update Integration Website" We transfer code and datas from Production to Integration. } 2007/8/17, Allie Micka <allie@pajunas.com>:
Michel Levy-Provençal wrote:
2/ When our sites will be online, we will need to continue the development. The only way we found to manage this is to lock param tables on the live instance. And when we'll want to update the live website, we'll have to : - update the php code (thanks svn) - and (this is quite touchy) update the param tables using a sql import....
At Advantage Labs, we find it quite handy to create a custom module for each project. Most Drupal developers probably do this anyway - for form_alter hooks, canned views, and other behavior overrides.
We also log configuration changes in that module's .install file. There, we can put variable_set's, sql queries, node_type_save()'s, whatever we want.
Deployment means: A. committing the SVN changes (and updating the instance, if necessary) B. running update.php
This is certainly more time-intensive, but it pays off if you have concurrent development and/or any sort of staging/QA process. You can consistently QA and commit independent changes
FWIW, we (and all other Advantage Labs Drupal hosting customers) also use a custom SVN process for Drupal management, where each commit by a developer results in 3 steps:
1. A pre-commit hook that "svn commit"'s the production site's data (for file uploads, database, etc. changes) 2. The actual commit by the developer, provided step #1 results in no merge errors. 3. An "svn update" on the production server instance to sync up to the developer's changes.
At that point, for the process you're describing, we'd run update.php to bring in the config changes. Also, the developer can get a refreshed db dump that includes more of the production data and config changes.
Allie Micka pajunas interactive, inc. http://www.pajunas.com
scalable web hosting and open source strategies
-- Michel Lévy-Provençal +336 23 93 60 16 | mikiane.com
On 8/21/07, Michel Levy-Provençal <mlevypro@gmail.com> wrote:
Another idea we had today about this question... This is a chalkboard solution. An idea... Something that has to be simplified and tested. We gonna continue the reflexion and try to implement the solution we have found after... If somebody has any idea to simplify the process, you are welcome...
Use the just released AutoPilot -- http://drupal.org/project/autopilot It does some interesting table inclusion / exclusion and works with SVN revisions to push between three systems as you describe. It's got rough edges and needs a fair amount of SysAdmin powers to get configured, but in it's ideal state you run build.example.com (a Drupal instance) and it manages remote systems for you. Needed: more work on content staging (dealing with the node table and friends) and a better / automatic way to look at DB schemas and infer what needs clobbering and what needs merging. -- Boris Mann Office 604-682-2889 Skype borismann http://www.bryght.com
Quoting Boris Mann <boris@bryght.com>:
Needed: more work on content staging (dealing with the node table and friends) and a better / automatic way to look at DB schemas and infer what needs clobbering and what needs merging.
Perhaps the http://drupal.org/project/importexportapi module could help with that. And add the excellent http://drupal.org/project/dba module to do an automated backup via the cron process. Earnie
Michel, I've felt this pain. And I wrote up my approach here: http://www.dave-cohen.com/node/1066 That's what I'm doing today on drupal 5.x development. I haven't checked my module into drupal.org because to make it work requires a bit of patching to drupal core. But I use it anyway, and suggest it to you. If you decide to go this route I'm happy to help. -Dave On Friday 17 August 2007 06:45, Michel Levy-Provençal wrote:
Hi,
We are 4 developpers using Drupal to realize Blogs and news portals. We use SVN to share drupal PHP code. Each developper has his own instance on his computer with his setting.php and mysql database.
This is fine for a one shot development... BUT, (there is a but...)
participants (6)
-
Allie Micka -
Andre Molnar -
Boris Mann -
Dave Cohen -
Earnie Boyd -
Michel Levy-Provençal