Best way to deploy Drupal.
Dear All, We are building a somewhat complex site based on Drupal which features redundant servers at all levels of the system. We are starting of with 12 machines. We currently have three environments: development, staging and production. We are having a hard time setting an automated deployment mechanism. I have searched on the archives but was able to find this thread only: http://lists.drupal.org/pipermail/support/2007-May/004711.html I have not been able to understand from that how other people out there are implementing automated deployment. I found a number of links outside drupal.org: http://stackoverflow.com/questions/377629/drupal-deployment-testing-dont-how... http://stackoverflow.com/questions/282858/drupal-source-control-strategy http://nicksergeant.com/blog/drupal/painless-drupal-revision-control-cvs-and... http://nicksergeant.com/blog/drupal/my-thoughts-small-scale-drupal-developme... http://www.workhabit.com/labs/autopilot http://www.dave-cohen.com/node/1066 http://www.garfieldtech.com/blog/drupal-dev-server I have been working with Drupal's DEPLOY Module (http://drupal.org/project/deploy). Also I am trying to contact Workhabit (i.e. Autopilot guys). I have not found yet a solid approach to automate deployment. Would anyone through me some pointers on how to do it? How are you guys doing this? Thank you so much, unai
"Automating" deployment? To what degree do you expect this to be automated? I can't think of any situation in which you'd want deployment to be done without observation and testing during the process. Can you elaborate on what sort of process you are expecting? -- Kathleen Murtagh On Mon, Jul 13, 2009 at 4:53 AM, Unai Rodriguez <me@u-journal.org> wrote:
Dear All,
We are building a somewhat complex site based on Drupal which features redundant servers at all levels of the system. We are starting of with 12 machines. We currently have three environments: development, staging and production.
We are having a hard time setting an automated deployment mechanism.
I have searched on the archives but was able to find this thread only: http://lists.drupal.org/pipermail/support/2007-May/004711.html
I have not been able to understand from that how other people out there are implementing automated deployment.
I found a number of links outside drupal.org:
http://stackoverflow.com/questions/377629/drupal-deployment-testing-dont-how... http://stackoverflow.com/questions/282858/drupal-source-control-strategy
http://nicksergeant.com/blog/drupal/painless-drupal-revision-control-cvs-and...
http://nicksergeant.com/blog/drupal/my-thoughts-small-scale-drupal-developme... http://www.workhabit.com/labs/autopilot http://www.dave-cohen.com/node/1066 http://www.garfieldtech.com/blog/drupal-dev-server
I have been working with Drupal's DEPLOY Module (http://drupal.org/project/deploy). Also I am trying to contact Workhabit (i.e. Autopilot guys).
I have not found yet a solid approach to automate deployment. Would anyone through me some pointers on how to do it? How are you guys doing this?
Thank you so much, unai -- [ Drupal support list | http://lists.drupal.org/ ]
Dear Kathleen, Thank you so much for your reply. I guess we would like our deployment process to be somewhat similar to what we use in other projects; a number of developers are writing code while testing on their local machine first (aka developer sandbox). Each developer's machine has a web server and database server with a copy of the code. Once a developer is sure that her stuff is working fine on her sandbox that is the time when she will commit it into our version control system (i.e. we use subversion). We have a server that takes care of the automated deployments. That is, any developer can access a web-based interface, input her user/password and click one button. This button will deploy the build which is currently on the trunk in subversion and put it into the development server; also some other things will take place like copying the proper settings.php file (in drupal's case) to the proper server(s). From the web-based interface a developer can deploy to any of the 3 environments: development, staging or production. Usually development is where developers test (plus their sandboxes). Our Quality Assurance team takes care of testing the staging and production servers. That means deployments to these two environments are more controlled. The deployment process is automated, meaning, you can push the code to any environment upon clicking one button. The process takes care of deploying the right code in the right place plus the configuration files (in this case settings.php). We have been doing this for a while, I think is pretty standard. Now, for Drupal is different since part of the "settings" are stored in the database, right? We are having a hard time getting the database settings to be applied over to the next environment (i.e. development to staging or staging to production). I have been testing the deploy module which is able to do something like this... My question is more along the lines of... What do you guys do? Do you just deploy manually (i.e. reproduce the settings from your sandboxes to development, then to staging then to production) or you found a way of having this automated? Thank you so much :-) unai On Mon, 13 Jul 2009 08:40:48 -0400, Kathleen Murtagh wrote:
"Automating" deployment? To what degree do you expect this to be automated?
I can't think of any situation in which you'd want deployment to be done without observation and testing during the process.
Can you elaborate on what sort of process you are expecting?
-- Kathleen Murtagh
On Mon, Jul 13, 2009 at 4:53 AM, Unai Rodriguez <me@u-journal.org> wrote:
Dear All,
We are building a somewhat complex site based on Drupal which features redundant servers at all levels of the system. We are starting of with 12 machines. We currently have three environments: development, staging and production.
We are having a hard time setting an automated deployment mechanism.
I have searched on the archives but was able to find this thread only: http://lists.drupal.org/pipermail/support/2007-May/004711.html
I have not been able to understand from that how other people out there are implementing automated deployment.
I found a number of links outside drupal.org:
http://stackoverflow.com/questions/377629/drupal-deployment-testing-dont-how...
http://stackoverflow.com/questions/282858/drupal-source-control-strategy
http://nicksergeant.com/blog/drupal/painless-drupal-revision-control-cvs-and...
http://nicksergeant.com/blog/drupal/my-thoughts-small-scale-drupal-developme...
http://www.workhabit.com/labs/autopilot http://www.dave-cohen.com/node/1066 http://www.garfieldtech.com/blog/drupal-dev-server
I have been working with Drupal's DEPLOY Module (http://drupal.org/project/deploy). Also I am trying to contact Workhabit (i.e. Autopilot guys).
I have not found yet a solid approach to automate deployment. Would anyone through me some pointers on how to do it? How are you guys doing this?
Thank you so much, unai -- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ]
There are three main methods for managing the database when deploying from one workspace to another. The most common is "write down, or script your database configuration settings and repeat them in the next workspace." This is where people generally start out as it is the easiest, conceptually, to understand. It is both prone to human error and tedious, though. The second method is "put everything in code and update functions." This is the most recommended method because it is scalable both for very small and very large projects. It is also the most complicated because it requires adept programmers and takes away all the nice gui tools that makes Drupal development so accessible. There is progress towards this front, though. There is a movement to provide more "exportables" that will allow a developer to use the web gui to configure something, export the settings, and place it into code. Views currently supports this method out of the box. The "features" module is starting to give exportable support to other areas. The last, and rarely used, method is "put the database in version control, and merge databases." This is probably the most difficult to understand conceptually, but fairly scriptable allowing non-programmer developers to use the web gui and contribute to a project. It is great for small projects, and becomes increasingly more difficult the larger and larger the database (the restriction is your developer's computing power, but a production database exceeding 500mb would start to require extra finesse to manage). The "dbscripts" module provides an extensive amount of scripts in order to use this method. I use my version control system to deploy to a workspace from the primary repository. I don't need to worry about settings.php because I do not include that in version control. I set the database configuration settings once, and then never erase that file again. If you do have other settings being used in settings.php, you could put the database connection settings, and any other workspace-specific settings in a separate file and include that file in settings.php. Manually updating the workspace-specific settings would then only be required when they change (e.g.: adding Google Analytics to production, while keeping it disabled in development and testing). I don't use a "one click" deploy, however, I generally just have to run "svn update" or "svn switch". If you use the "put everything in code and update functions" method, you can install drush and have drush run update.php for you. Your script would then just say something like, "svn update && drush updatedb" or "svn switch && drush updatedb". If you use the "put the database in version control, and merge databases" method, then you would restore the appropriate database for the given workspace. For development and testing, your script would just say something like, "svn update && ./dbscripts/restore.php none". For production, you would first merge the development and production databases, commit the merged databases and then deploy. The final deploy script would be something like, "svn switch <release branch> && ./dbscripts/restore.php production min". -- Kathleen Murtagh On Mon, Jul 13, 2009 at 9:04 AM, Unai Rodriguez <me@u-journal.org> wrote:
Dear Kathleen,
Thank you so much for your reply.
I guess we would like our deployment process to be somewhat similar to what we use in other projects; a number of developers are writing code while testing on their local machine first (aka developer sandbox). Each developer's machine has a web server and database server with a copy of the code. Once a developer is sure that her stuff is working fine on her sandbox that is the time when she will commit it into our version control system (i.e. we use subversion).
We have a server that takes care of the automated deployments. That is, any developer can access a web-based interface, input her user/password and click one button. This button will deploy the build which is currently on the trunk in subversion and put it into the development server; also some other things will take place like copying the proper settings.php file (in drupal's case) to the proper server(s).
From the web-based interface a developer can deploy to any of the 3 environments: development, staging or production. Usually development is where developers test (plus their sandboxes). Our Quality Assurance team takes care of testing the staging and production servers. That means deployments to these two environments are more controlled.
The deployment process is automated, meaning, you can push the code to any environment upon clicking one button. The process takes care of deploying the right code in the right place plus the configuration files (in this case settings.php).
We have been doing this for a while, I think is pretty standard.
Now, for Drupal is different since part of the "settings" are stored in the database, right? We are having a hard time getting the database settings to be applied over to the next environment (i.e. development to staging or staging to production).
I have been testing the deploy module which is able to do something like this... My question is more along the lines of... What do you guys do? Do you just deploy manually (i.e. reproduce the settings from your sandboxes to development, then to staging then to production) or you found a way of having this automated?
Thank you so much :-) unai
On Mon, 13 Jul 2009 08:40:48 -0400, Kathleen Murtagh wrote:
"Automating" deployment? To what degree do you expect this to be automated?
I can't think of any situation in which you'd want deployment to be done without observation and testing during the process.
Can you elaborate on what sort of process you are expecting?
-- Kathleen Murtagh
On Mon, Jul 13, 2009 at 4:53 AM, Unai Rodriguez <me@u-journal.org> wrote:
Dear All,
We are building a somewhat complex site based on Drupal which features redundant servers at all levels of the system. We are starting of with 12 machines. We currently have three environments: development, staging and production.
We are having a hard time setting an automated deployment mechanism.
I have searched on the archives but was able to find this thread only: http://lists.drupal.org/pipermail/support/2007-May/004711.html
I have not been able to understand from that how other people out there are implementing automated deployment.
I found a number of links outside drupal.org:
http://stackoverflow.com/questions/377629/drupal-deployment-testing-dont-how...
http://stackoverflow.com/questions/282858/drupal-source-control-strategy
http://nicksergeant.com/blog/drupal/painless-drupal-revision-control-cvs-and...
http://nicksergeant.com/blog/drupal/my-thoughts-small-scale-drupal-developme...
http://www.workhabit.com/labs/autopilot http://www.dave-cohen.com/node/1066 http://www.garfieldtech.com/blog/drupal-dev-server
I have been working with Drupal's DEPLOY Module (http://drupal.org/project/deploy). Also I am trying to contact Workhabit (i.e. Autopilot guys).
I have not found yet a solid approach to automate deployment. Would anyone through me some pointers on how to do it? How are you guys doing this?
Thank you so much, unai -- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ] -- [ Drupal support list | http://lists.drupal.org/ ]
Dear Kathleen, If we want to go for the second method (i.e. "put everything in code and update functions"), how do we do it? Is all this about using "Hook Update" (http://api.drupal.org/api/function/hook_update)? Thank you, unai On Mon, 13 Jul 2009 09:52:13 -0400, Kathleen Murtagh wrote:
There are three main methods for managing the database when deploying from one workspace to another.
The most common is "write down, or script your database configuration settings and repeat them in the next workspace." This is where people generally start out as it is the easiest, conceptually, to understand. It is both prone to human error and tedious, though.
The second method is "put everything in code and update functions." This is the most recommended method because it is scalable both for very small and very large projects. It is also the most complicated because it requires adept programmers and takes away all the nice gui tools that makes Drupal development so accessible. There is progress towards this front, though. There is a movement to provide more "exportables" that will allow a developer to use the web gui to configure something, export the settings, and place it into code. Views currently supports this method out of the box. The "features" module is starting to give exportable support to other areas.
The last, and rarely used, method is "put the database in version control, and merge databases." This is probably the most difficult to understand conceptually, but fairly scriptable allowing non-programmer developers to use the web gui and contribute to a project. It is great for small projects, and becomes increasingly more difficult the larger and larger the database (the restriction is your developer's computing power, but a production database exceeding 500mb would start to require extra finesse to manage). The "dbscripts" module provides an extensive amount of scripts in order to use this method.
I use my version control system to deploy to a workspace from the primary repository. I don't need to worry about settings.php because I do not include that in version control. I set the database configuration settings once, and then never erase that file again. If you do have other settings being used in settings.php, you could put the database connection settings, and any other workspace-specific settings in a separate file and include that file in settings.php. Manually updating the workspace-specific settings would then only be required when they change (e.g.: adding Google Analytics to production, while keeping it disabled in development and testing).
I don't use a "one click" deploy, however, I generally just have to run "svn update" or "svn switch".
If you use the "put everything in code and update functions" method, you can install drush and have drush run update.php for you. Your script would then just say something like, "svn update && drush updatedb" or "svn switch && drush updatedb".
If you use the "put the database in version control, and merge databases" method, then you would restore the appropriate database for the given workspace. For development and testing, your script would just say something like, "svn update && ./dbscripts/restore.php none". For production, you would first merge the development and production databases, commit the merged databases and then deploy. The final deploy script would be something like, "svn switch <release branch> && ./dbscripts/restore.php production min".
-- Kathleen Murtagh
On Mon, Jul 13, 2009 at 9:04 AM, Unai Rodriguez <me@u-journal.org> wrote:
Dear Kathleen,
Thank you so much for your reply.
I guess we would like our deployment process to be somewhat similar to what we use in other projects; a number of developers are writing code while testing on their local machine first (aka developer sandbox). Each developer's machine has a web server and database server with a copy of the code. Once a developer is sure that her stuff is working fine on her sandbox that is the time when she will commit it into our version control system (i.e. we use subversion).
We have a server that takes care of the automated deployments. That is, any developer can access a web-based interface, input her user/password and click one button. This button will deploy the build which is currently on the trunk in subversion and put it into the development server; also some other things will take place like copying the proper settings.php file (in drupal's case) to the proper server(s).
From the web-based interface a developer can deploy to any of the 3 environments: development, staging or production. Usually development is where developers test (plus their sandboxes). Our Quality Assurance team takes care of testing the staging and production servers. That means deployments to these two environments are more controlled.
The deployment process is automated, meaning, you can push the code to any environment upon clicking one button. The process takes care of deploying the right code in the right place plus the configuration files (in this case settings.php).
We have been doing this for a while, I think is pretty standard.
Now, for Drupal is different since part of the "settings" are stored in the database, right? We are having a hard time getting the database settings to be applied over to the next environment (i.e. development to staging or staging to production).
I have been testing the deploy module which is able to do something like this... My question is more along the lines of... What do you guys do? Do you just deploy manually (i.e. reproduce the settings from your sandboxes to development, then to staging then to production) or you found a way of having this automated?
Thank you so much :-) unai
On Mon, 13 Jul 2009 08:40:48 -0400, Kathleen Murtagh wrote:
"Automating" deployment? To what degree do you expect this to be automated?
I can't think of any situation in which you'd want deployment to be done without observation and testing during the process.
Can you elaborate on what sort of process you are expecting?
-- Kathleen Murtagh
On Mon, Jul 13, 2009 at 4:53 AM, Unai Rodriguez <me@u-journal.org> wrote:
Dear All,
We are building a somewhat complex site based on Drupal which features redundant servers at all levels of the system. We are starting of with 12 machines. We currently have three environments: development, staging and production.
We are having a hard time setting an automated deployment mechanism.
I have searched on the archives but was able to find this thread only: http://lists.drupal.org/pipermail/support/2007-May/004711.html
I have not been able to understand from that how other people out there are implementing automated deployment.
I found a number of links outside drupal.org:
http://stackoverflow.com/questions/377629/drupal-deployment-testing-dont-how...
http://stackoverflow.com/questions/282858/drupal-source-control-strategy
http://nicksergeant.com/blog/drupal/painless-drupal-revision-control-cvs-and...
http://nicksergeant.com/blog/drupal/my-thoughts-small-scale-drupal-developme...
http://www.workhabit.com/labs/autopilot http://www.dave-cohen.com/node/1066 http://www.garfieldtech.com/blog/drupal-dev-server
I have been working with Drupal's DEPLOY Module (http://drupal.org/project/deploy). Also I am trying to contact Workhabit (i.e. Autopilot guys).
I have not found yet a solid approach to automate deployment. Would anyone through me some pointers on how to do it? How are you guys doing this?
Thank you so much, unai -- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ] -- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ]
Yes, when using that method, hook_update will become your best friend. Not everything you configure will require a hook_update, though. You can maximize the API of the modules you are using in order to get many of the results you want. Ultimately, the idea is that you will 100% separate the construction of the site from its content and users. With that method, you would be able to start with an empty database, fill in some content and everything would "just work." -- Kathleen Murtagh On Mon, Jul 13, 2009 at 10:10 AM, Unai Rodriguez <me@u-journal.org> wrote:
Dear Kathleen,
If we want to go for the second method (i.e. "put everything in code and update functions"), how do we do it? Is all this about using "Hook Update" (http://api.drupal.org/api/function/hook_update)?
Thank you, unai
On Mon, 13 Jul 2009 09:52:13 -0400, Kathleen Murtagh wrote:
There are three main methods for managing the database when deploying from one workspace to another.
The most common is "write down, or script your database configuration settings and repeat them in the next workspace." This is where people generally start out as it is the easiest, conceptually, to understand. It is both prone to human error and tedious, though.
The second method is "put everything in code and update functions." This is the most recommended method because it is scalable both for very small and very large projects. It is also the most complicated because it requires adept programmers and takes away all the nice gui tools that makes Drupal development so accessible. There is progress towards this front, though. There is a movement to provide more "exportables" that will allow a developer to use the web gui to configure something, export the settings, and place it into code. Views currently supports this method out of the box. The "features" module is starting to give exportable support to other areas.
The last, and rarely used, method is "put the database in version control, and merge databases." This is probably the most difficult to understand conceptually, but fairly scriptable allowing non-programmer developers to use the web gui and contribute to a project. It is great for small projects, and becomes increasingly more difficult the larger and larger the database (the restriction is your developer's computing power, but a production database exceeding 500mb would start to require extra finesse to manage). The "dbscripts" module provides an extensive amount of scripts in order to use this method.
I use my version control system to deploy to a workspace from the primary repository. I don't need to worry about settings.php because I do not include that in version control. I set the database configuration settings once, and then never erase that file again. If you do have other settings being used in settings.php, you could put the database connection settings, and any other workspace-specific settings in a separate file and include that file in settings.php. Manually updating the workspace-specific settings would then only be required when they change (e.g.: adding Google Analytics to production, while keeping it disabled in development and testing).
I don't use a "one click" deploy, however, I generally just have to run "svn update" or "svn switch".
If you use the "put everything in code and update functions" method, you can install drush and have drush run update.php for you. Your script would then just say something like, "svn update && drush updatedb" or "svn switch && drush updatedb".
If you use the "put the database in version control, and merge databases" method, then you would restore the appropriate database for the given workspace. For development and testing, your script would just say something like, "svn update && ./dbscripts/restore.php none". For production, you would first merge the development and production databases, commit the merged databases and then deploy. The final deploy script would be something like, "svn switch <release branch> && ./dbscripts/restore.php production min".
-- Kathleen Murtagh
On Mon, Jul 13, 2009 at 9:04 AM, Unai Rodriguez <me@u-journal.org> wrote:
Dear Kathleen,
Thank you so much for your reply.
I guess we would like our deployment process to be somewhat similar to what we use in other projects; a number of developers are writing code while testing on their local machine first (aka developer sandbox). Each developer's machine has a web server and database server with a copy of the code. Once a developer is sure that her stuff is working fine on her sandbox that is the time when she will commit it into our version control system (i.e. we use subversion).
We have a server that takes care of the automated deployments. That is, any developer can access a web-based interface, input her user/password and click one button. This button will deploy the build which is currently on the trunk in subversion and put it into the development server; also some other things will take place like copying the proper settings.php file (in drupal's case) to the proper server(s).
From the web-based interface a developer can deploy to any of the 3 environments: development, staging or production. Usually development is where developers test (plus their sandboxes). Our Quality Assurance team takes care of testing the staging and production servers. That means deployments to these two environments are more controlled.
The deployment process is automated, meaning, you can push the code to any environment upon clicking one button. The process takes care of deploying the right code in the right place plus the configuration files (in this case settings.php).
We have been doing this for a while, I think is pretty standard.
Now, for Drupal is different since part of the "settings" are stored in the database, right? We are having a hard time getting the database settings to be applied over to the next environment (i.e. development to staging or staging to production).
I have been testing the deploy module which is able to do something like this... My question is more along the lines of... What do you guys do? Do you just deploy manually (i.e. reproduce the settings from your sandboxes to development, then to staging then to production) or you found a way of having this automated?
Thank you so much :-) unai
On Mon, 13 Jul 2009 08:40:48 -0400, Kathleen Murtagh wrote:
"Automating" deployment? To what degree do you expect this to be automated?
I can't think of any situation in which you'd want deployment to be done without observation and testing during the process.
Can you elaborate on what sort of process you are expecting?
-- Kathleen Murtagh
On Mon, Jul 13, 2009 at 4:53 AM, Unai Rodriguez <me@u-journal.org> wrote:
Dear All,
We are building a somewhat complex site based on Drupal which features redundant servers at all levels of the system. We are starting of with 12 machines. We currently have three environments: development, staging and production.
We are having a hard time setting an automated deployment mechanism.
I have searched on the archives but was able to find this thread only: http://lists.drupal.org/pipermail/support/2007-May/004711.html
I have not been able to understand from that how other people out there are implementing automated deployment.
I found a number of links outside drupal.org:
http://stackoverflow.com/questions/377629/drupal-deployment-testing-dont-how...
http://stackoverflow.com/questions/282858/drupal-source-control-strategy
http://nicksergeant.com/blog/drupal/painless-drupal-revision-control-cvs-and...
http://nicksergeant.com/blog/drupal/my-thoughts-small-scale-drupal-developme...
http://www.workhabit.com/labs/autopilot http://www.dave-cohen.com/node/1066 http://www.garfieldtech.com/blog/drupal-dev-server
I have been working with Drupal's DEPLOY Module (http://drupal.org/project/deploy). Also I am trying to contact Workhabit (i.e. Autopilot guys).
I have not found yet a solid approach to automate deployment. Would anyone through me some pointers on how to do it? How are you guys doing this?
Thank you so much, unai -- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ] -- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ] -- [ Drupal support list | http://lists.drupal.org/ ]
Unai, Kathleen is describing two different tools that are both useful in trying to achieve the results you are after. To expand a bit on the "putting everything in code" approach: a view created by the views module GUI interface might present some page or block to your liking, that view is basically inside that environment's db. If you install enable the views exporter module, you will have access to then export that view as code, which will basically mean you can copy+paste it somewhere. But where? ...Into your module's hook_views_default_views() function. (http://drupal.org/node/99568). You've now taken your views "settings" out of the db, and placed them into code that you can manage and deploy from your svn repo. For larger applications with many views, we find it cleaner and helpful to place each coded view in its own view_name.views.inc.php file in a views/ sub-directory in our module's main directory. At the bottom of each individual view file, we add a necessary line that doesn't get exported with the the view. This line populates the initially empty $views array when the file is included in the hook: <code> $views[$view->name] = $view; </code> Then, our hook_views_default_views() function looks like follows: <code> function my_module_views_default_views() { $view_files = file_scan_directory(drupal_get_path('module', 'my_module') . '/views', '.*\.views\.inc\.php'); $views = array(); foreach ($view_files as $file => $data) { include $file; } return $views; } </code> This export feature is key to getting different modules "settings" out of the db and into a deployable form of code. Seth Unai Rodriguez wrote:
Dear Kathleen,
If we want to go for the second method (i.e. "put everything in code and update functions"), how do we do it? Is all this about using "Hook Update" (http://api.drupal.org/api/function/hook_update)?
Thank you, unai
On Mon, 13 Jul 2009 09:52:13 -0400, Kathleen Murtagh wrote:
There are three main methods for managing the database when deploying from one workspace to another.
The most common is "write down, or script your database configuration settings and repeat them in the next workspace." This is where people generally start out as it is the easiest, conceptually, to understand. It is both prone to human error and tedious, though.
The second method is "put everything in code and update functions." This is the most recommended method because it is scalable both for very small and very large projects. It is also the most complicated because it requires adept programmers and takes away all the nice gui tools that makes Drupal development so accessible. There is progress towards this front, though. There is a movement to provide more "exportables" that will allow a developer to use the web gui to configure something, export the settings, and place it into code. Views currently supports this method out of the box. The "features" module is starting to give exportable support to other areas.
The last, and rarely used, method is "put the database in version control, and merge databases." This is probably the most difficult to understand conceptually, but fairly scriptable allowing non-programmer developers to use the web gui and contribute to a project. It is great for small projects, and becomes increasingly more difficult the larger and larger the database (the restriction is your developer's computing power, but a production database exceeding 500mb would start to require extra finesse to manage). The "dbscripts" module provides an extensive amount of scripts in order to use this method.
I use my version control system to deploy to a workspace from the primary repository. I don't need to worry about settings.php because I do not include that in version control. I set the database configuration settings once, and then never erase that file again. If you do have other settings being used in settings.php, you could put the database connection settings, and any other workspace-specific settings in a separate file and include that file in settings.php. Manually updating the workspace-specific settings would then only be required when they change (e.g.: adding Google Analytics to production, while keeping it disabled in development and testing).
I don't use a "one click" deploy, however, I generally just have to run "svn update" or "svn switch".
If you use the "put everything in code and update functions" method, you can install drush and have drush run update.php for you. Your script would then just say something like, "svn update && drush updatedb" or "svn switch && drush updatedb".
If you use the "put the database in version control, and merge databases" method, then you would restore the appropriate database for the given workspace. For development and testing, your script would just say something like, "svn update && ./dbscripts/restore.php none". For production, you would first merge the development and production databases, commit the merged databases and then deploy. The final deploy script would be something like, "svn switch <release branch> && ./dbscripts/restore.php production min".
-- Kathleen Murtagh
On Mon, Jul 13, 2009 at 9:04 AM, Unai Rodriguez <me@u-journal.org> wrote:
Dear Kathleen,
Thank you so much for your reply.
I guess we would like our deployment process to be somewhat similar to what we use in other projects; a number of developers are writing code while testing on their local machine first (aka developer sandbox). Each developer's machine has a web server and database server with a copy of the code. Once a developer is sure that her stuff is working fine on her sandbox that is the time when she will commit it into our version control system (i.e. we use subversion).
We have a server that takes care of the automated deployments. That is, any developer can access a web-based interface, input her user/password and click one button. This button will deploy the build which is currently on the trunk in subversion and put it into the development server; also some other things will take place like copying the proper settings.php file (in drupal's case) to the proper server(s).
From the web-based interface a developer can deploy to any of the 3 environments: development, staging or production. Usually development is where developers test (plus their sandboxes). Our Quality Assurance team takes care of testing the staging and production servers. That means deployments to these two environments are more controlled.
The deployment process is automated, meaning, you can push the code to any environment upon clicking one button. The process takes care of deploying the right code in the right place plus the configuration files (in this case settings.php).
We have been doing this for a while, I think is pretty standard.
Now, for Drupal is different since part of the "settings" are stored in the database, right? We are having a hard time getting the database settings to be applied over to the next environment (i.e. development to staging or staging to production).
I have been testing the deploy module which is able to do something like this... My question is more along the lines of... What do you guys do? Do you just deploy manually (i.e. reproduce the settings from your sandboxes to development, then to staging then to production) or you found a way of having this automated?
Thank you so much :-) unai
On Mon, 13 Jul 2009 08:40:48 -0400, Kathleen Murtagh wrote:
"Automating" deployment? To what degree do you expect this to be
automated?
I can't think of any situation in which you'd want deployment to be done without observation and testing during the process.
Can you elaborate on what sort of process you are expecting?
-- Kathleen Murtagh
On Mon, Jul 13, 2009 at 4:53 AM, Unai Rodriguez <me@u-journal.org> wrote:
Dear All,
We are building a somewhat complex site based on Drupal which features redundant servers at all levels of the system. We are starting of
with 12
machines. We currently have three environments: development, staging and production.
We are having a hard time setting an automated deployment mechanism.
I have searched on the archives but was able to find this thread only: http://lists.drupal.org/pipermail/support/2007-May/004711.html
I have not been able to understand from that how other people out
there are
implementing automated deployment.
I found a number of links outside drupal.org:
http://stackoverflow.com/questions/377629/drupal-deployment-testing-dont-how...
http://stackoverflow.com/questions/282858/drupal-source-control-strategy
http://nicksergeant.com/blog/drupal/painless-drupal-revision-control-cvs-and...
http://nicksergeant.com/blog/drupal/my-thoughts-small-scale-drupal-developme...
http://www.workhabit.com/labs/autopilot http://www.dave-cohen.com/node/1066 http://www.garfieldtech.com/blog/drupal-dev-server
I have been working with Drupal's DEPLOY Module (http://drupal.org/project/deploy). Also I am trying to contact
Workhabit
(i.e. Autopilot guys).
I have not found yet a solid approach to automate deployment.
Would anyone
through me some pointers on how to do it? How are you guys doing this?
Thank you so much, unai -- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ]
-- [ Drupal support list | http://lists.drupal.org/ ]
Dear All, Thank you so much for the contributions. I have classified different parts that currently compose our Drupal system along with a suggested method of "putting everything in code and update functions": a) THEMES.- We just copy the code over, there is no settings to be applied really b) VIEWS.- We will be using what is suggested here: http://treehouseagency.com/blog/steven-merrill/2008/11/05/speed-and-version-... (This is similar to what Seth sent but applied to Drupal 6 -- THANKS SETH!) c) MODULES DEVELOPED BY US FROM SCRATCH.- We will make sure that all the settings that need to be put into the database are being updated on the hook_update function. Ideally these settings could come from a configuration file in php. We will use DRUSH to invoke our update.php upon deployment, as suggested by Kathleen. d) 3rd PARTY MODULES (aka contributed modules by the Drupal Community).- I am not sure how to proceed with this one; for now I am trying to check if the modules that we use implement the hook_update function and modify if possible so the settings get loaded from configuration files, similar to (c). My questions are: 1) How would you handle (d)? What I have now in mind seems a bit tricky... 2) What do you think of the classification? Am I missing something? Does it make sense? Thank you so much. With Best Wishes, unai
Any value stored in the variables table can be set in code. Read the bottom of the settings.php file for details of implementation. Many many many modules store their values here. For your custom modules, you can set all your default variable values in code. Then, any time you need to change that default value, you'll do it in code rather than performing an action that would effect the database. Read through the API of the contributed modules and see what you can use. Use provided hooks whenever possible. If necessary, you can run functions as one-off update events. For example, I used simplenews' simplenews_subscribe_user() function as part of a script to initially subscribe all users. Othertimes I have run scripts to migrate values from one field to another field, then delete the old field. I don't have a recommended best practice for automating this, but I'd look into setting up a system that will run each script in succession and then store what point it is at (e.g.: something like Ruby on Rails' migrations). -- Kathleen Murtagh On Tue, Jul 14, 2009 at 6:05 AM, Unai Rodriguez <me@u-journal.org> wrote:
Dear All,
Thank you so much for the contributions.
I have classified different parts that currently compose our Drupal system along with a suggested method of "putting everything in code and update functions":
a) THEMES.- We just copy the code over, there is no settings to be applied really b) VIEWS.- We will be using what is suggested here:
http://treehouseagency.com/blog/steven-merrill/2008/11/05/speed-and-version-... (This is similar to what Seth sent but applied to Drupal 6 -- THANKS SETH!) c) MODULES DEVELOPED BY US FROM SCRATCH.- We will make sure that all the settings that need to be put into the database are being updated on the hook_update function. Ideally these settings could come from a configuration file in php. We will use DRUSH to invoke our update.php upon deployment, as suggested by Kathleen. d) 3rd PARTY MODULES (aka contributed modules by the Drupal Community).- I am not sure how to proceed with this one; for now I am trying to check if the modules that we use implement the hook_update function and modify if possible so the settings get loaded from configuration files, similar to (c).
My questions are:
1) How would you handle (d)? What I have now in mind seems a bit tricky...
2) What do you think of the classification? Am I missing something? Does it make sense?
Thank you so much.
With Best Wishes, unai -- [ Drupal support list | http://lists.drupal.org/ ]
Unai, (d) really is the tricky part. Some modules will play nice, others won't. Some of the easiest modules to configure in your code will be ones that use /system_settings_form()/ to return their administrative options and configuration forms. /system_settings_form()/ just stores things as variables. This means that, for a low maintenance module that does all of its settings configuration on its admin screen, the API to configure that module from other code consists of /variable_get()/, /variable_set()/, & /variable_del()/. I would not recommend doing this in the contrib module's .install file, ie: directly in that module's update_N hooks. If you take mighty_contrib_module.install and add the function /mighty_contrib_module_update_1()/ (or, more correctly, ...update_6101() ), things will work fine only until the next release of that module is available and the author has added his own first update hook. Besides the version control conflicts in the files themselves, there will be schema version fixing needing to be done in the system table. I would recommend creating your own local module that exists mostly to just contain the update_N hooks to configure other contrib modules (say, housekeeping.module). Kind of like the junk drawer in you kitchen... everything useful (and not) that's together in one place just to keep it out of the way... Assuming that mighty_contrib_module had an admin setting to set the max number of words allowed in something, we could find the name of the FAPI element, and then inside of housekeeping.install write: <code> function housekeeping_update_1() { variable_set('mighty_contrib_module_max_words', 100); $ret[] = array( 'success' => true, 'query' => 'this is just a message masquerading as called sql statement for reporting purposes.', ); return $ret; } </code> You'll find all sorts of things to shove into the housekeeping module ;) Including some core config stuff too. Looking at the classifications you've set out below, I'd say that (b) is really a subset of (d). When it comes to contrib modules, there are going to be two main categories: those that play nice and those that don't. views happens to fall into the 'plays nice' category. Others won't. It's an important distinction you'll need to make when you're deciding which contrib modules to use from your solution's onset, an important criteria that cannot be overlooked. Seth Unai Rodriguez wrote:
Dear All,
Thank you so much for the contributions.
I have classified different parts that currently compose our Drupal system along with a suggested method of "putting everything in code and update functions":
a) THEMES.- We just copy the code over, there is no settings to be applied really b) VIEWS.- We will be using what is suggested here: http://treehouseagency.com/blog/steven-merrill/2008/11/05/speed-and-version-... (This is similar to what Seth sent but applied to Drupal 6 -- THANKS SETH!) c) MODULES DEVELOPED BY US FROM SCRATCH.- We will make sure that all the settings that need to be put into the database are being updated on the hook_update function. Ideally these settings could come from a configuration file in php. We will use DRUSH to invoke our update.php upon deployment, as suggested by Kathleen. d) 3rd PARTY MODULES (aka contributed modules by the Drupal Community).- I am not sure how to proceed with this one; for now I am trying to check if the modules that we use implement the hook_update function and modify if possible so the settings get loaded from configuration files, similar to (c).
My questions are:
1) How would you handle (d)? What I have now in mind seems a bit tricky...
2) What do you think of the classification? Am I missing something? Does it make sense?
Thank you so much.
With Best Wishes, unai -- [ Drupal support list | http://lists.drupal.org/ ]
You might find http://drupal.org/project/drush important for this. -- Earnie -- http://r-feed.com/ -- http://for-my-kids.com/ -- http://www.4offer.biz/ -- http://give-me-an-offer.com/
participants (4)
-
Earnie Boyd -
Kathleen Murtagh -
Seth Freach -
Unai Rodriguez