Staging content to production servers
Alan -- thanks for this write up. I think the staging/production server issue is one that merits attention. A lot of times, it means we're running without a safety net -- it is too hard to merge (config, content) changes between a dev site and a live site...so lots of people just don't, or do extra work in re-doing the same config steps on a live site.
Boris/Alan I have had clients ask me on this as well. The idea is to have a production server than no one posts content to directly, and a staging server where content is posted, edited, refined, ..etc. Then a button is clicked to push selected content from the staging to the production. Perhaps a starting point would be something similar to the Node Import/Export that Jeremy Epstein is writing as part of SoC. If this is coupled with: a) an API that modules can use to push/pull their own data along with what core does. b) an XML RPC or some other protocol for sending the data from one server to the other. c) Users and roles as well. d) all of the above is configurable (push content only, content + comments, users only, user + roles, all the above) If this happens, we can have truly federated servers and other neat things. For example, this can be the solution for sharing of users and roles across sites. If the sites are multi site on the same physical server, or across the globe, it will still work. Just some thoughts.
*cough* publish.module subscribe.module *cough*
Alan -- thanks for this write up. I think the staging/production server issue is one that merits attention. A lot of times, it means we're running without a safety net -- it is too hard to merge (config, content) changes between a dev site and a live site...so lots of people just don't, or do extra work in re-doing the same config steps on a live site.
Boris/Alan
I have had clients ask me on this as well. The idea is to have a production server than no one posts content to directly, and a staging server where content is posted, edited, refined, ..etc. Then a button is clicked to push selected content from the staging to the production.
Perhaps a starting point would be something similar to the Node Import/Export that Jeremy Epstein is writing as part of SoC.
If this is coupled with:
a) an API that modules can use to push/pull their own data along with what core does.
b) an XML RPC or some other protocol for sending the data from one server to the other.
c) Users and roles as well.
d) all of the above is configurable (push content only, content + comments, users only, user + roles, all the above)
If this happens, we can have truly federated servers and other neat things. For example, this can be the solution for sharing of users and roles across sites. If the sites are multi site on the same physical server, or across the globe, it will still work.
Just some thoughts.
On 15-Jun-06, at 11:16 AM, Khalid B wrote:
Alan -- thanks for this write up. I think the staging/production server issue is one that merits attention. A lot of times, it means we're running without a safety net -- it is too hard to merge (config, content) changes between a dev site and a live site...so lots of people just don't, or do extra work in re-doing the same config steps on a live site.
Boris/Alan
I have had clients ask me on this as well. The idea is to have a production server than no one posts content to directly, and a staging server where content is posted, edited, refined, ..etc. Then a button is clicked to push selected content from the staging to the production.
OK...so for *content only*....you can use John Van Dyk's publish - subscribe modules to do this *today*. They are done for 4.6, and he is working on improvements in the 4.7 line. (well, content and taxonomy terms....)
Perhaps a starting point would be something similar to the Node Import/Export that Jeremy Epstein is writing as part of SoC.
If this is coupled with:
a) an API that modules can use to push/pull their own data along with what core does.
pub - sub doesn't yet have hooks. I have a request to "group enable" pub - sub, so it is OG aware. Haven't scheduled a call with JvD yet, but have talked to Moshe about it.
b) an XML RPC or some other protocol for sending the data from one server to the other.
publish - subscribe
c) Users and roles as well.
pub - sub has filters.
d) all of the above is configurable (push content only, content + comments, users only, user + roles, all the above)
pub-sub has filters. In some ways, the pub - sub filter system is VERY similar to "Views"...and lots of modules are views enabled. So there might be an interesting play here to pub - sub enable Views (I don't think pub - sub should necessarily depend on views...)
If this happens, we can have truly federated servers and other neat things. For example, this can be the solution for sharing of users and roles across sites. If the sites are multi site on the same physical server, or across the globe, it will still work.
Just some thoughts.
Boris Mann wrote:
OK...so for *content only*....you can use John Van Dyk's publish - subscribe modules to do this *today*. They are done for 4.6, and he is working on improvements in the 4.7 line.
(well, content and taxonomy terms....)
pub - sub doesn't yet have hooks. I have a request to "group enable" pub - sub, so it is OG aware. Haven't scheduled a call with JvD yet, but have talked to Moshe about it.
Ah, yes, pub - sub of course! Thanks, John and Boris, for reminding us. Making pub - sub group aware would absolutely rock. ..chrisxj P.S. /me thinks having a way to make it more easily known what we have for current capabilities would be Really Useful, especially after having spent time today using grep at the command line in the contrib repo looking for modules to do a specific thing.
Khalid B wrote:
I have had clients ask me on this as well. The idea is to have a production server than no one posts content to directly, and a staging server where content is posted, edited, refined, ..etc. Then a button is clicked to push selected content from the staging to the production.
Why are people using staging sites for this, instead of the workflow/publish/moderation queue abilities built into Drupal? Personally, I'm more interested in staging software changes and config changes, so I'm curious as to what is driving this other use. -- ..chrisxj
On 6/15/06, Chris Johnson <chris@tinpixel.com> wrote:
Why are people using staging sites for this, instead of the workflow/publish/moderation queue abilities built into Drupal?
I've worked in environments where it was a strict requirement. Relatively few people had access to the production servers, but we made changes on dev, tested the migration path when moving them to test-environment and then IT would promote things to pre-prod and (if successful) to production environment. That was a telecommunications company so it's understandable to see where they got that mentality, but they applied that stringent process to a non-critical software project I was working on. Frustrating as heck. Plus, the internal Drupal workflow only applies to nodes, not other configuration stuff. And i think the scope of this conversation includes "other configuration stuff" right? Greg -- Greg Knaddison | Growing Venture Solutions Denver, CO | http://growingventuresolutions.com Technology Solutions for Communities, Individuals, and Small Businesses
-----Oorspronkelijk bericht----- Van: development-bounces@drupal.org namens Greg Knaddison - GVS
I've worked in environments where it was a strict requirement. Relatively few people had access to the production servers, but we made changes on dev, tested the migration path when moving them to test-environment and then IT would promote things to pre-prod and (if successful) to production environment.
We are talking two different thing here. All big sites have a Dev, Test, Accept and Prod environment. But that is regarding configuration / application. Having a staging server for content (what does it look like, are the links working, do all the right users see it in the right way) is typically CMS 1999 way and can be skipped IMHO with workflow in a real CMS.
Greg Knaddison - GVS wrote:
On 6/15/06, Chris Johnson <chris@tinpixel.com> wrote:
Why are people using staging sites for this, instead of the workflow/publish/moderation queue abilities built into Drupal?
I've worked in environments where it was a strict requirement. Relatively few people had access to the production servers, but we made changes on dev, tested the migration path when moving them to test-environment and then IT would promote things to pre-prod and (if successful) to production environment.
Plus, the internal Drupal workflow only applies to nodes, not other configuration stuff. And i think the scope of this conversation includes "other configuration stuff" right?
I certainly understand the strict control of access to production servers in general. I've worked in environments like that, and believe in the benefits of having that tight control -- especially when I've been carrying the crisis pager for the software dev group. I definitely wanted to know who did what and when! It just seems like staging content, not config or software, on drupal is actually *more problemmatic* than actually using the built in tools for promoting content from author to editor to approval to publication on the production drupal site itself. I'm with you 100% on staging the other changes -- not that *that* is easy, either. :-)
On 15-Jun-06, at 12:04 PM, Chris Johnson wrote:
Khalid B wrote:
I have had clients ask me on this as well. The idea is to have a production server than no one posts content to directly, and a staging server where content is posted, edited, refined, ..etc. Then a button is clicked to push selected content from the staging to the production.
Why are people using staging sites for this, instead of the workflow/publish/moderation queue abilities built into Drupal?
Good question. I think part of it is seeing it "really" live -- being able to log out, see it as an anonymous users, and so on. You can't do it on a live site without it, being, um, live. And in part, this may be a discomfort with the way CMS is handled, vs. static content.
Personally, I'm more interested in staging software changes and config changes, so I'm curious as to what is driving this other use.
This discussion would be useful for the consulting list. Khalid, perhaps you can post to the group on Drupal.org and ask for feedback for what other people see a need for: http://groups.drupal.org/ consulting (with the note that pub - sub does all of it already :P) Cheers, -- Boris Mann Vancouver 778-896-2747 San Francisco 415-367-3595 SKYPE borismann http://www.bryght.com
Boris Mann wrote:
On 15-Jun-06, at 12:04 PM, Chris Johnson wrote:
Why are people using staging sites for this, instead of the workflow/publish/moderation queue abilities built into Drupal?
Good question. I think part of it is seeing it "really" live -- being able to log out, see it as an anonymous users, and so on. You can't do it on a live site without it, being, um, live.
Excellent point. Clearly there is a need to stage content in that case. Thanks much for pointing that out. ..chris
The Import / Export API will certainly be a great tool for migrating data from a staging environment to a prod environment. The API supports any data that is stored in the DB: that is, not just the usual nodes / users / comments / categories, but also variables (i.e. config settings), system files, block configuration, and anything else you can think of. As long as there's a definition available for the data structure, it can be exported / imported. There currently isn't support for properly exporting / importing serialized data, but adding support is simply a matter of defining 'serialized' as a new field type in the API. Cheers, Jaza. On 6/16/06, Khalid B <kb@2bits.com> wrote:
Alan -- thanks for this write up. I think the staging/production server issue is one that merits attention. A lot of times, it means we're running without a safety net -- it is too hard to merge (config, content) changes between a dev site and a live site...so lots of people just don't, or do extra work in re-doing the same config steps on a live site.
Boris/Alan
I have had clients ask me on this as well. The idea is to have a production server than no one posts content to directly, and a staging server where content is posted, edited, refined, ..etc. Then a button is clicked to push selected content from the staging to the production.
Perhaps a starting point would be something similar to the Node Import/Export that Jeremy Epstein is writing as part of SoC.
If this is coupled with:
a) an API that modules can use to push/pull their own data along with what core does.
b) an XML RPC or some other protocol for sending the data from one server to the other.
c) Users and roles as well.
d) all of the above is configurable (push content only, content + comments, users only, user + roles, all the above)
If this happens, we can have truly federated servers and other neat things. For example, this can be the solution for sharing of users and roles across sites. If the sites are multi site on the same physical server, or across the globe, it will still work.
Just some thoughts.
I've started to address this issue by reserving the first 1000 ids for the "staging" (or if you prefer, "install profile"). So for instance node nid 10 comes from the install profile, while node nid 1010 is on the production server only. Same goes for vocabulary with vid 10 (or 1010), menu with mid 10 (or 1010), etc. Later, I can add node 11 to the install profile and propigate it to my production site easily enough and without fear of overwriting anything that's been configured on the production server. You can read exactly how I do this here <http://dave-cohen.com/node/1066>. -Dave
Alan -- thanks for this write up. I think the staging/production server issue is one that merits attention. A lot of times, it means we're running without a safety net -- it is too hard to merge (config, content) changes between a dev site and a live site...so lots of people just don't, or do extra work in re-doing the same config steps on a live site.
Dave Cohen wrote:
I've started to address this issue by reserving the first 1000 ids for the "staging" (or if you prefer, "install profile"). So for instance node nid 10 comes from the install profile, while node nid 1010 is on the production server only. Same goes for vocabulary with vid 10 (or 1010), menu with mid 10 (or 1010), etc. Later, I can add node 11 to the install profile and propigate it to my production site easily enough and without fear of overwriting anything that's been configured on the production server.
You can read exactly how I do this here <http://dave-cohen.com/node/1066>.
wow, thats a very clever solution to the problem. i would totally support drupal patches which insure that ids start at 1000. i'm not so sure about a patch which adds unique keys to all tables. we need someone db person to chime in there. anyway, it would be handy if we supported this trick better (i.e. without patching). thanks for sharing. -moshe
Moshe Weitzman wrote:
Dave Cohen wrote:
I've started to address this issue by reserving the first 1000 ids for the "staging" (or if you prefer, "install profile"). So for instance node nid 10 comes from the install profile, while node nid 1010 is on the production server only. Same goes for vocabulary with vid 10 (or 1010), menu with mid 10 (or 1010), etc. Later, I can add node 11 to the install profile and propigate it to my production site easily enough and without fear of overwriting anything that's been configured on the production server.
You can read exactly how I do this here <http://dave-cohen.com/node/1066>.
wow, thats a very clever solution to the problem. i would totally support drupal patches which insure that ids start at 1000.
Dave - in your post it described a very real world scenario so having some changes on the core side, as Moshe suggested to work hand-in-hand with the profile install seems the way to go. But I wonder how is this going to work in a release upgrade when this feature is included, we can have some scripts to bump all existing nodes by 1000, but does it mean those millions of drupal nodes that have been indexed by google or being bookmarked will all be broken? Jenny
On Saturday 17 June 2006 09:45, Jenny Hsueh wrote:
Dave - in your post it described a very real world scenario so having some changes on the core side, as Moshe suggested to work hand-in-hand with the profile install seems the way to go.
But I wonder how is this going to work in a release upgrade when this feature is included, we can have some scripts to bump all existing nodes by 1000, but does it mean those millions of drupal nodes that have been indexed by google or being bookmarked will all be broken?
Jenny
Or those sites that have more than 1000 pages, and therefore would need more namespace room than that for a full mirror staging area? -- Larry Garfield AIM: LOLG42 larry@garfieldtech.com ICQ: 6817012 "If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it." -- Thomas Jefferson
On Saturday 17 June 2006 07:55, Larry Garfield wrote:
On Saturday 17 June 2006 09:45, Jenny Hsueh wrote:
But I wonder how is this going to work in a release upgrade when this feature is included, we can have some scripts to bump all existing nodes by 1000, but does it mean those millions of drupal nodes that have been indexed by google or being bookmarked will all be broken?
I suppose you could try something like that, although I imagined using my suggestion for new sites only. The google index would be wrong at first, but fixed after google's bot revisited the site. User bookmarks would remain broken.
Jenny
Or those sites that have more than 1000 pages, and therefore would need more namespace room than that for a full mirror staging area?
Of course 1000 was just an arbitrary choice that seemed like plenty for my needs. Others may want a higher number. -Dave
On Saturday 17 June 2006 12:19, Dave Cohen wrote:
Or those sites that have more than 1000 pages, and therefore would need more namespace room than that for a full mirror staging area?
Of course 1000 was just an arbitrary choice that seemed like plenty for my needs. Others may want a higher number.
-Dave
The problem is that I may not know in advance how many I'll need. That's especially true with a site where users can supply content arbitrarily, such as anything with a forum. If we want to have an integrated staging/live setup, then an alternate URL set would be more flexible. Eg, anything prefixed with /staging/ is the staging version of whatever comes after it. (I'm not entirely sure how one would do that, but it would be a better plan, and easier to mirror back and forth, methinks. -- Larry Garfield AIM: LOLG42 larry@garfieldtech.com ICQ: 6817012 "If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it." -- Thomas Jefferson
With my first drafst for fixtures I simply solved this by NOT using any IDs. Because fixtures are pure PHP, we can let Drupal create the IDs. No need for any hardcoded ID, if you design the architecture of the import-export of staging code well enough, IMO. Op zaterdag 17 juni 2006 05:41, schreef Moshe Weitzman:
wow, thats a very clever solution to the problem. i would totally support drupal patches which insure that ids start at 1000.
This works untill you want to test the import of 1500 nodes from some old site. Or develop a caching system for your large dataset. Starting at any high number will cause problems if you need a higher number. Yes, you can increase the initial high number. But this is not a solution. Just like, back in the days, we coded Qbasic like 10 foo 20 bar 30 goto 10 oh - whoops, I need baz bax and bay too: 10 foo 12 baz 14 bax 16 bay 20 bar 30 goto 10 At some moment you got stuck and had to renumber your whole listing. Right? At some moment 1000 will not cut it. Bèr I
On 18 Jun 2006, at 10:03 PM, Bèr Kessels wrote:
With my first drafst for fixtures I simply solved this by NOT using any IDs. Because fixtures are pure PHP, we can let Drupal create the IDs. No need for any hardcoded ID, if you design the architecture of the import- export of staging code well enough, IMO. I agree with the idea of imports / exports.
Shipping the defaults as php as opposed to sql makes a lot more sense. -- Adrian Rossouw Drupal developer and Bryght Guy http://drupal.org | http://bryght.com
Unfortunately, IDs get baked in to various places. For example, let's say my staging content includes a vocabulary and a Views module view to display the content of that vocabulary. The view is exportable as PHP, but if you look closely, you'll find the PHP includes something like: array ( 'tablename' => 'term_data', 'field' => 'vid', 'operator' => 'AND', 'options' => '', 'value' => array ( 0 => '7', ), Note the "7". That's the vocabulary's ID on the staging server. If, on the production server, the vid for that vocabulary is NOT 7, the view will not work as desired. I've found it convenient to hardcode IDs in various places of my own code. So I've grown fond of the notion that some important IDs will remain the same from staging to testing to production. I'm not saying it can't be done without hardcoding the IDs. But it's quite a challenge. -Dave On Monday 19 June 2006 05:12, Adrian Rossouw wrote:
On 18 Jun 2006, at 10:03 PM, Bèr Kessels wrote:
With my first drafst for fixtures I simply solved this by NOT using any IDs. Because fixtures are pure PHP, we can let Drupal create the IDs. No need for any hardcoded ID, if you design the architecture of the import- export of staging code well enough, IMO.
I agree with the idea of imports / exports.
Shipping the defaults as php as opposed to sql makes a lot more sense.
-- Adrian Rossouw Drupal developer and Bryght Guy http://drupal.org | http://bryght.com
Op maandag 19 juni 2006 15:24, schreef Dave Cohen:
I've found it convenient to hardcode IDs in various places of my own code. So I've grown fond of the notion that some important IDs will remain the same from staging to testing to production.
I'm not saying it can't be done without hardcoding the IDs. But it's quite a challenge.
You are right: in my import export stuff, I allow hardcoded IDs: $node->id = 1234 $node->title = 'foo' $node->uid = <token>first_user_uid</token> will be made into a post with nid 1234, title 'foo' and userID generated from the parser system in sympal scripts (ie replaced with a variable). If you omit $node->id it will be generated by Drupal. If you provide it, the system will try to do its best to either replace/alter the existing node, or to make a new one with that nid. Bèr
On 19 Jun 2006, at 7:01 PM, Bèr Kessels wrote:
You are right: in my import export stuff, I allow hardcoded IDs: $node->id = 1234 $node->title = 'foo' $node->uid = <token>first_user_uid</token>
or $node->nid = resolve_id('nid', 'identifier'); And then keep a special list of what id's the identifiers mean. When generating the output ( i don't like the word fixture ), it's a simple check to see if the id is in an identifier list. A single field / id combination can't have multiple identifiers though. -- Adrian Rossouw Drupal developer and Bryght Guy http://drupal.org | http://bryght.com
I've started to address this issue by reserving the first 1000 ids for the "staging" (or if you prefer, "install profile").
I think this is an excellent idea that perhaps makes my suggestion to split config vs. data tables into two databases obsolete. Some thoughts: - If we had data-driven database schemas, your drupal_base_db.txt script (or its built-in equivalent) would (a) not need to hard-code the list of tables not to copy (the schema would identify them) and (b) not need to use fragile-looking MySQL-specific regular expressions and SHOW queries to do its job. - Not every table needs an id field, only those for which you want to reserve some id space. Pure config tables (e.g. system) can be copied in their entirety. The schema could identify such tables so processing them correctly would be easy. Barry
On Saturday 17 June 2006 05:40, Barry Jaspan wrote:
I've started to address this issue by reserving the first 1000 ids for the "staging" (or if you prefer, "install profile").
I think this is an excellent idea that perhaps makes my suggestion to split config vs. data tables into two databases obsolete. Some thoughts:
- If we had data-driven database schemas, your drupal_base_db.txt script (or its built-in equivalent) would (a) not need to hard-code the list of tables not to copy (the schema would identify them) and (b) not need to use fragile-looking MySQL-specific regular expressions and SHOW queries to do its job.
Agreed that script could use some improvement. I was just happy to get it working enough for my needs. I learned mysql queries I never knew existed.
- Not every table needs an id field, only those for which you want to reserve some id space. Pure config tables (e.g. system) can be copied in their entirety. The schema could identify such tables so processing them correctly would be easy.
The db dump my script produces can be applied over and over, and should be applied each time the install profile is updated. It's like running update.php. But it works by using INSERT IGNOREs (or REPLACEs). If the table has no unique id, the script would end up adding duplicate rows to the production database. I'm not saying this is the ultimate solution, but as long as it works that way the tables must have unique ids. So for example, if I already have a production site, but I want to enable a new module, I'd start by enabling the module on the install profile server. Then I'd use my script to dump the database. The result would have a new row inserted into the system table (along with all the original rows). Then I run the script on the production server. The original rows have no effect thanks to INSERT IGNORE. The new row is inserted though, and the production server now has the module enabled. -Dave
Op donderdag 15 juni 2006 20:16, schreef Khalid B:
Perhaps a starting point would be something similar to the Node Import/Export that Jeremy Epstein is writing as part of SoC.
My idea is that we should start with the runtime environments "development", "testing" and "live". On my server I have these three available as subdomains too, where "development" is a standalone drupal install for all sites, "testing" is a testing.example.com multisite on the main installation and live is just example.com on that main installation. Together with some (ugly) scripts that extract stuff from database tables, and my initial fixtures code this works for me. I would love to turn these scripts into sympal script contribs, but so far lack of time and interest (I.E. "it works for me" and "drupal seems to go into a different direction wrt all this anyway") make me not want to contribute this atm. People whom are intersted are very welcome to lend a hand. In that case, please get me offlist. Then we can discuss where to take this. Bèr
participants (14)
-
Adrian Rossouw -
Barry Jaspan -
Boerland, Bert -
Boris Mann -
Bèr Kessels -
Chris Johnson -
Dave Cohen -
Greg Knaddison - GVS -
Jenny Hsueh -
Jeremy Epstein -
John VanDyk -
Khalid B -
Larry Garfield -
Moshe Weitzman