Distilling distributions from configured sites.
So on the topic of distributions (what I am going to call install profiles from now on), some code I recently wrote for http://drupal.org/node/25745, that allows you to very easily rip out the variables configured on the site and store them as defaults. function export_defaults() { global $conf; foreach ($conf as $name => $value) { $output .= "\$default['$name'] = " . $value . ";\n"; // doesn't work for objects... } return $output; } So that led me to think about what other things we have to import / export for new sites. I also discussed the same with Jaza (who is working on the import/export api.). He has targetted the following things in core that need to be copied : http://basement.greenash.net.au/soc2006/CurrentTodoList Jaza's system is far more complex than our meagre needs for the distributions right now, plus it won't be ready in time for 4.8.I also think my proposed approach has a lot of benefits for both our requirements for the distributions (being able to easily build and maintain them), the Drupal API in general and will actually probably make his life easier, possibly even fast tracking some of his code into core. Now my idea (which is very similar to how the views module does it) , is to use a set of functions such as : function drupal_export($resultset, $function, $id_field, $objects = TRUE) { $x = 0; while ($id = db_fetch_array($resultset)) { $object = $function($id[$id_field]); $output .= "\$object[$x] = new stdClass();\n"; foreach ($object as $field => $value) { if ($field != $id_field) { if (is_object($value)) { $output .= "\$object[$x]->$field = (object) " . var_export((array) $value, TRUE) . ";\n"; } else { $output .= "\$object[$x]->$field = " . var_export($value, TRUE) . ";\n"; } } } $x++; } return $output; } function drupal_import($objects, $function) { foreach ($objects as $object) { $function($object); } } for export : $result = db_query("select nid from node order by nid asc"); print drupal_export($result, 'node_load', 'nid')); // or rather save into a file and then on import : include_once 'distributions/default/node.import'; drupal_import($objects, 'node_save); How I see this working is that each module (or thing), will get placed in the distribution/mydistribution/<driver>.import files which are very simply created, and very simply modified. Since we kind of know what order things need to be imported in we will just recurse through the object and import them. These following things are already handled : system table = enabled modules, and themes. Handled by the .distribution file (until we can get dependencies into core that allows them to be configured in the .meta file (ie: requires:theme-bluemarine, blog)) variables (defaults) = handled by the distribution.inc file. see http://drupal.org/node/25745 The following things desperately need to be handled, cleanly and easily : profile-fields, users, node types, locale ,vocabularies, terms, blocks, aliases, files (more on this later) nodes , comments and finally menus. (in roughly that order.) I'm unsure about locale in that list. (maybe it should be a setting?) These are optional, but i'd like to see them : aggregator: feeds, items and categories .. could be really awesome for networks of sites. views : it already does this, but could be used as both the basis for the api, and one of the most practical uses of it. cck : might already do this, but it's important that it does. action/workflow : don't know if it uses it's own table structure or the variable table, but it would be useful for distribution. any other contrib module that would benefit from this. What this however requires , is that we have a consistent set of _load and _save functions, which are all equal to node_load. I feel this is something we should have to begin with. Additionally, especially with node_load and block_load, we will need a way to access the original php code for php filter fields, and not just the output. Also, we need to decide whether initial imports should support revisions (i am not sure they should). Perhaps we could even get away with just having a module_import and module_export hook, which could do the mappings to whatever it's internal functions currently are. This would be a non-invasive way to implement this functionality, and give us a very good idea of what we need to make consisten _save and _load functions for in future releases. Some of this work is possibly already done as part of jaza's import / export project. Some of the node related code might already be living in publish and subscribe. This moves us further away from doing database dumps, and makes distributions a lot easier to create (you would just run 'scripts/build_distro.php --site mysite.comnewdistroname'. And then new sites would use that as their basis. One of the more awesome side effects of this all, being we can also distribute each of these as packages on drupal.org, and teach the install system to install them all. It provides a clean way for modules to distribute views and cck node types etc, perhaps even custom blocks and templates (with the inline template editor i want to write) There are some issues with this on already running sites, with versioning of all these elements, but i believe the dependencies work will end up helping with this. One of the other things I would like to see us do, is to provide variables that have been submitted as part of the distribution specific form, as variables to the .import files. A perfect example of this is having the first node configured as "welcome to your site $edit[fullname]". Even the creation of the first user (for instance), could be handled via this. Another point I came across, is the files directory. I believe the installer should try and create it, and I also believe the distributions should be able to have default files that are copied to the directory. An example being that a brochureware distribution might ship with a customised logo/favicon, and possibly an initial node configured and made to be the front page, which has a nice piece of stock photography already present (giving people a presentable front page to work from). It might even ship with several different templates and allow you to select which one of them you want your front page to look like. That's a rather rambling post for what is actually really simple in concept, and if we build the framework within which to do it and have a developer per import / export object .. we should hopefully still be able to get something like this into 4.8... and then we can watch the distributions bloom =)
sounds like a great enhancement.
for export : $result = db_query("select nid from node order by nid asc"); print drupal_export($result, 'node_load', 'nid')); // or rather save into a file
and then on import : include_once 'distributions/default/node.import'; drupal_import($objects, 'node_save);
be careful with this. the node fields have not been sanitized at all i nthe DB. you are just including them without protection, AFAICT. there is probably an easy fix for this.
Another point I came across, is the files directory. I believe the installer should try and create it, and I also believe the distributions should be able to have default files that are copied to the directory.
yes. this is sticks out like a sore thumb given how smooth install is now.
Another point I came across, is the files directory. I believe the installer should try and create it, and I also believe the distributions should be able to have default files that are copied to the directory.
yes. this is sticks out like a sore thumb given how smooth install is now.
I mentioned this to eaton a little: * the installer should save to sites/domain.com/settings.php. * the installer should pre-create sites/domain.com/files/ * the installer should set that preference for you. -- Morbus Iff ( i am the sexy alpha geek ) Technical: http://www.oreillynet.com/pub/au/779 Culture: http://www.disobey.com/ and http://www.gamegrene.com/ icq: 2927491 / aim: akaMorbus / yahoo: morbus_iff / jabber.org: morbus
On 8/9/06, Morbus Iff <morbus@disobey.com> wrote:
Another point I came across, is the files directory. I believe the installer should try and create it, and I also believe the distributions should be able to have default files that are copied to the directory.
yes. this is sticks out like a sore thumb given how smooth install is now.
I mentioned this to eaton a little:
* the installer should save to sites/domain.com/settings.php.
It already does that, sort of, if the directory for the domain is created and if the settings.php is the default one. So, if what you are saying is it should create domain.com directory and settings.php, then I am with you on this. Long term, using domain.com is far better than default, since you can move from multi site to single site and vice versa seamlessly. If you use default, there is a lot of work involved and hence not seamless.
* the installer should pre-create sites/domain.com/files/
Fully agree. If the Apache user cannot create the directory, then it should tell the user to do so. While we are at it, why not add modules and themes as empty directories too?
You are right. I think the sites/all thing that is in HEAD now eliminates the need for site specific modules and themes, and keeps things separate from core as well (for upgrades). On 8/9/06, Adrian Rossouw <adrian@bryght.com> wrote:
On 8/9/06, Khalid B <kb@2bits.com> wrote:
While we are at it, why not add modules and themes as empty directories too?
I'd rather not create directories that are possibly not going to be used.
Eliminates the need *in many cases*... But certainly not in all cases. I can't imagine someone who wants to put together a web site not putting at least a few custom modules in, or one or two downloaded (or customized) themes. --Jeff
-----Original Message----- From: Khalid B [mailto:kb@2bits.com] Sent: Wednesday, August 09, 2006 9:53 AM To: Adrian Rossouw Cc: development@drupal.org Subject: Re: [development] Distilling distributions from configured sites.
You are right.
I think the sites/all thing that is in HEAD now eliminates the need for site specific modules and themes, and keeps things separate from core as well (for upgrades).
Yes. You are also right. What Adrian means that those who need them (the per site themes and modules directories) can add them, and there is no need to add them by default. On 8/9/06, Jeff Eaton <jeff@viapositiva.net> wrote:
Eliminates the need *in many cases*... But certainly not in all cases. I can't imagine someone who wants to put together a web site not putting at least a few custom modules in, or one or two downloaded (or customized) themes.
--Jeff
-----Original Message----- From: Khalid B [mailto:kb@2bits.com] Sent: Wednesday, August 09, 2006 9:53 AM To: Adrian Rossouw Cc: development@drupal.org Subject: Re: [development] Distilling distributions from configured sites.
You are right.
I think the sites/all thing that is in HEAD now eliminates the need for site specific modules and themes, and keeps things separate from core as well (for upgrades).
On Wed, August 9, 2006 9:35 am, Khalid B said:
On 8/9/06, Morbus Iff <morbus@disobey.com> wrote:
Another point I came across, is the files directory. I believe the installer should try and create it, and I also believe the distributions should be able to have default files that are copied to the directory.
yes. this is sticks out like a sore thumb given how smooth install is now.
I mentioned this to eaton a little:
* the installer should save to sites/domain.com/settings.php.
It already does that, sort of, if the directory for the domain is created and if the settings.php is the default one.
So, if what you are saying is it should create domain.com directory and settings.php, then I am with you on this.
Long term, using domain.com is far better than default, since you can move from multi site to single site and vice versa seamlessly. If you use default, there is a lot of work involved and hence not seamless.
What about moving from a test site to a live site, however? I do that all the time. We should have clear migration instructions available in a provided readme or even in settings.php itself, as well as on the site, if it's not going to be automated.
* the installer should pre-create sites/domain.com/files/
Fully agree.
+1
If the Apache user cannot create the directory, then it should tell the user to do so.
While we are at it, why not add modules and themes as empty directories too?
+1 I don't know if we want it to create sites/all/modules et al, however. I can see arguments both directions. --Larry Garfield
On 8/9/06, Moshe Weitzman <weitzman@tejasa.com> wrote:
sounds like a great enhancement.
be careful with this. the node fields have not been sanitized at all i nthe DB. you are just including them without protection, AFAICT. there is probably an easy fix for this.
Umm. the entire idea is that they haven't been sanitised. these are the fields that will be saved to the database. They are not for displaying. For example, if you export a custom block, you want to export the php that you entered, not the value the php generates.
Umm. the entire idea is that they haven't been sanitised.
these are the fields that will be saved to the database. They are not for displaying.
For example, if you export a custom block, you want to export the php that you entered, not the value the php generates.
i haven't really thought this through, but my concern is that some user submits a node with the text db_query("DELETE * FROM users"). then you do a var_export() and then you do an include() in the import script. i am worried that this import will unknowingly cause damage.
Op woensdag 9 augustus 2006 13:50, schreef Adrian Rossouw:
for export : $result = db_query("select nid from node order by nid asc"); print drupal_export($result, 'node_load', 'nid')); // or rather save into a
What I do in the fixtures is a lot simpler and a lot more general. The hardes part is finding what APIs store stuff, and what load "stuff". "Stuff" is here something like a node, a term, and so on. The idea is: $result = db_query('select nid from node where uid = 1'); while ($row = db_fetch_object($result)) { $nodes[$row->nid] = node_load($row->nid); } $out = var_export ($nodes, TRUE); file_put_content(... $out); viola! simple! works! a include_once($file) and then a foreach($nodes as $node) which node_save()s all the objects is all you need! http://be2.php.net/manual/en/function.var-export.php Bèr
participants (7)
-
Adrian Rossouw -
Bèr Kessels -
Jeff Eaton -
Khalid B -
Larry Garfield -
Morbus Iff -
Moshe Weitzman