I am writing to request comments on a proposed modification to Drupal core. Proposal: move the core search functionality in Drupal into an include file which can be specified on a per site basis. The actual functions used to return search results would not change, this proposed modification would simply put them all in a file which can be specified within settings.php. There are a number of reasons to consider doing this: 1) The flexibility to specify different search components would resolve some potential design deficiencies in the platform as it stands now. We have been doing some work integrating the Xapian search engine with Drupal. Instead of storing indexed results in the Drupal database, we are putting indexed content in Xapian instead. To get this to work properly, we have needed to patch core, which represents a deficiency both in terms of installation and long term maintenance of sites. Modifications to core mean people are working with unique distributions of Drupal instead of a common distribution, which are more complex to support. 2) This change would not differ significantly from design patterns currently in place within the framework. Other core components have already adopted this approach. An alternate cache.inc file can be specified within settings.php and is used by Robert Douglas' memcache module. Settings.php has had the ability to override system settings for a long time. 3) In some high traffic scenarios, database throughput becomes a bottleneck to the point where Drupal's current search features become unusable in the context in which they are deployed. Providing the flexibility to specify the underlying search components used for Drupal will provide system architects with additional options for scaling Drupal sites without losing characteristic features of the platform. In the case of Xapian, all database traffic related to search features are offloaded into an external database, which could represent significant gains in Drupal database throughput. 4) Open source search engines are becoming increasingly sophisticated, and there are a number of emerging systems coming of age which developers may want to integrate in the future. Developers are often loathe to want to hack core, for reasons ranging from inexperience to lack of support for core modifications within the Drupal community. Being able to modify the underlying search components without the need for a core patch greatly simplifies the process of integration and likely increases the likelihood developers will want to attempt integration projects. 5) Search remains an important component of any Web site. One of the great strengths of Drupal is the ability to provide an integrated set of tools for managing content. The most significant benefit to adopting this approach could be the ability to expand the range of options available to implementers at the time a site is being built. Providing numerous search engine configuration options with relative benefits and drawbacks increases the strength of the platform. Best Regards, Michael Haggerty _______________________________________________ Michael Haggerty | Managing Partner | Trellon, LLC web www.trellon.com | email mhaggerty@trellon.com tel 240 643 6561 | aim haggerty321
The main thing I would like to see is splitting search module into a base search api module and a node search module. The latter could be disabled and replaced with any number of alternative node search modules. If it's possible for Xapian search to use the search api, then this might be an even easier solution -- just turn modules on and off, no need to modify a settings file. --mark On Mon, Jun 2, 2008 at 11:24 AM, Michael Haggerty <mhaggerty-lists@trellon.com> wrote:
I am writing to request comments on a proposed modification to Drupal core. Proposal: move the core search functionality in Drupal into an include file which can be specified on a per site basis. The actual functions used to return search results would not change, this proposed modification would simply put them all in a file which can be specified within settings.php. There are a number of reasons to consider doing this: 1) The flexibility to specify different search components would resolve some potential design deficiencies in the platform as it stands now. We have been doing some work integrating the Xapian search engine with Drupal. Instead of storing indexed results in the Drupal database, we are putting indexed content in Xapian instead. To get this to work properly, we have needed to patch core, which represents a deficiency both in terms of installation and long term maintenance of sites. Modifications to core mean people are working with unique distributions of Drupal instead of a common distribution, which are more complex to support. 2) This change would not differ significantly from design patterns currently in place within the framework. Other core components have already adopted this approach. An alternate cache.inc file can be specified within settings.php and is used by Robert Douglas' memcache module. Settings.php has had the ability to override system settings for a long time. 3) In some high traffic scenarios, database throughput becomes a bottleneck to the point where Drupal's current search features become unusable in the context in which they are deployed. Providing the flexibility to specify the underlying search components used for Drupal will provide system architects with additional options for scaling Drupal sites without losing characteristic features of the platform. In the case of Xapian, all database traffic related to search features are offloaded into an external database, which could represent significant gains in Drupal database throughput. 4) Open source search engines are becoming increasingly sophisticated, and there are a number of emerging systems coming of age which developers may want to integrate in the future. Developers are often loathe to want to hack core, for reasons ranging from inexperience to lack of support for core modifications within the Drupal community. Being able to modify the underlying search components without the need for a core patch greatly simplifies the process of integration and likely increases the likelihood developers will want to attempt integration projects. 5) Search remains an important component of any Web site. One of the great strengths of Drupal is the ability to provide an integrated set of tools for managing content. The most significant benefit to adopting this approach could be the ability to expand the range of options available to implementers at the time a site is being built. Providing numerous search engine configuration options with relative benefits and drawbacks increases the strength of the platform. Best Regards, Michael Haggerty _______________________________________________ Michael Haggerty | Managing Partner | Trellon, LLC
web www.trellon.com | email mhaggerty@trellon.com tel 240 643 6561 | aim haggerty321
Please open an issue for this (if there isn't one already) and move the discussion there. Personally, without any deep research, I like this idea. Much like cache.inc in D6. - Ken On Mon, Jun 2, 2008 at 2:47 PM, mark burdett <mfburdett@gmail.com> wrote:
The main thing I would like to see is splitting search module into a base search api module and a node search module. The latter could be disabled and replaced with any number of alternative node search modules. If it's possible for Xapian search to use the search api, then this might be an even easier solution -- just turn modules on and off, no need to modify a settings file.
--mark
On Mon, Jun 2, 2008 at 11:24 AM, Michael Haggerty <mhaggerty-lists@trellon.com> wrote:
I am writing to request comments on a proposed modification to Drupal core. Proposal: move the core search functionality in Drupal into an include file which can be specified on a per site basis. The actual functions used to return search results would not change, this proposed modification would simply put them all in a file which can be specified within settings.php. There are a number of reasons to consider doing this: 1) The flexibility to specify different search components would resolve some potential design deficiencies in the platform as it stands now. We have been doing some work integrating the Xapian search engine with Drupal. Instead of storing indexed results in the Drupal database, we are putting indexed content in Xapian instead. To get this to work properly, we have needed to patch core, which represents a deficiency both in terms of installation and long term maintenance of sites. Modifications to core mean people are working with unique distributions of Drupal instead of a common distribution, which are more complex to support. 2) This change would not differ significantly from design patterns currently in place within the framework. Other core components have already adopted this approach. An alternate cache.inc file can be specified within settings.php and is used by Robert Douglas' memcache module. Settings.php has had the ability to override system settings for a long time. 3) In some high traffic scenarios, database throughput becomes a bottleneck to the point where Drupal's current search features become unusable in the context in which they are deployed. Providing the flexibility to specify the underlying search components used for Drupal will provide system architects with additional options for scaling Drupal sites without losing characteristic features of the platform. In the case of Xapian, all database traffic related to search features are offloaded into an external database, which could represent significant gains in Drupal database throughput. 4) Open source search engines are becoming increasingly sophisticated, and there are a number of emerging systems coming of age which developers may want to integrate in the future. Developers are often loathe to want to hack core, for reasons ranging from inexperience to lack of support for core modifications within the Drupal community. Being able to modify the underlying search components without the need for a core patch greatly simplifies the process of integration and likely increases the likelihood developers will want to attempt integration projects. 5) Search remains an important component of any Web site. One of the great strengths of Drupal is the ability to provide an integrated set of tools for managing content. The most significant benefit to adopting this approach could be the ability to expand the range of options available to implementers at the time a site is being built. Providing numerous search engine configuration options with relative benefits and drawbacks increases the strength of the platform. Best Regards, Michael Haggerty _______________________________________________ Michael Haggerty | Managing Partner | Trellon, LLC
web www.trellon.com | email mhaggerty@trellon.com tel 240 643 6561 | aim haggerty321
-- Ken Rickard agentrickard@gmail.com http://ken.therickards.com
I am writing to request comments on a proposed modification to Drupal core. Proposal: move the core search functionality in Drupal into an include file which can be specified on a per site basis.
I'm all for more flexible search and indexing. But I fail to see how it is more modular to move all the code from a module and into an include file. Modules can already by be swapped for alternate implementations using our usual modules page. So this would just move something that can be done in the UI into something that has to be done by hand. Probably you want to do some refactoring along the way which is the important part of the proposal.
I invite you to join the search group on g.d.o and make this proposal there: http://groups.drupal.org/node/4102 Please also read all of the blog entries on search that came out of the MINNESOTA SEARCH SPRINT: http://drupal.org/search/node/MINNESOTA+SEARCH+SPRINT http://www.google.com/search?hl=en&q=MINNESOTA+SEARCH+SPRINT&btnG=Google+Sea... I hope that you help us with this. We have two visions. We'd like to make it possible to extend/separate the indexing and/or extend/separate the UI. Earnest Berry started on separating the UI, but AFAIK nobody has started work on separating the indexing. Thanks! Doug Michael Haggerty wrote:
I am writing to request comments on a proposed modification to Drupal core.
Proposal: move the core search functionality in Drupal into an include file which can be specified on a per site basis. The actual functions used to return search results would not change, this proposed modification would simply put them all in a file which can be specified within settings.php.
There are a number of reasons to consider doing this:
1) The flexibility to specify different search components would resolve some potential design deficiencies in the platform as it stands now. We have been doing some work integrating the Xapian search engine with Drupal. Instead of storing indexed results in the Drupal database, we are putting indexed content in Xapian instead. To get this to work properly, we have needed to patch core, which represents a deficiency both in terms of installation and long term maintenance of sites. Modifications to core mean people are working with unique distributions of Drupal instead of a common distribution, which are more complex to support.
2) This change would not differ significantly from design patterns currently in place within the framework. Other core components have already adopted this approach. An alternate cache.inc file can be specified within settings.php and is used by Robert Douglas' memcache module. Settings.php has had the ability to override system settings for a long time.
3) In some high traffic scenarios, database throughput becomes a bottleneck to the point where Drupal's current search features become unusable in the context in which they are deployed. Providing the flexibility to specify the underlying search components used for Drupal will provide system architects with additional options for scaling Drupal sites without losing characteristic features of the platform. In the case of Xapian, all database traffic related to search features are offloaded into an external database, which could represent significant gains in Drupal database throughput.
4) Open source search engines are becoming increasingly sophisticated, and there are a number of emerging systems coming of age which developers may want to integrate in the future. Developers are often loathe to want to hack core, for reasons ranging from inexperience to lack of support for core modifications within the Drupal community. Being able to modify the underlying search components without the need for a core patch greatly simplifies the process of integration and likely increases the likelihood developers will want to attempt integration projects.
5) Search remains an important component of any Web site. One of the great strengths of Drupal is the ability to provide an integrated set of tools for managing content. The most significant benefit to adopting this approach could be the ability to expand the range of options available to implementers at the time a site is being built. Providing numerous search engine configuration options with relative benefits and drawbacks increases the strength of the platform.
Best Regards, Michael Haggerty _______________________________________________
Michael Haggerty | Managing Partner | Trellon, LLC
web www.trellon.com | email mhaggerty@trellon.com tel 240 643 6561 | aim haggerty321
-- Doug Green douggreen@douggreenconsulting.com 904-583-3342 Bringing Ideas to Life with Software Artistry and Invention...
1,2) great but Nix the include file, use modules instead. It would be much better implemented using modules and search hooks. Otherwise we'll run into another image.inc and the support issues of how to swap out search systems. You will also end up with the inability to use more than once search system at a time which already afflicts core image, database, and cache API's. I'd rather not see that design patter repeated again. 3,4,5) I think flexibility is reason enough without fluff bullet points saying flexibility, flexibility. .darrel. On Mon, Jun 2, 2008 at 5:14 PM, Doug Green < douggreen@douggreenconsulting.com> wrote:
I invite you to join the search group on g.d.o and make this proposal there:
http://groups.drupal.org/node/4102
Please also read all of the blog entries on search that came out of the MINNESOTA SEARCH SPRINT:
http://drupal.org/search/node/MINNESOTA+SEARCH+SPRINT
http://www.google.com/search?hl=en&q=MINNESOTA+SEARCH+SPRINT&btnG=Google+Sea...
I hope that you help us with this. We have two visions. We'd like to make it possible to extend/separate the indexing and/or extend/separate the UI. Earnest Berry started on separating the UI, but AFAIK nobody has started work on separating the indexing.
Thanks! Doug
Michael Haggerty wrote:
I am writing to request comments on a proposed modification to Drupal core.
Proposal: move the core search functionality in Drupal into an include file which can be specified on a per site basis. The actual functions used to return search results would not change, this proposed modification would simply put them all in a file which can be specified within settings.php.
There are a number of reasons to consider doing this:
1) The flexibility to specify different search components would resolve some potential design deficiencies in the platform as it stands now. We have been doing some work integrating the Xapian search engine with Drupal. Instead of storing indexed results in the Drupal database, we are putting indexed content in Xapian instead. To get this to work properly, we have needed to patch core, which represents a deficiency both in terms of installation and long term maintenance of sites. Modifications to core mean people are working with unique distributions of Drupal instead of a common distribution, which are more complex to support.
2) This change would not differ significantly from design patterns currently in place within the framework. Other core components have already adopted this approach. An alternate cache.inc file can be specified within settings.php and is used by Robert Douglas' memcache module. Settings.php has had the ability to override system settings for a long time.
3) In some high traffic scenarios, database throughput becomes a bottleneck to the point where Drupal's current search features become unusable in the context in which they are deployed. Providing the flexibility to specify the underlying search components used for Drupal will provide system architects with additional options for scaling Drupal sites without losing characteristic features of the platform. In the case of Xapian, all database traffic related to search features are offloaded into an external database, which could represent significant gains in Drupal database throughput.
4) Open source search engines are becoming increasingly sophisticated, and there are a number of emerging systems coming of age which developers may want to integrate in the future. Developers are often loathe to want to hack core, for reasons ranging from inexperience to lack of support for core modifications within the Drupal community. Being able to modify the underlying search components without the need for a core patch greatly simplifies the process of integration and likely increases the likelihood developers will want to attempt integration projects.
5) Search remains an important component of any Web site. One of the great strengths of Drupal is the ability to provide an integrated set of tools for managing content. The most significant benefit to adopting this approach could be the ability to expand the range of options available to implementers at the time a site is being built. Providing numerous search engine configuration options with relative benefits and drawbacks increases the strength of the platform.
Best Regards, Michael Haggerty _______________________________________________
Michael Haggerty | Managing Partner | Trellon, LLC
web www.trellon.com | email mhaggerty@trellon.com tel 240 643 6561 | aim haggerty321
-- Doug Green douggreen@douggreenconsulting.com 904-583-3342
Bringing Ideas to Life with Software Artistry and Invention...
participants (7)
-
Darrel O'Pry -
Doug Green -
Ken Rickard -
Khalid Baheyeldin -
mark burdett -
Michael Haggerty -
Moshe Weitzman