Most people don't start by rewriting large parts of core.

True, but it does pretty much what I need it to do. That last mile, support for multilingual content is a deal breaker for me.

Over the years, we actually have resolved to implement some processes. Feel free to suggest improvements, but for continued good working relationship I suggest you scratch "backwards compatibility", "responsibility", and related items from your vocabulary.

I think that is a good sum up of where my wishes and those of the community differ. I'm looking for something that I can feel safe about using to build my company's corporate site off of. I didn't realize that Drupal is so much more technology focused than support focused. So, I apologize for all the drama. I'll stop now.

IIRC, I approved your account.

Yep, and thanks. You told me to wait until after the drupal.org upgrades so I can change my CVS password... still waiting.

Sorry to say so, but no aim is achieved without effort. If you don't feel like doing grunt work once in a while, the processes that need a review are yours not ours.

I have to agree with this. On that note, isn't there a way to automate the grunt work a bit? This is what I wound up doing while building off of the 4.7 release since the database definitions changed with pretty much every cvs update... I got pretty tired of having to manually go through and recreate the DB's for drupal, reimporting content, re-enabling my modules, setting the themes, etc. So I wrote a drupal bootstrapper. Basically it would automate the process of configuring a clean source tree all the way up to my configured website. Why not do that with scratch.drupal.org, bootstrap it to a desired state, and then run selenium on it. It won't give big code problems, but it will at least ensure that what worked before is still working. Anyways, just an idea. Thanks. Ben

Please observe that we have a database upgrade system. You do not need to recreate the DB with every CVS update.

I know that exists. However CVS updates usually bring more than just database changes. I was talking about automating bug testing a bit, not just updating the database. The reason I use a bootstrap file is during testing I like to return the database to a consistent state at the start of each test run. That consistent state is whatever in the drupal db create file. So my bootstrap file would start with an empty database, create the drupal tables, create some users, enable custom modules, set system variables, insert base content, etc. At the end of the bootstrap, if all actions were successful, I can be pretty sure that my custom version of Drupal is ready to use. Now, theoritically, I should be able to run Selenium against my installation. Log in, Insert a few nodes, check that they were there, delete some nodes, etc. Now I can be sure that all the code is still working correctly with the updates from CVS. This way it doesn't matter how much things change between CVS updates. I can just take 2 minutes to run my bootstrap and then selenium if all test passed, I go have a coffee. I personally find this really useful, it's not for everybody. I don't like doing grunt work...rather drink coffee. Ben.

On Fri, 2006-02-24 at 21:16 -0800, Benson Wong wrote:

I think that is a good sum up of where my wishes and those of the community differ. I'm looking for something that I can feel safe about using to build my company's corporate site off of. I didn't realize that Drupal is so much more technology focused than support focused. So, I apologize for all the drama. I'll stop now.

I think you have good feed back. I think we're all just a little defensive today :)... it happens... regardless... Drupal's focus on head/beta versions is very technical.. No company will offer you support, or for the most part even access to their beta code. The support for 4.6 is much better, and the code base is much more stable.

...

IIRC, I approved your account.

Yep, and thanks. You told me to wait until after the drupal.org upgrades so I can change my CVS password... still waiting.

...

Sorry to say so, but no aim is achieved without effort. If you don't feel like doing grunt work once in a while, the processes that need a review are yours not ours.

I have to agree with this. On that note, isn't there a way to automate the grunt work a bit?

This is what I wound up doing while building off of the 4.7 release since the database definitions changed with pretty much every cvs update...

This will happen if you work with the beta version of just about any piece of software. I think there has been a lot of strain for people trying to go into production on the shoulders of 4.7/HEAD pre-release and beta.. Drupal, the released versions, are really not all that unstable or difficult to upgrade. The module updates even the formAPI changes are pretty well documented and should be quick to follow for most drupal module developers... Hacking core... well you've got moving target issues.

Thanks. Ben

Op zaterdag 25 februari 2006 06:16, schreef Benson Wong:

I got pretty tired of having to manually go through and recreate the DB's for drupal, reimporting content, re-enabling my modules, setting the themes, etc. So I wrote a drupal bootstrapper. Basically it would automate the process of configuring a clean source tree all the way up to my configured website.

Maybe you want to look at sympal_scripts? Maybe you want to even contribute there? It sounds you aer pursueing the same as I do. http://cvs.drupal.org/viewcvs/drupal/contributions/tricks/sympal_scripts/ http://drupal.org/project/sympal_scripts Next up there: fixtures[1]: store the status of a site (content, vars etc) as PHP code (not in XML, not in SQL, not even YAML, but the language we all speak: PHP), for easy reimport. Great for distros. Even greater for development environments or testbeds. [1] http://wiki.rubyonrails.com/rails/pages/ActiveRecordYamlFixtures -- [ Bèr Kessels | Drupal services www.webschuur.com ]

Benson Wong wrote:

...

Most people don't start by rewriting large parts of core.

True, but it does pretty much what I need it to do. That last mile, support for multilingual content is a deal breaker for me.

There are plenty of suggestions for that,

...

Over the years, we actually have resolved to implement some processes. Feel free to suggest improvements, but for continued good working relationship I suggest you scratch "backwards compatibility", "responsibility", and related items from your vocabulary.

I think that is a good sum up of where my wishes and those of the community differ. I'm looking for something that I can feel safe about using to build my company's corporate site off of. I didn't realize that Drupal is so much more technology focused than support focused. So, I apologize for all the drama. I'll stop now.

Well, my argument is that Drupal _is_ perfectly safe to build hatever site you want to build with with it. Why is that so? Because it is Open Source. If you don't like it, change it. If you can't do it yourself hire somebody, if that somebody gets run over by a truck, hire somebody else. Try to do that with commercial software. There, if somethign fails, you can only hope they'll fix it. And if the publicly offered support through the forums (which is pretty good) isn't good enough, try to get a service agreement with somebody.

...

IIRC, I approved your account.

Yep, and thanks. You told me to wait until after the drupal.org upgrades so I can change my CVS password... still waiting.

Ah, sorry, well, it wasn't me who forgot the password. ;^)

...

Sorry to say so, but no aim is achieved without effort. If you don't feel like doing grunt work once in a while, the processes that need a review are yours not ours.

I have to agree with this. On that note, isn't there a way to automate the grunt work a bit?

We have discussed some ways of automation, but nobody has yet found the time to implement them. Two topics I remember: -- unit tests (see http://cvs.drupal.org/viewcvs/drupal/contributions/modules/simpletest/) -- automated patch testing.

This is what I wound up doing while building off of the 4.7 release since the database definitions changed with pretty much every cvs update...

I got pretty tired of having to manually go through and recreate the DB's for drupal, reimporting content, re-enabling my modules, setting the themes, etc. So I wrote a drupal bootstrapper. Basically it would automate the process of configuring a clean source tree all the way up to my configured website.

There are talks for an installer. But our update.php should take care of db updates. Even from contrib nowadays.

Why not do that with scratch.drupal.org, bootstrap it to a desired state, and then run selenium on it. It won't give big code problems, but it will at least ensure that what worked before is still working.

What is selenium in this context?

Anyways, just an idea.

Which I am eager to evaluate. Cheers, Gerhard

On Sun, 26 Feb 2006 09:28:32 +0100 Dries Buytaert wrote:

The real question is: why is upgrading _that_ important?

I basically agree with everything you just said. I'd like to offer one possible answer to this real question you raised: "because every module maintainer has a different opinion about how to manage fixes and improvements in the different versions of their modules, and drupal can't do much of anything without at least *some* modules enabled." It seems that if you want anything improved about your 4.6 site, you either have to manually maintain a forked copy of the 4.6 module code base (which is what i'm doing, while trying to submit as many patches back to the issue queues as possible), or you have to upgrade to 4.7. A few anecdotes to illustrate the point from my recent history of interacting with different module maintainers: 1) A trivial patch I submitted to the project.module to fix a permission problem in 4.6 wasn't going to be committed (without some prodding) because "4.6 is frozen", even though the code was actually broken (one of the access control permissions didn't work at all if you tried to enable it) and a 1-line change got it working. 2) A very handy feature in simple_access.module was discussed at great length as a 4.6 improvement, but finally the maintainer just committed it to the 4.7 version of simple_access and closed the issue. To further complicate this, the fix was done in the same commit as porting the entire module to the 4.7 forms API, so it's now going to be a lot of effort to extract this feature (which many people using 4.6 wish they had) from this monolithic commit and get it working in 4.6 (it doesn't appear to require 4.7 forms for any real reason). Even if I did the work, it's unclear if the maintainer will actually apply it to 4.6. 3) The maintainer of signup.module wants to completely re-write and re-organize the entire 4.6 signup codebase, to make it more modular and off-load more of the forms-related stuff it's doing into modules it should depend on (partially in response to the avalanche of patches i've submitted that provide new functionality I needed). So, folks trying to use the "stable" 4.6 signup might find that one day, when they download the "4.6 signup" (since there's no versioning of modules, something that makes doing your own revision control with vendor branches more of a pain), all of a sudden the entire look and functionality of the module has completely changed from the last 4.6 site they setup, it now depends on 3 other modules, etc. So, 3 very handy modules have 3 completely different philosophies on what to do with their "stable" version. Project didn't even want to fix a known bug with a trivial solution, since (at first) that was considered too much of a change to a now-frozen stable release. simple_access seems willing to do bug fixes to 4.6, but anything new goes into 4.7. Signup seems willing to completely change everything, right in the middle of a "stable" version. I'm still not sure what I think about the fact that there's no consistency across modules in this respect. Of course, it's great that ultimately, everything is open source, and I can (and do) just get my own 4.6 site working exactly how I want, with some combination of stock modules, custom modifications, and occasionally, updates to the modules from the real maintainers when they fix or add something I care about. However, that's not an ideal model, since it makes the cost of using drupal pretty high. I happen to have been writing code and using CVS for well over a decade before I ever heard of drupal, so even though I had never written a line of php a few months ago, I could add a bunch of functionality I needed and fix a lot of bugs I found. But most people trying to use drupal to setup a website aren't going to have those skills and that experience to draw from (nor many thousands of dollars to hire someone who does). If one of the motivating philosophies behind drupal is "down with The Man, let's help the underdog" (which I totally support), that doesn't quite jive with "only if the underdog is already a computer programmer, or has a $20K budget for their website". It also complicates the question "why bother upgrading?". Depending on what modules you need, you're often faced with the difficult (and somewhat evil) choice of a) redo a lot of work someone already did to the 4.7 version or b) upgrade your entire site to 4.7. This is *not* an argument for crippling drupal by tying its hands to backwards compatibility. We've gone down that road for the university research project I work on for my day job, and it's a *major* waste of developer time, source of bloat, and maintenance costs. I'm just raising the issue that while the core has a clear philosophy and (mostly) clear process, the modules are all over the map, and precisely because of drupal's (correct) emphasis on modularity, core drupal *on its own* isn't very useful for doing much. I'm still too new around here to have any profound insights on what to do about all this, so unfortunately I don't (yet) have a concrete suggestion to share. Given the distributed ownership and maintenance of the contrib modules, I don't think it'd be possible (or wise) to enforce any kind of uniformity. I just wanted to offer my perspective as a new drupal user and developer, trying to keep a live site happy and stable in 4.6, but still getting new functionality or bug fixes I need to solve real problems I'm facing. I hope this is helpful input into the discussion... Thanks, -Derek

On Sun, 26 Feb 2006 17:05:45 -0800 "Benson Wong" wrote:

I dream of Drupal switching from cvs to svn.

i don't understand all the fascination with which revision control system to use. they all do roughly the same thing. most of our problems don't stem from what tool we're using to manage the revisions, it's the social policy/procedure for what (bug/feature), where (branch) and who makes the changes. that policy for the core is pretty clear, consistent, and good (from my limited contact). but, i don't see changing tools as a means to get closer to having a policy like that for the contrib modules. sure, svn gets some things better than cvs, and i'm sure baazar-ng is nice, too, but there's nothing fundamental about any of them that can solve the kinds of issues i'm talking about. most of the reasons i've heard for switching to svn or baazar-ng are things that can be done (albeit with a little effort) in cvs. maybe it's just that i've used cvs for *so long* and know nearly every trick there is.

Better access methods (https)

for developers with drupal.org source access, we could do cvs over ssh if we really cared.

, finer grain access control,

in cvs you can control as fine as you want, all the way down to: "this 1 user can only modify 1 file on 1 branch".

access control lists

not exactly sure what you mean by this, but i'm pretty sure whatever you have in mind is something you can already do in cvs (if you know every possible knob you can turn). it's also totally unrelated to a policy/vision for managing versions and changes within contrib.

It's very little work for a contributor to checkout the 4.6.5 version of their module, fix some bugs, and commit the changes just for that version of drupal. Much easier with SVN than CVS (I think).

no, this much cvs and svn can do with equal ease.

Another plus is that people who depend on their code can just export the correct tag.

cvs can do this, too.

ie: svn co https://svn.drupal.org/respositories/contributions/my_module/tags/ 4.6.5

cvs co -d... -r 4_6_5 contributions/my_module -derek p.s. does anyone actually care if i don't capitalize my emails to this list? this is one of the signs of a "good post" in the faq about development@drupal.org. when i had a period of bad wrist problems and ergonomic woes, one of the suggestions was to give up the shift key, since it puts extra strain on your fingers. i've been out of the habit ever since.

1) A trivial patch I submitted to the project.module to fix a permission problem in 4.6 wasn't going to be committed (without some prodding) because "4.6 is frozen", even though the code was actually broken (one of the access control permissions didn't work at all if you tried to enable it) and a 1-line change got it working.

Derek, I agree with your general point on 4.6. In the issue you reference, http://drupal.org/node/49408, I didn't mean that 4.6 was (formally) frozen--just that, to my knowledge, no one was doing much active maintenance. But this small distinction gets fairly near the core of the issues we're discussing. The situation is that a lot of (paid, economically significant) work depends on volunteer contributions--from Dries on down to module maintainers. The thing about volunteers is that you can't really require them to do work. They'll contribute largely what they want to, what interests and motivates them. Generally speaking, when there's limited time available, doing maintenance on old releases is often going to rate relatively low when measured up against, e.g., getting an innovative new version out. So maybe the question really becomes: how do we enable those who depend on old releases to contribute to maintaining them? Maybe Dries' welcome approach of having a 'release maintainer' is worth looking at here for more than core. Major users of modues could adopt old releases when the module author/maintainer no longer wishes to maintain them.

On Mon, 27 Feb 2006 23:30:57 -0500 Walt Daniels wrote:

I think there are bugs which deserve a similar high priority, e.g. the MySQL 5.12+ bug.

there's no 1 bug. this bug is spread out over numerous contrib modules. the best summary and solution (that i've seen) is described here: http://drupal.org/node/43735#comment-92865 starbow (the author of that post) and i have submitted patches to the event, image, taxonomy, and project modules. so far, the one for event was applied, but none of the others have been committed (yet). there are probably other modules where this is a problem (e.g. http://drupal.org/node/51696), but we've only found/fixed this bug for the modules we're actively using. the solution is usually very easy: you just have to re-order the tables in a SELECT query so that if you're JOIN'ing w/ the {node} table, that it comes first. otherwise, db_rewrite_sql() will break SQL standards compliance if you enable any of the node access modules (node_privacy_byrole, simple_access, taxonomy_access, etc), since they add an "INNER JOIN {node_access} ON na.nid = n.nid" before the "{node} n" in the query. it's really just up to each module maintainer to apply these patches that already exist, or, in cases where there's no patch yet, a relatively trivial change to the code can get their module to work with MySQL 5.0.12+. i'd be happy to help folks test patches to various modules if needed. -derek p.s. for the interested reader, here are links to the not-yet-applied patches: image -- http://drupal.org/node/49433 project -- http://drupal.org/node/50572 taxonomy -- http://drupal.org/node/49430

On Feb 27, 2006, at 8:30 PM, Walt Daniels wrote:

For instance there is a severe problem that prevents 4.6 working with MySQL 5.12+ and ISPs are converting to it.

http://dev.mysql.com/downloads/mysql/5.1.html NOTE: This alpha release, as any other pre-production release, should not be installed on production level systems or systems with critical data. It is good practice to back up your data before installing any new version of software. Kieran

On Mon, 2006-02-27 at 20:11 -0800, Nedjo Rogers wrote: <snip>

But this small distinction gets fairly near the core of the issues we're discussing. The situation is that a lot of (paid, economically significant) work depends on volunteer contributions--from Dries on down to module maintainers.

The thing about volunteers is that you can't really require them to do work. They'll contribute largely what they want to, what interests and motivates them. Generally speaking, when there's limited time available, doing maintenance on old releases is often going to rate relatively low when measured up against, e.g., getting an innovative new version out.

So maybe the question really becomes: how do we enable those who depend on old releases to contribute to maintaining them?

<snip> If your economic welfare depends on drupal, set up a cvs/svn/bzr repository with drupal 4.6 as a vendor branch, make the changes and bug fixes you need. You should be willing to contribute your upgrades back to the community and hope they get accepted back into the main tree. The fact is your business needs are probably immediate, the development communities needs are probably less immediate than your friday deadline. If you want to build your business on an open source foundation, be prepared to do some work. You can't guarantee that community process will conform to your businesses needs, or that if they do conform for a period, that they will stay that way. .darrel.

On Tue, 28 Feb 2006 12:11:19 -0500 "Darrel O'Pry" wrote:

If your economic welfare depends on drupal, set up a cvs/svn/bzr repository with drupal 4.6 as a vendor branch, make the changes and bug fixes you need.

for the record: a) i'm not an enterprise, just an individual user doing all this drupal work on my own time, for my own private site for a brazilian percussion ensemble i direct. ;) i'd like to start using drupal for my day job (academic staff @ the uw-madison comp sci department), but that's another story... b) setting up a cvs repository and importing drupal 4.6 as a vendor branch is exactly what i did. (side note: the fact that contrib modules don't have real version numbers makes using vendor branches more of a pain, since you have to tag based on the date you imported, not some reasonable version number of a given contrib module).

You should be willing to contribute your upgrades back to the community and hope they get accepted back into the main tree.

that's what i've been trying to do. my point is that it's not so easy to do this (especially given the divergent interests and policies of each contrib module maintainer), and it's time consuming and (so far) somewhat unrewarding work.

If you want to build your business on an open source foundation, be prepared to do some work.

i agree (even though i'm not a buisness). i already have done lots of work. since it's open source, and i believe in open source in general, i'm spending a bunch of *additional* effort trying to get my fixes and improvements back into the main version so everyone can benefit from them. i'm just trying to make suggestions about what we as a community can do to make that process easier. i don't think that just because people *can* fork any module and maintain their own version of it means that should be the first suggestion.

You can't guarantee that community process will conform to your businesses needs, or that if they do conform for a period, that they will stay that way.

i'm not motivated by having drupal conform to "buisness needs" per se, i'm just talking about basic usefulness to the world. my point boils down to this: because drupal is so modular, every drupal site depends on at least a few contrib modules. there's a clear social policy and process for changes to core that people can at least understand and live with. there's no such policy for contrib modules. since sites depend on both core and contrib, they're usually stuck in the middle of inconsistencies regarding when/where things get fixed or improved. i think it'd be in drupal's overall interests to try to close this gap in some way. i've been trying to come up with ideas and suggestions, but it's a hard problem, and i still feel very much like an outsider in this community. i'm not slaming anyone, or complaining, i'm just trying to help make improvements based on my own experience interacting with drupal so far. thanks, -derek

That would be a change in policy. Our current policy is to support the current stable release and the one before it. That would mean that 4.6 would be not supported after the release of the version after 4.7.

thats a result of limited resources for the security team. i suspect that 4.6 will live on for years because either someone funds the drupal.org security team or some volunteers to maintain the branch.

Where we often differ is in our culture of backward compatibility. From day one, I decided not to care about backward compatibility and it has been one of our core values ever since. (However, we only break your code, not your data.) Either you like that, or you don't.

Just to be clear, we did not do this for no reason. The reason for not maintaining backward compatibility is stop this as a hinderance for innovation. As painful as upgrades may be, the disregard for compatibility in core is one of the main reasons Drupal continues to develop new features, refactor existing ones, and be cutting edge, yet nimble