Why Drupal 4.7 is late? (long)
First of all, it is not late. It'll be out when it's done and noone said it otherwise. However, it was announced quite some ago that "In preparation of the Drupal 4.7.0 release, development of Drupal core will be frozen on September 1st." http://drupal.org/node/28466 On September 1st, Dries announced http://drupal.org/node/30176 that September 15 is the new code freeze date mainly due to form API and upgrade system improvements. He hoped to have a release candidate on October 7. Form API thread http://drupal.org/node/29465 started on August 23. The Form API was committed to core in October 7. In a mere three months it saw 48 revisions. Bootstrap, for example saw 80 in two years. Even node.module needed four months to reach v1.48 but that was almost five years ago and much, much less code and users were affected. Form API is a big change. Not just the way the forms are defined but the whole workflow is changed. You no longer have access to the POSTed information on form build time, you can interact with that later on via various callbacks. This immense change is percieved as hard but it is not. Aaron Welch (also known as crunchywelch) have converted nothing less than flexinode single handedly in three days time and this included several form API fixes -- this was back when form API itself was pretty bogus. His estimate (and mine) is that it can be done in a day now that form API is fixed up. Other modules have also been converted. Ecommerce is under way. Module developers, brace yourselves and try it. It's not hard if you first forget the old ways. But alas the form API conversion of core were not made by form API experts. You know why? It's so simple. Think a bit on it. I think you guessed it: because there were none (with Adrian being the exception, of course). But how could there be? It's a whole new API. We struggled, learned it, added stuff to the API itself when there was a need. So, there are quite a lot of bugs in core resulting from this conversion and the quality is not even. Currently, there are only a handful people who can -- or want? -- to hammer these bugs out. Everyone, it's impossible for a small handful of developers to fix them all. Often patch fixing efforts abandoned and I need to take over. I took the form API mantle but it's impossible to cope with the load! Currently, being a form API maintainer means that I need to understand _all_ core (there are approx. a thousand functions and some 32K LoC) on an astonishing level because various bugs that were introduced by form API fall somehow on me. If I'd be a millionarie with nothing else to do but fixing core bugs, it'd be a different situation but alas I need to earn money. Do you except the menu maintainer to fix all those menu permission issues just because menu system is used? So, I ask for the help of seasoned developers. Form API is not voodoo magic. It is not written in Brainfuck. It's PHP code which you guys can understand pretty well. It's even documented even if the documentation is a bit spotty. But there are good things also -- because fixing all this stuff took this long, the upgrade system could run its course and got commited on December 6. It's also a very good thing, now modules can use the upgrade system. This sounds a bit deceiving but if you think that there is no difference on a database level between create and alter then you'll quickly realize that this is a pillar for install system. And we can also say that when you have added a feature to your site by installing a new module, it's a new version of that particular site, it's indeed an upgrade. Thanks for bearing with me Karoly Negyesi Ps. If anyone wants to start bitching around "you should have released without form API", then first please show your comment with a September date asking for that on Drupal.org -- I met none in the roadmap posts or the form API thread.
This very useful and puts things into perspective very clearly - Thanks chx!
I found this module very, very useful to begin to port 4.6 code to the new form api: http://drupal.org/node/37457 It is a form updater module that would really help newcomers to the new form api as myself. Give it a try and you won't regret. Robert Garrigos Karoly Negyesi wrote:
First of all, it is not late. It'll be out when it's done and noone said it otherwise.
However, it was announced quite some ago that "In preparation of the Drupal 4.7.0 release, development of Drupal core will be frozen on September 1st." http://drupal.org/node/28466
On September 1st, Dries announced http://drupal.org/node/30176 that September 15 is the new code freeze date mainly due to form API and upgrade system improvements. He hoped to have a release candidate on October 7. Form API thread http://drupal.org/node/29465 started on August 23.
The Form API was committed to core in October 7. In a mere three months it saw 48 revisions. Bootstrap, for example saw 80 in two years. Even node.module needed four months to reach v1.48 but that was almost five years ago and much, much less code and users were affected.
Form API is a big change. Not just the way the forms are defined but the whole workflow is changed. You no longer have access to the POSTed information on form build time, you can interact with that later on via various callbacks. This immense change is percieved as hard but it is not. Aaron Welch (also known as crunchywelch) have converted nothing less than flexinode single handedly in three days time and this included several form API fixes -- this was back when form API itself was pretty bogus. His estimate (and mine) is that it can be done in a day now that form API is fixed up. Other modules have also been converted. Ecommerce is under way. Module developers, brace yourselves and try it. It's not hard if you first forget the old ways.
But alas the form API conversion of core were not made by form API experts. You know why? It's so simple. Think a bit on it. I think you guessed it: because there were none (with Adrian being the exception, of course). But how could there be? It's a whole new API. We struggled, learned it, added stuff to the API itself when there was a need. So, there are quite a lot of bugs in core resulting from this conversion and the quality is not even. Currently, there are only a handful people who can -- or want? -- to hammer these bugs out. Everyone, it's impossible for a small handful of developers to fix them all. Often patch fixing efforts abandoned and I need to take over. I took the form API mantle but it's impossible to cope with the load! Currently, being a form API maintainer means that I need to understand _all_ core (there are approx. a thousand functions and some 32K LoC) on an astonishing level because various bugs that were introduced by form API fall somehow on me. If I'd be a millionarie with nothing else to do but fixing core bugs, it'd be a different situation but alas I need to earn money. Do you except the menu maintainer to fix all those menu permission issues just because menu system is used?
So, I ask for the help of seasoned developers. Form API is not voodoo magic. It is not written in Brainfuck. It's PHP code which you guys can understand pretty well. It's even documented even if the documentation is a bit spotty.
But there are good things also -- because fixing all this stuff took this long, the upgrade system could run its course and got commited on December 6. It's also a very good thing, now modules can use the upgrade system. This sounds a bit deceiving but if you think that there is no difference on a database level between create and alter then you'll quickly realize that this is a pillar for install system. And we can also say that when you have added a feature to your site by installing a new module, it's a new version of that particular site, it's indeed an upgrade.
Thanks for bearing with me
Karoly Negyesi
Ps. If anyone wants to start bitching around "you should have released without form API", then first please show your comment with a September date asking for that on Drupal.org -- I met none in the roadmap posts or the form API thread.
But there are good things also -- because fixing all this stuff took this long, the upgrade system could run its course and got commited on December 6. It's also a very good thing, now modules can use the upgrade system.
Where can I find some docs of this upgrade system? I don't find it in the developers handbook. Thanks, Robrecht
there is no documentation but it works the same way as you can see it in drupal core webchick did an example of how a .install file should look like: http://drupal.org/node/43492 cu tobi Am Sunday 15 January 2006 11:19 schrieb Robrecht Jacques:
But there are good things also -- because fixing all this stuff took this long, the upgrade system could run its course and got commited on December 6. It's also a very good thing, now modules can use the upgrade system.
Where can I find some docs of this upgrade system? I don't find it in the developers handbook.
Thanks, Robrecht
2006/1/15, Tobias Maier <tobias.maier@gmail.com>:
Am Sunday 15 January 2006 11:19 schrieb Robrecht Jacques:
Where can I find some docs of this upgrade system? I don't find it in the developers handbook.
there is no documentation but it works the same way as you can see it in drupal core
Thanks. I'll check out the example and the way it works in core. That's enough docs for me. Maybe the "upgrading modules from 4.6 to 4.7" page could mention this new upgrade system though. It's a good thing these "place upgrade-<module>.php in your root and run it" instructions for modules can disappear.
webchick did an example of how a .install file should look like: http://drupal.org/node/43492
Robrecht
Well said Karoly. 4.7 is not late. It is done when it is done, and that is the way things are, and will be for some time. If some company wants it sooner, they can commit more resources to it (money, developers, technical writers, ...etc.).
Folks sorry to be rude, but this is off topic. So far, only Drumm has seeked me out in private and offered a helping hand in fixing bugs. There are enough bugs for a roman legion. All hands on board!
webchick did an example of how a .install file should look like: http://drupal.org/node/43492
If you're talking form api problems exclusively, I think I have enough of a grip to be useful. Is there any way I can identify them specifically? On 1/15/06, Karoly Negyesi <karoly@negyesi.net> wrote:
Folks sorry to be rude, but this is off topic.
So far, only Drumm has seeked me out in private and offered a helping hand in fixing bugs. There are enough bugs for a roman legion. All hands on board!
webchick did an example of how a .install file should look like: http://drupal.org/node/43492
On Sun, 15 Jan 2006 18:17:14 +0100, Earl Dunovant <prometheus6@gmail.com> wrote:
If you're talking form api problems exclusively, I think I have enough of a grip to be useful. Is there any way I can identify them specifically?
besides the obvious 'form' and 'edit' keyword I'm looking out for 'save' 'validation' , names of elements (checkbox, radio etc) and then just gut instinct, this http://drupal.org/node/44265 issue makes me think "hmm does the login submit work correctly? is the goto there?". Currently the biggest problem is editing comments. http://drupal.org/node/43325 there is ongoing work but it needs finishing. Then testing to hell. And some then testing would be nice... Thanks a lot Karoly Negyesi
On 15 Jan 2006, at 8:21 PM, Karoly Negyesi wrote:
On Sun, 15 Jan 2006 18:17:14 +0100, Earl Dunovant <prometheus6@gmail.com> wrote:
If you're talking form api problems exclusively, I think I have enough of a grip to be useful. Is there any way I can identify them specifically?
besides the obvious 'form' and 'edit' keyword I'm looking out for 'save' 'validation' , names of elements (checkbox, radio etc) and then just gut instinct, this http://drupal.org/node/44265 issue makes me think "hmm does the login submit work correctly? is the goto there?".
I have a patch here that moves the goto out of the _submit functions and into drupal_submit_form. The issue is that we allow you to have multiple submit functions, but the way it's coded now the first submit function you call very likely redirects you. The patch introduces a return value of the path to be redirected to for _submit function. It's almost done and I will upload it soon, my question is .. do you think it's a realistic default to force a drupal_goto() after all successful form submissions, as 99% of forms seem to use that. There's also the case of some forms misusing the _submit function to trigger a preview form which i need to look at still. -- Adrian Rossouw Drupal developer and Bryght Guy http://drupal.org | http://bryght.com
On Monday 16 January 2006 03:13, Adrian Rossouw wrote:
It's almost done and I will upload it soon, my question is ..
do you think it's a realistic default to force a drupal_goto() after all successful form submissions, as 99% of forms seem to use that.
Frankly I hate the goto() calls. :-) Doesn't that double the load on the server, since Drupal gets built up, processes the request, redirects, and then gets built up AGAIN to display the page it was already at? What's the advantage of the goto, when the downside is doubling the load on the system? -- Larry Garfield AIM: LOLG42 larry@garfieldtech.com ICQ: 6817012 "If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it." -- Thomas Jefferson
Frankly I hate the goto() calls. :-) Doesn't that double the load on the server, since Drupal gets built up, processes the request, redirects, and then gets built up AGAIN to display the page it was already at? What's the advantage of the goto, when the downside is doubling the load on the system?
AFAIK, this helps avoid Refresh and back button issues with form submissions, and also takes care of cache_clear_all calls which would require a reload.. -K
On 16 Jan 2006, at 5:11 PM, Larry Garfield wrote:
Frankly I hate the goto() calls. :-) Doesn't that double the load on the server, since Drupal gets built up, processes the request, redirects, and then gets built up AGAIN to display the page it was already at? What's the advantage of the goto, when the downside is doubling the load on the system? And changes in the form need to propogate.
Since the form is created before the submit is called (with no way to recreate it) meaning that when you display the form (after having submitted it) any changes you have made don't reflect yet. The fact that the values you are seeing as correct are just because they were in the _POST, and overrode the defaults provided (the things you need to change). -- Adrian Rossouw Drupal developer and Bryght Guy http://drupal.org | http://bryght.com
Frankly I hate the goto() calls. :-) Doesn't that double the load on the server, since Drupal gets built up, processes the request, redirects, and then gets built up AGAIN to display the page it was already at? What's the advantage of the goto, when the downside is doubling the load on the system?
1) After submitting a form, you are usually taken to the result at its normal viewing url (e.g. node/123) rather than the form's URL (node/add/page or node/123/edit). This makes URLs more consistent. 2) Any changes made to already-loaded Drupal pieces (e.g. the menu, the modules loaded, cache, the form itself, ...) will take effect immediately. 3) You can refresh the result without resubmitting the data (this used to be a huge problem on Drupal.org even when we were small). 4) You can provide permanent GET URLs for forms that are used for browsing only (e.g. search). The amount of work duplicated is not as big as you seem to think. The POST request only deals with submitting the data and saving it. The GET request afterwards only deals with loading the data and displaying. They are mostly complementary. Steven
On Jan 16, 2006, at 9:11 AM, Larry Garfield wrote:
Doesn't that double the load on the server, since Drupal gets built up, processes the request, redirects, and then gets built up AGAIN to display the page it was already at? What's the advantage of the goto, when the downside is doubling the load on the system?
While an obvious point, it's also worth mentioning that you're not doubling the entire site's load. The goto and the second call to Drupal only happen on form-posting activities, and the page content isn't downloaded the first time round. So while Drupal is getting built up twice, the expensive part of sending out the page request only occurs once. I know that you know this, but it helps to think of things in a cost/ benefit mindset. Most sites are read-heavy, so the redirect only occurs on a very small percentage of requests. Even an active site will see less than 1 post for every 500 views, so creating a 501th request is well worth it to reap the benefits already discussed in this thread. Allie Micka pajunas interactive, inc. http://www.pajunas.com scalable web hosting and open source strategies
Allie Micka wrote:
I know that you know this, but it helps to think of things in a cost/ benefit mindset. Most sites are read-heavy, so the redirect only occurs on a very small percentage of requests. Even an active site will see less than 1 post for every 500 views, so creating a 501th request is well worth it to reap the benefits already discussed in this thread.
FYI, in my tests, the expensive part of Drupal is the bootstrapping.
Earl Miles wrote:
Allie Micka wrote:
I know that you know this, but it helps to think of things in a cost/ benefit mindset. Most sites are read-heavy, so the redirect only occurs on a very small percentage of requests. Even an active site will see less than 1 post for every 500 views, so creating a 501th request is well worth it to reap the benefits already discussed in this thread.
FYI, in my tests, the expensive part of Drupal is the bootstrapping.
That is not the bootstrap process as such, but loading and parsing all the files, that is a general PHP issue. Two known solutions exist: Opcode caches or split mode. :) Cheers, Gerhard
Gerhard Killesreiter wrote:
FYI, in my tests, the expensive part of Drupal is the bootstrapping.
That is not the bootstrap process as such, but loading and parsing all the files, that is a general PHP issue. Two known solutions exist: Opcode caches or split mode. :)
Agreed. I read through a bunch of your posts about your experimentations with such, and decided it just wasn't worth bothering with. But it is worth remembering that drupal_goto is not a free operation.
Robrecht Jacques wrote:
Where can I find some docs of this upgrade system? I don't find it in the developers handbook.
contrib/docs/developer/hooks/update.php Looks like this was finally indexed by Drupal Docs, http://drupaldocs.org/api/head/function/hook_update_N, but it went and proved that HTML isn't allowed. I'll fix that in a bit unless someone else does. Btw, what version of Drupal is Drupal Docs running? I'll probably set up a copy locally if it is CVS. -- Neil Drumm http://delocalizedham.com/
Thank you Karoly, for this very well written mail! It clarifies a lot. But, (as always) I'd like to play the devils advocate here. As I recently wrote on my blog [1] there are a few things to consider: * Its done when its done is not a good attitude hen you want to keep professionals (aka those-with-agendas-plannings-and-deadlines) on board. They need insurance that their project can go live on Date X. * If you want people to help fixing bugs in HEAD, you must allow them to use head for development. IE: make it as stable as possible and give insurance about when things will be done. * If you don't provide that certainty, professional developers will have to decide to develop on and for the last stable release. Hence pulling away resources that *could* have helped with the HEAD. That could have fixed numerous bugs. I still see lots of 4.6 modules being released. Meaning that people still do no trust HEAD. So, I am not debating against the points you make. For the reasons you give for why we are where we are are very well put, and very true. I just hope, sincerely, that we learn from this; and never ever make the almost classic OSS mistake to slip in huge changes in the end. Again. And that we could reconsider some of our release cycle methods. For, I fear this is not a standalone incident, but, with the growth-rate of Drupal will rather become standard. Meaning there will probably be no 4.8 before 2007/8.
Ps. If anyone wants to start bitching around "you should have released without form API", then first please show your comment with a September date asking for that on Drupal.org -- I met none in the roadmap posts or the form API thread. This, however, is the only part I really disagree with. We all know that the FormAPI was met with great Resistance (hefty discussions at the drupalconference). Should we all archive our concerns? I am sure, that on IRC, in mails and maybe evenon Drupal.org I have been against this big change at the end of the cycle. But that does not mean that I had to (try to) stop the patch from going in, does it? Should we always vote against things that we a) do not yet fully understand or b) do not agree upon? It is a bad thing, IMO to have to -1 all decisions, that one does not really understand but that 'might possibly have a great impact, of which one thinks, it might turn out negative'. In that case one should not vote. Which, is not the same as a +1 !
[1] http://www.webschuur.com/node/410 Bèr -- PGP ber@webschuur.com http://www.webschuur.com/sites/webschuur.com/files/ber_webschuur.asc PGP berkessels@gmx.net http://www.webschuur.com/sites/webschuur.com/files/ber_gmx.asc
show your comment with a September date
This, however, is the only part I really disagree with. We all know that the FormAPI was met with great Resistance (hefty discussions at the drupalconference). Should we all archive our concerns? I am sure, that on IRC, in mails and maybe evenon Drupal.org I have been against this big change at the end of the cycle.
That was in October! I mean, by October it was quite clear that we are slipping but -- OK well, we got into a classical "dollar bidding" situation, and even I was unable to see that. But you were neither: http://drupal.org/node/29465#comment-44087 was written by you.
Bèr Kessels wrote:
Thank you Karoly, for this very well written mail! It clarifies a lot.
But, (as always) I'd like to play the devils advocate here.
I'd rather see you reviewing patches.
As I recently wrote on my blog [1] there are a few things to consider: * Its done when its done is not a good attitude hen you want to keep professionals (aka those-with-agendas-plannings-and-deadlines) on board. They need insurance that their project can go live on Date X.
There is no insurance. There won't ever be one. Look at how often large software companies postpone releases. And those have real clients. The Drupal project as such does not have clients. Your clients are your problem, nobody else's. Also, if you want to be Drupal a viable development plattform in the future it will be wise to invest some of your time into working on HEAD even if it is not tied to any current project.
* If you want people to help fixing bugs in HEAD, you must allow them to use head for development. IE: make it as stable as possible
We already do that.
and give insurance about when things will be done.
See above.
* If you don't provide that certainty, professional developers will have to decide to develop on and for the last stable release. Hence pulling away resources that *could* have helped with the HEAD. That could have fixed numerous bugs. I still see lots of 4.6 modules being released. Meaning that people still do no trust HEAD.
I see it the other way round: If you work a lot on head, you can judge whether you can use head for your clients. OTOH, I find it quite normal to use the latest stable release for a project.
So, I am not debating against the points you make. For the reasons you give for why we are where we are are very well put, and very true. I just hope, sincerely, that we learn from this; and never ever make the almost classic OSS mistake to slip in huge changes in the end. Again. And that we could reconsider some of our release cycle methods. For, I fear this is not a standalone incident, but, with the growth-rate of Drupal will rather become standard. Meaning there will probably be no 4.8 before 2007/8.
Ps. If anyone wants to start bitching around "you should have released without form API", then first please show your comment with a September date asking for that on Drupal.org -- I met none in the roadmap posts or the form API thread.
This, however, is the only part I really disagree with. We all know that the FormAPI was met with great Resistance (hefty discussions at the drupalconference).
The discussions were mainly related to a rather controversial idea some people had: Provide a backwards compatibility layer. I still can't believe how much time I spent argueing against something that in the end turned out to be not possible. To incorporate the forms API was neccessary from a security point of view. Anybody who did not contribute to rolling out a Drupal security release at 4am should simply stfu when it comes to security issues.
Should we all archive our concerns? I am sure, that on
lists.drupal.org/pipermail/development/ :p
IRC, in mails and maybe evenon Drupal.org I have been against this big change at the end of the cycle. But that does not mean that I had to (try to) stop the patch from going in, does it?
I didn't notice much of your opposition. Good for you. :p
Should we always vote against things that we a) do not yet fully understand or b) do not agree upon? It is a bad thing, IMO to have to -1 all decisions, that one does not really understand but that 'might possibly have a great impact, of which one thinks, it might turn out negative'. In that case one should not vote. Which, is not the same as a +1 !
Whatever. Cheers, Gerhard
On 15/01/06, Karoly Negyesi <karoly@negyesi.net> wrote:
First of all, it is not late. It'll be out when it's done and noone said it otherwise.
I might be speaking out of turn here, but I totally agree with this sentiment. However, when is done done? For any software project, regardless of how simple or complex it is, or what model it follows, there will _always_ be bugs. So, it's something of a catch-22 that the longer the delay, the more bugs are found and the longer the delay and .. There needs to be a line drawn somewhere.. I realise you've linked to a couple of issues on here, but I'm sure there are more. Is there a list of issues, that if solved will lead to a 4.7 release? If not, shouldn't there be one? I've also read on the forum that the bug-fixing parties concentrate on all outstanding bugs and patches, regardless of whether they are to be included in 4.7. So, as someone has already attempted to organise, there should be a Priority level named "Hyper-critical" or something similar, that can only be set/unset by the primary core developers. This should allow better control of the project's workflow and enable everybody to concentrate on getting 4.7 out the door. Or in the meantime, a list of such issues could be posted, publicised and regularly updated somewhere.. Anyways, I'd love to help if/where I am able to, but only if I know what to help with exactly. IMO, we just need to tackle this in a systematic manner. Thanks for listening :) Regards, Karthik.
On 1/16/06, Karthik <narakasura@gmail.com> wrote:
I realise you've linked to a couple of issues on here, but I'm sure there are more. Is there a list of issues, that if solved will lead to a 4.7 release? If not, shouldn't there be one?
Bugzilla has a flag which can be applied to bugs which indicate that they "block" a specific release. This flag is restricted to a few users. It would be nice if Drupal used Bugzilla but since it doesn't perhaps the issue system could add this feature. Cheers, Chris
participants (17)
-
Adrian Rossouw -
Allie Micka -
Bèr Kessels -
Chris Cook -
Earl Dunovant -
Earl Miles -
Gerhard Killesreiter -
Karoly Negyesi -
Karthik -
Khalid B -
Larry Garfield -
Neil Drumm -
puregin -
Robert Garrigós Castro -
Robrecht Jacques -
Steven Wittens -
Tobias Maier