RFC: letting modules phone home to check for new releases
(if you're impatient and won't read this message ;) please comment here: http://drupal.org/node/94154 -- "module page could say if new releases are available"). one of the many benefits of the new release system is that contrib maintainers can now make an official release when they think their users should upgrade. this is particularly important for security fixes. my goal is for the admin/build/modules page to tell you if you're running out-of-date code. project.module just has to provide an RSS feed for new release nodes and/or an XML-RPC interface so that something on the modules page can either display the aggregated feed items in a block, or the UI could directly display the needs-update info. for example, you visit your modules page and the row for one of your installed contribs is now bright red with a giant warning: "SECURITY UPDATE RELEASED FOR THIS VERSION, YOU ARE URGED TO UPGRADE IMMEDIATELY" wouldn't that be slick? ;) my goal is to get this into core someday. it's too late in 5.x for that. however, since the modules page is all happy FAPI, we could do this via a form_alter()'ing contrib for 5.x and see that eventually migrate into core in 6.x or beyond. why am i asking for comments? there needs to be some more data in the .info files for 5.x modules if a contrib is going to be able to do this properly. as of today, the packaging script on drupal.org that creates the downloadable tarballs will automatically append the version of your module into the .info files[1]. this is b/c the modules page in core is about to be enhanced to display this data[2]. however, just knowing the version and the human-readable name of the module isn't enough for a 5.x contrib to be able to find out if your code is out of date. we also need something along the lines of: ; Project identification Project home = drupal.org Project name = views then, the soon-to-be-written check_updates.module can get this info and use it to construct the right URLs for the XML-RPC calls or to find the right RSS feed, etc. the packaging script can easily add this info automatically, but i need to finalize exactly what the fields are going to be called and what data they should really hold. i don't want to just hard-code a full URL since the project.module changes aren't done to provide the info yet, and we might change our mind on exactly the right path to use (and XML-RPC vs. RSS, etc). also, we can't assume *every* module installed on your site is from drupal.org, since that would break things for 3rd-party modules or custom-built ones. in that case, "Project home" should point to something else, or not exist at all. similarly, using a "Project name" instead of a "Project nid" is both more sane for end users to grok, and provides a level of indirection in case project nodes ever move around, get deleted and re-created, whatever. i *really* want to get this data into the .info files ASAP so that there aren't many 5.x contribs out in the wild that are missing it. however, i don't want to just unilaterally decide the fields and format of the values without any input from the rest of you. so, please comment ASAP here: http://drupal.org/node/94154 thanks! -derek [1] http://drupal.org/node/97286 "automatically insert human-readable version into .info files" [2] http://drupal.org/node/87298 "Add version info to modules page"
Hi, The way I handle this on Sympal is far easier, maybe a good idea: I add a predefined feed in aggregator, and have a PHP block preconfigured that shows the last item in that feed to certain roles. If you like KISS something down this line may be more pragmatic and easier to achieve. It works rather well, be it, that I cannot (nor want to) focus on certain modules, something that you seem to want to introduce also. Bèr
Derek's plan is much more robust and not a lot more complicated than RSS. People ignore RSS. But if they look at their modules page and right next to the Views module it says "New version available", the'll notice that. I really like Derek's plan. Derek, the fields you proposed look sane an unobtrusive. How, though will it support the case you mentioned where the module comes from a 3rd party vendor? Don't you need an XML-RPC endpoint or or some other absolute URL? I know you mentioned you want to avoid hardcoding these, but is it reasonable to expect that you can avoid doing so? -Robert Bèr Kessels wrote:
Hi,
The way I handle this on Sympal is far easier, maybe a good idea: I add a predefined feed in aggregator, and have a PHP block preconfigured that shows the last item in that feed to certain roles.
If you like KISS something down this line may be more pragmatic and easier to achieve. It works rather well, be it, that I cannot (nor want to) focus on certain modules, something that you seem to want to introduce also.
Bèr
On 18 Nov 2006, at 11:39 AM, Robert Douglass wrote:
Derek, the fields you proposed look sane an unobtrusive. How, though will it support the case you mentioned where the module comes from a 3rd party vendor? Don't you need an XML-RPC endpoint or or some other absolute URL? I know you mentioned you want to avoid hardcoding these, but is it reasonable to expect that you can avoid doing so?
I think a block that shows up for uid no 1, which lists security announcements, would be cool too Kind of a security release announcement block. Even if it's just a little line of text in the body.
I think a block that shows up for uid no 1, which lists security announcements, would be cool too
Kind of a security release announcement block. Even if it's just a little line of text in the body.
Huh? Last I checked drupal.org/security had an RSS feed and Drupal core had an aggregator which can do blocks and we have excellent control over block visibility.
Op zaterdag 18 november 2006 11:44, schreef Karoly Negyesi:
Huh? Last I checked drupal.org/security had an RSS feed and Drupal core had an aggregator which can do blocks and we have excellent control over block visibility.
This is almost exactly what I mean. Its works now, needs no patches, no modules and no develoment/. It can be done now. Bèr -- Drupal, Ruby on Rails and Joomla! development: webschuur.com | Drupal hosting: sympal.nl
Bèr Kessels wrote:
Op zaterdag 18 november 2006 11:44, schreef Karoly Negyesi:
Huh? Last I checked drupal.org/security had an RSS feed and Drupal core had an aggregator which can do blocks and we have excellent control over block visibility.
This is almost exactly what I mean. Its works now, needs no patches, no modules and no develoment/. It can be done now.
Yes, but does it work for just the versions of modules you currently have on your site? No. The RSS feed works for ALL modules. That's great. There's 500 modules and in general I probably care about maybe 15 or 20. That's a lot of data to sift through, needlessly. RSS won't let me search for data on a specific module (Because RSS feeds only go back so far). What we're proposing here *is* simple, and it doesn't try to use RSS for something it's not actually particularly good for. If it were good for that, we would already have figured out a way to use it.
Huh? Last I checked drupal.org/security had an RSS feed and Drupal core had an aggregator which can do blocks and we have excellent control over block visibility.
This is almost exactly what I mean. Its works now, needs no patches, no modules and no develoment/. It can be done now.
Ah sorry. My mail has been misunderstood. I reacted to Adrian's mail re. security. In overall this solution is not enough because you many modules but by far not all from contribs, so neither a per module feed or an eat-it-all feed won't suffice. So definitely we need more code in project.
Op zaterdag 18 november 2006 18:41, schreef Karoly Negyesi:
In overall this solution is not enough because you many modules but by far not all from contribs, so neither a per module feed or an eat-it-all feed won't suffice. So definitely we need more code in project.
I never said it would be enough. All I said is that, instead of talking for ages here about that utopian solution, then get into developing some whole new release-pushing-xmlrpc-system, we can offer a default RSS feed with block NOW. As in: now. Not something in some future after some unknown development cycle. Bèr -- Drupal, Ruby on Rails and Joomla! development: webschuur.com | Drupal hosting: sympal.nl
What about simply having an iframe to https://drupal.org/modulecheck/modulename, which returns a "You're running the latest version" normally, or a button saying "click here to upgrade" otherwise. The button essentially pionts to the changelog / install notes. (later, the button can point to the installer which autoupgrades stuff). -Arnab On 11/19/06, Bèr Kessels <ber@webschuur.com> wrote:
Op zaterdag 18 november 2006 18:41, schreef Karoly Negyesi:
In overall this solution is not enough because you many modules but by far not all from contribs, so neither a per module feed or an eat-it-all feed won't suffice. So definitely we need more code in project.
I never said it would be enough. All I said is that, instead of talking for ages here about that utopian solution, then get into developing some whole new release-pushing-xmlrpc-system, we can offer a default RSS feed with block NOW.
As in: now. Not something in some future after some unknown development cycle.
Bèr --
Drupal, Ruby on Rails and Joomla! development: webschuur.com | Drupal hosting: sympal.nl
It looks like with the suggestions for XML-RPC calls would allow the modulecheck to be run with the cron hook. That way, everytime an admin looks at their modules page, it wouldn't be a call to drupal.org, like an iframe would. Also, I'm not crazy about iframes, but I could in be a minority on that point. On 11/19/06, Arnab Nandi <arnab@arnab.org> wrote:
What about simply having an iframe to https://drupal.org/modulecheck/modulename, which returns a "You're running the latest version" normally, or a button saying "click here to upgrade" otherwise. The button essentially pionts to the changelog / install notes. (later, the button can point to the installer which autoupgrades stuff).
-Arnab
On 11/19/06, Bèr Kessels <ber@webschuur.com> wrote:
Op zaterdag 18 november 2006 18:41, schreef Karoly Negyesi:
In overall this solution is not enough because you many modules but by far not all from contribs, so neither a per module feed or an eat-it-all feed won't suffice. So definitely we need more code in project.
I never said it would be enough. All I said is that, instead of talking for ages here about that utopian solution, then get into developing some whole new release-pushing-xmlrpc-system, we can offer a default RSS feed with block NOW.
As in: now. Not something in some future after some unknown development cycle.
Bèr --
Drupal, Ruby on Rails and Joomla! development: webschuur.com | Drupal hosting: sympal.nl
I don't like iframes either, but it is a lot simpler to set up than xml-rpc. Allthough, I would prefer a xmlrpc solution, or both. On 11/20/06, Corey Bordelon <corey.bordelon@gmail.com> wrote:
It looks like with the suggestions for XML-RPC calls would allow the modulecheck to be run with the cron hook. That way, everytime an admin looks at their modules page, it wouldn't be a call to drupal.org, like an iframe would. Also, I'm not crazy about iframes, but I could in be a minority on that point.
On 11/19/06, Arnab Nandi < arnab@arnab.org> wrote:
What about simply having an iframe to https://drupal.org/modulecheck/modulename, which returns a "You're running the latest version" normally, or a button saying "click here to upgrade" otherwise. The button essentially pionts to the changelog / install notes. (later, the button can point to the installer which autoupgrades stuff).
-Arnab
On 11/19/06, Bèr Kessels <ber@webschuur.com> wrote:
Op zaterdag 18 november 2006 18:41, schreef Karoly Negyesi:
In overall this solution is not enough because you many modules but by far not all from contribs, so neither a per module feed or an eat-it-all feed won't suffice. So definitely we need more code in project.
I never said it would be enough. All I said is that, instead of talking for ages here about that utopian solution, then get into developing some whole new release-pushing-xmlrpc-system, we can offer a default RSS feed with block NOW.
As in: now. Not something in some future after some unknown development cycle.
Bèr --
Drupal, Ruby on Rails and Joomla! development: webschuur.com | Drupal hosting: sympal.nl
-- // Johan Forngren /**************************************** Email: johan@forngren.com Site: http://johan.forngren.com/ Skype: forngren MSN: johan@forngren.com IRC: forngren @ freenode Jabber/GTalk: forngren@gmail.com ****************************************/
Johan Forngren wrote:
I don't like iframes either, but it is a lot simpler to set up than xml-rpc. Allthough, I would prefer a xmlrpc solution, or both.
Since Drupal already has a good xmlrpc library, xmlrpc is actually very simple. It's actually easier than trying to parse an RSS feed, in fact.
Indeed, but not all servers has xml-rpc or require some kind configuration to enable it. I think it's the masses who need this function the most, but also the ones who is less likely to set up xml-rpc. On 11/20/06, Earl Miles <merlin@logrus.com> wrote:
Johan Forngren wrote:
I don't like iframes either, but it is a lot simpler to set up than xml-rpc. Allthough, I would prefer a xmlrpc solution, or both.
Since Drupal already has a good xmlrpc library, xmlrpc is actually very simple. It's actually easier than trying to parse an RSS feed, in fact.
-- // Johan Forngren /**************************************** Email: johan@forngren.com Site: http://johan.forngren.com/ Skype: forngren MSN: johan@forngren.com IRC: forngren @ freenode Jabber/GTalk: forngren@gmail.com ****************************************/
On 11/20/06, Johan Forngren <johan@forngren.com> wrote:
Indeed, but not all servers has xml-rpc or require some kind configuration to enable it. I think it's the masses who need this function the most, but also the ones who is less likely to set up xml-rpc.
Provided that PHP has the proper extensions configured, Drupal "out of the box" with XML-RPC. I'm not sure what the exact percentage of hosts that have the xml extension but according to the PHP docs ( http://us3.php.net/xml ) it's enabled by default.
Bèr Kessels wrote:
Op zaterdag 18 november 2006 11:44, schreef Karoly Negyesi:
Huh? Last I checked drupal.org/security had an RSS feed and Drupal core had an aggregator which can do blocks and we have excellent control over block visibility.
This is almost exactly what I mean. Its works now, needs no patches, no modules and no develoment/. It can be done now.
Two other points. 1) It's hard to argue that highly targeted security announcements are a bad idea. If 1) drupal.org tracks what modules have security problems and the solution, and 2) a site can ask drupal.org for this information an all its modules, sites owners can be properly informed. We know that many people continue to run with old versions of the software. We can reduce that number a lot. 2) This information can be given via drupal_set_message on /admin, not via blocks which can be misconfigured due to user error. We can be certain that the right messages are delivered to the right people.
On Nov 18, 2006, at 4:59 AM, adrian rossouw wrote:
On 18 Nov 2006, at 11:39 AM, Robert Douglass wrote:
Derek, the fields you proposed look sane an unobtrusive. How, though will it support the case you mentioned where the module comes from a 3rd party vendor? Don't you need an XML-RPC endpoint or or some other absolute URL? I know you mentioned you want to avoid hardcoding these, but is it reasonable to expect that you can avoid doing so? I think a block that shows up for uid no 1, which lists security announcements, would be cool too
Sorry for being late on this, but I just wanted to comment that any such setup would be useful to be able to be configured for users other than 1. I rarely use user 1 -- only when I need to run update.php -- and as such, I would never see such a block. Thanks, Ricky
Just to clarify: the reason I think XML-RPC is superior to RSS in this case (and worth the extra effort), is that it can request information for the exact branches of the modules installed. We'd have to work extra hard to build a tailored RSS feed for every Drupal installation on the planet, add extra fields to the feed, parse them out etc., whereas with XML-RPC it is all much more comfortable. I like that Derek's proposal will allow us to display things like: "You are using views-4-7--1.1. views-4-7--1.2 is available. This is a security update" as well as: "You are using views-4-7--1.1. views-4-7--1.2 is the latest release in this branch. views-4-7--2.0 is also compatible with your Drupal installation and is available." Robert Douglass wrote:
Derek's plan is much more robust and not a lot more complicated than RSS. People ignore RSS. But if they look at their modules page and right next to the Views module it says "New version available", the'll notice that. I really like Derek's plan.
Derek, the fields you proposed look sane an unobtrusive. How, though will it support the case you mentioned where the module comes from a 3rd party vendor? Don't you need an XML-RPC endpoint or or some other absolute URL? I know you mentioned you want to avoid hardcoding these, but is it reasonable to expect that you can avoid doing so?
-Robert
Bèr Kessels wrote:
Hi,
The way I handle this on Sympal is far easier, maybe a good idea: I add a predefined feed in aggregator, and have a PHP block preconfigured that shows the last item in that feed to certain roles.
If you like KISS something down this line may be more pragmatic and easier to achieve. It works rather well, be it, that I cannot (nor want to) focus on certain modules, something that you seem to want to introduce also.
Bèr
Robert Douglass wrote:
Just to clarify: the reason I think XML-RPC is superior to RSS in this case (and worth the extra effort), is that it can request information for the exact branches of the modules installed. We'd have to work extra hard to build a tailored RSS feed for every Drupal installation on the planet, add extra fields to the feed, parse them out etc., whereas with XML-RPC it is all much more comfortable. I like that Derek's proposal will allow us to display things like:
"You are using views-4-7--1.1. views-4-7--1.2 is available. This is a security update" as well as: "You are using views-4-7--1.1. views-4-7--1.2 is the latest release in this branch. views-4-7--2.0 is also compatible with your Drupal installation and is available."
Also, the 'extra effort' is tiny. Seriously, support for xmlrpc is already there. Support for having Drupal phone home to drupal.org is already there. All that this proposal does is add a couple of extra xmlrpc methods that can be called, which allows for a contrib module to utilize these methods and let operators know about important data *that is tailored to them*.
my goal is to get this into core someday. it's too late in 5.x for that. however, since the modules page is all happy FAPI, we could do this via a form_alter()'ing contrib for 5.x and see that eventually migrate into core in 6.x or beyond.
Typically, modules will only go out of date long after you install them. So, you would only notice outdated modules when you come back to admin/modules for something else. Perhaps this could be added to the status report instead. We already have an 'outreach' from the status report to /admin for critical errors and we could do something similar for out of date versions. Given that version checking can be applied not only to modules, but also themes and even core, it seems a bit out of place on the modules page as well. Plus, it allows us to present it in a simple, compact format. Some examples: ------------ Drupal 5.2 (up to date) (Checkmark icon) ------------ Drupal 5.1 (minor update available) (Error icon) A new maintenance release is available for Drupal 5.x and should be installed immediately to prevent critical security bugs. ------------ Drupal 4.7.4 (major update available) (Warning icon) A new major release of Drupal is available which includes many new features. ------------ ------------ Modules Up to date ------------ Modules Minor updates available Updates are available for the following modules: * Pirate * Views (Warning icon) ------------ Modules Major updates available A critical security fix is available for the following modules and should be installed immediately: * Hack your site.module ------------ The idea of the status report has always been that it should only present important information. Contrast this with, for example, presenting a long table of module names with a column that says "up to date" or "updates available": it forces the admin to sift through the (possibly long) list him/herself. It is much more efficient to summarize the information. Of course, an additional outreach could be added to admin/modules too, but the update information should IMO not live there. Steven Wittens
On Nov 18, 2006, at 9:47 AM, Steven Wittens wrote:
Perhaps this could be added to the status report instead.
i'm down with this, only not "instead". personally, once check_updates.module (or whatever) does it's daily ping home and has a locally cached table of out-of-date modules (and potentially themes + core), i'd be thrilled to see the admin literally bombarded with the fact they've got out-of-date (potentially insecure) modules installed: 1) on the modules page (since it's about modules, and that's where they'll be seeing the versions of their installed modules in the first place). 2) in the status report (grand idea) 3) as a little block that automatically appears any time you visit admin/* then, it'd be pretty hard to miss the fact you're out of date, and that you should do something about it. none of this is meant to replace the drupal.org/security RSS feed, and people are encouraged to use that however they wish. this whole proposal just supplements that mechanism with a more direct approach that tells people exactly what they have to know. once we start putting more work into security audits of contrib, our poor security RSS feed is going to start turning into the boy that cried wolf. :( of course, we must continue, but more and more, SAs are going to go out and people are going to say "oh, i don't care about that". now that we have the means, i want to see us use them to put only the relevant data, and all of it, exactly where the admins need it. thanks, -derek
On 11/17/06, Derek Wright <drupal@dwwright.net> wrote:
i *really* want to get this data into the .info files ASAP so that there aren't many 5.x contribs out in the wild that are missing it. however, i don't want to just unilaterally decide the fields and format of the values without any input from the rest of you. so, please comment ASAP here:
Adding the extra information is a great idea...we have our own little repository / update system, and with a different "home", different sites could, for instance, keep different distributions up to date. HOWEVER, the phone home and XML-RPC stuff makes me *very* nervous from a security perspective. I would want to have some real hard core folks examine and document information flow end to end and looking for vulnerabilities -- ideally some external folks as well. We will need to review all Drupal.orgprocesses as well as the receiving code. There has been other talk about auto-downloading various information. Same comment there -- huge security risk, needs 100x as much review, and even then I'm nervous about it.... -- Boris Mann Vancouver 778-896-2747 San Francisco 415-367-3595 Skype borismann http://www.bryght.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I am currently working on a system to automatically install modules (i.e download and unpack into the appropriate directory) and upgrade modules (i.e. overwrite present files). The architecture is finished: - - A module for the GUI and the actual work (not fully implemented yet) - - A client that provides necessary information to the "GUI and functionality" module (one implementation is nearly finished, it lacks the ability to provide the actual module files) - - A server that serves information (module files, version information etc.) to client (not implemented, because the client does not need a server) My implementation of the client uses a working copy of the HEAD branch of CVS (that's why I don't need a server at the moment), but there are numerous other possibilities: - - CVS (without a working copy) - - XML-RPC - - FTP - - webcvs ... {whatever} I wouldn't have mentioned this effort in this early stage (it's far from alpha) but there are three points I have to make: 1. I wouldn't want us to duplicate code or go into different (a.k.a incompatible) directions. 2. My module would also benefit from more precise information in the .info-files 3. I, too, like to have access to the projects via XML-RPC. So, what do you think? I tried to understand the other posts of this discussion, but they seem to revolve around technical aspects that are not really an issue (at least not at this point of the development (sure, making it into core would be great)). Greetings, - -- Oswald Jaskolla Ingenieurbüro Richard Schieferdecker Kreuzherrenstraße 2 D-52062 Aachen Tel.: 02 41 / 409 54 43 Fax: 02 41 / 477 05 199 mobil: 01 64 / 941 06 75 eMail: oswald.jaskolla@schieferdecker.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFY0HEuinSHQ/4/T4RAtikAKCN8LUE8t0rEx7oYBpQpBo27O/0kgCglG65 95I4hnZxqP7LvhZLEZL/ePs= =p8Vl -----END PGP SIGNATURE-----
On Nov 21, 2006, at 10:13 AM, Oswald Jaskolla wrote:
So, what do you think?
i hope you don't take this personally, by i'm *very* opposed to the kind of system you're building. the security implications of giving your website permission to overwrite itself automatically are *HORRIFYING*. i'd never install such a thing, and i'd never advocate anyone else should install such a thing. the kind of system i'm building is just an automated way to tell the human site admins: "your code is out of date" (and if true, "and insecure") and nag them mercilessly until they upgrade the stale module(s) to the latest, secure version(s). it's still the human's task to perform the upgrade itself. this manual upgrade could itself be further automated, but a high- privileged admin user must run this automated script themselves, just like they have to run update.php themselves. building and providing a tool that can "do it all" for you is asking for security hell, and therefore defeats the purpose of what i'm trying to accomplish (make it easier and therefore more likely for drupal sites to remain secure). anyway, i'm willing to coordinate, and further discuss design/ implementation issues, but i can't emphasize enough how bad i think a fully-automated system for upgrading a drupal site would be. maybe i'm misunderstanding your design/proposal, but after re-reading your message a few times, it's pretty clear you're marching down the path towards what i'd consider "the dark side". ;) sorry, -derek
Derek Wright wrote:
On Nov 21, 2006, at 10:13 AM, Oswald Jaskolla wrote:
So, what do you think?
i hope you don't take this personally, by i'm *very* opposed to the kind of system you're building.
the security implications of giving your website permission to overwrite itself automatically are *HORRIFYING*. i'd never install such a thing, and i'd never advocate anyone else should install such a thing.
The underlying idea is so silly I'd not want us to host the resulting code on drupal.org.
the kind of system i'm building is just an automated way to tell the human site admins: "your code is out of date" (and if true, "and insecure") and nag them mercilessly until they upgrade the stale module(s) to the latest, secure version(s). it's still the human's task to perform the upgrade itself.
That's a nice thing and very useful. Do you plan to check if a module has been altered after download, too? Cheers, Gerhard
On Nov 21, 2006, at 5:42 PM, Gerhard Killesreiter wrote:
Do you plan to check if a module has been altered after download, too?
no, i wasn't planning to do that. the md5 hashes we have are on the whole tarball, not per file. so an easy, obvious way to detect this doesn't immediately spring to mind, short of doing a bunch more work to calculate, store, and advertise the per-file md5 hashes at d.o. [1] i was just planning to go with the version listed in the .info file and comparing that to what "home" thinks the latest version of that module is and reporting (in various places) if they disagree. in the first (and probably only) implementation, "home" would be drupal.org. if someone else wants to make what i'm doing work to phone home to other sites where other modules are hosted, they should feel free to do so. again, i'm happy to discuss/coordinate/share, but i'm not going to worry about it until people who really want this functionality a) exist, b) step forward, and c) do some of their own work to make it happen. cheers, -derek [1] i guess the packaging script could compute the md5 hashes for all the files in a tarball, and write a big section of those filename => md5 hash mappings into the .info file itself. then, check_updates.module could compare the values advertised in the .info file with the real values as installed on the file system. i'm not sure that's worth doing, but it'd probably work. ;)
Why so many hostile reactions to Oswald's project? Read his message again. He's not proposing a site that updates itself. He's promoting automatic downloads and automatic uprades. An administrator could select the modules to be installed or upgraded and it would happen automatically. What he asked for was collaboration on a system to detect which modules have updates available. What he's working on is inevitable. Manual module installation and upgrading creates many opportunities for inexperienced users to make mistakes.
Darren Oh wrote:
What he's working on is inevitable. Manual module installation and upgrading creates many opportunities for inexperienced users to make mistakes.
Yes, and dww and I (and webchick and neclimdul and a handfull of others too) have been working toward *this very goal* very hard for the last 4 or 5 months. None of this is occurring in a vacuum. One of the major motivations of the new release system project was put in place a method for your modules to tell if they're out of date. The map's been here for awhile and we've been working toward it. Just because some guy has some system that he's been working on in isolation doesn't mean we should all suddenly drop what we've been busting our asses on for months over. If he wants to join the work we've been doing, that's great, but he's got to join the work that's already been done. It's not like we've been working privately and behind the scenes. There's a huge trail of issues and conversations about this set of projects that have led up to it. Also, this quote:
So, what do you think? I tried to understand the other posts of this discussion, but they seem to revolve around technical aspects that are not really an issue (at least not at this point of the development (sure, making it into core would be great)).
If the technical aspects we're discussing aren't really an issue, I'm not sure what the issues are. The technical aspects we're discussing are, in my opinion, the last impediment to being able to actually write this system. The rest of it is just sitting down and writing a module to do it according to the spec we lay out in these discussions.
Darren Oh wrote:
Why so many hostile reactions to Oswald's project? Read his message again. He's not proposing a site that updates itself. He's promoting automatic downloads and automatic uprades.
Where's the difference between "a sites that updates itself" and "automatic upgrades"? I can't spot any.
An administrator could select the modules to be installed or upgraded and it would happen automatically.
What he asked for was collaboration on a system to detect which modules have updates available.
What he's working on is inevitable. Manual module installation and upgrading creates many opportunities for inexperienced users to make mistakes.
Inexperienced users shouldn't be running a public service such as a website. They should read and follow the docs and thus be no longer inexperienced. Cheers, Gerhard
Where's the difference between "a sites that updates itself" and "automatic upgrades"? I can't spot any.
Think about it. This is the process recommended in the manual: 1. Select the module you want to install or upgrade. 2. Download the module to your computer. 3. Unpack the module. 4. Start your FTP client. 5. Connect to your site. 6. Switch to the correct directory (whatever that is). 7. Upload the module directory to your site. 8. Enable the module (if installing) or run update.php (if upgrading). Steps 2 through 8 would become one step. Automatic installation or automatic upgrade seemed clear enough to me. What would you call it?
Darren Oh wrote:
Where's the difference between "a sites that updates itself" and "automatic upgrades"? I can't spot any.
Think about it. This is the process recommended in the manual:
1. Select the module you want to install or upgrade. 2. Download the module to your computer. 3. Unpack the module. 4. Start your FTP client. 5. Connect to your site. 6. Switch to the correct directory (whatever that is). 7. Upload the module directory to your site. 8. Enable the module (if installing) or run update.php (if upgrading).
Steps 2 through 8 would become one step. Automatic installation or automatic upgrade seemed clear enough to me. What would you call it?
For security and stability reasons it's more complex than that.
On Nov 21, 2006, at 10:55 PM, Earl Miles wrote: > Darren Oh wrote: >>> Where's the difference between "a sites that updates itself" and >>> "automatic upgrades"? I can't spot any. >> Think about it. This is the process recommended in the manual: >> 1. Select the module you want to install or upgrade. >> 2. Download the module to your computer. >> 3. Unpack the module. >> 4. Start your FTP client. >> 5. Connect to your site. >> 6. Switch to the correct directory (whatever that is). >> 7. Upload the module directory to your site. >> 8. Enable the module (if installing) or run update.php (if >> upgrading). >> Steps 2 through 8 would become one step. Automatic installation >> or automatic upgrade seemed clear enough to me. What would you >> call it? > > For security and stability reasons it's more complex than that. All right, then. Steps 2 through 7 could become one step. Administrators get a prompt when update.php needs to be run. The point is, what should this be called if not an automatic installation or upgrade? Maybe replace automatic with automated? Or scripted? Derek's concerns about permissions could be addressed by using FTP rather than PHP to replace files. The point is to making things easier for users. The system he's designing would help administrators select the modules to upgrade (step 1). Anyway, as has been said before, some system like this is inevitable in the future. I just hoped to dampen the negative reaction.
On Nov 21, 2006, at 8:21 PM, Darren Oh wrote:
Anyway, as has been said before, some system like this is inevitable in the future.
thanks in large part to the system i wrote and put in place. ;) now, i'm taking responsibility for that, and making sure that no one uses the fruits of my labors to build any insecure tools in the name of "usability".
I just hoped to dampen the negative reaction.
then help design a secure system that doesn't have the failings that are generating the negative reactions. we're not opposed to making it easier to update your site when it's out of date (as merlin said, a small # of us have been busting our asses to make that happen for months now). what we're opposed to are the careless approaches to security being put forward (no personal offense to anyone in particular). i want this to *help* the security team, not make our lives worse. that's why i'm reacting negatively. hope that's clear... thanks, -derek
On Tuesday 21 November 2006 22:27, Derek Wright wrote:
i want this to *help* the security team, not make our lives worse.
that's why i'm reacting negatively.
Drupal 5 now has the ability to run from the command line, right? So make the actual replace/upgrade process a command line only script. Checking which new modules are available can and should be an automated process. Actually doing the upgrade should be something you have go to well out of your way to do. For the sysadmins in the crowd, there's nothing wrong with putting "apt-get update" in your crontab. "apt-get upgrade" should only be run by the sysadmin as root, by hand, and therefore when you know you're doing something dangerous. :-) So: 1) System auto-updates list of latest versions and lets you know if action is necessary. 2) Admin goes to command line and runs drupal-get.php download 3) Admin sets site to offline. 4) Admin runs drupal-get.php backup 5) Admin runs drupal-get.php upgrade 6) Admin runs update.php (or that can be done automatically by #5 if it's safe) 7) Admin brings site back online. The hard parts (downloading and unpacking) are all automated, but only when the admin is on the command line (and therefore running as the owner of the files anyway), and therefore (hopefully) knows exactly what he's doing. Yes, this excludes people who are on a host that doesn't give them a shell. Given the file permissions involved, I don't really know of a way around that. *sits back to have the security holes pointed out in the above.* :-) -- Larry Garfield AIM: LOLG42 larry@garfieldtech.com ICQ: 6817012 "If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it." -- Thomas Jefferson
On Tue, 21 Nov 2006, Larry Garfield wrote:
For the sysadmins in the crowd, there's nothing wrong with putting "apt-get update" in your crontab. "apt-get upgrade" should only be run by the sysadmin as root, by hand, and therefore when you know you're doing something dangerous. :-)
The hard parts (downloading and unpacking) are all automated, but only when the admin is on the command line (and therefore running as the owner of the files anyway), and therefore (hopefully) knows exactly what he's doing. Yes, this excludes people who are on a host that doesn't give them a shell. Given the file permissions involved, I don't really know of a way around that.
*sits back to have the security holes pointed out in the above.* :-)
How many shared hosts have a command line feature for their users? This discussion is around easing the update pain for inexperienced users. They are surely not running their site on their own servers. The problem with the above, even if you only consider "apt-get update" automatically done is that incompatibilities can happen. Previous releases of Drupal have seen database updates needed between minor versions, and even big chunks of code replaced / modified. Like the XML-RPC layer completely replaced and released as a minor update. Or the form key requirement added and released as a minor update. There is a strong possibility that at the time when that update happens, your contrib modules are not yet up to date (especially those you might have added yourself, or downloaded and set up from a third party vendor - not drupal.org). Now think that you have a modified theme, not some stock theme from drupal.org. That cannot be updated, although you might need to add the form key stuff into the code. No automatic update will do this for you. The problem is that customized software like Drupal is extremely hard to update automatically (and most of the time it will fail due to being uncertain that some of your components which are not possible to update might not be compatible with stuff which is possible to update). And this is only about updating minor releases... As far as I see, noone saves you from testing the updates / upgrades yourself, and ensuring that everything works. There is a place for humans here. Gabor
And again... Op woensdag 22 november 2006 03:56, schreef Darren Oh:
Think about it. This is the process recommended in the manual:
1. Select the module you want to install or upgrade. 2. Download the module to your computer. 3. Unpack the module. 4. Start your FTP client. 5. Connect to your site. 6. Switch to the correct directory (whatever that is). 7. Upload the module directory to your site. 8. Enable the module (if installing) or run update.php (if upgrading).
Steps 2 through 8 would become one step. Automatic installation or automatic upgrade seemed clear enough to me. What would you call it
... we forget that a great lot of Drupas sites are, could be, or should be installed with TOOLS. As in: bryghts own tools, capistrano, ssh scripts, ispconfig installers, plesk API implementations etceteras. In that case the *properly designed* install and upgrade *framework* would allow webGUI tools like: 1. Select the module you want to install or upgrade. 2. Install the module in your site. 3. Enable the module (if installing). Or like 1. Type a domainname (example.com) 2. Select what type of site you want (from a list of install profiles and/or other CMSes) 3. Log in, into your newly installed and preconfigured site Such a webgui CAN (but should not, IMO) live inside your very own Drupal site. It can also be ran from within plesk, or ispconfig. Or on the CLI by one sysadmin with too much time. So, I again, plead for a framework. An API. A well thought out foundation. An architecture. Not a GUI installer that needs FTP and SSH access and that requires you to open up (rather) insecure permissions on certain files. Not an in-site-upgrade system that forces you to run with the most insecure permissions on teh intarweb. No. A framework. Then the Good Folks Who Develop Cool stuff can build install GUIs on top of that. Or a capistrano recipe. Or we can give the redistributors a good alternative to develop on top of, instead of bashing the hell out of them on digg and drupal.org for distributing "old" Drupal versions. Bèr -- Drupal, Ruby on Rails and Joomla! development: webschuur.com | Drupal hosting: sympal.nl
On 22 Nov 2006, at 11:41, Bèr Kessels wrote:
So, I again, plead for a framework. An API. A well thought out foundation. An architecture. Not a GUI installer that needs FTP and SSH access and that requires you to open up (rather) insecure permissions on certain files. Not an in-site-upgrade system that forces you to run with the most insecure permissions on teh intarweb. No. A framework.
I agree that we should play nice with third-party installers. That said, I think you might want to do a cost-benefit analysis: either work on your own Sympal scripts, or team up with Derek, Earl, webchick et al to govern and improve certain design aspects of the core installer. We both know what strategy is going to win in the long term. -- Dries Buytaert :: http://www.buytaert.net/
Op woensdag 22 november 2006 14:48, schreef Dries Buytaert:
We both know what strategy is going to win in the long term.
Yup. Be it that capistrano works NOW. And so do these scripts. they allow me to install stuff already. So on the "short term" I also know what strategy brings me most benefit. But that said: the main reason that I continued on the framework and API (commandline) road is not because I felt like "doing my own thing". The reason is twofold: When The installer was discussed, I explained my plans in rahter great detail. At that moment I could hardly show code. I could only show where I wanted to head and explain (which I did) why I thought that framework route was the one to take. People, including Dries, clearly indicated that this was not the desired route at that time: Drupal needed a web-based installer ASAP, not a framework in some future. Fine, but I decided to continue on what I felt the correct route, because I have no need (scratch itch) for a webbased installer. When the first installer was released we discussed a profile-management system (distros). By that time, I already had a working draft of my own installer with profiles. I presented that on this list, but it was ignored, with one exception: A technical sidenote by Moshe. And frankly I don't feel like spending loads of efforts in defending an idea, where people have little interest in. I prefer to spend that time on making stuff work better. Not on telling people that it might work better then something else. So I decided to let it go and see what would come out, and whether or not I would like the core distro-system better (I don't) and then convert my own into that. So here are my plans: * Move all my scripts to capistrano. It makes no sense to reinvent the wheel (sympal PHP scripts are a Drupal-specific PHP version of a much more mature Ruby system called capistrano) * During that time, keep on using the scripts, so possibly I will integrate the new profile/distro stuff in PHP, before moving over to capistrano. Questions: Would capistrano serve as a default Drupal install/update framework? Are we afraid of Ruby (as in: not rails, but the language)? Do we want to depend on a third party tool for our deployment framework? Personal my answers are: yes, no, yes. But I'd like to hear if this can even be considered for core :) Bèr
Bèr Kessels wrote: Note: I am not very much interested in installers/updaters since I mostly only run one site at a time.
Questions: Would capistrano serve as a default Drupal install/update framework? Are we afraid of Ruby (as in: not rails, but the language)? Do we want to depend on a third party tool for our deployment framework?
I don't think I'd use a tool written in "yet another language".
Personal my answers are: yes, no, yes. But I'd like to hear if this can even be considered for core :)
That's IMO a rather clear case of "Forget it". Cheers, Gerhard
I've got nothing against Ruby or any other language, but there's no way in hell I'd support making another language a requirement for Drupal. Gerhard Killesreiter wrote:
Bèr Kessels wrote:
Personal my answers are: yes, no, yes. But I'd like to hear if this can even be considered for core :)
That's IMO a rather clear case of "Forget it".
Cheers, Gerhard
On Wed, 22 Nov 2006, [iso-8859-1] B�r Kessels wrote:
Questions: Would capistrano serve as a default Drupal install/update framework? Are we afraid of Ruby (as in: not rails, but the language)? Do we want to depend on a third party tool for our deployment framework?
Personal my answers are: yes, no, yes. But I'd like to hear if this can even be considered for core :)
We are solving an end user problem with in installer/updater. How many of our users (and user prospects) will host on servers having capistrano installed? If we are depending on capistrano, how many servers will we be compatible with? Gabor
I'm a little late to nitpick details, but capistrano doesn't need to be on the host/server. It needs to be installed on the user's computer as a sort of client. The other software requirement (besides ruby) is a version control system that has the files you want to install elsewhere (that's probably optional, too). I believe capistrano uses ssh to login and run the commands in it's scripts. Think of it like a super ssh/scripting client that runs multiple commands on multiple machines in parallel. If that thought scares you, all I can say is "be afraid, be very afraid" ;) <opinion-based-almost-rant> I don't think Bèr wants capistrano to be the "one installer to rule them all" for everyone. I think our needs are similar; I need something that can allow me to deploy/update multiple drupal sites on multiple hosts. That's not the use case for the newbie setting up their first site. PEAR has an interesting installer, especially with the channels concept that was introduced in their last major update (over 1.5 years ago). Why can't we adapt a system to play nicely with that, so that a drupal install is as easy as "pear install drupal/drupal.core"? (I know their are other technical things needed before that can happen, but it's an option that we're getting very close to being able to do). If said framework existed, we could use the same infrastructure to power whatever "human-interaction required" update system that many seem to favor with the system for those of us who have "multiple sites to update and better things to do than be playing with some @!#$%^ tarballs of modules". </opinion-based-almost-rant> -- Corey Bordelon <http://drupal.org/user/23318> On 11/22/06, Gabor Hojtsy <gabor@hojtsy.hu> wrote:
On Wed, 22 Nov 2006, [iso-8859-1] Bèr Kessels wrote:
Questions: Would capistrano serve as a default Drupal install/update framework? Are we afraid of Ruby (as in: not rails, but the language)? Do we want to depend on a third party tool for our deployment framework?
Personal my answers are: yes, no, yes. But I'd like to hear if this can even be considered for core :)
We are solving an end user problem with in installer/updater. How many of our users (and user prospects) will host on servers having capistrano installed? If we are depending on capistrano, how many servers will we be compatible with?
Gabor
On 22 Nov 2006, at 16:27, Bèr Kessels wrote:
When The installer was discussed, I explained my plans in rahter great detail. At that moment I could hardly show code. I could only show where I wanted to head and explain (which I did) why I thought that framework route was the one to take. People, including Dries, clearly indicated that this was not the desired route at that time: Drupal needed a web-based installer ASAP, not a framework in some future.
I don't remember saying any of that, but my memory has failed on more than one occasion. However, I do remember advocating the need to play nice with installers like Debian's and Gentoo's. My opinions on this issue have been very clear for at least 6-12 months -- that is plenty of time to align your work with that of core. Of course, you're more than welcome to work on Sympal. There is nothing wrong with that. :)
So here are my plans: * Move all my scripts to capistrano. It makes no sense to reinvent the wheel (sympal PHP scripts are a Drupal-specific PHP version of a much more mature Ruby system called capistrano) * During that time, keep on using the scripts, so possibly I will integrate the new profile/distro stuff in PHP, before moving over to capistrano.
Have fun! I haven't looked at Capistrano, but we can't require people to install Ruby. Then again, if this is only meant to be used by a small subset of power-users, it might fly. For various reasons, I'd prefer a PHP-based solution though (eg. maintainability). -- Dries Buytaert :: http://www.buytaert.net/
Op donderdag 23 november 2006 08:56, schreef Dries Buytaert:
Then again, if this is only meant to be used by a small subset of power-users, it might fly. For various reasons, I'd prefer a PHP-based solution though (eg. maintainability).
Yup, me too. However, it seems to become more common over time. You will notice the Ruby (on rails) enabled webhosts popping up all over the place already. As said elsewhere in the thread, capistrano is not something to be installed on the server, but rather on a client. So the server neds no Ruby, the client needs that. Add to that, the fact that wordpress, Joomla! and most RoR apps are capistranised, and you can see a great opportunity. But all that in consideration, dependency on another language has two issues: we need to have people in the dev teams who speak such a language, and we will be requiring a lot more for a simple installation. Still, I think capistrano is very well suited for the task, it seems to become a sort of de-facto standard in deployment-land and it has all the stuff running and working, while such a PHP application is not even existing. More links and investigations for Drupal capistrano on http://groups.drupal.org/node/1461 Bèr -- Drupal, Ruby on Rails and Joomla! development: webschuur.com | Drupal hosting: sympal.nl
On Nov 21, 2006, at 9:01 PM, Derek Wright wrote:
On Nov 21, 2006, at 5:42 PM, Gerhard Killesreiter wrote:
Do you plan to check if a module has been altered after download, too?
no, i wasn't planning to do that. the md5 hashes we have are on the whole tarball, not per file. so an easy, obvious way to detect this doesn't immediately spring to mind, short of doing a bunch more work to calculate, store, and advertise the per-file md5 hashes at d.o. [1]
One thing about this is that it would be good to allow admins to say "yes, this file has been modified, but I know about it so don't tell me again". Occasionally I make small changes to a module (example: I modified biblio.module to hide ability to export XML, because I don't want it visible), and I don't want to be plagued by massive warnings (or worse, automatic disabling) because I customized a module. Thanks, Ricky
While I agree with Derek about the security issues there can be ways around it. For example, functions that perform the update do so only when a file (in the site root directory) contains some random characters chosen by the system for that upgrade session. The file must be manually created prior to the upgrade though a way to generate the file is provided. The administrator can only download to their computer and then upload to server. Derek is absolutely wright (sic) that security is a BIG issue. I'd also not use it without measures similar to the above. Cheers, -- Sammy Spets Synerger Pty Ltd http://synerger.com On 21-Nov-06 17:25, Derek Wright wrote:
On Nov 21, 2006, at 10:13 AM, Oswald Jaskolla wrote:
So, what do you think?
i hope you don't take this personally, by i'm *very* opposed to the kind of system you're building.
the security implications of giving your website permission to overwrite itself automatically are *HORRIFYING*. i'd never install such a thing, and i'd never advocate anyone else should install such a thing.
the kind of system i'm building is just an automated way to tell the human site admins: "your code is out of date" (and if true, "and insecure") and nag them mercilessly until they upgrade the stale module(s) to the latest, secure version(s). it's still the human's task to perform the upgrade itself.
this manual upgrade could itself be further automated, but a high- privileged admin user must run this automated script themselves, just like they have to run update.php themselves. building and providing a tool that can "do it all" for you is asking for security hell, and therefore defeats the purpose of what i'm trying to accomplish (make it easier and therefore more likely for drupal sites to remain secure).
anyway, i'm willing to coordinate, and further discuss design/ implementation issues, but i can't emphasize enough how bad i think a fully-automated system for upgrading a drupal site would be.
maybe i'm misunderstanding your design/proposal, but after re-reading your message a few times, it's pretty clear you're marching down the path towards what i'd consider "the dark side". ;)
sorry, -derek
On Nov 21, 2006, at 5:54 PM, Sammy Spets wrote:
The file must be manually created prior to the upgrade though a way to generate the file is provided.
you lost me. ;) if the admin has to manually type in the random key to allow the "automatic" update to do its thing, what's the point of calling it "automatic" and giving the website filesystem write access to clobber itself? why not just say "human intervention is REQUIRED"? 1) put site into system maintenance mode 2) backup DB and filesystem 3) run some "fetch_latest.php" script that knows all your installed modules, downloads the updated tarballs, unpacks them into place, etc 4) run update.php 5) test to make sure life is good 6) leave system maintenance mode and return to live operation. i'm happy to see steps #2 and #3 as automated as possible, but #3 should definitely run as the high-privileged admin's uid, not the uid of the webserver process itself. anything less would be uncivilized... ;) -derek
You're saying I called it automatic, when I didn't! We're talking about the same thing. My example is a way to ensure the update procedure is legitimately run by the admin user and thus can use the webserver process uid to replace files (which is necessary for the files to be usable by Drupal). It's an example and not a suggestion or a "do it this way". Aside from all that there is a need to reduce downtime of a site. Putting the site into maintenance mode and then downloading the replacement modules increases the downtime. Would be nice if the solution ends up with a way to download updated modules before going into maintenance mode. So an addendum to Derek's workflow: 1) download updated modules into a temporary filesystem location 2) put site into system maintenance mode 3) backup DB and filesystem 4) run some "upgrade.php" script that unpacks the modules into place 5) run update.php 6) test to make sure life is good 7) leave system maintenance mode and return to live operation. Cheers, -- Sammy Spets Synerger Pty Ltd http://synerger.com On 21-Nov-06 18:11, Derek Wright wrote:
On Nov 21, 2006, at 5:54 PM, Sammy Spets wrote:
The file must be manually created prior to the upgrade though a way to generate the file is provided.
you lost me. ;) if the admin has to manually type in the random key to allow the "automatic" update to do its thing, what's the point of calling it "automatic" and giving the website filesystem write access to clobber itself?
why not just say "human intervention is REQUIRED"?
1) put site into system maintenance mode 2) backup DB and filesystem 3) run some "fetch_latest.php" script that knows all your installed modules, downloads the updated tarballs, unpacks them into place, etc 4) run update.php 5) test to make sure life is good 6) leave system maintenance mode and return to live operation.
i'm happy to see steps #2 and #3 as automated as possible, but #3 should definitely run as the high-privileged admin's uid, not the uid of the webserver process itself.
anything less would be uncivilized... ;)
-derek
On Nov 21, 2006, at 6:46 PM, Sammy Spets wrote:
My example is a way to ensure the update procedure is legitimately run by the admin user and thus can use the webserver process uid to replace files (which is necessary for the files to be usable by Drupal).
no offense, but this is exactly the kind of sloppy approach to file system permissions that leads to sites getting hacked and drupal getting a bad name. :( your Drupal source files *DO NOT* have to be writable by the webserver process uid for "the files to be usable by Drupal" (!!!). the "files" directory is a special case, which is why a) it's been a source of numerous security issues and b) it should be handled with intensely defensive programming by anything that's touching it. that aside, your Drupal source files must be _readable_ by the webserver process uid, period. your Drupal source files *SHOULD* be _writable_ only by a *DIFFERENT* process uid, namely, your "high-privilege admin" user (could be root, but doesn't have to be, so long as it's *NOT* the same as your httpd). until people completely understand this, they'll a) continue to fill up this thread with confusing posts that don't advance the end goal, and b) continue to setup insecure websites. "automatic" upgrades, in whatever form, imply that some automatic service is going to be overwriting things in modules/* under your doc root. that requires that the automatic service has write access, and that means trouble. unless the human (with the different uid, and higher powers to do damage) runs a script to trigger the upgrade (after backups, etc), life sucks. so, if we're all clear on that, and we want to discuss exactly how much can be automated into this script, what the UI looks like, etc, etc, that's all fine and good, but a) it's premature (i'm just asking about agreement on what goes in the .info files to make all this possible) and b) it's a different discussion than "automatic site upgrades"... -derek p.s. sure, friendly amendment about downloading to a tmp directory and only putting the site in maintenance mode for as little time as possible is totally reasonable. i wasn't being as careful about the exact steps in that reply, since my main point was *THE HUMANS MUST DO IT* or life sucks. p.p.s. did i mention that human intervention is required for this to be a good idea? ;)
Again i'll emphasize that it was an example and NOT a suggestion. You're jumping all over it like it's on fire. A little hot under the collar? I'm going to bow out of this discussion while I still have a head on my shoulders. I'm obviously out of my league. -- Sammy Spets Synerger Pty Ltd http://synerger.com On 21-Nov-06 19:23, Derek Wright wrote:
On Nov 21, 2006, at 6:46 PM, Sammy Spets wrote:
My example is a way to ensure the update procedure is legitimately run by the admin user and thus can use the webserver process uid to replace files (which is necessary for the files to be usable by Drupal).
no offense, but this is exactly the kind of sloppy approach to file system permissions that leads to sites getting hacked and drupal getting a bad name. :(
your Drupal source files *DO NOT* have to be writable by the webserver process uid for "the files to be usable by Drupal" (!!!). the "files" directory is a special case, which is why a) it's been a source of numerous security issues and b) it should be handled with intensely defensive programming by anything that's touching it.
that aside, your Drupal source files must be _readable_ by the webserver process uid, period.
your Drupal source files *SHOULD* be _writable_ only by a *DIFFERENT* process uid, namely, your "high-privilege admin" user (could be root, but doesn't have to be, so long as it's *NOT* the same as your httpd).
until people completely understand this, they'll a) continue to fill up this thread with confusing posts that don't advance the end goal, and b) continue to setup insecure websites.
"automatic" upgrades, in whatever form, imply that some automatic service is going to be overwriting things in modules/* under your doc root. that requires that the automatic service has write access, and that means trouble.
unless the human (with the different uid, and higher powers to do damage) runs a script to trigger the upgrade (after backups, etc), life sucks.
so, if we're all clear on that, and we want to discuss exactly how much can be automated into this script, what the UI looks like, etc, etc, that's all fine and good, but a) it's premature (i'm just asking about agreement on what goes in the .info files to make all this possible) and b) it's a different discussion than "automatic site upgrades"...
-derek
p.s. sure, friendly amendment about downloading to a tmp directory and only putting the site in maintenance mode for as little time as possible is totally reasonable. i wasn't being as careful about the exact steps in that reply, since my main point was *THE HUMANS MUST DO IT* or life sucks.
p.p.s. did i mention that human intervention is required for this to be a good idea? ;)
On 22/11/06, Derek Wright <drupal@dwwright.net> wrote:
your Drupal source files *DO NOT* have to be writable by the webserver process uid for "the files to be usable by Drupal" (!!!). the "files" directory is a special case, which is why a) it's been a source of numerous security issues and b) it should be handled with intensely defensive programming by anything that's touching it.
Yep, a very important aspect of web application security is to protect the web application from itself. The web server can't tell the difference between a legitimate request to overwrite application files and one that has manipulated the application into doing so. Trust nothing (or more practically trust as little as possible). Ideally the web server and the web application shouldn't trust each other. Changes to the application should be done "out of band" ie not through the same medium the rest of the internet accesses the application. Of course, on a shared host there will be compromises that need to be made. For instance phpsuexec effectively violates the "out of band" idea by running the site as the "out of band" admin user, but it adds protection against other sites on a shared host which is arguably more important. When not on a shared host it would be better not to use phpsuexec so running the app and changing the app are separated from each other. Hopefully that wasn't offtopic. Apologies for rehashing something most of you are already fully aware of, I just wanted to reiterate what Derek was saying in a slightly different way to make sure any less experienced developers understood his point. -- Cheers Anton
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Wow, Oswald Jaskolla wrote:
I am currently working on a system to automatically install modules.
looks like I really hit a nerve there. So let me clarify a few things: - - Downloading and installing is only done on explicit request of the administrator. I am not Microsoft. - - Downloaded files are not less safe because they are downloaded via PHP. There is currently no checksumming available and apart from developers nobody looks into the code to see if it was tampered with. - - There are a lot of drupal installations for development and testing, that do not have the same security needs as production sites have. - - Typo3 does it. The only security issue remaining is having write access to the modules directory. If the actual downloading and unpacking is done via a one time cron job, this cron job could temporarily alter the access mode of the target directory, minimizing the time that the directory is writable. Greetings, - -- Oswald Jaskolla Ingenieurbüro Richard Schieferdecker Kreuzherrenstraße 2 52062 Aachen Tel.: 02 41 / 409 54 43 Fax: 02 41 / 477 05 199 mobil: 01 64 / 941 06 75 eMail: oswald.jaskolla@schieferdecker.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFZCSquinSHQ/4/T4RAsUmAJ4sTVuIs5eKpQgOCn9sZ6QvOub7YwCeN39w pnLSOei74O+fQkwTaHF1sho= =aIUQ -----END PGP SIGNATURE-----
write perms to modules directory from drupal as web server user is really hard for me to swallow.... any package managers like script should be run from the command line as a privileged user. should do it's set job and be bullet proof. On Wed, 2006-11-22 at 11:21 +0100, Oswald Jaskolla wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Wow,
Oswald Jaskolla wrote:
I am currently working on a system to automatically install modules.
looks like I really hit a nerve there. So let me clarify a few things:
- - Downloading and installing is only done on explicit request of the administrator. I am not Microsoft. - - Downloaded files are not less safe because they are downloaded via PHP. There is currently no checksumming available and apart from developers nobody looks into the code to see if it was tampered with. - - There are a lot of drupal installations for development and testing, that do not have the same security needs as production sites have. - - Typo3 does it.
The only security issue remaining is having write access to the modules directory. If the actual downloading and unpacking is done via a one time cron job, this cron job could temporarily alter the access mode of the target directory, minimizing the time that the directory is writable.
Greetings, - -- Oswald Jaskolla Ingenieurbüro Richard Schieferdecker Kreuzherrenstraße 2 52062 Aachen
Tel.: 02 41 / 409 54 43 Fax: 02 41 / 477 05 199 mobil: 01 64 / 941 06 75 eMail: oswald.jaskolla@schieferdecker.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFFZCSquinSHQ/4/T4RAsUmAJ4sTVuIs5eKpQgOCn9sZ6QvOub7YwCeN39w pnLSOei74O+fQkwTaHF1sho= =aIUQ -----END PGP SIGNATURE-----
On Nov 22, 2006, at 5:04 PM, Darrel O'Pry wrote:
write perms to modules directory from drupal as web server user is really hard for me to swallow....
any package managers like script should be run from the command line as a privileged user. should do it's set job and be bullet proof.
Let's not forget that very few users use the command line to work with Drupal. Let's also not make unnecessary assumptions about how an automated module install or upgrade would work. The security issues will be worked out if people share more ideas for how it can be done than for how it can't be done. Oswald was asking to collaborate. The negative reactions give the impression that some people would rather work on their own. Not very open source. Am I missing some history here?
On Wednesday 22 November 2006 17:33, Darren Oh wrote:
On Nov 22, 2006, at 5:04 PM, Darrel O'Pry wrote:
write perms to modules directory from drupal as web server user is really hard for me to swallow....
any package managers like script should be run from the command line as a privileged user. should do it's set job and be bullet proof.
Let's not forget that very few users use the command line to work with Drupal. Let's also not make unnecessary assumptions about how an automated module install or upgrade would work. The security issues will be worked out if people share more ideas for how it can be done than for how it can't be done.
I don't see how it can be done any other way. The apache user should NOT have write access to code files. To upgrade modules, you need write access to code files. Therefore, you either need to temporarily give apache write access to code files (which you can't do from within a web app running in apache, obviously) or run the upgrade as a user that already has write access to them. The Drupal 5 installer already says "please manually give apache write access" before it runs (something you need a command line for, or a toggle in an FTP client) and "hey, lock down access now" afterward for settings.php. That kinda works for one file, but for an entire file tree I don't see how that would work.
Oswald was asking to collaborate. The negative reactions give the impression that some people would rather work on their own. Not very open source. Am I missing some history here?
I think the "history" is "we're already working on it in way X, it's a hard problem, please don't confuse us with way Y that has problems that way X is already trying to address". Note that I am not part of the "we" in that sentence; I am just summarizing what I've seen said in the past year. :-) -- Larry Garfield AIM: LOLG42 larry@garfieldtech.com ICQ: 6817012 "If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it." -- Thomas Jefferson
Therefore, you either need to temporarily give apache write access to code files (which you can't do from within a web app running in apache, obviously) or run the upgrade as a user that already has write access to them.
Are we sure that you can't change the owner of the current process through Apache? We can execute arbitrary shell commands, if needed. The ideal solution could then be a script that can be invoked both from the web and from the command-line. Through the browser, it would ask for your local username/password, and then perform the upgrade tasks (only from a very limited set of commands, e.g. unpacking module files and copying them into the right dir). From the command-line, it would just assume the current user is the right one already. Steven Wittens
On Wednesday 22 November 2006 20:11, Steven Wittens wrote:
Are we sure that you can't change the owner of the current process through Apache? We can execute arbitrary shell commands, if needed. The ideal solution could then be a script that can be invoked both from the web and from the command-line.
As far as I know, the only way to change the user the process is running as is with the suexec apache module, in which case the process runs as the user that owns the PHP script that is running. Otherwise, changing the owner of a running process requires root access, something no web app should ever have. Think about it from the other side: If you had a PHP script that could decide to change the user it's running as to some arbitrary user, would YOU want it on your server? I wouldn't.
Through the browser, it would ask for your local username/password, and then perform the upgrade tasks (only from a very limited set of commands, e.g. unpacking module files and copying them into the right dir). From the command-line, it would just assume the current user is the right one already.
Steven Wittens
All I can think of here off the top of my head would be exec()ing su, but again any shared host that makes that possible I don't want to touch. Although, there are web control panels for the system itself, like webmin. I'm not entirely sure how they do their thing. That may be something to look into, but I still expect that any shared web host worth the money is going to not allow a normal user to run anything like that, on principle. -- Larry Garfield AIM: LOLG42 larry@garfieldtech.com ICQ: 6817012 "If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of every one, and the receiver cannot dispossess himself of it." -- Thomas Jefferson
Op donderdag 23 november 2006 03:33, schreef Larry Garfield:
Although, there are web control panels for the system itself, like webmin. I'm not entirely sure how they do their thing. That may be something to look into, but I still expect that any shared web host worth the money is going to not allow a normal user to run anything like that, on principle.
Webmin has its own webserver compiled, running on a different port. This server runs with root(alike) permissions. When someone compromises such a tool, he/she can do anything from within a browser. I have already played with another option, being a single instance of lighthttp running as a different user under a different UID, with root permissions. That lighthttp serves a single hardened Drupal site Once logged in on that site, certain modules can speak for example to sympal scripts, e.g. to install a new multisite. OR simply exectute exec() tasks on the server, as root. However, Drupal was/is not secure enough *IMHO* to handle such a critical task. E.g. too much issues with XSS and so were released last year, to serve such a critical task. But the idea works: Drupal can be used as a vhost management tool. NOTE: Webmin is not very secure either (see the long list of security issues on their site), but its architecture allows for better security configurations. Bèr -- Drupal, Ruby on Rails and Joomla! development: webschuur.com | Drupal hosting: sympal.nl
On Thu, 2006-11-23 at 03:11 +0100, Steven Wittens wrote:
Therefore, you either need to temporarily give apache write access to code files (which you can't do from within a web app running in apache, obviously) or run the upgrade as a user that already has write access to them.
Are we sure that you can't change the owner of the current process through Apache? We can execute arbitrary shell commands, if needed. The ideal solution could then be a script that can be invoked both from the web and from the command-line.
Through the browser, it would ask for your local username/password, and then perform the upgrade tasks (only from a very limited set of commands, e.g. unpacking module files and copying them into the right dir). From the command-line, it would just assume the current user is the right one already.
Steven Wittens
A well written & setuid script would do the job. You can call it through system or exec I believe. There are many applications that use this technique. The first that come to mind or qmail and majordomo. I believe windows even supports a run as user option for its files. Something to keep in mind about such a script is it should in no way receive input from the browser. It should only be triggered, and should still be paranoid about its config files... aka they're still owned by the proper non-apache uid, don't contain any sort of exploit like code... At the end of the day I'm more concerned about other websites on shared servers and partitioning risk between multiple sites on the same server. As it is any code injection bugs in drupal could me a lost db, and the real value of most websites is what's in the database. .darrel.
-----Original Message----- From: development-bounces@drupal.org on behalf of Darrel O'Pry Sent: Mon 11/27/2006 6:48 AM To: development@drupal.org Subject: Re: [development] RFC: letting modules phone home to checkfor new releases On Thu, 2006-11-23 at 03:11 +0100, Steven Wittens wrote:
Therefore, you either need to temporarily give apache write access to code files (which you can't do from within a web app running in apache, obviously) or run the upgrade as a user that already has write access to them.
Are we sure that you can't change the owner of the current process through Apache? We can execute arbitrary shell commands, if needed. The ideal solution could then be a script that can be invoked both from the web and from the command-line.
Through the browser, it would ask for your local username/password, and then perform the upgrade tasks (only from a very limited set of commands, e.g. unpacking module files and copying them into the right dir). From the command-line, it would just assume the current user is the right one already.
Steven Wittens
A well written & setuid script would do the job. You can call it through system or exec I believe. There are many applications that use this technique. The first that come to mind or qmail and majordomo. I believe windows even supports a run as user option for its files. Something to keep in mind about such a script is it should in no way receive input from the browser. It should only be triggered, and should still be paranoid about its config files... aka they're still owned by the proper non-apache uid, don't contain any sort of exploit like code... At the end of the day I'm more concerned about other websites on shared servers and partitioning risk between multiple sites on the same server. As it is any code injection bugs in drupal could me a lost db, and the real value of most websites is what's in the database. .darrel. and I'd be interested in seeing if it would work through IIS or Apache on Windows.
On 11/22/06, Darren Oh <darrenoh@sidepotsinternational.com> wrote:
On Nov 22, 2006, at 5:04 PM, Darrel O'Pry wrote:
write perms to modules directory from drupal as web server user is really hard for me to swallow....
any package managers like script should be run from the command line as a privileged user. should do it's set job and be bullet proof.
Let's not forget that very few users use the command line to work with Drupal. Let's also not make unnecessary assumptions about how an automated module install or upgrade would work. The security issues will be worked out if people share more ideas for how it can be done than for how it can't be done.
It's likely that the best path is command line tools first -- "the market" can build whatever GUI tools are desired on top of it. At some point, some of those may make it to Drupal directly, but taking the responsibility for remote upgrades is not something I would not necessarily want to expose the community to. Lots of people are still thinking about individual websites. Think Fantastico and other host scripts that can drive command line hooks into Drupal as privileged users. Other options -- like pointing your website at different repositories (supported by Derek's outlined framework) -- are also taken into consideration. Oswald was asking to collaborate. The negative reactions give the
impression that some people would rather work on their own. Not very open source. Am I missing some history here?
No, you're not....you're missing frustration with a core (in both senses of the word) group of people who have played nice and developed stuff directly for the core of Drupal, undergoing excruciating levels of issue threads to have all their work torn apart and improved upon. And then someone drops in with an alternate system. What would have been nice: Review the existing threads, post new issues pointing to sandbox / module code, and integrate. That can still happen....there is likely some good code to be integrated. Secondly, Oswald didn't cover much in the nature of security. That's what I put my 2¢ in on....security, security, security. -- Boris Mann Vancouver 778-896-2747 San Francisco 415-367-3595 Skype borismann http://www.bryght.com
The whole shared-hosting without shell access and restricted Apache server access to files has been well-analyzed and a solution implemented by Allie Micka of Pajunas Interactive -- and she's a great Drupal developer to boot. So if you want to solve this problem of creating install tools that will really work in a secure fashion on shared-hosting providers, it would be really worth the trouble to get her involved in the conversation. :-) ..chrisxj
There already is an automated installer.... sigh. Op dinsdag 21 november 2006 19:13, schreef Oswald Jaskolla:
I am currently working on a system to automatically install modules (i.e download and unpack into the appropriate directory) and upgrade modules (i.e. overwrite present files). The architecture is finished:
http://cvs.drupal.org/viewcvs/drupal/contributions/modules/sympal_scripts/mo... -- Drupal, Ruby on Rails and Joomla! development: webschuur.com | Drupal hosting: sympal.nl
Bèr, have you ever written a tutorial on how to use the Sympal scripts? I think a lot of people are unaware of what they can do and how you use them. Sorry if you have, and I missed it =) Bèr Kessels wrote:
There already is an automated installer.... sigh.
Op dinsdag 21 november 2006 19:13, schreef Oswald Jaskolla:
I am currently working on a system to automatically install modules (i.e download and unpack into the appropriate directory) and upgrade modules (i.e. overwrite present files). The architecture is finished:
http://cvs.drupal.org/viewcvs/drupal/contributions/modules/sympal_scripts/mo...
Op woensdag 22 november 2006 14:00, schreef Robert Douglass:
Bèr, have you ever written a tutorial on how to use the Sympal scripts? I think a lot of people are unaware of what they can do and how you use them. Sorry if you have, and I missed it =)
I haven't, mostly because not all stuff is made by me, and because it all is still alpha-stage. Meaning: stuff works allright, but stuff can change Just Like That, when I feel like it, or when I am in a bad mood :). Potentially letting you loose loads of data, for example. The closest you can get to a manual is this document of the commandline reference, for once its ready: http://webschuur.com/node/637 Its both a reference and a statusdoc, because I tell below the reference what is implemented and what not. Bèr
participants (24)
-
adrian rossouw -
andrew morton -
Anton -
Arnab Nandi -
Boris Mann -
Bèr Kessels -
Chris Johnson -
Corey Bordelon -
Darrel O'Pry -
Darren Oh -
Derek Wright -
Dries Buytaert -
Earl Miles -
Gabor Hojtsy -
Gerhard Killesreiter -
Johan Forngren -
Karoly Negyesi -
Larry Garfield -
Oswald Jaskolla -
Richard Morse -
Robert Douglass -
Sammy Spets -
Steven Peck -
Steven Wittens