Back to Curl-based content generation
Greetings, sorry if I couldn't answer earlier to all the feedback that came in the original thread a couple weeks ago. It looks like several things I wrote were simply ignored by some posters. I'm going to shortly sum them up again, and then ask one (last, I promise) related question. Quoting from several posts of the original thread:
Quite frankly, shell + curl are not an adequate tool for the job, unless you grok awk and sed really well. ... You will likely also need to have a cookie .... View Source. Search '<form' . You'll see everything that needs to be post'ed for a given form. ... Everything in drupal is processed through index.php, so you don't need a sequence of URLs for your post'ing either.
IIRC, I had already made clear, before getting these answers, that: - I'm already much more expert in awk/sed/bash/perl coding than with PHP, so a non-php solution is much more time-efficient for me, if at all possible. Not to mention that those tools/languages give, in the real world of desktop/usb key linux distros, more guarantees to work out of the box without tweaking or installing extra-packages than anything requiring php. - I know very well what HTML forms and HTTP cookies are, and have already done this with non-drupal websites. And the URLs you see in the browser when you add content by hand are NOT "index.php + some parameters" - (almost) the only solution I am interested into is how to add new content remotely, when I cannot alter in any way the configuration of the server where Drupal runs. I'm not rewriting all this to start a fight, really, just to clarify why I asked what I asked and what I already know, that is to save your time. This said, I do understand all the points about Drupal being more interested in offering a flexible API than in supporting this kind of things. So, I will now go back to my desk and put together, by myself, my custom shell+curl hack based on the one posted here by Martin Stadler (thanks, Martin!). However, let me ask you this: If you already said it, I do confess I didn't recognize it: what is the smallest possible set of Drupal(related) Php **files or libraries** that I should install on a Linux *desktop* without HTTP server but with a working PHP-CLI interpreter, to put together a PHP script which can log into a remote Drupal site and add nodes? Thanks, M. -- Your own civil rights and the quality of your life heavily depend on how software is used *around* you: http://digifreedom.net/node/84
On Thu, Feb 19, 2009 at 11:47 AM, M. Fioretti <mfioretti@nexaima.net> wrote:
Greetings,
sorry if I couldn't answer earlier to all the feedback that came in the original thread a couple weeks ago.
It looks like several things I wrote were simply ignored by some posters. I'm going to shortly sum them up again, and then ask one (last, I promise) related question.
Quoting from several posts of the original thread:
Quite frankly, shell + curl are not an adequate tool for the job, unless you grok awk and sed really well. ... You will likely also need to have a cookie .... View Source. Search '<form' . You'll see everything that needs to be post'ed for a given form. ... Everything in drupal is processed through index.php, so you don't need a sequence of URLs for your post'ing either.
IIRC, I had already made clear, before getting these answers, that:
- I'm already much more expert in awk/sed/bash/perl coding than with PHP, so a non-php solution is much more time-efficient for me, if at all possible. Not to mention that those tools/languages give, in the real world of desktop/usb key linux distros, more guarantees to work out of the box without tweaking or installing extra-packages than anything requiring php.
- I know very well what HTML forms and HTTP cookies are, and have already done this with non-drupal websites. And the URLs you see in the browser when you add content by hand are NOT "index.php + some parameters"
- (almost) the only solution I am interested into is how to add new content remotely, when I cannot alter in any way the configuration of the server where Drupal runs.
I'm not rewriting all this to start a fight, really, just to clarify why I asked what I asked and what I already know, that is to save your time.
This said, I do understand all the points about Drupal being more interested in offering a flexible API than in supporting this kind of things. So, I will now go back to my desk and put together, by myself, my custom shell+curl hack based on the one posted here by Martin Stadler (thanks, Martin!). However, let me ask you this:
If you already said it, I do confess I didn't recognize it: what is the smallest possible set of Drupal(related) Php **files or libraries** that I should install on a Linux *desktop* without HTTP server but with a working PHP-CLI interpreter, to put together a PHP script which can log into a remote Drupal site and add nodes?
Maybe I lost some part of this discussion, but I wonder: Why would you need any kind of server or PHP on your desktop to work with a remote Drupal web site? And how could a web site respond to requests from any of those? Are you talking about putting together a CLI web browser?
Thanks, M.
-- Your own civil rights and the quality of your life heavily depend on how software is used *around* you: http://digifreedom.net/node/84
M. wants to shove content into a remote Drupal site from his desktop. So he wants to know the minimum to install in terms of a script which bootstraps Drupal and uses curl to get the content and node_save to save it to the remote URL. I think this might be better served by running an XMLRPC or other server (see services module), then you can use anything you like to speak the same protocol. Victor Kane http://awebfactory.com.ar On Thu, Feb 19, 2009 at 9:04 AM, Cog Rusty <cog.rusty@gmail.com> wrote:
On Thu, Feb 19, 2009 at 11:47 AM, M. Fioretti <mfioretti@nexaima.net> wrote:
Greetings,
sorry if I couldn't answer earlier to all the feedback that came in the original thread a couple weeks ago.
It looks like several things I wrote were simply ignored by some posters. I'm going to shortly sum them up again, and then ask one (last, I promise) related question.
Quoting from several posts of the original thread:
Quite frankly, shell + curl are not an adequate tool for the job, unless you grok awk and sed really well. ... You will likely also need to have a cookie .... View Source. Search '<form' . You'll see everything that needs to be post'ed for a given form. ... Everything in drupal is processed through index.php, so you don't need a sequence of URLs for your post'ing either.
IIRC, I had already made clear, before getting these answers, that:
- I'm already much more expert in awk/sed/bash/perl coding than with PHP, so a non-php solution is much more time-efficient for me, if at all possible. Not to mention that those tools/languages give, in the real world of desktop/usb key linux distros, more guarantees to work out of the box without tweaking or installing extra-packages than anything requiring php.
- I know very well what HTML forms and HTTP cookies are, and have already done this with non-drupal websites. And the URLs you see in the browser when you add content by hand are NOT "index.php + some parameters"
- (almost) the only solution I am interested into is how to add new content remotely, when I cannot alter in any way the configuration of the server where Drupal runs.
I'm not rewriting all this to start a fight, really, just to clarify why I asked what I asked and what I already know, that is to save your time.
This said, I do understand all the points about Drupal being more interested in offering a flexible API than in supporting this kind of things. So, I will now go back to my desk and put together, by myself, my custom shell+curl hack based on the one posted here by Martin Stadler (thanks, Martin!). However, let me ask you this:
If you already said it, I do confess I didn't recognize it: what is the smallest possible set of Drupal(related) Php **files or libraries** that I should install on a Linux *desktop* without HTTP server but with a working PHP-CLI interpreter, to put together a PHP script which can log into a remote Drupal site and add nodes?
Maybe I lost some part of this discussion, but I wonder: Why would you need any kind of server or PHP on your desktop to work with a remote Drupal web site? And how could a web site respond to requests from any of those? Are you talking about putting together a CLI web browser?
Thanks, M.
-- Your own civil rights and the quality of your life heavily depend on how software is used *around* you: http://digifreedom.net/node/84
On Thu, Feb 19, 2009 13:04:56 PM +0200, Cog Rusty wrote:
Maybe I lost some part of this discussion, but I wonder: Why would you need any kind of server or PHP on your desktop to work with a remote Drupal web site?
because I want to add nodes from the command line at my desktop, for reasons not really relevant here (in short, it would integrate much, much better with my own, already existing document management/publishing/backup flow)... ...but, when I asked info to write my own bash/curl script to do this, many members of this list simply answered saying "install this or that drupal module or php library on the server" :-)
Are you talking about putting together a CLI web browser?
No, just an http client, ie a shell script which sends to Drupal the same HTML code that a real browser would transmit when I add a node by hand, via keyboard and mouse. The web is full of scripts like this, but doing it to drupal seems much harder/undocumented than in other cases. Marco -- Your own civil rights and the quality of your life heavily depend on how software is used *around* you: http://digifreedom.net/node/84
Quoting "M. Fioretti" <mfioretti@nexaima.net>:
On Thu, Feb 19, 2009 13:04:56 PM +0200, Cog Rusty wrote:
Maybe I lost some part of this discussion, but I wonder: Why would you need any kind of server or PHP on your desktop to work with a remote Drupal web site?
because I want to add nodes from the command line at my desktop, for reasons not really relevant here (in short, it would integrate much, much better with my own, already existing document management/publishing/backup flow)...
I add nodes to my site via a php script executed in batch. The script creates a node object and calls node_save(). No curl needed. Do you control the remote Drupal system?
...but, when I asked info to write my own bash/curl script to do this, many members of this list simply answered saying "install this or that drupal module or php library on the server" :-)
Well, that is how we modify Drupal you know. You create a module to do this or that.
Are you talking about putting together a CLI web browser?
No, just an http client, ie a shell script which sends to Drupal the same HTML code that a real browser would transmit when I add a node by hand, via keyboard and mouse. The web is full of scripts like this, but doing it to drupal seems much harder/undocumented than in other cases.
Undocumented because it is out of the norm. I suggest you create a php script for the server side that will create the node object and do the node_save(). Then create a curl script on the client side that will fill the post data for the server script. -- Earnie http://r-feed.com Make a Drupal difference and review core patches. -- http://for-my-kids.com/ -- http://give-me-an-offer.com/
You could always simply run a curl script that just mimics a login then page submission. On Thu, Feb 19, 2009 at 2:23 PM, Earnie Boyd <earnie@users.sourceforge.net>wrote:
Quoting "M. Fioretti" <mfioretti@nexaima.net>:
On Thu, Feb 19, 2009 13:04:56 PM +0200, Cog Rusty wrote:
Maybe I lost some part of this discussion, but I wonder: Why would
you need any kind of server or PHP on your desktop to work with a remote Drupal web site?
because I want to add nodes from the command line at my desktop, for reasons not really relevant here (in short, it would integrate much, much better with my own, already existing document management/publishing/backup flow)...
I add nodes to my site via a php script executed in batch. The script creates a node object and calls node_save(). No curl needed. Do you control the remote Drupal system?
...but, when I asked info to write my own bash/curl script to do this,
many members of this list simply answered saying "install this or that drupal module or php library on the server" :-)
Well, that is how we modify Drupal you know. You create a module to do this or that.
Are you talking about putting together a CLI web browser?
No, just an http client, ie a shell script which sends to Drupal the same HTML code that a real browser would transmit when I add a node by hand, via keyboard and mouse. The web is full of scripts like this, but doing it to drupal seems much harder/undocumented than in other cases.
Undocumented because it is out of the norm. I suggest you create a php script for the server side that will create the node object and do the node_save(). Then create a curl script on the client side that will fill the post data for the server script.
-- Earnie http://r-feed.com Make a Drupal difference and review core patches.
-- Ashraf Amayreh http://aamayreh.org
On Thu, Feb 19, 2009 14:54:51 PM +0200, Ashraf Amayreh wrote:
You could always simply run a curl script that just mimics a login then page submission.
honestly, this is getting funny now. This is exactly what *I* asked here weeks ago: info and documentation to put together just such a script with the minimum possible amount of manual trial and error, http headers sniffing and so on. Nothing more. Marco -- Your own civil rights and the quality of your life heavily depend on how software is used *around* you: http://digifreedom.net/node/84
M. Fioretti wrote:
On Thu, Feb 19, 2009 14:54:51 PM +0200, Ashraf Amayreh wrote:
You could always simply run a curl script that just mimics a login then page submission.
honestly, this is getting funny now. This is exactly what *I* asked here weeks ago: info and documentation to put together just such a script with the minimum possible amount of manual trial and error, http headers sniffing and so on. Nothing more.
Marco
Request the node/add/{node-type}page. Grep out all the input elements along with their default values. Change the needed default values (ie: title, body, etc.) then resubmit it via a post request to the same URL. If you want to catch error messages then you have to grep out the error division of the URL you submitted to. As far as what data is required - well that depends upon the site's configuration. It depends upon modules the site has enabled, how the content type is configured, what options are available to the current user's access level, etc. The only other option is the one stated numerous times - using a module. Either activate the Blog API module (and use XML-RPC), or use something like the services module. Now I have seen your reply about this:
No, as I already said several times in this thread. More exactly: I do control the one where I'd need this first, but not others on which I'll have to work soon, that's why I'm looking for one solution which doesn't rely on access to server. So, this wouldn't work, would it:
If you are doing this for other sites, then I can only assume it's because they are asking you to (if not then that opens up a whole new debate on ethics). If that's the case then can't you simple ask them to install a module for this, even if it's as simple as activating the BlogAPI module?
On Thu, Feb 19, 2009 at 8:57 AM, Jamie Holly <hovercrafter@earthlink.net>wrote:
M. Fioretti wrote:
No, as I already said several times in this thread. More exactly: I do
control the one where I'd need this first, but not others on which I'll have to work soon, that's why I'm looking for one solution which doesn't rely on access to server. So, this wouldn't work, would it:
If you are doing this for other sites, then I can only assume it's because they are asking you to (if not then that opens up a whole new debate on ethics).
Indeed. In fact, the whole conversation has inspired me to write a module that blocks flooding by tracking form tokens instead of hosts. I get that (attempted DOS via automated comment submissions) periodically.
Executing arbitrary stuff in Drupal via cURL? Simple. As in "SimpleTest". The entire testing suite is based on cURL. I recommend having a look at http://drupal.org/project/simpletest, resp. /modules/simpletest in CVS HEAD of Drupal core. To get a feeling of how to use it properly, I recommend writing some tests for D7. That way, you'll learn how to properly use the testing framework, and thus, how to use cURL for doing arbitrary stuff. We even have some issues prepared for you: http://drupal.org/project/issues/drupal/term/345 sun
On Thu, Feb 19, 2009 08:57:17 AM -0500, Jamie Holly wrote:
Request the node/add/{node-type}page. Grep out all the input elements along with their default values...
OK, thanks.
If you are doing this for other sites, then I can only assume it's because they are asking you to (if not then that opens up a whole new debate on ethics).
Ethics? Are you serious? I have several **real** accounts on several, independent Drupal-based websites. **real** meaning "I'm a real, honest, regularly registered and approved user on all those websites". I must or want add content to all those websites more or less regularly, say several times a week. And in at least one case I also need to put online as nodes (without, I repeat, any possibility to modify the server setup) lots of already existing text files. And I just happen to find a much smarter and more efficient use of my time to type in my full-screen, favourite, heavily customized text-editor and then launch some "publish-to-drupal-website" script than log in to cut-and-paste stuff or click drop-down menus every time. Period. What's wrong with such a wish? Above all, what's ethic got to do with this? The only issue may be, in general, if this lessened the income from web ads of a site. Rest assured this is not an issue for any of the websites where I need to do this. I perfectly know how bad it is for websites, I even publicly wrote about it 7 years ago (http://www.linuxjournal.com/article/5623). So don't worry, ethics is perfectly safe here. Incidentally, this is not the first, more or less explicit "why are you asking us help on how to spam drupal websites" kind of answer I've got in this thread, and it's starting to be offensive, both for me and for you Drupal developers. I write online for a living: would I be so idiot to mess my reputation in such a way? Even if I didn't write, if I were a spammer, would I be so idiot to come here with my real name to ask? If I were a spammer I would have already made all of this work without ****ever**** asking in public, because I'd have had a much more powerful incentive to spend days and nights making spambots work without letting everybody know about it.
If that's the case then can't you simple ask them to install a module for this, even if it's as simple as activating the BlogAPI module?
because some of those websites don't give a damn if I add content via browser, voodoo spells or anything in between, but simply won't change their default configuration for me. Sure, this is my problem only, it's just that it's tiring to repeat it many times. No problem, however. Marco -- Your own civil rights and the quality of your life heavily depend on how software is used *around* you: http://digifreedom.net/node/84
I'm sorry if you felt it an insult, but it is a legitimate concern . The reason the "common scripts" you found out there aren't working is not a default of Drupal, but rather a feature. That feature helps reduce spam, and you are asking the development list of Drupal how to circumvent this highly valuable feature. Perhaps your questions would be better raised in the support area of Drupal.org or support mailing list. Jamie Holly M. Fioretti wrote:
On Thu, Feb 19, 2009 08:57:17 AM -0500, Jamie Holly wrote:
Request the node/add/{node-type}page. Grep out all the input elements along with their default values...
OK, thanks.
If you are doing this for other sites, then I can only assume it's because they are asking you to (if not then that opens up a whole new debate on ethics).
Ethics? Are you serious? I have several **real** accounts on several, independent Drupal-based websites. **real** meaning "I'm a real, honest, regularly registered and approved user on all those websites".
I must or want add content to all those websites more or less regularly, say several times a week. And in at least one case I also need to put online as nodes (without, I repeat, any possibility to modify the server setup) lots of already existing text files. And I just happen to find a much smarter and more efficient use of my time to type in my full-screen, favourite, heavily customized text-editor and then launch some "publish-to-drupal-website" script than log in to cut-and-paste stuff or click drop-down menus every time. Period.
What's wrong with such a wish? Above all, what's ethic got to do with this? The only issue may be, in general, if this lessened the income from web ads of a site. Rest assured this is not an issue for any of the websites where I need to do this. I perfectly know how bad it is for websites, I even publicly wrote about it 7 years ago (http://www.linuxjournal.com/article/5623). So don't worry, ethics is perfectly safe here.
Incidentally, this is not the first, more or less explicit "why are you asking us help on how to spam drupal websites" kind of answer I've got in this thread, and it's starting to be offensive, both for me and for you Drupal developers. I write online for a living: would I be so idiot to mess my reputation in such a way? Even if I didn't write, if I were a spammer, would I be so idiot to come here with my real name to ask? If I were a spammer I would have already made all of this work without ****ever**** asking in public, because I'd have had a much more powerful incentive to spend days and nights making spambots work without letting everybody know about it.
If that's the case then can't you simple ask them to install a module for this, even if it's as simple as activating the BlogAPI module?
because some of those websites don't give a damn if I add content via browser, voodoo spells or anything in between, but simply won't change their default configuration for me. Sure, this is my problem only, it's just that it's tiring to repeat it many times. No problem, however.
Marco
I'm trying to clean-up this thread (I know I shoudn't)... Useful things first so people who don't care about nonsense can skip the rest: Marco wants to be able to post content to arbitrary (not under any control by him) Drupal sites via command line. There seem to be two options: 1) Use a script like http://drupal.org/node/80548 . If you for some reason don't want to use PHP it should be easy to extract the cURL code to use it in a shell script or whatever as I pointed out earlier on this list. 2) Use SimpleTest as this seems to contain this functionality also based on cURL. @Marco: What more do you need? What kind of better prepared solution do you expect? What kind of documentation is missing? You already said you would be fine with these resources. I don't quote understand why this is still an issue. Now comes the social part: I think Marco is right when he is unhappy with most of the answers to his question. It seems most people didn't read so why do you answer then? I'd like to see more respect. Ok, I stop here.
M. Fioretti wrote:
Incidentally, this is not the first, more or less explicit "why are you asking us help on how to spam drupal websites" kind of answer I've got in this thread, and it's starting to be offensive, both for me and for you Drupal developers. I write online for a living: would I be so idiot to mess my reputation in such a way? Even if I didn't write, if I were a spammer, would I be so idiot to come here with my real name to ask? If I were a spammer I would have already made all of this work without ****ever**** asking in public, because I'd have had a much more powerful incentive to spend days and nights making spambots work without letting everybody know about it.
Marco you are a relative unknown on this mailing list to me and others. Your drupal user history is entirely selfserving (http://drupal.org/user/166184/track) with out ever returning the favor to the community. Despite this many people have stepped forward to offer you ideas and help. Please understand that we all provide this advice and assistance out of the generosity of our hearts and the desire for our project to succeed. Your abrasiveness and ingratitude in response to this help are not appreciated nor welcome. Furthermore, your question IS suspicious without a better historical understanding of you as a drupal user/contributor so it should be met with the concern you have experienced. Several valid suggestions have been proposed and you would do well to investigate each of them. Please take any further discussion of it off the development list as this is a support question. -mf
On Thu, Feb 19, 2009 07:23:29 AM -0500, Earnie Boyd wrote:
Quoting "M. Fioretti" <mfioretti@nexaima.net>:
On Thu, Feb 19, 2009 13:04:56 PM +0200, Cog Rusty wrote:
Maybe I lost some part of this discussion, but I wonder: Why would you need any kind of server or PHP on your desktop to work with a remote Drupal web site?
No, as I already said several times in this thread. More exactly: I do control the one where I'd need this first, but not others on which I'll have to work soon, that's why I'm looking for one solution which doesn't rely on access to server. So, this wouldn't work, would it:
I add nodes to my site via a php script executed in batch. The script creates a node object and calls node_save().
because it has to run on the same server which hosts the drupal website, right?
Well, that is how we modify Drupal you know. You create a module to do this or that.
The only problem being that "we" only means drupal sysadmins, not drupal users. I don't want to modify anything in the drupal websites I need to use, they work just fine. I only want to write software which talks to them from remote computers, to add nodes. Thanks, Marco -- Your own civil rights and the quality of your life heavily depend on how software is used *around* you: http://digifreedom.net/node/84
Hello, all. To save time, I'll answer with only one messages to all the replies which came after the half-rant I sent here yesterday. Please read it all, it isn't that long. I do enough, in public, to support FOSS and related issues like digital rights (cfr http://mfioretti.net) to not worry at all if sometimes I *may* appear as a freeloader with just a "short, self-serving involvement in the community" of one of the many FOSS tools I use. So, no problem on that front, really. Just a passing note that a quick internet search would have definitely proved that *I* wouldn't come here or anywhere else to learn how to spam. I admit that in my *very* first messages I wasn't clear enough, partly because I didn't even know myself what I should exactly ask. What actually frustrated me in all this too long thread, enough to write what I wrote yesterday, is just two things: - honestly, it took too many messages to be told clearly (regardless of manners) that you simply do NOT want to make easier the way I'd like to interact with Drupal, and _why_ you think so. Please note that I'm NOT questioning at all your decision and motives here. I'm just saying "if I had been told all this explicitly the first day, we'd have all saved a lot of time". Maybe this decision and its rationale should be posted on the Drupal website. - regardless of the point above, I have gotten more than one reply where it was evident that the person answering to me simply hadn't bothered to read my message, the one he was replying to, from beginning to end. So, I hope you'll take all this as just a friendly suggestion to pay more attention and in general use the list more effectively in the future. Me, I have nothing to add, and certainly don't want to bicker anymore. Considering all the input I got, it looks that I need to try, at least for those websites where I can ask to change configuration, a command line script which talks remotely, via XML-RPC, to the blogapi module. If somebody can post me privately (**) relevant documentation or, even better, working code samples (Perl, Python, whatever)... thanks a lot in advance. The reason I'm asking is that the blogapi page at drupal.org documents how that code works internally and how it talks to drupal, but I haven't recognized any link to documentation useful to write the _client_ side. Be well, Marco (**) I will unsubscribe really soon, need to focus on other things now. -- Your own civil rights and the quality of your life heavily depend on how software is used *around* you: http://digifreedom.net/node/84
On Fri, Feb 20, 2009 at 11:32 AM, M. Fioretti <mfioretti@nexaima.net> wrote: ...snip...
- honestly, it took too many messages to be told clearly (regardless of manners) that you simply do NOT want to make easier the way I'd like to interact with Drupal, and _why_ you think so. Please note that I'm NOT questioning at all your decision and motives here. I'm just saying "if I had been told all this explicitly the first day, we'd have all saved a lot of time". Maybe this decision and its rationale should be posted on the Drupal website. ...snip...
Marco, the issue is not the people who do not want to do what you are looking for. There are things which all of us do not want to do,.and there is little that anyone can do about that. The issue is to find (or convert?) people who want to do what you are looking for. The rest is just talk.
On Thu, Feb 19, 2009 at 3:47 AM, M. Fioretti <mfioretti@nexaima.net> wrote:
- (almost) the only solution I am interested into is how to add new content remotely, when I cannot alter in any way the configuration of the server where Drupal runs.
Given the constraint that you cannot modify the server, it seems to me that the only way to do this is to mimic a user session via a client script. If you are only adding node then this shouldn't be too hard but it will take several requests and cookie management on the client. The session would look something like: POST login request POST node form submission (repeat node form POST for each node to be created) If you need to modify or delete nodes it becomes quite a bit harder since you will need to find the right node to modify before posting. Personally I would use perl on the client side if I had to do this since the client side tools are richer for perl than php. Now if I had the ability to alter the drupal installation I would use the services module and xmlrpc calls to get the data into drupal. I would still use perl for the xmlrpc client. -Craig
participants (11)
-
Ashraf Amayreh -
Cog Rusty -
Craig Forbes -
Daniel F. Kudwien -
Earl Dunovant -
Earnie Boyd -
Jamie Holly -
M. Fioretti -
Martin Stadler -
Michael Favia -
Victor Kane