[development] cURL and drupal_http_request do not properly download certain Google News feeds

[Moshe Weitzman]

for HTTP Monitor, I use the Net tab in Firebug or the Tamper Data
firebug extension.

On Tue, Jan 19, 2010 at 2:15 PM, Alex Barth <alex at developmentseed.org> wrote:
>
> After getting a report that
>
> http://news.google.com/news?pz=1&hl=ar&q=سوريا&cf=all&output=rss
>
> is not properly downloading with Feeds module, I dug deep and discovered
> that cURL and drupal_http_request() return an RSS feed with no items, while
> wget and PHP stream_get_contents() do return a full RSS feed with a number
> of items.
>
> Details here: http://drupal.org/node/689552
>
> I am unsure what is actually causing this peculiar behavior and I would
> appreciate people's input. The issue affects not just Feeds but any other
> Drupal module that downloads and processes Google News RSS feeds - including
> core aggregator.
>
> - This seems to be an issue where Google News decides, based on some request
> parameters, what content to return and what not - or am I missing something?
> - The user agent is the same in cases where the issue occurs and where it
> doesn't, I am using the same machine for all tests - what else could Google
> use to distinguish my requests?
> - Any tips on an 'HTTP monitor' I could be using to actually monitor
> outgoing HTTP requests from my local machine?
>
> Alex Barth
> http://www.developmentseed.org/blog
> tel (202) 250-3633
>
>
>
>
>