Odd PHP error_log entries (5 million in last 30 days)

[Krinkle]

It appears for each request to a website powered by adds nonsense log entries to the PHP error_log. Literally any request. 404 errors served by WordPress as well as regular page views of existing pages.

Sample:

2012/11/23 01:58:16 [error] 17739#0: *159536936 FastCGI sent in stderr: "PHP message: http://jqueryui.com/themeroller
PHP message: http://jqueryui.com/development
PHP message: http://jqueryui.com/support
PHP message: http://blog.jqueryui.com/
PHP message: http://jqueryui.com/about" while reading upstream, client: 192.156.102.101, server: api.jqueryui.com, request: "GET /slider/ HTTP/1.0", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "api.jqueryui.com", referrer: "..."

2012/11/23 01:58:51 [error] 17858#0: *204553 FastCGI sent in stderr: "PHP message: http://learn.jquery.com/javascript-101/operators/
PHP message: http://learn.jquery.com
PHP message: http://learn.jquery.com/about
PHP message: http://learn.jquery.com/contributing
PHP message: http://learn.jquery.com/style-guide" while reading response header from upstream, client: 125.22.78.82, server: *.jquery.com, request: "GET /javascript-101/operators/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "stage.learn.jquery.com", referrer: "..."

2012/11/23 01:57:23 [error] 17740#0: *159532723 FastCGI sent in stderr: "PHP message: http://qunitjs.com/
PHP message: http://qunitjs.com/
PHP message: http://qunitjs.com/cookbook/
PHP message: http://api.qunitjs.com/
PHP message: http://qunitjs.com/intro/
PHP message: http://qunitjs.com/extensions/" while reading upstream, client: 84.27.63.222, server: qunitjs.com, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "qunitjs.com", referrer: "..."

FastCGI obscures the error log by wrapping all errors from 1 request in one message.

The real PHP error_log reports this:

[23-Nov-2012 09:58:16 UTC] http://jqueryui.com/themeroller
[23-Nov-2012 09:58:16 UTC] http://jqueryui.com/development
[23-Nov-2012 09:58:16 UTC] http://jqueryui.com/support
[23-Nov-2012 09:58:16 UTC] http://blog.jqueryui.com/
[23-Nov-2012 09:58:16 UTC] http://jqueryui.com/about

These are really weird as they're not actual errors emitted by PHP, as those look like this:

[23-Nov-2012 07:54:46 UTC] PHP Notice:  Use of undefined constant vehicles in /var/lib/jenkins/.../docs/pages/dynamic-samples/category.php on line 53

So something somewhere is randomly writing a list of urls to sterr in WordPress, and doing so for every request (existing and non-existing pages) on all new websites. At least found these types of error log entires on all requests of:

• jqueryui.com
• api.jqueryui.com
• qunitjs.com
• api.qunitjs.com
• stage.learn.jquery.com
• learn.jquery.com

[Krinkle]

For those with access to our Splunk service:

Search sourcetype="apache_error" "PHP message: http://"

Piping it to timechart span=1h count tells me there are 5,000 to 20,000 new matches every hour which confirms that it is happening on all hits (except hits for the old websites and hits not served by php).

---

EDIT: I traced it back:

First match on November 13, 2012 at 7:00 AM:

[Krinkle]

Nailed it down to d78c2e3#L9R15

[Krinkle]

Confirmed. The filter has gone all the way down to zero new matches in the last 30 minutes: