ELB + nginx PUT requests cause 411 (was previously CSRF protection and DoS safety for data-remote links)

[raggi]

Updates:

Just to circle back, so that it's clear for everyone, this was a storm of crap that took us longer to track down, and no code changes are actually required. Sorry for the noise.

• Amazon ELB was the thing removing Content-Length: 0 from requests. It looked like a browser bug, but ELB was actually rewriting the requests. (Sigh) (https://forums.aws.amazon.com/thread.jspa?messageID=232444)
• There's something going on in a preproduction app that we have which appears to be preventing the CSRF protection from erroring when the CSRF headers are missing. I think this is unique to our app, we'll circle back on the rails side if it's not unique to something we've done.
• We had been running an aged version of rails.js for some reason, and in that version, the CSRF headers were missing - totally our issue.

Thanks for pushing back, and sorry again for the noise.

For the record, if someone is suffering 411s from nginx behind Amazon ELB, the cause is the ELB issue linked above. It is possible to hack in a solution of simply sending some body data, but we're actually considering moving away from ELB given that the "prioritized fix" has been open since Oct. '09.

---

Original report:

Some browsers incorrectly don't set a Content-Length when sending empty PUT requests. This affects jquery 1.5 and nginx. The Content-Length field is important in order to protect against sloworis style attacks, and this is why nginx returns a 411 in these cases.

There are several solutions to the problem, the most pertinent (IMO) is the one in this pull request, as it also allows for CSRF protection to be turned on for XHR PUT on the server side, which I would recommend...

Other solutions are:

raggi@d6271c5

and

http://blog.standalonecode.com/?p=16

One of my guys also just did this:

      } else {
        method = element.data('method');
        url = element.attr('href');
        data = null;
        if (method.toUpperCase() === 'PUT') {
          method = "POST";
          data.push({name: '_method', value: 'PUT'});
        }

But that makes me itch, as it's a workaround in the wrong direction and doesn't address other issues in this area.

Cheers!

[mislav]

I don't get this. More explanation pls?

The change in the pull req mostly just moves code around. I like the one-line fix that you mentioned better.

[shwoodard]

LOL...I make raggi itch and you like my solution. What to do, what to do...???

[shwoodard]

oh...nm...you like the one lone fix

[raggi]

mislav: well, there's an additional issue that rails is ignoring CSRF protection for XHR PUTs which is a security problem. I want to speak to Koz to verify this, though.

I can't produce you a test case because you don't have any integration tests against a webserver that's http/1.1 strict.

As stated, try using data-remote and data-method=PUT against a site running behind nginx, when using jquery-1.5, and you will see the 411 status code as a response. I think this affects most browsers, they are using a combo of keep-alive and PUT without sending a Content-Length, which is technically invalid, as per the http spec, and would make the server vulnerable to slow clients for these routes.

As for 'mostly moving code around', no, I avoided having to rewrite the form building in this fix, as opposed to rebuilding the data segment in other ways.

[mislav]

I don't think that you are wrong. The "Content-Length required" error response is very real and I've seen it in Webrick as well as nginx.

What I did request, and you didn't deliver, is step-by-step breakdown of the changes you did to make "Content-Length: 0" get sent. For instance, you talk about XHR request being affected by this, but your changes mostly concentrate around building the hidden form which has nothing to do with XHR. This is what confuses me

[raggi]

The semantic changes involve the code path that uses $.ajax, not the path that calls form.submit()

[mislav]

I've noticed. You still didn't explain. How does generating a hidden form for "data-remote" links help Ajax?

[raggi]

I generated the form simply to make the contents consistent, the same effect could be done a different way, but this was the 4 LoC version.

[raggi]

Hmm, for some reason, we had this missing: https://github.com/rails/jquery-ujs/blob/master/src/rails.js#L9-15

So maybe that part isn't necessary. I can check at some point when time is available.

[raggi]

There is definitely something else going on in the application where this came up with regard to CSRF, but it's not relevant here, and this patch is invalidated by code that was not present in our environment (we had an old rails.js). I'm going to close this and reopen one with the one-liner after i've had a chance to verify that works.

[raggi]

Just to circle back, so that it's clear for everyone, this was a storm of crap that took us longer to track down, and no code changes are actually required. Sorry for the noise.

• Amazon ELB was the thing removing Content-Length: 0 from requests. It looked like a browser bug, but ELB was actually rewriting the requests. (Sigh) (https://forums.aws.amazon.com/thread.jspa?messageID=232444)
• There's something going on in a preproduction app that we have which appears to be preventing the CSRF protection from erroring when the CSRF headers are missing. I think this is unique to our app, we'll circle back on the rails side if it's not unique to something we've done.
• We had been running an aged version of rails.js for some reason, and in that version, the CSRF headers were missing - totally our issue.

Thanks for pushing back, and sorry again for the noise.

For the record, if someone is suffering 411s from nginx behind Amazon ELB, the cause is the ELB issue linked above. It is possible to hack in a solution of simply sending some body data, but we're actually considering moving away from ELB given that the "prioritized fix" has been open since Oct. '09.

I've updated the original report as this is already coming up high in search results on the topic, and as such, may be valuable to someone in the future.