Setting up OAuth2 callbacks in Rails with HTTPS offloading on load balancers

“HTTPS everywhere” is not a luxury anymore. It is a necessity. Thankfully, obtaining an SSL certificate has become easier too, with initiatives such as Let’s Encrypt, GeoTrust, Positive SSL, StartSSL. Even cloud based services such as Cloudflare and Amazon AWS provide free SSL certificates to their customers.

####Here is setting some context to help the reader appreciate the discussion: We host our rails applications on Amazon AWS. We generally use three different environments - development, staging and production. Development environment is generally local to a developer while staging and production are hosted on the cloud. There is a minor difference in the way we configure our staging and production environments. Our staging environment typically contains a single machine instance hosting our application. This single instance is exposed to internet directly (has a public IP). On the other hand, our production environment typically contains a cluster of instances for the sake of horizontal scaling. These instances typically do not have a public IP and hence not exposed to internet directly. We put this cluster behind an internet-facing Elastic Load Balancer (ELB).

We use chef-solo to manage our cloud infrastructure as well as to deploy code to various environments.

#####The Problem Statement: For the sake of this discussion, we shall limit ourselves to configuring SSL certificates obtained from the two free providers, namely Let’s Encrypt and Amazon AWS.

Using Let’s Encrypt in a clustered setup is tricky, since you need to make one of the instances stateful, in the sense, one instance needs to be given the responsibility of obtaining and renewing SSL certificate from Let’s Encrypt. All other instances need to copy this certificate every time its renewed. This requirement unnecessarily complicates the setup and also takes away some amount of flexibility. Also, Let’s Encrypt does not issue wildcard certificates and the validity of a certificate is just 90 days

The certificates provisioned from the other provider, Amazon AWS, can only be installed on an ELB. Hence is best suited for our clustered setup, namely production. An added advantage is that Amazon can issue wildcard certificates. We could always add an ELB to our staging environment (even though we will never have more than one instance), but that costs extra money for no reason.

This leaves us with these options

We went ahead with this choice. Using chef to manage our setup came handy.

We first configured our Staging environment and everything worked as expected.

However, the same application, in production environment, started throwing CSRF detected Error whenever an OAuth2 callback happened. This was really strange. Our application integrated with two different OAuth providers, and the problem was consistent with both these providers.

#####What’s the issue?

The only difference between our Staging and Production setups was the ELB.

In production, we offloaded HTTPS at the ELB. Plain HTTP request would hit the NGINX web server, which in turn would reverse-proxy it to unicorn and rails.

CSRF detected was clearly an error emitting from the rails application. Not from NGINX, and not from the ELB.

A closer look would reveal that the rails application had no way to know if the callback was made on a http:// URL or a https:// URL, because it sees only HTTP (due to offloading). Was this the reason rails was unhappy?

OAuth2, by design, does not accept plain HTTP callbacks (unless it is to localhost).

####How do we move forward?

#####PoC to prove the theory

Just to confirm what we think is the cause, we enabled HTTPS on NGINX (like we did in our staging environment). This was in addition to HTTPS on the Load balancer. We reconfigured the Load Balancer to NOT offload HTTPS but forward the request as-is to NGINX.

What do we have now? The CSRF detected errors are gone. Application behaves just like it should.

This confirmed our theory.

But the question now is, how do we achieve our desired configuration of offloading HTTPS at the ELB ? Is it just not possible ?

The Solution

We have been using X-Forwarded-For header while reverse proxying to unicorn so that our rails application knows the client IP address (rather than the IP address of the Load Balancer). We need this for logging and tracking.

Could there be something on similar lines to tell the rails application that the request was not on HTTP but on HTTPS?

Sure there is. We had to set a header in our reverse proxy configuration:

X-Forwarded-Proto  to  https

For NGINX, we do it like this:

proxy_set_header X-Forwarded-Proto https;

Voila, Rails is happy and things are back to normal!

Details:

Csrf detected!

Rails bothers about SSL only at two places,

1. At environment config, force_ssl.
   
2. At external included Gem like Omniauth.
   

In Rails environment config.

config.force_ssl = true

This does the trick, but doesn’t seem like a good idea to enable this option in Rails because, we offload https at NGINX. For Rails, request came in http, so it does a permanent redirect to https, which ends in a infinite loop.

Our stack trace gave a clue that error might be inside omniauth gem.

As we dug inside the Gem and found out that Omniauth looks at these headers

lib/omniauth/strategy.rb#L493-L499

def ssl?
  request.env['HTTPS'] == 'on' ||
  request.env['HTTP_X_FORWARDED_SSL'] == 'on' ||
  request.env['HTTP_X_FORWARDED_SCHEME'] == 'https' ||
  (request.env['HTTP_X_FORWARDED_PROTO'] && request.env['HTTP_X_FORWARDED_PROTO'].split(',')[0] == 'https') ||
  request.env['rack.url_scheme'] == 'https'
end

This is where we found that setting up X_FORWARDED_PROTO to https should fix our problems.

Initially, this X_FORWARDED_PROTO was set to $scheme. Which will be http for production as https is offloaded at ELB.

Now, by setting X_FORWARDED_PROTO to https, we are making sure that redirects are happening on https.