DataGeek: Setting Up a Test Reverse Proxy/SSL Accelerator Environment (or for that matter, a budget production one)

In a previous post I talked about dealing with WCF services behind an F5 BIG-IP box (http://www.f5.com/products/big-ip/) that provides load balancing and SSL offloading. I’ve actually found and developed solutions to the problems but needed to be able to properly test them.
Given that, according to our operations people, BIG-IP boxes start at 5 figures and rapidly go up, they seemed unwilling to purchase one for me. Additionally, for security issues, and to remain in compliance with our SAS-70, I don’t have access to our production environment for testing.
First an explanation as to how our network is configured. There are a couple of approaches to load balancing. One common approach is to have the user browse to https://www.mywebsite.com. This actually points to a load balancer. Using whatever algorithm is configured, the load balancer then chooses the actual web server to use and redirects the user to it, such as https://www1.mywebsite.com.
In this case, the browser then resends the request to the server it was redirected to and interacts directly with it, no longer interacting with the load balancer. The diagram below outlines this approach to load balancing.

There are a couple of disadvantages to this:

All of the web servers must be directly accessible to the internet.
Each of the web servers needs to process SSL as well as performing the application functions.
SSL certificates must be managed on each of the web servers.

The approach we use is a little different. The diagram below outlines how our network is configured.

In this approach, the web client interacts only with the F5 reverse proxy. When the F5 receives a request, it decrypts the SSL and forwards the request to one of the load-balanced web servers as an unencrypted http request. The web server in turn returns the response to the F5 which encrypts it and sends it back to the web client. This has a couple of advantages:

The web servers are protected behind the firewall.
The SSL processing is handled by SSL hardware accelerators and the web servers can use their resources for the problem at hand.
The encryption certificates are managed in a single place.

So how to create a valid test environment without spending a fortune? The answer came with the Application Request Routing (ARR) module for IIS7 (http://www.iis.net/download/ApplicationRequestRouting). This is add-on module available from Microsoft for IIS that provides a host of features, including the ability to act as a reverse proxy and offload SSL processing.
Although you could install ARR on the same test server as your web service, I would recommend creating a dedicated server (pretty easy if you’re in a virtual environment using Hyper-V or VMWare). Once you’ve built your server (my setup is Win2k8 R2, IIS 7.5 and ARR), you’ll need to perform the following tasks:

Create an SSL certificate for your website.
Assign the certificate to a binding in IIS.
Setup the hosts file on your test client to point to your reverse proxy.
Setup the certificate as a trusted certificate on your test client.
Setup the hosts file on your proxy server to point to your web servers.
Configure ARR to forward your requests to the web servers.

Create an SSL Certificate for your Website

To do this you actually need to create two certificates. The first is a Certificate Authority (CA) certificate. Certificates issued by a CA are trusted by any client that trusts the CA. Commercial websites use certificates issued by commonly trusted CAs such as Verisign. Certificates issued by these commercial CAs are trusted by most client OSs and browsers.
In this case we don’t want or need to go through the hassle and cost getting a commercial certificate so we’ll in essence create our own CA and issue our own certificates. We’ll create this certificate on the proxy server using the makecert utility. This utility is installed with the Windows SDK which is installed with Visual Studio. If, like me, you’ve created a clean proxy server, you’ll want to install the Windows SDK 7.0, which is available here:
http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=3138
Once installed, the makecert.exe file can be found in the following folder (if using the default install):
C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin
To create your CA certificate, open Command Prompt as Administrator. Navigate to a folder in which you would like to save the certificate files and enter the following command line:

"C:\Program Files\Microsoft SDKs\Windows\v7.0\Bin\makecert.exe" -sv MyCACert.pvk -cy authority -r MyCACert.cer -a sha1 -n "CN=My Certification Authority" -ss my -sr localmachine

This will result in two consecutive prompts, one asking you to create a private key password and the other to enter it.

Once completed, two files will be created in the current folder, the certificate, MyCACert.cer, and the private key, MyCACert.pvk. The certificate has also been added to the local machine’s certificate store. You can view this by opening up the certificate manager for the local machine. Do this by typing mmc.exe at the command prompt or in the Start Menu/Search box. Click on File/Add or Remove Snap-ins. Choose the Certificates Snap-in and click Add. You will be prompted to choose which certificates to manage. Choose Computer Account. When asked to choose a computer, choose the Local computer (the default) and click Finish and OK.
In the certificates tree under Certificates (Local Computer)/Personal/Certificates you should see a My Certification Authority certificate. No need to do anything with it. Just checking that it was properly created.
Next we need to create the actual SSL certificate. To do this, go back to the command prompt and type:

"C:\Program Files\Microsoft SDKs\Windows\v7.0\Bin\makecert" -iv MyCACert.pvk -ic MyCACert.cer -cy end -pe -n CN="*.mydomain.com" -eku 1.3.6.1.5.5.7.3.1 -ss my -sr localmachine -sky exchange -sp "Microsoft RSA SChannel Cryptographic Provider" -sy 12

During this process you will be prompted for the private key password we entered for the CA certificate.

Note that we’re now passing the domain name “*.mydomain.com” that this certificate is assigned to. By prefacing it with an asterisk, we’re creating a wildcard certificate that can be used by any subdomain under “mydomain.com”. Note that it can only have a single wildcard segment. i.e. it can protect “www.mydomain.com” but not “www.server1.mydomain.com”. Also note that we’re referencing the MyCACert.pfk and MyCACert.cer certificate files to sign it with our previously created CA certificate.
If you go back to the mmc certificate manager you’ll now see a new certificate issued to “*.mydomain.com” under the Personal/Certificates folder.
We’re now done creating certificates. Time to use them.

Assign a Certificate to a Binding in IIS

Open up IIS Manager, click on the Default Web Site and click on Bindings under the right Actions panel.

In the Site Bindings window, click Add. In the Add Site Binding dialog, choose https in the Type dropdown, the *.mydomain.com certificate in the SSL certificate dropdown and type in the host name you’d like to assign it to.

Click OK and you’ve now created a binding to you’re service host with a valid SSL certificate assigned to it.

Setup the Hosts File on Your Test Client to Point to Your Reverse Proxy

Now that you’ve created a site binding with a host name you need a way to point to the proxy using the host name. Under the assumption that you, like myself, do not have ready access to the company DNS server, the next best way is to add it to your local hosts file.
Navigate to C:\Windows\System32\drivers\etc and open the hosts file in the text editor of your choice and add the following line:

192.168.3.157 pingservice.mydomain.com

Where the IP address is the IP of the proxy server. Note that I found I needed to explicitly give myself full control permissions on the file to save it.
Just to make sure it’s working, open the browser of your choice and browse to:

https://pingservice.mydomain.com

You should see the following page or your browsers equivalent version:

What this is telling us is that the host name is working and the certificate is installed and working, but that it has been issued by a CA that we don’t trust.

Setup the Certificate as Trusted on the Client

To get the client to trust the certificate, we need to import it into our local certificate store. Start by copying the MyCACert.cer file to your client. Next open up the certificate manager in MMC (same as on the server). In the certificate tree, navigate to Certificates (Local Computer)/Trusted Root Certification Authorities/Certificates, right click and choose All Tasks/Import.

Follow through the wizard, choosing the MyCACert.cer file when prompted for the certificate to import (click through the other dialogs leaving the defaults).

Now return to the browser and refresh the page. You should no longer get the certificate warning. (If you do you may want to close and reopen your browser).

Configure ARR to Forward Your Requests to the Web Servers

Now that everything is pointed in the right directions, it’s time to setup ARR to route the requests. On the proxy server, open up IIS Manager. With ARR installed, you’ll note that there’s a new node under the server, Server Farms.

Right click on the Server Farms and click on Create Server Farm. Enter the name of your server farm. It doesn’t have to be the host name, but that makes keeping track of them easier.

On the next dialog you’ll enter the target servers. Enter the host name or IP address of each server, clicking the Add button each time. The host name here can simply be the Windows server name. It doesn’t need to have any relation to the host name the client is requesting.

When you click Finish, you’ll be asked if you want to automatically create a URL Rewrite rule. Click Yes.

Once back in IIS Manager, expand out the Server Farms node and you should see the server farm you just created. Click on it and you’ll see a few things you can configure in the center pane. Double-click on the Routing Rules icon.

Double check that the Use URL Rewrite to inspect incoming requests and Enable SSL offloading check boxes are check and click the URL Rewrite link in the Actions pane.

On this page you’ll see the rewrite rule that the server farm creation wizard created. With the default rule, all requests to this server, regardless of host name, port or protocol will be routed to the server farm. To restrict the rule to the host name we want, click on Add.. in the Actions pane under the Conditions section.

In the Add Condition dialog, type {HTTP_HOST} in the Condition input text box, and the host name in the Pattern text box. Clicking OK will add this condition to the rule and restrict the rule to forward only requests with the pingservice.mydomain.com host name.

Finally, to enforce HTTPS on the proxy server, click on the Default Web Site in the server tree and double-click on the SSL Settings icon under the IIS group.

In the SSL Settings page, check the Require SSL check box and click Apply.

One last and very important note. The host name sent in the request to the web server is NOT the server name (server1.pingservice.mydomain.com or server2.pingservice.mydomain.com) but the original host name, pingservice.mydomain.com. As such, unless you’re routing to the Default Web Site, you’ll need to setup a http binding with a host name of pingservice.mydomain.com.
At this point you’re all set to test your service behind a reverse proxy.