The HTTP Server API

Over the course of the next few posts we’re going to take a look at a specific API in Windows, namely the HTTP Server API and we’ll explore how it can be used with a view to making it available to Python.

In this post we’ll introduce it by looking at the .NET HttpListener class, and we’ll write a simple C# program to  demonstrate it. Then we’ll look at the underlying HTTP Server API, and write a simple C++ program with almost identical functionality, but implemented in terms of the HTTP Server API.

The .NET HttpListener class

First, let’s introduce the API at a point where you may have used it without knowing. If you’re a .NET programmer then you may be aware of System.Net.HttpListener. This class, which is used by OWIN self-host, allows multiple HTTP servers to bind to a single port.

Here is an example C# program that uses HttpListener. It reads a URL from the command line and uses HttpListener to listen for incoming HTTP requests whose prefix is that URL. It loops, reading requests and responding to them with a simple message, until it encounters a DELETE verb, at which point it terminates.

using System;
using System.Net;
using System.Text;

namespace ListenerExample
{
    class Program
    {
        static void Main(string[] args)
        {
            // Create and start a listener.
            var listener = new HttpListener();
            var url = (args.Length > 0) ? args[0] : "http://localhost:9000/api/cs/";
            listener.Prefixes.Add(url);
            listener.Start();

            // Announce that it's running.
            Console.WriteLine("Listening. Please submit requests to: {0}", url);

            while (true)
            {
                // Wait for a request.
                var context = listener.GetContext();
                var request = context.Request;

                // Display some information about the request.
                Console.WriteLine("Full URL: {0}", request.Url.OriginalString);
                Console.WriteLine("    Path: {0}", request.Url.PathAndQuery);

                // Break from the loop if it's the poison pill (a DELETE request).
                if (request.HttpMethod == "DELETE")
                {
                    Console.WriteLine("Asked to stop.");
                    break;
                }

                // Send a response.
                var response = context.Response;
                string responseString = "Hello from C#";
                byte[] buffer = Encoding.UTF8.GetBytes(responseString);
                response.ContentLength64 = buffer.Length;
                response.ContentType = "text/html";
                var output = response.OutputStream;
                output.Write(buffer, 0, buffer.Length);
                output.Close();
                response.Close();
            }

            // Stop listening.
            listener.Stop();
        }
    }
}

Here it is listening for requests on http://localhost:9000/api/endpoint1/.

C:> ListenerExample http://localhost:9000/api/endpoint1/
Listening. Please submit requests to: http://localhost:9000/api/endpoint1/

If we send it a request with cURL then it responds, but only if the request’s URL starts with the URL that was supplied on the command line.

C:> curl http://localhost:9000/api/endpoint1/
Hello from C#

If we use a different prefix then we get a 404 error.

C:> curl http://localhost:9000/api/endpoint2/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Not Found</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Not Found</h2>
<hr><p>HTTP Error 404. The requested resource is not found.</p>
</BODY></HTML>

If we start another instance of the program listening on the same port but use a different URL prefix then it will work. For example, here it is listening to http://localhost:9000/api/endpoint2/.

C:> ListenerExample http://localhost:9000/api/endpoint2/
Listening. Please submit requests to: http://localhost:9000/api/endpoint2/

Now, because one instance of the program is listening on http://localhost:9000/api/endpoint1/ and the other is listening on http://localhost:9000/api/endpoint2/, we get a response from both endpoints.

C:> http://localhost:9000/api/endpoint1/
Hello from C#
C:> curl http://localhost:9000/api/endpoint2/
Hello from C#

The URL prefixes have to be unique, otherwise it will complain. For example, attempting to start the program listening on http://localhost:9000/api/endpoint1/ when another instance is already listening on the same endpoint will cause it to fail.

C:> ListenerExample http://localhost:9000/api/endpoint1/

Unhandled Exception: System.Net.HttpListenerException: Failed to listen on prefix 'http://localhost:9000/api/endpoint1/' because it conflicts with an existing registration on the machine.
 at System.Net.HttpListener.AddAllPrefixes()
 at System.Net.HttpListener.Start()
 at ListenerExample.Program.Main(String[] args) in C:\Users\Rod\OneDrive\Projects\HttpServer\ListenerExample\Program.cs:line 15

Similarly, if something is already listening on the port that doesn’t use the underlying API used by HttpListener, then it will fail. For example, let’s use Python to start an HTTP server listening on localhost:9000.

C:> python -m http.server --bind localhost 9000
Serving HTTP on 127.0.0.1 port 9000

Now that the port is in use, the program fails to start as it can’t bind to the port.

C:> ListenerExample http://localhost:9000/api/endpoint1/

Unhandled Exception: System.Net.HttpListenerException: The process cannot access the file because it is being used by another process
 at System.Net.HttpListener.AddAllPrefixes()
 at System.Net.HttpListener.Start()
 at ListenerExample.Program.Main(String[] args) in C:\Users\Rod\OneDrive\Projects\HttpServer\ListenerExample\Program.cs:line 15

This is a different error. Here, the Python HTTP server has bound to the port, so our program which uses HttpListener can’t bind to it.

If all goes well then the program will run forever, until it receives a DELETE request, at which point it releases its resources and stops. You can send a DELETE request with cURL as follows:

C:> curl -X DELETE http://localhost:9000/api/endpoint1/
curl: (56) Recv failure: Connection was reset

What’s going on?

We’ve established that programs that use HttpListener can co-exist on the same port as long as they all use HttpListener. Normally, this isn’t possible with sockets, so what is going on here?

Let’s add the -D flag to cURL to see the response headers.

C:> curl -D - http://localhost:9000/api/endpoint1/
HTTP/1.1 200 OK
Content-Length: 13
Content-Type: text/html
Server: Microsoft-HTTPAPI/2.0
Date: Sun, 22 May 2016 16:25:40 GMT

Hello from C#

The clue is in the “Server:” header which contains “Microsoft-HTTPAPI/2.0”.

Searching the .NET reference source for HttpListener reveals calls to functions such as HttpCreateRequestQueue() and HttpReceiveHttpRequest(). These functions are declared in UnsafeNativeMethods.cs, so clearly they’re not implemented in C#.

Further investigation reveals that they’re part of the Windows HTTP Server API.

The HTTP Server API

Several years ago, Microsoft added the HTTP Server API into Windows. It exists in XP and Server 2003, so it has been around for a long time. There are two versions of the API, and as HttpListener uses the second version, let’s do the same. The API itself runs in kernel mode, as shown in this diagram.

We already looked at a C# program that uses the HTTP Server API indirectly (via HttpListener), so let’s see what the C++ equivalent would look like.

#ifndef UNICODE
#define UNICODE
#endif

#ifndef _WIN32_WINNT
#define _WIN32_WINNT 0x0600
#endif

#ifndef WIN32_LEAN_AND_MEAN
#define WIN32_LEAN_AND_MEAN
#endif

#include <windows.h>
#include <http.h>
#include <stdio.h>
#include <stdlib.h>

#pragma comment(lib, "httpapi.lib")

int wmain(int argc, wchar_t **argv)
{
	// Initialize the API.
	ULONG result = 0;
	HTTPAPI_VERSION version = HTTPAPI_VERSION_2;
	result = HttpInitialize(version, HTTP_INITIALIZE_SERVER, 0);

	// Create server session.
	HTTP_SERVER_SESSION_ID serverSessionId;
	result = HttpCreateServerSession(version, &serverSessionId, 0);

	// Create URL group.
	HTTP_URL_GROUP_ID groupId;
	result = HttpCreateUrlGroup(serverSessionId, &groupId, 0);

	// Create request queue.
	HANDLE requestQueueHandle;
	result = HttpCreateRequestQueue(version, NULL, NULL, 0, &requestQueueHandle);

	// Attach request queue to URL group.
	HTTP_BINDING_INFO info;
	info.Flags.Present = 1;
	info.RequestQueueHandle = requestQueueHandle;
	result = HttpSetUrlGroupProperty(groupId, HttpServerBindingProperty, &info, sizeof(info));

	// Add URLs to URL group.
	PCWSTR url = (argc == 2) ? argv[1] : L"http://localhost:9000/api/cpp/";
	result = HttpAddUrlToUrlGroup(groupId, url, 0, 0);

	// Announce that it is running.
	wprintf(L"Listening. Please submit requests to: %s\n", url);

	for (;;)
	{
		// Wait for a request.
		HTTP_REQUEST_ID requestId = 0;
		HTTP_SET_NULL_ID(&requestId);
		int bufferSize = 4096;
		int requestSize = sizeof(HTTP_REQUEST) + bufferSize;
		BYTE *buffer = new BYTE[requestSize];
		PHTTP_REQUEST pRequest = (PHTTP_REQUEST)buffer;
		RtlZeroMemory(buffer, requestSize);
		ULONG bytesReturned;
		result = HttpReceiveHttpRequest(
			requestQueueHandle,
			requestId,
			HTTP_RECEIVE_REQUEST_FLAG_COPY_BODY,
			pRequest,
			requestSize,
			&bytesReturned,
			NULL
		);

		// Display some information about the request.
		wprintf(L"Full URL: %ws\n", pRequest->CookedUrl.pFullUrl);
		wprintf(L"    Path: %ws\n", pRequest->CookedUrl.pAbsPath);

		// Break from the loop if it's the poison pill (a DELETE request).
		if (pRequest->Verb == HttpVerbDELETE)
		{
			wprintf(L"Asked to stop.\n");
			break;
		}

		// Respond to the request.
		HTTP_RESPONSE response;
		RtlZeroMemory(&response, sizeof(response));
		response.StatusCode = 200;
		response.pReason = "OK";
		response.ReasonLength = (USHORT)strlen(response.pReason);

		// Add a header to the response.
		response.Headers.KnownHeaders[HttpHeaderContentType].pRawValue = "text/html";
		response.Headers.KnownHeaders[HttpHeaderContentType].RawValueLength = (USHORT)strlen(response.Headers.KnownHeaders[HttpHeaderContentType].pRawValue);

		// Add an entity chunk to the response.
		PSTR pEntityString = "Hello from C++";
		HTTP_DATA_CHUNK dataChunk;
		dataChunk.DataChunkType = HttpDataChunkFromMemory;
		dataChunk.FromMemory.pBuffer = pEntityString;
		dataChunk.FromMemory.BufferLength = (ULONG)strlen(pEntityString);
		response.EntityChunkCount = 1;
		response.pEntityChunks = &dataChunk;

		result = HttpSendHttpResponse(
			requestQueueHandle,
			pRequest->RequestId,
			0,
			&response,
			NULL,
			NULL,	// &bytesSent (optional)
			NULL,
			0,
			NULL,
			NULL
		);

		delete buffer;
	}

	// Remove URLs from URL group.
	result = HttpRemoveUrlFromUrlGroup(groupId, url, 0);

	// Detach the request queue from the URL group.
	info.Flags.Present = 0;
	info.RequestQueueHandle = NULL;
	result = HttpSetUrlGroupProperty(groupId, HttpServerBindingProperty, &info, sizeof(info));

	// Shut down the request queue.
	result = HttpShutdownRequestQueue(requestQueueHandle);

	// Close down the API.
	result = HttpTerminate(HTTP_INITIALIZE_SERVER, NULL);

	return 0;
}

If we run this, it behaves in much the same way as its C# equivalent. And when we invoke it with curl -D, we see that the “Server:” header is set to “Microsoft-HTTPAPI/2.0” as before.

C:> curl -sD - http://localhost:9000/api/endpoint1/
HTTP/1.1 200 OK
Content-Type: text/html
Server: Microsoft-HTTPAPI/2.0
Date: Sun, 22 May 2016 18:10:59 GMT
Content-Length: 14

Hello from C++

Similarly, if we run another instance, or run it alongside its C# equivalent, then they can all listen on the same port as long as their URL prefixes are unique.

Summary

This has been a quick introduction to the .NET HttpListener and the underlying Windows HTTP Server API, presented in the form of two simple, synchronous servers that appear to have identical behaviour. In the next post in this series we’ll look at how we might go about using the HTTP Server API from Python.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s