Understanding HTTP – The internet’s communication protocol

HyperText Transfer Protocol is one of the fundamentals of the internet. Every web developer, even front-end developers should at least have a basic understanding of what HTTP is. It is the mechanism which enabled computers talk to each other over the internet. It defines the format in which messages are passed on the internet. It is absolutely necessary for every web developer.

Background

Okay, so the internet began as a government project of the US around the 1960s-80s and reached commercial adoption around the 1990s. The internet is a huge interconnected web of computers. Each computer is assigned a unique address known as an IP address (e.g. 182.179.188.131). Computers on a network identify other computers on the same network by their unique IP addresses.

So, we have a mesh of wires connecting different computers. These wires go through oceans, through the ground, over and under buildings in many cases. How can we ensure that the signal sent by one computer will reach the other computer even though there may be many twists and turns and possibly other blockages in the wire?

That’s where the Transmission Control Protocol (TCP) comes in. The TCP protocol is designed to ensure that messages (packets as we call them computer networks) are reliably sent from one computer to another over a network. TCP forms the backbone of most of the internet.

Now, TCP’s responsibility is to ensure that any packet sent by a computer reaches reliably at its target IP / computer. However, TCP does not define the structure of the message. The structure of the messages being passed on the internet are defined by HTTP.

Client-server architecture

To understand HTTP we must first understand client-server architecture. In a client-server architecture we have one system or computer that is the client. The client requests some information from the server. The server is a system or computer whose job is to respond to client’s requests with the relevant data.

HTTP client server architecture
Client-server architecture with HTTP

The HTTP mechanism is comprised of a request-response cycle. The client can send an HTTP request to the server and it’s the server’s job to respond to the request with relevant data. Most HTTP responses contain HTML.

The internet is simply a huge network of clients and servers. If you type alazierplace.com into your browser’s address bar and press enter, your browser is the client and it will send an HTTP request to my server. My server then responds to the HTTP request by sending the HTML of my website in the response.

Types of HTTP Requests

Every HTTP request is a simple text file which is formatted in a certain way. The formatting is very well defined and documented by the W3C in the HTTP specification. If you ever want to know why something works the way it does in HTTP, The HTTP specification is the best place to find out. It is somewhat cumbersome to navigate it but I learn something new every time I have to check a reference for something. So, RTFM.

The HTTP specification defines a list of request types that are used to request data from HTTP servers. These are formally known as HTTP methods or verbs. The method or verb signifies the intent of the client when making a request. Here are the most common HTTP methods and their conventional use cases:

  • GET: GET requests are the most common types of requests on the web. They simply retrieve data from a server. 99% of the time they are used to request HTML. Every time you visit any website in your browser, the browser sends a GET request for the content of the site. These requests do not have a request body. Any arbitrary information sent through GET requests has to be appended to the URL as a query string. This makes GET requests very insecure for sending sensitive information such as emails and passwords.
  • POST: POST requests are used to send arbitrary amounts of data to the server. The data is sent as the request body and not in the query string. By convention, POST requests are used when a new entity is to be created on the server from submitted data.
  • PUT: PUT requests are identical to POST requests in that they are used to submit data to the server. However, the convention is that PUT requests should be used when we have to completely replace some already existing entity on the server.
  • PATCH: PATCH requests are used to update some existing entity on the server. They submit data in the request body similar to POST and PUT requests.
  • DELETE: DELETE requests are similar to GET requests in that they have no body. They are the opposite of GET. While GET is used to retrieve data, DELETE requests are used to delete data form the server (no duh?).

These are some of the common HTTP headers used. There are others like OPTIONS, HEAD, TRACE. Details here.

Structure of HTTP Requests

The structure of an HTTP request is broken down into the following parts.

  • The request line
  • Zero or more headers
  • A blank line denoting the beginning of the body
  • An optional message body

The following is an example of an HTTP GET request your browser sends when you type google.com into the browser.

GET / HTTP/2
Host: www.google.com
authority: www.google.com
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
accept-encoding: gzip, deflate
accept-language: en-US,en;q=0.9
 

And here is an example of and HTTP POST request with data in the body.

POST /test HTTP/1.1
Host: foo.example
Content-Type: application/x-www-form-urlencoded
Content-Length: 27

field1=value1&field2=value2

There is a subtle difference between GET and POST which I will explain soon. As you can see, an HTTP request is nothing more than plain text with some specific formatting. Let’s dissect this file.

Request Line

The very first line in an HTTP request is know as the request line.

GET / HTTP/2

It comprises of 3 parts:

  • The very first is the HTTP method used. It can be GET, PUT, POST, OPTIONS or anyone of these values. The method defines the type of request being sent to the server and dictates what data can or cannot be passed with the request.
  • Second, is the request URI. / and /test in our examples. This defines what file on the server the client is requesting.
  • Third, is the HTTP protocol version.

HTTP Headers

After the request line, we have a series of key-value pairs separated by a :. These are known as HTTP headers and are case-insensitive. Headers are how additional information is sent to the server. The complete list of compatible header fields can be found here. HTTP headers are well defined and can only contain specific values. It is possible to define custom headers using the X- syntax, e.g.

X-Powered-By: PHP/5.2.17

However it is not very widely used. Ever wondered how browser cookies are sent along with the request? A cookies header. Headers define meta information about the request, like what is the content-type of the request body. What are the acceptable response types etc.

Request Body

The last and final part of a request is the body. After the headers, there is a blank line which signifies that the headers are finished and the next data is the request body. The request body is used to send arbitrary data to the server. It could be JSON, simple text or even binary code.

The body is only enabled on certain HTTP methods. GET requests don’t have a body. POST, PUT and PATCH requests do. The type of data in the body is defined by the Content-Type header. If you’re sending JSON data to the server then the content-type would be application/json.

Structure of HTTP Responses

As stated, HTTP requests and responses are just plain text files. So the structure of an HTTP response is similar to a request with some differences. They also consist of 3 parts.

  • A status line
  • Zero or more headers
  • A blank line denoting the beginning of the body
  • An optional message body

Below is an example of an HTTP response.

HTTP/1.1 200 OK
Server: nginx/1.14.0 (Ubuntu)
Date: Sun, 19 May 2019 12:50:50 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Sun, 19 May 2019 12:49:34 GMT
Connection: keep-alive
ETag: "5ce150de-264"
Accept-Ranges: bytes

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Let’s break it down shall we?

Status Line

The status line is the first thing in an HTTP response

HTTP/1.1 200 OK

In order, it has 2 parts:

  • The HTTP version
  • The HTTP status code and its textual meaning

In the above example, HTTP/1.1 is the version and 200 OK is the status code and text.

Response Headers

Just like with the request, HTTP responses contain headers which contain metadata about the response. These are simply key-value pairs separated by a semi colon :.

We can see that the Content-Type header in the above example contains the value text/html denoting that is will contain HTML.

Response Body

The response body is such that it contains the actual response content sent by the server. Almost every HTTP response has a response body. The content of the response can be virtually anything. HTML, JSON, XML, you name it. However, mostly the HTTP responses contain HTML or JSON.

In the above example, the response body starts after the first blank and the response body contains the actual HTML that will be sent to the browser.

HTTP Status Codes

One of the most popular and useful things about HTTP is its use of status codes. Status code indicate the status of the requested resource or document. The status code is simply the number in the status line of a response and is always in the hundreds range, e.g. 200, 302, 404 etc.

Here are some of the most popular status codes. You might’ve seem some of them on the web.

  • 200 OK: Indicates a successful request and returns the response.
  • 301 Moved Permanently: The resource has been moved to a new location. Mostly used to redirect to the resource.
  • 400 Bad Request: Used when there is an error in the HTTP request.
  • 403 Forbidden: Indicates that the client does not have the necessary authorization to access the resource.
  • 404 Not Found: The resource is not found.
  • 418 I’m a teapot: Used when the server refuses to brew coffee because it’s a teapot. (Yes, this is real).
  • 500 Internal Server Error: Raised when the web server encounters and error that is not handled properly.

And the full list of codes can be found here.

Security

Now, there is a lot to be said about security in HTTP. HTTP does provide basic access authentication. What that means is that the client would have to supply a username and a password to get access to the resource on the server. But that is just a rudimentary mechanism and we need better security in modern apps.

HTTP was created in such a way that makes it easy to read by humans (i.e. uses plain text and formatting). However, this introduces some complications. By default HTTP does not encode or encrypt the message or headers in anyway. So, if a request is sent to a server through a client, that request goes through a network of routers and switches and servers. Any one of those servers can read the content or message of the request or response. So if I send the username and password of my bank account through an HTTP post request, any router or server that comes between the bank’s server and my computer can read my credentials. There are ways to overcome this which I will talk about below.

A Stateless protocol. HTTP is designed as a stateless protocol. What this means is that in a pure HTTP request/response cycle, the client and the server do not retain any information about each other. It is not the responsibility of the HTTP protocol itself to recognize the authenticity of the client or the server. While this decision has decreased the implementation complexity of the protocol, I believe it introduces chances of attacks such as phishing attacks and cross-site request forgery attacks. So web developers have developed techniques to ensure such security. In my opinions, such security measures should be built-in to the protocol.

Best Practices

Here I will list down what I think are best practices when using HTTP. Whether it be spinning up a traditional template based app, creating an API or creating a microservices based architecture, here is what I think we should do.

  1. Use proper HTTP status codes: HTTP status codes were defined for a reason, and they form a standard set of rules that most systems on the web follow. So we should use proper HTTP codes in our APIs and servers. If it’s a success return 200, if it’s a redirect return 302 and if it’s not there, return 404. This enables us to build more robust clients and servers.
  2. Always use SSL/TSL TLS: As I said, HTTP traffic is plain text so anyone or any system can read it. To prevent this we should always use an SSL certificate to ensure that the request and response are completely encrypted and only readable to the intended client and server.
  3. Use CSRF security: HTTP being stateless, we can’t ensure the validity of clients on servers. So we should use a CSRF token or similar mechanism to prevent cross-site request forgery attacks.
  4. Use JWTs for secure APIs: When building APIs the best way to grant secure access to clients are through JSON web tokens. JWTs should be used with a secure algorithm and secret key. Also, they should be provided in the Authorization header instead of browser cookies.