Introduction

Have you ever wondered what exactly happens behind the scenes when you type a website's URL (e.g., www.chaicode.com) and press enter?

What role does the OSI model play in data transfer, and how do concepts like DNS resolution, the 3-way TCP handshake, the TLS handshake, HTTP/HTTPS, sessions etc. fit together to form a clear picture of the HTTP request-response cycle?

In this article, we will explore and touch on these concepts to get a clear overall view of the networking fundamentals involved in an HTTP request-response cycle.

1. Role of The Developers

Consider the following Scenario:

• Developers create an application, say, Chaicode.

• They host it (let's say) on a server with the IP 142.250.190.14 (assume).

• They purchase a domain www.chaicode.com from (let's say) GoDaddy.com.

• They then associate the IP with the domain in their GoDaddy’s account by configuring DNS record settings as well as authoritative name servers.

• GoDaddy as the registrar, automatically updates the TLD server to reflect that Godaddy’s servers are now the authoritative servers for chaicode.com

• They obtain a TLS digital certificate, signed by a trusted Certificate Authority (e.g., Let's Encrypt, DigiCert) so as to enable HTTPS.

• Once everything is in place, the application is released for the users. Users can then initiate the HTTP request-response cycle to interact and use the application over the network.

Before diving into, how the user request travels between the client and the server, let’s first get a brief on the following concepts (discussed in the following 3 sections):

• OSI model and its 7 layers— a conceptual framework, essential for understanding data transfer

• HTTP ( HyperText Transfer Protocol)

• Understanding HTTP : An Analogy

2. OSI (Open Systems Intercommunication) Model and Its 7 Layers

Application Layer

• This layer is closest to the user, providing an interface between the user and the network.

• It handles high-level protocols such as HTTP/HTTPS.

Presentation Layer

• This layer ensures that the data sent from the application layer is understandable by the receiving end.

• It handles data translation, compression, and ensures encryption.

  • Examples: ASCII, UTF-8, binary image formats.

Session Layer

• This layer ensures that the connection remains open for the duration of the interaction.

Transport Layer

• This layer manages the data packets, also called segments.

• 💡

  In case of TCP, there can be latency(delay) in the data being received, but unlike UDP, there will be no packet losses.

Network Layer

• This layer is responsible for adding and encapsulating headers consisting source and destination IP addresses.

• It is responsible for routing, i.e determines the best path for data transfer using routers and switches.

Data Link Layer

• This layer breaks the data from the Network layer into smaller, manageable frames.

  💡

  To locate the exact device in a network, IP address alone is not sufficient (since it is dynamic and can change), hence MAC addresses are needed as they remain constant

• Hence, along with the source & destination IP addresses, the data link layer adds its own headers, consisting MAC addresses to identify the sender (browser) and receiver (say, local router) within the local network.

• The local router then uses its routing table to direct data packets to their next destination.

Physical Layer

• Finally all the data, is offloaded on this layer and this is the layer that handles the actual transmission of raw bits (0s and 1s) over physical mediums like Ethernet or fiber optic cables.

3. HTTP (Hypertext Transfer Protocol)

• HTTP is a protocol for transferring hypertext documents(say, pages linked to each other using clickable hypertext) over the web.

• The term “Hypertext” includes, not just text but also clickable images, buttons, and other elements that can navigate users to different pages.

• HTTP Versions: HTTP/0.9, HTTP/1.0, HTTP/1.1, HTTP/2, HTTP/3.

• What changed from HTTP/1 to HTTP/1.1 ?

  • HTTP/1.1 allows multiple requests/responses to be sent over a single TCP connection, reducing overhead, i.e, there is no need of re-establishing TCP connection before each and every request.

• What changed from HTTP/1.1 to HTTP/2 ?

  • HTTP/2 supports data encryption, compression and provides multiplexing, allowing simultaneous requests over a single connection

    For example: On loading a webpage that requires HTML, CSS, JS, and multiple images— all resources are sent simultaneously in HTTP/2. The server responds with all the resources at once, each in separate streams, without waiting for others to finish, whereas in case of HTTP/1.1 each resource waits for the previous resource's request to get completed before it can be requested.

• HTTP/3 works on QUIC protocol that works on UDP

4. Understanding HTTP : An Analogy

A n a l o g y: Imagine you’re boarding a train. Every time you travel, there are certain set of rules you must follow— like carrying a valid ticket and an identity proof of acceptable format(for instance, a PAN card, and not your Class-1 ID card). However, simply stating the rules isn’t enough; the rules need to be enforced. That's where the TTE (Train Ticket Examiner) comes in, who checks both your ticket and your ID to ensure everything is in order.

Regardless of how many times you board the train, the TTE doesn’t retain any personal information about you. Every time, you are treated as a new passenger. His sole focus is on verifying your ticket and ID for that particular journey. Alongside him, you may also spot policemen on the train, who ensures the security of your journey— similar to how HTTPS secures data through TLS encryption.

Now, let’s think of this analogy in terms of HTTP.

• Passenger —> is like the request.

• The train —> represents the network.

• The set of rules —> is HTTP.

• The web server (such as Nginx) or the HTTP server (like Node.js’s HTTP module) —> plays the role of the TTE, by enforcing the HTTP protocols, parsing requests into their components (headers, body, Url, query params, method etc.).

• Just like the TTE, these servers doesn’t store any user specific information about past requests—each request is treated independently. This is why HTTP is called a stateless protocol.

Statelessness of the protocol keeps it lightweight, scalable, and efficient. Imagine if the TTE had to remember every single passenger’s details for every journey. That would create an enormous amount of overhead. Similarly, adding stateful management at the protocol level would complicate HTTP.

However, while HTTP doesn’t manage state, maintaining state is crucial from the applications’ perspective. That’s why state management happens at the application level, often through sessions(discussed below in detail) or tokens, ensuring that the information about user interactions is retained across multiple requests.

5. User Request Initiation and The Need For DNS

Now that we have a brief understanding on concepts like OSI model and HTTP(discussed in the previous 3 sections), we can now explore the journey of the HTTP request from this section onwards.

Consider the following Scenario:

• A user enters www.chaicode.com in the URL bar of their browser and presses enter.

• The browser attempts to resolve the domain to an IP address using local cache.

• If it’s unsuccessful in resolving from the local cache, this is where the role of DNS(Domain Name Server) resolution comes into picture.

DNS Resolution: What and Why

• DNS translates human-readable domain names (e.g., chaicode.com) into IP addresses (let’s assume, 142.250.190.14).

• It simplifies browsing by allowing users to access websites with simple easy-to-remember names instead of complex IP addresses.

Key DNS Records

How the domain www.chaicode.com is resolved to an IP Address(say, 142.250.190.14 )

• The browser, unable to resolve the IP from its local cache, now queries a DNS resolver (Provided by your ISP or public DNS like Google DNS or Cloudflare DNS).

• The resolver contacts the nearest of the 13 root name servers (SEE Map)and asks about the TLD server, responsible for managing the top level domains ( .com in this case).

  • Examples of TLD(Top level domains) include : .com, .org, .ai etc.

• Next, the TLD server directs it to the list of chaicode.com’s authoritative name servers(configured in DNS settings by the Developers in their Godaddy’s account).

• The resolver queries the first authoritative server (for eg: ns1.godaddy.com) from the list, provided by the TLD server earlier. The server provides the A Record (the IP address), which is sent back to the browser.

  💡

  In case the first authoritative server fails for some reason, the resolver will query the next one from the list(eg: ns2.godaddy.com), but the process will not be blocked.

• Once the domain is resolved, the browser prepares an HTTP request, for example:

    GET/index.html HTTP/2
    Host : www.chaicode.com
  

  💡

  NOTE: Even if the website uses HTTPS, the browser still prepares the initial request as HTTP/2 and not HTTPS/2. The key difference is that HTTPS is HTTP over a secure connection established using TLS (Transport Layer Security), but the request itself doesn't include "HTTPS/2" in the headers.

6. 3-Way TCP Handshake

• Before the prepared http request(in the previous section) can be sent to the server, a reliable connection between client and server is required. This is ensured by establishing a 3-way TCP handshake through the following 3 steps:

  1. Client sends a SYN packet to initiate the connection (It’s similar to asking: “Am I audible?”).

  2. Server responds with SYN-ACK, thereby acknowledging the synchronization request from the client and simultaneously agreeing to initiate the connection (It’s similar to saying: “Yes, you are audible. But am I also audible ?”).

  3. Client then finally replies with ACK, to acknowledge the server’s SYN and completes the 3-way TCP handshake (It’s similar to confirming: “Yes, you too are audible and we can now resume our conversation.”).

     • Note : Without the final ACK from the client, the server wouldn’t know if the client is ready to complete the handshake and establish connection.

7. TLS Handshake - For Enabling HTTP(s)

• The purpose of Transport Layer Security(TLS, previously called SSL) handshake is to setup a secure encryption, for data transfer, over the already established TCP connection

  • TLS Versions : TLS 1.0, TLS 1.1, TLS 1.2, and TLS 1.3

• It involves verifying each other's identities (via digital certificate) and agreeing upon encryption methods, between the client and the server

• Once the handshake is complete, both sides have a secure channel ready for communication.

• Client sends a client hello message, containing a random value and cipher suits(list of encryption algorithms) for the server to choose from

    ClientHello {
        CipherSuites: RSA_SHA_AES,
        Random: Client_Random_Value
        // More relevant info
    }
  

• Server responds back with the chosen cipher suit(say, RSA_SHA_AES), its own random value, and the digital certificate(consisting of a public key) signed by trusted Certificate Authority

    ServerHello {
        CipherSuite: RSA_SHA_AES, // chooses this one from the ones provided by client
        Random: Server_Random_Value
        // More relevant info
    }
  
    Certificate {
        Public Key: Public_Key
        Issuer: DigiCert // assume for example
        Signature: DigiCer's digital signature
    }
  

• Client has preloaded list of trusted CAs(Certificate Authorities) and thus verifies the identity using the signed digital Certificate

• Client creates another random value, called the pre-master secret and encrypts it using the public key from the certificate and RSA Algorithm, then sends it back to the server, where the server decrypts it using its private key. Since the encryption happens using public key and decryption happens using private key, this type of encryption is called Asymmetric encryption.

  💡

  RSA is also used when you generate SSH key pair for setting up your git-github authentication, where you set the public key in your Github account.

• At this stage, both the client and the server have their own random values, each other's random values and the pre-master secret, using which, along with the SHA hashing algorithm they both generate a shared session key.

• Finally to complete the handshake, client uses SHA to hash all the handshake messages(clientHello, serverHello etc.), to ensure integrity(that the message hasn't been tampered with, during transmission), includes this hash in its finished message and then encrypts the finished message using AES algorithm & sends the Client finished message to the server

• The server does the same process and sends the server finished message to the client to complete the TLS handshake.

• Since the AES encryption is done using the same shared session key, this encryption is called Symmetric encryption.

In short : HTTP + TLS = HTTPs

To explore 💡 : How is the TLS certificate obtained from trusted Certificate Authorities like DigiCert, Let’s Encrypt etc. and how does the browser/client recognize the trusted certificates ?

9. Processing The Request and Managing State Using Sessions

• Once the TLS handshake is complete, the request is finally then sent to the server, where it is handled by web server(listening on port 80) or an http server, which parses the request.

• The request is then processed by the app server(express-js) which processes it to execute the application's business logic.

  • NOTE : express is not just app server but does also play the role of http or web server(to serve static files using express.static())

• Often times, users create an account on the application, through which the state(of their activities on the application) is managed.

• Since http is stateless, a session can be used to store and manage the user-specific data on the server

• Server creates a sessionID for the generated session and usually encapsulates it in a cookie

• Once the http request is processed, a response is prepared, that has the response data(may be list of products corresponding to GET request), response message, http code(eg: 200), cookie(containing the sessionID) etc.

• This response is sent back to the client, which then stores the cookie

  

  • NOTE : Sharing cookie(s) can compromise your account.

• The client automatically includes the cookie(containing the sessionID) in subsequent requests to the server, and the server uses it to retrieve the session data, thereby managing the state of the user(eg: authenticating the user using sessionID)

• However, for the state or the user-data to persist, post the session is closed(say, after the user logs out of the application), it needs to be stored in a Database (eg: MongoDB).

• This gives us an approximate idea as to how the http request-response cycle works and what its journey looks like.

Final Thoughts

To summarize, in this article, we explored the journey of an HTTP request-response cycle— from user input to data transfer and response, simultaneously touching concepts like DNS, handshakes etc. Understanding these underlying processes helps in building the foundation for creating efficient, secure, and scalable web applications.