TLS (Transport Layer Security) and the TLS 1.3 Handshake

1. Overview

A. Definition and Background

TLS (Transport Layer Security) is a cryptographic protocol that provides confidentiality, integrity, and authentication between two communicating parties on top of a reliable transport layer such as TCP; it is the standardized, evolved successor to Netscape's SSL (Secure Sockets Layer) as taken up by the IETF. In short, TLS lets a party "verify that the peer really is that server (authentication) using a public-key certificate, exchange a session key safely in that process to encrypt subsequent data (confidentiality), and detect any tampering of messages (integrity)" — it is a security layer that sits between applications and the transport layer. Today the 'S' in HTTPS — the encryption of web, API, mail, and VPN traffic — essentially all runs on TLS.

The fundamental reason TLS emerged is that TCP/IP, the foundational protocol of the Internet, was designed assuming plaintext transmission. On the early web, login passwords and credit-card numbers flowed across the network in the clear, and any man-in-the-middle who intercepted the packets could read them directly. Netscape released SSL 2.0 in 1994, but it had so many design flaws that it was soon replaced by SSL 3.0 (1996); in 1999 the IETF standardized this as TLS 1.0 (RFC 2246), neutralizing the trademark and governance. After TLS 1.1 (2006) and TLS 1.2 (2008, RFC 5246), a decade later came TLS 1.3 (2018, RFC 8446). Version 1.3 is less a simple version bump than a redesign that fundamentally stripped away the accumulated vulnerabilities (BEAST, POODLE, downgrade attacks) and the slow handshake.

B. Necessity

The necessity of TLS is explained by the three pillars of security — confidentiality, integrity, and authentication (the C and I of CIA, plus authentication). First, confidentiality prevents eavesdropping on public networks. On physically shared media such as public Wi-Fi, without TLS a session cookie or authentication token is stolen as-is and the account is hijacked. Second, integrity detects a man-in-the-middle tampering with a response body or an in-transit amount. Third, authentication guarantees via a certificate that the peer the user connected to really is "that bank's server," blocking connections to phishing or impersonating servers.

In addition, TLS is necessary as a foundational control for regulation and compliance. The Personal Information Protection Act, [[isms-p]], and PCI-DSS explicitly require encryption in transit, and the browser industry has effectively mandated universal encryption (HTTPS Everywhere) by displaying "Not Secure" warnings on HTTP sites and allowing HTTP/2 and HTTP/3 only over TLS. In Korea too, e-government and financial services are guided to allow only TLS 1.2 or higher and to disable the weak SSL 3.0/TLS 1.0, so TLS has become not a choice but a baseline premise.

C. Key Characteristics

TLS's characteristics can be distilled into three. The first is its hybrid cryptographic structure: slow public-key operations (the asymmetric side of [[symmetric-asymmetric-encryption]]) are used only for session-key agreement and server authentication, while the actual bulk data is encrypted with a fast symmetric key (such as AES) — a [[digital-envelope]]-style combination. The second is negotiation-based flexibility: during the handshake both sides pick a mutually supported version, cipher suite, and key-exchange method, so aging algorithms can be replaced. The third is layer independence: TLS operates regardless of the upper protocol (HTTP, SMTP, IMAP), protecting diverse applications with a single security layer. Together these three make TLS the de facto standard of Internet security — "flexible through negotiation, efficient through hybrid cryptography, and universal through layer separation."

2. Overall Structure and Components of TLS

TLS should be understood not as a single procedure but as a layered structure consisting of the lower Record Protocol (a transport layer that encrypts and fragments all data and carries it) and the upper subprotocols that run on it (Handshake, Alert, ChangeCipherSpec, Application Data). Below is an overall structural diagram of the TLS protocol stack and the relationships among its components.

flowchart TB
  subgraph APP["Application layer"]
    HTTP["HTTP/SMTP/IMAP, etc."]
  end
  subgraph TLS["TLS layer"]
    subgraph SUB["Upper subprotocols"]
      HS["Handshake(key agreement·auth)"]
      AL["Alert(alerts·closure)"]
      AD["Application Data(encrypted transfer)"]
    end
    REC["Record Protocol(fragment·compress·MAC·encrypt)"]
    SUB --> REC
  end
  subgraph NET["Transport·network layer"]
    TCP["TCP(reliable transport)"]
    IP["IP"]
  end
  HTTP --> HS
  HTTP --> AD
  REC --> TCP --> IP

The core of this structure is that the Record Protocol breaks every upper message into record units and encrypts each with a sequence number and authentication tag. Even handshake messages, once a key is agreed, are encrypted and delivered at the record layer, and application data is likewise protected in the same record format. In other words, the Handshake is the "control plane that agrees on which key to protect with," and the Record is the "data plane that enforces that protection" — the roles are separated.

A. Record Protocol — the Data Plane

The Record Protocol fragments the byte stream descending from above into records of a fixed size (up to 2^14 bytes), applies authentication and encryption including a sequence number to each record, and passes it down to TCP. Through TLS 1.2 a mix of "MAC-then-Encrypt" combinations and AEAD coexisted, but TLS 1.3 permits AEAD (Authenticated Encryption with Associated Data) only, guaranteeing confidentiality and integrity in a single operation. AEAD binds encryption and authentication-tag generation into one algorithm (AES-GCM, ChaCha20-Poly1305), removing the room for attacks (POODLE, Lucky13) that exploited subtle differences in the order of past padding and MAC handling.

In practice, the Record Protocol's design bears directly on performance. If a record is too large, the latency to the first byte grows; too small, and header and tag overhead increases. CDNs and web servers therefore use a "record-size adaptation" technique that starts with small records to render the first screen quickly and gradually grows them. Netflix and Google, for example, use dynamic record sizing to reduce perceived latency in video streaming.

Furthermore, the AEAD authentication tag attached to each record (16 bytes for AES-GCM) and the record header impose relatively large overhead in communications with many small messages. For instance, sending IoT sensor data of a few dozen bytes as an individual record per message can make the tag and header share exceed the body, so constrained embedded environments require design considerations such as batching messages or using a lightweight profile. Thus the choice of Record-layer parameters is decided in tandem with non-functional requirements of bandwidth, latency, and power.

B. Handshake Protocol — the Control Plane

The Handshake Protocol is the heart of TLS, responsible for version and cipher-suite negotiation, key exchange, server (and if needed client) authentication, and session-key derivation. In TLS 1.3 this process is greatly simplified and completes in a single 1-RTT (one round trip). The step-by-step detail is explained in the diagram below. Of the remaining subprotocols, Alert handles error and session-closure notices (e.g., close_notify), and ChangeCipherSpec, which signaled a cipher switch through TLS 1.2, remains only as a compatibility dummy in TLS 1.3.

The naming of cipher suites also took on a different meaning across the two versions, which matters in practice. TLS 1.2's TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 packs 'key exchange (ECDHE) + authentication (RSA) + symmetric cipher (AES-128-GCM) + hash (SHA256)' all into one string, so the number of combinations exploded and weak combinations could creep in among them. TLS 1.3, by contrast, separates key exchange and authentication out of the cipher suite — negotiating them via the supported_groups and signature_algorithms extensions — and leaves only the AEAD algorithm and hash in the cipher-suite string, as in TLS_AES_128_GCM_SHA256. This separation simplifies negotiated combinations and makes it hard to stray from safe defaults.

3. The TLS 1.3 Handshake Procedure

The greatest innovation of the TLS 1.3 handshake is that the client sends its key-exchange material (key_share) together in the very first message, cutting TLS 1.2's 2-RTT to 1-RTT. Below is the detailed flow of a typical 1-RTT full handshake.

sequenceDiagram
  participant C as "Client"
  participant S as "Server"
  C->>S: ClientHello(versions·cipher suites·supported_groups·key_share)
  Note over S: Select common parameters + generate own key_share
  S->>C: ServerHello(selected cipher·key_share)
  Note over C,S: Derive ECDHE shared secret → derive handshake keys via HKDF
  S->>C: {EncryptedExtensions · Certificate · CertificateVerify · Finished}
  Note over C: Verify certificate chain + check CertificateVerify signature
  C->>S: {Finished} (+ optional client certificate)
  Note over C,S: Application traffic keys fully derived
  C->>S: Application Data(encrypted)
  S->>C: Application Data(encrypted)

A. ClientHello and ServerHello — the Start of Negotiation

The handshake begins with the client sending a ClientHello. It carries the list of supported TLS versions, the list of preferred cipher suites, the elliptic-curve groups to be used for key exchange (supported_groups, e.g., X25519, secp256r1), and the public key material (key_share) for those groups. In TLS 1.2 key material was exchanged only after the server's response, but in 1.3 the client throws its material in advance "assuming this curve will be used," saving one round trip.

Beyond key-exchange material, the ClientHello carries several extensions. Among them, SNI (Server Name Indication) is an essential extension that, in virtual-hosting environments serving many domains on one IP/port, tells the server "give me the certificate for this domain," while ALPN (Application-Layer Protocol Negotiation) agrees in advance on an upper protocol such as HTTP/2 or HTTP/3 at the handshake stage, eliminating an extra round trip. The problem is that SNI is exposed in plaintext, so "which site one is connecting to" is visible as-is to eavesdroppers and censorship devices — and the ECH described later is exactly the attempt to encrypt this last plaintext metadata.

The server responds with ServerHello, finalizing one mutually supported version and cipher suite from the received lists and returning its own key_share. If the server does not support the group the client sent, it does one more round trip with a HelloRetryRequest specifying a different group, but since most modern clients send X25519 by default this renegotiation is rare. Version downgrade defense matters at this stage: TLS 1.3 embeds a specific constant at the tail of the ServerHello random so that the fact "I support a higher version but was pulled down to a lower one" is detected at the Finished stage, blocking forced-downgrade attacks like the past FREAK and Logjam.

B. Key Exchange and Key Derivation — ECDHE and HKDF

TLS 1.3 effectively unifies the key-exchange method to (EC)DHE families such as ECDHE (Elliptic-Curve Diffie-Hellman, Ephemeral) and completely abolishes the old static RSA key exchange. This is one of 1.3's most important security improvements. In the static RSA scheme, a single leak of the server's private key allowed retroactive decryption of all previously collected traffic. ECDHE, by contrast, generates and discards an ephemeral key pair for each session, so PFS (Perfect Forward Secrecy) is guaranteed and past sessions remain safe even if the server's private key leaks in the future.

Both sides combine the peer's key_share with their own secret to derive the same shared secret, then, rather than using it directly, pass it through HKDF (HMAC-based Key Derivation Function) to progressively derive purpose-specific keys (handshake keys, application traffic keys, resumption PSK, etc.). HKDF normalizes entropy in two phases, 'extract' and 'expand,' and separates keys by label, implementing the key-separation principle so that the exposure of one key does not spread to keys for other purposes. That much of the handshake is encrypted early with these handshake keys is also characteristic of 1.3 — even the server certificate is encrypted, reducing metadata exposure to eavesdroppers.

C. Certificate Verification and Finished — Confirming Trust

The server sends Certificate (the X.509 certificate chain) and CertificateVerify. CertificateVerify is a value the server signs with its private key over the entire handshake so far; the client verifies this signature with the certificate's public key, thereby confirming that "the peer presenting the certificate is the real server that actually holds that private key." The client verifies the certificate chain up to a trust root (the Root CA of [[pki]]) and checks the domain-name (SAN) match, validity period, and revocation status (OCSP/CRL). If this verification fails or is loose, a man-in-the-middle attack succeeds, so mobile apps sometimes strengthen defense with certificate pinning that pins a specific certificate or public key.

The practical implications of certificate verification are far from trivial. In 2011 the certificate authority DigiNotar was breached and forged certificates were issued for Google domains, exposing the Gmail traffic of hundreds of thousands of people in Iran to a man-in-the-middle attack — an incident that drove home that "TLS's trust ultimately depends on the health of the entire CA ecosystem." From this lesson, Certificate Transparency (CT) logs were introduced, recording and monitoring all issued certificates in a public log so that fraudulent issuance can be detected early. In other words, one must understand that TLS's authentication trust is not a single point in the protocol but an ecosystem trust jointly upheld by CAs, CT, and browser root stores.

Finally, both sides exchange a Finished message. Finished is a MAC over the hash of all handshake messages exchanged so far, mutually confirming that the negotiation was not tampered with midway. From this point on, the application traffic keys are activated and actual data flows encrypted. In a mutual-authentication (mTLS) environment where the server must also authenticate the client, the server sends a CertificateRequest and the client additionally presents its own certificate and CertificateVerify — this is used centrally in service-to-service authentication of a [[zero-trust]] architecture.

D. Session Resumption and 0-RTT — Performance Optimization

TLS 1.3 provides PSK (Pre-Shared Key)-based session resumption so that reconnecting to a peer it has already handshaked with does not repeat the whole process. After the first connection the client keeps a resumption ticket (NewSessionTicket) issued by the server and, on the next connection, presents it to restore the session quickly without asymmetric operations. Going further, 0-RTT (Zero Round-Trip Time) lets the client, on reconnection, carry actual application data along with the ClientHello, making the round-trip latency effectively zero.

However, 0-RTT carries an inherent risk. Early data sent over 0-RTT is vulnerable to replay attacks: if an attacker replays the same request, the server may process it twice. Therefore 0-RTT must be allowed only for idempotent requests ([[idempotency]]) such as lookups (GET), and the server must control it so that it is not applied to non-idempotent requests such as payments or state changes. This is a classic trade-off where 'latency reduction' and 'replay safety' conflict, so CDNs and large services enable 0-RTT only selectively.

4. Comparison of TLS 1.2 and TLS 1.3

Viewing the difference between the two versions through the lens of 'why it changed,' 1.3 was designed to "simply remove the insecure options." 1.2 was flexible, allowing dozens of cipher suites and RSA/DHE/ECDHE/static/ephemeral key exchange, but that flexibility became "room to pick a weak combination," a hotbed of downgrade and weak-cipher attacks. 1.3 drastically shrank the cipher suites to about five AEAD options, limited key exchange to (EC)DHE, and removed renegotiation and compression, forcing 'safe defaults' such that misconfiguration itself is impossible.

Category	TLS 1.2	TLS 1.3
Handshake round trips	2-RTT	1-RTT (0-RTT on resumption)
Key exchange	RSA·DHE·ECDHE (static RSA allowed)	(EC)DHE only — PFS enforced
Symmetric cipher	CBC (MAC-then-Encrypt)·AEAD mix	AEAD only (GCM·ChaCha20)
Number of cipher suites	dozens (many weak combinations)	reduced to about five
Renegotiation/compression	allowed (attack surface)	removed
Handshake encryption	much in plaintext (certificate exposed)	early encryption (certificate protected)

The practical implications for performance are clear. A handshake shortened to 1-RTT greatly improves perceived response speed in mobile and high-latency environments. For example, on a mobile line with 100 ms round-trip latency, cutting one round trip saves 100 ms per first connection, and this effect accumulates on web pages that fetch many resources. Indeed, Google and Cloudflare reported that handshake latency fell by about half after migrating to TLS 1.3. On the security side too, enforcing PFS and removing weak ciphers structurally shrank the incident surface.

That said, migration also comes with practical constraints. Enterprise environments have many middleboxes that terminate and inspect TLS, and a 'protocol ossification' problem emerged where these failed to recognize 1.3 properly and broke the handshake. TLS 1.3 leaving a ChangeCipherSpec dummy record and moving the version marker into an extension (supported_versions) was also a design to make packets look 'like 1.2' to old middleboxes and secure compatibility. Moreover, organizations such as finance that audited traffic by passive decryption based on static RSA found that method impossible once PFS was enforced, and had to redesign their audit architecture around endpoint agents or TLS-terminating proxies. This is a representative case where a security improvement forced a change in operational practice.

5. Advanced Topics — Major Attacks and Latest Trends

The history of TLS is a history of attack and defense. Downgrade attacks (FREAK, Logjam) manipulated the negotiation to pull it down to weak export-grade ciphers; POODLE exploited the CBC padding-handling flaw of SSL 3.0, and BEAST the CBC IV predictability of TLS 1.0. The 2014 Heartbleed was not a flaw in the TLS protocol itself but an implementation bug in OpenSSL's Heartbeat extension (a missing bounds check) that leaked server memory — an incident that taught that "even a safe protocol is fatal if the implementation is wrong." TLS 1.3 removed by design those that can be blocked at the protocol level (CBC, renegotiation, compression, downgrade), while Heartbleed-class implementation flaws are addressed by library patches and adoption of memory-safe languages (such as the Rust-based rustls).

A TLS fingerprinting (JA3/JA3S) that turns the handshake's plaintext nature to advantage is also actively used in practice. The list, order, and cipher-suite combination of extensions carried in the ClientHello have a pattern unique to each client implementation, so hashing them can identify "whether this connection is a normal browser or known malware/bot" even though the traffic is encrypted. Security operations and bot blocking use this as a detection signal, and conversely attackers try to evade detection by mimicking a normal browser (fingerprint spoofing). This is a representative case showing that 'encryption hides the content but leaves metadata patterns,' paradoxically underscoring the need for the metadata protection ECH aims at.

The most noteworthy of the latest trends is the transition to post-quantum cryptography (PQC). A large-scale quantum computer could break the discrete-logarithm problem that ECDHE relies on, so the threat of 'Harvest Now, Decrypt Later' is materializing. In response, Google and Cloudflare began large-scale application of a hybrid key exchange (X25519+ML-KEM, formerly Kyber) combining existing X25519 with a lattice-based algorithm to Chrome and Edge traffic from 2023–2024 (see [[post-quantum-crypto]]). In addition, ECH (Encrypted Client Hello), which encrypts even the SNI (the connecting domain) of the ClientHello, has been standardized and deployed to strengthen connection-metadata privacy, and UDP-based [[quic-http3]] internalizes the TLS 1.3 handshake into the transport layer to shorten connection establishment even further. mTLS has established itself as the default communication security of [[zero-trust]] and service meshes.

6. Considerations and Implications

The following are considerations when designing and operating TLS from a professional-engineer perspective.

• Adoption strategy (enforce safe defaults): New systems should make TLS 1.3 the default, keeping 1.2 only for backward compatibility and fully disabling SSL 3.0/TLS 1.0/1.1 and weak cipher suites such as CBC and RC4. Server configuration should be standardized not by guesswork but by targeting an 'A grade' as a metric with verification tools such as Mozilla's SSL Configuration Generator and SSL Labs, and HSTS headers should forbid plaintext connections themselves.

• Trade-off (performance vs. safety): 0-RTT dramatically reduces latency but carries replay risk, so a selective policy is needed that allows it only for idempotent requests and disables it for non-idempotent APIs. Likewise, the lifetime and key-rotation cycle of session-resumption tickets must be balanced between 'reconnection performance' and 'risk of weakening PFS'; failing to rotate the ticket encryption key (STEK) periodically undermines the forward secrecy of resumed segments.

• Implementation/operational security (certificate lifecycle): Many TLS incidents arise not from the protocol but from expired certificates, weak private-key storage, and loose chain verification. Certificates should be auto-issued and renewed via ACME (Let's Encrypt) to eliminate expiry incidents, private keys kept in an HSM/KMS, and internal service-to-service communication equipped with a scheme that auto-rotates short-lived certificates (such as SPIFFE/SPIRE). To guard against implementation flaws (Heartbleed class), library CVEs should be continuously tracked and patched.

• Conflict between visibility and regulation (decryption operations): Universal encryption raises privacy but lowers traffic visibility in security monitoring ([[siem]], IDS) and fault analysis. Organizations must therefore design TLS termination and re-encryption points at the perimeter, minimize the scope of decryption so that it does not conflict with personal data and regulation, and tightly control access to decryption keys and logs. Designing both visibility assurance and privacy protection at once is the key.

• Outlook and related technologies (quantum-transition roadmap): In response to the 'Harvest Now, Decrypt Later' threat, a crypto-agility roadmap should be established that preemptively transitions to hybrid PQC key exchange, starting with data requiring long-term confidentiality (medical, state secrets). Governance that abstracts algorithms so they can be swapped easily by configuration and periodically checks the PQC support status of certificates and libraries will be a core task for the next decade.

References

• IETF, "RFC 8446: The Transport Layer Security (TLS) Protocol Version 1.3": https://datatracker.ietf.org/doc/html/rfc8446
• IETF, "RFC 5246: The TLS Protocol Version 1.2": https://datatracker.ietf.org/doc/html/rfc5246
• Cloudflare, "A Detailed Look at RFC 8446 (a.k.a. TLS 1.3)": https://blog.cloudflare.com/rfc-8446-aka-tls-1-3/
• Mozilla, "Security/Server Side TLS": https://wiki.mozilla.org/Security/Server_Side_TLS
• Cloudflare, "The TLS Post-Quantum Experiment": https://blog.cloudflare.com/post-quantum-for-all/

---

In one line: TLS is the de facto standard of Internet security, providing confidentiality, integrity, and authentication over TCP with hybrid cryptography; TLS 1.3 is a redesign that cut the handshake to 1-RTT and left only (EC)DHE and AEAD to enforce forward secrecy, raising both safety and performance, and the remaining tasks are controlling 0-RTT replay, automating the certificate lifecycle, and transitioning to post-quantum cryptography.