Performance study of kernel TLS handshakes
Abstract
Tempesta TLS, a part of Tempesta FW, implements TLS handshakes in the Linux kernel and focuses on performance to filter out application layer DDoS attacks. We started development from the fork of mbed TLS library, but it took significant effort to make it fast, so we ended up with full reworking of the library. The design of Tempesta TLS was discussed in our previous work [17]. The main focus of this paper is to explore how much performance we can get for TLS handshake itself and the whole application, on example of HTTPS server, by moving TLS handshake into the kernel space. While performance optimization of the TLS handshakes mathematics is still in progress, we observed that Tempesta TLS can establish 40-80% more TLS connections per second than OpenSSL/Nginx and provide up to x4 lower latency in some tests.
Keywords
User space TLS handshakes, TLS, Linux kernel, fast computations on elliptic curves.
KPTI impact on performance
An in-kernel TLS handshake implementation does not need to switch between the kernel and user spaces, so KPTI is not involved. Our goal was to see how performance of a user-space HTTPS server suffers from KPTI. We expected to see high KPTI impact on the performance due to many network I/O system calls and a lot of memory allocations. But our measurements showed less than 4% performance drop on system with security mitigations enabled.
| TLS | w/o KPTI | w/KPTI | Delta |
|---|---|---|---|
| 1.2 | 8'735 | 8'401 | -3.82% |
| 1.3 | 7'734 | 7'473 | -3.37% |
TLS handshake involves not so many network I/O system calls, so the performance impact was not very high. All the hottest functions are related to cryptographic operations and memory allocations.
Functions Overhead
| Functions | Overhead |
|---|---|
| libcrypto.so: | |
| __ecp_nistz256_mul_montx, | |
| __ecp_nistz256_sqr_montx, | |
| sha256_block_data_order_avx2, | |
| ecp_nistz256_avx2_gather_w7, | |
| OPENSSL_cleanse, | |
| ecp_nistz256_ord_sqr_montx, | |
| ecp_nistz256_point_doublex, | |
| __ecp_nistz256_sub_fromx, | |
| __ecp_nistz256_mul_by_2x, | |
| ecp_nistz256_point_addx, | |
| ecp_nistz256_point_add_affinex, | |
| aesni_ecb_encrypt, | |
| BN_num_bits_word, | |
| EVP_MD_CTX_reset | 30.7% |
| libc.so: _int_malloc, _int_free, | |
| malloc, malloc_consolidate, cfree, | |
| __memmove_avx_unaligned_erms, | |
| __memset_avx2_unaligned_erms | 13.2% |
| kernel: do_syscall_64, entry_SYSCALL_64, prepare_exit_to_usermode, | |
| syscall_return_via_sysret, common_interrupt | 4.5% |
| nginx: - | 0% |
Since performance drop for the user-space HTTPS server is not high in this scenario, user-space HTTPS server can not benefit a lot by disabling KPTI.
The kernel and TLS versions impact on performance
As Tempesta TLS was initially based on Linux kernel 4.14 and will be ported on the next LTS release after Linux 5.7, we also compared TLS handshake performance on those kernel versions. All mitigations available in both kernels was switched on.
| Kernel | Handshakes/s | 95P Latency, ms |
|---|---|---|
| TLS 1.2 New sessions | ||
| 4.14 | 7’624 | 498 |
| 5.7 | 7’284 | 466 |
| -4.5% | -6.4% | |
| TLS 1.2 Session Resumption | ||
| 4.14 | 19’203 | 246 |
| 5.7 | 17’452 | 112 |
| -9% | -54% | |
| TLS 1.3 New sessions | ||
| 4.14 | 7’147 | 315 |
| 5.7 | 6’811 | 466 |
| -4.7% | +47% | |
| TLS 1.3 Session Resumption | ||
| 4.14 | 6’472 | 287 |
| 5.7 | 6’183 | 342 |
| -4.4% | +19% |
The 5.7 kernel in all tests makes 4.5-9% less handshakes in second than the 4.14 kernel. It’s also interesting that the latency on TLS 1.2 is significantly lower on 5.7 kernel while TLS 1.3 shows the opposite picture.
TLS handshake in a full HTTPS transaction
Long living connections can not benefit from handshake optimisations since the handshake takes a relatively small part of a communication process. Short living connections show the opposite picture: the handshake overhead is huge enough to dominate the communication process. To prove that an HTTPS transaction performance depends on handshake processing we compared a server performance in the scenarios when a client opens and immediately drops a TLS connection right after a TLS connection was established (no data transfer), right after a 1KB server response transmission, or right after 10KB server response transmission. In all the scenarios, the client sends HTTP/1.1 request with the usual headers set for popular browsers.
| Handshakes/s | Resp 1Kb/s | Resp 10Kb/s |
|---|---|---|
| Plain HTTP | ||
| N/A | 77’109 | 55’060 |
| HTTPS with TLS 1.2(new sessions) | ||
| 7’225 | 6’402 | 6’238 |
| HTTPS with TLS 1.2(session resumption) | ||
| 17’274 | 13’472 | 12’630 |
The TLS handshake overhead is high enough to reduce performance on encrypted short living connections by x12 for 1KB responses, and by x8.5 for 10KB responses. TLS session resumption reduces the difference by x2, but the handshake still costs a lot.
Compare the kernel- and user-space TLS handshakes
To compare in-kernel and user-space TLS handshakes performance it is required to benchmark basic cryptographic operations first. For elliptic cryptography they are ECDSA, Elliptic Curve Digital Signature Algorithm, and ECDHE, Ellipticcurve DiffieHellman key agreement protocol.
| ECDSA, op/s | ECDHE, op/s | |
|---|---|---|
| OpenSSL 1.1.1d | 36’472 | 16’619 |
| WolfSSL* | 41’527 | 55’548 |
| Tempesta TLS** | 27’261 | 6’690 |
| * before the non constant-time fixed point multiplication was fixed. | ||
| ** including ephemeral keys generation. |
Conclusion
In-kernel handshakes allow to serve more requests per second in all test cases. While we saw 40-80% performance improvement for the benchmarks in a virtual environment and session resumptions on bare-metal, we observed only 6% performance improvement for new TLS sessions in a bare-metal environment.
The performance of Tempesta TLS is affected by the heavy cryptographic computations performed in SoftIRQ, which can introduce latencies under particular conditions. Further optimizations are necessary to improve overall performance and mitigate any potential issues around TLS handshakes for various environments.