<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html lang="en" style='--code-editor-font: var(--default-mono-font, "GitLab Mono"), JetBrains Mono, Menlo, DejaVu Sans Mono, Liberation Mono, Consolas, Ubuntu Mono, Courier New, andale mono, lucida console, monospace;'>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<title>
GitLab
</title>
<style data-premailer="ignore" type="text/css">
a { color: #1068bf; }
</style>
<style>img {
max-width: 100%; height: auto;
}
body {
font-size: .875rem;
}
body {
-webkit-text-shadow: rgba(255,255,255,.01) 0 0 1px;
}
body {
font-family: "GitLab Sans",-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Noto Sans",Ubuntu,Cantarell,"Helvetica Neue",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji"; font-size: inherit;
}
</style>
</head>
<body style='font-size: inherit; -webkit-text-shadow: rgba(255,255,255,.01) 0 0 1px; font-family: "GitLab Sans",-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Noto Sans",Ubuntu,Cantarell,"Helvetica Neue",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";'>
<div class="content">
<p class="details" style="font-style: italic; color: #626168;">
<a href="https://gitlab.com/berrange">Daniel P. Berrangé</a> created an issue: <a href="https://gitlab.com/gnutls/gnutls/-/issues/1717">#1717</a>
</p>
<div class="md" style="position: relative; z-index: 1; color: #3a383f; word-wrap: break-word;">
<h2 dir="auto" style="margin-top: 0; margin-bottom: 10px;" align="initial">
<a href="#description-of-problem" aria-hidden="true" class="anchor" id="user-content-description-of-problem" style="margin-top: 0; margin-left: -20px; text-decoration: none; outline: none; position: absolute; width: 20px;"></a>Description of problem:</h2>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">Note, this bug reports comes out of a QEMU bug we've been struggling to resolve for a few years <a href="https://gitlab.com/qemu-project/qemu/-/issues/1937" data-reference-type="issue" data-original="https://gitlab.com/qemu-project/qemu/-/issues/1937" data-link="false" data-link-reference="true" data-issue="136443921" data-project="11167699" data-iid="1937" data-namespace-path="qemu-project/qemu" data-project-path="qemu-project/qemu" data-issue-type="issue" data-container="body" data-placement="top" title="Live migration with TLS fail (GNUTLS AUTO_REKEY)" class="gfm gfm-issue" style="margin-top: 0;">qemu-project/qemu#1937</a></p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">I've also seen <a href="https://gitlab.com/gnutls/gnutls/-/issues/1567" data-reference-type="issue" data-original="https://gitlab.com/gnutls/gnutls/-/issues/1567" data-link="false" data-link-reference="true" data-issue="151096010" data-project="179611" data-iid="1567" data-namespace-path="gnutls/gnutls" data-project-path="gnutls/gnutls" data-issue-type="issue" data-container="body" data-placement="top" title="SEGFAULT in libgnutls30 during multithreaded call of `gnutls_record_send`" class="gfm gfm-issue" style="margin-top: 0;">#1567</a> talking about SEGVs, which may well ultimately be the same problem as this bug. I've never managed to create any SEGV in my own testing though, only error return codes.</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">The GNUTLS docs about thread safety have a list of conditions which, if satisfied, are expected to allow a <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; margin-top: 0; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>gnutls_session_t</code> object to be used for concurrent I/O from 2 threads (ie for parallel <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>gnutls_record_send</code> & <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>gnutls_record_recv</code>, but <strong style="font-weight: 600;">nothing</strong> else):</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial"><a href="https://gnutls.org/manual/gnutls.html#Thread-safety" rel="nofollow noreferrer noopener" target="_blank" style="margin-top: 0;">https://gnutls.org/manual/gnutls.html#Thread-safety</a></p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">AFAICT, even if an application follows that guidance, usage of <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; margin-top: 0; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>gnutls_session_t</code> from 2 threads for concurrent send & recv remains unsafe by default.</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">At the very least this appears always unsafe if TLS 1.3 is negotiated for the connection, I've not checked if older TLS protocol versions are similarly impacted by default.</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">In my testing any TLS 1.3 rekeying message sent <strong style="font-weight: 600; margin-top: 0;">or</strong> received, will corrupt the internal session state if there are concurrent send & recv operations taking place in separate threads.</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">If the application sets <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; margin-top: 0; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>GNUTLS_NO_AUTO_REKEY</code> on their <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>gnutls_session_t</code> object, that will merely prevent <em>their end</em> of the connection <em>initiating</em> a rekey operation.</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">The remote peer may still initiate a rekey operation, either automatically or manually. In addition the rekey request from the remote peer, might have set the flag to require the local peer to initiate its own rekey for the other direction of the channel.</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">IOW setting <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; margin-top: 0; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>GNUTLS_NO_AUTO_REKEY</code> is insufficient to prevent rekeying in either direction on a TLS 1.3 session.</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">As an alternative an application could arrange for <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; margin-top: 0; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>CHACHA20-POLY1305</code> to be the only listed algorithm in the priority string, since that algorithm does not require automatic rekeying. The problem with this is that there's no guarantee the remote peer will support this - only <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>AES-128</code> is mandated by the RFC IIUC, or an admin defined priority string on the remote peer may prevent <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>CHACHA20-POLY1305</code> being offered.</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">Lets assume the handshake does negotiate <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; margin-top: 0; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>CHACHA20-POLY1305</code>, such that auto-rekeying is not required. This is still not sufficient to guarantee that an application's usage of <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>gnutls_session_t</code> from multiple threads for send & recv is safe. The remote implementation may have (redundantly) enabled auto-rekeying for <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>CHACHA20-POLY1305</code>, or the remote peer's application code could initiate a manual rekeying at any time regardless of cipher choice.</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">AFAICT, the only way to safely use <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; margin-top: 0; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>gnutls_session_t</code> for concurrent I/O from two threads is to have an application level mutex acquired & released around <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>gnutls_record_send</code> and <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>gnutls_record_recv</code> calls, but that has problems which make it suboptimal:</p>
<ul dir="auto" style="text-align: initial; list-style-type: disc; margin: 0 0 1rem; padding: 0;">
<li style="margin-top: 0; line-height: 1.6em; margin-left: 25px; padding-left: 3px;">It eliminates concurrency entirely defeating the point of using 2 threads</li>
<li style="line-height: 1.6em; margin-left: 25px; padding-left: 3px;">It entirely prevents all progress if one thread sits in a blocking <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; margin-top: 0; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; white-space: pre-wrap; overflow-wrap: break-word; word-break: keep-all; padding: 0.125rem 0.25rem;'>gnutls_record_recv</code> while wanting to also call <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; white-space: pre-wrap; overflow-wrap: break-word; word-break: keep-all; padding: 0.125rem 0.25rem;'>gnutls_record_send</code>
</li>
</ul>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">To mitigate the second problem there are two options</p>
<ul dir="auto" style="text-align: initial; list-style-type: disc; margin: 0 0 1rem; padding: 0;">
<li style="margin-top: 0; line-height: 1.6em; margin-left: 25px; padding-left: 3px;">Use poll() to detect readability/writability before calling gnutls_record_send/recv, so they (hopefully) don't block.</li>
<li style="line-height: 1.6em; margin-left: 25px; padding-left: 3px;">Register custom push/pull functions which then release + reaquire the mutex either side of the recv/send sockets syscalls.</li>
</ul>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">The second option would rely on GNUTLS' concurrency problems being confined to exclusively <strong style="font-weight: 600; margin-top: 0;">after</strong> the syscall completes, or exclusively <strong style="font-weight: 600;">before</strong> the syscall is invoked. If there is anything in GNUTLS rekey process which relies on state being maintained across the OS syscall, then this release+reacquire dance would still be potentially unsafe. My testing so far though, hasn't exposed a problem with release+reacquire dance inside the push+pull functions.</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">Both mitigations share the problem, however, that the application mutex is preventing any concurrency on the cipher operations. Even with hardware accelerated AES, cipher operations can be a performance bottleneck, so neither are really viable workarounds if needing to maximise concurrency in the general case.</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">Conceptually I also don't like the idea of setting GNUTLS_NO_AUTO_REKEY as that is disabling a cryptographic security recommendation from the TLS 1.3 RFC, and thus not rekeying could be considered a security bug / CVE in any app that does that.</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">AFAICT, the only way to fix this without impacting concurrency/performance, is for GNUTLS to have its own private mutex(es) or rw-lock to protect its internal session state across the rekey handling, when there are concurrent send/recv operations taking place.</p>
<h2 dir="auto" style="margin-top: 20px; margin-bottom: 10px;" align="initial">
<a href="#version-of-gnutls-used" aria-hidden="true" class="anchor" id="user-content-version-of-gnutls-used" style="margin-top: 0; margin-left: -20px; text-decoration: none; outline: none; position: absolute; width: 20px;"></a>Version of gnutls used:</h2>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">Fedora build of 3.8.9 and upstream git master build of <a href="https://gitlab.com/gnutls/gnutls/-/commit/af6a39894e0dc8e1dd3f9690f7fc011d0ffe86b5" data-reference-type="commit" data-original="af6a39894e0dc8e1dd3f9690f7fc011d0ffe86b5" data-link="false" data-link-reference="false" data-commit="af6a39894e0dc8e1dd3f9690f7fc011d0ffe86b5" data-project="179611" data-container="body" data-placement="top" title="Merge branch 'zfridric_devel' into 'master'" class="gfm gfm-commit has-tooltip" style='font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; font-size: 95%; margin-top: 0;'>af6a3989</a></p>
<h2 dir="auto" style="margin-top: 20px; margin-bottom: 10px;" align="initial">
<a href="#distributor-of-gnutls-eg-ubuntu-fedora-rhel" aria-hidden="true" class="anchor" id="user-content-distributor-of-gnutls-eg-ubuntu-fedora-rhel" style="margin-top: 0; margin-left: -20px; text-decoration: none; outline: none; position: absolute; width: 20px;"></a>Distributor of gnutls (e.g., Ubuntu, Fedora, RHEL)</h2>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">Fedora and Upstream.</p>
<h2 dir="auto" style="margin-top: 20px; margin-bottom: 10px;" align="initial">
<a href="#how-reproducible" aria-hidden="true" class="anchor" id="user-content-how-reproducible" style="margin-top: 0; margin-left: -20px; text-decoration: none; outline: none; position: absolute; width: 20px;"></a>How reproducible:</h2>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">Non-deterministic frequency, but will always eventually hit when TLS 1.3 is negotiated and concurrent <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; margin-top: 0; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>gnutls_record_{send,recv}</code> calls are made from 2 threads.</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">Steps to Reproduce:
Use the following demo program which acts as both a single threaded server and two-threaded client (one send, one recv)</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial"><a href="https://gitlab.com/-/project/179611/uploads/d9a9465521cc287814085ec4e49b9fd3/tlsrekey.c" data-canonical-src="/uploads/d9a9465521cc287814085ec4e49b9fd3/tlsrekey.c" data-link="true" class="gfm" style="margin-top: 0;">tlsrekey.c</a></p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">A default build of it will operate correctly, since all rekeying is disabled on server and client.</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">To demonstrate the flaw with client initiated auto-rekeying</p>
<div class="gl-relative markdown-code-block js-markdown-code">
<pre class="code highlight js-syntax-highlight language-plaintext" v-pre="true" style='display: block; font-size: 14px; color: #3a383f; line-height: 1.6em; overflow-x: auto; border-radius: .25rem; position: relative; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; margin: 0 0 1rem; padding: 12px; border: 1px solid #dcdcde;'><code style='font-size: inherit; color: inherit; word-wrap: normal; word-break: keep-all; border-radius: .25rem; background-color: inherit; white-space: pre; margin-top: 0; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: normal; padding: unset;'><span id="LC1" class="line" lang="plaintext" style="margin-top: 0;">$ gcc -g -Wall -lgnutls -o tlsrekey tlsrekey.c -DCLIENT_AUTO_REKEY</span>
<span id="LC2" class="line" lang="plaintext">$ ./tlsrekey </span>
<span id="LC3" class="line" lang="plaintext">1569408: client sender</span>
<span id="LC4" class="line" lang="plaintext">1569407: client receiver</span>
<span id="LC5" class="line" lang="plaintext">1569409: server echo</span>
<span id="LC6" class="line" lang="plaintext">1569408: send: The specified session has been invalidated for some reason.</span>
<span id="LC7" class="line" lang="plaintext">1569407: recv: Decryption has failed.</span></code></pre>
<copy-code></copy-code><insert-code-snippet></insert-code-snippet>
</div>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">This shows the client gnutls_record_send() call failing with invalid session, and the client gnutls_record_recv() call failing with bad decryption. It suggests the client's automatic rekey has corrupted its own session state</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">To demonstrate the flaw with server initiated auto-rekeying</p>
<div class="gl-relative markdown-code-block js-markdown-code">
<pre class="code highlight js-syntax-highlight language-plaintext" v-pre="true" style='display: block; font-size: 14px; color: #3a383f; line-height: 1.6em; overflow-x: auto; border-radius: .25rem; position: relative; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; margin: 0 0 1rem; padding: 12px; border: 1px solid #dcdcde;'><code style='font-size: inherit; color: inherit; word-wrap: normal; word-break: keep-all; border-radius: .25rem; background-color: inherit; white-space: pre; margin-top: 0; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: normal; padding: unset;'><span id="LC1" class="line" lang="plaintext" style="margin-top: 0;">$ gcc -g -Wall -lgnutls -o tlsrekey tlsrekey.c -DSERVER_AUTO_REKEY</span>
<span id="LC2" class="line" lang="plaintext">$ ./tlsrekey </span>
<span id="LC3" class="line" lang="plaintext">1569654: client receiver</span>
<span id="LC4" class="line" lang="plaintext">1569655: client sender</span>
<span id="LC5" class="line" lang="plaintext">1569656: server echo</span>
<span id="LC6" class="line" lang="plaintext">1569656: echo recv: Decryption has failed.</span></code></pre>
<copy-code></copy-code><insert-code-snippet></insert-code-snippet>
</div>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">This shows the server gnutls_record_recv() call failing with bad decryption. The server is single threaded, however, so this is not a server flaw. It suggests the threaded client has sent bad data after receiving the server's rekey operation.</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">To demonstrate the flaw with server initiated manual rekeying</p>
<div class="gl-relative markdown-code-block js-markdown-code">
<pre class="code highlight js-syntax-highlight language-plaintext" v-pre="true" style='display: block; font-size: 14px; color: #3a383f; line-height: 1.6em; overflow-x: auto; border-radius: .25rem; position: relative; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; margin: 0 0 1rem; padding: 12px; border: 1px solid #dcdcde;'><code style='font-size: inherit; color: inherit; word-wrap: normal; word-break: keep-all; border-radius: .25rem; background-color: inherit; white-space: pre; margin-top: 0; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: normal; padding: unset;'><span id="LC1" class="line" lang="plaintext" style="margin-top: 0;">$ gcc -g -Wall -lgnutls -o tlsrekey tlsrekey.c -DSERVER_MANUAL_REKEY</span>
<span id="LC2" class="line" lang="plaintext">$ ./tlsrekey </span>
<span id="LC3" class="line" lang="plaintext">1569673: client receiver</span>
<span id="LC4" class="line" lang="plaintext">1569674: client sender</span>
<span id="LC5" class="line" lang="plaintext">1569675: server echo</span>
<span id="LC6" class="line" lang="plaintext">1569675: echo recv: Decryption has failed.</span></code></pre>
<copy-code></copy-code><insert-code-snippet></insert-code-snippet>
</div>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">This shows the same behaviour as the server auto-rekeying, likely triggering client session state corruption.</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">Note the response to server rekeying is somewhat non-deterministic - it often takes many rekeying requests from the server before the client goes wrong. IME, frequency of problems can be improved with <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; margin-top: 0; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>GNUTLS_DEBUG_LEVEL=5</code> since that tweaks the timing of the race.</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">In testing auto-rekeying failures, I advise making it rekey more often than every 16 million records to speed things up. The following patch does that, and also adds some printfs which make the race hit more frequently (less verbose that using <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; margin-top: 0; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>GNUTLS_DEBUG_LEVEL</code> for this purpose):</p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial"><a href="https://gitlab.com/-/project/179611/uploads/d34e69334e51db76ad0e96ddd23e328a/rekey-faster.patch" data-canonical-src="/uploads/d34e69334e51db76ad0e96ddd23e328a/rekey-faster.patch" data-link="true" class="gfm" style="margin-top: 0;">rekey-faster.patch</a></p>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial">When building the test program <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; margin-top: 0; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>-DLOCK</code> and <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>-DUNLOCK_IO</code> can be defined to implement the workarounds discussed above and show that GNUTLS no longer corrupts its state. As mentioned though, these workarounds harm concurrency of I/O and/or cipher operations.</p>
<h2 dir="auto" style="margin-top: 20px; margin-bottom: 10px;" align="initial">
<a href="#actual-results" aria-hidden="true" class="anchor" id="user-content-actual-results" style="margin-top: 0; margin-left: -20px; text-decoration: none; outline: none; position: absolute; width: 20px;"></a>Actual results:</h2>
<p dir="auto" style="color: #3a383f; margin: 0 0 1rem;" align="initial"><code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; margin-top: 0; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>gnutls_record_recv</code> and <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>gnutls_recv_send</code> called concurrently from 2 threads, will periodically fail when a TLS 1.3 rekeying operation is performed, whether initiated by the local or remote peer, whether automatic or manually initiated by the remote peer.</p>
<h2 dir="auto" style="margin-top: 20px; margin-bottom: 10px;" align="initial">
<a href="#expected-results" aria-hidden="true" class="anchor" id="user-content-expected-results" style="margin-top: 0; margin-left: -20px; text-decoration: none; outline: none; position: absolute; width: 20px;"></a>Expected results:</h2>
<p dir="auto" style="color: #3a383f; margin: 0;" align="initial"><code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; margin-top: 0; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>gnutls_record_recv</code> and <code style='font-size: 90%; color: #18171d; word-wrap: break-word; border-radius: .25rem; background-color: #ececef; font-weight: inherit; white-space: break-spaces; word-break: break-all; font-family: "GitLab Mono", "JetBrains Mono", "Menlo", "DejaVu Sans Mono", "Liberation Mono", "Consolas", "Ubuntu Mono", "Courier New", "andale mono", "lucida console", monospace; font-variant-ligatures: none; overflow-wrap: break-word; padding: 0.125rem 0.25rem;'>gnutls_recv_send</code> can be safely called concurrently from 2 threads, provided they are the only GNUTLS APIs used (as per <a href="https://gnutls.org/manual/gnutls.html#Thread-safety" rel="nofollow noreferrer noopener" target="_blank">https://gnutls.org/manual/gnutls.html#Thread-safety</a>)</p>
</div>
</div>
<div class="footer" style="margin-top: 10px;">
<p style="font-size: small; color: #626168;">
—
<br>
Reply to this email directly or <a href="https://gitlab.com/gnutls/gnutls/-/issues/1717">view it on GitLab</a>.
<br>
You're receiving this email because of your account on <a target="_blank" rel="noopener noreferrer" href="https://gitlab.com">gitlab.com</a>. <a href="https://gitlab.com/-/sent_notifications/47bf9e4c4d66ce90c069bd06575faa07/unsubscribe" target="_blank" rel="noopener noreferrer">Unsubscribe</a> from this thread · <a href="https://gitlab.com/-/profile/notifications" target="_blank" rel="noopener noreferrer" class="mng-notif-link">Manage all notifications</a> · <a href="https://gitlab.com/help" target="_blank" rel="noopener noreferrer" class="help-link">Help</a>
<span style="color: transparent; font-size: 0; display: none; overflow: hidden; opacity: 0; width: 0; height: 0; max-width: 0; max-height: 0;">
Notification message regarding https://gitlab.com/gnutls/gnutls/-/issues/1717 at 1749815067
</span>
<script type="application/ld+json">{"@context":"http://schema.org","@type":"EmailMessage","action":{"@type":"ViewAction","name":"View Issue","url":"https://gitlab.com/gnutls/gnutls/-/issues/1717"}}</script>
</p>
</div>
</body>
</html>