<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html lang="en" style='--code-editor-font: var(--default-mono-font, "Menlo"), DejaVu Sans Mono, Liberation Mono, Consolas, Ubuntu Mono, Courier New, andale mono, lucida console, monospace;'>
<head>
<meta content="text/html; charset=US-ASCII" http-equiv="Content-Type">
<title>
GitLab
</title>
<style data-premailer="ignore" type="text/css">
a { color: #1068bf; }
</style>
<style>img {
max-width: 100%; height: auto;
}
body {
font-size: 0.875rem;
}
body {
-webkit-text-shadow: rgba(255,255,255,0.01) 0 0 1px;
}
body {
font-family: var(--default-regular-font, -apple-system),BlinkMacSystemFont,"Segoe UI",Roboto,"Noto Sans",Ubuntu,Cantarell,"Helvetica Neue",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji"; font-size: inherit;
}
</style>
</head>
<body style='font-size: inherit; -webkit-text-shadow: rgba(255,255,255,0.01) 0 0 1px; font-family: var(--default-regular-font, -apple-system),BlinkMacSystemFont,"Segoe UI",Roboto,"Noto Sans",Ubuntu,Cantarell,"Helvetica Neue",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";'>
<div class="content">
<p class="details" style="font-style: italic; color: #737278;">
<a href="https://gitlab.com/rwmjones">Richard W.M. Jones</a> created an issue: <a href="https://gitlab.com/gnutls/gnutls/-/issues/1470">#1470</a>
</p>
<div class="md" style="color: #333238; word-wrap: break-word;">
<h2 dir="auto" style="font-size: 1.5em; font-weight: 600; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-color: #e6e6ea; border-bottom-style: solid; color: #333238; margin: 0 0 16px;" align="initial">
<a id="user-content-description-of-problem" class="anchor" href="#description-of-problem" aria-hidden="true" style="margin-top: 0; float: left; margin-left: -20px; text-decoration: none; outline: none;"></a>Description of problem:</h2>
<p dir="auto" style="color: #333238; margin: 0 0 16px;" align="initial">Since both <code style='font-size: 0.875rem; color: #1f1e24; word-wrap: break-word; background-color: #ececef; border-radius: 4px; margin-top: 0; font-weight: inherit; font-family: var(--default-mono-font, "Menlo"),"DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; white-space: pre-wrap; overflow-wrap: break-word; word-break: keep-all; padding: 2px 4px;'>nbdkit</code> and <code style='font-size: 0.875rem; color: #1f1e24; word-wrap: break-word; background-color: #ececef; border-radius: 4px; font-weight: inherit; font-family: var(--default-mono-font, "Menlo"),"DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; white-space: pre-wrap; overflow-wrap: break-word; word-break: keep-all; padding: 2px 4px;'>libnbd</code> support gnutls + kTLS, I can test (k)TLS sending and receiving between our server and client. I have found that if the sending process is using kTLS, then the data gets desynchronised under load. However receiving using kTLS seems fine. It seems there is a problem in gnutls_record_send (or perhaps the kernel related to sendmsg when using kTLS).</p>
<p dir="auto" style="color: #333238; margin: 0 0 16px;" align="initial">Here is my test:</p>
<div class="gl-relative markdown-code-block js-markdown-code">
<pre lang="plaintext" class="code highlight js-syntax-highlight language-plaintext" data-canonical-lang="" v-pre="true" style='display: block; font-size: 0.875rem; color: #333238; line-height: 1.6em; overflow-x: auto; border-radius: 4px; position: relative; font-family: var(--default-mono-font, "Menlo"),"DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; margin: 0 0 16px; padding: 12px; border: 1px solid #dcdcde;'><code style='font-size: 0.875rem; color: inherit; word-wrap: normal; word-break: keep-all; background-color: inherit; border-radius: 4px; white-space: pre; margin-top: 0; font-family: var(--default-mono-font, "Menlo"),"DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; overflow-wrap: normal; padding: unset;'><span id="LC1" class="line" lang="plaintext" style="margin-top: 0;">$ psktool -u alice -p keys.psk</span>
<span id="LC2" class="line" lang="plaintext">$ nbdkit --tls=require --tls-psk=keys.psk pattern 10G \</span>
<span id="LC3" class="line" lang="plaintext"> --run 'nbdcopy -p --no-extents "nbds://alice@localhost?tls-psk-file=keys.psk" null:'</span></code></pre>
<copy-code></copy-code>
</div>
<p dir="auto" style="color: #333238; margin: 0 0 16px;" align="initial">It copies 10G of pattern data from nbdkit (server) to nbdcopy (client) and then throws it away (<code style='font-size: 0.875rem; color: #1f1e24; word-wrap: break-word; background-color: #ececef; border-radius: 4px; margin-top: 0; font-weight: inherit; font-family: var(--default-mono-font, "Menlo"),"DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; white-space: pre-wrap; overflow-wrap: break-word; word-break: keep-all; padding: 2px 4px;'>null:</code>) over TCP localhost port 10809.</p>
<p dir="auto" style="color: #333238; margin: 0 0 16px;" align="initial">The pattern data is generated by <a href="https://libguestfs.org/nbdkit-pattern-plugin.1.html" rel="nofollow noreferrer noopener" target="_blank" style="margin-top: 0;">https://libguestfs.org/nbdkit-pattern-plugin.1.html</a></p>
<p dir="auto" style="color: #333238; margin: 0 0 16px;" align="initial">Without kTLS it works fine. With kTLS it fails.</p>
<p dir="auto" style="color: #333238; margin: 0 0 16px;" align="initial">I also hacked gnutls so I could disable kTLS selectively. Using LD_LIBRARY_PATH I am able to selectively enable and disable kTLS at either end. I found that it only fails if the <strong style="font-weight: bold; margin-top: 0;">sender</strong> is using kTLS, not if the sender is using userspace TLS and the receiver is using kTLS. This seems to indicate the problem happens on the sending side.</p>
<p dir="auto" style="color: #333238; margin: 0 0 16px;" align="initial">I also used strace to show that it is a desynchronisation problem, because we can see the patterns produced by <a href="https://libguestfs.org/nbdkit-pattern-plugin.1.html" rel="nofollow noreferrer noopener" target="_blank" style="margin-top: 0;">https://libguestfs.org/nbdkit-pattern-plugin.1.html</a> in what is supposed to be an NBD reply header.</p>
<h2 dir="auto" style="font-size: 1.5em; font-weight: 600; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-color: #e6e6ea; border-bottom-style: solid; color: #333238; margin: 24px 0 16px;" align="initial">
<a id="user-content-version-of-gnutls-used" class="anchor" href="#version-of-gnutls-used" aria-hidden="true" style="margin-top: 0; float: left; margin-left: -20px; text-decoration: none; outline: none;"></a>Version of gnutls used:</h2>
<p dir="auto" style="color: #333238; margin: 0 0 16px;" align="initial">gnutls @ <a href="https://gitlab.com/gnutls/gnutls/-/commit/496b4bb357adfbaadf068377bdad5b9cc51f5128" data-reference-type="commit" data-original="496b4bb357adfbaa" data-link="false" data-link-reference="false" data-project="179611" data-commit="496b4bb357adfbaadf068377bdad5b9cc51f5128" data-container="body" data-placement="top" title="Merge branch 'master' into 'master'" class="gfm gfm-commit has-tooltip" style='font-family: var(--default-mono-font, "Menlo"),"DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; font-variant-ligatures: none; font-size: 95%; margin-top: 0;'>496b4bb3</a>
kernel 6.2.0-0.rc7.20230206gitd2d11f342b17.50.fc38.x86_64</p>
<h2 dir="auto" style="font-size: 1.5em; font-weight: 600; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-color: #e6e6ea; border-bottom-style: solid; color: #333238; margin: 24px 0 16px;" align="initial">
<a id="user-content-distributor-of-gnutls-eg-ubuntu-fedora-rhel" class="anchor" href="#distributor-of-gnutls-eg-ubuntu-fedora-rhel" aria-hidden="true" style="margin-top: 0; float: left; margin-left: -20px; text-decoration: none; outline: none;"></a>Distributor of gnutls (e.g., Ubuntu, Fedora, RHEL)</h2>
<p dir="auto" style="color: #333238; margin: 0;" align="initial">Fedora, but self-built.</p>
</div>
</div>
<div class="footer" style="margin-top: 10px;">
<p style="font-size: small; color: #737278;">
—
<br>
Reply to this email directly or <a href="https://gitlab.com/gnutls/gnutls/-/issues/1470">view it on GitLab</a>.
<br>
You're receiving this email because of your account on <a target="_blank" rel="noopener noreferrer" href="https://gitlab.com">gitlab.com</a>. <a href="https://gitlab.com/-/sent_notifications/4b8e934fad1b46eda91ae7fcc2ad95c8/unsubscribe" target="_blank" rel="noopener noreferrer">Unsubscribe</a> from this thread · <a href="https://gitlab.com/-/profile/notifications" target="_blank" rel="noopener noreferrer" class="mng-notif-link">Manage all notifications</a> · <a href="https://gitlab.com/help" target="_blank" rel="noopener noreferrer" class="help-link">Help</a>
<script type="application/ld+json">{"@context":"http://schema.org","@type":"EmailMessage","action":{"@type":"ViewAction","name":"View Issue","url":"https://gitlab.com/gnutls/gnutls/-/issues/1470"}}</script>
</p>
</div>
</body>
</html>