Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added static/img/ui-capture/ui-capture-flow-detail.webp
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added static/img/ui-capture/ui-capture-flows.webp
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added static/img/ui-capture/ui-capture-key-created.webp
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added static/img/ui-capture/ui-capture-setup-card.webp
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added static/img/ui-capture/ui-capture-status.webp
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,5 @@ Linux:
| <img src="https://th.bing.com/th/id/R.7802b52b7916c00014450891496fe04a?rik=r8GZM4o2Ch1tHQ&riu=http%3a%2f%2f1000logos.net%2fwp-content%2fuploads%2f2017%2f03%2fLINUX-LOGO.png&ehk=5m0lBvAd%2bzhvGg%2fu4i3%2f4EEHhF4N0PuzR%2fBmC1lFzfw%3d&risl=&pid=ImgRaw&r=0" width="10" height="10" alt="Linux" /> **Linux** | <img src="https://upload.wikimedia.org/wikipedia/commons/e/e5/Green_tick_pointed.svg" width="20" height="20" alt="Supported" /> | <img src="https://upload.wikimedia.org/wikipedia/commons/e/e5/Green_tick_pointed.svg" width="20" height="20" alt="Supported" /> | Linux kernel 5.10 or higher |

Keploy runs natively on macOS (Apple Silicon / arm64) and Windows (x86‑64) — neither has eBPF, so it intercepts traffic in userspace on both, and neither needs `sudo` or Administrator. Natively on macOS and Windows, Keploy understands HTTP/HTTPS, MySQL and MongoDB calls; calls to other services — PostgreSQL, Redis, Kafka, gRPC and the like — are captured only as raw bytes and usually don't replay, so for an app that runs in containers or depends on them, use Docker. Docker, Lima (macOS) and WSL (Windows) remain supported alternatives, and WSL is the route on Windows on ARM. On an Intel Mac use Lima: the Docker route on macOS still runs the native CLI on the host, so it is Apple Silicon only too.

Low-latency recording (`keploy record --low-latency`, or a Kubernetes Sidecar with `low_latency_mode`) and the Kubernetes DaemonSet agent capture with eBPF hooks that arm64 kernels support only from 6.4 (x86-64 kernels from 5.10). On an older arm64 kernel, record without low latency, and use Sidecar mode without `low_latency_mode` in Kubernetes.
22 changes: 22 additions & 0 deletions versioned_docs/version-4.0.0/keploy-explained/common-errors.md
Original file line number Diff line number Diff line change
Expand Up @@ -224,12 +224,34 @@ The same setup connects normally when you run it without Keploy.

See [MySQL port detection](../running-keploy/configuration-file.md#mysql-port-detection) for how Keploy identifies the protocol during Record and recovers the port during Test.

### 13. Capture loss warning: "proxyless capture could not record an app's bytes"

#### Description:

While recording with `keploy record --low-latency`, a Kubernetes Sidecar with `low_latency_mode`, or the Kubernetes DaemonSet agent, Keploy logs:

```
proxyless capture could not record an app's bytes: the app sent them with sendfile(2), ...
```

The recording is missing those bytes, and the test cases or mocks that carried them may be incomplete.

#### Possible Cause:

- The app moved the bytes without copying them through its own memory: `sendfile(2)` (for example nginx `sendfile on`, Tomcat's default sendfile, or Go's `http.ServeFile`), `splice(2)` (for example Go's `io.Copy` between two connections), or io_uring registered buffers.
- Less often: a `MSG_TRUNC` receive, which needs no change, or a page Keploy's hook could not read, logged as an error.

#### Solution:

- Follow the `toRecordIt` field, if the line has one. [Capture loss warnings](../running-keploy/capture-loss.md), which the line's `docs` field links, describes each cause and what to do about it.

If you’re still encountering issues after trying these solutions, feel free to reach out to the Keploy team on [Slack](https://keploy.io/slack).

Happy Testing!

## Related

- [Capture loss warnings](/docs/running-keploy/capture-loss/) — bytes the eBPF capture cannot record, and how to record them.
- [Debugger Guide](/docs/keploy-explained/debugger-guide/) — debug Keploy with the VS Code debugger.
- [Running Keploy on Windows in WSL](/docs/keploy-explained/windows-wsl/) — WSL setup and fixes.
- [Running Keploy on macOS in a Linux VM (Lima)](/docs/keploy-explained/mac-linux/) — macOS setup with Lima.
Expand Down
3 changes: 3 additions & 0 deletions versioned_docs/version-4.0.0/quickstart/k8s-proxy.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,7 @@ The Keploy Proxy supports two ways to capture traffic from your application Pods
| What happens on `Start Recording` | The proxy injects the agent and rolls the application Deployment. | The proxy creates a `RecordingSession` Custom Resource. The DaemonSet picks it up and programs its BPF target maps to capture matching Pods on each node. |
| Pod mutation on the application namespace | Required (`patch` on Deployments). | **Not required.** Application Pods are never modified. |
| Application restart at recording start | Yes, on first recording. | No. |
| Node kernel | Linux 5.10 or later. Low-latency mode needs 6.4 or later on arm64. | Linux 5.10 or later on x86-64, 6.4 or later on arm64. |
| Best for | Dev/staging, teams happy to grant write RBAC to Keploy on the application namespace. | Production with read-only RBAC on the application namespace; environments where rolling the application Pod has unacceptable cost; or when you want cluster-mode auto-replay (replay runs in a separate cluster you provide). |

The screenshots below show the **Sidecar** flow because that is the default. To use **DaemonSet** mode instead, set the daemonset values when you run the Helm command in step 4 below—every other step is identical.
Expand Down Expand Up @@ -244,6 +245,8 @@ If you want to use **DaemonSet mode** instead of the default Sidecar mode, appen
--set daemonset.crds.install=true
```

The DaemonSet's eBPF capture needs a node kernel of 5.10 or later on x86-64, and 6.4 or later on arm64. On arm64 nodes with an older kernel, use Sidecar mode without low-latency mode.

After install you should see a per-node `k8s-proxy-daemonset-*` Pod alongside the regular proxy Deployment:

```bash
Expand Down
211 changes: 211 additions & 0 deletions versioned_docs/version-4.0.0/running-keploy/capture-loss.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,211 @@
---
id: capture-loss
title: Fix "could not record an app's bytes" warnings
sidebar_label: Capture loss warnings
description: Fix the Keploy warning that it could not record an app's bytes — the sendfile, splice, io_uring and MSG_TRUNC calls behind it, and what to change in the app.
tags:
- troubleshooting
- recording
- ebpf
- kubernetes
keywords:
- keploy capture loss
- could not record an app's bytes
- payloads are missing from the recordings
- sendfile
- splice
- io_uring
- nginx sendfile
- keploy low latency recording
- keploy daemonset recording
---

When Keploy records with its eBPF capture (`keploy record --low-latency`, a
Kubernetes Sidecar with `low_latency_mode`, and the DaemonSet agent), it
records what your app's send and receive calls copy between the app's memory
and its sockets. A few calls move bytes without copying them through the
app's memory, so there is nothing for Keploy to read. Keploy does not record
those bytes. It marks the gap on the connection so the protocol parser
resyncs past it (or stops on that connection) instead of reading garbage, and
it tells you once per app and cause in a warning like this:

```text
🐰 Keploy: 2026-10-07T10:06:43.043391152Z WARN proxyless capture could not record an app's bytes: the app sent them with sendfile(2), which moves a file's pages straight into the socket — those bytes are missing from the recording; the gap is marked on the connection so its parser resyncs or stops there {"pod": "web-7c9d6b5f4-x2kqp", "process": "nginx", "pid": 4242, "connID": 2563235, "connection": "10.0.0.5:8080", "direction": "response", "lostBytes": 16384, "ktls": false, "toRecordIt": "turn sendfile off for traffic Keploy records — nginx: sendfile off; …", "docs": "https://keploy.io/docs/running-keploy/capture-loss/#sendfile"}
```

The fields:

- `pod`, `process` and `pid` name the app. `pod` is empty outside the
Kubernetes DaemonSet agent.
- `connection` is the server end of the connection: the address the app
dialed, or, on a connection the app accepted, its own listening address.
`connID` is Keploy's ID for the connection.
- `direction`, on a TCP connection, is `request` (client to server) or
`response` (server to client). On a connection using the kernel's TLS
(`"ktls": true`), it is the app's `sent` or `received`. Page references
also come from UDP and Unix socket calls, and `MSG_TRUNC` from UDP; there
`direction` is the call's (`request` for a send, `response` for a receive)
and `connection` may not be a server's address.
- `lostBytes` is the size of the one call that triggered the warning. Later
losses are counted in the periodic warning below.
- `toRecordIt` says what to change so the bytes are recorded.
- `docs` links the section of this page on the cause.

Every minute in which more bytes were lost, Keploy also logs a periodic
warning for each cause, across all apps:

```text
🐰 Keploy: 2026-10-07T10:07:43.043391152Z WARN proxyless capture: payloads are missing from the recordings: the app sent them with sendfile(2), which moves a file's pages straight into the socket; each was marked lost on its connection, whose parser resyncs or stops there {"newlyLost": 3, "lost": 12, "lostBytes": 196608, "toRecordIt": "turn sendfile off for traffic Keploy records — nginx: sendfile off; …", "docs": "https://keploy.io/docs/running-keploy/capture-loss/#sendfile"}
```

`newlyLost` counts the calls since the last report, and `lost` and `lostBytes`
count every call and byte since the agent started.

## Find the cause and fix it

Find the words your warning uses, and go to that section:

| The warning says | Section |
| --------------------------------------------------------------- | --------------------------------------------------------------------------- |
| "the app sent them with sendfile(2)" | [Turn off sendfile](#sendfile) |
| "the app moved them from a pipe into the socket with splice(2)" | [Copy through a buffer instead of splicing into the socket](#splice-send) |
| "the app took them out of the socket into a pipe" | [Copy through a buffer instead of splicing out of the socket](#splice-recv) |
| "a send or receive of the app's moved them by page references" | [Use plain send and receive calls](#kernel-pages) |
| "the app received them with MSG_TRUNC" | [Leave MSG_TRUNC receives as they are](#msg-trunc) |
| "their page was not resident to the capture hook" | [Report a page-not-resident error](#read-fault) |
| "Failed to attach do_splice_direct and do_splice", at startup | [Check for unmarked losses at startup](#unmarked) |
| "Failed to attach sock_splice_read", at startup | [Check for unmarked losses at startup](#unmarked) |

### Turn off sendfile {#sendfile}

The app sent a file's pages straight into the socket, so they were never in
its memory. Turn sendfile off for the traffic Keploy records:

- nginx: `sendfile off;`
- Apache: `EnableSendfile Off`
- Tomcat: `useSendfile="false"` on the Connector (on by default)
- gunicorn: `--no-sendfile`
- aiohttp: set `AIOHTTP_NOSENDFILE=1`
- Java: don't use `FileChannel.transferTo` or Netty's `FileRegion` for this
traffic.
- Go, on plain TCP (`crypto/tls` never uses sendfile). These use sendfile:
`http.ServeFile`, `http.FileServer` over `http.Dir`, `http.ServeContent` of
an `*os.File`, `io.Copy` from an `*os.File` into a `net.Conn`, or into an
`http.ResponseWriter` whose response is not chunked (`Content-Length` is
set, or the client speaks HTTP/1.0), and an `http.Client` request whose
`Body` is an `*os.File` with `ContentLength` set. `io.CopyN` and
`io.CopyBuffer` take the same path; passing your own buffer does not
avoid it. Hide the socket or the file from the copy:

```go
// http.ServeFile
http.ServeFile(struct{ http.ResponseWriter }{w}, r, name)

// http.FileServer
h := http.FileServer(root)
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
h.ServeHTTP(struct{ http.ResponseWriter }{w}, r)
})

// io.Copy from a file, into a connection or a ResponseWriter
io.Copy(struct{ io.Writer }{conn}, f)
io.Copy(struct{ io.Writer }{w}, f)

// an http.Client request body read from a file (io.NopCloser(f) alone is
// not enough: net/http unwraps it)
req.Body = io.NopCloser(struct{ io.Reader }{f})
```

### Copy through a buffer instead of splicing into the socket {#splice-send}

The app moved bytes from a pipe into the socket with `splice(2)` or io_uring's
`IORING_OP_SPLICE`. Copy through a buffer instead:

- Go, never on `crypto/tls`: `io.Copy` into a TCP connection from another TCP
or Unix stream connection splices. Hide both ends from it:
`io.Copy(struct{ io.Writer }{dst}, struct{ io.Reader }{src})`.
- HAProxy: leave `option splice-*` off.

### Copy through a buffer instead of splicing out of the socket {#splice-recv}

The app took received bytes out of the socket with `splice(2)` or
`sendfile(2)`, typically on to a file or another socket, without ever holding
them in its own memory. Copy through a buffer instead:

- Go, never on `crypto/tls`: `io.Copy` from a TCP connection into another TCP
or Unix stream connection, as a forwarding proxy does, or into an
`*os.File`, splices. Hide both ends from it:
`io.Copy(struct{ io.Writer }{dst}, struct{ io.Reader }{src})`.
- HAProxy: leave `option splice-*` off.

### Use plain send and receive calls {#kernel-pages}

The bytes moved by page references the kernel holds, not through the app's
memory. This happens with io_uring registered (fixed) buffers. It also
happens, from kernel 6.5, with a sendfile or splice send when Keploy could not
attach its splice hooks (see [Check for unmarked losses at startup](#unmarked)).

Use plain `send`/`recv`, or io_uring without registered buffers, for the
traffic Keploy records. For sendfile or splice, turn them off as described
above.

### Leave MSG_TRUNC receives as they are {#msg-trunc}

The app received with `MSG_TRUNC`, which discards the bytes without copying
them. There is nothing to change: the app never read them, and the
connection's parser resyncs past the gap.

### Report a page-not-resident error {#read-fault}

This one is logged as an error, once per agent rather than once per app: a
page of the payload could not be read without faulting it in, which Keploy's
hook cannot do. This is a limitation of the capture, not something the
app chose. The error names no app; its fields are `shard`, `connID`,
`direction` (`0` for request, `1` for response), `lostBytes`, `destPort`,
`ktls` and `docs`.

If it recurs, report it with the agent's logs on
[Slack](https://keploy.io/slack) or as a
[GitHub issue](https://github.com/keploy/keploy/issues).

## Check for unmarked losses at startup {#unmarked}

Keploy marks sendfile and splice losses through hooks on the kernel's sendfile
and splice paths. When one of them cannot attach, the agent warns once at
startup, and some losses are then not marked at all, with no warning for each
connection:

- `Failed to attach do_splice_direct and do_splice`: before kernel 6.5, what
sendfile and splice send goes unmarked. From 6.5, the kernel hands those
bytes to Keploy's data hooks, which mark them as
[page references](#kernel-pages).
- `Failed to attach sock_splice_read`: what splice takes out of a socket goes
unmarked, on any kernel.

If you see either, turn sendfile and splice off for the traffic Keploy
records, as described above, even when no capture loss warning follows.

## Know what TLS changes

TLS through a library (OpenSSL, Go's `crypto/tls`, Java's JSSE) encrypts in
the app's memory, so it does not take the sendfile or splice paths, unless the
library hands encryption to the kernel. OpenSSL 3 does that with its kTLS
option, for example nginx with `ssl_conf_command Options KTLS;`. That is the
kernel's TLS (kTLS): Keploy marks those bytes on the kTLS stream
(`"ktls": true`), and turning sendfile off applies there too.

## Understand why these bytes cannot be read

sendfile and splice move page references inside the kernel: from a file's
page cache, or from a socket's receive queue, into a pipe or another socket.
The bytes never pass through a buffer of the app's that Keploy's hook could
read. Keploy hooks these calls on every kernel its eBPF capture runs on
(x86-64 from 5.10, arm64 from 6.4). Where the hooks attach (the agent warns at
startup when one does not), Keploy knows exactly which bytes were lost and on
which connection, and says so instead of recording a stream with silent holes.

## Related

- [Troubleshooting guide](../keploy-explained/common-errors.md)
- [DaemonSet recording architecture](k8s-proxy-daemonset-architecture.md)
- [Platform requirements](../concepts/platform-requirements.md)
Loading
Loading