Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ Organized by area of interest, with a focus on machine learning and AI:
- **DevOps & Tools**: Git commands, configurations, and workflows
- **Roadmap**: A competency matrix tracking progress toward Senior
- **Troubleshooting**: Real problems hit and how they got fixed
- **Achievements**: Things I built or improved, with the numbers
- **And more, as new areas come up!**

## Features
Expand Down
111 changes: 111 additions & 0 deletions docs/achievements/draftr-p95-latency-background-s3-deletes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
---
tags:

- achievements
- python
- fastapi
- aws
- s3
- observability
- grafana
- performance

---

# Draftr: Cut a Slow Endpoint's p95 from ~2.3 s to ~0.24 s by Moving S3 Deletes to a Background Task

Used the telemetry FastAPI/OpenTelemetry gives almost for free, wired it into Grafana, and
spotted that one endpoint (`PUT /api/drawings/{id}/commit`) had a p95 around 2.3 s while
every other route was far faster. The cause was synchronous S3 `DeleteObject` calls on the
request path; moving them to a FastAPI `BackgroundTask` dropped that endpoint's p95 to
roughly 0.24 s.

- **Project:** [draftr](https://github.com/igormcsouza/draftr) — private Excalidraw
workspace (Next.js + FastAPI on AWS serverless)
- **Date:** 2026-10-04
- **Role:** sole developer — instrumentation, diagnosis, fix

## Context

Draftr's backend is a FastAPI app on AWS Lambda (container image, behind the Lambda Web
Adapter running `uvicorn`), storing drawing scenes and thumbnails in S3 and metadata in
a database. Traces, metrics, and logs go over OTLP to Grafana Cloud.

`/commit` is the save endpoint: the client uploads the new scene (and thumbnail) to S3
with presigned URLs, then calls `/commit` to atomically point the drawing at the new
version. Once committed, the *previous* scene/thumbnail objects are no longer needed and
get deleted.

## Problem

After wiring the telemetry into Grafana I looked at p95 latency per route. Everything was
comfortably fast except `/commit`:

| Route | p95 (around deploy time) |
| --- | --- |
| `PUT /api/drawings/{id}/commit` | **~2.3 s** |
| `GET /api/drawings/{id}` | ~0.09 s |
| `GET /api/library` | ~0.04 s |
| `POST /api/drawings/{id}/upload-urls` | ~0.69 s |

(Pulled from Grafana Cloud Prometheus, `http_server_request_duration_seconds_bucket`.
Low traffic, so these are small samples — directionally right, not benchmark-grade.)

One outlier route is exactly what per-route percentiles are for: an average over all
endpoints would have hidden it.

## What I Did

Read the handler. After the database commit succeeded, it deleted the superseded objects
from S3 inline, before returning:

```python
# before: the client waits for S3 deletes it doesn't care about
blobs.delete(*(k for k in (old_scene, old_thumb) if k not in (body.scene_key, body.thumb_key)))
return {"version": version}
```

The client doesn't need the delete to finish to know the commit succeeded — the data is
already safe. So the delete is cleanup, not part of the answer. FastAPI's
`BackgroundTasks` runs a callable after the response is sent:

```python
@router.put("/drawings/{id}/commit")
def commit(id: str, body: s.Commit, bg: BackgroundTasks, ...):
...
# after the response
bg.add_task(blobs.delete, *(k for k in (old_scene, old_thumb) if k not in (body.scene_key, body.thumb_key)))
return {"version": version}
```

Applied the same change to `set_thumb`, which had the identical pattern. Commit:
[`38704ed`](https://github.com/igormcsouza/draftr/commit/38704ed) ("Run commit/thumb S3
deletes after the response").

## Result

Same endpoint, p95 in Grafana (20-minute rate window, 2026-10-04):

- Before the change: **~2.3 s** (2.28–2.35 s)
- After the change: **~0.24 s**

Roughly a 10× improvement on the one slow endpoint, with a three-line diff per handler.

## Takeaways

- **Per-route percentiles find what averages hide.** One slow route among fast ones only
shows up when you break latency down by route and look at p95, not the mean.
- **Ask whether the client needs the work to finish.** Cleanup, notifications, and cache
invalidation usually don't belong on the response path.
- **Trade-off I accepted:** background deletes are best-effort. On Lambda the execution
environment can be frozen after the response is returned, so a delete may be delayed or
skipped, leaving an orphaned object. That's acceptable here because orphans are only
wasted storage (S3 versioning plus a 7-day retention rule keeps recovery possible), but
it would not be acceptable for work that must happen. For that, use a queue.
- **Interview version:** "I added observability, found one endpoint 5-10× slower than the
rest from its p95, traced it to synchronous S3 deletes on the request path, and moved
them off it — p95 went from ~2.3 s to ~0.24 s."

## Related Articles

- [Troubleshooting](../troubleshooting/index.md)
84 changes: 84 additions & 0 deletions docs/achievements/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
---
title: Achievements
tags:

- achievements
- overview

---

# Achievements

A running log of things I built, fixed, or improved that are worth remembering — with the
numbers and the reasoning attached. The point is simple: when someone asks "what have you
done?" or "tell me about a time you...", the answer is a search away instead of a
half-remembered story.

This is different from [Troubleshooting](../troubleshooting/index.md) (a bug I hit and how
the fix works, written so it's findable by symptom) and from the
[Roadmap](../roadmap/index.md) (progress against the Senior competency matrix). An
achievement entry is written from the *outcome*: what changed, by how much, and what I'd
say about it in an interview.

## When to Add an Entry

Add one whenever something ships that I'd be happy to talk about later — a measurable
improvement, a hard problem solved, a project milestone. Capture it the same day, while
the numbers are still in the dashboards.

## How to Add One

1. Create a file directly under `docs/achievements/`, named for the outcome (e.g.
`draftr-p95-latency-background-s3-deletes.md`).

1. Use the structure below: Summary, Context, Problem, What I Did, Result, Takeaways.

1. Add tags (at minimum `achievements`, plus the relevant technologies).

1. Link the entry from the [Entries](#entries) list below, newest first.

### Entry Template

````markdown
---
tags:

- achievements
- technology

---

# Outcome-Focused Title

One-line summary with the headline number.

- **Project:** name (link to the repo)
- **Date:** YYYY-MM-DD
- **Role:** what I owned

## Context

What the system is and why this mattered.

## Problem

What was wrong, and how I noticed it.

## What I Did

The investigation and the change.

## Result

Before/after numbers, with the source (dashboard, query, time window).

## Takeaways

What I learned, trade-offs accepted, and how I'd tell this story in an interview.
````

## Entries

Newest first.

- [Draftr: Cut a Slow Endpoint's p95 from ~2.3 s to ~0.24 s by Moving S3 Deletes to a Background Task](draftr-p95-latency-background-s3-deletes.md) — 2026-10-04
8 changes: 7 additions & 1 deletion docs/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ the right folder," something's wrong — fix the process, not the habit.
Is it a specific bug/error you hit and fixed?
→ docs/troubleshooting/ (see its own template)

Is it something you built/improved that you'll want to recall later (with numbers)?
→ docs/achievements/ (see its own template)

Is it a progress update on your Senior-engineer roadmap?
→ docs/roadmap/index.md (edit the matrix + add a Day X entry)

Expand Down Expand Up @@ -162,11 +165,14 @@ to get subtly wrong in raw markdown.

## 8. Special Sections

Two sections have their own dedicated format, different from a standard topic article:
Three sections have their own dedicated format, different from a standard topic article:

- **[Troubleshooting](troubleshooting/index.md)** — one file per fixed problem, using
[the entry template in the troubleshooting index](troubleshooting/index.md#how-to-add-one)
(Problem / Environment / Root Cause / Fix / Prevention).
- **[Achievements](achievements/index.md)** — one file per accomplishment, using
[the entry template in the achievements index](achievements/index.md#how-to-add-one)
(Context / Problem / What I Did / Result / Takeaways).
- **[Roadmap](roadmap/index.md)** — a single page you edit in place (competency matrix +
Day X log), not a new file per update.

Expand Down
1 change: 1 addition & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ specialized in machine learning and AI:
- **[DevOps & Tools](knowledge/devops-tools/index.md)**: Git, tooling, and workflows
- **[Roadmap](roadmap/index.md)**: A competency matrix tracking progress toward Senior
- **[Troubleshooting](troubleshooting/index.md)**: Real problems hit and how they got fixed
- **[Achievements](achievements/index.md)**: Things I built or improved, with the numbers
- **And more, as new areas come up!**

## Navigation
Expand Down
Loading