diff --git a/README.md b/README.md index bebafc2..0dfce61 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,7 @@ Organized by area of interest, with a focus on machine learning and AI: - **DevOps & Tools**: Git commands, configurations, and workflows - **Roadmap**: A competency matrix tracking progress toward Senior - **Troubleshooting**: Real problems hit and how they got fixed +- **Achievements**: Things I built or improved, with the numbers - **And more, as new areas come up!** ## Features diff --git a/docs/achievements/draftr-p95-latency-background-s3-deletes.md b/docs/achievements/draftr-p95-latency-background-s3-deletes.md new file mode 100644 index 0000000..5b044a5 --- /dev/null +++ b/docs/achievements/draftr-p95-latency-background-s3-deletes.md @@ -0,0 +1,111 @@ +--- +tags: + +- achievements +- python +- fastapi +- aws +- s3 +- observability +- grafana +- performance + +--- + +# Draftr: Cut a Slow Endpoint's p95 from ~2.3 s to ~0.24 s by Moving S3 Deletes to a Background Task + +Used the telemetry FastAPI/OpenTelemetry gives almost for free, wired it into Grafana, and +spotted that one endpoint (`PUT /api/drawings/{id}/commit`) had a p95 around 2.3 s while +every other route was far faster. The cause was synchronous S3 `DeleteObject` calls on the +request path; moving them to a FastAPI `BackgroundTask` dropped that endpoint's p95 to +roughly 0.24 s. + +- **Project:** [draftr](https://github.com/igormcsouza/draftr) — private Excalidraw + workspace (Next.js + FastAPI on AWS serverless) +- **Date:** 2026-10-04 +- **Role:** sole developer — instrumentation, diagnosis, fix + +## Context + +Draftr's backend is a FastAPI app on AWS Lambda (container image, behind the Lambda Web +Adapter running `uvicorn`), storing drawing scenes and thumbnails in S3 and metadata in +a database. Traces, metrics, and logs go over OTLP to Grafana Cloud. + +`/commit` is the save endpoint: the client uploads the new scene (and thumbnail) to S3 +with presigned URLs, then calls `/commit` to atomically point the drawing at the new +version. Once committed, the *previous* scene/thumbnail objects are no longer needed and +get deleted. + +## Problem + +After wiring the telemetry into Grafana I looked at p95 latency per route. Everything was +comfortably fast except `/commit`: + +| Route | p95 (around deploy time) | +| --- | --- | +| `PUT /api/drawings/{id}/commit` | **~2.3 s** | +| `GET /api/drawings/{id}` | ~0.09 s | +| `GET /api/library` | ~0.04 s | +| `POST /api/drawings/{id}/upload-urls` | ~0.69 s | + +(Pulled from Grafana Cloud Prometheus, `http_server_request_duration_seconds_bucket`. +Low traffic, so these are small samples — directionally right, not benchmark-grade.) + +One outlier route is exactly what per-route percentiles are for: an average over all +endpoints would have hidden it. + +## What I Did + +Read the handler. After the database commit succeeded, it deleted the superseded objects +from S3 inline, before returning: + +```python +# before: the client waits for S3 deletes it doesn't care about +blobs.delete(*(k for k in (old_scene, old_thumb) if k not in (body.scene_key, body.thumb_key))) +return {"version": version} +``` + +The client doesn't need the delete to finish to know the commit succeeded — the data is +already safe. So the delete is cleanup, not part of the answer. FastAPI's +`BackgroundTasks` runs a callable after the response is sent: + +```python +@router.put("/drawings/{id}/commit") +def commit(id: str, body: s.Commit, bg: BackgroundTasks, ...): + ... + # after the response + bg.add_task(blobs.delete, *(k for k in (old_scene, old_thumb) if k not in (body.scene_key, body.thumb_key))) + return {"version": version} +``` + +Applied the same change to `set_thumb`, which had the identical pattern. Commit: +[`38704ed`](https://github.com/igormcsouza/draftr/commit/38704ed) ("Run commit/thumb S3 +deletes after the response"). + +## Result + +Same endpoint, p95 in Grafana (20-minute rate window, 2026-10-04): + +- Before the change: **~2.3 s** (2.28–2.35 s) +- After the change: **~0.24 s** + +Roughly a 10× improvement on the one slow endpoint, with a three-line diff per handler. + +## Takeaways + +- **Per-route percentiles find what averages hide.** One slow route among fast ones only + shows up when you break latency down by route and look at p95, not the mean. +- **Ask whether the client needs the work to finish.** Cleanup, notifications, and cache + invalidation usually don't belong on the response path. +- **Trade-off I accepted:** background deletes are best-effort. On Lambda the execution + environment can be frozen after the response is returned, so a delete may be delayed or + skipped, leaving an orphaned object. That's acceptable here because orphans are only + wasted storage (S3 versioning plus a 7-day retention rule keeps recovery possible), but + it would not be acceptable for work that must happen. For that, use a queue. +- **Interview version:** "I added observability, found one endpoint 5-10× slower than the + rest from its p95, traced it to synchronous S3 deletes on the request path, and moved + them off it — p95 went from ~2.3 s to ~0.24 s." + +## Related Articles + +- [Troubleshooting](../troubleshooting/index.md) diff --git a/docs/achievements/index.md b/docs/achievements/index.md new file mode 100644 index 0000000..cb780bc --- /dev/null +++ b/docs/achievements/index.md @@ -0,0 +1,84 @@ +--- +title: Achievements +tags: + +- achievements +- overview + +--- + +# Achievements + +A running log of things I built, fixed, or improved that are worth remembering — with the +numbers and the reasoning attached. The point is simple: when someone asks "what have you +done?" or "tell me about a time you...", the answer is a search away instead of a +half-remembered story. + +This is different from [Troubleshooting](../troubleshooting/index.md) (a bug I hit and how +the fix works, written so it's findable by symptom) and from the +[Roadmap](../roadmap/index.md) (progress against the Senior competency matrix). An +achievement entry is written from the *outcome*: what changed, by how much, and what I'd +say about it in an interview. + +## When to Add an Entry + +Add one whenever something ships that I'd be happy to talk about later — a measurable +improvement, a hard problem solved, a project milestone. Capture it the same day, while +the numbers are still in the dashboards. + +## How to Add One + +1. Create a file directly under `docs/achievements/`, named for the outcome (e.g. + `draftr-p95-latency-background-s3-deletes.md`). + +1. Use the structure below: Summary, Context, Problem, What I Did, Result, Takeaways. + +1. Add tags (at minimum `achievements`, plus the relevant technologies). + +1. Link the entry from the [Entries](#entries) list below, newest first. + +### Entry Template + +````markdown +--- +tags: + +- achievements +- technology + +--- + +# Outcome-Focused Title + +One-line summary with the headline number. + +- **Project:** name (link to the repo) +- **Date:** YYYY-MM-DD +- **Role:** what I owned + +## Context + +What the system is and why this mattered. + +## Problem + +What was wrong, and how I noticed it. + +## What I Did + +The investigation and the change. + +## Result + +Before/after numbers, with the source (dashboard, query, time window). + +## Takeaways + +What I learned, trade-offs accepted, and how I'd tell this story in an interview. +```` + +## Entries + +Newest first. + +- [Draftr: Cut a Slow Endpoint's p95 from ~2.3 s to ~0.24 s by Moving S3 Deletes to a Background Task](draftr-p95-latency-background-s3-deletes.md) — 2026-10-04 diff --git a/docs/contributing.md b/docs/contributing.md index 74ba22c..3c54126 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -19,6 +19,9 @@ the right folder," something's wrong — fix the process, not the habit. Is it a specific bug/error you hit and fixed? → docs/troubleshooting/ (see its own template) +Is it something you built/improved that you'll want to recall later (with numbers)? + → docs/achievements/ (see its own template) + Is it a progress update on your Senior-engineer roadmap? → docs/roadmap/index.md (edit the matrix + add a Day X entry) @@ -162,11 +165,14 @@ to get subtly wrong in raw markdown. ## 8. Special Sections -Two sections have their own dedicated format, different from a standard topic article: +Three sections have their own dedicated format, different from a standard topic article: - **[Troubleshooting](troubleshooting/index.md)** — one file per fixed problem, using [the entry template in the troubleshooting index](troubleshooting/index.md#how-to-add-one) (Problem / Environment / Root Cause / Fix / Prevention). +- **[Achievements](achievements/index.md)** — one file per accomplishment, using + [the entry template in the achievements index](achievements/index.md#how-to-add-one) + (Context / Problem / What I Did / Result / Takeaways). - **[Roadmap](roadmap/index.md)** — a single page you edit in place (competency matrix + Day X log), not a new file per update. diff --git a/docs/index.md b/docs/index.md index 18b5044..197759c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -17,6 +17,7 @@ specialized in machine learning and AI: - **[DevOps & Tools](knowledge/devops-tools/index.md)**: Git, tooling, and workflows - **[Roadmap](roadmap/index.md)**: A competency matrix tracking progress toward Senior - **[Troubleshooting](troubleshooting/index.md)**: Real problems hit and how they got fixed +- **[Achievements](achievements/index.md)**: Things I built or improved, with the numbers - **And more, as new areas come up!** ## Navigation