Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ weights. Candidate generation, letterbox
`content`, and the retention-gated ranking path are checked against FocalNet
Python goldens.

`make model-focalnet` downloads FocalNet weights for local runs. They are not
required for the default tests.

To also run the real YuNet and U²-Net models:

```sh
Expand Down
55 changes: 27 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,19 @@

`autogravity` is a small Go HTTP service that finds the best crop focus in an
image. It prioritizes a confidently detected face using YuNet, then falls back
to the weighted centroid of the strongest U²-Net salient region. Results are
normalized X/Y coordinates. It never crops, stores, identifies, or modifies the
submitted image.
to the weighted centroid of the strongest U²-Net salient region. Set
`MODEL_BACKEND=focalnet` to use [FocalNet](https://github.com/appwrite/focalnet),
Appwrite's own model, instead. Results are normalized X/Y coordinates. It never
crops, stores, identifies, or modifies the submitted image.

## Requirements

- Go 1.25 or newer
- The included INT8 U²-Net model (approximately 42 MiB) and YuNet face detector
(approximately 230 KiB). `make model` verifies them and downloads/verifies the
FP32 U²-Net fallback (approximately 168 MiB).
- Optional: Appwrite's FocalNet model (approximately 19 MiB). Download it with
`make model-focalnet`.
- An ONNX Runtime shared library. Version 1.23.2 is used by the Docker image and
matches the pinned Go binding.

Expand All @@ -32,14 +35,18 @@ export ONNXRUNTIME_LIB=/absolute/path/to/libonnxruntime.dylib # macOS
make model
make build
./autogravity

# Optional: Appwrite's FocalNet backend
make model-focalnet
MODEL_BACKEND=focalnet ./autogravity
```

The server listens on `:8080`. These environment variables are available:

| Variable | Default | Purpose |
| --- | --- | --- |
| `ADDR` | `:8080` | HTTP listen address |
| `MODEL_BACKEND` | `u2net` | `u2net` (YuNet + U²-Net) or `focalnet` (published human-ranking model) |
| `MODEL_BACKEND` | `u2net` | `u2net` (YuNet + U²-Net) or `focalnet` (Appwrite's model) |
| `MODEL_PRECISION` | `int8` | `int8` or `fp32` for the U²-Net backend |
| `MODEL_PATH` | unset | Explicit model path; overrides `MODEL_PRECISION` / FocalNet default |
| `FACE_MODEL_PATH` | `models/face_detection_yunet_2023mar.onnx` | YuNet face-detection model path (U²-Net backend only) |
Expand Down Expand Up @@ -98,11 +105,10 @@ the tighter constraint.

The image bundles the verified INT8 and face models and downloads the verified
FP32 model and CPU-only ONNX Runtime during the build. FocalNet weights are
optional in PR images: `make model-focalnet` verifies the published graph and
stages it at `models/optional/focalnet-human.onnx`. Docker copies that file
only when its SHA-256 matches `2026-09-14-rc1`. The contract dummy is never
accepted. The default `MODEL_BACKEND=u2net` still works if the file is absent;
`MODEL_BACKEND=focalnet` fails at startup. Release images set
optional in PR images: `make model-focalnet` downloads them and stages a copy
at `models/optional/focalnet-human.onnx`. Docker copies that file only when its
SHA-256 matches. The default `MODEL_BACKEND=u2net` still works if the file is
absent; `MODEL_BACKEND=focalnet` fails at startup. Release images set
`REQUIRE_FOCALNET_MODEL=1` so they cannot publish without the real weights.
U²-Net precision selection is by environment, not architecture.

Expand All @@ -111,7 +117,7 @@ docker build -t autogravity .
docker run --rm -p 8080:8080 autogravity
# Select FP32 without rebuilding:
docker run --rm -p 8080:8080 -e MODEL_PRECISION=fp32 autogravity
# Use the published FocalNet human-ranking model:
# Use Appwrite's FocalNet model:
docker run --rm -p 8080:8080 -e MODEL_BACKEND=focalnet autogravity
```

Expand Down Expand Up @@ -204,26 +210,23 @@ are reused safely across requests.

## FocalNet backend

[FocalNet](https://github.com/appwrite/focalnet) publishes a format-v2 human
crop-ranking ONNX graph. Set `MODEL_BACKEND=focalnet` to load
`models/focalnet-human.onnx` instead of YuNet + U²-Net. YuNet is not run;
faces are already fused into the 64×64 importance map. `/analyze` generates
the same candidate crops as FocalNet's Python runtime, scores them with the
trained ranking head, keeps candidates within 0.05 of the best importance
retention, and returns the selected crop's center as `gravity`.
[FocalNet](https://github.com/appwrite/focalnet) is Appwrite's own model. Set
`MODEL_BACKEND=focalnet` to load `models/focalnet-human.onnx` instead of YuNet +
U²-Net. It picks a crop and returns that crop's center as `gravity`. Faces are
already part of the model, so YuNet is not run.

The evaluated artifact is the FocalNet GitHub release `2026-09-14-rc1`
The weights come from the public FocalNet GitHub release `2026-09-14-rc1`
(`focalnet-human.onnx`, SHA-256
`59164c601c98cea3f62b25166710831dac63e1a872fc64767c65316ad5385439`).
It is not vendored in git. Download it with `make model-focalnet`, then:
They are not stored in git. Download them with `make model-focalnet`, then:

```sh
export MODEL_BACKEND=focalnet
./autogravity
```

Pass `?aspect_ratio=16:9` (or a positive float such as `1.5`) to rank a
non-square crop. The default is `1:1`. Example response:
Pass `?aspect_ratio=16:9` (or a positive number such as `1.5`) for a non-square crop.
The default is `1:1`. Example response:

```json
{
Expand All @@ -234,18 +237,14 @@ non-square crop. The default is `1:1`. Example response:
"left": 120,
"top": 40,
"width": 480,
"height": 480,
"height": 270,
"retained_importance": 0.88
}
}
```

The ONNX contract is RGB NCHW `image` `[1,3,256,256]`, padded `boxes`
`[1,128,4]`, letterbox `content` `[1,4]` in, and `importance` `[1,1,64,64]`
plus `crop_scores` `[1,128]` out. Autogravity unpads the letterboxed map,
applies FocalNet's candidate and retention-gate ranking, and uses the
selected crop center as the gravity point. Preprocessing matches FocalNet's
Python runtime (`imaging.py`) via the existing Go letterbox path at 256×256.
See [models/README.md](models/README.md) for the ONNX contract and checksum
details.

## Telemetry and shutdown

Expand Down
1 change: 1 addition & 0 deletions docs/src/lib/site.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ export const NAV = {
start: [
{ href: '#overview', label: 'Overview' },
{ href: '#face-priority', label: 'Face priority' },
{ href: '#focalnet', label: 'FocalNet' },
{ href: '#preview', label: 'Storage preview' },
{ href: '#install', label: 'Install' },
{ href: '#config', label: 'Configuration' },
Expand Down
89 changes: 79 additions & 10 deletions docs/src/routes/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,13 @@ function DocsPage() {
<p className="doc-lead">
A small Go HTTP service that finds the best crop focus in an image.
It prioritizes confidently detected faces with YuNet, then falls
back to U²-Net saliency. It never crops, stores, identifies, or
modifies the submitted image.
back to U²-Net saliency. You can also switch to Appwrite&apos;s
FocalNet model. It never crops, stores, identifies, or modifies the
submitted image.
</p>
<div className="pill-row">
<span className="pill">Face-first · saliency fallback</span>
<span className="pill">JPEG · PNG · WebP</span>
<span className="pill">JPEG · PNG · WebP · GIF</span>
<span className="pill">CPU-only</span>
</div>
</section>
Expand All @@ -45,6 +46,28 @@ function DocsPage() {
</p>
</section>

<section id="focalnet" className="doc-section">
<SectionTitle
kicker="How it works"
title="Appwrite's FocalNet model"
lead="FocalNet is Appwrite's own model. It picks a crop and uses the crop's center as the focal point, without a separate face detector."
/>
<CodePanel label="Run FocalNet">
<code>
<span className="prompt">$ </span>make model-focalnet
{'\n'}
<span className="prompt">$ </span>MODEL_BACKEND=focalnet
./autogravity
{'\n'}
<span className="prompt">$ </span>docker run --rm -p 8080:8080 \
{'\n'}
{' '}-e MODEL_BACKEND=focalnet \
{'\n'}
{' '}ghcr.io/appwrite/autogravity
</code>
</CodePanel>
</section>

<section id="preview" className="doc-section">
<SectionTitle
kicker="Integration"
Expand Down Expand Up @@ -84,7 +107,8 @@ function DocsPage() {
<p className="doc-copy">
Or build from source with Go 1.25 or newer.{' '}
<InlineCode>make model</InlineCode> downloads and verifies the ONNX
models.
models. Use <InlineCode>make model-focalnet</InlineCode> if you want
Appwrite&apos;s FocalNet weights.
</p>
<CodePanel label="Source">
<code>
Expand Down Expand Up @@ -117,6 +141,13 @@ function DocsPage() {
u2net (YuNet + U²-Net) or focalnet
</span>
</div>
<div className="data-table-row">
<span className="accent">MODEL_PRECISION</span>
<span className="accent">int8</span>
<span className="data-table-desc">
int8 or fp32 for the U²-Net backend
</span>
</div>
<div className="data-table-row">
<span className="accent">MODEL_PATH</span>
<span className="accent">unset</span>
Expand Down Expand Up @@ -155,13 +186,13 @@ function DocsPage() {
<SectionTitle
kicker="Reference"
title="API"
lead="Send a JPEG, PNG, or WebP image as a multipart image field, or as the raw request body."
lead="Send a JPEG, PNG, WebP, or GIF image as a multipart image field, or as the raw request body."
/>

<HttpEndpoint
method="POST"
path="/analyze"
description="Returns a prioritized face center or the strongest salient region's weighted focal point."
description="Returns a face center, a saliency focal point, or a FocalNet crop center."
/>

<div className="code-grid">
Expand Down Expand Up @@ -228,13 +259,51 @@ function DocsPage() {
applied before analysis. On the default backend a reliable face
supplies its bounding-box center; otherwise U²-Net supplies the
saliency centroid. Set <InlineCode>MODEL_BACKEND=focalnet</InlineCode>{' '}
to use the distilled importance model instead —{' '}
<InlineCode>source</InlineCode> is then <InlineCode>focalnet</InlineCode>{' '}
and YuNet is not consulted. Confidence is that strategy&apos;s
model score, not an identity match or a calibrated probability.
to use Appwrite&apos;s model instead.{' '}
<InlineCode>source</InlineCode> is then{' '}
<InlineCode>focalnet</InlineCode> and the response includes a{' '}
<InlineCode>crop</InlineCode> rectangle. Confidence is that
strategy&apos;s model score, not an identity match or a calibrated
probability.
</p>
</div>

<CodePanel label="FocalNet request">
<code>
curl -sS -X POST \{'\n'}
{' '}http://localhost:8080/analyze?aspect_ratio=16:9 \{'\n'}
{' '}-F <span className="str">'image=@photo.jpg'</span>
</code>
</CodePanel>
<CodePanel label="FocalNet response">
<code>
{'{'}{'\n'}
{' '}<span className="key">"gravity"</span>: {'{'}{'\n'}
{' '}<span className="key">"x"</span>:{' '}
<span className="num">0.52</span>,{'\n'}
{' '}<span className="key">"y"</span>:{' '}
<span className="num">0.41</span>{'\n'}
{' '}{'}'},{'\n'}
{' '}<span className="key">"confidence"</span>:{' '}
<span className="num">0.91</span>,{'\n'}
{' '}<span className="key">"source"</span>:{' '}
<span className="str">"focalnet"</span>,{'\n'}
{' '}<span className="key">"crop"</span>: {'{'}{'\n'}
{' '}<span className="key">"left"</span>:{' '}
<span className="num">120</span>,{'\n'}
{' '}<span className="key">"top"</span>:{' '}
<span className="num">40</span>,{'\n'}
{' '}<span className="key">"width"</span>:{' '}
<span className="num">480</span>,{'\n'}
{' '}<span className="key">"height"</span>:{' '}
<span className="num">270</span>,{'\n'}
{' '}<span className="key">"retained_importance"</span>:{' '}
<span className="num">0.88</span>{'\n'}
{' '}{'}'}{'\n'}
{'}'}
</code>
</CodePanel>

<HttpEndpoint
method="GET"
path="/healthz"
Expand Down
13 changes: 13 additions & 0 deletions docs/src/styles.css
Original file line number Diff line number Diff line change
Expand Up @@ -356,6 +356,18 @@ a:hover {
color: var(--muted);
}

.doc-copy a,
.doc-lead a {
color: var(--pink-soft);
text-decoration: underline;
text-underline-offset: 0.18em;
}

.doc-copy a:hover,
.doc-lead a:hover {
color: var(--aw-pink);
}

.doc-copy--flush {
max-width: none;
}
Expand Down Expand Up @@ -884,6 +896,7 @@ a:hover {
}

.data-table-row .accent {
overflow-wrap: anywhere;
color: var(--fg);
font-family: var(--font-mono);
font-size: 0.8125rem;
Expand Down
19 changes: 9 additions & 10 deletions models/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ obscured faces may not reach the threshold.

## FocalNet

FocalNet is an optional `MODEL_BACKEND=focalnet` path. The published
human-ranking graph is `focalnet-human.onnx` from the
FocalNet is Appwrite's own model, used when `MODEL_BACKEND=focalnet`. The
weights are `focalnet-human.onnx` from the public
[FocalNet `2026-09-14-rc1` release](https://github.com/appwrite/focalnet/releases/tag/2026-09-14-rc1)
(20,398,992 bytes). `make model-focalnet` downloads it, verifies this SHA-256,
and stages a copy at `models/optional/focalnet-human.onnx` for Docker:
Expand All @@ -68,12 +68,11 @@ contract-compatible placeholder lives at `internal/focalnet/testdata/dummy.onnx`
for Go tests; it is image-independent and must not be copied into
`models/focalnet-human.onnx` or a production image. Docker accepts only the
published checksum above. The artifact is redistributed under the MIT license
in `FOCALNET_LICENSE`.
in `FOCALNET_LICENSE`. See [appwrite/focalnet](https://github.com/appwrite/focalnet)
for training code and results.

The FocalNet GitHub release is currently private, so Autogravity's default
`GITHUB_TOKEN` cannot download it. PR image builds omit the weights rather than
substituting the dummy; the default U²-Net backend still starts.
`MODEL_BACKEND=focalnet` fails at startup if the file is missing. Published
release images require the real checksum (`REQUIRE_FOCALNET_MODEL=1`) and a
token that can read `appwrite/focalnet` (set repo secret
`FOCALNET_GITHUB_TOKEN`, or run `make model-focalnet` before `docker build`).
PR image builds still start without the file. `MODEL_BACKEND=focalnet` fails at
startup if it is missing. Release images require the real checksum
(`REQUIRE_FOCALNET_MODEL=1`). Run `make model-focalnet` before `docker build`,
or set repo secret `FOCALNET_GITHUB_TOKEN` if a workflow cannot download the
release.
Loading