Skip to content

Commit 0f7b452

Browse files
committed
docs: agent images AdminForth/1955/implement-ability-for-agent-pl
Image attachments section of the agent plugin: the turns column, the Upload plugin on it, read-only turns and checkpoints, attachmentsField and the imageAttachments limits. strictContentType is required for S3; the storage adapters docs name the versions it is available since.
1 parent 086240e commit 0f7b452

2 files changed

Lines changed: 77 additions & 2 deletions

File tree

‎documentation/docs/tutorial/06-Adapters/04-storage-adapters.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ new AdminForthAdapterS3Storage({
3030
}),
3131
```
3232

33-
Before turning it on, update `@adminforth/bulk-ai-flow`, `@adminforth/markdown`, `@adminforth/rich-editor` and `@adminforth/oauth`: their older versions send the type reported by the browser, and the upload fails when it differs from the one picked by the extension. Your own code that uploads files to the URL from the [upload plugin API](../09-Plugins/05-1-upload-api.md) has to send the `contentType` returned with the URL as well. While the option is off, the adapter logs a warning on start. It will be on by default in the next major version.
33+
The option is available since `@adminforth/storage-adapter-amazon-s3` 1.3.0. Before turning it on, update `@adminforth/bulk-ai-flow`, `@adminforth/markdown`, `@adminforth/rich-editor` and `@adminforth/oauth`: their older versions send the type reported by the browser, and the upload fails when it differs from the one picked by the extension. Your own code that uploads files to the URL from the [upload plugin API](../09-Plugins/05-1-upload-api.md) has to send the `contentType` returned with the URL as well. While the option is off, the adapter logs a warning on start. It will be on by default in the next major version.
3434

3535
## S3 compatible adapter
3636
```bash
@@ -39,7 +39,7 @@ pnpm add @adminforth/storage-adapter-s3-compatible
3939

4040
Provides S3 compatible interface for object storage services such as MinIO, Wasabi, Cloudflare R2 or other third-party S3 providers.
4141

42-
It supports the same `strictContentType` option as the Amazon S3 adapter, see [Strict content type](#strict-content-type).
42+
It supports the same `strictContentType` option as the Amazon S3 adapter since version 1.2.0, see [Strict content type](#strict-content-type).
4343

4444
### How the adapter works
4545

‎documentation/docs/tutorial/09-Plugins/01-agent.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -302,6 +302,81 @@ Each item in `modes` defines a user-selectable preset in the chat UI. The select
302302

303303
The plugin adds a chat surface to the admin UI, keeps session history per admin user, and shows a mode picker when `modes` are configured.
304304

305+
## Image attachments
306+
307+
Users can paste, attach or drop images (PNG, JPEG, WebP) in the chat, up to 5 per message, and the agent sees them. The images are stored by the [Upload plugin](./05-0-upload.md) on a column of the turns resource.
308+
309+
1. Add a nullable column for the paths of the images to the turns table and describe it as a string array:
310+
311+
```ts title="./resources/turns.ts"
312+
{
313+
name: 'attachments',
314+
type: AdminForthDataTypes.JSON,
315+
isArray: {
316+
enabled: true,
317+
itemType: AdminForthDataTypes.STRING,
318+
},
319+
showIn: { list: false, show: true, edit: false, create: false, filter: false },
320+
},
321+
```
322+
323+
2. Install the Upload plugin on this column, with images only:
324+
325+
```ts title="./resources/turns.ts"
326+
plugins: [
327+
new UploadPlugin({
328+
pathColumnName: 'attachments',
329+
storageAdapter: new AdminForthAdapterS3Storage({
330+
// ...bucket, region, credentials
331+
strictContentType: true,
332+
}),
333+
allowedFileExtensions: ['png', 'jpg', 'jpeg', 'webp'],
334+
filePath: ({ originalExtension }) => `agent_attachments/${randomUUID()}.${originalExtension}`,
335+
}),
336+
],
337+
```
338+
339+
With an S3 storage `strictContentType: true` is required: the browser uploads the image straight to the bucket, and without it any content can be stored under an image name, see [Strict content type](../06-Adapters/04-storage-adapters.md#strict-content-type). The option is available since `@adminforth/storage-adapter-amazon-s3` 1.3.0 and `@adminforth/storage-adapter-s3-compatible` 1.2.0.
340+
341+
3. Make the turns and checkpoints resources read-only in the admin panel. The agent writes them itself, while create or edit rights on the turns resource would also let admins download any file of its storage through the Upload plugin:
342+
343+
```ts title="./resources/turns.ts"
344+
options: {
345+
allowedActions: {
346+
create: false,
347+
edit: false,
348+
delete: false,
349+
},
350+
},
351+
```
352+
353+
4. Pass the column to the plugin:
354+
355+
```ts
356+
turnResource: {
357+
// ...
358+
attachmentsField: 'attachments',
359+
},
360+
```
361+
362+
Before the upload the browser scales an image down to fit 1920×1080 and converts it to WebP (to JPEG in a browser which can not encode WebP, e.g. Safari). The plugin accepts images up to 3.75 MB: the model gets them as base64, and the smallest per-image limit of the providers is 5 MB. Every image costs tokens on every model call, so the model sees an image for a while after it was sent, and only the latest few at once. Older images are replaced with a note, and the agent loads them again with the `view_image` tool when they matter for the answer.
363+
364+
The limits are options of the plugin, each a whole number from 1:
365+
366+
```ts
367+
new AdminForthAgent({
368+
// ...
369+
imageAttachments: {
370+
// images one message can have, 5 by default
371+
maxPerMessage: 5,
372+
// minutes the model sees an image after it was sent or loaded again with view_image, 30 by default
373+
visibleMinutes: 30,
374+
// images the model sees at once, the latest ones, 5 by default
375+
maxVisible: 5,
376+
},
377+
}),
378+
```
379+
305380
## Chat surfaces (Telegram, etc.)
306381

307382
By default, the Agent plugin exposes a chat surface inside the AdminForth admin UI.

0 commit comments

Comments
 (0)