From cc312a71e54961c230cf2683ca87c2269ffa31ed Mon Sep 17 00:00:00 2001 From: Dylan Corrales Date: Fri, 6 Nov 2026 16:21:02 -0500 Subject: [PATCH 01/11] Adding Ambarella S3L support A HAL for the Ambarella S3L (S3LM) over its IAV driver (/dev/iav), for ARM with either float ABI: it drives the driver through ioctls and links no vendor library. It is picked when the IAV driver reports an S3L (AMBARELLA_CHIP in /proc/ambarella/iav). - VIN mode and frame rate, mirror/flip through VIN SET_MIRROR, the source buffers, preview, and one H.264 stream read from the bitstream buffer (RTSP, MP4). Encoder settings are the driver's defaults for now. MJPEG is skipped on S3L. - ISP configuration: system.sensor_config names an S3LISP03 file (made from options, with no vendor data, by the generator OpenIPC's ambarella-osdrv-s3l package carries in src/isp), loaded into the IMG buffer before the first ENABLE_PREVIEW and applied with the IMG 'p' set ioctls. Without it, the one for the sensor the VIN reports is loaded, /etc/sensors/.bin, as OpenIPC's S3L package generates them. Its table pointers are IMG-buffer offsets with a relocation list, and the HAL adds the buffer's address as the driver reports it, so the file does not depend on where the buffer is. The file's sizes and offsets are checked against the IMG buffer, and against the blocks the HAL keeps below the command rings, before anything is copied or applied. The IMG command rings are zeroed first: after a cold boot DRAM holds garbage there that the DSP would otherwise read as commands. - 3A in Divinus: DSP statistics (CONFIG_STATISTICS / GET_STATISTICS, two slices per frame stitched), auto exposure through the VIN AGC/shutter ioctls every Ambarella sensor driver implements, with anti-flicker, auto white balance from the per-tile raw G/R, G/B median (handed to the AE thread under a lock, so both gains come from one frame), and bad-pixel thresholds that follow the gain. New isp options auto_exposure, exposure_target and auto_white_balance. - The DSP takes the 3A statistics setup into effect only when preview starts, and the setup can only be sent once preview runs. In the first preview after the microcode is loaded, the SDK's 2018/10/28 284712 (which OpenIPC ships) sends no statistics, and this camera's own 267837 sends stale ones: the same three buffers over and over. So the setup is sent, left in place for a frame, and preview started again before any stream encodes, on every start (about 0.15 s). A driver that refuses to go back to idle (an emulated DSP does) leaves preview running, and the HAL carries on with a warning. - Grayscale for night mode, by swapping the chroma-scale curve. - The sensor name for /api/status, from the VIN device info. - build.sh s3l (and CI): OpenIPC's toolchain.ambarella-s3l. The S3L is a Cortex-A9 with VFPv3; the armhf-musl toolchain emits VFPv4 fused multiply-adds, which fault here. - No audio yet: with audio enabled, Divinus warns and runs without it. Saving the configuration keeps audio.enable as it was set, so the choice is not lost (app_config.audio_requested). - The main source buffer is sized for the H.264/H.265 stream, and the VIN runs at the fastest enabled stream's frame rate. - IAV state changes (idle, preview, encode start/stop) run with signals blocked: the driver waits for the DSP interruptibly, and a signal in that window (a service stop repeating SIGTERM) left IAV stuck, refusing every later start until a reboot. Tested on an Amcrest IP4M-1026E (Dahua IPC-HFW1430SN, S3L + OV4689): H.264 2688x1512 over RTSP from a cold boot, AE/AWB, night-mode grayscale, stop/start/restart cycles, with both the SDK microcode (284712) and the camera's own (267837). In the first preview after boot, the 3A statistics now change from frame to frame with both (with 267837 they had cycled through three fixed buffers). Builds for all build.yml targets, s3l among them (VFPv3/NEONv1). --- .github/workflows/build.yml | 2 +- README.md | 2 + build.sh | 8 +- doc/config.md | 7 +- src/app_config.c | 13 +- src/app_config.h | 5 + src/hal/amba/s3l_aaa.c | 280 ++++++++++++++ src/hal/amba/s3l_hal.c | 718 ++++++++++++++++++++++++++++++++++++ src/hal/amba/s3l_hal.h | 55 +++ src/hal/amba/s3l_iav.h | 95 +++++ src/hal/support.c | 3 + src/hal/support.h | 3 + src/hal/types.h | 1 + src/media.c | 83 ++++- src/server.c | 4 +- 15 files changed, 1269 insertions(+), 10 deletions(-) create mode 100644 src/hal/amba/s3l_aaa.c create mode 100644 src/hal/amba/s3l_hal.c create mode 100644 src/hal/amba/s3l_hal.h create mode 100644 src/hal/amba/s3l_iav.h diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 35232c1f..ede26860 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -17,7 +17,7 @@ jobs: - name: Build run: | - list="ak39xx fhan-v2 gm813x hisi-v1 hisi-v2 hisi-v2a hisi-v4 hisi-v4a inge-t31 rv11xx star6 star6e" + list="ak39xx fhan-v2 gm813x hisi-v1 hisi-v2 hisi-v2a hisi-v4 hisi-v4a inge-t31 rv11xx s3l star6 star6e" for lib in $list; do sh build.sh $lib zip $lib divinus* diff --git a/README.md b/README.md index f8097cb8..cd3a776c 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,7 @@ In spite of these design choices, Divinus boasts numerous features that cater to | infinity6c[^18] | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | | mercury6[^19] | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | | sun8iw21p1[^20] | ↻ | ↻ | ↻ | ↻ | ↻ | +| S3L[^21] | ↻ | ↻ | ↻ | ✔️ | ↻ | _✔️ - supported, ↻ - in development, ✗ - unsupported, ⁿ/ₐ - not supported by hardware_ @@ -73,6 +74,7 @@ _* At the moment, text, RGB or bitfield bitmaps and PNG overlays are handled, mo [^18]: SSC377\(D/DE/QE\) and SSC378\[DE/QE\] [^19]: SSC359G [^20]: V851S\(E\) and V853 +[^21]: Ambarella S3L (S3LM), via the IAV driver; the ISP configuration and 3A (AE, AWB) run in Divinus ### Documentation diff --git a/build.sh b/build.sh index 30dd7a81..bb37da64 100755 --- a/build.sh +++ b/build.sh @@ -60,6 +60,10 @@ elif [ "$1" = "fhan-v1" ]; then elif [ "$1" = "armv6-musl" ] || [ "$1" = "fhan-v2" ]; then PRE="openipc-linux-musleabi" toolchain toolchain.fullhan-fh8852v200 arm +elif [ "$1" = "armhf-vfp3" ] || [ "$1" = "s3l" ]; then + # Ambarella S3L: a Cortex-A9 with VFPv3. The armhf-musl toolchain targets VFPv4 with fused + # multiply-add, which faults (SIGILL) here. + toolchain toolchain.ambarella-s3l arm elif [ "$1" = "mips-musl" ] || [ "$1" = "inge-t31" ]; then toolchain toolchain.ingenic-t31 mipsel elif [ "$1" = "riscv64-musl" ] || [ "$1" = "cvitek" ]; then @@ -68,8 +72,8 @@ elif [ "$1" = "riscv64-musl" ] || [ "$1" = "cvitek" ]; then toolchain riscv64-lp64d--musl--stable-2024.05-1 riscv64 else echo "Usage: $0 [arm-glibc|arm-musl3|arm-musl4|arm9-glibc|arm9-musl3|arm9-musl4|arm9-uclibc|" - echo " armhf-glibc|armhf-musl|armhf-uclibc|armv6-musl|mips-musl|riscv64-musl|ak39xx|cvitek|gm813x|" + echo " armhf-glibc|armhf-musl|armhf-uclibc|armhf-vfp3|armv6-musl|mips-musl|riscv64-musl|ak39xx|cvitek|gm813x|" echo " hisi-v1|hisi-v2|hisi-v2a|hisi-v3|hisi-v3a|hisi-v4|hisi-v4a|fhan-v1|fhan-v2|inge-t31|rv11xx|" - echo " star6|star6e] (debug)" + echo " s3l|star6|star6e] (debug)" rm -f .previous fi diff --git a/doc/config.md b/doc/config.md index d4343c2b..c4ad563b 100644 --- a/doc/config.md +++ b/doc/config.md @@ -4,7 +4,7 @@ This document describes the fields that can be found within a configuration file ## System section -- **sensor_config**: Path to the sensor calibration or configuration file, if applicable (e.g., `/etc/sensors/imx415.bin`). +- **sensor_config**: Path to the sensor calibration or configuration file, if applicable (e.g., `/etc/sensors/imx415.bin`). On Ambarella S3L, an ISP configuration file (S3LISP03, as OpenIPC's S3L package generates them); when it is missing, the one for the detected sensor, `/etc/sensors/.bin` (OpenIPC's S3L package generates `ov4689.bin`), is used. One of them is required: without an ISP configuration the S3L's DSP produces no image, so Divinus stops with an error. - **web_port**: Port number for the web server (default: `80`). - **web_whitelist**: Array of up to 4 IP addresses or domains allowed to access the web server. - **web_enable_auth**: Boolean to enable authentication on the API, live stream and WebUI endpoints (default: `false`). @@ -35,7 +35,10 @@ This document describes the fields that can be found within a configuration file - **mirror**: Boolean to turn on image mirroring (default: `false`). - **flip**: Boolean to turn on image flipping (default: `false`). -- **antiflicker**: Antiflicker setting in Hz (default: `60`). +- **antiflicker**: Antiflicker setting in Hz (default: `60`); the S3L's own auto exposure honours 50 or 60 and treats anything else as off. +- **auto_exposure**: Boolean, Divinus's own auto exposure on platforms whose HAL runs its 3A itself (Ambarella S3L; default: `true`). +- **exposure_target**: Target mean luma for that auto exposure, up to 235; 0 or anything below 17 uses the platform default (55 on the S3L; default: `0`). +- **auto_white_balance**: Boolean, Divinus's own auto white balance on those platforms (default: `true`). ## mDNS section diff --git a/src/app_config.c b/src/app_config.c index 9d163672..a3204312 100644 --- a/src/app_config.c +++ b/src/app_config.c @@ -119,6 +119,9 @@ int app_config_save(void) { fprintf(file, " mirror: %s\n", app_config.mirror ? "true" : "false"); fprintf(file, " flip: %s\n", app_config.flip ? "true" : "false"); fprintf(file, " antiflicker: %d\n", app_config.antiflicker); + fprintf(file, " auto_exposure: %s\n", app_config.auto_exposure ? "true" : "false"); + fprintf(file, " exposure_target: %d\n", app_config.exposure_target); + fprintf(file, " auto_white_balance: %s\n", app_config.auto_white_balance ? "true" : "false"); fprintf(file, "mdns:\n"); fprintf(file, " enable: %s\n", app_config.mdns_enable ? "true" : "false"); @@ -156,7 +159,7 @@ int app_config_save(void) { } fprintf(file, "audio:\n"); - fprintf(file, " enable: %s\n", app_config.audio_enable ? "true" : "false"); + fprintf(file, " enable: %s\n", app_config.audio_requested ? "true" : "false"); fprintf(file, " bitrate: %d\n", app_config.audio_bitrate); fprintf(file, " gain: %d\n", app_config.audio_gain); fprintf(file, " srate: %d\n", app_config.audio_srate); @@ -265,6 +268,7 @@ enum ConfigError app_config_parse(void) { app_config.sensor_config[0] = 0; app_config.audio_enable = false; + app_config.audio_requested = false; app_config.audio_bitrate = 128; app_config.audio_gain = 0; app_config.jpeg_enable = false; @@ -280,6 +284,9 @@ enum ConfigError app_config_parse(void) { app_config.mirror = false; app_config.flip = false; app_config.antiflicker = 0; + app_config.auto_exposure = true; + app_config.exposure_target = 0; + app_config.auto_white_balance = true; app_config.night_mode_enable = false; app_config.ir_sensor_pin = 999; @@ -412,6 +419,9 @@ enum ConfigError app_config_parse(void) { if (err != CONFIG_OK) goto RET_ERR; parse_int(&ini, "isp", "antiflicker", -1, 60, &app_config.antiflicker); + parse_bool(&ini, "isp", "auto_exposure", &app_config.auto_exposure); + parse_int(&ini, "isp", "exposure_target", 0, 235, &app_config.exposure_target); + parse_bool(&ini, "isp", "auto_white_balance", &app_config.auto_white_balance); parse_bool(&ini, "mdns", "enable", &app_config.mdns_enable); @@ -502,6 +512,7 @@ enum ConfigError app_config_parse(void) { } parse_bool(&ini, "audio", "enable", &app_config.audio_enable); + app_config.audio_requested = app_config.audio_enable; if (app_config.audio_enable) { parse_int(&ini, "audio", "bitrate", 32, 320, &app_config.audio_bitrate); parse_int(&ini, "audio", "gain", -60, 30, &app_config.audio_gain); diff --git a/src/app_config.h b/src/app_config.h index 65056455..4e0bef7d 100644 --- a/src/app_config.h +++ b/src/app_config.h @@ -42,6 +42,9 @@ struct AppConfig { bool mirror; bool flip; int antiflicker; + bool auto_exposure; + int exposure_target; + bool auto_white_balance; // [osd] bool osd_enable; @@ -78,6 +81,8 @@ struct AppConfig { // [audio] bool audio_enable; + // audio.enable as configured: audio_enable is cleared for the run on a platform without audio + bool audio_requested; unsigned int audio_bitrate; int audio_gain; unsigned int audio_srate; diff --git a/src/hal/amba/s3l_aaa.c b/src/hal/amba/s3l_aaa.c new file mode 100644 index 00000000..60cab580 --- /dev/null +++ b/src/hal/amba/s3l_aaa.c @@ -0,0 +1,280 @@ +#if defined(__arm__) + +#include "s3l_hal.h" + +#include +#include +#include + +#define S3L_CFA_REC 12672 +#define S3L_STAT_DATA 0x82 + +struct s3l_aaa_frame { + unsigned int n_tiles; + float gr[1024], gb[1024]; +}; + +static inline unsigned int rd16(const unsigned char *p) { return p[0] | p[1] << 8; } + +static void s3l_aaa_parse(const unsigned char *cfa, struct s3l_aaa_frame *f) +{ + f->n_tiles = 0; + unsigned int slices = cfa[0x36] ? cfa[0x36] : 1; + if (slices > 2) slices = 2; + for (unsigned int k = 0; k < slices; k++) { + const unsigned char *c = cfa + k * S3L_CFA_REC; + unsigned int awb_n = c[0x30] * c[0x31]; + if (awb_n > 1024) continue; + double awb_px = (double)rd16(c + 0x04) * rd16(c + 0x06); + unsigned int awb_sh = rd16(c + 0x0c); + if (awb_sh > 15) continue; + for (unsigned int i = 0; i < awb_n; i++) { + const unsigned char *t = c + S3L_STAT_DATA + 10 * i; + if (rd16(t + 8)) continue; + double r = (double)(rd16(t) << awb_sh) / awb_px, g = (double)(rd16(t + 2) << awb_sh) / awb_px, + b = (double)(rd16(t + 4) << awb_sh) / awb_px; + if (g > 20 && r > 1 && b > 1 && f->n_tiles < 1024) { + double q = g / r, p = g / b; + if (q > 0.8 && q < 4 && p > 0.8 && p < 4) { f->gr[f->n_tiles] = q; f->gb[f->n_tiles] = p; f->n_tiles++; } + } + } + } +} + +int s3l_aaa_config(int fd) +{ + struct __attribute__((packed)) { + uint32_t cmd_code, flags, fifo[4]; + uint16_t awb[10], ae[8], af[8]; + } c; + memset(&c, 0, sizeof(c)); + c.flags = 1 | 1 << 8; + uint16_t awb[10] = { 32, 32, 0, 0, 128, 128, 128, 128, 0, 16383 }; + uint16_t ae[8] = { 12, 8, 0, 0, 341, 512, 0, 16383 }; + uint16_t af[8] = { 12, 8, 0, 0, 341, 512, 341, 512 }; + memcpy(c.awb, awb, sizeof(awb)); + memcpy(c.ae, ae, sizeof(ae)); + memcpy(c.af, af, sizeof(af)); + if (ioctl(fd, S3L_IOC_IMG_CONFIG_STATISTICS, &c) < 0) { + HAL_WARNING("s3l_aaa", "CONFIG_STATISTICS failed: %s\n", strerror(errno)); + return EXIT_FAILURE; + } + return EXIT_SUCCESS; +} + +/* Written by the netlink thread, read by the AE thread: the lock keeps both gains from one frame */ +static pthread_mutex_t _dsp_mtx = PTHREAD_MUTEX_INITIALIZER; +static float _dsp_wr, _dsp_wb; +static unsigned int _dsp_seq, _dsp_tiles; + +static int cmp_f(const void *a, const void *b) +{ + float x = *(const float *)a, y = *(const float *)b; + return (x > y) - (x < y); +} + +void s3l_aaa_frame(const unsigned char *cfa) +{ + struct s3l_aaa_frame f; + + s3l_aaa_parse(cfa, &f); + + if (f.n_tiles >= 32) { + qsort(f.gr, f.n_tiles, sizeof(float), cmp_f); + qsort(f.gb, f.n_tiles, sizeof(float), cmp_f); + pthread_mutex_lock(&_dsp_mtx); + _dsp_wr = f.gr[f.n_tiles / 2]; + _dsp_wb = f.gb[f.n_tiles / 2]; + _dsp_tiles = f.n_tiles; + _dsp_seq++; + pthread_mutex_unlock(&_dsp_mtx); + } +} + +static pthread_t _ae_tid; +static volatile char _ae_run; +static char _opt_ae = 1, _opt_awb = 1; +static int _opt_target, _opt_flicker; + +void s3l_aaa_options(char ae, int target, char awb, int antiflicker) +{ + _opt_ae = ae; _opt_awb = awb; _opt_target = target; _opt_flicker = antiflicker; +} + +#define VIN_GET_AAAINFO 0x80447313u +#define VIN_GET_AGC 0x80287307u +#define VIN_GET_SHUTTER 0x800c7306u +#define VIN_SET_AGC_SHUTTER 0x40107308u + +static int _vin_fd = -1; +static unsigned int _line_ticks, _max_rows, _gain_max_idx; +static double _step_db; + +static int sensor_init(int fd, unsigned int *rows, double *gain) +{ + uint32_t aaa[17] = { 0 }, agc[10] = { 0 }, sh[3] = { 0 }; + _vin_fd = fd; + if (ioctl(fd, VIN_GET_AAAINFO, aaa) == 0 && ioctl(fd, VIN_GET_AGC, agc) == 0 && aaa[0x2c / 4] && + aaa[0x38 / 4] && agc[0x10 / 4] && agc[0x8 / 4] >= agc[0xc / 4]) { + _line_ticks = aaa[0x2c / 4]; + _max_rows = aaa[0x38 / 4]; + _step_db = agc[0x10 / 4] / 16777216.0; + _gain_max_idx = (agc[0x8 / 4] - agc[0xc / 4]) / agc[0x10 / 4]; + *rows = ioctl(fd, VIN_GET_SHUTTER, sh) == 0 && sh[1] ? sh[1] / _line_ticks : _max_rows / 2; + *gain = pow(10, (int32_t)(agc[0x4 / 4] - agc[0xc / 4]) / 16777216.0 / 20); + return 0; + } + return -1; +} + +static void sensor_apply(unsigned int rows, double gain) +{ + int idx = (int)lround(20 * log10(gain) / _step_db); + if (idx < 0) idx = 0; + if ((unsigned int)idx > _gain_max_idx) idx = _gain_max_idx; + uint32_t a[4] = { 0, rows, (uint32_t)idx, 2 }; + ioctl(_vin_fd, VIN_SET_AGC_SHUTTER, a); +} + +static void *s3l_ae_thread(void *arg) +{ + int fd = (int)(intptr_t)arg >> 8, fps = (int)(intptr_t)arg & 0xff; + int ae = _opt_ae, awb = _opt_awb; + double target = _opt_target > 16 ? _opt_target : 55; + int flicker = _opt_flicker == 50 || _opt_flicker == 60 ? _opt_flicker : 0; + uint32_t qb[16] = { 0 }; + static uint32_t d[300]; + unsigned char *m = MAP_FAILED; + if (ioctl(fd, S3L_IOC_QUERY_BUF, qb) == 0) + m = mmap64(NULL, qb[1], PROT_READ, MAP_SHARED, fd, qb[2]); + if (m == MAP_FAILED) { + HAL_WARNING("s3l_aaa", "No preview buffer to measure, auto exposure and white balance off\n"); + /* Still the one thread that applies ISP changes (night mode, grayscale) */ + while (_ae_run) { usleep(100000); s3l_isp_flush(); } + return NULL; + } + + unsigned int lines = 1, max_lines = 1, flick_lines = 0; + double gain = 1.0, max_gain = 1.0; + if (sensor_init(fd, &lines, &gain) == 0) { + max_lines = _max_rows; + max_gain = pow(10, _gain_max_idx * _step_db / 20); + + if (flicker) flick_lines = (unsigned int)lround(512e6 / (2.0 * flicker) / _line_ticks); + } else if (ae) { + HAL_WARNING("s3l_aaa", "The sensor driver reports no shutter/gain limits, auto exposure off\n"); + ae = 0; + } + double total = lines * gain; + + double wr = 1.5, wb = 1.8; + s3l_isp_get_wb(&wr, &wb); + + int dbp_base = s3l_isp_dbp_mode() ? s3l_isp_dbp_base() : 0, dbp_step = -1; + + HAL_INFO("s3l_aaa", "AE %s via %s (target Y %.0f, %d fps, max %u lines, line %u ticks, gain 1..%.1fx, " + "anti-flicker %d Hz = %u lines), AWB %s, DBP-vs-gain %s (base %d)\n", ae ? "on" : "off", + "VIN ioctls", target, fps, max_lines, _line_ticks, max_gain, flicker, flick_lines, + awb ? "on" : "off", dbp_base ? "on" : "off", dbp_base); + + for (unsigned int n = 0; _ae_run; n++) { + usleep(100000); + memset(d, 0, sizeof(d)); + d[0] = 3; d[2] = 0; + if (ioctl(fd, S3L_IOC_QUERY_DESC, d) < 0) { s3l_isp_flush(); continue; } + uint32_t y = d[3], uv = d[4], w = d[5], h = d[6], pitch = d[7]; + if (!w || !h || w > pitch || (uint64_t)y + (uint64_t)pitch * h > qb[1] || + (uint64_t)uv + (uint64_t)pitch * ((h + 1) / 2) > qb[1]) { s3l_isp_flush(); continue; } + + double wsum = 0, wcnt = 0; + unsigned long cnt = 0, clip = 0, su = 0, sv = 0, cuv = 0, su2 = 0, sv2 = 0, cuv2 = 0; + for (uint32_t r = 8; r < h; r += 16) + for (uint32_t c = 8; c < w; c += 16) { + unsigned int Y = m[y + r * pitch + c]; + double wt = (r > h / 4 && r < 3 * h / 4 && c > w / 4 && c < 3 * w / 4) ? 2 : 1; + wsum += wt * Y; wcnt += wt; cnt++; + if (Y >= 235) clip++; + if (Y > 40 && Y < 220) { + unsigned int U = m[uv + (r / 2) * pitch + (c & ~1u)], V = m[uv + (r / 2) * pitch + (c & ~1u) + 1]; + su2 += U; sv2 += V; cuv2++; + if (abs((int)U - 128) + abs((int)V - 128) < 24) { su += U; sv += V; cuv++; } + } + } + double mean = wsum / wcnt, clip_frac = (double)clip / cnt; + + static unsigned int last_seq; + pthread_mutex_lock(&_dsp_mtx); + unsigned int seq = _dsp_seq; + int dsp_fresh = seq != last_seq && _dsp_tiles >= 32; + double tr = _dsp_wr, tb = _dsp_wb; + pthread_mutex_unlock(&_dsp_mtx); + if (awb && n % 3 == 0 && dsp_fresh) { + last_seq = seq; + tr = fmin(fmax(tr, 0.8), 3.5); tb = fmin(fmax(tb, 0.8), 3.5); + if (tr / tb > 2.5) tr = tb * 2.5; + if (tr / tb < 0.4) tb = tr / 0.4; + if (fabs(tr - wr) > 0.005 || fabs(tb - wb) > 0.005) { + wr += 0.3 * (tr - wr); wb += 0.3 * (tb - wb); + s3l_isp_set_wb(wr, wb); + } + } else if (awb && n % 3 == 0) { + if (cuv < 200) { su = su2; sv = sv2; cuv = cuv2; } + if (cuv > 200) { + double eu = (double)su / cuv - 128, ev = (double)sv / cuv - 128; + if (fabs(eu) > 0.7 || fabs(ev) > 0.7) { + wr *= exp(-0.03 * ev); wb *= exp(-0.03 * eu); + + wr = fmin(fmax(wr, 0.8), 3.5); wb = fmin(fmax(wb, 0.8), 3.5); + if (wr / wb > 2.5) wr = wb * 2.5; + if (wr / wb < 0.4) wb = wr / 0.4; + s3l_isp_set_wb(wr, wb); + } + } + } + + if (!ae) { s3l_isp_flush(); continue; } + double tgt = target * fmin(1.0, fmax(0.6, 1 - 4 * (clip_frac - 0.01))); + if (fabs(mean - tgt) < 3) { s3l_isp_flush(); continue; } + double cur = pow(fmax(mean - 16, 0.5) / 219, 1 / 0.45), want = pow((tgt - 16) / 219, 1 / 0.45); + double ratio = sqrt(fmin(1.6, fmax(0.6, want / cur))); + total = fmin(fmax(total * ratio, 1), max_lines * max_gain); + lines = total < max_lines ? (unsigned int)total : max_lines; + if (flick_lines && flick_lines <= max_lines && lines >= flick_lines) + lines = lines / flick_lines * flick_lines; + if (lines < 1) lines = 1; + gain = fmin(fmax(total / lines, 1.0), max_gain); + sensor_apply(lines, gain); + + if (dbp_base) { + int step = gain < 2 ? 0 : gain < 4 ? 1 : gain < 8 ? 2 : 3; + if (step != dbp_step) { + int thr = (int)lround(dbp_base * sqrt((double)(1 << step))); + s3l_isp_set_dbp(thr); + dbp_step = step; + } + } + s3l_isp_flush(); + } + munmap(m, qb[1]); + return NULL; +} + +int s3l_ae_start(int fd, int fps) +{ + _ae_run = 1; + if (pthread_create(&_ae_tid, NULL, s3l_ae_thread, (void *)(intptr_t)(fd << 8 | (fps & 0xff)))) { + _ae_run = 0; + HAL_DANGER("s3l_aaa", "Starting the AE thread failed!\n"); + return EXIT_FAILURE; + } + return EXIT_SUCCESS; +} + +void s3l_ae_stop(void) +{ + if (!_ae_run) return; + _ae_run = 0; + pthread_join(_ae_tid, NULL); +} + +#endif diff --git a/src/hal/amba/s3l_hal.c b/src/hal/amba/s3l_hal.c new file mode 100644 index 00000000..02150e59 --- /dev/null +++ b/src/hal/amba/s3l_hal.c @@ -0,0 +1,718 @@ +#if defined(__arm__) + +#include "s3l_hal.h" + +#include +#include +#include +#include +#include +#include + +hal_chnstate s3l_state[S3L_VENC_CHN_NUM] = {0}; +int (*s3l_aud_cb)(hal_audframe*); +int (*s3l_vid_cb)(char, hal_vidstream*); + +static int _s3l_fd = -1; +static unsigned char *_s3l_bsb; +static unsigned int _s3l_bsb_len; +static unsigned short _s3l_main_w, _s3l_main_h; + +static char _s3l_isp_path[128]; +static unsigned char *_s3l_img; +static unsigned int _s3l_img_len, _s3l_img_phys, _s3l_liso_off, _s3l_upd_slot; + +static int _s3l_nl_sock = -1; +static pthread_t _s3l_nl_tid, _s3l_stats_tid; +static volatile char _s3l_aaa_run, _s3l_previewing; + +static unsigned int s3l_state_get(void) +{ + unsigned int state = 0xdeadbeef; + ioctl(_s3l_fd, S3L_IOC_GET_IAV_STATE, &state); + return state; +} + +/* IAV waits for the DSP interruptibly: a signal there leaves it stuck until a reboot */ +static int s3l_ioctl_nosig(unsigned long cmd, void *arg) +{ + sigset_t all, old; + int ret, err; + + sigfillset(&all); + pthread_sigmask(SIG_BLOCK, &all, &old); + ret = ioctl(_s3l_fd, cmd, arg); + err = errno; + pthread_sigmask(SIG_SETMASK, &old, NULL); + errno = err; + return ret; +} + +static int s3l_ioctl(unsigned long cmd, void *arg, const char *what) +{ + int state_change = cmd == S3L_IOC_ENTER_IDLE || cmd == S3L_IOC_ENABLE_PREVIEW; + + if ((state_change ? s3l_ioctl_nosig(cmd, arg) : ioctl(_s3l_fd, cmd, arg)) < 0) { + HAL_DANGER("s3l_iav", "%s failed: %s\n", what, strerror(errno)); + return -1; + } + return 0; +} + +void s3l_hal_deinit(void) +{ + if (_s3l_fd >= 0) close(_s3l_fd); + _s3l_fd = -1; +} + +int s3l_hal_init(void) +{ + if ((_s3l_fd = open("/dev/iav", O_RDWR)) < 0) + HAL_ERROR("s3l_iav", "Cannot open /dev/iav: %s\n", strerror(errno)); + return EXIT_SUCCESS; +} + +static int s3l_nl_send(uint32_t type, uint32_t dir, uint32_t cmd, uint32_t status) +{ + struct { struct nlmsghdr hdr; struct s3l_nl_msg msg; } buf; + struct sockaddr_nl to = { .nl_family = AF_NETLINK }; + + memset(&buf, 0, sizeof(buf)); + buf.hdr.nlmsg_len = NLMSG_LENGTH(sizeof(buf.msg)); + buf.hdr.nlmsg_pid = getpid(); + buf.msg.pid = getpid(); + buf.msg.port = S3L_NL_PORT_IMAGE; + buf.msg.type = type; + buf.msg.dir = dir; + buf.msg.cmd = cmd; + buf.msg.status = status; + return sendto(_s3l_nl_sock, &buf, buf.hdr.nlmsg_len, 0, + (struct sockaddr *)&to, sizeof(to)); +} + +static void *s3l_nl_thread(void *arg) +{ + (void)arg; + while (_s3l_aaa_run) { + struct { struct nlmsghdr hdr; struct s3l_nl_msg msg; unsigned char extra[256]; } buf; + ssize_t n = recv(_s3l_nl_sock, &buf, sizeof(buf), 0); + if (n < 0) { + /* ENOBUFS: messages were dropped while this thread was starved; the session stands */ + if (errno == EINTR || errno == EAGAIN || errno == ENOBUFS) continue; + HAL_WARNING("s3l_aaa", "Image netlink receive failed: %s\n", strerror(errno)); + break; + } + if (n < (ssize_t)NLMSG_LENGTH(24)) continue; + + if (buf.msg.type == 1 && buf.msg.dir == 0) + s3l_nl_send(1, 1, buf.msg.cmd, 0); + } + return NULL; +} + +static void *s3l_stats_thread(void *arg) +{ + static unsigned char rgb[262144], cfa[262144], hist[65536], hdr[65536]; + unsigned char rv, cv, hv, dv; + unsigned int pts; + unsigned long long mpts; + void *stats[10] = { &rv, &cv, &hv, &dv, rgb, cfa, hist, hdr, &pts, &mpts }; + (void)arg; + + while (_s3l_aaa_run) { + /* Asked outside preview, the driver waits its 5 s out even if preview starts meanwhile */ + if (!_s3l_previewing) { usleep(20000); continue; } + rv = cv = 0; + if (ioctl(_s3l_fd, S3L_IOC_GET_STATISTICS, stats) < 0) + usleep(30000); + else if (cv && rv) + s3l_aaa_frame(cfa); + } + return NULL; +} + +static int s3l_aaa_start(void) +{ + struct sockaddr_nl me = { .nl_family = AF_NETLINK, .nl_pid = getpid() }; + struct timeval tv = { .tv_sec = 0, .tv_usec = 500000 }; + + if ((_s3l_nl_sock = socket(AF_NETLINK, SOCK_RAW, S3L_NL_PORT_IMAGE)) < 0) + HAL_ERROR("s3l_aaa", "Image netlink socket failed: %s\n", strerror(errno)); + if (bind(_s3l_nl_sock, (struct sockaddr *)&me, sizeof(me)) < 0) + HAL_ERROR("s3l_aaa", "Image netlink bind failed: %s\n", strerror(errno)); + setsockopt(_s3l_nl_sock, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv)); + if (s3l_nl_send(0, 0, 0 , 0) < 0) + HAL_ERROR("s3l_aaa", "Image netlink connect failed: %s\n", strerror(errno)); + + _s3l_aaa_run = 1; + if (pthread_create(&_s3l_nl_tid, NULL, s3l_nl_thread, NULL) || + pthread_create(&_s3l_stats_tid, NULL, s3l_stats_thread, NULL)) + HAL_ERROR("s3l_aaa", "Starting the 3A threads failed\n"); + return EXIT_SUCCESS; +} + +static void s3l_aaa_stop(void) +{ + if (!_s3l_aaa_run) return; + _s3l_aaa_run = 0; + pthread_join(_s3l_nl_tid, NULL); + pthread_join(_s3l_stats_tid, NULL); + close(_s3l_nl_sock); + _s3l_nl_sock = -1; +} + +/* Below the rings the HAL keeps its own blocks (LISO update slots, DBP sets, the mono curve) */ +#define S3L_IMG_SCRATCH_OFF (S3L_IMG_RINGS_OFF - 0x5000) + +static int s3l_img_load(unsigned char *img, unsigned int len, uint32_t phys, const char *path) +{ + struct s3l_isp_hdr hdr; + struct s3l_isp_op *ops = NULL; + uint32_t *rels = NULL; + static unsigned char page[4096]; + unsigned int written = 0, liso = 0; + int ret = EXIT_FAILURE; + FILE *file = fopen(path, "rb"); + + _s3l_liso_off = 0; + + if (!file) { + int e = errno; + HAL_WARNING("s3l_isp", "Cannot open %s: %s\n", path, strerror(e)); + return EXIT_FAILURE; + } + if (fread(&hdr, sizeof(hdr), 1, file) != 1 || memcmp(hdr.magic, S3L_ISP_MAGIC, 8) || + !hdr.ops || hdr.ops > 256 || hdr.rels > S3L_ISP_RELS_MAX || + hdr.img_off > S3L_IMG_SCRATCH_OFF || hdr.img_len > S3L_IMG_SCRATCH_OFF - hdr.img_off || + hdr.img_off + hdr.img_len > len) { + HAL_WARNING("s3l_isp", "%s is not an S3L ISP configuration (%s)\n", path, S3L_ISP_MAGIC); + goto out; + } + uint32_t img_off = hdr.img_off; + if (!(ops = malloc(hdr.ops * sizeof(*ops))) || + fread(ops, sizeof(*ops), hdr.ops, file) != hdr.ops || + (hdr.rels && (!(rels = malloc(hdr.rels * sizeof(*rels))) || + fread(rels, sizeof(*rels), hdr.rels, file) != hdr.rels))) + goto out; + + for (unsigned int o = 0; o < hdr.img_len; o += sizeof(page)) { + size_t want = MIN(sizeof(page), hdr.img_len - o), got = fread(page, 1, want, file); + if (got != want) { + HAL_WARNING("s3l_isp", "%s is cut short\n", path); + goto out; + } + + memcpy(img + img_off + o, page, got); + written++; + } + + /* Table pointers are IMG offsets in the file: point them at the buffer where the driver put it */ + for (unsigned int i = 0; i < hdr.rels; i++) { + uint32_t p; + if (rels[i] & 3 || rels[i] < img_off || rels[i] + 4 > img_off + hdr.img_len) goto bad_rel; + memcpy(&p, img + rels[i], 4); + if (p < img_off || p >= img_off + hdr.img_len) goto bad_rel; + p += phys; + memcpy(img + rels[i], &p, 4); + continue; +bad_rel: + HAL_WARNING("s3l_isp", "%s: relocation %u is outside the configuration\n", path, i); + goto out; + } + + for (unsigned int i = 0; i < hdr.ops; i++) { + uintptr_t blk = (uintptr_t)img + ops[i].offset; + int r; + if (ops[i].offset < img_off || ops[i].offset >= img_off + hdr.img_len || + ops[i].offset >= S3L_IMG_SCRATCH_OFF || (ops[i].kind != 's' && ops[i].kind != 'u')) goto out; + /* The LISO block is read and rewritten here later: all 0x400 bytes must be in the image */ + if (ops[i].kind == 's' && ops[i].nr == 0x60) { + if (ops[i].offset + 0x400 > img_off + hdr.img_len || ops[i].offset + 0x400 > S3L_IMG_SCRATCH_OFF) goto out; + liso = ops[i].offset; + } + if (ops[i].kind == 's') + r = ioctl(_s3l_fd, S3L_IOC_IMG_SET(ops[i].nr), blk); + else { + uint32_t arg[8] = { (uint32_t)blk, ops[i].flags }; + r = ioctl(_s3l_fd, S3L_IOC_IMG_UPDATE(ops[i].nr), arg); + } + if (r < 0) { + HAL_WARNING("s3l_isp", "ISP op %u ('%c' %#x) failed: %s\n", + i, ops[i].kind, ops[i].nr, strerror(errno)); + goto out; + } + } + HAL_INFO("s3l_isp", "Loaded %s (%u ops, %u pages)\n", path, hdr.ops, written); + _s3l_liso_off = liso; + ret = EXIT_SUCCESS; +out: + free(rels); + free(ops); + fclose(file); + return ret; +} + +static int s3l_img_prepare(void) +{ + uint32_t qb[16] = { S3L_BUFFER_IMG }; + unsigned char *img; + + if (s3l_ioctl(S3L_IOC_QUERY_BUF, qb, "QUERY_BUF(IMG)")) + return EXIT_FAILURE; + if (qb[1] < S3L_IMG_RINGS_OFF + S3L_IMG_RINGS_LEN) + HAL_ERROR("s3l_isp", "IMG buffer too small (%#x)\n", qb[1]); + img = mmap64(NULL, qb[1], PROT_READ | PROT_WRITE, MAP_SHARED, _s3l_fd, qb[2]); + if (img == MAP_FAILED) + HAL_ERROR("s3l_isp", "Mapping the IMG buffer failed: %s\n", strerror(errno)); + + /* After a cold boot the DSP would read uninitialised DRAM here as commands */ + memset(img + S3L_IMG_RINGS_OFF, 0, S3L_IMG_RINGS_LEN); + + /* Otherwise the one for the sensor the VIN reports, as OpenIPC's S3L package installs them, + if the VIN named one (a plain name: it becomes part of a path) */ + char def[64] = ""; + int named = *sensor && strcmp(sensor, "unidentified"); + for (const char *c = sensor; named && *c; c++) + named = isalnum((unsigned char)*c) || *c == '_' || *c == '-'; + if (named) + snprintf(def, sizeof(def), S3L_ISP_DIR "%s.bin", sensor); + /* Without one the DSP produces no frames at all, and a preview started that way cannot be left + until a reboot: stop here, with the DSP still idle */ + if ((!*_s3l_isp_path || s3l_img_load(img, qb[1], qb[2], _s3l_isp_path)) && + (!*def || !strcmp(_s3l_isp_path, def) || s3l_img_load(img, qb[1], qb[2], def))) { + munmap(img, qb[1]); + HAL_ERROR("s3l_isp", "No ISP configuration loaded (system.sensor_config%s%s); " + "the DSP produces no image without one\n", *def ? ", " : "", def); + } + + _s3l_img = img; + _s3l_img_len = qb[1]; + _s3l_img_phys = qb[2]; + return EXIT_SUCCESS; +} + +#define S3L_UPD_LISO_SLOTS (S3L_IMG_RINGS_OFF - 0x2000) +#define S3L_UPD_DBP_SETS (S3L_IMG_RINGS_OFF - 0x4000) +#define S3L_UPD_MONO_CURVE (S3L_IMG_RINGS_OFF - 0x5000) + +static unsigned int _s3l_dbp_set; +static uint8_t _s3l_pend_15c, _s3l_pend_1f4; + +static int s3l_isp_commit(uint8_t upd_15c, uint8_t upd_1f4) +{ + if (!_s3l_img || !_s3l_liso_off) return EXIT_FAILURE; + unsigned char *blk = _s3l_img + S3L_UPD_LISO_SLOTS + (_s3l_upd_slot++ & 1) * 0x400; + memcpy(blk, _s3l_img + _s3l_liso_off, 0x400); + blk[0x15c] |= upd_15c; + blk[0x1f4] |= upd_1f4; + uint32_t arg[8] = { (uint32_t)(uintptr_t)blk, 0x1 }; + if (ioctl(_s3l_fd, S3L_IOC_IMG_UPDATE(0x60), arg) < 0) { + static char told; /* retried every flush until it works: say it once */ + if (!told++) HAL_WARNING("s3l_isp", "LISO update failed: %s\n", strerror(errno)); + return EXIT_FAILURE; + } + return EXIT_SUCCESS; +} + +int s3l_isp_flush(void) +{ + /* Acquire: the block edits made before a bit was set are seen with it */ + uint8_t a = __atomic_exchange_n(&_s3l_pend_15c, 0, __ATOMIC_ACQUIRE); + uint8_t b = __atomic_exchange_n(&_s3l_pend_1f4, 0, __ATOMIC_ACQUIRE); + if (!a && !b) return EXIT_SUCCESS; + if (s3l_isp_commit(a, b)) { + /* Kept for the next flush: some changes (grayscale) are made only once */ + __atomic_fetch_or(&_s3l_pend_15c, a, __ATOMIC_RELAXED); + __atomic_fetch_or(&_s3l_pend_1f4, b, __ATOMIC_RELAXED); + return EXIT_FAILURE; + } + return EXIT_SUCCESS; +} + +int s3l_isp_get_wb(double *r, double *b) +{ + uint32_t g[4]; + if (!_s3l_img || !_s3l_liso_off) return EXIT_FAILURE; + memcpy(g, _s3l_img + _s3l_liso_off + 0xe4, sizeof(g)); + *r = g[0] / 65536.0; *b = g[3] / 65536.0; + return EXIT_SUCCESS; +} + +int s3l_isp_set_wb(double r, double b) +{ + if (!_s3l_img || !_s3l_liso_off) return EXIT_FAILURE; + uint32_t g[4] = { (uint32_t)(r * 65536), 0x10000, 0x10000, (uint32_t)(b * 65536) }; + memcpy(_s3l_img + _s3l_liso_off + 0xe4, g, sizeof(g)); + __atomic_fetch_or(&_s3l_pend_15c, 0x02, __ATOMIC_RELEASE); + return EXIT_SUCCESS; +} + +int s3l_isp_dbp_mode(void) +{ + return _s3l_img && _s3l_liso_off ? _s3l_img[_s3l_liso_off + 0x1f0] & 7 : 0; +} + +int s3l_isp_dbp_base(void) +{ + if (!_s3l_img || !_s3l_liso_off || !_s3l_img_phys) return 0; + uint32_t p; + memcpy(&p, _s3l_img + _s3l_liso_off + 0x160, 4); + if (p < _s3l_img_phys || p - _s3l_img_phys + 2 > _s3l_img_len) return 0; + return _s3l_img[p - _s3l_img_phys] | _s3l_img[p - _s3l_img_phys + 1] << 8; +} + +int s3l_isp_set_dbp(int thr) +{ + if (!_s3l_img || !_s3l_liso_off || !_s3l_img_phys) return EXIT_FAILURE; + if (thr < 0) thr = 0; + if (thr > 255) thr = 255; + unsigned int set = S3L_UPD_DBP_SETS + (_s3l_dbp_set++ & 1) * 0x800; + uint16_t *dark = (uint16_t *)(_s3l_img + set), *hot = (uint16_t *)(_s3l_img + set + 0x300); + for (int i = 0; i < 384; i++) dark[i] = hot[i] = thr; + uint32_t pd = _s3l_img_phys + set, ph = _s3l_img_phys + set + 0x300; + memcpy(_s3l_img + _s3l_liso_off + 0x160, &pd, 4); + memcpy(_s3l_img + _s3l_liso_off + 0x164, &ph, 4); + __atomic_fetch_or(&_s3l_pend_1f4, 0x03, __ATOMIC_RELEASE); + return EXIT_SUCCESS; +} + +static uint32_t _s3l_chroma_curve; + +void s3l_channel_grayscale(char enable) +{ + if (!_s3l_img || !_s3l_liso_off || !_s3l_img_phys) return; + uint32_t cur, mono = _s3l_img_phys + S3L_UPD_MONO_CURVE; + memcpy(&cur, _s3l_img + _s3l_liso_off + 0x120, 4); + if (enable) { + if (cur == mono) return; + _s3l_chroma_curve = cur; + memset(_s3l_img + S3L_UPD_MONO_CURVE, 0, 256); + memcpy(_s3l_img + _s3l_liso_off + 0x120, &mono, 4); + } else { + if (cur != mono || !_s3l_chroma_curve) return; + memcpy(_s3l_img + _s3l_liso_off + 0x120, &_s3l_chroma_curve, 4); + } + __atomic_fetch_or(&_s3l_pend_15c, 0x10, __ATOMIC_RELEASE); +} + +int s3l_channel_bind(char index) +{ + (void)index; + return EXIT_SUCCESS; +} + +int s3l_channel_unbind(char index) +{ + (void)index; + return EXIT_SUCCESS; +} + +int s3l_pipeline_create(short width, short height, char mirror, char flip, char framerate) +{ + static unsigned char buf[4096]; + struct s3l_vin_mode mode = { 0, 0, 0, 0 }; + struct s3l_vin_fps fps = { 0, S3L_VIN_FPS(framerate > 0 ? framerate : 30) }; + struct s3l_vin_info info = { 0 }; + unsigned int state; + uint32_t v[2]; + + if (s3l_aaa_start()) + return EXIT_FAILURE; + + if ((state = s3l_state_get()) != S3L_IAV_STATE_IDLE && + s3l_ioctl(S3L_IOC_ENTER_IDLE, 0, "ENTER_IDLE")) + return EXIT_FAILURE; + + if (s3l_ioctl(S3L_IOC_VIN_SET_MODE, &mode, "VIN_SET_MODE") || + s3l_ioctl(S3L_IOC_VIN_SET_FPS, &fps, "VIN_SET_FPS") || + s3l_ioctl(S3L_IOC_VIN_GET_VIDEOINFO, &info, "VIN_GET_VIDEOINFO")) + return EXIT_FAILURE; + if (!info.width || !info.height) + HAL_ERROR("s3l_iav", "The sensor driver reports no video mode\n"); + + { + uint32_t mir[3] = { 0, (uint32_t)((flip ? 1 : 0) | (mirror ? 2 : 0)), 255 }; + /* Every time, off included: the VIN keeps the last pattern across runs */ + s3l_ioctl(S3L_IOC_VIN_SET_MIRROR, mir, "VIN_SET_MIRROR"); + } + + _s3l_main_w = (width > 0 && width < (int)info.width) ? width : info.width; + _s3l_main_h = (height > 0 && height < (int)info.height) ? height : info.height; + HAL_INFO("s3l_iav", "VIN %ux%u, main buffer %ux%u\n", + info.width, info.height, _s3l_main_w, _s3l_main_h); + + memset(buf, 0, sizeof(buf)); + if (s3l_ioctl(S3L_IOC_GET_SYSRES, buf, "GET_SYSTEM_RESOURCE")) + return EXIT_FAILURE; + buf[S3L_SYSRES_ENC_MODE] = 0; + buf[S3L_SYSRES_MAX_NUM_ENC] = 2; + v[0] = _s3l_main_w; v[1] = _s3l_main_h; + memcpy(buf + S3L_SYSRES_BUF_MAX(0), v, 8); + memcpy(buf + S3L_SYSRES_STREAM_MAX(0), v, 8); + v[0] = 704; v[1] = 576; + memcpy(buf + S3L_SYSRES_STREAM_MAX(1), v, 8); + memset(buf + S3L_SYSRES_BUF_MAX(2), 0, 24); + memset(buf + S3L_SYSRES_STREAM_MAX(2), 0, 16); + v[0] = 1; memcpy(buf + S3L_SYSRES_C0, v, 4); + v[0] = 0; memcpy(buf + S3L_SYSRES_DC, v, 4); + memcpy(buf + S3L_SYSRES_EXTRA_A4, v, 4); + memcpy(buf + S3L_SYSRES_EXTRA_D8, v, 4); + if (s3l_ioctl(S3L_IOC_SET_SYSRES, buf, "SET_SYSTEM_RESOURCE")) + return EXIT_FAILURE; + + memset(buf, 0, sizeof(buf)); + if (s3l_ioctl(S3L_IOC_GET_SRCBUF_SETUP, buf, "GET_SRCBUF_SETUP")) + return EXIT_FAILURE; + for (int i = 0; i < 4; i++) { + uint32_t type; + memcpy(&type, buf + S3L_SRCBUF_TYPE(i), 4); + if (type == S3L_SRCBUF_TYPE_PREVIEW) { + type = 0; + memcpy(buf + S3L_SRCBUF_TYPE(i), &type, 4); + } + } + v[0] = _s3l_main_w; v[1] = _s3l_main_h; + memcpy(buf + S3L_SRCBUF_SIZE(0), v, 8); + memcpy(buf + S3L_SRCBUF_MAIN_WIN_A, v, 8); + + for (int k = 0; k < 4; k++) + memcpy(buf + S3L_SRCBUF_SUB_INPUT(k), v, 8); + v[0] = info.width; v[1] = info.height; + memcpy(buf + S3L_SRCBUF_MAIN_WIN_B, v, 8); + v[0] = 640; v[1] = 480; + memcpy(buf + S3L_SRCBUF_SIZE(1), v, 8); + if (s3l_ioctl(S3L_IOC_SET_SRCBUF_SETUP, buf, "SET_SRCBUF_SETUP")) + return EXIT_FAILURE; + + if (s3l_img_prepare()) + return EXIT_FAILURE; + + if (s3l_ioctl(S3L_IOC_ENABLE_PREVIEW, 0, "ENABLE_PREVIEW")) + return EXIT_FAILURE; + + /* The DSP takes the 3A statistics setup into effect only when preview starts, and it can + only be sent once preview runs: in the first preview after the microcode is loaded, the + SDK's 284712 sends no statistics and this camera's own 267837 sends stale ones. So the + setup is sent, left in place for a frame, and preview started again, before any stream + is encoding. Every later preview would do without it; it costs about 0.15 s. If the + driver refuses to go back to idle (an emulated DSP does), preview is still running: + carry on with the statistics as they are. */ + s3l_aaa_config(_s3l_fd); + usleep(100000); + if (s3l_ioctl_nosig(S3L_IOC_ENTER_IDLE, 0) < 0) { + int err = errno; + HAL_WARNING("s3l_iav", "Restarting preview for the 3A statistics was refused (%s); " + "they may stay stale until the next start\n", strerror(err)); + } else { + if (s3l_ioctl(S3L_IOC_ENABLE_PREVIEW, 0, "ENABLE_PREVIEW")) + return EXIT_FAILURE; + s3l_aaa_config(_s3l_fd); + } + _s3l_previewing = 1; + s3l_ae_start(_s3l_fd, framerate > 0 ? framerate : 30); + + { + uint32_t qb[16] = { S3L_BUFFER_BSB }; + if (s3l_ioctl(S3L_IOC_QUERY_BUF, qb, "QUERY_BUF(BSB)")) + return EXIT_FAILURE; + _s3l_bsb_len = qb[1]; + _s3l_bsb = mmap64(NULL, qb[1], PROT_READ, MAP_SHARED, _s3l_fd, qb[2]); + if (_s3l_bsb == MAP_FAILED) { + _s3l_bsb = NULL; + HAL_ERROR("s3l_iav", "Mapping the bitstream buffer failed: %s\n", strerror(errno)); + } + } + + return EXIT_SUCCESS; +} + +void s3l_pipeline_destroy(void) +{ + s3l_ae_stop(); + if (_s3l_bsb) munmap(_s3l_bsb, _s3l_bsb_len); + _s3l_bsb = NULL; + _s3l_previewing = 0; + if (s3l_state_get() != S3L_IAV_STATE_IDLE) + s3l_ioctl_nosig(S3L_IOC_ENTER_IDLE, 0); + s3l_aaa_stop(); +} + +int s3l_video_create(char index, hal_vidconfig *config) +{ + static unsigned char buf[256]; + uint32_t v[4]; + + if (index < 0 || index >= S3L_VENC_CHN_NUM) + HAL_ERROR("s3l_venc", "Only %d encoder channel(s) are supported!\n", S3L_VENC_CHN_NUM); + if (config->codec != HAL_VIDCODEC_H264) + HAL_ERROR("s3l_venc", "Only H.264 is supported so far!\n"); + + memset(buf, 0, sizeof(buf)); + buf[S3L_STRFMT_ID] = index; + if (s3l_ioctl(S3L_IOC_GET_STREAM_FORMAT, buf, "GET_STREAM_FORMAT")) + return EXIT_FAILURE; + v[0] = S3L_STREAM_TYPE_H264; + memcpy(buf + S3L_STRFMT_TYPE, v, 4); + v[0] = 0; + memcpy(buf + S3L_STRFMT_BUF_ID, v, 4); + v[0] = _s3l_main_w; v[1] = _s3l_main_h; v[2] = 0; v[3] = 0; + memcpy(buf + S3L_STRFMT_ENC_WIN, v, 16); + if (s3l_ioctl(S3L_IOC_SET_STREAM_FORMAT, buf, "SET_STREAM_FORMAT")) + return EXIT_FAILURE; + + memset(buf, 0, sizeof(buf)); + buf[0] = index; + if (s3l_ioctl(S3L_IOC_GET_H264_CFG, buf, "GET_H264_CFG") || + s3l_ioctl(S3L_IOC_SET_H264_CFG, buf, "SET_H264_CFG")) + return EXIT_FAILURE; + + if (s3l_ioctl_nosig(S3L_IOC_START_ENCODE, (void *)(1u << index)) < 0) + HAL_ERROR("s3l_venc", "Starting encoder %d failed: %s\n", index, strerror(errno)); + + s3l_state[index].payload = config->codec; + s3l_state[index].enable = 1; + return EXIT_SUCCESS; +} + +int s3l_video_destroy(char index) +{ + if (index < 0 || index >= S3L_VENC_CHN_NUM) return EXIT_FAILURE; + if (s3l_state[index].enable) + s3l_ioctl_nosig(S3L_IOC_STOP_ENCODE, (void *)(1u << index)); + s3l_state[index].enable = 0; + s3l_state[index].payload = HAL_VIDCODEC_UNSPEC; + return EXIT_SUCCESS; +} + +int s3l_video_destroy_all(void) +{ + for (char i = 0; i < S3L_VENC_CHN_NUM; i++) + s3l_video_destroy(i); + return EXIT_SUCCESS; +} + +void s3l_video_request_idr(char index) +{ + (void)index; +} + +void *s3l_video_thread(void) +{ + static uint32_t desc[S3L_SZ_QUERYDESC / 4 + 64]; + unsigned char *wrapBuf = NULL; + unsigned int wrapLen = 0; + + while (keepRunning) { + if (!_s3l_bsb || !s3l_state[0].enable) { usleep(100000); continue; } + + memset(desc, 0, sizeof(desc)); + desc[0] = S3L_DESC_FRAME; + desc[S3L_DESC_W_ID] = 0xffffffff; + desc[S3L_DESC_W_TIMEOUT_MS] = 500; + if (ioctl(_s3l_fd, S3L_IOC_QUERY_DESC, desc) < 0) { + if (errno != EAGAIN && errno != ETIMEDOUT && errno != EINTR) { + HAL_WARNING("s3l_venc", "Querying a frame failed: %s\n", strerror(errno)); + usleep(100000); + } + continue; + } + + unsigned int off = desc[S3L_DESC_W_DATA_OFF], size = desc[S3L_DESC_W_SIZE]; + if (off >= _s3l_bsb_len || !size || size > _s3l_bsb_len) continue; + + unsigned char *data = _s3l_bsb + off; + if (off + size > _s3l_bsb_len) { + if (wrapLen < size) { + free(wrapBuf); + wrapBuf = malloc(wrapLen = size); + if (!wrapBuf) { wrapLen = 0; continue; } + } + unsigned int first = _s3l_bsb_len - off; + memcpy(wrapBuf, data, first); + memcpy(wrapBuf + first, _s3l_bsb, size - first); + data = wrapBuf; + } + + if (!s3l_vid_cb) continue; + + struct timespec ts; + clock_gettime(CLOCK_MONOTONIC, &ts); + + hal_vidstream outStrm; + hal_vidpack outPack[1]; + memset(outPack, 0, sizeof(outPack)); + outStrm.count = 1; + outStrm.seq = desc[S3L_DESC_W_FRAME_NUM]; + outStrm.pack = outPack; + outPack[0].data = data; + outPack[0].length = size; + outPack[0].offset = 0; + outPack[0].timestamp = ts.tv_sec * 1000000ull + ts.tv_nsec / 1000; + + unsigned int n = 0, sc, scanOff = 0; + while (n < 7 && (sc = nal_find_startcode(data, scanOff, size)) < size) { + unsigned int scLen = (data[sc + 2] == 1) ? 3 : 4; + if (sc + scLen >= size) break; + outPack[0].nalu[n].type = data[sc + scLen] & 0x1F; + outPack[0].nalu[n].offset = sc; + n++; + scanOff = sc + scLen; + } + outPack[0].naluCnt = n; + outPack[0].nalu[n].offset = size; + for (unsigned int i = 0; i < n; i++) + outPack[0].nalu[i].length = outPack[0].nalu[i + 1].offset - outPack[0].nalu[i].offset; + + (*s3l_vid_cb)(0, &outStrm); + } + + free(wrapBuf); + HAL_INFO("s3l_venc", "Shutting down encoding thread...\n"); + return NULL; +} + +void s3l_system_deinit(void) +{ +} + +int s3l_system_init(char *snrConfig) +{ + unsigned int state = s3l_state_get(); + snprintf(_s3l_isp_path, sizeof(_s3l_isp_path), "%s", snrConfig ? snrConfig : ""); + if (state == 0xdeadbeef) + HAL_ERROR("s3l_iav", "Cannot read the IAV state; are the IAV modules and DSP ucode loaded?\n"); + HAL_INFO("s3l_iav", "IAV state %#x\n", state); + + unsigned char info[60] = { 0 }; + if (!ioctl(_s3l_fd, S3L_IOC_VIN_GET_DEVINFO, info) && info[8]) + snprintf(sensor, sizeof(sensor), "%.*s", (int)sizeof(sensor) - 1, (char *)info + 8); + return EXIT_SUCCESS; +} + +int s3l_system_identify(void) +{ + FILE *file; + char line[64], soc[16] = ""; + + if (access("/dev/iav", F_OK) || access("/dev/ucode", F_OK)) + return 0; + if (!(file = fopen("/proc/ambarella/iav", "r"))) + return 0; + while (fgets(line, sizeof(line), file)) + if (sscanf(line, "AMBARELLA_CHIP=%15s", soc) == 1) + break; + fclose(file); + if (strncmp(soc, "S3L", 3)) + return 0; + + plat = HAL_PLATFORM_S3L; + strcpy(family, "ambarella"); + strcpy(chip, soc); + + chnCount = S3L_VENC_CHN_NUM; + chnState = (hal_chnstate*)s3l_state; + vid_thread = s3l_video_thread; + return 1; +} + +#endif diff --git a/src/hal/amba/s3l_hal.h b/src/hal/amba/s3l_hal.h new file mode 100644 index 00000000..5d0656fa --- /dev/null +++ b/src/hal/amba/s3l_hal.h @@ -0,0 +1,55 @@ +#pragma once + +#include "s3l_iav.h" + +#include "../globals.h" +#include "../tools.h" +#include "../../fmt/nal.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define S3L_VENC_CHN_NUM 1 + +extern hal_chnstate s3l_state[S3L_VENC_CHN_NUM]; +extern int (*s3l_aud_cb)(hal_audframe*); +extern int (*s3l_vid_cb)(char, hal_vidstream*); + +void s3l_hal_deinit(void); +int s3l_hal_init(void); + +void s3l_channel_grayscale(char enable); +int s3l_channel_bind(char index); +int s3l_channel_unbind(char index); + +int s3l_pipeline_create(short width, short height, char mirror, char flip, char framerate); +void s3l_pipeline_destroy(void); + +int s3l_video_create(char index, hal_vidconfig *config); +int s3l_video_destroy(char index); +int s3l_video_destroy_all(void); +void s3l_video_request_idr(char index); +void *s3l_video_thread(void); + +int s3l_aaa_config(int fd); +int s3l_ae_start(int fd, int fps); +int s3l_isp_set_wb(double r, double b); +int s3l_isp_get_wb(double *r, double *b); +int s3l_isp_dbp_mode(void); +int s3l_isp_dbp_base(void); +int s3l_isp_set_dbp(int thr); +int s3l_isp_flush(void); +void s3l_aaa_options(char ae, int target, char awb, int antiflicker); +void s3l_ae_stop(void); +void s3l_aaa_frame(const unsigned char *cfa); + +void s3l_system_deinit(void); +int s3l_system_init(char *snrConfig); +int s3l_system_identify(void); diff --git a/src/hal/amba/s3l_iav.h b/src/hal/amba/s3l_iav.h new file mode 100644 index 00000000..4a1f9920 --- /dev/null +++ b/src/hal/amba/s3l_iav.h @@ -0,0 +1,95 @@ +#pragma once + +#include + +#define S3L_IOC_GET_IAV_STATE 0x80045610u +#define S3L_IOC_ENTER_IDLE 0x00005611u +#define S3L_IOC_ENABLE_PREVIEW 0x40045612u +#define S3L_IOC_START_ENCODE 0x40045613u +#define S3L_IOC_STOP_ENCODE 0x40045614u +#define S3L_IOC_SET_SYSRES 0x40045615u +#define S3L_IOC_GET_SYSRES 0xc0045615u +#define S3L_IOC_QUERY_BUF 0xc0045618u +#define S3L_IOC_QUERY_DESC 0xc0045619u +#define S3L_IOC_SET_SRCBUF_SETUP 0x40045630u +#define S3L_IOC_GET_SRCBUF_SETUP 0xc0045630u +#define S3L_IOC_SET_STREAM_FORMAT 0x40045634u +#define S3L_IOC_GET_STREAM_FORMAT 0xc0045634u +#define S3L_IOC_SET_H264_CFG 0x40045638u +#define S3L_IOC_GET_H264_CFG 0xc0045638u +#define S3L_IOC_GET_STATISTICS 0x80047080u +#define S3L_IOC_IMG_CONFIG_STATISTICS 0x40047001u + +#define S3L_IOC_VIN_SET_MODE 0x40107303u +#define S3L_IOC_VIN_SET_FPS 0x40087304u +#define S3L_IOC_VIN_GET_DEVINFO 0x803c7301u +#define S3L_IOC_VIN_GET_FPS 0x80087304u +#define S3L_IOC_VIN_GET_VIDEOINFO 0x80247305u +#define S3L_IOC_VIN_SET_MIRROR 0x400c730au + +#define S3L_SZ_SYSRES 464 +#define S3L_SZ_SRCBUF_SETUP 232 +#define S3L_SZ_STREAM_FMT 36 +#define S3L_SZ_H264_CFG 52 +#define S3L_SZ_QUERYDESC 960 + +#define S3L_IAV_STATE_INIT 0xff +#define S3L_IAV_STATE_IDLE 0 +#define S3L_IAV_STATE_PREVIEW 1 +#define S3L_IAV_STATE_ENCODING 2 + +#define S3L_BUFFER_BSB 1 +#define S3L_DESC_FRAME 0 +#define S3L_STREAM_TYPE_H264 1 + +#define S3L_SRCBUF_TYPE_PREVIEW 2 + +#define S3L_VIN_FPS(fps) ((uint32_t)((512000000ull + (fps) / 2) / (fps))) + +struct s3l_vin_mode { uint32_t vsrc_id, mode, hdr_mode, bits; }; +struct s3l_vin_fps { uint32_t vsrc_id, fps; }; +struct s3l_vin_info { uint32_t vsrc_id, video_mode, width, height, fps, rsvd[4]; }; + +#define S3L_SYSRES_ENC_MODE 0x00 +#define S3L_SYSRES_MAX_NUM_ENC 0x01 +#define S3L_SYSRES_BUF_MAX(i) (0x04 + 8 * (i)) +#define S3L_SYSRES_STREAM_MAX(i) (0x3c + 8 * (i)) +#define S3L_SYSRES_EXTRA_A4 0xa4 +#define S3L_SYSRES_C0 0xc0 +#define S3L_SYSRES_EXTRA_D8 0xd8 +#define S3L_SYSRES_DC 0xdc + +#define S3L_SRCBUF_SIZE(i) (0x00 + 8 * (i)) +#define S3L_SRCBUF_MAIN_WIN_A 0x30 +#define S3L_SRCBUF_MAIN_WIN_B 0x38 +#define S3L_SRCBUF_SUB_INPUT(k) (0x48 + 16 * (k)) +#define S3L_SRCBUF_TYPE(i) (0xa8 + 4 * (i)) + +#define S3L_STRFMT_ID 0x00 +#define S3L_STRFMT_TYPE 0x04 +#define S3L_STRFMT_BUF_ID 0x08 +#define S3L_STRFMT_ENC_WIN 0x0c + +#define S3L_DESC_W_ID 2 +#define S3L_DESC_W_TIMEOUT_MS 3 +#define S3L_DESC_W_DATA_OFF 4 +#define S3L_DESC_W_PIC_TYPE 6 +#define S3L_DESC_W_FRAME_NUM 7 +#define S3L_DESC_W_SIZE 13 + +#define S3L_BUFFER_IMG 8 +#define S3L_IMG_RINGS_OFF 0x500000 +#define S3L_IMG_RINGS_LEN 0x3000 +#define S3L_IOC_IMG_SET(nr) (0x40017000u | (nr)) +#define S3L_IOC_IMG_UPDATE(nr) (0x40047000u | (nr)) + +#define S3L_ISP_MAGIC "S3LISP03" +#define S3L_ISP_DIR "/etc/sensors/" +#define S3L_ISP_RELS_MAX 1024 +/* Offsets are from the start of the IMG buffer; each relocation is a pointer word holding an IMG offset */ +struct s3l_isp_hdr { char magic[8]; uint32_t ops, img_len, img_off, rels; }; +struct s3l_isp_op { uint8_t kind, nr, rsvd[2]; uint32_t offset, flags; }; + +#define S3L_NL_PORT_IMAGE 20 +#define S3L_NL_MSG_LEN 128 +struct s3l_nl_msg { uint32_t pid, port, type, dir, cmd, status; uint8_t pad[S3L_NL_MSG_LEN - 24]; }; diff --git a/src/hal/support.c b/src/hal/support.c index 0292bc44..b5d66b83 100644 --- a/src/hal/support.c +++ b/src/hal/support.c @@ -17,6 +17,9 @@ unsigned long long lastMillisTemp = 0; float lastReadTemp = 0.0 / 0.0; void hal_identify(void) { +#if defined(__arm__) + if (s3l_system_identify()) return; +#endif #if defined(__ARM_PCS_VFP) if (i6_system_identify()) return; if (i6c_system_identify()) return; diff --git a/src/hal/support.h b/src/hal/support.h index 4ed80370..d2588b91 100644 --- a/src/hal/support.h +++ b/src/hal/support.h @@ -7,6 +7,9 @@ #include #include +#if defined(__arm__) +#include "amba/s3l_hal.h" +#endif #if defined(__ARM_PCS_VFP) #include "star/i3_hal.h" #include "star/i6_hal.h" diff --git a/src/hal/types.h b/src/hal/types.h index 694b93ca..1ba326f1 100644 --- a/src/hal/types.h +++ b/src/hal/types.h @@ -14,6 +14,7 @@ typedef enum { HAL_PLATFORM_I6C, HAL_PLATFORM_M6, HAL_PLATFORM_RK, + HAL_PLATFORM_S3L, HAL_PLATFORM_T31, HAL_PLATFORM_V1, HAL_PLATFORM_V2, diff --git a/src/media.c b/src/media.c index 461ed13c..fb305198 100644 --- a/src/media.c +++ b/src/media.c @@ -253,6 +253,14 @@ int save_video_stream(char index, hal_vidstream *stream) { int media_start(void) { int ret = EXIT_SUCCESS; +#if defined(__arm__) + /* Before the stream destinations are told about an audio track */ + if (plat == HAL_PLATFORM_S3L && app_config.audio_enable) { + HAL_WARNING("media", "No audio on this platform yet, continuing without it\n"); + app_config.audio_enable = false; + } +#endif + for (int i = 0; app_config.stream_dests[i] && *app_config.stream_dests[i]; i++) { if (STARTS_WITH(app_config.stream_dests[i], "rtmp://")) { flv_set_config(app_config.mp4_width, app_config.mp4_height, app_config.mp4_fps, @@ -327,6 +335,9 @@ void request_idr(void) { break; } if (index != -1) switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: s3l_video_request_idr(index); break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: i6_video_request_idr(index); break; case HAL_PLATFORM_I6C: i6c_video_request_idr(index); break; @@ -355,6 +366,9 @@ void request_idr(void) { void set_grayscale(bool active) { pthread_mutex_lock(&chnMtx); switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: s3l_channel_grayscale(active); break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: i6_channel_grayscale(active); break; case HAL_PLATFORM_I6C: i6c_channel_grayscale(active); break; @@ -394,6 +408,9 @@ int take_next_free_channel(bool mainLoop) { int create_channel(char index, short width, short height, char framerate, char jpeg) { switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: return EXIT_SUCCESS; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: return i6_channel_create(index, width, height, jpeg); case HAL_PLATFORM_I6C: return i6c_channel_create(index, width, height, jpeg); @@ -427,6 +444,9 @@ int create_channel(char index, short width, short height, char framerate, char j int bind_channel(char index, char framerate, char jpeg) { switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: return s3l_channel_bind(index); +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: return i6_channel_bind(index, framerate); case HAL_PLATFORM_I6C: return i6c_channel_bind(index, framerate); @@ -453,6 +473,9 @@ int bind_channel(char index, char framerate, char jpeg) { int unbind_channel(char index, char jpeg) { switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: return s3l_channel_unbind(index); +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: return i6_channel_unbind(index); case HAL_PLATFORM_I6C: return i6c_channel_unbind(index); @@ -479,6 +502,9 @@ int unbind_channel(char index, char jpeg) { int media_video_disable(char index, char jpeg) { switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: return s3l_video_destroy(index); +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: return i6_video_destroy(index); case HAL_PLATFORM_I6C: return i6c_video_destroy(index); @@ -544,6 +570,15 @@ int media_audio_enable(void) { if (audioOn) return ret; +#if defined(__arm__) + /* No audio on S3L yet: off for this run, so nothing serves or advertises an empty track */ + if (plat == HAL_PLATFORM_S3L) { + HAL_WARNING("media", "No audio on this platform yet, continuing without it\n"); + app_config.audio_enable = false; + return EXIT_SUCCESS; + } +#endif + if (ringbuf_init(&audRing, AUD_RING_CAP)) HAL_ERROR("media", "Audio queue initialization failed!\n"); @@ -672,6 +707,9 @@ int media_mjpeg_enable(void) { config.maxBitrate = app_config.mjpeg_bitrate * 5 / 4; switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: ret = s3l_video_create(index, &config); break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: ret = i6_video_create(index, &config); break; case HAL_PLATFORM_I6C: ret = i6c_video_create(index, &config); break; @@ -765,6 +803,9 @@ int media_mp4_enable(void) { config.maxBitrate = app_config.mp4_bitrate * 5 / 4; switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: ret = s3l_video_create(index, &config); break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: ret = i6_video_create(index, &config); break; case HAL_PLATFORM_I6C: ret = i6c_video_create(index, &config); break; @@ -818,6 +859,9 @@ int sdk_start(void) { int ret; switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: ret = s3l_hal_init(); break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I3: ret = i3_hal_init(); break; case HAL_PLATFORM_I6: ret = i6_hal_init(); break; @@ -846,6 +890,12 @@ int sdk_start(void) { ret, errstr(ret)); switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: + s3l_aud_cb = save_audio_stream; + s3l_vid_cb = save_video_stream; + break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: i6_aud_cb = save_audio_stream; @@ -912,6 +962,9 @@ int sdk_start(void) { } switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: ret = s3l_system_init(app_config.sensor_config); break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I3: ret = i3_system_init(); break; case HAL_PLATFORM_I6: ret = i6_system_init(); break; @@ -951,6 +1004,17 @@ int sdk_start(void) { short framerate = MAX(app_config.mp4_fps, app_config.mjpeg_fps); switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: + s3l_aaa_options(app_config.auto_exposure, app_config.exposure_target, + app_config.auto_white_balance, app_config.antiflicker); + /* The main buffer is the H.264/H.265 stream's: a larger MJPEG shares it at its size */ + ret = s3l_pipeline_create(app_config.mp4_enable ? app_config.mp4_width : width, + app_config.mp4_enable ? app_config.mp4_height : height, app_config.mirror, app_config.flip, + /* The VIN runs at the fastest enabled stream's rate */ + MAX(app_config.mp4_enable ? app_config.mp4_fps : 0, + app_config.mjpeg_enable ? app_config.mjpeg_fps : 0) ?: framerate); break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: ret = i6_pipeline_create(0, width, height, app_config.mirror, app_config.flip, framerate); break; @@ -1007,7 +1071,9 @@ int sdk_start(void) { if (app_config.mp4_enable && (ret = media_mp4_enable())) HAL_ERROR("media", "MP4 initialization failed with %#x!\n", ret); - if (app_config.mjpeg_enable && (ret = media_mjpeg_enable())) + if (app_config.mjpeg_enable && plat == HAL_PLATFORM_S3L) + HAL_WARNING("media", "MJPEG is not supported on this platform yet, skipping\n"); + else if (app_config.mjpeg_enable && (ret = media_mjpeg_enable())) HAL_ERROR("media", "MJPEG initialization failed with %#x!\n", ret); if (app_config.jpeg_enable && (ret = jpeg_init())) @@ -1034,7 +1100,8 @@ int sdk_start(void) { pthread_attr_destroy(&thread_attr); } - if (!access(app_config.sensor_config, F_OK) && !sleep(1)) + /* S3L reads its ISP configuration itself, when the pipeline is created */ + if (plat != HAL_PLATFORM_S3L && !access(app_config.sensor_config, F_OK) && !sleep(1)) switch (plat) { #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I3: i3_config_load(app_config.sensor_config); break; @@ -1061,6 +1128,9 @@ int sdk_stop(void) { jpeg_deinit(); switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: s3l_video_destroy_all(); break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: i6_video_destroy_all(); break; case HAL_PLATFORM_I6C: i6c_video_destroy_all(); break; @@ -1085,6 +1155,9 @@ int sdk_stop(void) { } switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: s3l_pipeline_destroy(); break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: i6_pipeline_destroy(); break; case HAL_PLATFORM_I6C: i6c_pipeline_destroy(); break; @@ -1115,6 +1188,9 @@ int sdk_stop(void) { pthread_join(ispPid, NULL); switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: s3l_system_deinit(); break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I3: i3_system_deinit(); break; case HAL_PLATFORM_I6: i6_system_deinit(); break; @@ -1149,6 +1225,9 @@ int sdk_stop(void) { } switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: s3l_hal_deinit(); break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I3: i3_hal_deinit(); break; case HAL_PLATFORM_I6: i6_hal_deinit(); break; diff --git a/src/server.c b/src/server.c index 0b113a98..4f966b36 100644 --- a/src/server.c +++ b/src/server.c @@ -891,9 +891,9 @@ void respond_request(http_request_t *req) { app_config.audio_bitrate = (unsigned int)result; } else if (EQUALS(key, "enable")) { if (EQUALS_CASE(value, "true") || EQUALS(value, "1")) - app_config.audio_enable = 1; + app_config.audio_enable = app_config.audio_requested = 1; else if (EQUALS_CASE(value, "false") || EQUALS(value, "0")) - app_config.audio_enable = 0; + app_config.audio_enable = app_config.audio_requested = 0; } else if (EQUALS(key, "gain")) { long result; if (parse_ranged(value, -60, 30, &result)) From 9240439bd35e85a8d0d971046dafac5b405378f7 Mon Sep 17 00:00:00 2001 From: Dylan Corrales Date: Fri, 6 Nov 2026 16:23:44 -0500 Subject: [PATCH 02/11] S3L: MJPEG on the second encoder stream MJPEG was skipped on S3L. It now runs as its own IAV stream (MJPEG stream type) from the second source buffer, sized to mjpeg width/height (s3l_sub_size(), called before the pipeline is created), with the quality from mjpeg.qfactor (which media_mjpeg_enable passes on for every platform since #68). When MJPEG is the only stream, the main buffer already has its size and a sub-buffer cannot match it, so MJPEG encodes from the main buffer instead. The frame thread routes each frame by the stream id in its descriptor and only splits NAL units for H.264. Tested on the IP4M-1026E: H.264 2688x1512 over RTSP plus 640x360 MJPEG at 20 fps, qfactor 30 and 90 giving the matching JPEG quantisation tables (8.8 KB and 27 KB frames); MJPEG-only at 1280x720. --- src/hal/amba/s3l_hal.c | 58 ++++++++++++++++++++++++++++++++---------- src/hal/amba/s3l_hal.h | 3 ++- src/hal/amba/s3l_iav.h | 4 +++ src/media.c | 6 ++--- 4 files changed, 54 insertions(+), 17 deletions(-) diff --git a/src/hal/amba/s3l_hal.c b/src/hal/amba/s3l_hal.c index 02150e59..0fa8cd43 100644 --- a/src/hal/amba/s3l_hal.c +++ b/src/hal/amba/s3l_hal.c @@ -17,6 +17,8 @@ static int _s3l_fd = -1; static unsigned char *_s3l_bsb; static unsigned int _s3l_bsb_len; static unsigned short _s3l_main_w, _s3l_main_h; +static unsigned short _s3l_sub_w = 640, _s3l_sub_h = 480; +static char _s3l_mjpeg_buf = 1; static char _s3l_isp_path[128]; static unsigned char *_s3l_img; @@ -407,6 +409,14 @@ int s3l_channel_unbind(char index) return EXIT_SUCCESS; } +void s3l_sub_size(short width, short height) +{ + if (width > 0 && height > 0) { + _s3l_sub_w = (width + 15) & ~15; + _s3l_sub_h = (height + 7) & ~7; + } +} + int s3l_pipeline_create(short width, short height, char mirror, char flip, char framerate) { static unsigned char buf[4096]; @@ -441,6 +451,13 @@ int s3l_pipeline_create(short width, short height, char mirror, char flip, char HAL_INFO("s3l_iav", "VIN %ux%u, main buffer %ux%u\n", info.width, info.height, _s3l_main_w, _s3l_main_h); + /* A sub-buffer must be smaller than the main one */ + _s3l_mjpeg_buf = _s3l_sub_w < _s3l_main_w && _s3l_sub_h < _s3l_main_h; + if (!_s3l_mjpeg_buf) { + _s3l_sub_w = (_s3l_main_w / 2) & ~15; + _s3l_sub_h = (_s3l_main_h / 2) & ~7; + } + memset(buf, 0, sizeof(buf)); if (s3l_ioctl(S3L_IOC_GET_SYSRES, buf, "GET_SYSTEM_RESOURCE")) return EXIT_FAILURE; @@ -449,7 +466,8 @@ int s3l_pipeline_create(short width, short height, char mirror, char flip, char v[0] = _s3l_main_w; v[1] = _s3l_main_h; memcpy(buf + S3L_SYSRES_BUF_MAX(0), v, 8); memcpy(buf + S3L_SYSRES_STREAM_MAX(0), v, 8); - v[0] = 704; v[1] = 576; + v[0] = _s3l_sub_w; v[1] = _s3l_sub_h; + memcpy(buf + S3L_SYSRES_BUF_MAX(1), v, 8); memcpy(buf + S3L_SYSRES_STREAM_MAX(1), v, 8); memset(buf + S3L_SYSRES_BUF_MAX(2), 0, 24); memset(buf + S3L_SYSRES_STREAM_MAX(2), 0, 16); @@ -479,7 +497,7 @@ int s3l_pipeline_create(short width, short height, char mirror, char flip, char memcpy(buf + S3L_SRCBUF_SUB_INPUT(k), v, 8); v[0] = info.width; v[1] = info.height; memcpy(buf + S3L_SRCBUF_MAIN_WIN_B, v, 8); - v[0] = 640; v[1] = 480; + v[0] = _s3l_sub_w; v[1] = _s3l_sub_h; memcpy(buf + S3L_SRCBUF_SIZE(1), v, 8); if (s3l_ioctl(S3L_IOC_SET_SRCBUF_SETUP, buf, "SET_SRCBUF_SETUP")) return EXIT_FAILURE; @@ -544,26 +562,38 @@ int s3l_video_create(char index, hal_vidconfig *config) if (index < 0 || index >= S3L_VENC_CHN_NUM) HAL_ERROR("s3l_venc", "Only %d encoder channel(s) are supported!\n", S3L_VENC_CHN_NUM); - if (config->codec != HAL_VIDCODEC_H264) - HAL_ERROR("s3l_venc", "Only H.264 is supported so far!\n"); + if (config->codec != HAL_VIDCODEC_H264 && config->codec != HAL_VIDCODEC_MJPG) + HAL_ERROR("s3l_venc", "Only H.264 and MJPEG are supported so far!\n"); + char mjpeg = config->codec == HAL_VIDCODEC_MJPG; memset(buf, 0, sizeof(buf)); buf[S3L_STRFMT_ID] = index; if (s3l_ioctl(S3L_IOC_GET_STREAM_FORMAT, buf, "GET_STREAM_FORMAT")) return EXIT_FAILURE; - v[0] = S3L_STREAM_TYPE_H264; + v[0] = mjpeg ? S3L_STREAM_TYPE_MJPEG : S3L_STREAM_TYPE_H264; memcpy(buf + S3L_STRFMT_TYPE, v, 4); - v[0] = 0; + char sub = mjpeg && _s3l_mjpeg_buf; + v[0] = sub; memcpy(buf + S3L_STRFMT_BUF_ID, v, 4); - v[0] = _s3l_main_w; v[1] = _s3l_main_h; v[2] = 0; v[3] = 0; + v[0] = sub ? _s3l_sub_w : _s3l_main_w; + v[1] = sub ? _s3l_sub_h : _s3l_main_h; + v[2] = 0; v[3] = 0; memcpy(buf + S3L_STRFMT_ENC_WIN, v, 16); if (s3l_ioctl(S3L_IOC_SET_STREAM_FORMAT, buf, "SET_STREAM_FORMAT")) return EXIT_FAILURE; memset(buf, 0, sizeof(buf)); buf[0] = index; - if (s3l_ioctl(S3L_IOC_GET_H264_CFG, buf, "GET_H264_CFG") || - s3l_ioctl(S3L_IOC_SET_H264_CFG, buf, "SET_H264_CFG")) + if (mjpeg) { + if (s3l_ioctl(S3L_IOC_GET_MJPEG_CFG, buf, "GET_MJPEG_CFG")) + return EXIT_FAILURE; + unsigned char quality = MAX(config->minQual, config->maxQual); + if (quality >= 1 && quality <= 100) + buf[5] = quality; + if (s3l_ioctl(S3L_IOC_SET_MJPEG_CFG, buf, "SET_MJPEG_CFG")) + return EXIT_FAILURE; + } else if (s3l_ioctl(S3L_IOC_GET_H264_CFG, buf, "GET_H264_CFG") || + s3l_ioctl(S3L_IOC_SET_H264_CFG, buf, "SET_H264_CFG")) return EXIT_FAILURE; if (s3l_ioctl_nosig(S3L_IOC_START_ENCODE, (void *)(1u << index)) < 0) @@ -603,7 +633,7 @@ void *s3l_video_thread(void) unsigned int wrapLen = 0; while (keepRunning) { - if (!_s3l_bsb || !s3l_state[0].enable) { usleep(100000); continue; } + if (!_s3l_bsb || !(s3l_state[0].enable || s3l_state[1].enable)) { usleep(100000); continue; } memset(desc, 0, sizeof(desc)); desc[0] = S3L_DESC_FRAME; @@ -633,7 +663,8 @@ void *s3l_video_thread(void) data = wrapBuf; } - if (!s3l_vid_cb) continue; + unsigned int id = desc[S3L_DESC_W_ID]; + if (!s3l_vid_cb || id >= S3L_VENC_CHN_NUM || !s3l_state[id].enable) continue; struct timespec ts; clock_gettime(CLOCK_MONOTONIC, &ts); @@ -650,7 +681,8 @@ void *s3l_video_thread(void) outPack[0].timestamp = ts.tv_sec * 1000000ull + ts.tv_nsec / 1000; unsigned int n = 0, sc, scanOff = 0; - while (n < 7 && (sc = nal_find_startcode(data, scanOff, size)) < size) { + while (desc[S3L_DESC_W_STREAM_TYPE] == S3L_STREAM_TYPE_H264 && + n < 7 && (sc = nal_find_startcode(data, scanOff, size)) < size) { unsigned int scLen = (data[sc + 2] == 1) ? 3 : 4; if (sc + scLen >= size) break; outPack[0].nalu[n].type = data[sc + scLen] & 0x1F; @@ -663,7 +695,7 @@ void *s3l_video_thread(void) for (unsigned int i = 0; i < n; i++) outPack[0].nalu[i].length = outPack[0].nalu[i + 1].offset - outPack[0].nalu[i].offset; - (*s3l_vid_cb)(0, &outStrm); + (*s3l_vid_cb)(id, &outStrm); } free(wrapBuf); diff --git a/src/hal/amba/s3l_hal.h b/src/hal/amba/s3l_hal.h index 5d0656fa..ba539408 100644 --- a/src/hal/amba/s3l_hal.h +++ b/src/hal/amba/s3l_hal.h @@ -16,7 +16,7 @@ #include #include -#define S3L_VENC_CHN_NUM 1 +#define S3L_VENC_CHN_NUM 2 extern hal_chnstate s3l_state[S3L_VENC_CHN_NUM]; extern int (*s3l_aud_cb)(hal_audframe*); @@ -29,6 +29,7 @@ void s3l_channel_grayscale(char enable); int s3l_channel_bind(char index); int s3l_channel_unbind(char index); +void s3l_sub_size(short width, short height); int s3l_pipeline_create(short width, short height, char mirror, char flip, char framerate); void s3l_pipeline_destroy(void); diff --git a/src/hal/amba/s3l_iav.h b/src/hal/amba/s3l_iav.h index 4a1f9920..b856c9e2 100644 --- a/src/hal/amba/s3l_iav.h +++ b/src/hal/amba/s3l_iav.h @@ -17,6 +17,8 @@ #define S3L_IOC_GET_STREAM_FORMAT 0xc0045634u #define S3L_IOC_SET_H264_CFG 0x40045638u #define S3L_IOC_GET_H264_CFG 0xc0045638u +#define S3L_IOC_SET_MJPEG_CFG 0x40045639u +#define S3L_IOC_GET_MJPEG_CFG 0xc0045639u #define S3L_IOC_GET_STATISTICS 0x80047080u #define S3L_IOC_IMG_CONFIG_STATISTICS 0x40047001u @@ -41,6 +43,7 @@ #define S3L_BUFFER_BSB 1 #define S3L_DESC_FRAME 0 #define S3L_STREAM_TYPE_H264 1 +#define S3L_STREAM_TYPE_MJPEG 2 #define S3L_SRCBUF_TYPE_PREVIEW 2 @@ -73,6 +76,7 @@ struct s3l_vin_info { uint32_t vsrc_id, video_mode, width, height, fps, rsvd[4]; #define S3L_DESC_W_ID 2 #define S3L_DESC_W_TIMEOUT_MS 3 #define S3L_DESC_W_DATA_OFF 4 +#define S3L_DESC_W_STREAM_TYPE 5 #define S3L_DESC_W_PIC_TYPE 6 #define S3L_DESC_W_FRAME_NUM 7 #define S3L_DESC_W_SIZE 13 diff --git a/src/media.c b/src/media.c index fb305198..acc412dd 100644 --- a/src/media.c +++ b/src/media.c @@ -1008,6 +1008,8 @@ int sdk_start(void) { case HAL_PLATFORM_S3L: s3l_aaa_options(app_config.auto_exposure, app_config.exposure_target, app_config.auto_white_balance, app_config.antiflicker); + if (app_config.mjpeg_enable) + s3l_sub_size(app_config.mjpeg_width, app_config.mjpeg_height); /* The main buffer is the H.264/H.265 stream's: a larger MJPEG shares it at its size */ ret = s3l_pipeline_create(app_config.mp4_enable ? app_config.mp4_width : width, app_config.mp4_enable ? app_config.mp4_height : height, app_config.mirror, app_config.flip, @@ -1071,9 +1073,7 @@ int sdk_start(void) { if (app_config.mp4_enable && (ret = media_mp4_enable())) HAL_ERROR("media", "MP4 initialization failed with %#x!\n", ret); - if (app_config.mjpeg_enable && plat == HAL_PLATFORM_S3L) - HAL_WARNING("media", "MJPEG is not supported on this platform yet, skipping\n"); - else if (app_config.mjpeg_enable && (ret = media_mjpeg_enable())) + if (app_config.mjpeg_enable && (ret = media_mjpeg_enable())) HAL_ERROR("media", "MJPEG initialization failed with %#x!\n", ret); if (app_config.jpeg_enable && (ret = jpeg_init())) From 10888b9bf01189a924ebb63f2babbd89662399a1 Mon Sep 17 00:00:00 2001 From: Dylan Corrales Date: Fri, 6 Nov 2026 16:24:44 -0500 Subject: [PATCH 03/11] S3L: H.264 bitrate, CBR/VBR, GOP and profile from the config; forced IDR The mp4 bitrate, mode, GOP and profile were ignored: the encoder ran on the driver defaults (8 Mbit/s smart CBR, GOP 30, Main). Before encoding starts the H.264 config now takes the GOP length (an IDR every GOP) and the profile, and the bitrate stream config the average bitrate in bit/s with CBR or VBR, read-modify-write so the driver's QP limits stay. Other modes keep the driver's smart CBR, with a warning. request_idr() now forces an IDR through the stream config, so a new RTSP client no longer waits for the next GOP. The S3L's IAV encodes Main or High only: Baseline is encoded as Main, with a warning. Tested on the IP4M-1026E: 4096 kbit/s CBR, GOP 40, High measures about 4.2 Mbit/s with keyframes exactly every 40 frames; 2048 kbit/s VBR measures 2.2 Mbit/s; a second RTSP client connecting mid-stream puts an IDR off the GOP grid in the first client's stream. Profiles 1 and 2 come out as Main and High; profile 0 comes out as Main. --- src/hal/amba/s3l_hal.c | 34 ++++++++++++++++++++++++++++++---- src/hal/amba/s3l_iav.h | 9 +++++++++ 2 files changed, 39 insertions(+), 4 deletions(-) diff --git a/src/hal/amba/s3l_hal.c b/src/hal/amba/s3l_hal.c index 0fa8cd43..8340710e 100644 --- a/src/hal/amba/s3l_hal.c +++ b/src/hal/amba/s3l_hal.c @@ -592,9 +592,30 @@ int s3l_video_create(char index, hal_vidconfig *config) buf[5] = quality; if (s3l_ioctl(S3L_IOC_SET_MJPEG_CFG, buf, "SET_MJPEG_CFG")) return EXIT_FAILURE; - } else if (s3l_ioctl(S3L_IOC_GET_H264_CFG, buf, "GET_H264_CFG") || - s3l_ioctl(S3L_IOC_SET_H264_CFG, buf, "SET_H264_CFG")) - return EXIT_FAILURE; + } else { + uint32_t *w = (uint32_t *)buf; + if (s3l_ioctl(S3L_IOC_GET_H264_CFG, buf, "GET_H264_CFG")) + return EXIT_FAILURE; + if (config->gop > 0) + w[S3L_H264_W_GOP] = (w[S3L_H264_W_GOP] & 0xffff) | (uint32_t)config->gop << 16; + /* IAV's 1 Main and 2 High: the S3L's has no Baseline (0 is refused) */ + if (config->profile == HAL_VIDPROFILE_BASELINE) + HAL_WARNING("s3l_venc", "No H.264 Baseline profile on S3L, encoding Main\n"); + w[S3L_H264_W_IDR_PROFILE] = (w[S3L_H264_W_IDR_PROFILE] & ~0xffffu) | 1 | + (uint32_t)(config->profile == HAL_VIDPROFILE_HIGH ? 2 : 1) << 8; + if (s3l_ioctl(S3L_IOC_SET_H264_CFG, buf, "SET_H264_CFG")) + return EXIT_FAILURE; + + uint32_t rc[32] = { (uint32_t)index, S3L_STMCFG_H264_BITRATE }; + if (config->bitrate > 0 && !s3l_ioctl(S3L_IOC_GET_STREAM_CFG, rc, "GET_STREAM_CFG(bitrate)")) { + rc[3] = config->bitrate * 1000; + if (config->mode == HAL_VIDMODE_CBR || config->mode == HAL_VIDMODE_VBR) + rc[4] = (rc[4] & ~0xffu) | (config->mode == HAL_VIDMODE_CBR ? S3L_BRC_CBR : S3L_BRC_VBR); + else + HAL_WARNING("s3l_venc", "Only CBR and VBR on S3L, keeping the encoder's own rate control\n"); + s3l_ioctl(S3L_IOC_SET_STREAM_CFG, rc, "SET_STREAM_CFG(bitrate)"); + } + } if (s3l_ioctl_nosig(S3L_IOC_START_ENCODE, (void *)(1u << index)) < 0) HAL_ERROR("s3l_venc", "Starting encoder %d failed: %s\n", index, strerror(errno)); @@ -623,7 +644,12 @@ int s3l_video_destroy_all(void) void s3l_video_request_idr(char index) { - (void)index; + uint32_t cfg[32] = { (uint32_t)index, S3L_STMCFG_H264_FORCE_IDR, 1 }; + + if (index < 0 || index >= S3L_VENC_CHN_NUM || !s3l_state[index].enable || + s3l_state[index].payload != HAL_VIDCODEC_H264) + return; + s3l_ioctl(S3L_IOC_SET_STREAM_CFG, cfg, "SET_STREAM_CFG(force IDR)"); } void *s3l_video_thread(void) diff --git a/src/hal/amba/s3l_iav.h b/src/hal/amba/s3l_iav.h index b856c9e2..8afeb5f7 100644 --- a/src/hal/amba/s3l_iav.h +++ b/src/hal/amba/s3l_iav.h @@ -17,6 +17,8 @@ #define S3L_IOC_GET_STREAM_FORMAT 0xc0045634u #define S3L_IOC_SET_H264_CFG 0x40045638u #define S3L_IOC_GET_H264_CFG 0xc0045638u +#define S3L_IOC_SET_STREAM_CFG 0x40045636u +#define S3L_IOC_GET_STREAM_CFG 0xc0045636u #define S3L_IOC_SET_MJPEG_CFG 0x40045639u #define S3L_IOC_GET_MJPEG_CFG 0xc0045639u #define S3L_IOC_GET_STATISTICS 0x80047080u @@ -45,6 +47,13 @@ #define S3L_STREAM_TYPE_H264 1 #define S3L_STREAM_TYPE_MJPEG 2 +#define S3L_STMCFG_H264_BITRATE 0x1001 +#define S3L_STMCFG_H264_FORCE_IDR 0x1002 +#define S3L_BRC_CBR 1 +#define S3L_BRC_VBR 3 + +#define S3L_H264_W_GOP 1 +#define S3L_H264_W_IDR_PROFILE 2 #define S3L_SRCBUF_TYPE_PREVIEW 2 #define S3L_VIN_FPS(fps) ((uint32_t)((512000000ull + (fps) / 2) / (fps))) From 1607be1635f800c0e37e08e769acceaa969d1ea8 Mon Sep 17 00:00:00 2001 From: Dylan Corrales Date: Fri, 6 Nov 2026 16:46:24 -0500 Subject: [PATCH 04/11] S3L: MJPEG above 720x720 from the third source buffer The second source buffer is limited to 720x720 (IAV: "cannot be larger than 720x720"), so an MJPEG stream of 1280x720 or more could not start. Sizes above that now use the third source buffer, which takes up to the sensor size. Source buffers that are not used are switched off rather than left as the previous run configured them; IAV keeps that setup, and a stale third buffer larger than the main one made SET_SRCBUF_SETUP fail. Tested on the IP4M-1026E with H.264 2688x1512: MJPEG 640x360 (second buffer), 1280x720 and 1920x1080 (third buffer) at 20 fps, and MJPEG-only at 1280x720 right after a 1920x1080 run. --- src/hal/amba/s3l_hal.c | 35 ++++++++++++++++++++++++++--------- src/hal/amba/s3l_iav.h | 2 ++ 2 files changed, 28 insertions(+), 9 deletions(-) diff --git a/src/hal/amba/s3l_hal.c b/src/hal/amba/s3l_hal.c index 8340710e..97633acd 100644 --- a/src/hal/amba/s3l_hal.c +++ b/src/hal/amba/s3l_hal.c @@ -451,11 +451,18 @@ int s3l_pipeline_create(short width, short height, char mirror, char flip, char HAL_INFO("s3l_iav", "VIN %ux%u, main buffer %ux%u\n", info.width, info.height, _s3l_main_w, _s3l_main_h); - /* A sub-buffer must be smaller than the main one */ - _s3l_mjpeg_buf = _s3l_sub_w < _s3l_main_w && _s3l_sub_h < _s3l_main_h; + /* A sub-buffer must be smaller than the main one, and the second one at most 720x720 */ + if (_s3l_sub_w >= _s3l_main_w || _s3l_sub_h >= _s3l_main_h) + _s3l_mjpeg_buf = 0; + else if (_s3l_sub_w > S3L_SRCBUF1_MAX || _s3l_sub_h > S3L_SRCBUF1_MAX) + _s3l_mjpeg_buf = 2; + else + _s3l_mjpeg_buf = 1; + unsigned short b1w = _s3l_mjpeg_buf == 1 ? _s3l_sub_w : MIN(640, (_s3l_main_w / 2) & ~15); + unsigned short b1h = _s3l_mjpeg_buf == 1 ? _s3l_sub_h : MIN(480, (_s3l_main_h / 2) & ~7); if (!_s3l_mjpeg_buf) { - _s3l_sub_w = (_s3l_main_w / 2) & ~15; - _s3l_sub_h = (_s3l_main_h / 2) & ~7; + _s3l_sub_w = _s3l_main_w; + _s3l_sub_h = _s3l_main_h; } memset(buf, 0, sizeof(buf)); @@ -466,11 +473,14 @@ int s3l_pipeline_create(short width, short height, char mirror, char flip, char v[0] = _s3l_main_w; v[1] = _s3l_main_h; memcpy(buf + S3L_SYSRES_BUF_MAX(0), v, 8); memcpy(buf + S3L_SYSRES_STREAM_MAX(0), v, 8); - v[0] = _s3l_sub_w; v[1] = _s3l_sub_h; - memcpy(buf + S3L_SYSRES_BUF_MAX(1), v, 8); - memcpy(buf + S3L_SYSRES_STREAM_MAX(1), v, 8); memset(buf + S3L_SYSRES_BUF_MAX(2), 0, 24); memset(buf + S3L_SYSRES_STREAM_MAX(2), 0, 16); + v[0] = b1w; v[1] = b1h; + memcpy(buf + S3L_SYSRES_BUF_MAX(1), v, 8); + v[0] = _s3l_sub_w; v[1] = _s3l_sub_h; + if (_s3l_mjpeg_buf == 2) + memcpy(buf + S3L_SYSRES_BUF_MAX(2), v, 8); + memcpy(buf + S3L_SYSRES_STREAM_MAX(1), v, 8); v[0] = 1; memcpy(buf + S3L_SYSRES_C0, v, 4); v[0] = 0; memcpy(buf + S3L_SYSRES_DC, v, 4); memcpy(buf + S3L_SYSRES_EXTRA_A4, v, 4); @@ -497,8 +507,15 @@ int s3l_pipeline_create(short width, short height, char mirror, char flip, char memcpy(buf + S3L_SRCBUF_SUB_INPUT(k), v, 8); v[0] = info.width; v[1] = info.height; memcpy(buf + S3L_SRCBUF_MAIN_WIN_B, v, 8); - v[0] = _s3l_sub_w; v[1] = _s3l_sub_h; + v[0] = b1w; v[1] = b1h; memcpy(buf + S3L_SRCBUF_SIZE(1), v, 8); + for (int i = 2; i < 4; i++) { + char used = i == 2 && _s3l_mjpeg_buf == 2; + v[0] = used ? _s3l_sub_w : 0; v[1] = used ? _s3l_sub_h : 0; + memcpy(buf + S3L_SRCBUF_SIZE(i), v, 8); + v[0] = used ? S3L_SRCBUF_TYPE_ENCODE : 0; + memcpy(buf + S3L_SRCBUF_TYPE(i), v, 4); + } if (s3l_ioctl(S3L_IOC_SET_SRCBUF_SETUP, buf, "SET_SRCBUF_SETUP")) return EXIT_FAILURE; @@ -572,7 +589,7 @@ int s3l_video_create(char index, hal_vidconfig *config) return EXIT_FAILURE; v[0] = mjpeg ? S3L_STREAM_TYPE_MJPEG : S3L_STREAM_TYPE_H264; memcpy(buf + S3L_STRFMT_TYPE, v, 4); - char sub = mjpeg && _s3l_mjpeg_buf; + char sub = mjpeg ? _s3l_mjpeg_buf : 0; v[0] = sub; memcpy(buf + S3L_STRFMT_BUF_ID, v, 4); v[0] = sub ? _s3l_sub_w : _s3l_main_w; diff --git a/src/hal/amba/s3l_iav.h b/src/hal/amba/s3l_iav.h index 8afeb5f7..3397770b 100644 --- a/src/hal/amba/s3l_iav.h +++ b/src/hal/amba/s3l_iav.h @@ -54,7 +54,9 @@ #define S3L_H264_W_GOP 1 #define S3L_H264_W_IDR_PROFILE 2 +#define S3L_SRCBUF_TYPE_ENCODE 1 #define S3L_SRCBUF_TYPE_PREVIEW 2 +#define S3L_SRCBUF1_MAX 720 #define S3L_VIN_FPS(fps) ((uint32_t)((512000000ull + (fps) / 2) / (fps))) From 089c4550fc777ce5325a0143575a5c9ae044f117 Mon Sep 17 00:00:00 2001 From: Dylan Corrales Date: Fri, 6 Nov 2026 16:50:33 -0500 Subject: [PATCH 05/11] S3L: JPEG snapshots /image.jpg (and the ONVIF snapshot URI) did not work on S3L. Snapshots now follow T31: with MJPEG on they reuse the MJPEG stream at its resolution; otherwise jpeg_init() opens its own encoder stream, an MJPEG-type IAV stream at jpeg width/height and qfactor running at about 1 fps. A snapshot waits for that stream's next frame, which the frame thread copies only while a snapshot is pending; frames of the snapshot stream are not passed on as video. Every stream now also sets its frame-rate divider explicitly, since IAV keeps per-stream settings across runs and a stream left at the snapshot rate would otherwise stay at 1 fps. Tested on the IP4M-1026E with H.264 2688x1512 alongside: own snapshot stream at 1280x720 / q85 and 1920x1080 (5 of 5 snapshots, about 0.8 s each); with MJPEG 640x360 on, snapshots from it (0.1 s each) and MJPEG at 20 fps; MJPEG back at 20 fps after a snapshot-stream run. --- README.md | 2 +- src/hal/amba/s3l_hal.c | 74 +++++++++++++++++++++++++++++++++++++++--- src/hal/amba/s3l_hal.h | 1 + src/hal/amba/s3l_iav.h | 1 + src/jpeg.c | 17 ++++++++++ src/media.c | 2 ++ 6 files changed, 91 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index cd3a776c..e78ce643 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ In spite of these design choices, Divinus boasts numerous features that cater to | infinity6c[^18] | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | | mercury6[^19] | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | | sun8iw21p1[^20] | ↻ | ↻ | ↻ | ↻ | ↻ | -| S3L[^21] | ↻ | ↻ | ↻ | ✔️ | ↻ | +| S3L[^21] | ↻ | ✔️ | ✔️ | ✔️ | ↻ | _✔️ - supported, ↻ - in development, ✗ - unsupported, ⁿ/ₐ - not supported by hardware_ diff --git a/src/hal/amba/s3l_hal.c b/src/hal/amba/s3l_hal.c index 97633acd..99f774b3 100644 --- a/src/hal/amba/s3l_hal.c +++ b/src/hal/amba/s3l_hal.c @@ -17,6 +17,7 @@ static int _s3l_fd = -1; static unsigned char *_s3l_bsb; static unsigned int _s3l_bsb_len; static unsigned short _s3l_main_w, _s3l_main_h; +static char _s3l_fps = 30; static unsigned short _s3l_sub_w = 640, _s3l_sub_h = 480; static char _s3l_mjpeg_buf = 1; @@ -544,7 +545,8 @@ int s3l_pipeline_create(short width, short height, char mirror, char flip, char s3l_aaa_config(_s3l_fd); } _s3l_previewing = 1; - s3l_ae_start(_s3l_fd, framerate > 0 ? framerate : 30); + _s3l_fps = framerate > 0 ? framerate : 30; + s3l_ae_start(_s3l_fd, _s3l_fps); { uint32_t qb[16] = { S3L_BUFFER_BSB }; @@ -579,9 +581,10 @@ int s3l_video_create(char index, hal_vidconfig *config) if (index < 0 || index >= S3L_VENC_CHN_NUM) HAL_ERROR("s3l_venc", "Only %d encoder channel(s) are supported!\n", S3L_VENC_CHN_NUM); - if (config->codec != HAL_VIDCODEC_H264 && config->codec != HAL_VIDCODEC_MJPG) - HAL_ERROR("s3l_venc", "Only H.264 and MJPEG are supported so far!\n"); - char mjpeg = config->codec == HAL_VIDCODEC_MJPG; + if (config->codec != HAL_VIDCODEC_H264 && config->codec != HAL_VIDCODEC_MJPG && + config->codec != HAL_VIDCODEC_JPG) + HAL_ERROR("s3l_venc", "Only H.264, MJPEG and JPEG are supported so far!\n"); + char mjpeg = config->codec != HAL_VIDCODEC_H264; memset(buf, 0, sizeof(buf)); buf[S3L_STRFMT_ID] = index; @@ -637,6 +640,13 @@ int s3l_video_create(char index, hal_vidconfig *config) if (s3l_ioctl_nosig(S3L_IOC_START_ENCODE, (void *)(1u << index)) < 0) HAL_ERROR("s3l_venc", "Starting encoder %d failed: %s\n", index, strerror(errno)); + uint32_t fps[32] = { (uint32_t)index, S3L_STMCFG_FPS }; + if (!s3l_ioctl(S3L_IOC_GET_STREAM_CFG, fps, "GET_STREAM_CFG(fps)")) { + fps[2] = 1; + fps[3] = config->codec == HAL_VIDCODEC_JPG ? _s3l_fps : 1; + s3l_ioctl(S3L_IOC_SET_STREAM_CFG, fps, "SET_STREAM_CFG(fps)"); + } + s3l_state[index].payload = config->codec; s3l_state[index].enable = 1; return EXIT_SUCCESS; @@ -659,6 +669,43 @@ int s3l_video_destroy_all(void) return EXIT_SUCCESS; } +static pthread_mutex_t _s3l_snap_mtx = PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t _s3l_snap_cond = PTHREAD_COND_INITIALIZER; +/* One snapshot at a time: jpeg_get() serialises callers with jpeg_mutex */ +static hal_jpegdata *_s3l_snap; +static int _s3l_snap_id, _s3l_snap_ret; + +int s3l_video_snapshot_grab(hal_jpegdata *jpeg) +{ + struct timespec until; + int id = -1, ret; + + for (int i = 0; i < S3L_VENC_CHN_NUM; i++) { + if (!s3l_state[i].enable) continue; + if (s3l_state[i].payload == HAL_VIDCODEC_JPG) { id = i; break; } + if (s3l_state[i].payload == HAL_VIDCODEC_MJPG) id = i; + } + if (id < 0) + HAL_ERROR("s3l_venc", "No MJPEG or JPEG stream to take a snapshot from!\n"); + if (s3l_state[id].payload == HAL_VIDCODEC_MJPG) { + static char told; + if (!told++) + HAL_INFO("s3l_venc", "Snapshots come from the MJPEG stream, at its size and quality\n"); + } + + clock_gettime(CLOCK_REALTIME, &until); + until.tv_sec += 2; + pthread_mutex_lock(&_s3l_snap_mtx); + _s3l_snap = jpeg; + _s3l_snap_id = id; + _s3l_snap_ret = EXIT_FAILURE; + while (_s3l_snap && !pthread_cond_timedwait(&_s3l_snap_cond, &_s3l_snap_mtx, &until)); + _s3l_snap = NULL; + ret = _s3l_snap_ret; + pthread_mutex_unlock(&_s3l_snap_mtx); + return ret; +} + void s3l_video_request_idr(char index) { uint32_t cfg[32] = { (uint32_t)index, S3L_STMCFG_H264_FORCE_IDR, 1 }; @@ -738,7 +785,24 @@ void *s3l_video_thread(void) for (unsigned int i = 0; i < n; i++) outPack[0].nalu[i].length = outPack[0].nalu[i + 1].offset - outPack[0].nalu[i].offset; - (*s3l_vid_cb)(id, &outStrm); + pthread_mutex_lock(&_s3l_snap_mtx); + if (_s3l_snap && _s3l_snap_id == (int)id) { + if (_s3l_snap->length < size) { + unsigned char *grown = realloc(_s3l_snap->data, size); + if (grown) { _s3l_snap->data = grown; _s3l_snap->length = size; } + } + if (_s3l_snap->length >= size) { + memcpy(_s3l_snap->data, data, size); + _s3l_snap->jpegSize = size; + _s3l_snap_ret = EXIT_SUCCESS; + } + _s3l_snap = NULL; + pthread_cond_signal(&_s3l_snap_cond); + } + pthread_mutex_unlock(&_s3l_snap_mtx); + + if (s3l_state[id].payload != HAL_VIDCODEC_JPG) + (*s3l_vid_cb)(id, &outStrm); } free(wrapBuf); diff --git a/src/hal/amba/s3l_hal.h b/src/hal/amba/s3l_hal.h index ba539408..01beaa05 100644 --- a/src/hal/amba/s3l_hal.h +++ b/src/hal/amba/s3l_hal.h @@ -34,6 +34,7 @@ int s3l_pipeline_create(short width, short height, char mirror, char flip, char void s3l_pipeline_destroy(void); int s3l_video_create(char index, hal_vidconfig *config); +int s3l_video_snapshot_grab(hal_jpegdata *jpeg); int s3l_video_destroy(char index); int s3l_video_destroy_all(void); void s3l_video_request_idr(char index); diff --git a/src/hal/amba/s3l_iav.h b/src/hal/amba/s3l_iav.h index 3397770b..5d772a5c 100644 --- a/src/hal/amba/s3l_iav.h +++ b/src/hal/amba/s3l_iav.h @@ -47,6 +47,7 @@ #define S3L_STREAM_TYPE_H264 1 #define S3L_STREAM_TYPE_MJPEG 2 +#define S3L_STMCFG_FPS 0x0000 #define S3L_STMCFG_H264_BITRATE 0x1001 #define S3L_STMCFG_H264_FORCE_IDR 0x1002 #define S3L_BRC_CBR 1 diff --git a/src/jpeg.c b/src/jpeg.c index b848fea8..06104b3a 100644 --- a/src/jpeg.c +++ b/src/jpeg.c @@ -11,6 +11,11 @@ int jpeg_init() { pthread_mutex_lock(&jpeg_mutex); switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: + if (app_config.mjpeg_enable) goto active; + break; +#endif #if defined(__arm__) && !defined(__ARM_PCS_VFP) case HAL_PLATFORM_AK: goto active; case HAL_PLATFORM_GM: goto active; @@ -38,6 +43,9 @@ int jpeg_init() { config.minQual = config.maxQual = app_config.jpeg_qfactor; switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: ret = s3l_video_create(jpeg_index, &config); break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: ret = i6_video_create(jpeg_index, &config); break; case HAL_PLATFORM_I6C: ret = i6c_video_create(jpeg_index, &config); break; @@ -81,6 +89,12 @@ void jpeg_deinit() { pthread_mutex_lock(&jpeg_mutex); switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: + if (app_config.mjpeg_enable) goto active; + s3l_video_destroy(jpeg_index); + break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: i6_video_destroy(jpeg_index); break; case HAL_PLATFORM_I6C: i6c_video_destroy(jpeg_index); break; @@ -134,6 +148,9 @@ int jpeg_get(short width, short height, char quality, char grayscale, int ret; switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: ret = s3l_video_snapshot_grab(jpeg); break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: ret = i6_video_snapshot_grab(jpeg_index, quality, jpeg); break; case HAL_PLATFORM_I6C: ret = i6c_video_snapshot_grab(jpeg_index, quality, jpeg); break; diff --git a/src/media.c b/src/media.c index acc412dd..1b786797 100644 --- a/src/media.c +++ b/src/media.c @@ -1010,6 +1010,8 @@ int sdk_start(void) { app_config.auto_white_balance, app_config.antiflicker); if (app_config.mjpeg_enable) s3l_sub_size(app_config.mjpeg_width, app_config.mjpeg_height); + else if (app_config.jpeg_enable) + s3l_sub_size(app_config.jpeg_width, app_config.jpeg_height); /* The main buffer is the H.264/H.265 stream's: a larger MJPEG shares it at its size */ ret = s3l_pipeline_create(app_config.mp4_enable ? app_config.mp4_width : width, app_config.mp4_enable ? app_config.mp4_height : height, app_config.mirror, app_config.flip, From d0b11510a87f4c271118cb3492d6c300b56a2dbc Mon Sep 17 00:00:00 2001 From: Dylan Corrales Date: Fri, 6 Nov 2026 17:09:56 -0500 Subject: [PATCH 06/11] S3L: H.265 mp4.codec H265 was refused on S3L. The encoder supports it: it now runs as an IAV H.265 stream, configured like H.264 through the H.265 config ioctl (GOP length, an IDR every GOP; the driver's Main profile) and the H.265 stream configs (bitrate with CBR/VBR, forced IDR), which mirror the H.264 ones at +0x2000. IAV hands an H.265 frame over one slice at a time (three slices at 2688x1512, each with its own descriptor carrying the slice index and count). The frame thread collects a frame's slices, starting only at slice 0, and passes the whole frame on with one timestamp and its H.265 NAL types. Tested on the IP4M-1026E: 2688x1512 H.265 Main over RTSP at 20 fps, 4096 kbit/s CBR measuring about 4.1 Mbit/s, keyframes exactly every 40 frames, a forced IDR when a second client connects, recordings that decode without errors, /video.mp4, alongside MJPEG, and restarts. H.264 unchanged. --- src/hal/amba/s3l_hal.c | 66 +++++++++++++++++++++++++++++------------- src/hal/amba/s3l_iav.h | 6 ++++ 2 files changed, 52 insertions(+), 20 deletions(-) diff --git a/src/hal/amba/s3l_hal.c b/src/hal/amba/s3l_hal.c index 99f774b3..753c0c24 100644 --- a/src/hal/amba/s3l_hal.c +++ b/src/hal/amba/s3l_hal.c @@ -581,16 +581,16 @@ int s3l_video_create(char index, hal_vidconfig *config) if (index < 0 || index >= S3L_VENC_CHN_NUM) HAL_ERROR("s3l_venc", "Only %d encoder channel(s) are supported!\n", S3L_VENC_CHN_NUM); - if (config->codec != HAL_VIDCODEC_H264 && config->codec != HAL_VIDCODEC_MJPG && - config->codec != HAL_VIDCODEC_JPG) - HAL_ERROR("s3l_venc", "Only H.264, MJPEG and JPEG are supported so far!\n"); - char mjpeg = config->codec != HAL_VIDCODEC_H264; + char h265 = config->codec == HAL_VIDCODEC_H265; + char mjpeg = config->codec == HAL_VIDCODEC_MJPG || config->codec == HAL_VIDCODEC_JPG; + if (config->codec != HAL_VIDCODEC_H264 && !h265 && !mjpeg) + HAL_ERROR("s3l_venc", "Only H.264, H.265, MJPEG and JPEG are supported!\n"); memset(buf, 0, sizeof(buf)); buf[S3L_STRFMT_ID] = index; if (s3l_ioctl(S3L_IOC_GET_STREAM_FORMAT, buf, "GET_STREAM_FORMAT")) return EXIT_FAILURE; - v[0] = mjpeg ? S3L_STREAM_TYPE_MJPEG : S3L_STREAM_TYPE_H264; + v[0] = mjpeg ? S3L_STREAM_TYPE_MJPEG : h265 ? S3L_STREAM_TYPE_H265 : S3L_STREAM_TYPE_H264; memcpy(buf + S3L_STRFMT_TYPE, v, 4); char sub = mjpeg ? _s3l_mjpeg_buf : 0; v[0] = sub; @@ -614,19 +614,22 @@ int s3l_video_create(char index, hal_vidconfig *config) return EXIT_FAILURE; } else { uint32_t *w = (uint32_t *)buf; - if (s3l_ioctl(S3L_IOC_GET_H264_CFG, buf, "GET_H264_CFG")) + if (s3l_ioctl(h265 ? S3L_IOC_GET_H265_CFG : S3L_IOC_GET_H264_CFG, buf, "GET_H26X_CFG")) return EXIT_FAILURE; if (config->gop > 0) w[S3L_H264_W_GOP] = (w[S3L_H264_W_GOP] & 0xffff) | (uint32_t)config->gop << 16; - /* IAV's 1 Main and 2 High: the S3L's has no Baseline (0 is refused) */ - if (config->profile == HAL_VIDPROFILE_BASELINE) - HAL_WARNING("s3l_venc", "No H.264 Baseline profile on S3L, encoding Main\n"); - w[S3L_H264_W_IDR_PROFILE] = (w[S3L_H264_W_IDR_PROFILE] & ~0xffffu) | 1 | - (uint32_t)(config->profile == HAL_VIDPROFILE_HIGH ? 2 : 1) << 8; - if (s3l_ioctl(S3L_IOC_SET_H264_CFG, buf, "SET_H264_CFG")) + w[S3L_H264_W_IDR_PROFILE] = (w[S3L_H264_W_IDR_PROFILE] & ~0xffu) | 1; + if (!h265) { + /* IAV's 1 Main and 2 High: the S3L's has no Baseline (0 is refused) */ + if (config->profile == HAL_VIDPROFILE_BASELINE) + HAL_WARNING("s3l_venc", "No H.264 Baseline profile on S3L, encoding Main\n"); + w[S3L_H264_W_IDR_PROFILE] = (w[S3L_H264_W_IDR_PROFILE] & ~0xff00u) | + (uint32_t)(config->profile == HAL_VIDPROFILE_HIGH ? 2 : 1) << 8; + } + if (s3l_ioctl(h265 ? S3L_IOC_SET_H265_CFG : S3L_IOC_SET_H264_CFG, buf, "SET_H26X_CFG")) return EXIT_FAILURE; - uint32_t rc[32] = { (uint32_t)index, S3L_STMCFG_H264_BITRATE }; + uint32_t rc[32] = { (uint32_t)index, h265 ? S3L_STMCFG_H265_BITRATE : S3L_STMCFG_H264_BITRATE }; if (config->bitrate > 0 && !s3l_ioctl(S3L_IOC_GET_STREAM_CFG, rc, "GET_STREAM_CFG(bitrate)")) { rc[3] = config->bitrate * 1000; if (config->mode == HAL_VIDMODE_CBR || config->mode == HAL_VIDMODE_VBR) @@ -708,19 +711,21 @@ int s3l_video_snapshot_grab(hal_jpegdata *jpeg) void s3l_video_request_idr(char index) { - uint32_t cfg[32] = { (uint32_t)index, S3L_STMCFG_H264_FORCE_IDR, 1 }; - if (index < 0 || index >= S3L_VENC_CHN_NUM || !s3l_state[index].enable || - s3l_state[index].payload != HAL_VIDCODEC_H264) + (s3l_state[index].payload != HAL_VIDCODEC_H264 && s3l_state[index].payload != HAL_VIDCODEC_H265)) return; + uint32_t cfg[32] = { (uint32_t)index, s3l_state[index].payload == HAL_VIDCODEC_H265 ? + S3L_STMCFG_H265_FORCE_IDR : S3L_STMCFG_H264_FORCE_IDR, 1 }; s3l_ioctl(S3L_IOC_SET_STREAM_CFG, cfg, "SET_STREAM_CFG(force IDR)"); } void *s3l_video_thread(void) { static uint32_t desc[S3L_SZ_QUERYDESC / 4 + 64]; - unsigned char *wrapBuf = NULL; - unsigned int wrapLen = 0; + unsigned char *wrapBuf = NULL, *frameBuf = NULL; + unsigned int wrapLen = 0, frameLen = 0, frameCap = 0; + int frameNext = -1; + unsigned int frameId = 0; while (keepRunning) { if (!_s3l_bsb || !(s3l_state[0].enable || s3l_state[1].enable)) { usleep(100000); continue; } @@ -756,6 +761,25 @@ void *s3l_video_thread(void) unsigned int id = desc[S3L_DESC_W_ID]; if (!s3l_vid_cb || id >= S3L_VENC_CHN_NUM || !s3l_state[id].enable) continue; + if (desc[S3L_DESC_W_STREAM_TYPE] == S3L_STREAM_TYPE_H265) { + unsigned int slice = (desc[S3L_DESC_W_SLICE] >> 16) & 3, slices = (desc[S3L_DESC_W_SLICE] >> 18) & 3; + if (!slice) frameNext = 0, frameLen = 0, frameId = id; + if (slice != frameNext || id != frameId) { frameNext = -1; continue; } + if (frameLen + size > frameCap) { + unsigned char *grown = realloc(frameBuf, frameLen + size); + if (!grown) { frameNext = -1; continue; } + frameBuf = grown; + frameCap = frameLen + size; + } + memcpy(frameBuf + frameLen, data, size); + frameLen += size; + frameNext++; + if (frameNext < (int)slices) continue; + frameNext = -1; + data = frameBuf; + size = frameLen; + } + struct timespec ts; clock_gettime(CLOCK_MONOTONIC, &ts); @@ -771,11 +795,12 @@ void *s3l_video_thread(void) outPack[0].timestamp = ts.tv_sec * 1000000ull + ts.tv_nsec / 1000; unsigned int n = 0, sc, scanOff = 0; - while (desc[S3L_DESC_W_STREAM_TYPE] == S3L_STREAM_TYPE_H264 && + char hevc = desc[S3L_DESC_W_STREAM_TYPE] == S3L_STREAM_TYPE_H265; + while ((desc[S3L_DESC_W_STREAM_TYPE] == S3L_STREAM_TYPE_H264 || hevc) && n < 7 && (sc = nal_find_startcode(data, scanOff, size)) < size) { unsigned int scLen = (data[sc + 2] == 1) ? 3 : 4; if (sc + scLen >= size) break; - outPack[0].nalu[n].type = data[sc + scLen] & 0x1F; + outPack[0].nalu[n].type = hevc ? (data[sc + scLen] >> 1) & 0x3F : data[sc + scLen] & 0x1F; outPack[0].nalu[n].offset = sc; n++; scanOff = sc + scLen; @@ -806,6 +831,7 @@ void *s3l_video_thread(void) } free(wrapBuf); + free(frameBuf); HAL_INFO("s3l_venc", "Shutting down encoding thread...\n"); return NULL; } diff --git a/src/hal/amba/s3l_iav.h b/src/hal/amba/s3l_iav.h index 5d772a5c..d5c05b90 100644 --- a/src/hal/amba/s3l_iav.h +++ b/src/hal/amba/s3l_iav.h @@ -20,6 +20,8 @@ #define S3L_IOC_SET_STREAM_CFG 0x40045636u #define S3L_IOC_GET_STREAM_CFG 0xc0045636u #define S3L_IOC_SET_MJPEG_CFG 0x40045639u +#define S3L_IOC_SET_H265_CFG 0x4004563cu +#define S3L_IOC_GET_H265_CFG 0xc004563cu #define S3L_IOC_GET_MJPEG_CFG 0xc0045639u #define S3L_IOC_GET_STATISTICS 0x80047080u #define S3L_IOC_IMG_CONFIG_STATISTICS 0x40047001u @@ -46,10 +48,13 @@ #define S3L_DESC_FRAME 0 #define S3L_STREAM_TYPE_H264 1 #define S3L_STREAM_TYPE_MJPEG 2 +#define S3L_STREAM_TYPE_H265 3 #define S3L_STMCFG_FPS 0x0000 #define S3L_STMCFG_H264_BITRATE 0x1001 #define S3L_STMCFG_H264_FORCE_IDR 0x1002 +#define S3L_STMCFG_H265_BITRATE 0x3001 +#define S3L_STMCFG_H265_FORCE_IDR 0x3002 #define S3L_BRC_CBR 1 #define S3L_BRC_VBR 3 @@ -89,6 +94,7 @@ struct s3l_vin_info { uint32_t vsrc_id, video_mode, width, height, fps, rsvd[4]; #define S3L_DESC_W_TIMEOUT_MS 3 #define S3L_DESC_W_DATA_OFF 4 #define S3L_DESC_W_STREAM_TYPE 5 +#define S3L_DESC_W_SLICE 16 #define S3L_DESC_W_PIC_TYPE 6 #define S3L_DESC_W_FRAME_NUM 7 #define S3L_DESC_W_SIZE 13 From d686d3fabc282d10a5053bd6fb4a8bec54e2af34 Mon Sep 17 00:00:00 2001 From: Dylan Corrales Date: Fri, 6 Nov 2026 17:17:41 -0500 Subject: [PATCH 07/11] S3L: frame timestamps from the encoder's PTS Frames were stamped with the time the frame thread read their descriptor. After a large keyframe the next frames wait in IAV's queue while the keyframe is sent, so they were stamped late and then in a burst (steps of 9963, 2033, 1466 instead of 4500 at 20 fps); recordings logged "non monotonically increasing dts" around every keyframe and probing guessed 60 fps. Frames now carry the 90 kHz PTS from their descriptor, which the RTP layer expects ("the encoder's capture time"); the read time stays as the fallback when a descriptor has none. Tested on the IP4M-1026E: H.264 and H.265 RTSP recordings step exactly 4500 per frame at 20 fps with no ffmpeg warnings (5-8 per 10 s before), and /video.mp4 reports 20 fps; MJPEG and snapshots unchanged. --- src/hal/amba/s3l_hal.c | 4 +++- src/hal/amba/s3l_iav.h | 1 + 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/src/hal/amba/s3l_hal.c b/src/hal/amba/s3l_hal.c index 753c0c24..f51f496c 100644 --- a/src/hal/amba/s3l_hal.c +++ b/src/hal/amba/s3l_hal.c @@ -780,6 +780,7 @@ void *s3l_video_thread(void) size = frameLen; } + unsigned long long pts = (unsigned long long)desc[S3L_DESC_W_PTS + 1] << 32 | desc[S3L_DESC_W_PTS]; struct timespec ts; clock_gettime(CLOCK_MONOTONIC, &ts); @@ -792,7 +793,8 @@ void *s3l_video_thread(void) outPack[0].data = data; outPack[0].length = size; outPack[0].offset = 0; - outPack[0].timestamp = ts.tv_sec * 1000000ull + ts.tv_nsec / 1000; + /* The encoder's PTS counts 90 kHz ticks; Divinus wants microseconds */ + outPack[0].timestamp = pts ? pts * 100 / 9 : ts.tv_sec * 1000000ull + ts.tv_nsec / 1000; unsigned int n = 0, sc, scanOff = 0; char hevc = desc[S3L_DESC_W_STREAM_TYPE] == S3L_STREAM_TYPE_H265; diff --git a/src/hal/amba/s3l_iav.h b/src/hal/amba/s3l_iav.h index d5c05b90..63d3afd0 100644 --- a/src/hal/amba/s3l_iav.h +++ b/src/hal/amba/s3l_iav.h @@ -97,6 +97,7 @@ struct s3l_vin_info { uint32_t vsrc_id, video_mode, width, height, fps, rsvd[4]; #define S3L_DESC_W_SLICE 16 #define S3L_DESC_W_PIC_TYPE 6 #define S3L_DESC_W_FRAME_NUM 7 +#define S3L_DESC_W_PTS 8 #define S3L_DESC_W_SIZE 13 #define S3L_BUFFER_IMG 8 From 00f86c138cfb9fc6b15929799426744e51ae901d Mon Sep 17 00:00:00 2001 From: Dylan Corrales Date: Fri, 6 Nov 2026 21:56:13 -0500 Subject: [PATCH 08/11] S3L: on-screen display Regions were not drawn on S3L. They now use IAV's encode overlay: up to 10 areas per stream (one per OSD region), 8-bit indexed pixels with a 256-entry CLUT of V, U, Y and alpha, in the overlay buffer (QUERY_BUF 4). The S3L driver's overlay insert is 248 bytes, with 10 areas against the S2L SDK's 4. Each ARGB1555 bitmap becomes indexed colour: index 0 is transparent, opaque colours get BT.601 limited-range entries at the region's opacity, and an image with more than 255 colours falls back to 3-3-2. Every region has two pages, so an update writes the idle one and the DSP never reads a half-written bitmap. Areas follow the driver's rules (even x and width, y and height multiples of 4) and are clipped to the stream. The regions are drawn on the H.264/H.265 stream and put back whenever it starts. IAV keeps overlays across runs, and an area that does not fit a stream stops it from starting, so every stream's overlay is cleared when preview starts. Tested on the IP4M-1026E at 2688x1512: a timestamp updating every second, text with an outline at half opacity, a PNG with transparent parts and over 255 colours, regions clipped at the right and bottom edges, an off-frame one skipped, odd positions snapped, a region removed, and the overlay kept across an encoder restart from /api/mp4. Builds for all 12 targets in build.yml. --- README.md | 4 +- src/hal/amba/s3l_hal.c | 184 +++++++++++++++++++++++++++++++++++++++++ src/hal/amba/s3l_hal.h | 4 + src/hal/amba/s3l_iav.h | 10 +++ src/region.c | 15 ++++ 5 files changed, 215 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index e78ce643..7ddc6bc2 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ In spite of these design choices, Divinus boasts numerous features that cater to | infinity6c[^18] | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | | mercury6[^19] | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | | sun8iw21p1[^20] | ↻ | ↻ | ↻ | ↻ | ↻ | -| S3L[^21] | ↻ | ✔️ | ✔️ | ✔️ | ↻ | +| S3L[^21] | ↻ | ✔️ | ✔️ | ✔️ | ✔️ | _✔️ - supported, ↻ - in development, ✗ - unsupported, ⁿ/ₐ - not supported by hardware_ @@ -74,7 +74,7 @@ _* At the moment, text, RGB or bitfield bitmaps and PNG overlays are handled, mo [^18]: SSC377\(D/DE/QE\) and SSC378\[DE/QE\] [^19]: SSC359G [^20]: V851S\(E\) and V853 -[^21]: Ambarella S3L (S3LM), via the IAV driver; the ISP configuration and 3A (AE, AWB) run in Divinus +[^21]: Ambarella S3L (S3LM), via the IAV driver; the ISP configuration and 3A (AE, AWB) run in Divinus; the on-screen display is drawn on the H.264/H.265 stream ### Documentation diff --git a/src/hal/amba/s3l_hal.c b/src/hal/amba/s3l_hal.c index f51f496c..e979f434 100644 --- a/src/hal/amba/s3l_hal.c +++ b/src/hal/amba/s3l_hal.c @@ -29,6 +29,17 @@ static int _s3l_nl_sock = -1; static pthread_t _s3l_nl_tid, _s3l_stats_tid; static volatile char _s3l_aaa_run, _s3l_previewing; +static unsigned char *_s3l_ovl; +static unsigned int _s3l_ovl_len, _s3l_ovl_slot; +static struct { + hal_rect rect; + unsigned char opacity, page, set; + struct s3l_ovl_area area; +} _s3l_rgn[S3L_OVL_AREAS]; +static pthread_mutex_t _s3l_rgn_mtx = PTHREAD_MUTEX_INITIALIZER; + +static void s3l_region_apply(void); + static unsigned int s3l_state_get(void) { unsigned int state = 0xdeadbeef; @@ -526,6 +537,12 @@ int s3l_pipeline_create(short width, short height, char mirror, char flip, char if (s3l_ioctl(S3L_IOC_ENABLE_PREVIEW, 0, "ENABLE_PREVIEW")) return EXIT_FAILURE; + /* IAV keeps overlays across runs, and one that does not fit a stream stops it from starting */ + for (uint32_t i = 0; i < S3L_VENC_CHN_NUM; i++) { + struct s3l_ovl off = { .stream = i }; + ioctl(_s3l_fd, S3L_IOC_SET_OVERLAY, &off); + } + /* The DSP takes the 3A statistics setup into effect only when preview starts, and it can only be sent once preview runs: in the first preview after the microcode is loaded, the SDK's 284712 sends no statistics and this camera's own 267837 sends stale ones. So the @@ -652,6 +669,11 @@ int s3l_video_create(char index, hal_vidconfig *config) s3l_state[index].payload = config->codec; s3l_state[index].enable = 1; + if (!mjpeg) { + pthread_mutex_lock(&_s3l_rgn_mtx); + s3l_region_apply(); + pthread_mutex_unlock(&_s3l_rgn_mtx); + } return EXIT_SUCCESS; } @@ -719,6 +741,168 @@ void s3l_video_request_idr(char index) s3l_ioctl(S3L_IOC_SET_STREAM_CFG, cfg, "SET_STREAM_CFG(force IDR)"); } +static int s3l_region_map(void) +{ + if (_s3l_ovl) + return EXIT_SUCCESS; + uint32_t qb[16] = { S3L_BUFFER_OVERLAY }; + if (s3l_ioctl(S3L_IOC_QUERY_BUF, qb, "QUERY_BUF(overlay)")) + return EXIT_FAILURE; + if (qb[1] <= S3L_OVL_DATA_OFF) { + HAL_DANGER("s3l_rgn", "The overlay buffer is too small (%#x)\n", qb[1]); + return EXIT_FAILURE; + } + unsigned char *m = mmap64(NULL, qb[1], PROT_READ | PROT_WRITE, MAP_SHARED, _s3l_fd, qb[2]); + if (m == MAP_FAILED) { + HAL_DANGER("s3l_rgn", "Mapping the overlay buffer failed: %s\n", strerror(errno)); + return EXIT_FAILURE; + } + _s3l_ovl = m; + _s3l_ovl_len = qb[1]; + // Two pages per region, so an update never rewrites what the DSP is reading + _s3l_ovl_slot = ((qb[1] - S3L_OVL_DATA_OFF) / (S3L_OVL_AREAS * 2)) & ~31u; + return EXIT_SUCCESS; +} + +// The overlay is per stream: put the regions on every running H.264/H.265 stream +static void s3l_region_apply(void) +{ + struct s3l_ovl ovl; + + if (!_s3l_ovl) + return; + for (char i = 0; i < S3L_VENC_CHN_NUM; i++) { + if (!s3l_state[i].enable || (s3l_state[i].payload != HAL_VIDCODEC_H264 && + s3l_state[i].payload != HAL_VIDCODEC_H265)) + continue; + memset(&ovl, 0, sizeof(ovl)); + ovl.stream = i; + for (int a = 0; a < S3L_OVL_AREAS; a++) + if (_s3l_rgn[a].set) { + ovl.area[a] = _s3l_rgn[a].area; + ovl.flags = 1; + } + s3l_ioctl(S3L_IOC_SET_OVERLAY, &ovl, "SET_OVERLAY_INSERT"); + } +} + +// ARGB1555 to a CLUT entry: V, U, Y (BT.601, limited range), alpha +static void s3l_region_clut(unsigned char *e, unsigned short px, unsigned char alpha) +{ + int r = (px >> 10 & 31) * 255 / 31, g = (px >> 5 & 31) * 255 / 31, b = (px & 31) * 255 / 31; + e[0] = 128 + ((112 * r - 94 * g - 18 * b + 128) >> 8); + e[1] = 128 + ((-38 * r - 74 * g + 112 * b + 128) >> 8); + e[2] = 16 + ((66 * r + 129 * g + 25 * b + 128) >> 8); + e[3] = alpha; +} + +int s3l_region_create(char handle, hal_rect rect, short opacity) +{ + if (handle < 0 || handle >= S3L_OVL_AREAS) + return EXIT_FAILURE; + pthread_mutex_lock(&_s3l_rgn_mtx); + _s3l_rgn[handle].rect = rect; + _s3l_rgn[handle].opacity = MIN(MAX(opacity, 0), 255); + pthread_mutex_unlock(&_s3l_rgn_mtx); + return EXIT_SUCCESS; +} + +int s3l_region_setbitmap(char handle, hal_bitmap *bitmap) +{ + static short index[32768]; + unsigned short *src = bitmap ? bitmap->data : NULL; + + if (handle < 0 || handle >= S3L_OVL_AREAS || !src || !bitmap->dim.width || !bitmap->dim.height) + return EXIT_FAILURE; + pthread_mutex_lock(&_s3l_rgn_mtx); + if (s3l_region_map()) { + pthread_mutex_unlock(&_s3l_rgn_mtx); + return EXIT_FAILURE; + } + + // Areas start at an even x and a y multiple of 4, span an even width and a height multiple of 4 + unsigned int x = _s3l_rgn[handle].rect.x & ~1, y = _s3l_rgn[handle].rect.y & ~3; + if (x >= _s3l_main_w || y >= _s3l_main_h) { + _s3l_rgn[handle].set = 0; + s3l_region_apply(); + pthread_mutex_unlock(&_s3l_rgn_mtx); + return EXIT_FAILURE; + } + unsigned int w = MIN(bitmap->dim.width, _s3l_main_w - x); + unsigned int h = MIN(bitmap->dim.height, (_s3l_main_h - y) & ~3); + unsigned int width = (w + 1) & ~1, height = (h + 3) & ~3, pitch = (width + 31) & ~31; + if (pitch * height > _s3l_ovl_slot) { + height = (_s3l_ovl_slot / pitch) & ~3; + h = MIN(h, height); + HAL_WARNING("s3l_rgn", "Region %d is too large, keeping its first %u rows\n", handle, h); + } + if (!h) { + _s3l_rgn[handle].set = 0; + s3l_region_apply(); + pthread_mutex_unlock(&_s3l_rgn_mtx); + return EXIT_FAILURE; + } + + unsigned char page = _s3l_rgn[handle].page ^ 1, alpha = _s3l_rgn[handle].opacity; + unsigned int slot = handle * 2 + page; + unsigned char *clut = _s3l_ovl + slot * S3L_OVL_CLUT_LEN; + unsigned char *dst = _s3l_ovl + S3L_OVL_DATA_OFF + slot * _s3l_ovl_slot; + + // Index 0 is transparent; up to 255 colours exactly, beyond that 3-3-2 + memset(clut, 0, S3L_OVL_CLUT_LEN); + clut[0] = clut[1] = 128; + clut[2] = 16; + memset(index, 0xff, sizeof(index)); + int colours = 0, quantise = 0; + for (unsigned int i = 0; i < w * h && !quantise; i++) { + unsigned short px = src[(i / w) * bitmap->dim.width + i % w]; + if (!(px & 0x8000) || index[px & 0x7fff] >= 0) + continue; + if (colours == 255) + quantise = 1; + else { + index[px & 0x7fff] = ++colours; + s3l_region_clut(clut + 4 * colours, px, alpha); + } + } + if (quantise) + for (int i = 1; i < 256; i++) { + int r = (i - 1) >> 5 & 7, g = (i - 1) >> 2 & 7, b = (i - 1) & 3; + s3l_region_clut(clut + 4 * i, (r * 31 / 7) << 10 | (g * 31 / 7) << 5 | b * 31 / 3, alpha); + } + + memset(dst, 0, pitch * height); + for (unsigned int r = 0; r < h; r++) + for (unsigned int c = 0; c < w; c++) { + unsigned short px = src[r * bitmap->dim.width + c]; + if (!(px & 0x8000)) + continue; + dst[r * pitch + c] = quantise ? MIN(1 + ((px >> 12 & 7) << 5 | (px >> 7 & 7) << 2 | (px >> 3 & 3)), 255) : + index[px & 0x7fff]; + } + + _s3l_rgn[handle].area = (struct s3l_ovl_area){ .enable = 1, .width = width, .pitch = pitch, + .height = height, .size = pitch * height, .x = x, .y = y, + .clut = slot * S3L_OVL_CLUT_LEN, .data = dst - _s3l_ovl }; + _s3l_rgn[handle].page = page; + _s3l_rgn[handle].set = 1; + s3l_region_apply(); + pthread_mutex_unlock(&_s3l_rgn_mtx); + return EXIT_SUCCESS; +} + +void s3l_region_destroy(char handle) +{ + if (handle < 0 || handle >= S3L_OVL_AREAS) + return; + pthread_mutex_lock(&_s3l_rgn_mtx); + if (_s3l_rgn[handle].set) { + _s3l_rgn[handle].set = 0; + s3l_region_apply(); + } + pthread_mutex_unlock(&_s3l_rgn_mtx); +} + void *s3l_video_thread(void) { static uint32_t desc[S3L_SZ_QUERYDESC / 4 + 64]; diff --git a/src/hal/amba/s3l_hal.h b/src/hal/amba/s3l_hal.h index 01beaa05..aa3473bb 100644 --- a/src/hal/amba/s3l_hal.h +++ b/src/hal/amba/s3l_hal.h @@ -40,6 +40,10 @@ int s3l_video_destroy_all(void); void s3l_video_request_idr(char index); void *s3l_video_thread(void); +int s3l_region_create(char handle, hal_rect rect, short opacity); +int s3l_region_setbitmap(char handle, hal_bitmap *bitmap); +void s3l_region_destroy(char handle); + int s3l_aaa_config(int fd); int s3l_ae_start(int fd, int fps); int s3l_isp_set_wb(double r, double b); diff --git a/src/hal/amba/s3l_iav.h b/src/hal/amba/s3l_iav.h index 63d3afd0..8546d869 100644 --- a/src/hal/amba/s3l_iav.h +++ b/src/hal/amba/s3l_iav.h @@ -113,6 +113,16 @@ struct s3l_vin_info { uint32_t vsrc_id, video_mode, width, height, fps, rsvd[4]; struct s3l_isp_hdr { char magic[8]; uint32_t ops, img_len, img_off, rels; }; struct s3l_isp_op { uint8_t kind, nr, rsvd[2]; uint32_t offset, flags; }; +#define S3L_IOC_SET_OVERLAY 0x40045660u +#define S3L_IOC_GET_OVERLAY 0xc0045660u +#define S3L_BUFFER_OVERLAY 4 +#define S3L_OVL_AREAS 10 +#define S3L_OVL_CLUT_LEN 1024 +#define S3L_OVL_DATA_OFF 0x8000 +struct s3l_ovl_area { uint16_t enable, width, pitch, height; uint32_t size; uint16_t x, y; uint32_t clut, data; }; +struct s3l_ovl { uint32_t stream, flags; struct s3l_ovl_area area[S3L_OVL_AREAS]; }; +_Static_assert(sizeof(struct s3l_ovl) == 248, "IAV overlay insert is 248 bytes on S3L"); + #define S3L_NL_PORT_IMAGE 20 #define S3L_NL_MSG_LEN 128 struct s3l_nl_msg { uint32_t pid, port, type, dir, cmd, status; uint8_t pad[S3L_NL_MSG_LEN - 24]; }; diff --git a/src/region.c b/src/region.c index b6e7ca17..e2bd5ca9 100644 --- a/src/region.c +++ b/src/region.c @@ -393,6 +393,12 @@ found_font:; hal_rect rect = { .height = bitmap.dim.height, .width = bitmap.dim.width, .x = osds[id].posx, .y = osds[id].posy }; switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: + s3l_region_create(id, rect, osds[id].opal); + s3l_region_setbitmap(id, &bitmap); + break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: i6_region_create(id, rect, osds[id].opal); @@ -479,6 +485,12 @@ found_font:; hal_rect rect = { .height = bitmap.dim.height, .width = bitmap.dim.width, .x = osds[id].posx, .y = osds[id].posy }; switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: + s3l_region_create(id, rect, osds[id].opal); + s3l_region_setbitmap(id, &bitmap); + break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: i6_region_create(id, rect, osds[id].opal); @@ -548,6 +560,9 @@ found_font:; } else switch (plat) { +#if defined(__arm__) + case HAL_PLATFORM_S3L: s3l_region_destroy(id); break; +#endif #if defined(__ARM_PCS_VFP) case HAL_PLATFORM_I6: i6_region_destroy(id); break; case HAL_PLATFORM_I6C: i6c_region_destroy(id); break; From 76f6834e1c7271a4cd56ec1193081d50f7fc9e94 Mon Sep 17 00:00:00 2001 From: Dylan Corrales Date: Fri, 6 Nov 2026 22:15:07 -0500 Subject: [PATCH 09/11] S3L: a smaller H.264/H.265 stream beside the main one An H.264/H.265 stream could only be the main one, from the main source buffer. Now the HAL allows three encode streams, and one smaller than the main picture is encoded from the source buffer of its size: the second one, or the third when MJPEG put its own size there. The second buffer is limited to 720x720 and its size is fixed when preview starts, so it is planned then. MJPEG or the snapshots keep it at their size when they are at most 720x720; otherwise, or when neither is on, it is 640 wide at the main aspect (640x360 for 2688x1512), so a 640x360 sub-stream, the size the RTSP sub stream of OpenIPC/divinus#62 asks for, finds it. A size no buffer has is refused with the sizes there are (media.c then stops Divinus starting, as for any stream it cannot create). The on-screen display stays on the stream from the main buffer, and the frame thread now watches all three streams. Divinus itself creates no second H.264/H.265 stream yet: that comes with the sub-stream plumbing of OpenIPC/divinus#62. Until then, this path is taken only when a streamer creates one. Tested on the IP4M-1026E, with a local hook in media.c creating a 640x360 H.264 sub-stream next to the 2688x1512 main stream (main at 20 fps throughout): alone; with MJPEG at 1280x720 (three streams) and at 640x360 (sharing the buffer); with 1 fps JPEG snapshots at 1920x1080; and with MJPEG at 640x480, where the sub-stream is refused and, with the hook, the rest runs. Without a sub-stream, the board's default (main stream, OSD) runs as before. Builds for all 12 targets in build.yml. --- src/hal/amba/s3l_hal.c | 51 +++++++++++++++++++++++++++++++++--------- src/hal/amba/s3l_hal.h | 2 +- 2 files changed, 41 insertions(+), 12 deletions(-) diff --git a/src/hal/amba/s3l_hal.c b/src/hal/amba/s3l_hal.c index e979f434..6f9dd720 100644 --- a/src/hal/amba/s3l_hal.c +++ b/src/hal/amba/s3l_hal.c @@ -18,8 +18,9 @@ static unsigned char *_s3l_bsb; static unsigned int _s3l_bsb_len; static unsigned short _s3l_main_w, _s3l_main_h; static char _s3l_fps = 30; -static unsigned short _s3l_sub_w = 640, _s3l_sub_h = 480; +static unsigned short _s3l_sub_w, _s3l_sub_h; static char _s3l_mjpeg_buf = 1; +static unsigned short _s3l_buf1_w, _s3l_buf1_h; static char _s3l_isp_path[128]; static unsigned char *_s3l_img; @@ -37,6 +38,7 @@ static struct { struct s3l_ovl_area area; } _s3l_rgn[S3L_OVL_AREAS]; static pthread_mutex_t _s3l_rgn_mtx = PTHREAD_MUTEX_INITIALIZER; +static char _s3l_chn_buf[S3L_VENC_CHN_NUM]; static void s3l_region_apply(void); @@ -464,14 +466,23 @@ int s3l_pipeline_create(short width, short height, char mirror, char flip, char info.width, info.height, _s3l_main_w, _s3l_main_h); /* A sub-buffer must be smaller than the main one, and the second one at most 720x720 */ - if (_s3l_sub_w >= _s3l_main_w || _s3l_sub_h >= _s3l_main_h) + char sized = _s3l_sub_w && _s3l_sub_h; + if (sized && (_s3l_sub_w >= _s3l_main_w || _s3l_sub_h >= _s3l_main_h)) _s3l_mjpeg_buf = 0; - else if (_s3l_sub_w > S3L_SRCBUF1_MAX || _s3l_sub_h > S3L_SRCBUF1_MAX) + else if (sized && (_s3l_sub_w > S3L_SRCBUF1_MAX || _s3l_sub_h > S3L_SRCBUF1_MAX)) _s3l_mjpeg_buf = 2; else _s3l_mjpeg_buf = 1; - unsigned short b1w = _s3l_mjpeg_buf == 1 ? _s3l_sub_w : MIN(640, (_s3l_main_w / 2) & ~15); - unsigned short b1h = _s3l_mjpeg_buf == 1 ? _s3l_sub_h : MIN(480, (_s3l_main_h / 2) & ~7); + /* Otherwise the second buffer is 640 wide at the main aspect, shared by a sub-stream and + MJPEG or the snapshots when they were not given a size */ + unsigned short b1w = _s3l_mjpeg_buf == 1 && sized ? _s3l_sub_w : MIN(640, (_s3l_main_w / 2) & ~15); + unsigned short b1h = _s3l_mjpeg_buf == 1 && sized ? _s3l_sub_h : (b1w * _s3l_main_h / _s3l_main_w + 7) & ~7; + _s3l_buf1_w = b1w; + _s3l_buf1_h = b1h; + if (!sized) { + _s3l_sub_w = b1w; + _s3l_sub_h = b1h; + } if (!_s3l_mjpeg_buf) { _s3l_sub_w = _s3l_main_w; _s3l_sub_h = _s3l_main_h; @@ -481,7 +492,7 @@ int s3l_pipeline_create(short width, short height, char mirror, char flip, char if (s3l_ioctl(S3L_IOC_GET_SYSRES, buf, "GET_SYSTEM_RESOURCE")) return EXIT_FAILURE; buf[S3L_SYSRES_ENC_MODE] = 0; - buf[S3L_SYSRES_MAX_NUM_ENC] = 2; + buf[S3L_SYSRES_MAX_NUM_ENC] = S3L_VENC_CHN_NUM; v[0] = _s3l_main_w; v[1] = _s3l_main_h; memcpy(buf + S3L_SYSRES_BUF_MAX(0), v, 8); memcpy(buf + S3L_SYSRES_STREAM_MAX(0), v, 8); @@ -492,7 +503,10 @@ int s3l_pipeline_create(short width, short height, char mirror, char flip, char v[0] = _s3l_sub_w; v[1] = _s3l_sub_h; if (_s3l_mjpeg_buf == 2) memcpy(buf + S3L_SYSRES_BUF_MAX(2), v, 8); + /* Streams 1 and 2 each take MJPEG, the snapshots or a sub-stream */ + v[0] = MAX(_s3l_sub_w, b1w); v[1] = MAX(_s3l_sub_h, b1h); memcpy(buf + S3L_SYSRES_STREAM_MAX(1), v, 8); + memcpy(buf + S3L_SYSRES_STREAM_MAX(2), v, 8); v[0] = 1; memcpy(buf + S3L_SYSRES_C0, v, 4); v[0] = 0; memcpy(buf + S3L_SYSRES_DC, v, 4); memcpy(buf + S3L_SYSRES_EXTRA_A4, v, 4); @@ -610,10 +624,21 @@ int s3l_video_create(char index, hal_vidconfig *config) v[0] = mjpeg ? S3L_STREAM_TYPE_MJPEG : h265 ? S3L_STREAM_TYPE_H265 : S3L_STREAM_TYPE_H264; memcpy(buf + S3L_STRFMT_TYPE, v, 4); char sub = mjpeg ? _s3l_mjpeg_buf : 0; + unsigned short w = sub ? _s3l_sub_w : _s3l_main_w, h = sub ? _s3l_sub_h : _s3l_main_h; + if (!mjpeg && config->width && config->height && + (config->width < _s3l_main_w || config->height < _s3l_main_h)) { + if (config->width == _s3l_buf1_w && config->height == _s3l_buf1_h) + sub = 1, w = _s3l_buf1_w, h = _s3l_buf1_h; + else if (_s3l_mjpeg_buf == 2 && config->width == _s3l_sub_w && config->height == _s3l_sub_h) + sub = 2, w = _s3l_sub_w, h = _s3l_sub_h; + else + HAL_ERROR("s3l_venc", "No source buffer is %dx%d (main %ux%u, second %ux%u)\n", + config->width, config->height, _s3l_main_w, _s3l_main_h, _s3l_buf1_w, _s3l_buf1_h); + } v[0] = sub; memcpy(buf + S3L_STRFMT_BUF_ID, v, 4); - v[0] = sub ? _s3l_sub_w : _s3l_main_w; - v[1] = sub ? _s3l_sub_h : _s3l_main_h; + v[0] = w; + v[1] = h; v[2] = 0; v[3] = 0; memcpy(buf + S3L_STRFMT_ENC_WIN, v, 16); if (s3l_ioctl(S3L_IOC_SET_STREAM_FORMAT, buf, "SET_STREAM_FORMAT")) @@ -667,6 +692,7 @@ int s3l_video_create(char index, hal_vidconfig *config) s3l_ioctl(S3L_IOC_SET_STREAM_CFG, fps, "SET_STREAM_CFG(fps)"); } + _s3l_chn_buf[index] = sub; s3l_state[index].payload = config->codec; s3l_state[index].enable = 1; if (!mjpeg) { @@ -764,7 +790,7 @@ static int s3l_region_map(void) return EXIT_SUCCESS; } -// The overlay is per stream: put the regions on every running H.264/H.265 stream +// The overlay is per stream: put the regions on the H.264/H.265 streams from the main buffer static void s3l_region_apply(void) { struct s3l_ovl ovl; @@ -772,7 +798,7 @@ static void s3l_region_apply(void) if (!_s3l_ovl) return; for (char i = 0; i < S3L_VENC_CHN_NUM; i++) { - if (!s3l_state[i].enable || (s3l_state[i].payload != HAL_VIDCODEC_H264 && + if (!s3l_state[i].enable || _s3l_chn_buf[i] || (s3l_state[i].payload != HAL_VIDCODEC_H264 && s3l_state[i].payload != HAL_VIDCODEC_H265)) continue; memset(&ovl, 0, sizeof(ovl)); @@ -912,7 +938,10 @@ void *s3l_video_thread(void) unsigned int frameId = 0; while (keepRunning) { - if (!_s3l_bsb || !(s3l_state[0].enable || s3l_state[1].enable)) { usleep(100000); continue; } + char active = 0; + for (char i = 0; i < S3L_VENC_CHN_NUM; i++) + active |= s3l_state[i].enable; + if (!_s3l_bsb || !active) { usleep(100000); continue; } memset(desc, 0, sizeof(desc)); desc[0] = S3L_DESC_FRAME; diff --git a/src/hal/amba/s3l_hal.h b/src/hal/amba/s3l_hal.h index aa3473bb..fe4eb3cb 100644 --- a/src/hal/amba/s3l_hal.h +++ b/src/hal/amba/s3l_hal.h @@ -16,7 +16,7 @@ #include #include -#define S3L_VENC_CHN_NUM 2 +#define S3L_VENC_CHN_NUM 3 extern hal_chnstate s3l_state[S3L_VENC_CHN_NUM]; extern int (*s3l_aud_cb)(hal_audframe*); From 5755065f416e4defdb95eb31e3d3a600571f3aac Mon Sep 17 00:00:00 2001 From: Dylan Corrales Date: Fri, 6 Nov 2026 22:32:36 -0500 Subject: [PATCH 10/11] S3L: frame rates per stream, within the encoder's load limit IAV refuses to start a stream when all streams together exceed the DSP's encoder load, in macroblocks per second (570600 on S3L, "1080p60 + 480p60"). Each stream's frame rate was set only after it started, at the VIN's, so the check counted every stream at the full rate: MJPEG at the main size (or the yaml template's 3840x2160, which falls back to it) beside the 2688x1512 main stream never started. The frame rate is now set before the start, as the configured fraction of the VIN's (snapshots 1 fps as before). The HAL adds up the streams' load first: MJPEG and the snapshots that would not fit run at the highest frame rate that does, with a warning (and are refused if none does); an H.264/H.265 stream that would not fit is refused with the numbers, which stops Divinus starting. A stream asking for more than the rate the pipeline started with runs at that rate, with a warning. A stream is enabled before its encoder starts, so its first frame (the IDR) is not dropped. Tested on the IP4M-1026E: MJPEG at 3840x2160 and 20 fps runs at 2688x1512 and 15 fps next to the main stream at 20 fps (it did not start before); MJPEG at 1920x1080 and 15 fps gives exactly 15 fps; with a 640x360 sub-stream and full-size MJPEG, MJPEG drops to 14 fps; the board's default is unchanged. The refusal of an H.264/H.265 stream was not hit, as no case here adds one after the budget is used. Builds for all 12 targets in build.yml. --- src/hal/amba/s3l_hal.c | 42 +++++++++++++++++++++++++++++++++++++----- src/hal/amba/s3l_iav.h | 3 +++ 2 files changed, 40 insertions(+), 5 deletions(-) diff --git a/src/hal/amba/s3l_hal.c b/src/hal/amba/s3l_hal.c index 6f9dd720..2a60e1f3 100644 --- a/src/hal/amba/s3l_hal.c +++ b/src/hal/amba/s3l_hal.c @@ -21,6 +21,7 @@ static char _s3l_fps = 30; static unsigned short _s3l_sub_w, _s3l_sub_h; static char _s3l_mjpeg_buf = 1; static unsigned short _s3l_buf1_w, _s3l_buf1_h; +static unsigned int _s3l_load[S3L_VENC_CHN_NUM]; static char _s3l_isp_path[128]; static unsigned char *_s3l_img; @@ -682,19 +683,49 @@ int s3l_video_create(char index, hal_vidconfig *config) } } - if (s3l_ioctl_nosig(S3L_IOC_START_ENCODE, (void *)(1u << index)) < 0) - HAL_ERROR("s3l_venc", "Starting encoder %d failed: %s\n", index, strerror(errno)); - + /* Frame rate as a fraction of the VIN's, set before the start: IAV checks the total encoder + load (macroblocks per second over all streams) against what the DSP can do */ + unsigned int multi = 1, div = 1, mbs = ((w + 15) >> 4) * ((h + 15) >> 4), others = 0; + if (config->codec == HAL_VIDCODEC_JPG) + div = _s3l_fps; + else if (config->framerate > 0 && config->framerate < _s3l_fps) + multi = config->framerate, div = _s3l_fps; + else if (config->framerate > _s3l_fps) + HAL_WARNING("s3l_venc", "Stream %d runs at %d fps, the rate the pipeline was started with, " + "until Divinus restarts\n", index, _s3l_fps); + for (char i = 0; i < S3L_VENC_CHN_NUM; i++) + if (i != index && s3l_state[i].enable) + others += _s3l_load[i]; + if (others + mbs * _s3l_fps * multi / div > S3L_ENC_LOAD_MAX) { + unsigned int fit = others < S3L_ENC_LOAD_MAX ? (S3L_ENC_LOAD_MAX - others) / mbs : 0; + if (!mjpeg || !fit) + HAL_ERROR("s3l_venc", "Encoder %d (%ux%u at %u fps) exceeds the encoder load left " + "(%u of %u macroblocks/s)\n", index, w, h, _s3l_fps * multi / div, + S3L_ENC_LOAD_MAX - MIN(others, S3L_ENC_LOAD_MAX), S3L_ENC_LOAD_MAX); + HAL_WARNING("s3l_venc", "Encoder %d (%ux%u) runs at %u fps to fit the encoder load\n", + index, w, h, fit); + multi = fit, div = _s3l_fps; + } uint32_t fps[32] = { (uint32_t)index, S3L_STMCFG_FPS }; if (!s3l_ioctl(S3L_IOC_GET_STREAM_CFG, fps, "GET_STREAM_CFG(fps)")) { - fps[2] = 1; - fps[3] = config->codec == HAL_VIDCODEC_JPG ? _s3l_fps : 1; + fps[2] = multi; + fps[3] = div; s3l_ioctl(S3L_IOC_SET_STREAM_CFG, fps, "SET_STREAM_CFG(fps)"); } + /* Enabled first, so the frame thread keeps the stream's first frames (its IDR); its buffer + is known by then, for the on-screen display */ + pthread_mutex_lock(&_s3l_rgn_mtx); _s3l_chn_buf[index] = sub; s3l_state[index].payload = config->codec; s3l_state[index].enable = 1; + pthread_mutex_unlock(&_s3l_rgn_mtx); + if (s3l_ioctl_nosig(S3L_IOC_START_ENCODE, (void *)(1u << index)) < 0) { + s3l_state[index].enable = 0; + HAL_ERROR("s3l_venc", "Starting encoder %d failed: %s\n", index, strerror(errno)); + } + + _s3l_load[index] = mbs * _s3l_fps * multi / div; if (!mjpeg) { pthread_mutex_lock(&_s3l_rgn_mtx); s3l_region_apply(); @@ -710,6 +741,7 @@ int s3l_video_destroy(char index) s3l_ioctl_nosig(S3L_IOC_STOP_ENCODE, (void *)(1u << index)); s3l_state[index].enable = 0; s3l_state[index].payload = HAL_VIDCODEC_UNSPEC; + _s3l_load[index] = 0; return EXIT_SUCCESS; } diff --git a/src/hal/amba/s3l_iav.h b/src/hal/amba/s3l_iav.h index 8546d869..cafe1ecb 100644 --- a/src/hal/amba/s3l_iav.h +++ b/src/hal/amba/s3l_iav.h @@ -113,6 +113,9 @@ struct s3l_vin_info { uint32_t vsrc_id, video_mode, width, height, fps, rsvd[4]; struct s3l_isp_hdr { char magic[8]; uint32_t ops, img_len, img_off, rels; }; struct s3l_isp_op { uint8_t kind, nr, rsvd[2]; uint32_t offset, flags; }; +/* Encoder load limit, in macroblocks per second over all streams (IAV: "1080p60 + 480p60") */ +#define S3L_ENC_LOAD_MAX 570600 + #define S3L_IOC_SET_OVERLAY 0x40045660u #define S3L_IOC_GET_OVERLAY 0xc0045660u #define S3L_BUFFER_OVERLAY 4 From 843a43c4f21d94cabc3c6d3045dc0fd9d2fab633 Mon Sep 17 00:00:00 2001 From: Dylan Corrales Date: Fri, 6 Nov 2026 22:55:39 -0500 Subject: [PATCH 11/11] S3L: main stream sizes the source buffers cannot take as they are The main source buffer scales the VIN down by less than 4, and buffers and encode streams are a multiple of 16 wide and of 8 high. A main stream outside that, such as 640x360 or 960x540 on a 2688x1512 sensor, failed SET_SRCBUF_SETUP and Divinus exited without video. A size four or more times below the VIN is now encoded as a centred window of the smallest main buffer there can be (688x384 here), so only the edges are cropped, and sizes off the 16x8 grid are rounded up with a warning (960x540 is encoded as 960x544); the requested size is matched against the buffers the same way. The second source buffer, at least 240x240 in this mode, is left off when it would be smaller, and MJPEG or the snapshots use the main buffer. Its type is now set on every start, as for the third and fourth: IAV keeps it across runs, so a run with it off left the next one unable to encode from it. An MJPEG or JPEG size too small for a sub-buffer (at least 240x240, and scaled down from the main buffer by less than 6, IAV's check_zoom_limit) is scaled up at its aspect with a warning, and one not below the main buffer warns that it is encoded at the main buffer's size. The README notes that a stream's size, or a higher frame rate, takes a restart of Divinus to change. Tested on the IP4M-1026E: main streams of 640x360 (exited before), 960x540 (encoded 960x544), 656x370 (a 656x376 window of a 688x384 buffer), 1280x720 and 2688x1512; MJPEG-only 960x540; MJPEG at 640x360 on the second buffer before and after a run with it off; MJPEG and JPEG at 160x120 (SET_SRCBUF_SETUP failed before; now 464x344 with a warning), and MJPEG at 1920x1080 beside a 1280x720 main stream (warns, encodes 1280x720). --- README.md | 2 +- src/hal/amba/s3l_hal.c | 80 ++++++++++++++++++++++++++++++++++-------- src/hal/amba/s3l_iav.h | 1 + 3 files changed, 68 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 7ddc6bc2..7f240f4d 100644 --- a/README.md +++ b/README.md @@ -74,7 +74,7 @@ _* At the moment, text, RGB or bitfield bitmaps and PNG overlays are handled, mo [^18]: SSC377\(D/DE/QE\) and SSC378\[DE/QE\] [^19]: SSC359G [^20]: V851S\(E\) and V853 -[^21]: Ambarella S3L (S3LM), via the IAV driver; the ISP configuration and 3A (AE, AWB) run in Divinus; the on-screen display is drawn on the H.264/H.265 stream +[^21]: Ambarella S3L (S3LM), via the IAV driver; the ISP configuration and 3A (AE, AWB) run in Divinus; the on-screen display is drawn on the H.264/H.265 stream; JPEG snapshots come from the MJPEG stream (its size and quality) when it is on, otherwise from a 1 fps JPEG stream at jpeg.width/height and jpeg.qfactor (a snapshot request's own size, quality and grayscale are not applied); changing a stream's size, enabling one that was off at start, or raising its frame rate needs a restart of Divinus ### Documentation diff --git a/src/hal/amba/s3l_hal.c b/src/hal/amba/s3l_hal.c index 2a60e1f3..6d4cd4c4 100644 --- a/src/hal/amba/s3l_hal.c +++ b/src/hal/amba/s3l_hal.c @@ -22,6 +22,7 @@ static unsigned short _s3l_sub_w, _s3l_sub_h; static char _s3l_mjpeg_buf = 1; static unsigned short _s3l_buf1_w, _s3l_buf1_h; static unsigned int _s3l_load[S3L_VENC_CHN_NUM]; +static unsigned short _s3l_crop_w, _s3l_crop_h; static char _s3l_isp_path[128]; static unsigned char *_s3l_img; @@ -463,13 +464,47 @@ int s3l_pipeline_create(short width, short height, char mirror, char flip, char _s3l_main_w = (width > 0 && width < (int)info.width) ? width : info.width; _s3l_main_h = (height > 0 && height < (int)info.height) ? height : info.height; + /* The main buffer scales down by less than 4: a smaller main stream is a centred window + of the smallest main buffer there can be */ + _s3l_crop_w = _s3l_crop_h = 0; + if (_s3l_main_w * 4 <= info.width || _s3l_main_h * 4 <= info.height) { + _s3l_crop_w = _s3l_main_w; + _s3l_crop_h = _s3l_main_h; + _s3l_main_w = MIN((MAX(_s3l_main_w, (info.width / 4 + 16) & ~15) + 15) & ~15, info.width); + _s3l_main_h = MIN((MAX(_s3l_main_h, (info.height / 4 + 8) & ~7) + 7) & ~7, info.height); + HAL_WARNING("s3l_iav", "%ux%u is 4 times or more below the VIN, cropping it from %ux%u\n", + _s3l_crop_w, _s3l_crop_h, _s3l_main_w, _s3l_main_h); + } + /* Buffers and streams are a multiple of 16 wide and of 8 high: other sizes are rounded up */ + unsigned short *rw = _s3l_crop_w ? &_s3l_crop_w : &_s3l_main_w, *rh = _s3l_crop_w ? &_s3l_crop_h : &_s3l_main_h; + if ((*rw & 15) || (*rh & 7)) { + unsigned short ow = *rw, oh = *rh; + *rw = MIN((*rw + 15) & ~15, info.width); + *rh = MIN((*rh + 7) & ~7, info.height); + HAL_WARNING("s3l_iav", "%ux%u is not a multiple of 16x8, encoding %ux%u\n", ow, oh, *rw, *rh); + } HAL_INFO("s3l_iav", "VIN %ux%u, main buffer %ux%u\n", info.width, info.height, _s3l_main_w, _s3l_main_h); /* A sub-buffer must be smaller than the main one, and the second one at most 720x720 */ char sized = _s3l_sub_w && _s3l_sub_h; - if (sized && (_s3l_sub_w >= _s3l_main_w || _s3l_sub_h >= _s3l_main_h)) + /* A sub-buffer is at least 240x240 and scaled down from the main one by less than 6 + (IAV's check_zoom_limit): a smaller MJPEG or JPEG size is scaled up at its aspect */ + unsigned int minw = MAX(S3L_SRCBUF_MIN, _s3l_main_w / 6 + 1), minh = MAX(S3L_SRCBUF_MIN, _s3l_main_h / 6 + 1); + if (sized && (_s3l_sub_w < minw || _s3l_sub_h < minh)) { + unsigned int w = _s3l_sub_w, h = _s3l_sub_h; + if (w < minw) { h = (h * minw + w - 1) / w; w = minw; } + if (h < minh) { w = (w * minh + h - 1) / h; h = minh; } + HAL_WARNING("s3l_iav", "%ux%u is too small to scale to from %ux%u, encoding %ux%u\n", + _s3l_sub_w, _s3l_sub_h, _s3l_main_w, _s3l_main_h, (w + 15) & ~15, (h + 7) & ~7); + _s3l_sub_w = (w + 15) & ~15; + _s3l_sub_h = (h + 7) & ~7; + } + if (sized && (_s3l_sub_w >= _s3l_main_w || _s3l_sub_h >= _s3l_main_h)) { + HAL_WARNING("s3l_iav", "%ux%u is not below the main buffer, encoding it at %ux%u\n", + _s3l_sub_w, _s3l_sub_h, _s3l_main_w, _s3l_main_h); _s3l_mjpeg_buf = 0; + } else if (sized && (_s3l_sub_w > S3L_SRCBUF1_MAX || _s3l_sub_h > S3L_SRCBUF1_MAX)) _s3l_mjpeg_buf = 2; else @@ -478,11 +513,15 @@ int s3l_pipeline_create(short width, short height, char mirror, char flip, char MJPEG or the snapshots when they were not given a size */ unsigned short b1w = _s3l_mjpeg_buf == 1 && sized ? _s3l_sub_w : MIN(640, (_s3l_main_w / 2) & ~15); unsigned short b1h = _s3l_mjpeg_buf == 1 && sized ? _s3l_sub_h : (b1w * _s3l_main_h / _s3l_main_w + 7) & ~7; + /* Sub-buffers are at least 240x240: below a 480x480 main buffer the second one is left off */ + if (!(_s3l_mjpeg_buf == 1 && sized) && (b1w < S3L_SRCBUF_MIN || b1h < S3L_SRCBUF_MIN)) + b1w = b1h = 0; _s3l_buf1_w = b1w; _s3l_buf1_h = b1h; if (!sized) { - _s3l_sub_w = b1w; - _s3l_sub_h = b1h; + _s3l_mjpeg_buf = b1w ? 1 : 0; + _s3l_sub_w = b1w ? b1w : _s3l_main_w; + _s3l_sub_h = b1h ? b1h : _s3l_main_h; } if (!_s3l_mjpeg_buf) { _s3l_sub_w = _s3l_main_w; @@ -536,6 +575,9 @@ int s3l_pipeline_create(short width, short height, char mirror, char flip, char memcpy(buf + S3L_SRCBUF_MAIN_WIN_B, v, 8); v[0] = b1w; v[1] = b1h; memcpy(buf + S3L_SRCBUF_SIZE(1), v, 8); + /* IAV keeps the types across runs, so set them every time */ + v[0] = b1w ? S3L_SRCBUF_TYPE_ENCODE : 0; + memcpy(buf + S3L_SRCBUF_TYPE(1), v, 4); for (int i = 2; i < 4; i++) { char used = i == 2 && _s3l_mjpeg_buf == 2; v[0] = used ? _s3l_sub_w : 0; v[1] = used ? _s3l_sub_h : 0; @@ -625,22 +667,30 @@ int s3l_video_create(char index, hal_vidconfig *config) v[0] = mjpeg ? S3L_STREAM_TYPE_MJPEG : h265 ? S3L_STREAM_TYPE_H265 : S3L_STREAM_TYPE_H264; memcpy(buf + S3L_STRFMT_TYPE, v, 4); char sub = mjpeg ? _s3l_mjpeg_buf : 0; - unsigned short w = sub ? _s3l_sub_w : _s3l_main_w, h = sub ? _s3l_sub_h : _s3l_main_h; - if (!mjpeg && config->width && config->height && - (config->width < _s3l_main_w || config->height < _s3l_main_h)) { - if (config->width == _s3l_buf1_w && config->height == _s3l_buf1_h) + unsigned short w = sub ? _s3l_sub_w : _s3l_main_w, h = sub ? _s3l_sub_h : _s3l_main_h, x = 0, y = 0; + /* Requested sizes are rounded up to 16x8 as the buffers were */ + unsigned short cw = (config->width + 15) & ~15, ch = (config->height + 7) & ~7; + if (!mjpeg && _s3l_crop_w && cw == _s3l_crop_w && ch == _s3l_crop_h) { + w = _s3l_crop_w; + h = _s3l_crop_h; + x = ((_s3l_main_w - w) / 2) & ~1; + y = ((_s3l_main_h - h) / 2) & ~1; + } else if (!mjpeg && config->width && config->height && + (cw < _s3l_main_w || ch < _s3l_main_h)) { + if (cw == _s3l_buf1_w && ch == _s3l_buf1_h) sub = 1, w = _s3l_buf1_w, h = _s3l_buf1_h; - else if (_s3l_mjpeg_buf == 2 && config->width == _s3l_sub_w && config->height == _s3l_sub_h) + else if (_s3l_mjpeg_buf == 2 && cw == _s3l_sub_w && ch == _s3l_sub_h) sub = 2, w = _s3l_sub_w, h = _s3l_sub_h; else - HAL_ERROR("s3l_venc", "No source buffer is %dx%d (main %ux%u, second %ux%u)\n", - config->width, config->height, _s3l_main_w, _s3l_main_h, _s3l_buf1_w, _s3l_buf1_h); + HAL_ERROR("s3l_venc", "No source buffer is %ux%u (main %ux%u, second %ux%u)\n", + cw, ch, _s3l_main_w, _s3l_main_h, _s3l_buf1_w, _s3l_buf1_h); } v[0] = sub; memcpy(buf + S3L_STRFMT_BUF_ID, v, 4); v[0] = w; v[1] = h; - v[2] = 0; v[3] = 0; + v[2] = x; + v[3] = y; memcpy(buf + S3L_STRFMT_ENC_WIN, v, 16); if (s3l_ioctl(S3L_IOC_SET_STREAM_FORMAT, buf, "SET_STREAM_FORMAT")) return EXIT_FAILURE; @@ -879,15 +929,17 @@ int s3l_region_setbitmap(char handle, hal_bitmap *bitmap) } // Areas start at an even x and a y multiple of 4, span an even width and a height multiple of 4 + /* Clipped to the main stream: the whole main buffer, or the window of it a smaller one is */ + unsigned int sw = _s3l_crop_w ? _s3l_crop_w : _s3l_main_w, sh = _s3l_crop_w ? _s3l_crop_h : _s3l_main_h; unsigned int x = _s3l_rgn[handle].rect.x & ~1, y = _s3l_rgn[handle].rect.y & ~3; - if (x >= _s3l_main_w || y >= _s3l_main_h) { + if (x >= sw || y >= sh) { _s3l_rgn[handle].set = 0; s3l_region_apply(); pthread_mutex_unlock(&_s3l_rgn_mtx); return EXIT_FAILURE; } - unsigned int w = MIN(bitmap->dim.width, _s3l_main_w - x); - unsigned int h = MIN(bitmap->dim.height, (_s3l_main_h - y) & ~3); + unsigned int w = MIN(bitmap->dim.width, sw - x); + unsigned int h = MIN(bitmap->dim.height, (sh - y) & ~3); unsigned int width = (w + 1) & ~1, height = (h + 3) & ~3, pitch = (width + 31) & ~31; if (pitch * height > _s3l_ovl_slot) { height = (_s3l_ovl_slot / pitch) & ~3; diff --git a/src/hal/amba/s3l_iav.h b/src/hal/amba/s3l_iav.h index cafe1ecb..0762f38f 100644 --- a/src/hal/amba/s3l_iav.h +++ b/src/hal/amba/s3l_iav.h @@ -63,6 +63,7 @@ #define S3L_SRCBUF_TYPE_ENCODE 1 #define S3L_SRCBUF_TYPE_PREVIEW 2 #define S3L_SRCBUF1_MAX 720 +#define S3L_SRCBUF_MIN 240 #define S3L_VIN_FPS(fps) ((uint32_t)((512000000ull + (fps) / 2) / (fps)))