OMX is not sending frames

[gamelaster]

SoC: S3

I have verified that camera works good.

# ffmpeg -s 640x480 -r 15 -i /dev/video0 -vcodec h264 out5.mp4

ffmpeg version 4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9.3.0 (Buildroot 2020.08-951-g20ddd63292-dirty)
  configuration: --enable-cross-compile --cross-prefix=/home/gamiee/Desktop/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf- --sysroot=/home/gamiee/Desktop/buildroot/output/host/arm-buildroot-linux-gnueabihf/sysroot --host-cc=/usr/bin/gcc --arch=arm --target-os=linux --disable-stripping --pkg-config=/home/gamiee/Desktop/buildroot/output/host/bin/pkg-config --disable-static --enable-shared --prefix=/usr --enable-avfilter --disable-version3 --enable-logging --enable-optimizations --disable-extra-warnings --enable-avdevice --enable-avcodec --enable-avformat --enable-network --disable-gray --enable-swscale-alpha --disable-small --enable-dct --enable-fft --enable-mdct --enable-rdft --disable-crystalhd --disable-dxva2 --enable-runtime-cpudetect --disable-hardcoded-tables --disable-mipsdsp --disable-mipsdspr2 --disable-msa --enable-hwaccels --disable-cuda --disable-cuvid --disable-nvenc --disable-avisynth --disable-frei0r --disable-libopencore-amrnb --disable-libopencore-amrwb --disable-libdc1394 --disable-libgsm --disable-libilbc --disable-libvo-amrwbenc --disable-symver --disable-doc --enable-gpl --enable-nonfree --enable-ffmpeg --disable-ffplay --disable-avresample --disable-ffprobe --disable-libxcb --disable-postproc --disable-swscale --enable-indevs --disable-alsa --enable-outdevs --enable-pthreads --disable-zlib --disable-bzlib --disable-libfdk-aac --disable-libcdio --disable-gnutls --disable-openssl --disable-libdrm --disable-libopenh264 --disable-vaapi --disable-vdpau --disable-mmal --disable-omx --disable-omx-rpi --disable-libopencv --disable-libopus --disable-libvpx --disable-libass --disable-libbluray --disable-libmfx --disable-librtmp --disable-libmp3lame --disable-libmodplug --disable-libspeex --disable-libtheora --disable-libwavpack --disable-iconv --disable-libfreetype --disable-fontconfig --disable-libopenjpeg --disable-libx264 --disable-libx265 --disable-libdav1d --disable-x86asm --disable-mmx --disable-sse --disable-sse2 --disable-sse3 --disable-ssse3 --disable-sse4 --disable-sse42 --disable-avx --disable-avx2 --enable-armv6 --enable-vfp --enable-neon --disable-altivec --extra-libs=-latomic --enable-pic --cpu=cortex-a7 --enable-omx
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libswresample   3.  7.100 /  3.  7.100
[video4linux2,v4l2 @ 0x5b380] ioctl(VIDIOC_G_PARM): Inappropriate ioctl for device
[video4linux2,v4l2 @ 0x5b380] Time per frame unknown
Input #0, video4linux2,v4l2, from '/dev/video0':
  Duration: N/A, start: 117.711179, bitrate: N/A
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 640x480, 15 tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_omx))
Press [q] to stop, [?] for help
[h264_omx @ 0x5f2e0] libOMX_Core.so not found
 Leaving OMX_GetComponentsOfRole
[h264_omx @ 0x5f2e0] Using OMX.allwinner.video.encoder.avc
debug  : omx_venc <__AwOmxVencComponentCreate:2501>: COMPONENT_CREATE
debug  : cedarc <AwOmxComponentInit:26>:OMXCORE: aw_omx_component_init 67098

debug  : omx_venc <__AwOmxVencInit:392>: COMPONENT_INIT
debug  : ionAlloc <__GetIonMemOpsS:985>:*** get __GetIonMemOpsS ***
debug  : ionAlloc <ion_alloc_open:134>:begin ion_alloc_open

debug  : cedarc <VeSetSpeed:1559>: *** set ve freq to 360 Mhz ***
debug  : cedarc <VeInitialize:1198>: ve init ok

debug  : ionAlloc <ion_alloc_open:175>:** phy offset = 40000000
debug  : cedarc <VeRelease:1253>: ve release ok

debug  : cedarc <CdcMessageQueueCreate:47>:nMessageSize = 20
debug  : cedarc <CdcMessageQueueCreate:47>:nMessageSize = 20
debug  : cedarc <updateOmxDebugFlag:238>:DEBUG_TYPE is (null).
debug  : cedarc <AwOmxComponentSetCallbacks:309>:OMXCORE: aw_omx_component_set_callbacks 67098, b6233a2c , 5f690

debug  : omx_venc <__AwOmxVencSetCallbacks:2317>:===== vdec set callbacks***************
debug  : omx_venc <__AwOmxVencGetParameter:953>:InPort definition: width = 0, height = 0
debug  : omx_venc <__AwOmxVencGetParameter:965>:OutPort definition: width = 176, height = 144
debug  : omx_venc <__AwOmxVencSetParameter:1335>:init_input_port: stride = 640, width = 640, height = 480
debug  : omx_venc <__AwOmxVencSetParameter:1341>:set video port format 0x13
debug  : omx_venc <__AwOmxVencGetParameter:953>:InPort definition: width = 640, height = 480
debug  : omx_venc <__AwOmxVencGetParameter:965>:OutPort definition: width = 176, height = 144
debug  : omx_venc <__AwOmxVencSetParameter:1405>:init_output_port: framerate: 983040(>>16) 15, output width 640 height 480.
debug  : omx_venc <__AwOmxVencGetParameter:965>:OutPort definition: width = 640, height = 480
debug  : omx_venc <__AwOmxVencAllocateBuffer:1963>:allocate buffer, nPortIndex: InPort, nSizeBytes: 460800
debug  : omx_venc <__AwOmxVencAllocateBuffer:1963>:allocate buffer, nPortIndex: InPort, nSizeBytes: 460800
debug  : omx_venc <__AwOmxVencAllocateBuffer:1963>:allocate buffer, nPortIndex: OutPort, nSizeBytes: 460800
debug  : omx_venc <__AwOmxVencAllocateBuffer:1963>:allocate buffer, nPortIndex: OutPort, nSizeBytes: 460800
debug  : omx_venc <__AwOmxVencAllocateBuffer:1963>:allocate buffer, nPortIndex: OutPort, nSizeBytes: 460800
debug  : omx_venc <__AwOmxVencAllocateBuffer:1963>:allocate buffer, nPortIndex: OutPort, nSizeBytes: 460800
debug  : omx_venc <processThreadCommand:3317>:x set state command, cmd = main_thread_cmd_set_state, pCmdData = OMX_StateIdle.
debug  : omx_venc <processThreadCommand:3317>:x set state command, cmd = main_thread_cmd_set_state, pCmdData = OMX_StateExecuting.
debug  : omx_venc <ComponentVencThread:4316>:(vdrvThread receive cmd[venc_cmd_open]
debug  : cedarc <LogVersionInfo:40>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cedar Codec <<<<<<<<<<<<<<<<<<<<<<<<<<<<
tag   : createBranch_customize_sylix
branch: CedarC-v1.1.9
commit: 4a182bd42c931bea2f77b0ff9a531d233df8541e
date  : Thu May 25 15:34:03 2017 +0800
author: jenkins8080
patch :
----------------------------------------------------------------------

debug  : cedarc <VeSetSpeed:1559>: *** set ve freq to 360 Mhz ***
debug  : cedarc <VeInitialize:1198>: ve init ok

debug  : ionAlloc <__GetIonMemOpsS:985>:*** get __GetIonMemOpsS ***
debug  : ionAlloc <ion_alloc_open:134>:begin ion_alloc_open

debug  : omx_venc <init_h264_param:2820>:profile-venc=66, profile-omx=1, frame_rate:983040, bit_rate:200000, idr:12, eColorFormat:00000013

debug  : omx_venc <init_h264_param:2946>:Enable VBR. QpMin: 20. QpMax: 45. Bitrate: 200000.
debug  : cedarc <determineVencColorFormat:167>:color format: VENC_PIXEL_YUV420P/I420
debug  : omx_venc <openVencDriver:3184>:omx_venc base_config info: src_wxh:640x480, dis_wxh:640x480, stride:640

debug  : ionAlloc <__GetIonMemOpsS:985>:*** get __GetIonMemOpsS ***
debug  : omx_venc <openVencDriver:3221>:AVC 4 byte interleave nalu Disable.
debug  : cedarc <VideoEncInit:195>: (f:VideoEncInit, l:195)
debug  : cedarc <VideoEncInit:210>: (f:VideoEncInit, l:210)
debug  : cedarc <BitStreamCreate:102>: BitStreamCreate OK

[petit-miner]

Yeah, I also have the same issue with an USB Webcam as Input. If you use ffmpegs builtin Test Video the Video engine produces encoded H.264 just fine. You can try for yourself:
ffmpeg -f lavfi -i testsrc=duration=2:size=1280x720:rate=30 testsrc.h264 -y

[niyazFattahov]

Yeah, I also have the same issue with an USB Webcam as Input. If you use ffmpegs builtin Test Video the Video engine produces encoded H.264 just fine. You can try for yourself: ffmpeg -f lavfi -i testsrc=duration=2:size=1280x720:rate=30 testsrc.h264 -y

This works for me:
ffmpeg -f v4l2 -r 30 -t 10 -s 640x480 -i /dev/video0 -pix_fmt yuv420p -c:v h264_omx output.avi -y

[gamelaster]

Hmm, I already abandoned idea of using their OMX driver, since it didn't worked both on FFMPEG but also GStreamer.
@niyazFattahov I will probably try it again with your ffmpeg configuration.

[niyazFattahov]

@petit-miner what board you use with s3 chip? I also wanna try it for video encoding tests. I cannot find any board with this chip

[gamelaster]

@niyazFattahov PineCube have S3, and I'm testing everything on it.

[petit-miner]

@petit-miner what board you use with s3 chip? I also wanna try it for video encoding tests. I cannot find any board with this chip

I'm testing the Video encoder on a custom V3s board.

[niyazFattahov]

Hmm, I already abandoned idea of using their OMX driver, since it didn't worked both on FFMPEG but also GStreamer. @niyazFattahov I will probably try it again with your ffmpeg configuration.

do you try to use cedrus codec with v4l2_m2m ?

[petit-miner]

Yeah, I also have the same issue with an USB Webcam as Input. If you use ffmpegs builtin Test Video the Video engine produces encoded H.264 just fine. You can try for yourself: ffmpeg -f lavfi -i testsrc=duration=2:size=1280x720:rate=30 testsrc.h264 -y

This works for me: ffmpeg -f v4l2 -r 30 -t 10 -s 640x480 -i /dev/video0 -pix_fmt yuv420p -c:v h264_omx output.avi -y

Yeah, this works for me too, but I'm not very happy about the performance. I can encode a rawvideo with 720p however not in realtime. I get like ~8 FPs I tried baseline / medium / High but no difference in speed. The produced H264 files look fine and sharp and without any glitches. H264_omx hasn't much options to tweak:
Encoder h264_omx [OpenMAX IL H.264 video encoder]:
General capabilities: delay
Threading capabilities: none
Supported pixel formats: yuv420p
h264_omx AVOptions:
-omx_libname ED.V.... OpenMAX library name
-omx_libprefix ED.V.... OpenMAX library prefix
-zerocopy E..V.... Try to avoid copying input frames if possible (from 0 to 1) (default 0)
-profile E..V.... Set the encoding profile (from -99 to 100) (default -99)
baseline E..V....
main E..V....
high E..V....
Does anyone get any better performance?

[gamelaster]

@petit-miner there might be some problem in OMX driver. So far, when using raw CedarC library, the encoding is quite fast, but I seen some weird latencies too when testing. I need to finish my gstreamer plugin for libcedarc and that will show if it's capable of 720p live stream

[petit-miner]

okay, good to know. Where can I find this raw CedarC library? And how can I build it? I just need to capture 720p @ 30fps and encode it to H.264.

[gamelaster]

@petit-miner it's this repository 😊 OMX library is just additional wrapper for OMX support, which uses CedarC API

[petit-miner]

@petit-miner it's this repository 😊 OMX library is just additional wrapper for OMX support, which uses CedarC API

I know this is maybe a bit much to ask, but could you help me to compile the encoderdemo? I used these commands to compile:
./bootstrap
./configure --host=arm-linux-gnueabihf --enable-static
make ARCH=arm
Making all in memory
make[2]: Entering directory '/home/mthuerm/libcedarc/memory'
CC libMemAdapter_la-memoryAdapter.lo
CC libMemAdapter_la-ionAlloc.lo
CCLD libMemAdapter.la
/usr/bin/ld: cannot find -lVE
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:441: libMemAdapter.la] Error 1
make[2]: Leaving directory '/home/mthuerm/libcedarc/memory'
make[1]: *** [Makefile:398: all-recursive] Error 1
make[1]: Leaving directory '/home/mthuerm/libcedarc'
make: *** [Makefile:330: all] Error 2

Furthermore it seems like the Makefile in /demo/vencoderDemo/ doesn't get updated to my build environment.
It would be very nice if you could help me.

[gamelaster]

@petit-miner Do you have libVE.so in /usr/local/lib or /usr/lib directory?

[petit-miner]

Yeah I got that fixed now. If I use the encoderdemo in this repo the encoded video is super fast. I use a 17 second raw yuv420p video and convert it to H264 / MJPEG and I only get a video which is only a handful of frames long. Colours look fine in those few frames.
The same happens with ffmpeg: (I also found that the bottleneck with ffmpeg was the generation of the testsrc video and not the encoder)
ffmpeg -f lavfi -i testsrc=duration=10:size=1280x720:rate=30 -c:v rawvideo -pix_fmt yuv420p test.yuv
and then pass it trough the encoder with ffmpeg:
ffmpeg -re -f rawvideo -pix_fmt yuv420p -s:v 1280x720 -r 30 -i test.yuv output.h264

The video also seems speed up like crazy and is much shorter as my raw yuv420p testfile. I also verified that the testfile looks ok. I tried adding -re to ffmpeg to slow it down somewhat but it didn't help at all.
Funny enough when we overloaded the CPU with generating the testsrc video directly, the encoded video file looked fine and I didn't have such issues. Maybe it has something to do with the fact that the encoder only received 14 frames per second when the CPU was bottlenecked?
Do you have an idea?

[gamelaster]

@petit-miner about using encoding test, when you play raw .h264, there isn't information about frame rate, thus VLC / any media player picks how fast it will playback. About other things, I need to test it, but sadly, many things on my task list right now, but eventually I will get on it and will let you know about results, but mostly, the performance and quality was good.