[Request]: Could You Add overlay support for FFMPEG on AMF or VAAPI?

[alessblaze]

Is your feature request related to a problem? Please describe.
So i might want to transcode Adaptive Streams and Overlay insertion in stream is very common
Describe the solution you'd like
it would be very nice to have a way to do hardware decode overlay and output so i wont have to do much cpu work on this.
Describe alternatives you've considered
alessblaze/FFmpeg@36979e2 something like this but i am incapable of getting it fully right for all sort of streams.

Additional context
It would be very nice to have atleast overlay support for the GPUs.

[MikhailAMD]

I assume you want this for Linux. It is possible to make overlay compositor entirely on AMF compute interface but FFmpeg already has ff_vf_overlay_vulkan. There are currently issues with interop between AMF and Vulkan but they will be resolved. Would ff_vf_overlay_vulkan work for you?

[alessblaze]

i could not abstract out completely to be honest, i did try to use that but there is specific issue in hardware decoder which for some type of videos show green output at bottom which i tracked down to the hardware decode itself. we are using vulkan here, i am not very sure if i am using OS dependant code as it relies on amf but yea which was why i had to keep testing and finally i got some good output with a transport stream. but i did try using ffmpeg overlay. i could have had abstracted it out with ffmpeg provided methods but there seems to be some bad issues on the hardware decoder itself. even in vaapi scale downscaling needed to be some divisible by 32 in order to not have green bar on right. so i believe whatever technical implementation you might want to do its fine as long as we can access vulkan compute currently so other things or custom code could work. i wasnt able to load a kernel by the amf source with opencl syntax. neither source nor binary both ended up in segfault but i have very less experience in this hardware so it would be fine as long as it works and gives more than 1.5x output speed at 4k50.

[alessblaze]

pure vulkan paths have half of not implemented errors on vanilla builds.

[nyanmisaka]

Mesa 26.2 already supports vf_overlay_vaapi. https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/41090

[alessblaze]

not implemented in amd driver i believe, have you tried ? can't build whole rocm driver stack for this.

[nyanmisaka]

not implemented in amd driver i believe, have you tried ? can't build whole rocm driver stack for this.

Certainly. I helped them implement this. I backported and built minimal radeonsi and radv drivers to use it.
https://github.com/jellyfin/jellyfin-ffmpeg/blob/11665b360f1e2ced2f851e018ed611ddfd5cbeb5/docker-build.sh#L495

[alessblaze]

madam, kindly stay off, here we are talking about how we can abstract my amd pipeline, i appriceiate your contribution, but it doesn't solve how each hardware provider implements it, i can write your implementation in one day but i appreciate your vision, i am talking to the AMD representative of how we can make it better okay? i am sure you may have done great, but kindly open the ticket yourself on your own isssue, thank you.

[alessblaze]

do not want to be ungrateful to be honest, it is hard i know, kindly come with some poc, nvidia or amd is not your sibling neither US is. i know there can be endless abstractions. but does the hardware supports it? thats your problem. talk after you yourself is able to achieve a+b=csquared
i am sorry. you just needed to stay quiet to a senior engineer talking, if not, do it yourself. i write 10 standards a day but would the provider listen to me . no.. thats how it it is. but i believe amd will solve it sooner or later. till then sweetie do your own thing.

[alessblaze]

https://digital.alessmicro.com/posts/blog-delivering-live-tv-over-http-trial-runs-of-broadcast-technologies i have done lower latencies than this not with the help of you or your 2030-plz implement my standard thing. id do it myself so stay off. we aint doing it on someone else's project. we have everything that plex has, so leave off. we starting off. so i personally am not ungrateful of what helped me hereby amd. i have bunches of amd rocm ports for ai for right after . from pytorch to converting 7b parameter models with 8 gb vram. https://huggingface.co/aless2212 here. do not give your opinion. i know how things work. i want to know when the company wants do it. i can get 20 nvidia working cards okay? stay with your streaming. iam not mad just dissapointed.

[MikhailAMD]

Here are three different but related topics:

1. Nyanmisaka and myself point to exiting solutions and asking why they do not work. Usually there is little value to repeat existing code.
2. Composer / overlay on AMF Compute API would make sense in a stand-alone application. Having it in FFmpeg, IMHO, has little value, better to improve existing filters.
3. Regarding green bottom lines: they usually happen when processing code (overlay) doesn't take into account allocation alignment. All HW decoders allocate output images/textures/surfaces aligned by 16 or 32, width and height, due to HW optimizations. Software frameworks like FFmpeg or AMF carry crop information that needs to be used in processing: for example, in shaders. You didn't include shader source code in the commit but you would need to pass crop information and take it into account. In AMF API cropping information can be acquired from AMFPlane interface GetOffsetX, GetOffsetY, GetWidth, GetHeight. FFmpeg's AVFrame has it in width, height.

[alessblaze]

do you think , i do not knot know?

[alessblaze]

you sure you want this right? a random woman writing a {a+1} and and a {+2 } nvidia. if she was so good she she should have had her own. but nvm. i understand. i will shift the whole pipeline to nvidia. so i wont need to listen to this while you ask your senior engineers and that Chinese girl about what is wrong. i dont give a shit,. i am not your friend.

[alessblaze]

i am writing my own issue. i do not want anyone else talking. its as simple as it sounds.

[alessblaze]

https://www.wowza.com/docs/how-to-add-graphic-overlays-to-live-streams-with-wowza-transcoder
https://antmedia.io/marketplace/stamp-app/
https://techdocs.akamai.com/ivm/docs/add-logo-watermark
https://docs.aws.amazon.com/mediaconvert/latest/ug/video-overlays.html .. huge value in broadcasting, in dvb alike broadcast, the overlays are done in broadcasters encoder but the carrier marks like dvb operator, i do have images to show already but considering existing copyrights i cannot show someone's logo here. but its very common. do check nvidia side code once in ffmpeg.