I just used a live ubuntu image and it works out of the box with the same setup. No idea what's the problem with fedora. Unfortunately I'll move to ubuntu now.
Maybe you can find a guide/tutorial on how to set it up?
Usually you need the correct packages installed on your system to enable something like VAAPI or QSV. Then you need a version of ffmpeg with that enabled. And then configure it in Jellyfin correctly.
I don't have any specific insights on how to do it with Fedora. I suppose it's very similar to how it's done on other Linux distros.
You probably need to get codecs/drivers from RPMfusion to get it to work. Fedora doesn't include much by default. https://rpmfusion.org/Howto/Multimedia
Have you tried using the linuxserver.io Docker image? It has the latest drivers for hardware encoding included. I couldn't get HW encoding with the official image to work but this one worked without any manual setup. You still have to forward devices to the Docker container though.
Funny, i couldn't get HW encoding to work with the linuxserver.io docker image, but the exact same compose file, except it's using the official image, works just fine without any issues.
I had it in podman compose first. That didn't even return the proper error messages and just skipped them if there was any. I can't recommend it. It works on ubuntu.
you will need to make sure that jellyfin uses a version of ffmpeg that actually uses your graphics card - you might need to compile ffmpeg with the corresponding flags
I am running Plex with an Intel A40 in Ubuntu server. Worked well for me as Ubuntu had the drivers baked in before they made there way into a Debian release.
I know my setup with intel integrated gpu worked prior to the release pf 10.9. Now I can't get transcoding to work. In the comments they suggest the kernel version has something to do with it but for me it didn't fix it. I'll have to troubleshoot further today
Meanwhile transcoding works fine in Plex, so I feel it may be something specific to jellyfin