I went out and got the AMD 7800xt to do ultrafast AV1 encoding of large h264 and hevc files.
I am able to select VCN acceleration options in Windows but not Linux Handbrake.
I have tried the flatpak and both the Pop_OS and ppa deb packages of Handbrake. I have the latest mesa driver and am running the most current version of PopOS 22.04. I've had no issues gaming at all, just with picking the hardware encoder in Handbrake. Any ideas or rabbit holes I can go down?
Check which version of ffmpeg it's using and whether it has hardware acceleration for that codec.
Also bear in mind that you can't hardware decode and hardware encode at the same time on the same device. If that's what you're trying to do it's probably falling back on software silently in Windows instead of telling you.
Because one process will be running on the hardware, because of the way it works it can't really share that hardware between processes. I'm not sure if that's entirely a hardware limitation, but it seems to be enough of one that software hasn't overcome it.
You need to use amdgpu-pro, according to the page here. I tried installing the pro drivers on Linux Mint a few weeks ago and still couldn't get hardware acceleration in Handbrake to work, but it might be worth a shot.