On mine it was playing with some /sys/bus/usb/devices/1-4/power/wakeup or something, (use the right device number from lsusb), also there is some settings in the BIOS to power or not USB when sleeping, obviously if some USB port have no power, wakeup cannot work :) Every setup/BIOS is kinda different though.
And I have had the same problem as OP with just knocking the desk and the computer wakes up and it is annoying!
My latest trackball I switched it to Bluetooth mode so when PC is sleeping, no BT at all.
What system do you have? Laptop or desktop, if laptop which one, if desktop which mobo do you have? Check your bios for any sleep options, specifically S0 vs S3 standby.
S0 standby vs S3 standby can change things a lot. If you're in S3 standby then it's up to your bios/firmware to handle waking from sleep. If you're in S0 standby then your OS is in control.
Yeah this is a hardware thing. If there's no bios setting you might just be able to switch your mouse to a different usb port, most motherboards can only wake from 2 of them
I've been working on a fix for a similar issue, but it might not apply because my issue is caused by my mouse's wireless reciever. I wrote a little script to replace the bluetooth wakup config file, and it works when I manually run it. However, this resets every time I restart and I'm having difficulty getting it to run on startup, probably because it requires sudo.