For complete fairness to everyone we should tax every vehicle on road or path from scooters and bikes to b-doubles based on the 4th power of axle load to properly account for the impact on road maintenance costs. Then additional levies for disproportionate environmental costs and harm to vulnerable road users. Keep the overall tax amount the same but shift the burden so people with smaller vehicles pay substantially less than they do now. And then add strict liability for anything much larger than a kei car.
Since the amount of damage caused by bikes to infrastructure would be small, the govt could just slap a tax for the cost to the purchase price of every bike. Not saying I'd agree with it but there are ways to make all road users pay their share.