YSK: Your Lemmy activities (e.g. downvotes) are far from private
Edit: obligatory explanation (thanks mods for squaring me away)...
What you see via the UI isn't "all that exists". Unlike Reddit, where everything is a black box, there are a lot more eyeballs who can see "under the hood". Any instance admin, proper or rogue, gets a ton of information that users won't normally see. The attached example demonstrates that while users will only see upvote/downvote tallies, admins can see who actually performed those actions.
Edit: To clarify, not just YOUR instance admin gets this info. This is ANY instance admin across the Fediverse.
You'd still need to not vote on anything for that to have an impact.
When you perform any action on the site that needs to be replicated to other instances (votes, comments, posts - and edits/deletions to these) the instance you make the action from sends to the "home" instance of that community some data that says "Hey, this person your-instance.tld/u/username did this $ACTION", and then that data is forwarded from the community's instance to all subscribers of that community (so that things like upvotes/downvotes display in sync... well, that is the intention).
If you're curious about the technical details of how that data looks, you can take a look at the ActivityStream specification (note that not every activity type on here is actually implemented by Lemmy, the link is to show the general structure of the data).