https://snikket.org/ (xmpp based) is perfect for that. Matrix will work, but you will likely reach the limits of your Raspi with it fairly soon if you allow federation with other servers.
E2E is complicated, if you self-host for a group, having TLS and encrypting data at rest (storage) may be enough. Get a threat model. That being said, I would recommend snikket.org which is a superset of extensions over XMPP which is the open source IM that was the base of almost every app out there. Matrix and Rocket are both alright too. Depends too on your resources, synapse requires too much RAM (or so I heard)
I know it's not self hosted but why not Signal? Matrix is demanding on a SBC and your family would probably get the 'unable to decrypt message, please re-verify keys' error that happens in encrypted matrix group chats and Element does not have the best UI especially if you want your grandma to use it.
I recommend Matrix with the Conduit server. This server requires almost no resources and even runs on a Raspberry Pi.
Cinny works perfectly as a desktop client (in case you want to escape from the ubiquitous Element). And for mobile I would use Element for Android/iOS although FluffyChat also works very well.
For me you can try to host a SimpleX server and then connect to it (with SimpleX it is pretty as much secure to run its one server than use a public one).
Or maybe use XMPP but try to use a good encryption protocol. This option is great in term of power efficiency, XMPP would run great on a RasPI
You could try Jami. It's peer to peer, so essentially any participants are self-hosting it. Its E2E encrypted, supports group messaging, voice and video calling, has easily 'linkable' mobile and desktop apps for all platforms and requires no email address or phone number to use. It's also the only messenger I'm aware of which is endorsed by the Free Software Foundation. I highly recommend it 👌
However, I wouldn't host on a Raspberry Pi or even at home for that matter. Get a VPS and host it there. Linode even has a one click app install for Simplex Chat
Self host beeper... I love matrix but people struggle with maintaining a key and password .. beeper simplifies this and has other chat protocols (bridges) that might entice normies.