Their ideas are as interesting as they are frightening

I think the screams would be muffled but still audible. Until the bag is closed, the extradimensional space inside is connected to the outside world. However, since sound can only escape from the opening and not the sides of the bag, I would rule that it is much quieter, granting disadvantage on checks to hear the scream.

This is an eminently reasonable ruling and I's appreciates yous
It's up to the DM if it comes across like a portal to enter the bag or an impossible space like the TARDIS. Air does not flow between as you can suffocate and air would carry the sound, but I always rule it as feeling more like in impossibly large space rather than a magical portal.
- Idk, that feels a bit too technical. Trying to apply real world physics to D&D breaks way more things than just this, so we gotta be careful when to do it. Gotta keep just enough believability and consistency, without letting it ruin the feel.
  
  For me, the fact that "starwars lasers go pew pew" is enough reason to disregard how sound actually works in real life, lol.
- I always interpreted the suffocation to be only while the bag is closed. It doesn't make much sense if physical objects can enter the bag but not air. Why not?

You could just kill the guard by leaving them in the bag

I would immediately shift your alignment to chaotic evil for it because you're literally making them suffocate to death over a period of about ten minutes, but it's still RAW.

Except if the guard manages to rip a hole in the bag from the inside, they can destroy the bag and everything inside it will get scattered across the Astral Plane. Pretty bad for the guard, but better than a slow death being suffocated. Plus you don't need to eat or breathe on the Astral, so they could live there indefinitely.
- Tearing the bag from the inside seems like an easier said than done proposition.
Would a person die in a bag of holding or is it like the Astral plane and bodily needs are paused?
- Yes. It's specifies there's ten minutes of air. But the bag woodt need to be closed to not have air exchange.
- I've always imagined a bag of holding to be like some kind of stasis.
Bag of holding around the head. Tear it.
- Faster in theory but I'd be incredibly tempted to have it create an astral plane flavored headless horseman enemy
This sort of thing always seem silly to me. Most of the people you kill go to the lower planes. What is ten minutes of suffocation compared to that?
- Bro if you don't get how painful suffocation can be just sympathetically I'm not sure I can explain why doing that to someone on purpose is so heinous

If it "holds" the screams as if it was an item, would you be able to pull a scream out of the bag at will? Does than then mean that you can use the bag to prepare verbal components of spells?

Anything you can do, the DM can do, too. Treat things like this like war crimes or your next TPK will be a goblin horde taking out the party one by one with sneak attacks and bags of holding.

If you think that you are the players enemy instead of guide maybe.
- I understand that it's not a competition or a win/lose kind of game. Personally, I use TRRPGs as story engines for my make-believe characters, but if my players decide to throw jank at me, I have permission to throw jank at them.
That seems like a good idea for parties that constantly exploit everything they can. One off creativity should probably be rewarded though.
Honestly if you wanna give your party a bag of holding, sticking it on a bugbear assassin who uses this tactic and then uses the bag to dispose of bodies is a fun way to introduce it.

Anything the players do, I be DM can do too, yt if it uses a niche magic item, the DM better be prepared for it to end up in the hands of the players.
- So true, and your players will cherish it for the sentiment of the victory.

It would silence as many screams as hands you are loosing pulling items from it. Which is zero.

The Dungeons and Daddies guys took out a CR26 Vampire Lord at Level 2 with this one neat trick...

I would say yes, it's a poket dimension with no air so the sound would have no medium to flow through. Thank you for coming to my Ted talk

Seriously, super fun info for later!

This is funny, I see your point but would rule the opposite. Because the bag wouldn't be able to close due to a neck being in the way it wouldn't seal the pocket dimension and I would rule the guard would still be able to breath (and scream) through the bag.
- I do believe you are correct, thank you for correcting me good sir
It has enough air for one creature to breath for 10 minutes. And since the bag isn't closed when it's on the victim's head there is a connection between the pocket dimension and the material world through which sound waves may travel.
- I do believe you are correct, thank you for correcting me good sir
If there’s no air, wouldn’t it ruin a bunch of stuff you put in? Any liquids would evaporate, wood could deteriorate, etc. Is it also cold? That also causes changes to a bunch of stuff.

When you put his head in the bag you are also putting the air around his head in the bag so any "sound doesn't travel in a vacuum" arguments will be nonsense.

I imagine it would be just as loud as if someone poked their head out of a porthole of a ship and yelled outside. Someone inside the room of the ship could still hear it even though the victims head is poking through the porthole.

I thought it wouldn't make sense at first, but thinking about it more I think it does. Sound can't travel through the walls of the bug. It probably echos inside it, but there's not a lot of air around their neck for it to come out through, and sound doesn't travel well between gases and solids, so most of the sound would be absorbed. It wouldn't be perfect, but it would make it a lot quieter.