Lemmy.world (and some others) were hacked

Very impressed by how quickly action has been taken by this and other instances to patch the issue.

I wish hackers would invest their time in clearing credit card debt, deleting hospital fees, or something else that actually serves the public good, instead of hacking ordinary people just trying to get by.

Thanks Ruud for fixing it! Just a reminder guys that If you are using a third party app you need to login again.

what steps are being taken to ensure it doesn't happen again? was any personal data compromised for users?

IMPORTANT ANNOUNCEMENT: My account was not among those hacked. Any random bullshit appearing in my post/comment history was written by me.

First - really good summary and sounds like everyone is working hard.

Cross posting the below comment.

Under GDPR if you have had a data breach you have a legal obligation to assess whether you need to report it and you must make the report within 72 hours of discovering the breach.

There are other types of reportable breaches too, I only mention data as it sounds most likely. You may or may not be subject to PECR which may also have been breached although less likely. I don’t really have enough familiarity with the regulation to discuss that one.

If you are not sure if there has been a breach you may also need to discuss it with the relevant body or make a report.

Please can you update what action you have taken regarding this and if the incident was reportable or not and the reasons why. Edit - from that new information, it sounds like this is a reportable breach.

For a full understanding, it would be good to know if you had 2FA enabled on the compromised account particularly as it had admin privileges and if so how 2FA was circumvented with this exploit.

It would also be good to know what measures you have in place to prevent the same or other malicious attempts on your Open Collective and Patreon accounts as issues with those are potentially more serious. They may not be vulnerable to this, but it is going to be reassuring to know there is good security practice, 2FA protection etc enabled and you have robust procedures in place.

Thanks for letting us know - this is the kind of transparency that I wish the world had more of!

So what happened:

• Someone posted a post.
• The post contained some instruction to display custom emoji.
• So far so good.
• There is a bug in JavaScript (TypeScript) that runs on client's machine (arbitrary code execution?).
• The attacker leveraged the bug to grab victim's JWT (cookie) when the victim visited the page with that post.
• The attacker used the grabbed JWTs to log-in as victim (some of them were admins) and do bad stuff on the server.

Am I right?

I'm old-school developer/programmer and it seems that web is peace of sheet. Basic security stuff violated:

• User provided content (post using custom emojis) caused havoc when processing (doesn't matter if on server or on client). This is lack of sanitization of user-provided-data.
• JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS. In case of log-in, in HTTPS POST request and in case of response of successful log-in, in HTTPS POST response. Then, in case of requesting web page, again, it should be handled in HTTPS GET request. This is lack of using least permissions as possible, JS should not have access to cookies.
• How the attacker got those JWTs? JavaScript sent them to him? Web browser sent them to him when requesting resources form his server? This is lack of site isolation, one web page should not have access to other domains, requesting data form them or sending data to them.
• The attacker logged-in as admin and caused havoc. Again, this should not be possible, admins should have normal level of access to the site, exactly the same as normal users do. Then, if they want to administer something, they should log-in using separate username + password into separate log-in form and display completely different web page, not allowing them to do the actions normal users can do. You know, separate UI/applications for users and for admins.

Am I right? Correct me if I'm wrong.

Again, web is peace of sheet. This would never happen in desktop/server application. Any of the bullet points above would prevent this from happening. Even if the previous bullet point failed to do its job. Am I too naïve? Maybe.

Marek.

Damn, I go to bed early and I miss everything! Thanks for the quick resolution and transparent disclosure, this place is great!

Thank you for your work 🙏

This is really good to see such transparency from admins

Love the transparency!

Good thing we all use randomly generated passwords for every account and always remember to change them every few months.

Can we get another admin to sign off on this being authentic? In other words, short of a signed GPG signature how do we trust announcements after a breach where admin accounts are compromised?

So thats why MalwareBytes gave me this message yesterday.

Thanks for the work. As a heads up it appears most of the block instances are back however I believe explodingheads is still missing which you may want to confirm.

EDIT: it has been added back to the block list.

the details of the vulnerability are already known now anyway since there's a fix that was proposed on the Lemmy GitHub so I don't think it will hurt others to talk about it

FYI: I had to clear my lemmy.world cookies in order to be able to successfully log back in.

(This was with Firefox)

(Edit: I also shift-clicked reload, which somebody pointed out does clean the cache for that page, so I also cleaned the cache).

Thank the heavens the meme community stayed safe through this without my daily dose of cybersecurity memes idk how I would function ;)

How do we know that this isn't a fake announcement as well, trying to give us a sense of security???

Just kidding, thanks for letting us know! Thank god I haven't been too active the last few days! Can't afford my credentials being leaked, maybe I should be proactive and change my password anyways.

Had to clear my browser catch to log in, Jerboa still shows as not logged in even after logging out which you do by clicking the hamburger menu then click the top banner to change/log out of accounts. This post is a test to see if my account works again via browser lol.

Edit: clearing app data/cache for Jerboa fixed the login issue.

Do we have any details on how Michelle's account was compromised? Right now in the GitHub issue about the vulnerability they're clueless about how the custom emoji exploit could be performed without first an already compromised admin account.

EDIT: yeah here's how: https://github.com/LemmyNet/lemmy-ui/issues/1895#issuecomment-1629326627

You do NOT need an admin account to do that. Any normal user could have done that.

Hopefully with more attention on the source code scary hacks like this doesn’t happen again.

Took me a bit to realize I actually had to log out and log back in on Jerboa since it looked like I was still logged in but some interactions didn't work

One wonders how much spez paid for that hack.

You guys really have my highest respect for spending so much time to keep this running, despite all the recent trouble and now even an attack.

Thank you very much <3 You guys are awesome and I really appreciate how publicly you deal with this.

Can I ask some possibly dumb questions?

1. What is JWT?
2. Was any private user data compromised, and if so will users be informed?
3. Is there anything regular users can do to avoid their data being compromised? For example, not accessing lemmy on certain web browsers?

Thank you!

That was scary and exciting. Response seems competent and transparent. I ❤️ this place.

I think this is a strong reminder: We shouldn't put all our eggs in one basket. This will happen again. Unlike Reddit, we don't need to concentrate all communities on one instance. We should all make an effort to spread out. Some other general use instances are:

• https://lemm.ee
• https://lemmy.one
• https://sh.itjust.works
• https://sopuli.xyz

Again, for those new, you can post content to any of these instances and interact with content from other instances at the same time, just like you can send an email from your Gmail account to your ProtonMail account.

Well done all involved. Sounds like it was caught and mitigated quickly

So, do we change passwords, esp those who logged on during the attack? (I created this acct right before the attack happened tho.)

Despite the fact that Lemmy is a fairly new piece of software, which makes these issues more likely, I am really grateful for it being open source, and I really appreciate this level of transparency.

Thank you for the transparency and swift solution!

Any truth to what I've heard this may have been done by a group we defederated with?

Thanks for fixing and being so open about it

Good job. I don't understand very much of that, so that makes me all the more grateful. Thank you.

I just disabled whole "/admin" section on my instance and added nice message 😆

How does this impact those using mobile apps like Jerboa or Liftoff, instead of the website directly?

Permanently Deleted

At least now we can mark off the "disruptive website defacement attack" line on the checklist of (relatively) new website growing pains. Better to have them make lots of noise and get fixed quickly than quietly do sneaky things in the background.

The quick fix is much appreciated, thank you and everyone that helped for your hard work!

Thanks for your efforts. I know that Lemmy was put in place rather quickly as a Reddit alternative. But I'm genuinely hopeful that this will be a good alternative.

Thanks for the transparancy about this.

On Liftoff, I had to clear cache and storage in order to log back in. Still having issues with the website on Chrome, which keeps telling me I'm not logged in after clearing cache and logging back in.

Thanks for the transparency and the insights / recommendations, good behaviour! Is this the reason why my Memmy doesn't show anything?

Could admins sign announcements with a PGP key to mitigate false admin posts and the consequences this might have? Or is this no longer necessary?

One thing I don't get. Custom emojis can only be created by an admin, but you're saying an admin's account here got compromised because of that and not the other way around. Does that mean that an evil instance set a custom emoji with the injected JavaScript and propagated it to the federated instances?

Does an admin account have any permissions to view email addresses or data of registered users?

Did MichelleG not have 2FA enabled?

Now that this has happened, it's be worth pushing this issue through as high priority. If HttpOnly was enabled, then an admin takeover would not have been possible.

https://github.com/LemmyNet/lemmy-ui/issues/1252

That doesn't surprise me. Especially the "homemade" instances. The documentation is severely lacking and I had to fix lots of stuff in the instructions with try&despair to make my instance run.

There's not a great focus in security if your application starts with "step 1: install docker"

Permanently Deleted

Ugh, people should not go after systems trying to give a free service to the internet. It just ruins everything.

Thanks for the great work. The response time was awesome, considering you were asleep as well.

Thanks for the transparency.

How do we know you're the real you? This all could be part of the plan!

Great lemmies! Thanks for uniting us.

Soo it looks like the entry for this instance was also changed on https://lemmyverse.net/ . At least I hope it is the hack

Yah, I noticed my Lemmies auto-corrupted to Lemurs.

I don't care. I'm keeping it.

Lemurs are cute.

I had to create a new account. I tried enabling 2FA on my main account a week ago, but was never able to generate a token. Now when I try logging in it is asking for my 2FA token. Is there any way to get my account back. I'm a moderator of a community.

Once again, thank you guys for all that you do. As many other people are saying, appreciate the transparency about these things.

Permanently Deleted

I can only log in on incognito mode, which makes me think my cookie has been stolen or whatever. So my question is, what should I be doing about that?

Amazing how you quickly reacted to this!! Bravo!!

TIP: if you can't login after what happened, clear out your browser cache including ALL cookies, that fixes it (it did for me at least). I believe it's also advisable to change lemmy password.

This is why I've decided against running my own Lemmy instance. Too much work to have to keep up constantly with updating, too big of an attractive target for attackers.

Interesting.

Attackers started changing site settings and posting fake announcements etc

So at least that wasn't 100% malicious, otherwise they could've kept the vuln hidden and just collect data and whatnot.

On the other hand, who cared enough about Lemmy to hack it? Weird.

Pardon the ignorance, but how do I know if I was compromised? what do?

As someone in EU I didn't even realized there was an issue. Well done and great reaction time! Also thank you for the transparency 👑

I can't log into my account anymore, this one is a new one I've just made. I tried to reset my password but nothing came in the mailbox. I can still see comments and posts from that account though.

It's this one:

And I don't know why but I can't save the profile pic for this account.

Edit: Nvm, I use another email to sign up for Lemmy and forgot about it

This is so sad lmao rip. With any site growing as fast as these instances (because of the Reddit folk) Ig these attacks are to be expected. Hope everyone's accounts and personal info are okay

Rock on, Rudd.

With the JWT secret rotation, shouldn't everyone be forced to re-login? I'm posting with my existing session without any changes.

If I'm not from lemmy.world and visited a lemmy.world community via my home instance, does the hacker gain access to my account?

If I, while logged in to my home instance, accessed lemmy.world in another tab, does the hacker gain access to my account?

Does this hack infect devices used to access lemmy.world?

Sorry for noob questions, I'm just worried.

Thanks for the transparency. Was having issues with Lemmy, now seems everything back to normal. Got a question, Just to add an extra layer of security, Do i need to use ToR or VPN with Lemmy ?

Must have been jealous spez

I was wondering what’s going on. I was trying to read the post about Elon and Mr Beast and when I clicked on the link to open the comments, I got redirected to the lemon party nonsense. Not the Elon Musk shenanigans I expected. Weirdly, I could open the direct link to my profile, but not the main lemmy.world page

Well that's just great it really is a shame though how some people would actively want to ruin something free like this just because they can.

Permanently Deleted

It seems there is no way in Lemmy to invalidate all your session cookies? Without that, how can you secure an account which has a stolen session cookie?

Thank you for taking the time to update this :) Hope everything will be sorted out without people being scared. As a layman, was any user data compromised?

Excellent, thanks for the quick response ruud and admins.

I noticed this morning for a small amount of my posts with pictures, maybe 5-10%, the pictures were deleted or missing. Not sure if this is related to the incident.

Here's a relevant post that talked about this with @AlmightySnoo@lemmy.world I think is worth looking into for anyone curious what exactly happened.

https://sh.itjust.works/post/923025

please don't visit the legal section of the website or anything confirmed compromised if anything.

I wasn't using webpage, I was only using mobile app (Connect). Could my coockie be also stolen that way or was that only possible on webpage?

Thanks all working again. Had to clear my browser cache in order to login again and had to resign in to memmy too.

I guess its early days for lemmy for incidents like this, fingers crossed something like this doesn't happen again :)

Is it possible cookies for other websites were scraped? I was logged in to .world at the time; I have logged out of all accounts, and reset passwords as a precaution, but want to know if I should be on the lookout from this.

Poor beehaw.org is still down

I found this in my private messages, when an attack was happening I messaged the guy “are you ok” and he replied back to me with an image of my own message… I wonder if this was similar to what was done here? Was 8 days ago

What are the risks for people who use Jerboa for Lemmy? I logged put and back in and there doesn't seem to be any issues, so are the app users excluded from this?

A lot of images seems to be gone from posts in /c/pics is this related to the hack or the cleanup after?

Would it be a good idea to force a login if the users IP or device suddenly changes?

Thanks for the info, Ruud. I just put in for a monthly donation to you all -- I appreciate you.

TY to everyone itt who commented on how to fix the 3rd party app issues.

I was panicking when liftoff went wonky

I had an issue of being logged out of my account and could not log back in, after closing and reopening the site, closing browser, etc until I cleared my cookies, then it let me back in. If that helps anyone.

Congratulations everyone on the quick fix/mitigation!

Had to re-login in the Connect app

Testing… I have to keep trying this because memmy is being a dick.

Not too promising, hopefully it's fixed quickly.

That explains why I had to clear my browser cache, I was unable to login until I did.

I’ve been unable to login on desktop since this happened. Only been able to login via Memmy on IOS.

I put in my info and it kicks me back to the front page and doesn’t log me in.

I’ve tried clearing cache too

EDIT: Switching browser to Edge seemed to let me. Weird. Even reinstalled Firefox and still won't let me.

Seem to have a hard time loging in

I heard that some instances were defaced. Any examples of this? I wasn't online this noon so I never got to see any action.

Thanks for the quick reaction and TRANSPARENCY!!

Passwords were leaked?

I hope devs will examine all parts of the code that display content to make sure proper sanitization

Um, probably coincidence or a false posi, but malwarebytes is labeling lemmy.worldtoday as being compromised / malicious when following external links, it's only popped up twice, but here's a slightly redacted log file:

-Log Details- Protection Event Date: 7/10/23 Protection Event Time: 1:24 PM

-Software Information- Version: 4.5.33.272 Components Version: 1.0.2069 Update Package Version: 1.0.72209 License: Premium

-System Information- OS: Windows 11 (Build 22621.1928) CPU: x64 File System: NTFS User: System

-Blocked Website Details- Malicious Website: 1 , C:\Program Files\Google\Chrome Beta\Application\chrome.exe, Blocked, -1, -1, 0.0.0, ,

-Website Data- Category: Compromised Domain: lemmy.today

(end)

You guys are quick!

Thanks for the quick response! This admin team rules!

My main is locked up and it sucks, i hope this all gets fixed.

Hmm. Liftoff won't let me post but shows logged in and as a newbie be damned if I can find where to log out.

May I ask was is the JWS coockinand if it is automatically changed or if we have to change it in a way?

Thank you for your fast answer!

So that was why the logo and name was changed to israel. And for some reason getting redirected to a gif that was from lemmy

It seems that I lost all my subs. There were not many but still annoying.

E: Still subbed but can't see those in Voyager.

Should I change passwords or no?

Damnit, spez.

Is a password change advised? How does the JWT cookie and exploit effect apps eg Jerboa?

I appreciate the transparency. Hopefully with more eyes on the source code hacks like this will not happen again.

Thanks for the post-mortem and the quick fix! Glad you guys around to help battle test Lemmy's code.

Good shit! Thanks for keeping things up and the pretty quick response as well.

I’d like to logout, then log back in, because I can’t upvote / downvote- how do I logout? I can’t seem to find a logout button.

Occasional cookie deletions I understand, but will sign-ins persist in the future?

Is this why Jerboa seems to not work any more? It keeps insisting I'm not logged in, when I am, showing me as anonymous, but also showing my profile details, not letting me interact with things, etc... It's been a big problem these past few days making Lemmy unusable :-(

This seems way worse than they are making it sound.

I was unable to log in, it looped me & said I logged in, but did not. I read this post, cleared my cache, and I was able to log in (and change my password).

Because I am obsessed with bugles, any comment or post I make that does not manage to fit bugles in somewhere (because I always have room for bugles) will be an imposter!

Thank you for the transparency and keeping my nefarious bugle consumption private!

Maybe there needs to be a quick rundown how to actually log out and in on clients, seems you can't with jerboa without just wiping the app, and wefwef, you need to delete all accounts.

I see some instances are throwing server errors

thanks to admin team for resolving that quickly

How are you preventing it to happen again until a patch is released from devs?

Thanks.

I lost some of my post history. Is there a data issue that's come from this? Why are my comments gone?

Is there a rough time range when it happened? and any news about other big instances like lemmy.ml? Are those safe? Currently they are not on the same version as lemmy.world.

Is this why I can't log in on Chrome? I switched to Firefox and it worked.

Great job everyone! Keep it up! Love the transparency!

Thanks to everyone involved for the quick response 👍

I see you, Imposter.

Thanks for the update, I appreciate the transparency.

Well done on acting on it so quickly. I think I did see some of the fake announcements you were referring too but were taken down very quickly. Keep up the good work team and thanks for everything you are doing!

Is this why I had to sign in and out of my account on liftoff?

I couldn't comment untill I did that. There may be others!

******* This happened to me, one of my posts had it's photo deleted (I didn't delete it), then when I replaced it, the next time I checked the entire post had been deleted.

Thanks for fixing it.