Lemmy.world (and some others) were hacked

Very impressed by how quickly action has been taken by this and other instances to patch the issue.

Very, seems like great work.
Hijacking the top comment to say I had problems with logging in to Lemmy.world today and liftoff was failing in odd ways.
I had to go into my web browser and clear my site cookies for lemmy.world to let me log in there.
In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)
- I’m on iOS with the Memmy app. It’s a work in progress that’s officially unfinished so I’m not surprised but it has also been a bit buggy. Doesn’t seem that I can log out without deleting and reinstalling the app so hopefully this doesn’t happen too often XD
- thanks for posting this, I wouldn't have figured that out lol
- In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)
  Good PSA. It took me a bit to figure it out, the app doesn't make this obvious.
- Oh, I was wondering why it was showing me as logged in but wouldn't let me upvote due to not being logged in. Your liftoff psa just cleared that right up for me, thanks!
- How have I never thought of comment hijacking?!
uh, why did you have negative one dislike?
- Negative one upvotes would mean that enough people disliked me/another poster to bring my upvote total to zero. (Upvotes and likes are effectively the same thing, it’s just a naming convention). Reddit totals them up and seemingly Lemmy does as well.

I wish hackers would invest their time in clearing credit card debt, deleting hospital fees, or something else that actually serves the public good, instead of hacking ordinary people just trying to get by.

- As I pointed out in the thread it was probably a few Lemmy users themselves that did it.
- I thought we weren’t going to be Reddit-brained over here
- Nah. There's far too much risk for Reddit to be involved. If even one hacker spilled the beans it'd cause a massive panic for Reddit investors.
Deleting hospital fees/debt is very dangerous... In many HUGE regions in the US there's only one hospital and if that hospital suddenly can't pay its bills it could shut down, leaving a whole lot of completely innocent people in a very sad, people-are-dying sort of state.
In fact, something like this already happened:
https://www.cbsnews.com/chicago/news/st-maragrets-health-central-illinois-hospital-closing/
Hospitals are special in that they're often evil organizations (not all though) that are some of the easiest to hack but also provide critical services to the most vulnerable. One should tread lightly. Political solutions are better (hack some politicians that are against healthcare reform instead).
Clearing credit card debt via hacking is nearly impossible but I agree it would be a much more ethical choice for hackers to target. I used to work for the credit card industry. My unique insider perspective, deep industry knowledge, and personal experience is here to let you know they suck. They are just as evil and unethical and unnecessary as everyone thinks they are! Seriously: If Visa, MasterCard, American Express, and all the lesser players suddenly disappeared the world would be a better place.
Before that can happen though people need a backup payment method that doesn't go through their systems and no: Cash won't work (there's not enough in circulation and it's dangerous to carry large amounts of it). The credit card companies know this threat exists which is why they lobbied Florida (and probably other states) to outlaw alternative, government-run forms of payment (e.g. central bank currency).
As soon as people have a widely accepted payment option that doesn't go through Visa and MasterCard's middlemen (e.g. First Data) then hackers can take their gloves off! Until then though... Let's keep the payment infrastructure working, OK? Thanks!
There's no limit to the amount of good deeds hackers can do though. So let's encourage that! For example, there's plenty of cartels and evil religious organizations (e.g. Taliban, ISIS, Mormon Church, Prosperity Gospel scam artists) that have plenty of money to spare and enormous attack surfaces 👍
- Hospitals are special in that they’re often evil organizations
  Just want to state the obvious and say, this is pretty much only the case in the US.
- I think the alternative payment systems in the developing countries are actually good. UPI in India is very utilitarian. China also has the wechat thing. I guess the issue with these are that they are not universal and limited to a single country.
Considering some of the targeted instances and the stuff they left behind, it was likely some nazi.
“Marty we started this journey together!”
“It was a prank, Cos’!”
Ribbit
clearing credit card debt, deleting hospital fees, or something else that actually serves the public good,
Inflation does very clearly not serve the public good. That aside, causing havoc in banks and medical institutions would have other unpleasant effects.
- How about cleaning the bottom 10%'s debt, with the earings from one week of the top 0.1%?
- !badeconomics@lemmy.world

Thanks Ruud for fixing it! Just a reminder guys that If you are using a third party app you need to login again.

Thank you.
In case anyone else is having trouble logging in, my password wasn't working so I had to reset it from the website.
- So now I can log in via a 3rd party app but not the website (with the new password that I reset via the website.)
  I'm currently posting from the 3rd party app. Digging around to try and find 2FA settings for Lemmy.
Further 3rd party heads up -- for us nontech refugees:
If it looks like you are logged in, you may not be. I use Connect, and at your reminder, I clicked my acct and it says I was logged in. I tried to comment that Connect login was working, and my comment didn't show up.
I tried again, only to see an ”error: not logged in” message pop up.
Signed out, signed in again manually, and all is well.
So do a double check, Lemurs. Trust in your actions, not your eyes.
- Can confirm. Was like: “Memmy is fine!” — **Narrator:**nope. It was not.
- Thanks for the info! Here is my test to see if I'm actually logged in.
For capable people this is a minor annoyance but whenever there’s an “everyone needs to login again” issue, we will lose mere mortal users. In this case it wasn’t even clear that was needed - I appeared to be logged in but nothing worked. Ordinary users give up over things like this. I’ve seen it happen many times on sites where I had access to the analytics. I hope we regard this as a really bad thing to be avoided at all costs and not a “no big deal, just log in again.” Easy for you, easy for me, many others will just bail.
Thanks for the heads-up.
Thank you!

what steps are being taken to ensure it doesn't happen again? was any personal data compromised for users?

Good point, I'll update the post.
- Also I am curious, what's the easiest way to currently reach the admins in case this happens again somehow? Two of them on their account have been seemingly inactive for a month and as per your own statement you rarely check your notifications and dms. Is there a discord somewhere for it?
- So all our cookies are negated now with the JWT changed, and we just needed to login again? Can attackers have stolen our cookies in order to use our accounts to post as if it was us? I'm sure they were only interested in admin cookies, so most others were "useless" to them? I see nothing wrong with my posts so I should be safe, right?
- Nice work on the recovery, especially from a 0-day.

IMPORTANT ANNOUNCEMENT: My account was not among those hacked. Any random bullshit appearing in my post/comment history was written by me.

First - really good summary and sounds like everyone is working hard.

Cross posting the below comment.

Under GDPR if you have had a data breach you have a legal obligation to assess whether you need to report it and you must make the report within 72 hours of discovering the breach.

There are other types of reportable breaches too, I only mention data as it sounds most likely. You may or may not be subject to PECR which may also have been breached although less likely. I don’t really have enough familiarity with the regulation to discuss that one.

If you are not sure if there has been a breach you may also need to discuss it with the relevant body or make a report.

Please can you update what action you have taken regarding this and if the incident was reportable or not and the reasons why. Edit - from that new information, it sounds like this is a reportable breach.

For a full understanding, it would be good to know if you had 2FA enabled on the compromised account particularly as it had admin privileges and if so how 2FA was circumvented with this exploit.

It would also be good to know what measures you have in place to prevent the same or other malicious attempts on your Open Collective and Patreon accounts as issues with those are potentially more serious. They may not be vulnerable to this, but it is going to be reassuring to know there is good security practice, 2FA protection etc enabled and you have robust procedures in place.

Thanks for the info. We're looking into this.
If a valid browser token gets stolen like in this case, then MFA won't do much good because the stolen token will already have been authenticated. Linus Tech Tips experienced the same thing recently, you can check out their channel.
- That makes sense, thanks so much - there's a few good explanations here which really help! Would it be right in saying that all affected servers should be logging off all users - some have but not sure if all.
Out of curiosity, where would the regulators go for a case like this? There's no "company" running it per. se.
- It seems the general consensus is GDPR applies even to OSS non company entities, but it would appear that there's very little being done to honor it.
  https://www.zwilnik.com/better-social-media/activitypub-conference-2019/oss-compliance-with-privacy-by-default-and-design/#:~:text=Although%20GDPR%20directly%20applies%20only,sysadmins%2C%20including%20in%20the%20Fediverse.
  This article outlines Fediverse and responsibilities, I think it mostly requires someone to file a lawsuit before there's any action.
  In another case a man had cameras in his back yard that could also see a public area and was fined and forced to move them.
  https://www.termsfeed.com/blog/gdpr-exemptions/
  Mainly it just seems to be fodder to be used in lawsuits to make people comply with others security wishes. Not certain how all that works since cities are covered in public cameras.
- I am not sure how a platform like this will work with GDPR - each server will be responsible themselves, but how it works with the flow of data between servers and who the regulators would have cases against - I think that is to be tested at some point.
- They will go after a person instead.
Can 2FA be enabled for all users? I don't see the link to activate it after saving.
edit
Yeah, this doesn’t work at all. The apps don’t open links anymore. I tried some github site that reads the link and generates a QR, but the codes don’t work. This is a complete waste of time.
- Just reload the settings page after saving and you'll see the activation link. Just now enabled 2FA for my account.
Just curious if you turn yourself in to police everytime you speeding.
- This is not about turning you in, this is about protecting your users who all possibly just became victims of a crime, and for good reasons it's not fully upon you to decide whether the possible consequences of this are serious for those users.
- It's more that many people expect those handling their data to be seen to follow the correct procedures and be trusted to handle the data in a fair, transparent, safe and secure way - and in addition to protecting their users, companies are probably encouraged to abide by the regulations because it is very easy for anyone to report where they think action needs to be taken, and regulatory bodies may be more lenient where correct process has been followed.
  If I chance a speeding or parking ticket I can't be fined nearly 20 million pounds, although I wouldn't trust some parking companies not to try it! (I'm not saying that would be the case in this instance.)
  https://gdpr.eu/fines/

Thanks for letting us know - this is the kind of transparency that I wish the world had more of!

So what happened:

Someone posted a post.
The post contained some instruction to display custom emoji.
So far so good.
There is a bug in JavaScript (TypeScript) that runs on client's machine (arbitrary code execution?).
The attacker leveraged the bug to grab victim's JWT (cookie) when the victim visited the page with that post.
The attacker used the grabbed JWTs to log-in as victim (some of them were admins) and do bad stuff on the server.

Am I right?

I'm old-school developer/programmer and it seems that web is peace of sheet. Basic security stuff violated:

User provided content (post using custom emojis) caused havoc when processing (doesn't matter if on server or on client). This is lack of sanitization of user-provided-data.
JavaScript (TypeScript) has access to cookies (and thus JWT). This should be handled by web browser, not JS. In case of log-in, in HTTPS POST request and in case of response of successful log-in, in HTTPS POST response. Then, in case of requesting web page, again, it should be handled in HTTPS GET request. This is lack of using least permissions as possible, JS should not have access to cookies.
How the attacker got those JWTs? JavaScript sent them to him? Web browser sent them to him when requesting resources form his server? This is lack of site isolation, one web page should not have access to other domains, requesting data form them or sending data to them.
The attacker logged-in as admin and caused havoc. Again, this should not be possible, admins should have normal level of access to the site, exactly the same as normal users do. Then, if they want to administer something, they should log-in using separate username + password into separate log-in form and display completely different web page, not allowing them to do the actions normal users can do. You know, separate UI/applications for users and for admins.

Am I right? Correct me if I'm wrong.

Again, web is peace of sheet. This would never happen in desktop/server application. Any of the bullet points above would prevent this from happening. Even if the previous bullet point failed to do its job. Am I too naïve? Maybe.

Marek.

Damn, I go to bed early and I miss everything! Thanks for the quick resolution and transparent disclosure, this place is great!

Thank you for your work 🙏

This is really good to see such transparency from admins

Love the transparency!

Good thing we all use randomly generated passwords for every account and always remember to change them every few months.

Can we get another admin to sign off on this being authentic? In other words, short of a signed GPG signature how do we trust announcements after a breach where admin accounts are compromised?

Good point. I did post about this on Mastodon @mwadmin@mastodon.world
- Thanks for the reply!

So thats why MalwareBytes gave me this message yesterday.

Wow that's cool. How did malwarebytes know the website was compromised ?
- I think it sees that the browser is trying to execute code that is suspicious (the payload of the XSS was pretty obvious).
- probably looking for obvious patterns like "onload="... in image and link tags, because an onload event handler would usually never be put in those tags otherwise so the only plausible explanation is that it's a XSS attack
- I wish I knew. I tried logging into Lemmy yesterday and I was kept giving this message. I thought it might be relevant and saved this snip. I am only about to post this pic now. I did whitelist Lemmy on Malwarebytes after as well.
Interestingly, here is a log when browsing Lemmy over the last week or so.
Believe the derp.foo and .today are both federated instances. Don't know what the other rows are.
- Admin of derp.foo here. My best guess is Hetzner gave me an IP that had been used to host a botnet C&C before. As a precaution I switched to a new VPS; please contact me via matrix at @irdc:tchncs.de if the problem persists.

Thanks for the work. As a heads up it appears most of the block instances are back however I believe explodingheads is still missing which you may want to confirm.

EDIT: it has been added back to the block list.

Hey how do you check on that?
As of the time of me posting this comment, exploding heads is appearing in my feed with some anti lgbt posts. Idk what’s going on because I’m pretty sure they’re supposed to be defederated currently

the details of the vulnerability are already known now anyway since there's a fix that was proposed on the Lemmy GitHub so I don't think it will hurt others to talk about it

Could you please link the issue? Thanks!
- https://github.com/LemmyNet/lemmy-ui/pull/1897/files found it myself

FYI: I had to clear my lemmy.world cookies in order to be able to successfully log back in.

(This was with Firefox)

(Edit: I also shift-clicked reload, which somebody pointed out does clean the cache for that page, so I also cleaned the cache).

Thank the heavens the meme community stayed safe through this without my daily dose of cybersecurity memes idk how I would function ;)

How do we know that this isn't a fake announcement as well, trying to give us a sense of security???

Just kidding, thanks for letting us know! Thank god I haven't been too active the last few days! Can't afford my credentials being leaked, maybe I should be proactive and change my password anyways.

Had to clear my browser catch to log in, Jerboa still shows as not logged in even after logging out which you do by clicking the hamburger menu then click the top banner to change/log out of accounts. This post is a test to see if my account works again via browser lol.

Edit: clearing app data/cache for Jerboa fixed the login issue.

Thanks for this comment, I had to wipe cache and cookies to login on desktop. Wasn't sure what was preventing me from logging in.

Do we have any details on how Michelle's account was compromised? Right now in the GitHub issue about the vulnerability they're clueless about how the custom emoji exploit could be performed without first an already compromised admin account.

EDIT: yeah here's how: https://github.com/LemmyNet/lemmy-ui/issues/1895#issuecomment-1629326627

You do NOT need an admin account to do that. Any normal user could have done that.

Hopefully with more attention on the source code scary hacks like this doesn’t happen again.

One wonders how much spez paid for that hack.

Took me a bit to realize I actually had to log out and log back in on Jerboa since it looked like I was still logged in but some interactions didn't work

Yeah, I'm not sure what was going on.

Can I ask some possibly dumb questions?

What is JWT?
Was any private user data compromised, and if so will users be informed?
Is there anything regular users can do to avoid their data being compromised? For example, not accessing lemmy on certain web browsers?

Thank you!

JWT stands for JSON Web Token. It's basically a way for a server (lemmy's) to put a piece of information in your browser in a way that makes sure it come from the server. It (usually) uses some form of digital signature. You can think of it as a note someone gave you with their signature, assuming said signature is very hard/impossible to counterfeit. The next time you see that person, they don't have to remember you, they just have to check the signature. If it is valid, anything written on the note is taken at face value.
When you connect to a site, there are a few steps to validate that you are who you say you are (identification and authentication). Something like inputing you login/password. Since it would be tedious to do that on every requests, the first time you give your login/password to the server (this is the simplified version, this exchange is a bit more complex usually) the server gives you that JWT. For every subsequent requests, your browser automatically send that JWT that is simple to handle but hard to counterfeit, and the server safely knows that you're whoever is written in that JWT.
I assume there will be a post here when more details are known, or that this post itself will be updated. As with any online service, it's up to the service to decide if they want to communicate. (it may also be a legal requirements in some places to tell user when such an event occurs). Since we're talking about obtaining other user's authentication token including an admin, it is safe to assume that whatever an admin can see has leaked. This can range from basic user informations to more private stuff, although I am not familiar with the software behind lemmy. Note that this is a worst-case scenario; an admin impersonator could have access to anything an admin could see, it does not mean they immediately dumped everything. It depends on their motivation.
Protection against this kind of stuff Compromission of the JWT can happen in many ways and I don't know which way was used. But if there's a flaw in the software used (the lemmy's client-side code, for example) there is not much you can do. JWT can leak through many things :
server compromission (out of your control)
client-side compromission (only happens when using a browser; applications that uses API should be less susceptible to that)
vulnerable extension: if you have browser extensions, they can easily peek into what's happening in any given page (that's their whole purpose). Malicious extensions, or extensions that allow outsider's some kind of control over them can leak data
browser vulnerability: keep your browser up-to-date, and (this is controversial) stick to a family of well-enough known browser. That obscure browser that have 20 users worldwide and is based on a three years old version of chromium is not the best thing to use
keep your data safe: only put the minimum required amount of data on any service. For lemmy, I assume an email address and your login/password is the bare minimum (well the email is already extra, but it's very convenient to have). Some services really likes to get everything they can out of you.
Basically, stay up to date and don't use shady stuff. Easy to say, I know.
- This is great thank you!
- This answered all of my questions and really helped me understand what happened, thank you so much for having the patience to walk me through it!
  It sounds like using apps to access Lemmy is more secure than signing in on your mobile browser (which is what I'm currently doing.)
  I tried a couple apps last week but didn't like them, I think I'll revisit that today.
  Thanks again!
1 - jwt is the authentication cookie u get when u sign in with ur password, ur browser stores the jwt and then any further interaction authenticates using it
2 - yes, userscripts, extensions, custom frontends, apps, all apps have access to jwt
- 2 - yes, userscripts, extensions, custom frontends, apps, all apps have access to jwt
  HttpOnly flag not being set? That would protect against userscript, web apps, and extensions without extended permissions being able to access it.

You guys really have my highest respect for spending so much time to keep this running, despite all the recent trouble and now even an attack.

Thank you very much <3 You guys are awesome and I really appreciate how publicly you deal with this.

That was scary and exciting. Response seems competent and transparent. I ❤️ this place.

I think this is a strong reminder: We shouldn't put all our eggs in one basket. This will happen again. Unlike Reddit, we don't need to concentrate all communities on one instance. We should all make an effort to spread out. Some other general use instances are:

Again, for those new, you can post content to any of these instances and interact with content from other instances at the same time, just like you can send an email from your Gmail account to your ProtonMail account.

Despite the fact that Lemmy is a fairly new piece of software, which makes these issues more likely, I am really grateful for it being open source, and I really appreciate this level of transparency.

Well done all involved. Sounds like it was caught and mitigated quickly

So, do we change passwords, esp those who logged on during the attack? (I created this acct right before the attack happened tho.)

No, passwords weren't compromised
I think it's good practice to change passwords after an attack no matter what
So, do we change passwords
If you don't use a randomly generated password, it's a good idea to change it anyway. Not because of this specific attack but in general. For the longest time the Lemmy software was just a hobby of a very small group of individuals. While the back-end is written in Rust and probably more robust than the PHP code over at Kbin, I don't think a proper security review was ever conducted, so there's a not so small chance there will be some additional growing pains in the somewhat near future.
According to the admin, no, but changing your password and keeping your data safer is always totally fine to do and you should probably do it every once in a while regardless.

Thank you for the transparency and swift solution!

Vulnerability strikes. Open source's lightning response strikes back. Again.
@nonagonOrc @ruud 👍🏻

Any truth to what I've heard this may have been done by a group we defederated with?

Thanks for fixing and being so open about it

How does this impact those using mobile apps like Jerboa or Liftoff, instead of the website directly?

Check the pinned post on liftoff community page.
https://lemmy.world/post/1292303
As a safety precaution logged-on sessions on many servers have been cancelled and you are required to logon again.
- Thanks, I'll do that. Curiously, the lemmy.ml account keeps working, wonder what it depends on.
as someone who uses the app, extremely little effect from my experience, I didn't notice something was wrong at all until people pointed it out due to how liftoff does the whole sidebar thing for the instance.
It's still better to change your account password and clear your cache.
Was wondering this myself. Is there a way for users who where exposed to know about it?
(Edit) Eg if the exploit was through a post get notified if they saw the post?
- apparently they posted it as a weird image or emoji that looked like this:
- There is no need to get notified, they didn't steal passwords, just session cookies. Most (all?) servers have invalidated all the user login cookies, but if you are in doubt, just logging out and back in should be enough to get a new cookie.

I just disabled whole "/admin" section on my instance and added nice message 😆

Good job. I don't understand very much of that, so that makes me all the more grateful. Thank you.

Thanks for the transparancy about this.

Good communication is key to effective incident response. This is a good example of that. Kudos.

On Liftoff, I had to clear cache and storage in order to log back in. Still having issues with the website on Chrome, which keeps telling me I'm not logged in after clearing cache and logging back in.

The quick fix is much appreciated, thank you and everyone that helped for your hard work!

At least now we can mark off the "disruptive website defacement attack" line on the checklist of (relatively) new website growing pains. Better to have them make lots of noise and get fixed quickly than quietly do sneaky things in the background.

Thanks for your efforts. I know that Lemmy was put in place rather quickly as a Reddit alternative. But I'm genuinely hopeful that this will be a good alternative.

Could admins sign announcements with a PGP key to mitigate false admin posts and the consequences this might have? Or is this no longer necessary?

It's probably a good idea to have official announcements be signed, that way it's obvious when they're actually posting officially or if they are compromised.
What's the difference? JWT is already cryptographically signed, but tokens were stolen. That's the issue.
- No one store their PGP private key in cookies, at least I hope so.
  Ideally someone wants to send signed message doesn't store the signing key in their browser, sandbox their browser even.
- PGP private keys are harder to steal than JWTs, as they are not generally stored as a long-term cookie but briefly just to sign something. Through XSS (the vulnerability in this case), cookies are relatively easy to steal, but to steal a PGP key would require a more complex script able to steal the key at the time it is loaded in the browser (assuming the signing feature is implemented in the browser). It's a bit more sophisticated, but not totally bulletproof.
I think the lemmy.world admin posted on his official Mastodon.
https://mastodon.world/@mwadmin/110688515627268847

One thing I don't get. Custom emojis can only be created by an admin, but you're saying an admin's account here got compromised because of that and not the other way around. Does that mean that an evil instance set a custom emoji with the injected JavaScript and propagated it to the federated instances?

From the fix, I believe the custom emojis were not double checked after a user submits a post. The post data was used to display the emojis, and thus allowing injection.
The fix now is to search the emojis in the custom emojis list from the backend rather than the user post.

That doesn't surprise me. Especially the "homemade" instances. The documentation is severely lacking and I had to fix lots of stuff in the instructions with try&despair to make my instance run.

There's not a great focus in security if your application starts with "step 1: install docker"

Does an admin account have any permissions to view email addresses or data of registered users?

Did MichelleG not have 2FA enabled?

Now that this has happened, it's be worth pushing this issue through as high priority. If HttpOnly was enabled, then an admin takeover would not have been possible.

https://github.com/LemmyNet/lemmy-ui/issues/1252

The JWT exploit bypasses 2FA requirements. It basically steals your active session and allows a third party to use it.
- Good point. I suppose the only way to fix that particular issue to disallow cookie authentications from a new location
To answer one question, the admins are able to view email addresses I believe. My knowledge comes from "I read it in a comment awhile ago that sounded credible" so I could be wrong.

Great lemmies! Thanks for uniting us.

How do we know you're the real you? This all could be part of the plan!

Ugh, people should not go after systems trying to give a free service to the internet. It just ruins everything.

Thanks for the great work. The response time was awesome, considering you were asleep as well.

Thanks for the transparency.

I had to create a new account. I tried enabling 2FA on my main account a week ago, but was never able to generate a token. Now when I try logging in it is asking for my 2FA token. Is there any way to get my account back. I'm a moderator of a community.

Yah, I noticed my Lemmies auto-corrupted to Lemurs.

I don't care. I'm keeping it.

Lemurs are cute.

Soo it looks like the entry for this instance was also changed on https://lemmyverse.net/ . At least I hope it is the hack

Once again, thank you guys for all that you do. As many other people are saying, appreciate the transparency about these things.

I can only log in on incognito mode, which makes me think my cookie has been stolen or whatever. So my question is, what should I be doing about that?

Amazing how you quickly reacted to this!! Bravo!!

TIP: if you can't login after what happened, clear out your browser cache including ALL cookies, that fixes it (it did for me at least). I believe it's also advisable to change lemmy password.

see the GitHub repo, it's new
It's not fixed yet in the current version
Concerns were posted a few days ago, but no POC that used the exact same attack as we saw here. Basically, there were some warnings, and work was underway that would have prevented this, but it was not done fast enough. There is a patch now, that will take a while to roll out, plus a renewed focus on general and related issues.

As someone in EU I didn't even realized there was an issue. Well done and great reaction time! Also thank you for the transparency 👑

Pardon the ignorance, but how do I know if I was compromised? what do?

This is why I've decided against running my own Lemmy instance. Too much work to have to keep up constantly with updating, too big of an attractive target for attackers.

This is so sad lmao rip. With any site growing as fast as these instances (because of the Reddit folk) Ig these attacks are to be expected. Hope everyone's accounts and personal info are okay

Interesting.

Attackers started changing site settings and posting fake announcements etc

So at least that wasn't 100% malicious, otherwise they could've kept the vuln hidden and just collect data and whatnot.

On the other hand, who cared enough about Lemmy to hack it? Weird.

On the other hand, who cared enough about Lemmy to hack it? Weird.
This one: https://lemmy.world/u/LMAO
- Why those kind of people exist? GET A F"+$+ LIFE
- Wow ... what a loser.

I can't log into my account anymore, this one is a new one I've just made. I tried to reset my password but nothing came in the mailbox. I can still see comments and posts from that account though.

It's this one:

And I don't know why but I can't save the profile pic for this account.

Edit: Nvm, I use another email to sign up for Lemmy and forgot about it

You need to delete all cookies for lemmy.world in your browser, then log in again.
- nvm, I used my other email for Lemmy and completely forgot about it :v , I can log back in now
  I think I fucked up, I check the 2FA on the new account but didn't click on send 2FA code, now I can't log back in to edit the comment
  edit: I have another tab opened with that account and it's still fine, I've just edited the comment

Must have been jealous spez

If I'm not from lemmy.world and visited a lemmy.world community via my home instance, does the hacker gain access to my account?

If I, while logged in to my home instance, accessed lemmy.world in another tab, does the hacker gain access to my account?

Does this hack infect devices used to access lemmy.world?

Sorry for noob questions, I'm just worried.

afaik, exploit does not pass through federation. but you should change your password just in case.
it doesn't and probably cannot infect your device
I am still not sure about it, but if a compromised comment reached your instance (through federation) and users in your instances viewed that comment, they have been hacked too.
MAYBE you are safe If your instance has no custom emojis enabled.

With the JWT secret rotation, shouldn't everyone be forced to re-login? I'm posting with my existing session without any changes.

FWIW,, I had to re-log
better to re-login i guess

Thanks for the transparency. Was having issues with Lemmy, now seems everything back to normal. Got a question, Just to add an extra layer of security, Do i need to use ToR or VPN with Lemmy ?

Rock on, Rudd.

Well that's just great it really is a shame though how some people would actively want to ruin something free like this just because they can.

I wasn't using webpage, I was only using mobile app (Connect). Could my coockie be also stolen that way or was that only possible on webpage?

No, the vulnerability was due to a client-side bug in the Lemmy web UI. Mobile apps render content in a different way, and are not vulnerable to this kind of attack (apart from in exceptional circumstances).
Should probably log out and back in still though.

Thanks all working again. Had to clear my browser cache in order to login again and had to resign in to memmy too.

I guess its early days for lemmy for incidents like this, fingers crossed something like this doesn't happen again :)

It's a nice reminder that those with the skills but not the bad intentions would be welcome to look through the source code for vulnerabilities and report/patch anything they might find. :)

Excellent, thanks for the quick response ruud and admins.

Here's a relevant post that talked about this with @AlmightySnoo@lemmy.world I think is worth looking into for anyone curious what exactly happened.

https://sh.itjust.works/post/923025

please don't visit the legal section of the website or anything confirmed compromised if anything.

Thank you for taking the time to update this :) Hope everything will be sorted out without people being scared. As a layman, was any user data compromised?

I noticed this morning for a small amount of my posts with pictures, maybe 5-10%, the pictures were deleted or missing. Not sure if this is related to the incident.

It seems there is no way in Lemmy to invalidate all your session cookies? Without that, how can you secure an account which has a stolen session cookie?

I found this in my private messages, when an attack was happening I messaged the guy “are you ok” and he replied back to me with an image of my own message… I wonder if this was similar to what was done here? Was 8 days ago

Possible that they've had access for days, and different accounts were breached at different times.
No that was something else
- Thanks for clarifying 🙏
So that banned troll is back? https://lemmy.world/u/LMAO

A lot of images seems to be gone from posts in /c/pics is this related to the hack or the cleanup after?

I heard there was some sort of database rollback to an uncompromised snapshot.

Thanks for the info, Ruud. I just put in for a monthly donation to you all -- I appreciate you.

Congratulations everyone on the quick fix/mitigation!

I had an issue of being logged out of my account and could not log back in, after closing and reopening the site, closing browser, etc until I cleared my cookies, then it let me back in. If that helps anyone.

Likewise. I tried to log in but nothing would happen. I had to clear out my browser cookies first.
- I'm just glad the account isn't gone completely.

What are the risks for people who use Jerboa for Lemmy? I logged put and back in and there doesn't seem to be any issues, so are the app users excluded from this?

There is no risk for people who were using apps like Jerboa. This was a web specific bug. And now that the login tokens have been invalidated, there isn't anything for web users to worry about either.
- Thank you.

Testing… I have to keep trying this because memmy is being a dick.

Would it be a good idea to force a login if the users IP or device suddenly changes?

Is it possible cookies for other websites were scraped? I was logged in to .world at the time; I have logged out of all accounts, and reset passwords as a precaution, but want to know if I should be on the lookout from this.

No, in general it's not possible because the code in a page cannot access cookies that are bound to other domains. It is only possible if the "other" site misconfigured its own cookies (which is really not likely for stuff you would care about).

Had to re-login in the Connect app

TY to everyone itt who commented on how to fix the 3rd party app issues.

I was panicking when liftoff went wonky

Poor beehaw.org is still down

Thanks for the quick reaction and TRANSPARENCY!!

Um, probably coincidence or a false posi, but malwarebytes is labeling lemmy.~~world~~today as being compromised / malicious when following external links, it's only popped up twice, but here's a slightly redacted log file:

-Log Details- Protection Event Date: 7/10/23 Protection Event Time: 1:24 PM

-Software Information- Version: 4.5.33.272 Components Version: 1.0.2069 Update Package Version: 1.0.72209 License: Premium

-System Information- OS: Windows 11 (Build 22621.1928) CPU: x64 File System: NTFS User: System

-Blocked Website Details- Malicious Website: 1 , C:\Program Files\Google\Chrome Beta\Application\chrome.exe, Blocked, -1, -1, 0.0.0, ,

-Website Data- Category: Compromised Domain: lemmy.today

(end)

Seem to have a hard time loging in

Passwords were leaked?

I hope devs will examine all parts of the code that display content to make sure proper sanitization

That explains why I had to clear my browser cache, I was unable to login until I did.

You guys are quick!

I’ve been unable to login on desktop since this happened. Only been able to login via Memmy on IOS.

I put in my info and it kicks me back to the front page and doesn’t log me in.

I’ve tried clearing cache too

EDIT: Switching browser to Edge seemed to let me. Weird. Even reinstalled Firefox and still won't let me.

I heard that some instances were defaced. Any examples of this? I wasn't online this noon so I never got to see any action.

No need if it was JWT token. After you use your password to log in, the server send your browser/app a JWT token. It uses this token whenever it requests anything from the server, to show that your logged in for this session, and the server can look at the token and tell who it gave the token to, show it knows you're logged in.
All that is to say, logging out will mark any leaked token as no longer valid, and when you log back in you'll have a new, non-hacked login token.
- I mean, I want to see screenshots of the "fake announcements" the hackers posted through compromised admin accounts

Not too promising, hopefully it's fixed quickly.

Hmm. Liftoff won't let me post but shows logged in and as a newbie be damned if I can find where to log out.

Should I change passwords or no?

I appreciate the transparency. Hopefully with more eyes on the source code hacks like this will not happen again.

Is a password change advised? How does the JWT cookie and exploit effect apps eg Jerboa?

You will have to login again for those apps. As far as we know, the exploit doesn't allow someone to actually steal your password directly, just the session you were logged into.
However, it is my personal opinion that you should change your password anyway out of an abundance of caution.

Thanks for the post-mortem and the quick fix! Glad you guys around to help battle test Lemmy's code.

Damnit, spez.

Thanks for the quick response! This admin team rules!

May I ask was is the JWS coockinand if it is automatically changed or if we have to change it in a way?

My main is locked up and it sucks, i hope this all gets fixed.

So that was why the logo and name was changed to israel. And for some reason getting redirected to a gif that was from lemmy

Thank you for your fast answer!

It seems that I lost all my subs. There were not many but still annoying.

E: Still subbed but can't see those in Voyager.

I thought I’d lost mine too, and when I checked the community I wasn’t subbed. I could still view my profile, comments and posts though.
I cleared the cache, then tried to post here and it said I was logged out (even though I could see all my activity except subs). I couldn’t see any way to logout, so I edited my profile and re-entered my password then hit save. That seems to have fixed it, now I can post and my subs are back.

I’d like to logout, then log back in, because I can’t upvote / downvote- how do I logout? I can’t seem to find a logout button.

Are you on the web or an app?
I'm using Liftoff app, and I had to clear my storage and cache to log out.
- Where do yoh do that on Liftoff? I ended up reinstalling the app.

Good shit! Thanks for keeping things up and the pretty quick response as well.

Occasional cookie deletions I understand, but will sign-ins persist in the future?

Thanks for the update, I appreciate the transparency.

Thanks for keeping us up dated!

is that why I can't log into my lemmy.world account?

ok not a problem anymore. seems like I just had to clear my cache and it let me log in

******* This happened to me, one of my posts had it's photo deleted (I didn't delete it), then when I replaced it, the next time I checked the entire post had been deleted.

Is this why I had to sign in and out of my account on liftoff?

I couldn't comment untill I did that. There may be others!

I had a similar issue where my subscriptions were blank. A logout and re-login fixed it. Thanks.

Well done on acting on it so quickly. I think I did see some of the fake announcements you were referring too but were taken down very quickly. Keep up the good work team and thanks for everything you are doing!

Thanks to everyone involved for the quick response 👍

Is there a rough time range when it happened? and any news about other big instances like lemmy.ml? Are those safe? Currently they are not on the same version as lemmy.world.

2:11 UTC is my first record of the event taking place, but keep in mind the attacker could have injected code long before without noticeable impacts. There's no way to be completely certain they didn't steal tokens and access accounts before they made themselves known.

I lost some of my post history. Is there a data issue that's come from this? Why are my comments gone?

If it is only recent post history, maybe it was purged along with many malicious comments/posts
- I know but this was over a week of comments lost.

How are you preventing it to happen again until a patch is released from devs?

We removed the vulnerability
It's open source, they can just fix it themselves until it's released. :)

thanks to admin team for resolving that quickly

I was unable to log in, it looped me & said I logged in, but did not. I read this post, cleared my cache, and I was able to log in (and change my password).

Thanks for fixing it.

Is this why Jerboa seems to not work any more? It keeps insisting I'm not logged in, when I am, showing me as anonymous, but also showing my profile details, not letting me interact with things, etc... It's been a big problem these past few days making Lemmy unusable :-(

This seems way worse than they are making it sound.

Is that why Liftoff wasn't loading?

Maybe there needs to be a quick rundown how to actually log out and in on clients, seems you can't with jerboa without just wiping the app, and wefwef, you need to delete all accounts.

I see some instances are throwing server errors

👍

Still unable to log in. Is this everyone or just me?

Thanks.

Because I am obsessed with bugles, any comment or post I make that does not manage to fit bugles in somewhere (because I always have room for bugles) will be an imposter!

Thank you for the transparency and keeping my nefarious bugle consumption private!

Is this why I can't log in on Chrome? I switched to Firefox and it worked.

Great job everyone! Keep it up! Love the transparency!

I see you, Imposter.

that is why I got logged off from my account this morning!! impressed by the rapid intervention!! Good job lemmy team!

Test

lemmy explorer still acting kinda funny, not sure who is the person in charge to inform

i checked it now and looks okay.. https://i.imgur.com/a95CO6o.png
EDIT: i was looking at the wrong page.. yeah i still see the same malicious picture on world - https://i.imgur.com/sJYoels.png
- i cleared cache still the same lol

I couldn't login last weekend, couldn't that be te reason

One of the reasons I used a throwaway email here.

cool

Had an issue at work not long ago involving stolen tokens and back then it looked as if the token was scraped along with a lot of other web traffic and then about 12 days later they gained access.

Luckily the tokens have been invalidated by updating the secret

Been busy for lemmy team lately hopefully it's not too much

So is it safe to log back in?

Yes

Huh, i think i got lucky by forgetting that there is something i can consume other than youtube

I had to clear jerboa's app data so I could log in again. so strange

Is that why I got logged off?

Yes.
Also it looks like some images from posts are kind of gone
- Thank you. I’ll be changing my password.

Thanks for the quick response. Do we know if there was any data leak?

Thanks for being open about this and quick to fix it!

Should apps have logged themselves out?

Does this have anything to do with the Wefwef app name change?

Thanks for the update. Can you update us on whether or not you are planning to block threads.net?