Skip Navigation
Lemmy Support @lemmy.ml DJArbz @lemmy.notmy.cloud

Lots of Operation Timed Out messages

I just started to self host my instance this week. Is it normal to get a lot of these error messages? My instance seems to be working from an end users POV.

Jul 28 11:22:48 oracle lemmy_server[2229781]: 2023-07-28T16:22:48.239200Z  WARN lemmy_server::root_span_builder: Other errors which are not explicitly handled
Jul 28 11:22:48 oracle lemmy_server[2229781]:
Jul 28 11:22:48 oracle lemmy_server[2229781]: Caused by:
Jul 28 11:22:48 oracle lemmy_server[2229781]:     0: error sending request for url (https://lemmy.blahaj.zone/comment/1464434): operation timed out
Jul 28 11:22:48 oracle lemmy_server[2229781]:     1: operation timed out
Jul 28 11:22:48 oracle lemmy_server[2229781]:    0: lemmy_apub::activities::voting::vote::verify
Jul 28 11:22:48 oracle lemmy_server[2229781]:              at crates/apub/src/activities/voting/vote.rs:57
Jul 28 11:22:48 oracle lemmy_server[2229781]:    1: lemmy_apub::activities::community::announce::receive
Jul 28 11:22:48 oracle lemmy_server[2229781]:              at crates/apub/src/activities/community/announce.rs:141
Jul 28 11:22:48 oracle lemmy_server[2229781]:    2: lemmy_server::root_span_builder::HTTP request
Jul 28 11:22:48 oracle lemmy_server[2229781]:            with http.method=POST http.scheme="https" http.host=lemmy.notmy.cloud http.target=/inbox otel.kind="server" request_id=15087d78-d6d4-46c8-97b1-40aadc9598bd
Jul 28 11:22:48 oracle lemmy_server[2229781]:              at src/root_span_builder.rs:16
Jul 28 11:22:48 oracle lemmy_server[2229781]: LemmyError { message: None, inner: Other errors which are not explicitly handled
Jul 28 11:22:48 oracle lemmy_server[2229781]:
Jul 28 11:22:48 oracle lemmy_server[2229781]: Caused by:
Jul 28 11:22:48 oracle lemmy_server[2229781]:     0: error sending request for url (https://lemmy.blahaj.zone/comment/1464434): operation timed out
Jul 28 11:22:48 oracle lemmy_server[2229781]:     1: operation timed out, context: SpanTrace [{ target: "lemmy_apub::activities::voting::vote", name: "verify", file: "crates/apub/src/activities/voting/vote.rs", line: 57 }, { target: "lemmy_apub::activities::community::announce", name: "receive", file: "crates/apub/src/activities/community/announce.rs", line: 141 }, { target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=lemmy.notmy.cloud http.target=/inbox otel.kind=\"server\" request_id=15087d78-d6d4-46c8-97b1-40aadc9598bd", file: "src/root_span_builder.rs", line: 16 }] }
1
1 comments

  • Hi. This is typical right now of lemmy fderation based on my own personal testing.

    1. There is a 10 second timeout in HTTP that I think is on the short side.
    2. The peer servers your lemmy instance is trying to reach are suffering from big PostgreSQL overload due to some TRIGGER logic in lemmy 0.18.2 and earlier. Hopefully the 0.18.3 release will be fixed up and deployed on many of these sites fixing the worst of it.

    As you also can see, one single http timeout spits out tons of log lines. I created a GitHub issue to try and get that down to one line so my server isn't constantly writing to logs. https://github.com/LemmyNet/lemmy/issues/3627

    1. There were massive PostgreSQL mistakes in lemmy_server causing constant overload with even very modest number of comments/posts/instances in the table. 0.18.3 improves the situation. The servers you are connecting to you are constantly crashing.

    2. There is a 10-second timeout build into the HTTP timeout code in lemmy_server - and I think it is too short. I have opened a GitHub issue: https://github.com/LemmyNet/lemmy/issues/3627