Slyke @ Slyke @lemmy.ml Posts 1Comments 6Joined 2 yr. ago
Figured it out! It was Lemmy UI. I was using an older config version. It appears that newer version of Lemmy UI use different environment variables:
I was using:
LEMMY_EXTERNAL_HOST LEMMY_INTERNAL_HOST
Instead of:
LEMMY_UI_LEMMY_INTERNAL_HOST LEMMY_UI_LEMMY_EXTERNAL_HOST
Figured it out! It was Lemmy UI. I was using an older config version. It appears that newer version of Lemmy UI use different environment variables:
I was using:
LEMMY_EXTERNAL_HOST LEMMY_INTERNAL_HOST
Instead of:
LEMMY_UI_LEMMY_INTERNAL_HOST LEMMY_UI_LEMMY_EXTERNAL_HOST
Is the API meant to return 404 on the main route? Here's the text when hitting the API, and UI directly, and also the status codes returned. This is not going through NGINX:
bastion:~# curl http://lemmy-ui:1234/ <!DOCTYPE html> <html > <head> <script nonce="undefined"> window.isoData = {"path":"\u002F","site_res":undefined,"routeData":{},"errorPageData":{},"showAdultConsentModal":false}; if (!document.documentElement.hasAttribute("data-bs-theme")) { const light = window.matchMedia("(prefers-color-scheme: light)").matches; document.documentElement.setAttribute("data-bs-theme", light ? "light" : "dark"); } </script> <!-- A remote debugging utility for mobile --> <script src="//cdn.jsdelivr.net/npm/eruda"></script><script>eruda.init();</script> <!-- Custom injected script --> <title data-inferno-helmet="true"> </title> <style> #app[data-adult-consent] { filter: blur(10px); -webkit-filter: blur(10px); -moz-filter: blur(10px); -o-filter: blur(10px); -ms-filter: blur(10px); pointer-events: none; } </style> <!-- Required meta tags --> <meta name="Description" content="Lemmy"> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"> <link id="favicon" rel="shortcut icon" type="image/x-icon" href=/static/c15a0eb1/assets/icons/favicon.svg /> <!-- Web app manifest --> <link rel="manifest" href="/manifest.webmanifest" /> <link rel="apple-touch-icon" href=/static/c15a0eb1/assets/icons/apple-touch-icon.png /> <link rel="apple-touch-startup-image" href=/static/c15a0eb1/assets/icons/apple-touch-icon.png /> <!-- Styles --> <link rel="stylesheet" type="text/css" href="/static/c15a0eb1/styles/styles.css" /> <!-- Current theme and more --> <link rel="stylesheet" type="text/css" href="/css/themes/darkly.css" /> </head> <body > <noscript> <div class="alert alert-danger rounded-0" role="alert"> <b>Javascript is disabled. Actions will not work.</b> </div> </noscript> <div id='root'><div class="lemmy-site" id="app"><button class="btn skip-link bg-light position-absolute start-0 z-3" type="button">Jump to content</button><div class="shadow-sm"><nav class="navbar navbar-expand-md navbar-light p-0 px-3 container-lg" id="navbar"><a class="d-flex align-items-center navbar-brand me-md-3 active" aria-current="true" style="" id="navTitle" href="/"></a><button class="navbar-toggler border-0 p-1" type="button" aria-label="menu" data-tippy-content="Expand here" data-bs-toggle="collapse" data-bs-target="#navbarDropdown" aria-controls="navbarDropdown" aria-expanded="false"><svg class="icon"><use xlink:href="/static/c15a0eb1/assets/symbols.svg#icon-menu"></use><div class="visually-hidden"><title>menu</title></div></svg></button><div class="collapse navbar-collapse my-2" id="navbarDropdown"><ul class="me-auto navbar-nav" id="navbarLinks"><li class="nav-item"><a class="nav-link" title="Communities" href="/communities">Communities</a></li><li class="nav-item"><a class="nav-link" title="Create Post" href="/create_post">Create Post</a></li><li class="nav-item"><a class="nav-link d-inline-flex align-items-center d-md-inline-block" title="Support Lemmy" href="https://join-lemmy.org/donate"><svg class="icon small"><use xlink:href="/static/c15a0eb1/assets/symbols.svg#icon-heart"></use><div class="visually-hidden"><title>heart</title></div></svg><span class="d-inline ms-1 d-md-none ms-md-0">Support Lemmy</span></a></li></ul><ul class="navbar-nav" id="navbarIcons"><li class="nav-item" id="navSearch"><a class="nav-link d-inline-flex align-items-center d-md-inline-block" title="Search" href="/search"><svg class="icon"><use xlink:href="/static/c15a0eb1/assets/symbols.svg#icon-search"></use><div class="visually-hidden"><title>search</title></div></svg><span class="d-inline ms-1 d-md-none ms-md-0">Search</span></a></li><li class="nav-item"><a class="nav-link" title="Login" href="/login">Login</a></li><li class="nav-item"><a class="nav-link" title="Sign Up" href="/signup">Sign Up</a></li></ul></div></nav></div><div class="mt-4 p-0 fl-1"><div class="error-page container-lg text-center"><h1>Error!</h1><p class="p-4">There was an error on the server. Try refreshing your browser. If that doesn't work, come back at a later time. If the problem persists, you can seek help in the <a href="https://lemmy.ml/c/lemmy_support">Lemmy support community</a> or <a href="https://lemmy.ml/c/lemmy_support">Lemmy Matrix room</a>.</p></div></div><footer class="app-footer container-lg navbar navbar-expand-md navbar-light navbar-bg p-3"><div class="navbar-collapse"><ul class="navbar-nav ms-auto"><li class="nav-item"><span class="nav-link">UI: 0.19.9</span></li><li class="nav-item"><span class="nav-link">BE: </span></li><li class="nav-item"><a class="nav-link" href="/modlog">Modlog</a></li><li class="nav-item"><a class="nav-link" href="https://join-lemmy.org/docs/en/index.html">Docs</a></li><li class="nav-item"><a class="nav-link" href="https://github.com/LemmyNet">Code</a></li><li class="nav-item"><a class="nav-link" href="https://join-lemmy.org/">join-lemmy.org</a></li></ul></div></footer></div></div> <script defer src='/static/c15a0eb1/js/client.js'></script> </body> </html> bastion:~# curl http://lemmy:8536/ { "@context": [ "https://join-lemmy.org/context.json", "https://www.w3.org/ns/activitystreams" ], "type": "Application", "id": "https://mydomain.xyz/", "name": "MYDOMAIN", "preferredUsername": "mydomain.xyz", "inbox": "https://mydomain.xyz/inbox", "outbox": "https://mydomain.xyz/site_outbox", "publicKey": { "id": "https://mydomain.xyz/#main-key", "owner": "https://mydomain.xyz/", "publicKeyPem": "-----BEGIN PUBLIC KEY-----<SNIP>\n-----END PUBLIC KEY-----\n" }, "language": [], "published": "2025-02-17T15:42:53.548004Z" bastion:~# curl -I http://lemmy:8536/ HTTP/1.1 404 Not Found content-length: 0 cache-control: public, max-age=60 vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers date: Mon, 17 Feb 2025 20:06:03 GMT bastion:~# curl -I http://lemmy-ui:1234/ HTTP/1.1 500 Internal Server Error X-Powered-By: Express Content-Type: text/html; charset=utf-8 Content-Length: 5253 ETag: W/"1485-j9uo4fH1oBpLTZEATf6UDwWGY5M" Date: Mon, 17 Feb 2025 20:06:22 GMT Connection: keep-alive Keep-Alive: timeout=5
It seems Lemmy-UI is returning 500. This was run from a pod within the same Docker network:
# curl -I http://lemmy-ui:1234/ HTTP/1.1 500 Internal Server Error X-Powered-By: Express Content-Security-Policy: default-src 'self'; manifest-src *; connect-src *; img-src * data: blob:; script-src 'self' 'nonce-42930e15580b5234226abe8789f2c66f'; style-src 'self' 'unsafe-inline'; form-action 'self'; base-uri 'self'; frame-src *; media-src * data: Cache-Control: public, max-age=60 Content-Type: text/html; charset=utf-8 Content-Length: 5194 ETag: W/"144a-xS0fkaSDORsFqG8LgYer3xVM4T8" Date: Mon, 17 Feb 2025 19:48:22 GMT Connection: keep-alive Keep-Alive: timeout=5
But I see no errors from the pod itself:
$ docker compose logs lemmy-ui lemmy-ui-1 | Lemmy-ui v0.19.9 started listening on http://0.0.0.0:1234/ lemmy-ui-1 | 31 translation imports verified. lemmy-ui-1 | date-fns "zh" failed: unexpected format lemmy-ui-1 | 1 out of 94 date-fns imports failed. lemmy-ui-1 | 4 highlight.js imports verified. (Only testing 4 samples.)
NGINX logs:
75.154.243.157 - - [17/Feb/2025:19:45:43 +0000] "GET / HTTP/2.0" 500 5284 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:43 +0000] "GET /css/themes/darkly.css HTTP/2.0" 200 45877 "https://mydomain.xyz/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:43 +0000] "GET /css/themes/darkly.css.map HTTP/2.0" 400 24 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:43 +0000] "GET /manifest.webmanifest HTTP/2.0" 500 21 "https://mydomain.xyz/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:45 +0000] "GET /service-worker.js HTTP/2.0" 304 0 "https://mydomain.xyz/service-worker.js" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:48 +0000] "GET /login HTTP/2.0" 500 5325 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:48 +0000] "GET /css/themes/darkly.css HTTP/2.0" 200 45877 "https://mydomain.xyz/login" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:48 +0000] "GET /css/themes/darkly.css.map HTTP/2.0" 400 24 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:48 +0000] "GET /manifest.webmanifest HTTP/2.0" 500 21 "https://mydomain.xyz/login" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36" 75.154.243.157 - - [17/Feb/2025:19:45:50 +0000] "GET /service-worker.js HTTP/2.0" 304 0 "https://mydomain.xyz/service-worker.js" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
Any page I go to returns a 500. I suspect it's some nginx config not sending something to the backend or something. But still troubleshooting