Skip Navigation
Lemmy Support @lemmy.ml ZMonster @lemmy.world

Please, need help deploying a lemmy instance...

I have tried the docker, ansible, and scratch methods. I have been troubleshooting for a month now. I have gotten nowhere. I need someone to help walk me through how to deploy a lemmy server because the guides are absolute trash.

Please help. I'm wasting money running this VPS and for literally nothing.

Edit: So, I've tried the ansible method, but I can't access my server this way. It just keeps saying "UNREACHABLE". I have generated a dozen keys, none of them work. I have NO PROBLEMS with ssh in Putty. I can use Putty all day. Putty works fine using my ssh key. Ansible does not. No amount of new keys has made any difference. I have countless keys in my stupid droplet because of this hacky garbage.

53
53 comments
  • Do you have a local device you can try all this on first so you're not wasting money?

    I agree the guides aren't great, but they assume you have some experience doing this stuff.

    • but they assume you have some experience doing this stuff

      I can appreciate that, but I don't. So, if you are saying that I should just quit because I'm not an experience web developer, then thanks, and no thanks.

      For the better part of the last two decades, once or twice a year I throw a linux distro on a computer to see how it is. And of those several dozen times, EVERY SINGLE TIME, I have encountered an issue that needed troubleshooting and was never resolved. And every single time, I have found myself being advised that "you just have to know how to do this already". So I get what you are saying, but I don't. I don't have experience with this because it always ends up like this. I fucking hate windows, but it works. I can google a problem and find a solution within minutes. Literally any problem has a solution. Not so with linux. Just a lot of people telling me I just need to "know more".

  • I am willing to try any method you prefer. I've failed at all of them and have no preference. I don't care how I get it working, I just need to see something work. Please, please, please help!!!

    • Have you tried Lemmy-Easy-Deploy? This is not the method I used to deploy my own instance but I heard from some that this one is easy to use. It might be worth trying.

      • Yes, it worked, but as I've told others, the dev did not make it for a production use case. They said they intended it to be used by "people in college that are just testing". Several features don't work and are not going to be supported, like SMTP, which I will require. They also said they didn't intend to have more than one user of the instance... No idea who this usecase is for...

  • It seems like you have the keys configured for putty but not whatever SSH client Ansible uses.

    It's going to sound unhelpful, but have you tried running the Ansible stuff from WSL? This is tested on Linux (and macOS probably), but probably not Windows.

    You may also want to nuke the box of any previous state if you can. The NGINX errors you're getting are that it expects to find a LetsEncrypt certificate that doesn't exist yet. Typically you need to start NGINX without it first then certbot will generate it and update the NGINX config to use it.

    Some people have been using Lemmy-Easy-Deploy, it might be worth a try. It's supposed to be easier and more turnkey. The author is also very open to suggestions and help if the instructions are unclear.

    The docs are definitely geared towards people with existing server admin experience at the moment. There's talk on Matrix to improve those.

    Please post full logs of what you're running, it will be easier for people to troubleshoot, including everything before. The extra context can be very helpful to know what Ansible did before and give clues as to what it did previously that might fail. That stuff is a moving target, it gets updated frequently and may fail under some unanticipated conditions.

    • It’s going to sound unhelpful, but have you tried running the Ansible stuff from WSL? This is tested on Linux (and macOS probably), but probably not Windows. That's exactly what I'm using, WSL Ubuntu.

      And I've tried Lemmy-Easy-Deploy, and it worked fine, but the developer told me that they didn't intend for this to be used for an actual deployment instance with many users. I plan to scale this as large as I need. They also said that it did not support SMTP and that they had no intention of supporting SMTP. But without SMTP, users won't be able to confirm emails or reset passwords... So it just won't work.

      Please post full logs of what you’re running, it will be easier for people to troubleshoot, including everything before. The extra context can be very helpful to know what Ansible did before and give clues as to what it did previously that might fail. That stuff is a moving target, it gets updated frequently and may fail under some unanticipated conditions.

      Good point. I can do that. I have made so many impromptu modifications to my current server that troubleshooting it any further would surely cause errors. So I will blow up the droplet, start fresh, and get it to the first error and post back. Thanks for the advice!

      • Hmm, I see. In that case I'm afraid you will definitely need to get familiar with a lot of stuff really fast, because there's a lot of rough edges right now. It's super beta software after all. Even the admins of lemmy.world and beehaw.org are hitting pretty complex problems that needs a lot of digging. You'll definitely need to become fairly intimate with how Lemmy works under the hood if you want to scale up to hundreds of users. Especially the pict-rs storage as it grows pretty fast. People have been having to migrate it to object-storage.

        You can always join the install support Matrix room for additional help: https://matrix.to/#/#lemmy-support-new-instance:discuss.online. Or the whole Lemmy Matrix Space at #lemmy-space:matrix.org.

        I don't want to demotivate you or anything - it's a great learning experience! But beware you'll definitely have to troubleshoot and spend some time to keep it running smoothly. It's far from set and forget at the moment, particularly at scale.


        But one thing at a time, lets see how Ansible does on a blank droplet, and lets figure out how to fix your Ansible problems first before getting into manual installs and scaling up.

  • Could you tell us what has failed / did not work on your previous attempts? Also, what setup did you use, what reverse proxy you had in front of Lemmy if any, etc.

    • Okay, so it looks like nginx is a reverse proxy. That is the step I was on. It doesn't work. Last thing I did was to modify the docker-compose.yml file and change the port from 80 to 1236. I was told to do that by someone who has been trying to troubleshoot with me for 2 weeks now. It doesn't say to do that anywhere, but they told me to so I did. Then they said to "add this to my upstream" but I haven't a fucking clue what that means so I don't know what to do next. There is a config file in etc/nginx/sites-enabled/nginx.conf that the guide directed me to edit. This is the guide that I was directed to use to set up nginx. I was on the step where it requests the nginx status and mine just says "inactive". I promise, I am following these guides PRECISELY. I don't know what I am doing wrong, but all the solutions that I have received are things that the guide DOES NOT say to do. So I don't know where I am. I am happy to start fresh. You just let me know.

      • It doesn't tell you to actually run nginx, it assumes it will automatically run when installed. I don't know why, that sounds like dumb behaviour even if it were correct. You are right about the guides being trash.

        I recommend getting used to package manager (apt, dpkg) and system daemon/init system (systemd - accessed via systemctl) and then ignoring that guide. Installing and running nginx isn't complex enough to warrant a guide; installing packages and running services, in general, are just. Configuring nginx, however... If you know the concepts, it's pretty easy. The concepts are hard.

      • In my opinion its best not to touch the nginx that's set up by Lemmy and it's better to have another reverse-proxy in front of it.

        I'll try to come up with an solution later in the day, gotta do my daily at work.

    • Okay, I keep getting this question re: "reverse proxys", but I'm having trouble answering this because the guide does not use that language. There are no steps that direct me to edit, add, modify a "reverse proxy". So if there is a step in the guide that you know to be a "reverse proxy" then please tell me what it is. I am following the guides EXACTLY. There are a lot of missing steps, but most of these are things that were implied by the guide and not explicitly stated.

      I have tried the docker method. I have gotten close with this, as in, I can use the IP to access the instance, but I have not been able to get the domain to work. My A record is set up correctly. I used lemmy-easy-deploy to get it working the first time and it DID, but that dev does not support instances made for actual deployment so many features just don't work. It's not feasible at all. But my domain DID work. So I know my A record is set up correctly.

      I have tried the scratch method. That was a nightmare and nothing worked as expected. I am more than happy to try if you are willing to help me negotiate it.

      I have tried the ansible method. My local PC will not connect to the server with ansible. I have created a dozen new keys and NONE of them work. Ansible WILL NOT CONNECT. I have been using Putty to access the server and it works flawlessly, all day, every day. So I obviously have a working key. Ansible is not working. It just keeps saying "UNREACHABLE". I have googled this and found no solution that even remotely addressed what I am trying to do.

      I don't know what to do so you tell me and I'll fucking do it. Would you like me to start with the docker method? Would you like me to try the ansible method? Would you like me to try the scratch method? You tell me. I don't know. I don't care. I just want to get it working.

  • So, here's something that might work. I tested it on my local machine, up to Caddy but without HTTPS, but I'm confident it'll work once deployed on a server.

    Prerequisites:

    • Server with Docker and docker-compose installed
    • Ports 80 and 443 open and directed at your server
    • A domain name pointing to your server

    Setup

    First, create a folder and download the following files:

    Then, generate passwords for PostgreSQL and your admin user, store them somewhere safe.

    Config changes

    lemmy.hjson

    You'll want to change the admin_username, admin_password and site_name to match your primary user's credentials and the name you want to give your instance.

    Then, change hostname to match your domain name: if it is sub.domain.tld then it should read hostname: "sub.domain.tld".

    The base config file does not have proper configuration for the database, so you'll have to edit the database field as follows with the password you previously created:

      database: {
        host: postgres
        database: "lemmy"
        user: "lemmy"
        password: "POSTGRES_PWD" # Change for your password
      }
    

    Additionally, if you want to send emails for registration confirmation and password resets, add the following before the closing } and change to match your email provider configuration.

      email: {
          # Hostname and port of the smtp server
          smtp_server: "SMTP_SERVER"
          # Login name for smtp server
          smtp_login: "SMTP_LOGIN"
          # Password to login to the smtp server
          smtp_password: "SMTP_PASSWORD"
          # Address to send emails from, eg "noreply@your-instance.com"
          smtp_from_address: "SMTP_LOGIN"
          # Whether or not smtp connections should use tls. Can be none, tls, or starttls
          tls_type: "starttls"
        }
    

    docker-compose.yml

    By default the compose file is meant to build a development version of Lemmy, we will change this by removing the blocks with build and uncomment those with image. Note: think to update the images to 0.18.2 since it fixes some vulnerabilities.

    Also, since we will use a reverse proxy and I don't now if your server has a firewall, we should remove the ports blocks which are used to expose the services' ports on the host.

    Finally, make sure to change the POSTGRES_PASSWORD field to match the PostgreSQL password you set in lemmy.hjson.

    It should look something like that:

    version: "3.7"
    
    x-logging: &default-logging
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "4"
    
    services:
      proxy:
        image: nginx:1-alpine
        volumes:
          - ./nginx.conf:/etc/nginx/nginx.conf:ro,Z
        restart: always
        depends_on:
          - pictrs
          - lemmy-ui
        logging: *default-logging
    
      lemmy:
        # use "image" to pull down an already compiled lemmy. make sure to comment out "build".
        image: dessalines/lemmy:0.18.2
        # platform: linux/x86_64 # no arm64 support. uncomment platform if using m1.
        # use "build" to build your local lemmy server image for development. make sure to comment out "image".
        # run: docker compose up --build
    
        # this hostname is used in nginx reverse proxy and also for lemmy ui to connect to the backend, do not change
        hostname: lemmy
        restart: always
        environment:
          - RUST_LOG="warn,lemmy_server=debug,lemmy_api=debug,lemmy_api_common=debug,lemmy_api_crud=debug,lemmy_apub=debug,lemmy_db_schema=debug,lemmy_db_views=debug,lemmy_db_views_actor=debug,lemmy_db_views_moderator=debug,lemmy_routes=debug,lemmy_utils=debug,lemmy_websocket=debug"
          - RUST_BACKTRACE=full
        volumes:
          - ./lemmy.hjson:/config/config.hjson:Z
        depends_on:
          - postgres
          - pictrs
        logging: *default-logging
    
      lemmy-ui:
        # use "image" to pull down an already compiled lemmy-ui. make sure to comment out "build".
        image: dessalines/lemmy-ui:0.18.2
        # platform: linux/x86_64 # no arm64 support. uncomment platform if using m1.
        # use "build" to build your local lemmy ui image for development. make sure to comment out "image".
        # run: docker compose up --build
    
        # build:
        #   context: ../../lemmy-ui # assuming lemmy-ui is cloned besides lemmy directory
        #   dockerfile: dev.dockerfile
        environment:
          # this needs to match the hostname defined in the lemmy service
          - LEMMY_UI_LEMMY_INTERNAL_HOST=lemmy:8536
          # set the outside hostname here
          - LEMMY_UI_LEMMY_EXTERNAL_HOST=localhost:1236
          - LEMMY_UI_HTTPS=false
          - LEMMY_UI_DEBUG=true
        depends_on:
          - lemmy
        restart: always
        logging: *default-logging
        init: true
    
      pictrs:
        image: asonix/pictrs:0.4.0-beta.19
        # this needs to match the pictrs url in lemmy.hjson
        hostname: pictrs
        # we can set options to pictrs like this, here we set max. image size and forced format for conversion
        # entrypoint: /sbin/tini -- /usr/local/bin/pict-rs -p /mnt -m 4 --image-format webp
        environment:
          - PICTRS_OPENTELEMETRY_URL=http://otel:4137
          - PICTRS__API_KEY=API_KEY
          - RUST_LOG=debug
          - RUST_BACKTRACE=full
          - PICTRS__MEDIA__VIDEO_CODEC=vp9
          - PICTRS__MEDIA__GIF__MAX_WIDTH=256
          - PICTRS__MEDIA__GIF__MAX_HEIGHT=256
          - PICTRS__MEDIA__GIF__MAX_AREA=65536
          - PICTRS__MEDIA__GIF__MAX_FRAME_COUNT=400
        user: 991:991
        volumes:
          - ./volumes/pictrs:/mnt:Z
        restart: always
        logging: *default-logging
    
      postgres:
        image: postgres:15-alpine
        # this needs to match the database host in lemmy.hson
        # Tune your settings via
        # https://pgtune.leopard.in.ua/#/
        # You can use this technique to add them here
        # https://stackoverflow.com/a/30850095/1655478
        hostname: postgres
        command:
          [
            "postgres",
            "-c",
            "session_preload_libraries=auto_explain",
            "-c",
            "auto_explain.log_min_duration=5ms",
            "-c",
            "auto_explain.log_analyze=true",
            "-c",
            "track_activity_query_size=1048576",
          ]
        environment:
          - POSTGRES_USER=lemmy
          - POSTGRES_PASSWORD=password # Change with your password
          - POSTGRES_DB=lemmy
        volumes:
          - ./volumes/postgres:/var/lib/postgresql/data:Z
        restart: always
        logging: *default-logging
    

    Caddy

    For the final touch, we are going to setup Caddy, a reverse proxy with HTTPS support out of the box.

    First, create a file nammed Caddyfile and write the following in it:

    sub.domain.tld {
    	reverse_proxy http://proxy:1236
    }
    

    Make sure to match your actual domain name.

    Finally, update the docker-compose.yml file to add the following at the end (make sure that it's correctly tabulated)

      caddy:
        image: caddy:2.6.4
        restart: unless-stopped
        ports:
          - "80:80"
          - "443:443"
          - "443:443/udp"
        depends_on:
          - proxy
        volumes:
          - ./Caddyfile:/etc/caddy/Caddyfile:ro
          - caddy_data:/data
          - caddy_config:/config
    volumes:
      caddy_data:
      caddy_config:
    

    Launching the instance

    Before starting the stack, we have a few things left to do:

    • Create the folders for pictrs and postgres to store their data: mkdir -p volumes/postgres volumes/pictrs
    • Change the owner of volumes/pictrs: sudo chown -R 991:991 pictrs

    Finally, to start everything: docker compose up -d

  • No to demotivate you from trying it, but if you really want to use your instance instead of working on it, may I suggest you try a managed hosting provider?

    Communick has managed lemmy instances which are affordable, you don't need to worry about updates or backups and offer a free 14-day trial.

    (Obvious disclosure, I run this service)

  • putty

    That's your problem. Putty does keys differently to normal ssh.

    ansible

    docker

    .

    So, why can't you install lemmy? I'm going to assume you're using ubuntu. If you are willing to swap to arch (which I think DO offers), I recommend it. Get an AUR helper installed (a bit of hassle, I like trizen) and install lemmy from AUR. You'll need to bump the commit unless you like 0.17.3. I'll be doing that tonight so I can post relevant bits. If not, post errors.

    Or just keep doing containers so you can have weird and wonderful errors like no dns.

You've viewed 53 comments.