why can't I connect to my ssh server UNLESS I enter eval "$(ssh-agent -s)" first?

The whole point of ssh-agent is to remember your passphrase. If you don't want to do that your problem might be that for some reason ssh client doesn't pick up your key. Try defining it for the host

Also, there's -v flag for ssh. Use it to debug what's going on when it doesn't try to use your key

okay I tried that, using -i to specify private key. I get the same thing: blank / blinking cursor. When I use verbose -v flag, I see that in all cases (using -i, the config file, and originally) it ends with these two lines (after about 50 lines) :

debug1: Offering public key: /home/username/.ssh/id_rsa RSA SHA256:j3MUkYzhTrjC6PHkIbre3O(etc) agent debug1: Server accepts key: /home/username/.ssh/id_rsa RSA SHA256:j3MUkYzhTrjC6PHkIbre3OT(etc) agent

where (etc) is some redacted text. It seems the server is ACCEPTING the key, which is nice. But then it's still a blinking cursor...
- It seems the server is ACCEPTING the key
  
  Check if it is true. In the server logs.
The whole point of ssh-agent is to remember your passphrase.

replace passphrase with private key and you're very correct.

passphrases used to login to servers using PasswordAuthentication are not stored in the agent. i might be wrong with technical details on how the private key is actually stored in RAM by the agent, but in the context of ssh passphrases that could be directly used for login to servers, saying the agent stores passphrases is at least a bit misleading.

what you want is:

use Key authentication, not passwords

disable passwordauthentication on the server when you have setup and secured (some sort of backup) ssh access with keys instead of passwords.

if you always want to provide a short password for login, then don't use an agent, i.e. unset that environment variable and check ssh_config

give your private key a password that fits your needs (average time it shoulf take attackers to guess that password vs your time you need overall to exchange the pubkey on all your servers)

change the privatekey every time immediately after someone might have had access to the password protected privkey file

do not give others access to your account on your pc to not have to change your private key too often.

also an idea:

use a token that stores the private key AND is PIN protected as in it would lock itself upon a few tries with a wrong pin. this way the "password" needed to enter for logins can be minimal while at the same time protecting the private key from beeing copied. but even then one should not let others have access to the same machine (of course not as root) or account (as user, but better not at all) as an unlocked token could also possibly be used to place a second attacker provided key on the server you wanted to protect.

all depends on the level of security you want to achieve. additional TOTP could improve security too (but beware that some authenticator providers might have "sharing" features which could compromise the TOTP token even before its first use.
FWIW, I've found that the -v flag often doesn't say why it's not using your key, just that it isn't using your key and it has fallen back to password authentication.

It's usually not terribly helpful for figuring out why it's not using your key, just that it's not using your key, which you kind of already know if it's prompting you for a password. lol
- Because it's basically axiomatic: ssh uses all keys it knows about. The system can't tell you why it's not using something it doesn't know it should be able to use. You can give a -i for the certificate to check if it doesn't know it because the content is broken or the location.
  
  That said: this doesn't make -v more useful for cases like this, just because there's a reason!
Not OP but everytime I used the verbose output of ssh it didn't help me one bit. Even adding outrageous verbosity I was still quite confused on what step failed and which didn't.

I'm probably just bad at understanding SSH but i don't know it seems like ssh workflow includes many trial and error until it finds a way to connect.

Imo the verbose output of SSH is often not very helpful if you don't know very well ssh in the first place. Obviously it is still worth a shot and a good advice but don't expect ssh to clearly state what is going on :)
- Well, you have configuration and flag options to define what is it supposed to be trying to use. What order, I think too. But definitely understanding SSH a little bit will make the log more understandable. As with everything tbh :D

Maybe ssh can't find the key automatically. What is the path to your private key?

I'm pretty sure I generated it to ~/.ssh/id_rsa which I think the default location. It is also the location shown in the terminal image in my post.
- I think some distros disable using RSA by default. Might need to use it explicitly.

As mentioned, -v (or -vv) helps to analyze the situation.

My theory is that you already have something providing ssh agent service, but that process is somehow stuck, and when ssh tries to connect it, it doesn't respond to the connect, or it accepts the connection but doesn't actually interact with ssh. Quite possibly ssh doesn't have a timeout for interacting with ssh-agent.

Using eval $(ssh-agent -s) starts a new ssh agent and replaces the environment variables in question with the new ones, therefore avoiding the use of the stuck process.

If this is the actual problem here, then before running the eval, echo $SSH_AUTH_SOCK would show the path of the existing ssh agent socket. If this is the case, then you can use lsof $SSH_AUTH_SOCK to see what that process is. Quite possibly it's provided by gnome-keyring-daemon if you're running Gnome. As to why that process would not be working I don't have ideas.

Another way to analyze the problem is strace -o logfile -f ssh .. and then check out what is at the end of the logfile. If the theory applies, then it would likely be a connect call for the ssh-agent.

which ssh

which ssh-agent
- I guess it's worth checking if those names point to the expected binaries, but I also think it would be highly unlikely they would be anything else than just /usr/bin/ssh and /usr/bin/ssh-agent.
I didn't really follow the former part, but I can give you this:

strace -o logfile -f ssh -p 8322 pi@192.168.2.223 of when I get blank
- Please don't ignore the advice about SSH_AGENT_SOCK. It'll tell yoy what's going on (but not why).
- At the end of the log you find:
  
  822413 connect(4, {sa_family=AF_UNIX, sun_path="/run/user/1000/gcr/ssh"}, 110) = 0 ... 822413 read(4,
  
  meaning it's trying to interact with the ssh-agent, but it (finally) doesn't give a response.
  
  Use the lsof command to figure out which program is providing the agent service and try to resolve issue that way. If it's not the OpenSSH ssh-agent, then maybe you can disable its ssh-agent functionality and use real ssh-agent in its place..
  
  My wild guess is that the program might be trying to interactively verify the use of the key from you, but it is not succeeding in doing that for some reason.
My theory is that you already have something providing ssh agent service

in the past some xserver environments started an ssh-agent for you just in case of, and for some reason i don't remember that was annoying and i disabled it to start my agent in my shell environment as i wanted it.

also a possibility is tharlt there are other agents like the gpg-agent that afaik also handles ssh keys.

but i would also look into $HOME/.ssh/config if there was something configured that matches the hostname, ip, or with wildcards* parts of it, that could interfere with key selection as the .ssh/id_rsa key should IMHO always be tried if key auth is possible and no (matching) key is known to the ssh process, that is unless there already is something configured...

not sure if a system-wide /etc/ssh/ssh_config would interfere there too, maybe have a look there too. as this behaviour seems a bit unexpected if not configured specially to do so.
- I am not sure I "solved" this but when I add this to my startup script for my terminal (~/.zshrc):
  
  SSH_AUTH_SOCK=/tmp/ssh-agent-$USER-socket export SSH_AUTH_SOCK
  
  it works then. I am not sure I'm still using the ssh agent, but at least it also does not cache my passphrase.

Without the ssh-agent invocation:

what does ssh-add -L show?
what is the original SSH_AUTH_SOCK value?
what is listening to that? (Use lsof)

This kind of stuff often happens because there's a ton of terrible advice online about managing ssh-agent - make sure there's none if that baked into your shellrc.

ssh-add -L ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqS5l(redacted)f0phb8x+fUV1w== username@computername

echo $SSH_AUTH_SOCK /run/user/1000/gcr/ssh

lsof $SSH_AUTH_SOCK COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME gcr-ssh-a 778406 username 3u unix 0x000000007e25ee6b 0t0 30290 /run/user/1000/gcr/ssh type=STREAM (LISTEN) gcr-ssh-a 778406 username 6u unix 0x0000000020f5b559 0t0 2096642 /run/user/1000/gcr/ssh type=STREAM (CONNECTED) gcr-ssh-a 778406 username 12u unix 0x00000000a6756d60 0t0 2100347 /run/user/1000/gcr/ssh type=STREAM (CONNECTED) gcr-ssh-a 778406 username 15u unix 0x00000000625cb05a 0t0 2261237 /run/user/1000/gcr/ssh type=STREAM (CONNECTED) gcr-ssh-a 778406 username 21u unix 0x00000000d0b214f9 0t0 2261238 /run/user/1000/gcr/ssh type=STREAM (CONNECTED) gcr-ssh-a 778406 username 23u unix 0x00000000a2f197fe 0t0 2349665 /run/user/1000/gcr/ssh type=STREAM (CONNECTED) gcr-ssh-a 778406 username 27u unix 0x00000000da22a130 0t0 2349668 /run/user/1000/gcr/ssh type=STREAM (CONNECTED) gcr-ssh-a 778406 username 29u unix 0x000000004f7a1723 0t0 2365382 /run/user/1000/gcr/ssh type=STREAM (CONNECTED) gcr-ssh-a 778406 username 33u unix 0x00000000e26976b3 0t0 2365389 /run/user/1000/gcr/ssh type=STREAM (CONNECTED) gcr-ssh-a 778406 username 35u unix 0x00000000b8185a8a 0t0 2375648 /run/user/1000/gcr/ssh type=STREAM (CONNECTED) gcr-ssh-a 778406 username 39u unix 0x00000000ba41030c 0t0 2375649 /run/user/1000/gcr/ssh type=STREAM (CONNECTED) gcr-ssh-a 778406 username 41u unix 0x000000006867cb01 0t0 2380999 /run/user/1000/gcr/ssh type=STREAM (CONNECTED) gcr-ssh-a 778406 username 45u unix 0x0000000091384b95 0t0 2381008 /run/user/1000/gcr/ssh type=STREAM (CONNECTED) gcr-ssh-a 778406 username 47u unix 0x00000000d5b28b08 0t0 3729149 /run/user/1000/gcr/ssh type=STREAM (CONNECTED) gcr-ssh-a 778406 username 51u unix 0x00000000f65088aa 0t0 3731006 /run/user/1000/gcr/ssh type=STREAM (CONNECTED)

All before issuing the ssh-agent
- It's the gnome key ring ssh agent.
  
  It's possible that this has popped up a window asking gor permission / a passphrase / something and you're not seeing that.
- Search for /run/user/1000/gcr/ssh on the Internet. I'm on my phone and didn't find the solution, but I'm sure you'll find it.
- Okay, that agent process is running but it looks wedged: multiple connections to the socket seem to be opened, probably your other attempts to use ssh.
  
  The ssh-add output looks like it's responding a bit, however.
  
  I'd use your package manager to work out what owns it and go looking for open bugs in the tool.
  
  (Getting a trace of that process itself would be handy, while you're trying again. There may be a clue in its behaviour.)
  
  The server reaponse seems like the handshake process is close to completing. It's not immediately clear what's up there I'm afraid.

please, it's eval "$(ssh-agent -s)" (quotes!)

well seems to work without tho

edit: made no difference, but I changed it in the post title.
- Just because it works, doesn't mean it's right.
- I had a similar construct in my bashrc and forgot the quotes. It didn't throw an error but also didn't work. Took quite a while to find the issue. So personally, I would recommend trying to quote correctly whenever possible.
- I was unclear: I did not mean to imply that it will work with it.
  
  It's OT, but I'll clarify since it might be useful for people who find Bash cryptic.
  
  Thing is, roughly speaking:
  
  eval will evaluate its first argument as Bash code
  
  eval "$(any_command really)" will run run any_command really, take its output and then use it as first argument for eval. So the assumption is that any_command really must output a valid Bash code snippet.
  
  So what eval "$(ssh-agent -s)" really means is, "run ssh-agent -s, collect the output and run it right here, where we are calling eval. Compare to ssh-agent -s | bash -- this would also run ssh-agent output but it would run it in a new process--a child process of the current process---so the whatever the snippet would be, it would have no way of affecting state of the parent program, which is why it's safer.
  
  Aside: The reason we need eval in this case is that we actually need to affect state of the program: that's the whole point. We need to set several environment variables to values that ssh-agent "knows". Without eval we would have to "ask" ssh-agent separately for each value (I'm assuming it's not even supported) and then set all these envvars using eg. export keyword. Using eval we let ssh-agent dictate the whole process: which variables are going to be set to what values, with the caveat that if compromised, it could do "evil" stuff like setting PATH to override common commands with compromised code. etc.
  
  So what's the problem with the quotes? The Shell syntax, foo "$(bar baz)" will make sure that the thing between quotes is
  
  kept verbatim
  
  treated as a single argument, even if it contains newlines (with some ugly exceptions to this regarding the final newline)
  
  Now without quotes, Bash (as well as POSIX shell) actually have several things they can do with the output (read man bash for full list, but keep it for a long rainy evening). Some of it involves substituting eg. values like * with matching filenames, some of it may involve actually splitting the output to separate arguments based on spaces or other special characters (which can even be different characters depending on current state, see IFS and the likes).
  
  You can see the difference, if you run eg. printf '[%s]\n' instead of eval. This printf syntax will simply print all of following arguments on a separate line, adding braces before and after. You can compare
  
  printf '|%s|\n' $(ssh-agent -s) # printf will probably receive multiple extra arguments printf '|%s|\n' "$(ssh-agent -s)" # printf will receive just one extra argument (and print it as specified)
  
  (both of these commands should be safe as long as ssh-agent is not compromised and as long I have not made any terrible typo)

Can you try killall ssh on the client, and then try to ssh into the rpi again?

tried.

what are your ssh config settings: ~/.ssh/config or /etc/ssh/ssh_config

I just added the ~/.ssh/config file on client side:

Host pidoos HostName 192.168.2.223 User pi IdentityFile ~/.ssh/id_rsa

Same result.

The /etc/ssh/ssh_config is only relevant on the server side, right? Well, here it is.
- sshd_config is server side, ssh_config is client side AFAIK
  
  Your config looks pretty tame. Anything interesting in /etc/ssh/config.d/ ?

Where is the key? What are the permissions for it?

it got 600 both the private and public key, stored in ~/.ssh/
- Often permissions can be an issue. I'd check permission for directory and files and hinr directory for user and group.
  
  Sample permissions here: https://jonasbn.github.io/til/ssh/permissions_on_ssh_folder_and_files.html
  
  Many. Ssh issues I've had have been permissions issues.

are you using fish shell?

I have zsh
- https://github.com/ohmyzsh/ohmyzsh/tree/master/plugins/ssh-agent

The only reason ssh client would "hang" without any output is when it's waiting for external key storage to allow access. It's designed that way to give user some time to approve access to key storage.

It sometimes happen that the installed key storage is broken in a way that it fails to show user modal, for any reason (showing on wrong screen, wrong desktop, wrong activity, wrong framebuffer, ....)

One solution (that you already did) is to change the SSH agent env variable to point to different key storage.

Another would be (if possible) to uninstall the broken key storage if you don't use it. But it is sometimes needed/used by other apps.

It's overall good to notify/open bug on your distro issue tracker to notify that some packages are missconfigured (maybe have missing dependencies) or conflicts with other ones.

What happens if you run commands on that blinking cursor? E.g. it you run ls do you get an output? I've had that happen in the past, don't remember the reason though.

also no output

I am not sure I "solved" this but when I add this to my startup script for my terminal (~/.zshrc):

SSH_AUTH_SOCK=/tmp/ssh-agent-$USER-socket
export SSH_AUTH_SOCK

it works then. I am not sure I'm still using the ssh agent, but at least it also does not cache my passphrase (or private key in ram)

Try running ssh with -vv to get a better idea of the problem when no ssh agent is running.

Can you post the result of the env command as well? It sounds like your config is very minimal, but the fact that it's looking for a local Unix socket in the strace output is weird.

SYSTEMD_EXEC_PID=3980
SSH_AUTH_SOCK=/run/user/1000/gcr/ssh
SESSION_MANAGER=local/computername:@/tmp/.ICE-unix/3857,unix/computername:/tmp/.ICE-unix/3857
GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/668d083f_5746_4268_beac_1ab8e7e69305
GTK3_MODULES=xapp-gtk3-module
XDG_CURRENT_DESKTOP=GNOME
LANG=en_GB.UTF-8
GRADLE_HOME=/usr/share/java/gradle
LC_IDENTIFICATION=en_GB.UTF-8
DISPLAY=:1
QT_QPA_PLATFORMTHEME=qt5ct
COLORTERM=truecolor
QT_IM_MODULE=ibus
USER=username
LC_MEASUREMENT=en_GB.UTF-8
XDG_MENU_PREFIX=gnome-
HOME=/home/username
PWD=/home/username
DESKTOP_SESSION=gnome
XDG_SESSION_CLASS=user
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
LC_NUMERIC=en_GB.UTF-8
XDG_DATA_DIRS=/home/username/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share/:/usr/share/
WINDOWPATH=2
XDG_SESSION_DESKTOP=gnome
VTE_VERSION=7602
LC_TIME=en_GB.UTF-8
MAIL=/var/spool/mail/username
DEBUGINFOD_URLS=https://debuginfod.archlinux.org 
QT_AUTO_SCREEN_SCALE_FACTOR=1
LC_PAPER=en_GB.UTF-8
LOGNAME=username
MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.SettingsDaemon.MediaKeys.service/memory.pressure
MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=
XMODIFIERS=@im=ibus
GNOME_TERMINAL_SERVICE=:1.870
XDG_SESSION_TYPE=x11
SHELL=/bin/zsh
XDG_RUNTIME_DIR=/run/user/1000
LC_MONETARY=en_GB.UTF-8
EDITOR=/usr/bin/nano
PATH=/home/username/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/var/lib/flatpak/exports/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
USERNAME=username
GNOME_KEYRING_CONTROL=/run/user/1000/keyring
LC_TELEPHONE=en_GB.UTF-8
GDM_LANG=en_GB.UTF-8
LC_NAME=en_GB.UTF-8
MOTD_SHOWN=pam
TERM=xterm-256color
GDMSESSION=gnome
XAUTHORITY=/run/user/1000/gdm/Xauthority
LC_ADDRESS=en_GB.UTF-8
SHLVL=1
OLDPWD=/home/username
LESS_TERMCAP_mb=
LESS_TERMCAP_md=
LESS_TERMCAP_me=
LESS_TERMCAP_se=
LESS_TERMCAP_so=
LESS_TERMCAP_ue=
LESS_TERMCAP_us=
LESS=-R
LS_OPTIONS=--color=auto
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.7z=01;31:*.ace=01;31:*.alz=01;31:*.apk=01;31:*.arc=01;31:*.arj=01;31:*.bz=01;31:*.bz2=01;31:*.cab=01;31:*.cpio=01;31:*.crate=01;31:*.deb=01;31:*.drpm=01;31:*.dwm=01;31:*.dz=01;31:*.ear=01;31:*.egg=01;31:*.esd=01;31:*.gz=01;31:*.jar=01;31:*.lha=01;31:*.lrz=01;31:*.lz=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.lzo=01;31:*.pyz=01;31:*.rar=01;31:*.rpm=01;31:*.rz=01;31:*.sar=01;31:*.swm=01;31:*.t7z=01;31:*.tar=01;31:*.taz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tgz=01;31:*.tlz=01;31:*.txz=01;31:*.tz=01;31:*.tzo=01;31:*.tzst=01;31:*.udeb=01;31:*.war=01;31:*.whl=01;31:*.wim=01;31:*.xz=01;31:*.z=01;31:*.zip=01;31:*.zoo=01;31:*.zst=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:
P9K_SSH=0
_P9K_SSH_TTY=/dev/pts/0
P9K_TTY=old
_P9K_TTY=/dev/pts/0
_=/usr/bin/env

What happens if you do:

unset SSH_AUTH_SOCK

And then try to connect?

I have no idea why but ssh seems to not use keys with different names by default

can you expand on that? What do you mean different names? My PC has of course a different username than the server I'm connecting to. The label name at the end of the key is just a comment, so this is also not what you're referring to, I think.
- By default the key is named id.rsa and ssh-client may only load that. Or none at all. Very strange

@dysprosium ssh agent manages your ssh keys and automatically passes them as an identity when connecting to a server

If you want to connect without it, you can simply pass -i \<path to private key\> flag

okay I tried that, using -i to specify private key. I get the same thing: blank / blinking cursor. When I use verbose -v flag, I see that in BOTH cases (I see about 50 lines) it ends with these two lines:

debug1: Offering public key: /home/username/.ssh/id_rsa RSA SHA256:j3MUkYzhTrjC6PHkIbre3O(etc) agent debug1: Server accepts key: /home/username/.ssh/id_rsa RSA SHA256:j3MUkYzhTrjC6PHkIbre3OT(etc) agent

where (etc) is some redacted text. It seems the server is ACCEPTING the key, which is nice. But then it’s still a blinking cursor…
- @dysprosium Mind trying with -vvvv flag and sharing the output instead of -v?

Your shell for user pi may be broken. Try adding the shell command to your ssh command explicitly like

ssh pi@host /bin/sh

Or use /bin/bash