How to reconnect to a disconnected ssh session

Question

Is there a way to connect to an ssh session that was disconnected? We are having problems with our network connection to a remote site that we are working on separately; however, in the mean time we experience a large number of disconnects due to lost packets while connected to servers at the remote location. Many times the session stays active for a while, and sometimes it happens to be in the middle of some action (file editing, running some process, etc...) that I need to get back to rather than restart if possible.

Answer 1

Try to set ClientAliveInterval (e.g. 60) and TCPKeepAlive (yes or no) to appropriate values on the serverside sshd.conf .

This should keep your session alive even if the connection gets lost for minutes.

Answer 2

UPDATE: For an actual answer see zero_r's answer below

This isn't an answer, but a workaround. Use screen.

When you first log in, run screen. You get another shell, run commands in that. If you're disconnected, the screen process keeps the terminal alive so that your shell and the processes it is running don't fall over. When you reconnect, run 'screen -r' to resume.

There's a bunch more to configuring and using screen, but the above should workaround your problem.

Answer 3

As mentioned above, GNU Screen is the way to go. It allows you to have a 'screen session' on the remote box that you can run multiple commands in, via multiple 'screen windows'. This will simply detach if your parent SSH connection dies, keeping all the subprocesses running within it alive and well.

man screen is your friend as usual, and the OS package should be called screen if it is not installed by default.

Basics are:

• Start a screen session (on your remote host):

    $ screen
  

• Disconnect from your screen session: CTRL-A, d

• Reconnect to your screen session after logging back in again:

    $ screen -d -r
  

• Open another screen 'window': CTRL-A, c

• Cycle through your open screen windows: CTRL-A, space

There is lots of cool stuff you can do with screen. I've been using it for over 10 years, and I'm still finding out new features. It's my favourite Unix utility.

Answer 4

I can't believe no one has mentioned MOSH;

Mosh is a seperate protocol that can hook into the SSH login process, it keeps your session alive after days of disconnection, changing IP, high latency and so on. It is explained on the home page better than I can explain it so I have copied the description below. My experiences and advice are that I use it on my Android mobile, it's a life saver when travelling and SSH'ing. The same is true on my laptop when tethered with mobile on the train for example. I recommend compiling from source to get the latest version, the repo version for me inside Ubuntu has a few annoyances in it which are fixed in the newest version (at the time of writing).

Mosh (mobile shell)

Remote terminal application that allows roaming, supports intermittent connectivity, and provides intelligent local echo and line editing of user keystrokes.

Mosh is a replacement for SSH. It's more robust and responsive, especially over Wi-Fi, cellular, and long-distance links.

Mosh is free software, available for GNU/Linux, FreeBSD, Solaris, Mac OS X, and Android.

Features from the website:

• Change IP. Stay connected: Mosh automatically roams as you move between Internet connections. Use Wi-Fi on the train, Ethernet in a hotel, and LTE on a beach: you'll stay logged in. Most network programs lose their connections after roaming, including SSH and Web apps like Gmail. Mosh is different.

• Makes for sweet dreams: With Mosh, you can put your laptop to sleep and wake it up later, keeping your connection intact. If your Internet connection drops, Mosh will warn you — but the connection resumes when network service comes back.

• Get rid of network lag: SSH waits for the server's reply before showing you your own typing. That can make for a lousy user interface. Mosh is different: it gives an instant response to typing, deleting, and line editing. It does this adaptively and works even in full-screen programs like emacs and vim. On a bad connection, outstanding predictions are underlined so you won't be misled.

• No privileged code. No daemon: You don't need to be the superuser to install or run Mosh. The client and server are executables run by an ordinary user and last only for the life of the connection.

• Same login method: Mosh doesn't listen on network ports or authenticate users. The mosh client logs in to the server via SSH, and users present the same credentials (e.g., password, public key) as before. Then Mosh runs the mosh-server remotely and connects to it over UDP.

• Runs inside your terminal, but better: Mosh is a command-line program, like ssh. You can use it inside xterm, gnome-terminal, urxvt, Terminal.app, iTerm, emacs, screen, or tmux. But mosh was designed from scratch and supports just one character set: UTF-8. It fixes Unicode bugs in other terminals and in SSH.

• Control-C works great: Unlike SSH, mosh's UDP-based protocol handles packet loss gracefully, and sets the frame rate based on network conditions. Mosh doesn't fill up network buffers, so Control-C
  always works to halt a runaway process.

Answer 5

Sometimes I forgot to run screen too and losing my unfinished work. In this case, though we can't reattach to a broken SSH session, reparenting a running program to a new terminal and resuming what you were doing is still possible thanks to reptyr.

After accidentally disconnected from a SSH session, the first thing first is to run tmux (or whatever you like, but screen is not actively maintained and there are security flaws) lest the connection gets broken again. Then in the new session, run ps aux | grep {The process to be resumed} to get the PID. With the PID, you could try reptyr {PID} or reptyr -T {PID} (if there are subprocesses) to continue the work.

---

Additional notes:

Also, please consider setting ClientAliveInterval (e.g. 60) and TCPKeepAlive (yes) to appropriate values on the serverside sshd.conf as mentioned in zero_r's answer. Or, if you have to work in a network env with a sky-high packet loss rate like Starlink in storm, switch to MOSH, a SSH alternative but based on UDP.

Answer 6

autossh watches your connection and if it goes down, it reconnects. It is more reliable than keepalives. If you connect to a screen session, you'll continue right from where you disconnected (see rscreen that comes with autossh)

Answer 7

tmux

This one is a classic. Use it whenever you run the risk of losing connection to a terminal.

$ tmux
$ sh do_something_that_takes_forever
!! Connection fails so you reconnect once you notice
$ tmux ls
0: 1 windows (created Tue Aug 23 12:39:52 2011) [103x30]
$ tmux attach -t 0

Just like that, you're back in action.

Answer 8

Use reptyr to reparent a running program to a new terminal.

HOW TO:

Lets assume that we have connected to ssh from some previous IP address aaa.bbb.ccc.old,
then run aptitude process, and in the middle of the work that connection got broken.

We login with ssh again (optionally from a different IP address aaa.bbb.ccc.new).
The current connection uses pty/1 pseudo-terminal:

$ tty
/dev/pts/1

But there are 2 open connections, the current pts/1 one, and the old broken one using pts/0:

$ who
user     pts/0        2022-02-22 22:00 (aaa.bbb.ccc.old)
user     pts/1        2022-02-22 22:11 (aaa.bbb.ccc.new)

Lets list all shells and their subprocesses:

$ ps auf
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
user        2000  1.3  0.5  10168  5236 pts/1    Ss   22:11   0:00 -bash
user        2001  0.0  0.3  10620  3304 pts/1    R+   22:11   0:00  \_ ps auf
user        1000  0.0  0.5  10168  5132 pts/0    Ss   22:00   0:00 -bash
user        1001  5.8 16.0  46639 15839 pts/0    Sl+  22:00   0:03  \_ aptitude
root         820  0.0  0.1   5836  1840 tty1     Ss+  22:00   0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
root         810  0.0  0.2   7360  2260 ttyS0    Ss+  22:00   0:00 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220

There are 2 bash shells:
the current pts/1 one with PID 2000 running ps auf
and the old broken pts/0 one with PID 1000 running aptitude

Lets reconnect to the old one with reptyr:

$ sudo reptyr -T 1000

This way we gain access to the old shell and its subprocesses.
Since aptitude uses ncurses, you may need to press Ctrl+L or resize the terminal emulator's window to force it to repaint itself.
Now we can quit aptitude, and then exit the old shell (1000),
thus returning to our new shell (2000) with only 1 connection remaining, which is the current one:

$ who
user     pts/1        2022-02-22 22:11 (aaa.bbb.ccc.new)

Answer 9

I'd install and start screen to fix your problem. Screen will let you reconnect to a previous screen session.

Apart from that, screen also let's you do cool things like split your screen, view the console etc. You can find more info here and here.

For starters, if you get disconnected, you can use

screen -ls

to view your sessions and

screen -r ${session} 

to reconnect to a disconnected one.

Answer 10

While screen will keep your shell session open on the remote server if your ssh session drops, it won't do anything about the problem of ssh connections being dropped. As zero_r suggests, try tuning your ssh connection with keep alives and long timeouts.

I suggest you track down the cause of the lost packets causing the problems and fix that instead of working around it.

Answer 11

This answer is still relevant, screen exists, is supported, still does the job, etc, but there are other alternatives now: tmux offers extra features, and both it and screen are more commonly (in my experience) used wrapped in Byobu. All three are available in the standard repositories of major Linux distributions.

---

As others have pointed out, screen is generally the best solution for this and it adds a host of other useful features too.

You can setup your profile on the remote machine to automatically start and/or reconnect to screen on login, which saves you forgetting to start screen the one time you need it because you suffer a connection drop.

See http://tlug.dnho.net/?q=node/239 (or search Google for many other examples dones slightly different ways).

Answer 12

a more modern alternative to screen, alas not available to some types of "virtualisation" (e.g. in cygwin you can have "screen", but not "tmux" due to the way it's designed), but wherever you have the option to install tmux, I'd strongly recommend to go for that one insted of screen.

Answer 13

Here's another non-screen solution.

Putty tray allows you to do just that, it has a reconnect on disconnect option that isn't in regular or putty-ng

https://puttytray.goeswhere.com/

it's a (completely open source) fork of putty with other options, go to connection settings and there are 2 options, one for 'attempt to reconnect on connection failure', and 'attempt to reconnect on startup'.

Answer 14

Yet another option is to run Jupyter Notebook, which has its own pretty good web-based terminal emulator.

Upsides

• When I reattach my vpn I can re-open all my connections across several terminals on several hosts by opening all bookmarks in a folder.
• I can open the terminal on any device and it looks decent on any device

Downsides

• I do lose the console history (not the commands, just the scrolling) after reconnecting.

How to

1. On some_host, run jupyter-notebook --no-browser --port=$SOME_PORT &.

2. create session some_label in your browser by pointing to http://some_host:$SOMEPORT/terminals/some_label. You can now disconnect.

3. Bookmark your sessions. When you reconnect, you can open them all at once. If the tabs are already open, you will need to refresh them.

If $SOME_PORT is not open on some_host, then use ssh to port-forward and connect to http://localhost:$SOME_LOCAL_PORT/terminals/some_label instead:

ssh -q -N -f -L localhost:$SOME_LOCAL_PORT:localhost:$SOME_PORT $some_host

Answer 15

My problem was simple to solve. On my /etc/sysconfig/network-scripts/ifcfg-eth0 was wrong with the parameter IPADDR set to a static IP and BOOTPROTO=dhcp. After updating the BOOTPROTO=static parameter the problem was solved.