MOSH – a simple SSH replacement that works when network conditions are bad

We at Johnny’s Software Lab LLC are experts in performance. If performance is in any way concern in your software project, feel free to contact us.

SSH is a de facto standard when it comes remote connections. It is secure and ubiquitous and many software developers use it daily to connect to their build servers, test servers or testing boards.

However one drawback of SSH is that the connection drops far too often on unreliable networks. If you’ve used SSH for some time, you will know what I am talking about: if your remote system is not in the same room as you, or you switch networks, or you are using VPN then you have for sure experienced connection drops. These things are nasty because you lose all your environment or the test batch has stopped executing and you need to retest everything.

There are several ways to workaround this, for example screen or tmux commands in Linux, but today I am going to write about MOSH.

So what is MOSH?

MOSH stands for Mobile Shell. It advertises as a drop-in replacement for SSH. In the background, MOSH uses SSH for authentication so if you can already connect to your remote server using SSH, there shouldn’t be any difficulties in connecting using MOSH.

MOSH uses SSH for authentication, but after the user is authenticated, MOSH uses its own protocol to transmit data between local and remote machine that doesn’t rely on SSH. MOSH creates an instance on the remote machine that is running your commands; in case of a connection drop your local MOSH instance will patiently wait for the network connection to come back up and then reconnect to it. In the meantime the remote instance will keep your environment safe and running .

Is MOSH right for me?

I’ve been using MOSH for a long time, and event though its being advertised as a drop in replacement for SSH, it simply isn’t. I installed MOSH manually on our remote Linux server and I used Chrome client in Windows to connect to it. I used the tool for several months and I am still using it but there are problems. Let’s go through the list:

• Pros:
  • The connection doesn’t break, ever. If there is a connection drop, the remote process keeps on running. You can switch networks, put your computer to sleep, even after several hours it reconnects and everything is alive and well.
  • It’s simple to use. You could achieve the same functionality with screen or dmux commands in Linux, but these are more complicated to set up.
  • There is a feature called local echo. Even if the network lags a lot, you won’t see any delay in typing. It uses some kind of input prediction to display your character as soon as they enter them and this works great!
  • Windows client looks nice.
• Cons:
  • There is no scroll back. The authors have explained why this is the case and there are several workarounds for this. One is to use tee command to keep the output of important commands in a file, other is to use tmux etc. But I still miss the scroll back and there are times when this is a deal breaker.
  • On Windows Client, there is a bug with pasting as this sometimes doesn’t work.
  • Windows Client is very light, you cannot save configurations, and this is a problem if you are working with several remotes.
  • You can use MOSH in interactive mode, but you won’t get the error code of your command like you do in SSH. So it is difficult to use in scripts.
  • When the connection breaks and after a while MOSH reconnects, it won’t sync the complete content of the terminal, just the last screen that is visible. This is good because terminal is available immediately, but you cannot see anything that happened before the last screen.

Installation

I will assume you have a local machine that is Windows or Linux, and the remote machine you are using is Linux based. Installing MOSH is a trivial task in case you are the machine administrator, but if you don’t administrator rights you can still install MOSH by building from sources and installing it into your home directory. Linux installation contains both server and client, on Windows only client is available.

Official installation instructions are available at mosh.org. But the installation is straightforward, so I will go quickly through it here.

Windows installation

On Windows, there is no native MOSH client, instead MOSH client is provided as a Chrome app. Search google for “mosh for chrome“. Install the app, open the new browser tab and click the Apps button in the Bookmark bar. You should see Mosh as one of the installed applications.

Linux installation as administrator

Mosh is already contained in most Linux repositories. On my Ubutnu system, I installed it with:

$ sudo apt-get install mosh

Depending on the Linux distribution your commands may vary, but the instructions are available on mosh.org.

Linux installation as non-administrator

If you’ve installed MOSH the simple way, congratulations. You can skip this chapter. Otherwise keep on reading.

you can install mosh for yourself only if you are not the administrator of the machine. In my company the administrator is a grumpy guy who doesn’t like changes, and will only install software that is absolutely necessary. In that case, here is how to do it.

Dependencies

First, create a directory that will store your local instance of mosh and its dependecies. In my case, I store a lot of software that I use locally in /home/$USER/usr. You can change the place of storage to suit your needs.

$ mkdir /home/$USER/usr

It would be a very long article if I have to spell out how to compile all the dependencies mosh needs from source. Instead, I will assume your environment has installations of gcc, g++, make, autoconf, libtool and pkgconfig (this is quite standard stuff and most development environments have them).

Typically what is not installed by default is protocolbuffers. Protocolbuffers is a library that MOSH needs, to install it follow the bellow steps.

wget https://github.com/protocolbuffers/protobuf/archive/v3.11.4.tar.gz
tar xfv v3.11.4.tar.gz
cd protobuf-3.11.4/
./autogen.sh
./configure --prefix=/home/$USER/usr
make
make install

Second thing you might also need to install is ncurses library. This is the way to do it:

wget https://ftp.gnu.org/gnu/ncurses/ncurses-6.1.tar.gz
tar xfv ncurses-6.1.tar.gz
cd ncurses-6.1/
./configure --prefix=/home/$USER/usr --with-shared --without-debug  --without-normal
make -j8
make install

After this, you should have everything you need to install MOSH.

Installing and compiling MOSH

Download mosh source code. This is the latest release available when this article is written but feel free to check mosh.org for a newer releases.

wget https://mosh.org/mosh-1.3.2.tar.gz
tar xfv mosh-1.3.2.tar.gz
cd mosh-1.3.2/

Before starting the configure script, you need to set up the environment variables telling the script where your protobuf and ncurses libraries are:

export PKG_CONFIG_PATH=/home/$USER/usr/lib/pkgconfig
export PATH=/home/$USER/usr/bin:$PATH
export CXXFLAGS="-I/home/$USER/usr/include/ncurses -I/home/$USER/usr/include"

Start the configure script, and them make it and install it.

./configure --prefix=/home/$USER/usr --with-curses=/home/$USER/usr
make
make install

After installation, open your ~/.bashrc or other file and export the environment variables by appending following commands to the end of the file:

export PATH=/home/$USER/usr/bin/:$PATH
export LD_LIBRARY_PATH=/home/$USER/usr/lib:$LD_LIBRARY_PATH

Log out and then log back in. Now you should have everything installed. You can test your binaries:

/home/$USER/usr/bin/mosh
/home/$USER/usr/bin/mosh-server

Connecting to MOSH server

I am making an assumption that you can connect from your client machine to your server machine through SSH. Remember that MOSH uses SSH for authentication, and if SSH is down or misconfigured, you will have to fix it first.

Connecting from Windows

So you’ve installed the Chrome client, now you want to connect. Start up your MOSH client (start Chrome, open an empty tab, click Apps and then click Mosh). The windows will appear where you will need to type in your login parameters.

If on the remote side you’ve installed MOSH through repositories, you just need to fill out username and hostname.

If you’ve installed MOSH manually on the remote, you will also need to fill out Mosh server command properly. Here how it’s done:

LANG=en_US.UTF-8 LD_LIBRARY_PATH=/home/$USER/usr/lib /home/$USER/usr/bin/mosh-server

We need to specify this because mosh-server is not on it default location, and it needs additional libraries.

Now just click Connect.

Connecting from Linux

You start mosh client by typing mosh in the command line:

mosh user@host

This works if the remote mosh server is installed through repositories, otherwise you just need provide the server command parameter:

mosh --server="LANG=en_US.UTF-8 LD_LIBRARY_PATH=/home/$USER/usr/lib /home/$USER/usr/bin/mosh-server" user@host

After this, you should be logged into your server and you can start your work.

If you get an error saying mosh-server needs a UTF-8 native locale to run, just run the command like this:

LC_ALL="en_US.UTF-8" mosh user@host

Final Words

When I originally came across MOSH, I was very enthusiastic. During the pandemic we were working from home, the internet connection was unreliable and would drop every hour or so. Our build system uses a complicated setup through environment variables, and every time there was a connection drop we had to set up the environment all over again. MOSH did help with this, but only partially.

If there is a need to run a batch of long running tests on the remote, this solution is perfect. I normally redirect the output of the testing to a file and leave it running. In case of connection break the tests keep on running and I can inspect the output file later. However, in my every day compile-test-debug cycle on the remote server, the lack of scroll back limits the usability of MOSH. Yet, it can be used then as well with a bit of hacking, and I do use it in case of unstable connection.

So the verdict? MOSH definitely has something useful to offer, but the usefulness is limited. Though it is worth a try and can prove to be a very useful tool in some circumstances.

Please feel free to leave your experiences with MOSH in the comments.

Do you need to discuss a performance problem in your project? Or maybe you want a vectorization training for yourself or your team? Contact us
Or follow us on LinkedIn , Twitter or Mastodon and get notified as soon as new content becomes available.

Further Read

Mosh.org Official Page

Install script for MOSH