nohup over SSH

In case it helps anyone, I want to share about a problem I faced which had me stumped for a bit - before I encountered the solution, which is non-obvious to say the least.

You can run commands on a remote host over ssh like so:

ssh user@host "echo 'Hello World'"

Also, to run a command or script in the background, a frequently-seen command is the following:

nohup script &

where nohup ensures that the command isn’t interrupted by the hangup signal (which can occur due to a number of reasons, such as user logging off) while the ending & moves the shell command to the background.

However, the following does not work as expected:

ssh user@host "nohup ./script &"

The ssh session terminates after some time, but the script process is not started on the remote host.

It turns out that the reason why this occurs is that the ssh server refuses to lose any I/O to and from the newly created remote process - this causes it to hang as it gets stuck waiting. Eventually, the ssh session terminates and the process is killed.

To avoid this, and to have the command execute in the background, we need to redirect all 3 I/O streams (stdin, stdout, stderr) so that ssh no longer gets stuck waiting for these data streams:

ssh user@host "nohup ./script &> script.out < /dev/null &"

This creates the new process in the background as expected, and closes the ssh session.

(thanks askubuntu and wikipedia for pointing me in the right direction)