Discover how to implement simple job control in bash to run multiple processes simultaneously, restart them if they crash, and ensure they all terminate together.

Simple Job Control in Bash

In this guide, we’ll explore how to use bash to manage multiple processes as part of a system test suite. We’ll ensure they restart if they crash and stop together if the foreground process is terminated.

The Problem

For our system test suite, we need to run several programs concurrently, ensuring they restart upon crashing and all terminate if the foreground process stops. Traditional tools like monit, upstart, or systemd are unsuitable as these are local, non-system processes typically run from a Makefile. Even the forever tool was inadequate for non-node processes. Thus, we turn to bash.

The Plan

We will run a few netcat instances, each listening on a different port. These instances will act as stand-ins for our real processes. By avoiding the -k (keep listening) switch, we can quickly terminate them by sending data.

To send data, use the command:

echo "Hello World!" > /dev/tcp/localhost/9220

The Script

Here’s the bash script to manage the processes:

#!/bin/bash

# When the script exits, kill the current process group.
trap "kill -- -$BASHPID" EXIT

# Run the command in the background.
# If it stops, restart it.
(while true; do
    nc -l 9220
done) &

(while true; do
    nc -l 9221
done) &

# Wait indefinitely (for Ctrl+C).
cat

Adding a Delay Before Restart

To introduce a short delay before restarting the process, modify the script as follows:

(while true; do
    nc -l 9220
    sleep 1
done) &

Restart Only on Non-Zero Exit Status

To restart the process only if it exits with a non-zero status, use:

(while nc -l 9220; do
    sleep 1
done) &

By following these steps, you can efficiently manage multiple processes in bash, ensuring they restart if they crash and all stop together when needed.