Documentation

Table of contents:

• Documentation
  • Table of contents:
  • Service configuration
    • Service templating
    • Main section
      • Restart section
    • Healthiness Check
    • Failure section
    • Environment section
    • Termination section
    • ResourceLimit section
  • State machine
  • Horust’s configuration
  • Running a single command
  • Multiple service directories
  • horustctl: Checking system status
  • Plugins (WIP)

When starting horust, you can optionally specify where it should look for services and uses /etc/horust/services by default.

Service configuration

This section describes all the possible options you can put in a service.toml file. You should create one different service.toml for each command you want to run. Apart from the user parameter, everything should work even with an unprivileged user.

Service templating

Services can, but not have to, be templated. Currently, this feature works only via environment variables. The templating engine uses bash expansion mechanism. Each part of the service configuration can be used in tandem with the templating. Additionally, multiple variables can be safely used if needed. The engine does not support processing the shell queries, for example $(cat /proc/config.gz) will not be processed and will be used on face value.

Before loading the service file, Horust will internally search and replace every value with environment variable if it exists. In this case, the ${USER} block is replaced by the environment’s USER variable. Alternatively $USER without braces is also accepted, although heavily discouraged. Variable names must follow rules of the operating system, which means that, in case of bash, variables are case-sensitive, and usage of special characters is somewhat restricted. You can use any variable your system provides, or you also can set and/or export those before running Horust.

Below are some simple strings that will be properly templated. Simple feature usage in configuration is showcased in the provided sample service.

...
user = "${USER}"
stderr = "${HOME}${HORUST_LOGDIR}/stderr"
...

Main section

# name = "myname"
command = "/bin/bash -c 'echo hello world'"
start-delay = "2s"
start-after = ["database", "backend.toml"]
shutdown-after = ["database"]
stdout = "STDOUT"
stderr = "/var/logs/hello_world_svc/stderr.log"
stdout-rotate-size = "100MB"
stdout-should-append-timestamp-to-filename = false
user = "${USER}"
working-directory = "/tmp/"

• name = string: Name of the service. If missing, Horust will use the filename by default.
• command = string: Specify a command to run, or a full path. You can also add arguments. If a full path is not provided, the binary will be searched using the $PATH env variable.
• start-after = list<ServiceName>: Start after these other services. If service a should start after service b, then a will be started as soon as b is considered Running or Finished. If b goes in a FinishedFailed state (finished in an unsuccessful manner), a might not start at all.
• shutdown-after = list<ServiceName>: Shut down this service only after the listed services have stopped. This allows controlling the order in which services are terminated during a graceful shutdown. For example, if service vpn has shutdown-after = ["api.toml"], then vpn will only receive its termination signal after api has fully stopped. A second SIGTERM (forceful shutdown) bypasses this ordering.
• start-delay = time: Start this service with the specified delay. Check how to specify times here Queued starts are not prepared once shutdown begins.
• stdout = STDOUT|STDERR|file-path: Redirect stdout of this service. STDOUT and STDERR are special strings, pointing to stdout and stderr respectively. Otherwise, a file path is assumed.
• stdout-rotate-size = string: Chunk size of the file specified in stdout. Once the file grows above the specified size it will be closed and a new file will be created with a suffix .1. Once the new file also grows above the specified size it will also be closed and a next one will be created with the next suffix .2. This allows adding an external log rotation script, which can compress the old logs and maybe move them out to a different storage location. The size is parsed using bytefmt - for example 100 MB, 200 KB, 110 MIB or 200 GIB. If unset, the default value will be 100 MB.
• stdout_should_append_timestamp_to_filename = boolean: If true, the log file will get the timestamp of the run appended to the end. It’s helpful to avoid overwriting logs from different runs.
• stderr = STDOUT|STDERR|file-path: Redirect stderr of this service. Read stdout above for a complete reference.
• user = uid|username: Will run this service as this user. Either an uid or a username (check it in /etc/passwd)
• working-directory = string: Will run this command in this directory. Defaults to the working directory of the horust process.

Restart section

[restart]
strategy = "never"
backoff = "0s"
attempts = 0

• strategy = always|on-failure|never: Defines the restart strategy.

  • always: Failure or Success, it will always be restarted (bounded by attempts, see below).
  • on-failure: Only if it has failed (bounded by attempts, see below).
  • never: By default it won’t be restarted, no matter the exit status. It may still be restarted as a rapid-failure escape hatch if attempts is configured (see below).
• backoff = string: Use this time before retrying restarting the service.
• attempts = number: A budget that bounds how many times a service can fail rapidly (before it becomes stable) before it’s considered FinishedFailed. Default is 0, which means unbounded (the service is restarted forever according to its strategy). Attempts are useful when a service is failing too quickly: if you’re in a start-stop loop, this puts an end to it. A failure counts as “too quick” if it happens before the service became stable, i.e. before it reached the running state - see healthiness.healthy-after below to control how long a service must survive before it’s considered stable. Failing to spawn the process at all (for instance, a command that isn’t found) also counts against the budget. The budget applies to all three strategies:
  • For always and on-failure, attempts > 0 bounds the rapid-restart loop; attempts = 0 keeps restarting forever.
  • For never, attempts > 0 allows the service to be restarted up to that many times only when it fails too quickly; with attempts = 0 a never service is never restarted on failure. Once the service has become stable, never stops restarting it entirely. In every case, once the budget is exhausted the service is no longer restarted, and it terminates as FinishedFailed (or Finished, if its last exit was successful).

The delay between attempts is calculated as: backoff * attempts_made + start-delay, where attempts_made is capped at attempts (or at 10, when attempts = 0 means unbounded) so that a service which never becomes stable keeps being retried at a steady rate instead of drifting infinitely far apart. For instance, using:

• backoff = 1s
• attempts = 3
• start-delay = 1s”

Will wait 1 second and then start the service. If it doesn’t start:

• 1st attempt will start after 1*1 + 1 = 2 seconds.
• 2nd attempt will start after 1*2 + 1 = 3 seconds.
• 3d and last attempt will start after 1*3 +1 = 4 seconds.

If the attempts are over, then the service will be considered FailedFinished and won’t be restarted. The attempt count counts only rapid failures (failures that happen before the service becomes stable), and it is reset as soon as the service’s state changes to running (i.e. it becomes “green”). This state change is driven by the health-check component: a service with no health-check is considered Healthy and passes to the running state after healthiness.healthy-after (immediately by default, see the Healthiness section). Configure healthy-after to define how long a crash-prone service must survive before its restart budget is reset.

Healthiness Check

Only results from the current process attempt affect health; queued results from previous runs are ignored. If attempt identities are exhausted, Horust refuses the start, marks the service failed, and initiates graceful shutdown without reusing an identity or downgrading an already-forceful shutdown.

[healthiness]
http-endpoint = "http://localhost:8080/healthcheck"
file-path = "/var/myservice/up"
command = "curl -s localhost:8080/healthcheck"
max-failed = 3
healthy-after = "0s"

• http-endpoint = <http endpoint>: It will send an HEAD request to the specified http endpoint. 200 means the service is healthy, otherwise it will change the status to failure. This requires horust to be built with the http-healthcheck feature (included by default).
• file-path = /path/to/file: Before running the service, it will remove this file if it exists. Then, as soon as this file is created, the service will be considered running.
• command = your_command arg1 arg2 ...: It will run this command. If the exit status is 0, the service is considered healthy.
• max-failed = i32: How many unhealthy health-checks in a row are allowed before considering the service failed. Applies while starting too (about one check per second): raise it for services slow to become healthy.
• healthy-after = string: How long the process must stay alive before it’s considered healthy (and thus stably running, which resets the restart.attempts budget). Defaults to 0s, meaning the service is considered healthy immediately. This is most useful for services without an explicit health-check: raising it lets a crash loop deterministically exhaust its restart budget instead of resetting it on every quick (re)start.
• You can check the healthiness of your system using a http endpoint or a flag file.
• You can use the enforce dependency to kill every dependent system.

Failure section

[failure]
successful-exit-code = [0, 1, 255]
strategy = "ignore"

• successful-exit-code = [\<int>]: A comma separated list of exit code. Usually a program is considered failed if its exit code is different from zero. But not all fails are the same. With this parameter you can specify which exit codes will make this service considered as failed.

• strategy = shutdown|kill-dependents|ignore’: We might want to kill the whole system, or part of it, if some service fails. Default: ignore

  • kill-dependents: Dependents are all the services start after this one. So if service b has service a in its start-after section, and a has strategy=kill-dependents, then b will be stopped if a fails.
  • shutdown: Shut down all the services and exit Horust if this service has failed. Queued restarts are ignored during shutdown, preserving the service’s failed outcome.

Environment section

[environment]
keep-env = false
re-export = ["PATH", "DB_PASS"]
additional = { key = "value" } 

• keep-env = bool: default: false. Pass over all the environment variables. Regardless of the value of keep-env, the following keys will be updated / defined:
• USER
• HOSTNAME
• HOME
• PATH Use re-export for keeping them.
• re-export = [\<string>]: Environment variables to keep and re-export. This is useful for fine-grained exports or if you want for example to re-export the PATH.
• additional = { key = <string> }: Defined as key-values, other environment variables to use.

Termination section

[termination]
signal = "TERM"
wait = "10s"
die-if-failed = ["db.toml"]

• signal = "TERM|HUP|INT|QUIT|USR1|USR2|WINCH|...": The friendly signal used for shutting down the process. The full list of supported signal can be found here.
• wait = "time": How much time to wait before sending a SIGKILL after signal has been sent.
• die-if-failed = ["<service-name>"]: As soon as any of the services defined in this the array fails, this service will be terminated as well.

ResourceLimit section

[!NOTE] This feature requires running Horust as the root user or with related cgroups permissions. If you’re trying to use this feature in a container, you might need --privileged --cgroupns=host flags. Other solutions with container (haven’t been tested yet):

• podman: https://github.com/containers/podman/pull/9536
• containerd: https://github.com/containerd/containerd/issues/10924

[resource]
cpu = 0.5
memory = "100 MiB"
pids-max = 100

• cpu = float: The maximum CPUs that the service can use. If unset, there will be no limit for the CPU usage.
• memory = string: Size of the memory that the service can use. Exceeding this limit will cause Out-Of-Memory. The size is parsed using bytefmt - for example 100 MB, 200 KB, 110 MiB or 200 GiB. If unset, there will be no limit for the memory.
• pids-max = int: The maximum number of processes/threads that the service can create. If unset, there will be no limit.

---

State machine

You can compile this on https://state-machine-cat.js.org/

initial => Initial : "Will eventually be run";
Initial => Starting : "All dependencies are running, a thread has spawned and will run the fork/exec the process";
Initial => Finished : "System shutdown before service had a chance to run (Kill Event)"; 
Starting => Started : "The service has a pid";
Started => Running : "The service has met healthiness policy";
Started => Failed : "Service cannot be started";
Started => Success : "Service finished very quickly";
Failed => FinishedFailed : "Restart policy";
Started => InKilling : "Received a Kill event";
InKilling => Success : "Observed exit satisfies exit-code and health policies";
InKilling => Failed : "Observed exit fails exit-code or health policies";
Running => Failed  : "Exit status is not successful";
Running => Success  : "Exit status == 0";
Running => InKilling: "Received a Kill event";
Success => Initial : "Restart policy applied";
Success => Finished : "Based on restart policy";
Failed => Initial : "restart = always|on-failure";

Horust’s configuration

Horust can be configured by using the following parameters:

# Default time to wait after sending a `sigterm` to a process before sending a SIGKILL.
unsuccessful-exit-finished-failed = true

All the parameters can be passed via the cli (use horust --help) or via a config file. The default path for the config file is /etc/horust/horust.toml.

Running a single command

You can wrap a single command with horust by running:

./horust -- bash /tmp/myscript.sh

This is equivalent to running a single service defined as:

command= "bash /tmp/myscript.sh"

This will run the specified command as a one shot service, so it won’t be restarted after exiting.

Commands have precedence over services, so if you specify both a command and a services-path, the command will be executed and the --services-path is ignored.

Multiple service directories

You can you use the --services-path parameter to specify either a directory containing .toml services to run, or point it to a .toml service file to run.

You can specify multiple service directories by passing more than one --services-path arguments.

horust --services-path ./services/core --services-path ./services/extra --services-path ./my-service.toml

These directories are loaded at once and treated just like all *.toml files were in single shared directory. It means that for example service from ./services/extra can depend on service from ./services/core. The last parameter is used to load a single service file instead of a directory.

horustctl: Checking system status

Horustctl is a program that allows you to interact with horust. They communicate using Unix Domain Socket (UDS), and by default, horust stores the sockets in /var/run/horust. You can override the path by using the argument –uds-folder-path. Then you can use it, like this:

horustctl --uds-folder-path /tmp status myapp.toml

To check the status of your service. Currently, horustctl only supports querying for the service status. Additional capabilities are planned but not yet implemented.

Plugins (WIP)

Horust works via message passing, it should be fairly easy to plug additional components connected to its bus. At this time is unclear if there is the need for this. Please raise an issue if you’re interested in seeing this feature.