Report this

What is the reason for this report?

Understanding Systemd Units and Unit Files

Updated on March 16, 2026
Understanding Systemd Units and Unit Files

Introduction

Linux distributions widely use the systemd init system to manage services, devices, mounts, and boot states. In systemd, a unit is any resource the system knows how to operate on and manage. Units are defined by configuration files called unit files. This guide explains what systemd units and unit files are, where they live, how to author and customize them, and how to manage and troubleshoot them using systemctl, journalctl, and systemd-analyze. For day-to-day service control, see How To Use systemctl to Manage systemd Services and Units.

Key Takeaways

  • A systemd unit is a managed resource (service, socket, mount, timer, etc.); a unit file is the configuration file that defines that unit.
  • Place custom and override unit files under /etc/systemd/system/ so they are not overwritten by packages and take precedence over /lib/systemd/system/ (or /usr/lib/systemd/system/).
  • Use drop-in files in /etc/systemd/system/<unit>.d/*.conf to override specific directives; clear exec directives (e.g. ExecStart=) before redefining them.
  • Socket activation uses a .socket unit to create the socket and a .service unit that starts on demand; it improves boot parallelization and on-demand startup.
  • Dependency: Use Wants= for optional and Requires= for mandatory dependencies; use After=/ Before= for ordering. BindsTo= adds “stop when the other stops.”
  • systemctl enable sets boot-time activation (symlinks); systemctl start starts the unit now. systemctl stop stops it but does not remove enable symlinks.
  • After any unit file or drop-in change, run systemctl daemon-reload before starting, restarting, or enabling the unit.
  • Use journalctl -u \<unit\> and journalctl -u \<unit\> -f to view and follow logs; use systemd-analyze and systemd-analyze blame / critical-chain to diagnose boot performance and unit startup order.

Prerequisites

  • A Linux system running systemd (Ubuntu, Debian, Fedora, CentOS Stream, or Arch Linux)
  • A non-root user with sudo access
  • Basic familiarity with the terminal and Linux commands

If you are setting up a new server, see Initial Server Setup with Ubuntu first.

What Is systemd and Why It Replaced Traditional Init Systems

The core problem with SysVinit was sequencing: every service had to wait for the previous one to finish before starting. A SysVinit service script was often 50-100 lines of shell that handled start, stop, reload, and status logic manually, with no standard for dependency declaration or failure recovery.

systemd replaces that model. It reads declarative unit files, resolves the full dependency graph before starting anything, and activates units in parallel wherever ordering allows. A service that previously took 30 sequential seconds to reach a running state can reach it in under 10 seconds when its independent dependencies start simultaneously.

Beyond speed, systemd provides: integrated logging through the journal (no separate syslog configuration required), socket and path activation so services start only when needed, cgroup-based resource tracking so every process spawned by a service is accounted for, and security sandboxing through directives in the unit file itself.

systemd is the init system on Ubuntu, Debian, RHEL, Fedora, CentOS Stream, Arch Linux, and most other current Linux distributions.

What Is a systemd Unit

If you have run systemctl status nginx, the output you saw – the service state, PID, memory usage, recent log lines, and whether it is enabled at boot – describes a unit. A unit is the object systemd tracks and manages. A unit file is the configuration file on disk that tells systemd what the unit is, how to start it, what it depends on, and when it should run.

Units are not limited to services. systemd uses the same model to manage network sockets, filesystem mount points, swap devices, hardware devices, scheduled tasks, and system state checkpoints. Each type has its own unit file suffix and its own section in the unit file for type-specific configuration.

Ideas that other init systems might bundle into one script are often split into multiple units (for example, a .socket unit and a .service unit). That separation allows socket-based activation, parallel startup, and easier customization via drop-in overrides. Features that units enable include:

  • Socket-based activation: Sockets can be created early in the boot process; the associated service starts only when the socket is first used, improving parallelization and on-demand startup.
  • Security hardening: Directives such as NoNewPrivileges=, ProtectSystem=, and PrivateTmp= restrict what a service can access.
  • Drop-in overrides: You can override or extend unit behavior by adding files under /etc/systemd/system/<unit>.d/ without editing vendor unit files.

Path-based activation, device-based activation, template instances, and implicit dependency mapping are covered in the Unit Types and Anatomy sections below where the relevant unit file syntax is shown alongside them.

Where systemd Unit Files Live: Load Paths and Precedence

Unit files are read from several directories. When loading a unit, systemd searches these paths in a fixed precedence order and picks the first (highest-priority) complete unit file it finds as the main definition. It then applies any matching drop-in snippets from the corresponding *.d/ directories, also in precedence order. Knowing this order tells you where to place custom or modified unit files so they are not overwritten by package updates and so your changes take effect.

Directory Priority (1 = highest) When to use
/etc/systemd/system/ 1 Custom unit files and overrides. Survives reboots and is not overwritten by packages. Use for new services or to fully replace a vendor unit.
/run/systemd/system/ 2 Runtime-only units. Highest priority after /etc. Changes are lost on reboot. Used by systemd and by installers for temporary overrides.
/lib/systemd/system/ (or /usr/lib/systemd/system/) 3 Vendor- and package-supplied units. Do not edit; override via /etc/systemd/system/ or drop-in files instead.

On some distributions the vendor path is /usr/lib/systemd/system/ instead of /lib/systemd/system/. Both have the same precedence relative to /etc and /run.

Place custom unit files and drop-in directories under /etc/systemd/system/ so package updates do not overwrite them and your configuration persists across reboots.

Anatomy of a systemd Unit File

Unit files are plain-text, INI-style files. Structure is organized into sections, each starting with a section name in square brackets, e.g. [Unit], [Service], [Install]. Section names are case-sensitive. Between section headers, directives use a key-value format with an equals sign. In drop-in files, a directive can be cleared by assigning it an empty value before redefining it. See the Drop-In Overrides section below for the full pattern and a worked example.

Here is a minimal but complete unit file that runs a script at boot, so you have a concrete reference before the individual directives are explained:

[Unit]
Description=My startup script
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/myscript.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
  • [Unit] holds metadata and ordering. After=network.target means this unit starts after the network is up, but does not require it.
  • [Service] defines what runs. Type=oneshot means systemd waits for the script to exit before considering the unit started. RemainAfterExit=yes means the unit stays in an “active” state after the script finishes.
  • [Install] defines boot behavior. WantedBy=multi-user.target means systemctl enable will create a symlink that pulls this unit into the normal multi-user boot sequence.

Every systemd unit file follows this pattern. The sections and directives vary by unit type, but the three-section structure is consistent.

systemd accepts multiple boolean forms (1/yes/on/true and 0/no/off/false) and parses time values flexibly (unit-less numbers are seconds). Sections starting with X- are ignored by systemd and can be used for tooling or documentation.

[Unit] Section Directives

The [Unit] section holds metadata and relationships to other units. Common directives:

  • Description=: Short, human-readable description shown by systemctl status and similar tools.
  • Documentation=: URIs to man pages or documentation; exposed by systemctl status.
  • Requires=: Listed units must activate with this unit; if they fail, this unit fails. No ordering by default; use with After=.
  • Wants=: systemd will try to start listed units when this unit starts. If they fail or are missing, this unit still runs. Preferred for most optional dependencies.
  • BindsTo=: Like Requires=, but this unit is stopped when the listed unit stops.
  • After= / Before=: Ordering only. After= means “start after these units”; Before= means “these units start after this one.” They do not create a dependency by themselves.
  • Conflicts=: Listed units cannot run at the same time; starting this unit will stop them.
  • Condition...= / Assert...=: Condition directives skip the unit if not met; Assert directives cause activation to fail if not met.

[Install] Section Directives

The optional [Install] section defines what happens when you enable or disable the unit. Enabling a unit typically creates a symlink so the unit is pulled in at boot (e.g. by a target).

  • WantedBy=: When you run systemctl enable <unit>, a .wants directory is created under /etc/systemd/system for the listed target (e.g. multi-user.target.wants), and a symlink to this unit is placed there. Disabling removes the symlink.
  • RequiredBy=: Same idea as WantedBy= but creates a required dependency (.requires).
  • Also=: Additional units to enable or disable together with this one.
  • Alias=: Additional names under which this unit can be enabled. Allows multiple providers to satisfy a common name without conflicting.
  • DefaultInstance=: For template units, sets the fallback instance identifier used when no specific instance is provided at enable time.

Unit-Specific Sections: [Service], [Socket], and Others

Each unit type has a section named after the type (e.g. [Service], [Socket], [Timer]). The [Service] section is the one you will edit most often for daemon management.

The [Service] Section

Type= tells systemd how the service behaves so it can track the main process and consider the unit “started” correctly:

Type Use case
simple Default when ExecStart= is set. The process started by ExecStart= is the main process. Use when the service does not fork or when using socket activation.
forking Process forks and the parent exits; child is the real daemon. Set PIDFile= so systemd can track the child.
oneshot One-off task. systemd waits for the process to exit before considering the unit started. Use with RemainAfterExit=yes if the unit should stay “active” after the command exits.
dbus Service takes a name on D-Bus; systemd continues when the name appears. Set BusName= to that name.
notify Service sends a readiness notification (e.g. sd_notify) when started. systemd waits for it. Use for services that know when they are ready.
exec Like simple, but systemd considers the service started only after the ExecStart= process has successfully execve()d. This makes failures to exec the main binary visible immediately; it does not change how ExecStartPre= commands are ordered.
idle Start only after all active jobs are dispatched; mainly for console services.
notify-reload Service supports reload; systemd expects a notification after ExecReload=.

Execution directives:

  • ExecStart=: Full path and arguments for the main process. Prefix with - to allow non-zero exit without marking the unit failed. Only one ExecStart= per unit (except in oneshot).
  • ExecStartPre= / ExecStartPost=: Commands run before or after the main process. Multiple lines allowed; prefix with - to ignore failure.
  • ExecReload=: Command to reload configuration (e.g. send SIGHUP or run a reload script).
  • ExecStop=: Command to stop the service; if unset, systemd kills the main process.
  • Restart=: Controls when systemd automatically restarts the service. Common values:
    • no (default): Never restart.
    • on-failure: Restart if the process exits with a non-zero exit code, is killed by a signal, or hits a timeout. Use this for production services.
    • always: Restart regardless of how the process exited. Use for services that should never stay stopped.
    • on-success: Restart only if the process exited cleanly (exit code 0). Rarely needed.
    • on-abnormal: Restart on signals and timeouts but not on clean exit or failure exit codes. Use RestartSec= to set the wait time before the restart attempt (e.g. RestartSec=5).
  • TimeoutStartSec= / TimeoutStopSec= / TimeoutSec=: How long to wait before considering start/stop failed or killing the process.

Logging: Sending stdout and stderr to the journal makes debugging easier and keeps everything in one place:

StandardOutput=journal
StandardError=journal

Optional: SyslogIdentifier= sets the name used in journal entries for this service.

Understanding Unit Types in Detail

Units are categorized by type; the type is indicated by the filename suffix.

Type Suffix Purpose
Service .service Daemons and long-running processes.
Socket .socket Network or IPC sockets (and FIFOs) for socket activation.
Target .target Synchronization points and groups of units; used in the boot sequence.
Device .device Devices exposed by udev/sysfs that systemd manages for ordering or mounting.
Mount .mount A mount point managed by systemd (name = path with slashes as dashes).
Automount .automount Mount point that is mounted on first access; pairs with a .mount unit.
Swap .swap Swap space (file or device).
Path .path Path-based activation via inotify; triggers a matching .service by default.
Timer .timer Time-based activation (like cron); triggers an associated unit.
Slice .slice cgroup slice for resource limits (CPU, memory, I/O).
Scope .scope Grouping of externally created processes; created by systemd from bus/API, not from unit files.
Snapshot .snapshot Temporary snapshot of current unit state for rollback; does not persist across reboot.

.slice: Slices map to cgroup nodes and are used to limit or allocate CPU, memory, and I/O. Default slices include user.slice, system.slice, and machine.slice. Use case: cap resource usage for a set of services (e.g. a slice for batch jobs with MemoryMax= and CPUQuota=).

.scope: Scopes are created by systemd from external events (e.g. user sessions, container managers). You do not create .scope unit files; they represent groups of processes that were started outside systemd. Use case: viewing and constraining resources of a user session or a container.

.path: Monitors filesystem paths with inotify. When the path matches (e.g. file exists, directory not empty, file modified), systemd starts the associated unit (default: same name with .service). Use case: start a backup or sync service when a file appears in a directory, or when a file is written.

Path unit example: Run a processing script whenever a file appears in /var/spool/incoming/:

/etc/systemd/system/incoming-processor.path:

[Unit]
Description=Watch /var/spool/incoming for new files

[Path]
PathExistsGlob=/var/spool/incoming/*
Unit=incoming-processor.service

[Install]
WantedBy=multi-user.target

/etc/systemd/system/incoming-processor.service:

[Unit]
Description=Process files in /var/spool/incoming

[Service]
Type=oneshot
ExecStart=/usr/local/bin/process-incoming.sh

Enable the path unit, not the service directly:

sudo systemctl enable --now incoming-processor.path

.timer: Schedules activation of another unit (usually a .service) on a calendar (e.g. OnCalendar=*-*-* 02:00:00) or relative to boot/startup/unit active/inactive. Use case: replace cron for periodic tasks, with consistent logging and dependency integration.

To list all active and inactive timers:

systemctl list-timers --all
NEXT                          LEFT      LAST                          PASSED   UNIT           ACTIVATES
Thu 2099-01-01 02:30:00 UTC   14h left  Wed 2099-12-31 02:30:01 UTC   9h ago   backup.timer   backup.service

Timestamps reflect the system clock at the time of the command.

Timer unit example: Run a backup script every day at 2:30 AM. If the system was off at that time, run it immediately at next boot (Persistent=true):

/etc/systemd/system/backup.timer:

[Unit]
Description=Daily backup timer

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true

[Install]
WantedBy=timers.target

/etc/systemd/system/backup.service:

[Unit]
Description=Daily backup job

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh

Enable the timer (not the service; the timer activates it):

sudo systemctl enable --now backup.timer

.automount: Defines a mount point that is mounted automatically on first access. Must have a matching .mount unit. Use case: lazy-mounting network or removable storage so the mount happens only when something accesses the path.

Socket Activation (Working Example)

Socket activation separates “listening on a socket” from “running the service.” systemd creates the socket(s) early; the service starts only when the socket receives a connection (or the first datagram). That improves boot parallelization and allows on-demand startup so the service runs only when needed.

Example: A TCP socket on port 2000; one service instance starts per connection and receives the connection file descriptor from systemd.

Install socat if it is not already present:

# Debian / Ubuntu
sudo apt install socat

# Fedora / RHEL / CentOS Stream
sudo dnf install socat

# Arch Linux
sudo pacman -S socat

/etc/systemd/system/echo.socket:

[Unit]
Description=Echo socket for socket activation

[Socket]
ListenStream=2000
Accept=yes

[Install]
WantedBy=sockets.target

/etc/systemd/system/echo@.service (template: systemd passes the connection fd as fd 3 to each instance):

The service unit must be a template (name contains @) when Accept=yes is set. Using a non-template service name with Accept=yes will cause a failed activation.

[Unit]
Description=Echo service instance (socket-activated)

[Service]
Type=simple
ExecStart=/usr/bin/socat FD:3 STDIO
StandardInput=socket
StandardError=journal

With Accept=yes, systemd accepts each connection and starts an instance of echo@.service, passing the connection as file descriptor 3. socat FD:3 STDIO bridges that connection to the process stdin/stdout (echo behavior). The socket unit is started at boot or when you run systemctl start echo.socket; no service runs until the first client connects to port 2000.

Enable and start the socket only; service instances start on each connection:

sudo systemctl daemon-reload
sudo systemctl enable echo.socket
sudo systemctl start echo.socket
# echo@.service instances start automatically when clients connect to port 2000

To verify: echo hello | nc localhost 2000 (you should get hello back). Check with systemctl status echo.socket, list instances with systemctl status 'echo@*.service', and view logs for a specific instance with journalctl -u echo@<instance>.service.

Why it matters: Sockets can be created in parallel early in the boot sequence; daemons start only when needed, reducing boot time and idle resource use. This is a key difference from traditional init systems where the daemon itself had to create and bind the socket before accepting connections.

Template Unit Files and Instances

Template unit files use an @ in the name (e.g. example@.service) and can spawn multiple instances (e.g. example@8080.service, example@8081.service). Instances are often created as symlinks to the template. In the unit file, specifiers are replaced at runtime: %i is the instance name (the part between @ and .), %n is the full unit name, %p is the prefix before @. Use %i for port or config-specific behavior (e.g. ExecStart=/usr/bin/app --port %i). See man systemd.unit for the full specifier list.

Dependency Ordering: Wants, Requires, After, Before, and BindsTo

Dependency directives control which units are started or stopped together and in what order. Ordering directives (After=, Before=) do not by themselves cause units to start; combine them with Wants= or Requires=.

Directive Effect If dependent fails or stops
Wants= Start these units when this unit starts. Soft dependency. This unit still runs.
Requires= These units must start with this unit. Hard dependency. This unit fails to activate.
BindsTo= Like Requires=, plus: when the listed unit stops, this unit stops. This unit is stopped.
After= Start this unit after the listed units (ordering only). No dependency; use with Wants/Requires.
Before= Listed units start after this unit (ordering only). No dependency.
Conflicts= These units cannot run with this unit. Starting this unit stops them; starting them stops this unit.

Example: A web app might Want= nginx and After= nginx so nginx starts first, but the app still starts if nginx is missing or fails. A database client that must have the database could use Requires= and After= so the DB starts first and the client fails if the DB does not start.

Managing Units with systemctl

Use systemctl to start, stop, enable, disable, and inspect units. Common operations:

  • Start/stop/restart: systemctl start <unit>, systemctl stop <unit>, systemctl restart <unit>
  • Enable/disable at boot: systemctl enable <unit> creates symlinks so the unit is started by its target (e.g. multi-user.target); systemctl disable <unit> removes those links. Enable does not start the unit now; start does not enable it at boot.
  • Enable and start in one step: systemctl enable --now <unit> enables and immediately starts the unit. systemctl disable --now <unit> disables and stops it.
  • Status: systemctl status <unit> shows state, recent log lines, and whether the unit is enabled.
  • Inspect the resolved unit file: systemctl cat <unit> prints the full unit file as systemd has loaded it, including any active drop-in files appended at the bottom. Use this to confirm which file is in effect after creating or modifying a unit.
  • Check enabled/active state for scripting: systemctl is-active <unit> returns active or inactive (exit code 0 or 1); systemctl is-enabled <unit> returns enabled or disabled. Both are suitable for use in scripts.
  • Mask and unmask: systemctl mask <unit> prevents a unit from ever starting by symlinking it to /dev/null. It cannot be started manually or pulled in as a dependency. systemctl unmask <unit> reverses this. Use masking when disable is not sufficient, for example to prevent a conflicting service from being re-enabled by a package update.
  • Reload vs restart: systemctl reload <unit> sends a signal to the running process to re-read its configuration (requires ExecReload= in the unit file). systemctl restart <unit> stops and starts the process. Use reload when the service supports it to avoid dropping active connections.

Example status output:

● nginx.service - A high performance web server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled)
     Active: active (running) since Mon 2099-01-01 14:32:00 UTC; 2h ago
   Main PID: 1234 (nginx)
      Tasks: 5
     Memory: 12.5M
        CPU: 234ms

To confirm systemd is using your updated unit file after any change:

systemctl cat myapp.service

The output shows the resolved file path at the top and appends any active drop-ins. If your changes are not reflected, run sudo systemctl daemon-reload first.

Creating a Custom Service Unit File from Scratch

Here is a production-style .service unit for a hypothetical app that listens on a port, runs as a non-root user, and uses common hardening options.

Scenario: App binary at /opt/myapp/bin/myapp, config at /etc/myapp/config.yaml, should run as user myapp, restart on failure, and have restricted filesystem and privileges.

Before creating the unit file, create the system user the service will run as:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin myapp

The --system flag creates a system account with no login shell by default. --no-create-home skips the home directory. --shell /usr/sbin/nologin ensures the account cannot be used for interactive login.

Create /etc/systemd/system/myapp.service:

[Unit]
Description=MyApp daemon
Documentation=https://example.com/myapp/docs
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp --config /etc/myapp/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
TimeoutStartSec=10
TimeoutStopSec=10

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

# Hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/var/lib/myapp /etc/myapp
# Drop all capabilities; add specific ones back if the app needs them
# e.g. CapabilityBoundingSet=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=
AmbientCapabilities=
LockPersonality=yes
ProtectKernelTunables=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectClock=yes
PrivateDevices=yes
ProtectHostname=yes
ProtectKernelModules=yes
ProtectProc=invisible
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
SystemCallFilter=@system-service
SystemCallFilter=~@privileged
SystemCallFilter=~@resources
UMask=0077

[Install]
WantedBy=multi-user.target
  • Type=simple: Main process is the one from ExecStart=; no forking.
  • User= / Group=: Run as unprivileged user/group.
  • WorkingDirectory=: Process working directory.
  • ExecStart=: Full path and arguments; use ExecReload= with $MAINPID if the app supports reload.
  • Restart=on-failure and RestartSec=: Automatic restart with a short delay.
  • StandardOutput=journal / StandardError=journal: Send logs to the journal.
  • NoNewPrivileges=yes: Prevents privilege escalation.
  • ProtectSystem=strict: Makes most of the filesystem hierarchy (including /etc, /usr, /boot, /var) read-only; use ReadWritePaths= (and related directives) to explicitly allow writes to required directories (for example /var/lib/myapp).
  • PrivateTmp=yes: Private /tmp for the service.

Reload systemd, then enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now myapp.service
sudo systemctl status myapp.service

Verify the service is running and journal logging is working:

journalctl -u myapp.service -n 20

Using Drop-In Override Files to Customize Unit Behavior

Drop-in files let you override or extend specific directives of a unit without editing the vendor file in /lib/systemd/system/. They live under /etc/systemd/system/<unit>.d/ (e.g. nginx.service.d/) and have a .conf suffix. The recommended workflow is systemctl edit <unit>, which creates the directory and opens a file (default: override.conf).

When to use drop-ins: Change only a few options (e.g. environment variables, Restart=, timeouts, exec lines). Use a full unit file in /etc/systemd/system/ when you are replacing the entire unit or when the change set is large.

Clearing and redefining exec directives: To replace the main start command, you must clear it first, then set the new one:

[Service]
ExecStart=
ExecStart=/usr/local/bin/nginx -c /etc/nginx/nginx-custom.conf

Without the empty ExecStart=, most service types would end up with multiple ExecStart= definitions, which is invalid and causes systemd to error with “more than one ExecStart= setting”. The empty assignment clears the original command so only the new one from the drop-in is used. The same pattern applies to ExecStartPre=, ExecReload=, and ExecStop= when you override them.

Full workflow:

sudo systemctl edit nginx.service

This opens your default text editor ($EDITOR, typically nano or vi) with a pre-formatted template showing where to place your directives. After saving, systemd writes the result to /etc/systemd/system/nginx.service.d/override.conf. Run systemctl cat nginx.service afterward to confirm the override is merged correctly.

Add your directives, save, and exit. Then:

sudo systemctl daemon-reload
sudo systemctl restart nginx.service

Always run systemctl daemon-reload after creating or changing any unit file or drop-in; otherwise systemd will not apply the new configuration.

Viewing and Analyzing Logs with journalctl

The systemd journal stores logs from the kernel, systemd, and services that use StandardOutput=journal / StandardError=journal. Use How To Use journalctl to View and Manipulate systemd Logs for full detail; below are the most useful patterns for unit debugging.

  • Logs for one unit: journalctl -u <unit> (e.g. journalctl -u nginx.service).
  • Recent time window: journalctl -u nginx.service --since "1 hour ago" or --since "today".
  • Live tail: journalctl -u nginx.service -f (follow).
  • Last boot only: journalctl -b (current boot); journalctl -b -1 for previous boot.
  • Last lines + follow: journalctl -xe jumps to the end of the journal and adds explanatory catalog text for entries that have it. To follow live output, use -f instead.

Example output for a unit:

Jun 04 14:32:00 host nginx[1234]: Starting nginx...
Jun 04 14:32:00 host systemd[1]: Started A high performance web server.
Jun 04 14:32:01 host nginx[1234]: nginx/1.18.0

Use -o short-full for full timestamps, -n 100 to limit lines, and -p err to filter by priority.

Reading common log patterns:

When a unit fails to start, the journal typically shows one of these patterns:

ExecStart= command not found or permission denied:

myapp.service: Failed to execute command: No such file or directory
myapp.service: Failed at step EXEC spawning /opt/myapp/bin/myapp: No such file or directory

Check that the binary path is correct and executable: ls -l /opt/myapp/bin/myapp.

Service started but exited immediately:

myapp.service: Main process exited, code=exited, status=1/FAILURE
myapp.service: Failed with result 'exit-code'.

Run the ExecStart= command directly in your shell to see its output. The exit code and any error messages will appear in stdout/stderr.

Start request repeated too quickly (restart loop):

myapp.service: Start request repeated too quickly.
myapp.service: Failed with result 'exit-code'.
systemd[1]: Failed to start MyApp daemon.

The service is failing and Restart=on-failure is bringing it back immediately. Increase RestartSec= or fix the underlying failure first. Use journalctl -u myapp.service -n 50 to see the full exit sequence.

Diagnosing Boot Performance with systemd-analyze

systemd-analyze helps you see how long boot took and which units delayed it.

  • Total boot time: systemd-analyze
Startup finished in 2.345s (kernel) + 8.901s (userspace) = 11.246s
  • Time per unit: systemd-analyze blame (units that took the most time)
3.212s man-db.service
1.876s NetworkManager-wait-online.service
1.234s cloud-init.service
  892ms apt-daily.service
  ...
  • Critical chain for a unit: systemd-analyze critical-chain nginx.service shows the chain of units that had to finish before this one and their times:
The time when unit became active or started is printed after the "@" character.
multi-user.target @8.901s
└─nginx.service @8.234s +665ms
  └─network-online.target @8.230s
    └─NetworkManager-wait-online.service @6.354s +1.876s

Interpretation: NetworkManager-wait-online.service took about 1.9s and delayed network-online.target, which in turn delayed nginx.service. To speed up boot you can disable or relax units that are not required (e.g. disable NetworkManager-wait-online if you do not need “network is up” before starting services) or optimize slow units (e.g. delay or mask heavy ones like man-db.service).

FAQ

What is the difference between a systemd unit and a unit file?

A unit is the object systemd manages: the running (or stopped) service, socket, timer, etc. A unit file is the configuration file (e.g. myservice.service) that defines how that unit should behave: what to run, when to start, and how it relates to other units. One unit file can define one unit (or, with templates, many instance units).

What is the difference between Wants= and Requires=?

Wants= is a soft dependency: when this unit starts, systemd tries to start the listed units. If they fail or are missing, this unit still activates. Requires= is a hard dependency: the listed units must start successfully; if any of them fail, this unit fails to activate. Prefer Wants= unless the unit truly cannot function without the other.

Where should I place a custom unit file so it is not overwritten by package updates?

Put it in /etc/systemd/system/. That directory has the highest precedence and is intended for administrator and site-specific units. Package managers install units under /lib/systemd/system/ (or /usr/lib/systemd/system/), which do not overwrite files in /etc/systemd/system/.

How do I reload systemd after creating or modifying a unit file?

Run sudo systemctl daemon-reload. This reloads all unit files and drop-ins from disk. Then start, restart, or enable the unit as needed. Without daemon-reload, systemd keeps using the old in-memory configuration.

What is socket activation and when should I use it?

Socket activation means systemd creates and listens on the socket (e.g. TCP or Unix socket); the service process starts only when a connection (or first datagram) arrives. Use it when you want the service to start on demand, to improve boot parallelization (sockets created early, services later), or to hand off an already-bound socket to the daemon so it does not need to bind itself.

How do I prevent a unit from starting at boot even if another unit has a Wants= dependency on it?

Disable the unit: sudo systemctl disable <unit>. That removes the symlinks that pull it into the boot sequence. If another unit only Wants it (not Requires), that other unit will still start; the wanted unit simply will not be started by the dependency. If the other unit Requires it, the requiring unit may fail to activate. To fully prevent a unit from ever starting, you can mask it: sudo systemctl mask <unit> (replaces the unit with a link to /dev/null).

What is the difference between systemctl stop and systemctl disable?

systemctl stop stops the unit immediately but does not change whether it is enabled at boot. After reboot, an enabled unit will still start. systemctl disable removes the unit from the boot sequence (removes symlinks under /etc/systemd/system/*.wants/ or *.requires/) but does not stop it now. Use stop to turn it off for this boot; use disable to stop it from starting on future boots (and often both: stop then disable).

How do I check why a unit failed to start?

Run systemctl status <unit> to see the current state and the last few log lines. Then run journalctl -u <unit> -n 50 --no-pager (or journalctl -u <unit> -xe) to see more log output. Look for failed ExecStart= commands, missing dependencies, failed condition or assert directive checks, or permission errors. systemctl show <unit> --property=Result --property=ExecMainStatus can show the result and exit code of the main process.

Conclusion

Understanding systemd units and unit files is central to managing Linux services and the boot sequence. Unit files use a declarative, INI-style format so you can see dependencies, execution, and hardening in one place. Keeping customizations in /etc/systemd/system/ and using drop-in overrides keeps vendor units intact and makes upgrades predictable. Using socket activation, dependency ordering, and the journal with journalctl and systemd-analyze gives you control over when services start and how to debug them. For more on service management, see How To Use systemctl to Manage systemd Services and Units; for log analysis, see How To Use journalctl to View and Manipulate systemd Logs.

Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.

Learn more about our products

About the author(s)

Justin Ellingwood
Justin Ellingwood
Author
See author profile

Former Senior Technical Writer at DigitalOcean, specializing in DevOps topics across multiple Linux distributions, including Ubuntu 18.04, 20.04, 22.04, as well as Debian 10 and 11.

Vinayak Baranwal
Vinayak Baranwal
Editor
Technical Writer II
See author profile

Building future-ready infrastructure with Linux, Cloud, and DevOps. Full Stack Developer & System Administrator. Technical Writer @ DigitalOcean | GitHub Contributor | Passionate about Docker, PostgreSQL, and Open Source | Exploring NLP & AI-TensorFlow | Nailed over 50+ deployments across production environments.

Still looking for an answer?

Was this helpful?


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

This comment has been deleted

Your article is the best I have read on the subject. The only addition I see would be about the ‘systemctl show unitFile’ that displays all the configuration details about a unit file.

hi! nice article! Is it possible to make a .service to wait a HDD gets fully mounted before exec it? Today I add a sleep in pre start unit. I don’t use fstab. My O.S auto-mounts the HDD. I didn’t set anything. Tried add the media-HDD.mount in “After” but didn’t work. thanks!

Thanks for your article!

really helpful!

These tutorials are well written and super helpful. Thanks a lot!

Great article.

Formatting nitpick: in Types of Units, .service is missing a bullet.

The following locations should also be known for .service files:

Runtime

  • Runtime units: $XDG_RUNTIME_DIR/systemd/user

User (when using “systemctl --user”)

  • User Unit Files: $XDG_CONFIG_HOME/systemd/user
  • User Unit Files (when $XDG_CONFIG_HOME is not set): $HOME/.config/systemd/user

Override

  • Override Unit File Load Path: $SYSTEMD_UNIT_PATH

Reference: https://unix.stackexchange.com/a/367237

Very informative and deep article. Thanks.

Maybe it could be interesting to add something about how to run operations on services status, exploring also commands like systemd-delta and systemctl daemon-reload.

And maybe some real example at the end and some picture to have a fast re-read of the article, like for the unit files hierarchy :)

Best Regards

You left out %j, the part of %p after the last hyphen (if there is one, otherwise it is equal to %p). And %J of course.

I think it sad that systemd doesn’t have any string utilities to manipulate %I. E.g. if you pass two arguments as instance: “arg1 arg2” then there is no way to add a rule: After: foo@arg1.service because the only way to get to arg1 is with scripting (aka, as part of a ExecStart). And because of that, there is the question of how to mimic all of the other rules (like Wants:, Requires:, After: etc) with an ExecStart:

Creative CommonsThis work is licensed under a Creative Commons Attribution-NonCommercial- ShareAlike 4.0 International License.
Join the Tech Talk
Success! Thank you! Please check your email for further details.

Please complete your information!

The developer cloud

Scale up as you grow — whether you're running one virtual machine or ten thousand.

Get started for free

Sign up and get $200 in credit for your first 60 days with DigitalOcean.*

*This promotional offer applies to new accounts only.