linux-fundamentals▌
kernel, inodes, privileges, and systemd/daemons — the layer this round starts with, and the direct source of 5 of your 15 written questions.
Why this layer first
Kernel, inode, and systemd questions test whether you've actually operated a Linux box, not just used one through a managed platform. Every on-prem GPU server and every EC2 instance in bKash's fraud pipeline sits on exactly this stack — this is where memorized cloud-service trivia stops helping and real troubleshooting instinct starts mattering.
What's in this session
| page | covers | written q's |
|---|---|---|
| kernel(7) | kernel vs user space, syscalls, GPU driver relevance | — |
| inode(7) | metadata vs filename, the "disk full after rm" trap, inode exhaustion | 14, 15 |
| credentials(7) | UID/GID, permission bits, SUID/SGID/sticky, sudo vs su | — |
| systemd(1) | daemons, unit types, dependency ordering, journalctl | 5, 6, 10 |
kernel — the privileged core that mediates every process's access to hardware.
What it actually does
The kernel is the one piece of software that runs in a privileged CPU mode with direct hardware access. Everything else — your shell, Docker, Python, nginx — runs in user space and can only reach hardware indirectly, through the kernel.
Four jobs it owns: process scheduling (who gets CPU time), memory management (virtual memory, paging, isolating each process's RAM), device drivers (disks, NICs, and — for this role — GPUs), and the filesystem/network stack that turns raw blocks and packets into files and connections your code can use.
Kernel space vs. user space
User-space code never touches hardware directly. It asks the kernel via a system call — open(), read(), write(), fork(), exec(), mmap(), socket() — the kernel does the privileged work, and hands control back. That boundary is why a segfault in one Python process doesn't take the machine down, and why starting a new process always routes through fork() + execve() rather than any user-space shortcut.
Why it matters for a GPU/cloud role
The NVIDIA driver is a kernel module. nvidia-smi doesn't talk to the GPU — it talks to the kernel driver, which talks to the GPU. If a GPU "disappears" after an on-prem reboot, the first two commands are lsmod | grep nvidia and dmesg — before touching CUDA or PyTorch. Same instinct for a training job that dies with no stack trace: check dmesg | grep -i "killed process" before assuming a code bug. The kernel's OOM killer terminates processes silently when memory runs out.
"On our on-prem boxes, if a GPU disappears after a reboot, I wouldn't jump straight to CUDA — I'd check lsmod and dmesg first to confirm the kernel actually loaded the NVIDIA driver."
inode — the metadata record every file has. This one is genuinely real — man 7 inode on any Linux box covers exactly this page.
What an inode actually stores
Every file has an inode — a metadata record holding owner (UID/GID), permission bits, size, three timestamps (atime/mtime/ctime), a link count, and pointers to the actual data blocks. Notice what's not in there: the filename. Names live in the directory entry, a separate mapping of name → inode number. That's what ls -i shows you.
Why that distinction is the whole answer to Q14
Because the name is separate from the data, rm doesn't necessarily free space — it just deletes one directory entry and decrements the inode's link count. Data blocks are only released once the link count hits zero and no process still has the file open. Delete a 40GB dataset mid-training while a process still holds it open, and the space stays allocated until that process closes the file or dies.
Q: after training your server was full, you removed training data to clear storage — what command checks which processes are still holding disk?
A: lsof +L1 — open files with a link count under 1 (already deleted). Or the more memorable form: lsof | grep deleted. Fix: restart or reload the process holding it open.
The other inode gotcha: running out of them, not just space
A filesystem has a fixed inode count set at format time. You can hit 100% inode usage with plenty of raw disk space left, if something creates huge numbers of tiny files — a checkpoint directory dumping one small file per training step is a realistic ML example. df -h shows block/space usage; df -i shows inode usage. Check both when "no space left on device" shows up despite df -h looking fine.
Q: on a VM, why does /var fill up fast?
A: it holds variable runtime data — logs (/var/log, plus /var/log/journal if persistent journaling is on), package cache, and on a Docker host specifically /var/lib/docker: images, containers, and — with the default json-file log driver and no limits — unbounded container logs. Prevention: logrotate, journald's SystemMaxUse=, and Docker's log-opts max-size/max-file.
"If a fraud-scoring container's disk usage keeps climbing even after cleaning training artifacts, I'd check lsof +L1 before assuming it's a volume sizing problem — it's usually a process still holding a deleted file open."
credentials — the UID/GID rules that gate process privilege. Also real: man 7 credentials covers this territory.
The identity layer
Every process runs as a UID/GID, checked against /etc/passwd (user info), /etc/shadow (hashed passwords), and /etc/group. Every file's inode records an owner UID and group GID — that pairing is exactly what permission bits check against.
Permission bits
rwx for owner / group / other, expressed in octal (r=4, w=2, x=1). chmod 750 file → owner rwx, group r-x, other nothing. chown user:group file changes the owner/group.
The three special bits people forget
SUID (leading 4, e.g. 4755) — the file runs as its owner, not the invoking user. /usr/bin/passwd is root-owned with SUID set (-rwsr-xr-x — that s is the tell), which is how a regular user updates /etc/shadow — which they can't write directly — just by running passwd.
SGID (leading 2, e.g. 2775) — on a file, it runs as the file's group; on a directory, new files created inside inherit that directory's group instead of the creator's primary group. Useful for a shared GPU training directory where every teammate's output needs to stay group-readable.
Sticky bit (leading 1, e.g. 1777) — on a directory, only a file's owner (or root) can delete or rename it, even if others have write access to the directory. /tmp is the canonical example.
sudo vs su
| sudo | su | |
|---|---|---|
| scope | one command, as another user | entire shell session |
| policy | /etc/sudoers, per-user/command rules | needs the target user's password |
| audit trail | every invocation logged | no per-command log |
In any regulated environment — PCI-DSS relevant, in bKash's case — sudo is preferred specifically for that per-command audit trail.
Least privilege in practice
Service accounts — including containerized ones — should run with the minimum permissions they need: don't run containers as root, drop Linux capabilities you don't use, and reach for SGID group directories instead of loosening things to 777.
"Since we're handling transaction data, I'd default every service account and container to least privilege — sudo over shared su access for the audit trail, SGID group directories instead of world-writable folders for shared GPU training data."
systemd — the init system, PID 1. Also genuinely real: man systemd is a solid overview.
Daemon, defined
A background process with no controlling terminal, usually started at boot (or lazily, via socket activation) to provide a continuous service — sshd, dockerd, crond, a model-serving process running under supervision. The trailing "d" is naming convention, not a rule.
systemd, defined
The init system — PID 1 — on essentially every distro you'll touch here (Ubuntu, Amazon Linux 2/2023, RHEL). It replaced the older sequential SysV init scripts with a dependency graph: units declare what they need and when they should run relative to other units, and systemd parallelizes startup as much as that graph allows.
Q: what is a daemon? what is systemd?
A: a daemon is a background, terminal-detached process providing a continuous service. systemd is the init system (PID 1) that starts, stops, and supervises those daemons — including their startup ordering and dependencies.
Unit types
Ordering vs. dependency
The distinction that trips people up: After=/Before= is pure ordering — no dependency implied. Requires= is a hard dependency — if the required unit fails, this one stops too. Wants= is a soft dependency — preferred, but this unit survives even if the wanted one fails. [Install] WantedBy=multi-user.target is how systemctl enable actually wires a unit in.
Key commands
Q: command for printing the last 10 lines of a service's log?
A: if it logs through journald: journalctl -u service-name -n 10. If it writes to a flat file instead — nginx does, by default — tail -n 10 /var/log/nginx/error.log. Naming both shows you know logging isn't one-size-fits-all.
Q: a service needs to start only after successful login — what goes in the systemd config?
A: the simple answer is an After=/Wants= dependency in [Unit] on the relevant target. But if "successful login" means one specific user's login rather than just boot order, the mechanism is different: make it a user unit — ~/.config/systemd/user/app.service, with WantedBy=default.target under [Install], enabled via systemctl --user enable app. systemd only starts that user's instance once PAM (pam_systemd) creates a session at successful login — the unit literally cannot start before login succeeds.
Leading with that mechanism — not just the directive name — is what separates a memorized answer from a demonstrated one.
"For any service that should gate on a real event rather than boot order — a login, a mount becoming available — I'd reach for After=/Requires=, or a user unit tied to the PAM session if it's genuinely per-user, rather than a sleep-and-hope-it's-ready hack."
seven hands-on tasks — do these on any Linux box: a VM, WSL, or a free-tier EC2 instance. Progress saves automatically.
loading progress…