NextOS

A tiny 32-bit x86 hobby operating system written from scratch in C and assembly.

Build & run

Needs gcc (with -m32 support), GNU ld, make, and qemu-system-i386.

make run

make run builds the kernel and the programs in user/, installs them into /System/Commands and /Applications (and init into /System/Library/CoreServices) on disk.img with the host tool build/host/nxfs, copies the default configuration in rootfs/ into /Library where it isn't already (your edits are kept), and boots. It creates a 32 MiB disk.img (kept across make clean) and attaches it as the primary IDE master. Run make newdisk to start over with an empty disk. make run-serial boots the same disk with the serial port on your terminal, where there is a login too.

Tests, none of which need a display:

Command	What it runs
make test	the file system on the host against fake disks, including power failures after every write
make qtest	the real kernel under QEMU: processes, scheduling, fork and copy-on-write, exec, pipes, signals, memory on demand, threads, shared libraries, setuid (user/selftest.c)
make qtest-svc	init, mounts from Volumes, services and timers with a second disk, then checks both disks were left clean (user/svctest.c)
make qtest-doas	doas and its rules, as an ordinary user would use it (user/doastest.c)
make qtest-sh	the shell and the programs, from a shell script (tests/shtest)
make qtest-tty	logs in over the serial port and types: line editing, Ctrl+C/Ctrl+Z and jobs, less, top, edit, at and cron (tests/interactive.py; about two minutes)
make qtest-rust	Rust's std ported to NextOS (rust/std): files, threads, locks, channels, thread-locals, processes; needs nightly Rust
make qtest-dash	newlib and dash (ports/, downloaded and built the first time): dash runs tests/dashtest, a C test of newlib included

Where things are

The file system is laid out like a Mac's, with no Unix tree hidden underneath it: there is no /bin, /etc, /var, /tmp or /home. Every path below is in include/nextos/paths.h.

Path	Holds
/System/Commands	the command-line programs: the shell, ls, svc...
/System/Library/CoreServices/init	the first program
/Applications	the interactive programs (edit, less, more, top) and the demos
/Library/Accounts	Users (name:uid:home) and Passwords (root only)
/Library/Preferences	Volumes (disks to mount), Welcome (shown at boot), profile, doas.conf, crontab, logrotate
/Library/Services	the service units; Enabled/ says which start at boot
/Library/Logs	boot.log, auth.log, cron.log
/Library/Caches	state that means nothing after a reboot: what each service is doing, doas
/Library/Schedules	each user's Crontabs/ and the At/ jobs
/Library/Shared	data programs share (fonts, word lists, levels), one folder per program: Shared/NAME/
/Library/Temporary	anyone's scratch files
/Users	a home for each user, root's (/Users/root) too, and Shared
/Volumes	where other disks are mounted

$PATH is /System/Commands:/Applications, and scripts start with #!/System/Commands/sh.

Booting

The kernel starts one program, /System/Library/CoreServices/init (or whatever the boot loader's command line says with init=/path), as root, and then idles. init is the service manager: it mounts the disks listed in /Library/Preferences/Volumes, starts every enabled unit in dependency order, then stays to restart services that die, run timers and collect orphans. Among the units are a login (login) on each of the four consoles (Alt+F1..F4 switch between them) and one on the serial port: on first boot you choose a password for root; after logging in you get the shell as yourself in your home directory (shown as ~); exit or Ctrl+D logs out. Log in as root and create more accounts with useradd; each user gets /Users/<name>. If init is missing the kernel starts a root shell instead, so the system can be repaired. dmesg shows what the kernel said while it started.

Processes

NextOS is multitasking: up to 64 processes, scheduled preemptively (round robin, 50 ms slices). Each has its own address space, kernel stack, user, working directory, open files and environment. A program starts another with spawn (waiting for it, or with SPAWN_BG not waiting), or the Unix way with fork (the pages are shared copy-on-write) and exec; #! scripts run under the interpreter they name. The heap and the stack (up to 8 MiB) get pages when first touched. wait collects children and orphans are adopted by init.

Files and pipes. A descriptor points to an open file (read and written at an offset that dup and fork share), one end of a pipe, or a terminal. Children get their parent's descriptors, except those marked close-on-exec. A file removed while it is open lives until it is closed.

Signals. kill PID sends SIGTERM, kill -SIGNAL PID (or -PGID) any other. A program can catch them with signal, ignore or block them; the default ends it (with exit status 128 + the signal), stops it (SIGSTOP, SIGTSTP, SIGTTIN) or does nothing (SIGCHLD). Blocking calls are cut short (FS_EINTR) by a signal with a handler. alarm_ms sends SIGALRM.

Threads. A process can run more threads (thread_create, thread_join, thread_detach, thread_exit), scheduled like processes. They share its memory, open files, signal handlers, directory and user; each has its own stack, registers, signal mask and thread id (gettid; getpid is the process's). Each thread's %gs segment has a base of its own (set_tls), for thread-local storage. futex_wait/futex_wake put threads to sleep on a word of memory and wake them, and mutex_t and cond_t (in the library) are built on them. A signal sent to the process is taken by one of its threads; exit, a crash or a signal that ends the process ends every thread, and the process is over when its last thread ends. Only a process with one thread can exec. Threads count against the 64 slots of the process table.

Shared libraries. load_library("libfoo.so") loads an ELF shared object (built with -fPIC -shared) into the program, with the libraries it needs (DT_NEEDED), and gives a handle; library_symbol(handle, "name") finds a function or variable in it (handle 0: in any of them). The kernel's loader (src/dl.c) maps them in the library area (0x70000000 up, below the stack), applies their relocations (R_386_RELATIVE, _32, _PC32, _GLOB_DAT, _JMP_SLOT, all bound at once) and hands back their initialisers (DT_INIT, DT_INIT_ARRAY), which the library runs. Names without a / are looked for in /System/Library/Libraries, then /Library/Libraries. Programs are static, so a library calls back into the program through the symbols it exports: by default its libc (printf, malloc, strlen...), or whatever library_exports names. A library using a symbol nobody has isn't loaded. Libraries stay until the program ends or execs; fork keeps them. Thread-local variables in libraries aren't supported.

Unix programs. ports/ builds newlib, a full C library, and dash, a POSIX shell, for NextOS (ports/dash/build.sh; make install then puts dash in /System/Commands). C programs compiled with ports/bin/nextos-cc get stdio, printf with floats, math, fork, exec, waitpid, sigaction, opendir and the rest. open also knows /dev/null and /dev/tty, and kill(pid, 0) only checks. See ports/README.md.

Terminals and job control. Each terminal (the four consoles and the serial port) has a session, the login running on it, and a foreground process group, which gets what is typed: Ctrl+C sends it SIGINT and Ctrl+Z SIGTSTP. A process of a background job that reads the terminal is stopped (SIGTTIN). The shell puts each command line in a group of its own, so jobs, fg and bg work as in a Unix shell. Programs that handle Ctrl+C themselves (the shell, the editor) get it as a key. Input is a line at a time with editing (getkey gives single keys); on the serial port the screen operations become ANSI escape sequences.

Kernel code is never preempted; the file system and the disk driver, where a process sleeps waiting for the drive's interrupt, are kept to one process at a time by a sleeping lock.

Services

svc looks at and controls what init manages. Units are YAML files, /Library/Services/NAME.yaml (or .yml):

description: Console login
type: simple             # simple: a process that keeps running; oneshot: runs to completion
exec: /System/Commands/login   # or just login: found in /System/Commands or /Applications;
                               # or a list: [/System/Commands/cat, /Library/Preferences/Welcome]
restart:
  policy: always         # no | on-failure | always (gives up after 5 restarts in 10 s)
  delay: 500ms           # (or flat: restart: always, restart-delay: 500)
tty: tty2                # runs on a terminal, as a new session: tty1-tty4 or ttyS0 (yes: tty1)
requires: [other]        # started first; stops with it; if it fails, so does this unit
wants:                   # started first if possible
  - another
after: [motd]            # ordering only, when both are starting
user: alice              # run as this user
stop: cleanup            # run when the unit is stopped
remain: yes              # a oneshot stays active after it has run

A timer unit starts another one on a schedule:

type: timer
timer:
  unit: logrotate        # what it starts (a oneshot runs again each time)
  on-boot: 1m            # first, this long after boot
  every: 1h              # then this often (ms, s, m, h, d)

The YAML reader (user/lib/yaml.c) knows block and flow mappings and lists, quoted and plain strings, | and > blocks and comments, and reports mistakes with their line (svc check); it doesn't do anchors, tags or multi-line strings. Units in the older form, NAME.svc with key = value lines and the same settings, still work; if a unit has both files, the YAML one is used and svc check says the other is ignored.

Lines of /Library/Preferences/Volumes (DRIVE /path [noauto] [mkfs]) become mount units named after the path (/Volumes/Data is Volumes-Data.mount) and are mounted before anything else starts. A unit starts at boot when /Library/Services/Enabled/NAME exists. Commands: svc (list), svc status U, svc start|stop|restart U (stopping a unit stops what requires it first), svc enable|disable U, svc log [U], svc analyze (what started when, and how long it took), svc check (problems in the units and Volumes), and svc shutdown, which halt and reboot run to stop everything and unmount the disks before powering off.

The shell

/System/Commands/sh is a small Bourne-style shell:

Commands	cmd args, a; b, a && b, a || b, ! a, a &
Pipes and redirections	a | b | c, <, >, >>, 2>file, 2>&1, 3>file, <<EOF here-documents
Expansion	$VAR, ${VAR}, ${VAR:-x}, ${VAR:=x}, ${#VAR}, $?, $$, $!, $#, $@, $1..., $(cmd), $((1 + 2)), ~, * ? [a-z]
Quoting	'literal', "with $expansion", \x
Control	if/elif/else/fi, while, until, for x in ..., case ... esac, { ...; }, ( ... ), name() { ...; }

Built-ins: cd pwd echo export unset set shift read test [ true false break continue return source . eval exec type jobs fg bg wait kill history clear color calc about exit logout help. set -e stops a script at the first failing command and set -x traces. Everything else is found in $PATH (/System/Commands:/Applications). sh FILE ARGS runs a script, sh -c 'CMDS' a string, and interactive shells read /Library/Preferences/profile and ~/.shrc first.

Line editing: Left/Right, Home/End (or Ctrl+A/E), Backspace/Delete, Ctrl+U/K (delete to start/end), Ctrl+W (delete word), Ctrl+C (cancel), Ctrl+D (log out on an empty line), Ctrl+L (clear), Up/Down for history. Tab completes a command or path as far as it can, and lists the matches when several fit.

Programs

Files: ls [-la], cat [-n], cp [-r], mv, rm [-rf], mkdir [-p], rmdir, ln [-s], touch, chmod, chown, write, append, edit. Text: grep [-ivnclq] (regular expressions), head, tail [-f], sort [-rnu], uniq [-c], wc [-lwc], tee, seq, more, less. System: ps, top, kill, sleep, dmesg, date, env, uptime, mem, df, disks, mount, umount, mkfs, format, svc, crontab, at, halt, reboot. Users: whoami, id, users, useradd, userdel, passwd, su, doas. Programs that read text read standard input when given no files, so they work in pipes. help lists the shell's built-ins and the programs in /System/Commands and /Applications. reboot, halt, format, rdsec, wrsec, useradd, userdel, chown, mount, umount and mkfs need root.

cron and at

crond (the cron service) runs commands at set times, as their user, with sh -c, in their home directory, and appends what they print to /Library/Logs/cron.log.

• /Library/Preferences/crontab: MINUTE HOUR DAY MONTH WEEKDAY USER COMMAND lines; fields are numbers, *, lists 1,15, ranges 1-5 and steps */10. Times are UTC.
• crontab -l | -e | -r | FILE: your own table, the same without the USER field.
• at TIME COMMAND runs a command once: TIME is +30s, +5m, +2h, +1d or HH:MM; at -l lists your jobs and at -d ID removes one.

crontab and at are setuid root (the tables live where only root may write); they read your files with your own permissions.

edit

edit [FILE...] is a nano-like full-screen editor (a new file if FILE doesn't exist):

Key	Does
arrows, Home/End (or ^A/^E), PgUp/PgDn, Delete	move and delete
^S	save (an unnamed buffer asks for a name)
^X	save and close the file
^Q	close the file; asks before discarding changes (the editor ends with the last one)
^Z, ^Y	undo, redo (a run of typing, of deleting, a cut... is one step)
^O	open another file; ^N / ^P go to the next / previous one
^T	colours on or off (C sources, shell scripts and configuration files are coloured)
^W	search (Enter again repeats, wrapping round the file)
^K, ^U	cut a line (several in a row are collected), paste above the cursor
^G	go to a line

The editor's logic is in user/lib/editor.c and talks to the keyboard, screen and files through a small interface, so make test drives it with scripted keystrokes on the host. Long lines scroll sideways. As root (or with doas edit /Library/Preferences/doas.conf) it can change system files.

doas

doas [-u user] command runs a command as another user (root by default), as sudo would. It is a setuid-root program (the kernel runs a file whose mode has the setuid bit as its owner; chmod 4755, shown as s by ls). /Library/Preferences/doas.conf holds the rules, and the last one that matches wins:

permit [nopass] [persist] USER [as TARGET] [cmd COMMAND [args ARG...]]
deny   [nopass] [persist] USER [as TARGET] [cmd COMMAND [args ARG...]]

Unless the rule says nopass you type your own password (persist: not again for five minutes). cmd limits the command, args the exact arguments. Root can run anything. The file must be root's and not writable by others, and every decision goes to /Library/Logs/auth.log. doas -C checks the file, doas -L forgets your recent password, doas -s starts a shell. Services are controlled the same way: doas svc restart login.

What's inside

File	Purpose
src/boot.S	Multiboot header, stack setup, GDT/IDT loaders, interrupt stubs, the context switch
src/cpu.c	GDT (ring 0 and ring 3 segments), TSS, IDT, 8259 PIC remapping, exception/IRQ/syscall dispatch, panic screen
src/mm.c	Physical page allocator (bitmap) with counts for shared pages, paging, kernel heap, one address space per program, copy-on-write and pages on first touch
src/fs.c, src/fs_layout.h	ext2-style file system (version 5) with a journal and a block cache; see below
src/users.c	Accounts in /Library/Accounts/Users, salted SHA-256 password hashes in root-only /Library/Accounts/Passwords; makes the standard directories
include/nextos/paths.h	Where everything lives, for the kernel, the programs and the host tools
src/proc.c	ELF loader, process table and scheduler, spawn/fork/exec/wait/exit, threads and futexes, signals, process groups and sessions, the sleeping lock
src/file.c	Open files: files on disk, pipes and terminals behind file descriptors
src/tty.c	Terminals: sessions, the foreground group, Ctrl+C/Ctrl+Z, line input, the kernel log (kprintf, dmesg)
src/dl.c	The shared library loader: maps ELF shared objects and what they need, relocates them, finds symbols
src/syscall.c	System calls (int 0x80)
src/ata.c	ATA/IDE hard disk driver (PIO, LBA28): detects up to 4 drives, waits for each sector's interrupt
src/vga.c	Four virtual consoles on the VGA text screen
src/keyboard.c	PS/2 keyboard driver (shift, caps lock, Ctrl+letter, arrows, Home/End/Delete, Alt+F1..F4)
src/serial.c	COM1: a copy of the console, and a terminal of its own (escape sequences become keys)
src/timer.c, src/rtc.c	PIT timer at 100 Hz; the CMOS clock, read at boot
src/readline.c	Kernel line input for programs reading a terminal
src/string.c	Minimal libc string helpers
include/nextos/abi.h	System call numbers, error codes, signals and structs shared by the kernel and programs
user/sh.c	The shell
user/lib/	libc for programs (libc.c, util.c, threads and locks in thread.c, load_library in dl.c), the shell's line editor (readline.c), and libraries some programs share: the editor, the service engine, pagers, cron
user/*.c	The other programs, and demos hello note guess crash spin
user/init.c, user/svc.c, user/lib/svclib.c	The service manager (init), its command, and the unit engine they share
user/lib/yaml.c	A small YAML reader, for the service units
user/login.c, user/doas.c	The login, run as a service on each terminal; doas
user/edit.c, user/lib/editor*.c	The text editor and its keyboard/screen/file glue
user/crond.c, user/crontab.c, user/at.c	cron and at
rootfs/	Default /Library contents (Preferences and Services) copied into a disk image where missing
tools/nxfs.c	Host tool: mkfs, install, seed, chmod, ls and check for disk images. It compiles the kernel's own src/fs.c against a file-backed fake disk (tools/hostenv.c)
tools/fsck.c	Independent consistency checker for version 5 images (reachability, block ownership, link counts, bitmaps)
tests/fs_test.c, tests/edit_test.c, tests/yaml_test.c	File system, editor and YAML reader tests run on the host
tests/libs/	Shared libraries the self-test loads
ports/	newlib (a C library) and dash (a POSIX shell) for NextOS, with the glue between them and the kernel; see ports/README.md
tests/shtest, tests/interactive.py	The shell and programs, from a script; and from a terminal

Writing programs

Add user/<name>.c with a main(int argc, char **argv), include nextos.h, and make run; it shows up as /System/Commands/<name> (or in /Applications, if the Makefile lists it in APP_PROGS). Programs get printf/dprintf/snprintf, malloc/realloc/free, string functions, getenv/setenv, dates (gmtime_, format_time), readline, a line reader (reader_next) and each_input for "files or standard input", account lookups, find_command and wrappers for every system call in include/nextos/abi.h (spawn runs another program and waits for it; fork, execv, wait, kill, signal, pipe, dup2 and the rest are there too), threads with mutex_t and cond_t, and load_library. They run in ring 3 at 0x40000000 and can't touch kernel memory or I/O ports; a crash kills only the program.

File system layout (version 5)

In 1 KiB blocks: block 0 is the superblock, then the inode bitmap, the block bitmap, the inode table (one inode per 8 KiB of disk) and the data blocks, among them the journal (1/16 of the disk, between 32 KiB and 1 MiB). An inode is 64 bytes: uid, gid, mode, type, link count, size, modification time, 10 direct block pointers, a single-indirect and a double-indirect block, so a file is at most about 64 MiB. A directory is a file of 36-byte {inode, name} entries, including . and ... A symbolic link is an inode holding the path it points to; files may have several names (hard links). Files and directories are allocated from the bitmaps, so the only limits are the disk's free blocks and inodes. fs_layout.h has the exact structures. Blocks go through a small write-through cache. Version 3 disks (no journal, links or times) are upgraded when mounted, and versions 1 and 2 converted.

Upgrading from version 4

Version 4 had the same format but a Unix layout (/bin, /sbin, /etc, /home, /root, /tmp, /var, /mnt). The first mount of a version 4 disk holding a system (by the kernel, or by nxfs, so make run does it) moves everything to its new place: /etc/passwd becomes /Library/Accounts/Users, /etc/fstab /Library/Preferences/Volumes, /etc/svc /Library/Services, /home/bob /Users/bob, and so on (the table is relayout_moves in src/fs.c). Then the old paths inside the account file, the preferences, the units, the crontabs and each user's ~/.shrc are rewritten to the new ones. Each move is a transaction and the version number changes last, so an interrupted upgrade carries on at the next mount. If a new name is already taken the old file is left where it was and the kernel log says so. Other files of yours that name old paths (scripts starting #!/bin/sh, say) are not changed.

Crash safety

Every operation is a transaction. File contents are written to their blocks first (only to blocks that were free before, so a crash never shows a file another file's new data); the changed metadata (bitmaps, inodes, directories, indirect blocks) is collected and written to the journal, then the journal's descriptor with a checksum over everything (the commit), then each block to its place. A mount after a crash replays the last committed transaction, and an operation that failed part way is simply forgotten: every operation, a rename included, happens entirely or not at all, and no repair is needed.

Without a journal (the RAM disk, or an upgraded disk with no room for one) operations write data first, then the inode, then the directory entry, and remove the entry before freeing anything, so a power failure leaves the tree intact and at worst some blocks or inodes marked used that nothing refers to. The superblock records whether the file system was shut down cleanly (halt, reboot, umount and the host tool do that); mounting one that wasn't walks the whole tree and rebuilds the bitmaps and link counts from it, dropping anything inconsistent. That check also runs when the superblock asks for it: after a transaction too big for the journal was committed in parts, or when files removed while still open were left behind by a crash. It assumes a 512-byte sector is written entirely or not at all, which is why directory entries never straddle a sector. The tests cut the power after every sector write of each kind of operation, with and without a journal, and check that the disk comes back consistent.

More disks

disks lists the drives. A disk's file system is mounted over a directory of the root file system: mkfs 1 (only for a disk with nothing worth keeping), mkdir /Volumes/Data, mount 1 /Volumes/Data; mount and df show what is mounted and umount /Volumes/Data detaches it. Mounts are not remembered across reboots unless they are in /Library/Preferences/Volumes; after editing it, mount -a mounts what is new. The boot disk holding / is drive 0; the others have their own inodes and free space; mv between disks copies.

Packages

pkg installs, upgrades and removes packages. A package is one .pkg file built on the host with build/host/nxpkg build OUT.pkg NAME VERSION "DESCRIPTION" ROOTDIR: everything under ROOTDIR is installed at the same path (ROOTDIR/Applications/foo becomes /Applications/foo).

Command	Does
pkg install [-f] FILE.pkg...	install, or upgrade if the package is already installed (root only)
pkg remove NAME...	remove a package and the files it installed (root only)
pkg list / pkg info NAME / pkg files NAME	what is installed
pkg verify NAME	check that every file is there with the right size
pkg contents FILE.pkg	what a package file would install

What is installed is recorded in /Library/Packages/NAME/ (Info, Files). A file that belongs to one package is never overwritten by another, files that already exist are not overwritten without -f, an upgrade removes the files the new version no longer has, and system directories are never removed. make qtest-pkg tests it under QEMU.