Operating Systems
The Boundary Between Software and Hardware
Understanding device driver abstractions, user/kernel privilege modes, I/O registers, and protection boundaries.
Operating SystemsI/O ManagementProtectionSecurityDrivers
1. The Hardware Boundary
When an ordinary user application needs to perform an operation on a hardware device, it cannot communicate with the hardware directly:
Application
↓
"I need data from the device"
A beginner might imagine a direct software-to-device pathway:
Application
↓
Hardware
In a modern, protected operating system, this direct path is blocked. Allowing arbitrary, direct register access to every user program would lead to chaos. Instead, the OS inserts a structured boundary of interfaces and control mechanisms:
Application
↓
OS Interface / System Call
↓
Kernel
↓
Device Driver
↓
Hardware
The application requests a logical operation from the operating system. The operating system coordinates the request and delegates the device-specific operations to the driver, which contains the specialized commands required to interact with the target hardware.
2. What a Device Driver Actually Does
A device driver is a specialized kernel module that translates generic OS-level operations into hardware-specific control sequences. Different physical devices have entirely different characteristics:
* Registers (status, control, data register addresses)
* Command formats and protocols (e.g. formatting a command packet)
* Timing requirements and clock frequencies
* Hardware-specific quirks and initialization routines
Instead of requiring every application to understand these details, the driver presents a standardized interface to the operating system:
Generic Request (e.g. read, write)
↓
Kernel
↓
Device Driver (Hardware-specific translation)
↓
Device-specific operations
↓
Hardware
For instance, when reading from a serial port, the application simply executes:
Application
↓
read()
↓
Kernel
↓
UART Driver
↓
UART Hardware
The application does not know which registers to query or which hardware flags to check; it relies on the driver to handle the translation.
3. We Already Know How Hardware Coordinates
We have already seen how software and hardware coordinate through polling, interrupts, and DMA. Those mechanisms do not disappear when an operating system is introduced. Device drivers and the kernel use them underneath the higher-level OS abstraction.
OS
│
Device Driver
│
┌──────────┼──────────┐
↓ ↓ ↓
Polling Interrupt DMA
│ │ │
└──────────┼──────────┘
↓
Device
The driver coordinates operations using these fundamental coordination techniques:
* Polling: The device driver repeatedly reads status registers to check if a byte is ready, useful for ultra-low latency or simple hardware loops.
* Interrupts: The driver registers an Interrupt Service Routine (ISR). When a hardware event occurs (such as a keypress or network frame arrival), the device triggers an interrupt pin, notifying the CPU to pause execution and run the driver's handler code.
* DMA (Direct Memory Access): The driver configures a DMA controller with source/destination addresses. The hardware transfer executes directly between the device and RAM, bypassing the CPU to prevent it from stalling on large blocks of data.
4. EdgeCase — The I/O Journey
Use the simulator below to trace how a system call routes down through the OS kernel and device driver layers, and how hardware interrupts and DMA return data to the waiting application:
5. Why Does the OS Need a Protection Boundary?
If the application needs hardware data, why not let every program talk to the physical device directly?
If any program could read and write arbitrary device registers, several critical problems would arise:
Application A ──→ Device
Application B ──→ Device
Application C ──→ Device
* Interference: Application A might overwrite control registers while Application B is mid-transfer, crashing the device controller.
* Corruption: A bug in a user space program could corrupt the partition tables on a physical hard drive, ruining the storage structure for all other applications.
* Unauthorized Access: A program could bypass system policies to read private keys or user keystrokes directly from peripheral memory.
* System Control: Programs could disable CPU interrupts, locking up the machine entirely.
To maintain system stability, the operating system must stand as the sole gatekeeper between user software and hardware.
6. User Mode vs. Kernel Mode
To enforce this boundary, modern CPUs provide physical privilege rings. We have already seen this concept in action: applications run in a restricted state, while kernel code runs with full privileges.
┌─────────────────────────────┐
│ USER MODE │
│ │
│ Application A │
│ Application B │
│ Application C │
└──────────────┬──────────────┘
│
Controlled Interface (Syscall / Trap)
│
┌──────────────▼──────────────┐
│ KERNEL MODE │
│ │
│ OS + Drivers │
└──────────────┬──────────────┘
│
▼
HARDWARE
Applications execute in User Mode, where direct access to peripheral I/O ports or control registers is forbidden by the hardware. When an app needs to touch a device, it must execute a system call, causing a controlled trap into Kernel Mode, where the operating system evaluates the request and carries out the raw register manipulations safely.
7. What Protection Actually Provides
An operating system's security and protection subsystem provides several core guarantees:
Privilege
Restricts hazardous hardware operations (such as page table modifications or device register configurations) to trusted kernel code.
Isolation
Ensures that one process's memory space and files are completely hidden and protected from unauthorized modification by another process.
Resource Control
Prevents any single process from monopolizing physical devices, allocating disk blocks, or consuming unfair quantities of network bandwidth.
Permissions / Access Control
Validates user security tokens and group policies before permitting a system call to reach the underlying hardware drivers.
Controlled Entry
Forces all user-to-kernel transitions to pass through defined interrupt vectors, preventing malicious code from jumping directly into random kernel memory locations.
8. EdgeCase — The Protected Device
Use the simulator below to compare how the kernel evaluates hardware write requests from an authorized system task (Application A) versus an unauthorized guest task (Application B):
9. Connecting the Two Concepts
When we look at the complete OS stack, device management and security boundary checks work in unison:
APPLICATION
│
System Call
↓
KERNEL
/ \
/ \
Protection Device Driver
│ │
↓ ↓
Permission Poll / IRQ / DMA
│
↓
DEVICE
The key realization is that the OS is not merely a scheduler or a memory manager. It is also the controlled boundary between applications and the machine.
10. Embedded-System Connection
In raw bare-metal embedded firmware, there are often no privilege boundaries. Your software has direct access to the entire memory map. You configure hardware peripherals directly by writing to specialized memory addresses:
* General Purpose I/O (GPIO) direction registers
* Universal Asynchronous Receiver-Transmitter (UART) transmit/receive buffers
* Serial Peripheral Interface (SPI), I2C, and CAN controller control lines
* Analog-to-Digital Converter (ADC) channels, timers, and DMA channels
When you transition to an RTOS or a full operating system with a driver abstraction:
* Peripheral access is encapsulated inside API drivers (e.g. ioctl(), read(), write()).
* Under memory protection configurations (MPUs or MMUs), the OS blocks direct pointer write accesses to peripheral register addresses from application threads.
The amount of abstraction and protection depends entirely on the system architecture.
11. Summary of Concepts
| Property |
Core Responsibility |
Physical CPU Mechanism |
| I/O Device Management |
Abstraction and device configuration |
I/O Ports, DMA, Interrupt Lines |
| Protection / Security |
Isolation and access restriction |
Privilege Rings (User vs. Kernel), Page Tables |
We have now seen how an operating system manages execution, memory, files, storage, devices, and protection.
But not every operating system is designed with the same priorities.
Why do different types of operating systems exist in the first place?
Current Exploration
The Boundary Between Software and Hardware
