PCI devices interact with the BIOS/OS through a series of configuration registers that are read during power up. This permits discovery of the resources required by the device e.g. io memory and interrupts.
A device can have up to 6 base address registers (BAR0-5) which serve 2 purposes:
- Inform BIOS/OS about IO memory blocks (buffers, control registers etc.)
- BIOS/OS writes memory map values to BAR(s) during startup
Example - Xilinx Memory Board
Device has a single base register (BAR0) configured. It indicates the device has a 1Mi memory block to map into the processors address space. Address space is allocated on startup and the base address is written to BAR0 i.e. making it available to hardware on the device itself and OS device drivers.
This allocation is readily visible in /proc/iomem:
fe800000-fe8fffff : PCI Bus 0000:01
fe800000-fe8fffff : 0000:01:00.0
One thing I learned is that it is possible to read/write device memory without a device driver being present. You don’t need to write any kernel code. The device can be accessed directly from user space via the sysfs file system.
Linux uses a virtual file system (think memory disk) to make values in the kernel readily available to user space. Each attached device has a directory under /sys. For example, the Xilinx device is nested under /sys/bus/pci/devices/0000:01:00.0.
foo@mhv755:/sys/bus/pci/devices/0000:01:00.0$ ls -l
total 0
-rw-r--r-- 1 root root 4096 Nov 30 14:31 broken_parity_status
-r--r--r-- 1 root root 4096 Nov 30 14:31 class
-rw-r--r-- 1 root root 4096 Nov 30 10:16 config
-r--r--r-- 1 root root 4096 Nov 30 14:31 consistent_dma_mask_bits
-rw-r--r-- 1 root root 4096 Nov 30 14:31 d3cold_allowed
-r--r--r-- 1 root root 4096 Nov 30 14:31 device
-r--r--r-- 1 root root 4096 Nov 30 14:31 dma_mask_bits
-rw-r--r-- 1 root root 4096 Nov 30 14:31 driver_override
-rw-r--r-- 1 root root 4096 Nov 30 14:31 enable
-r--r--r-- 1 root root 4096 Nov 30 10:16 irq
-r--r--r-- 1 root root 4096 Nov 30 14:31 local_cpulist
-r--r--r-- 1 root root 4096 Nov 30 14:31 local_cpus
-r--r--r-- 1 root root 4096 Nov 30 14:31 modalias
-rw-r--r-- 1 root root 4096 Nov 30 14:31 msi_bus
-rw-r--r-- 1 root root 4096 Nov 30 14:31 numa_node
drwxr-xr-x 2 root root 0 Nov 30 14:31 power
--w--w---- 1 root root 4096 Nov 30 14:31 remove
--w--w---- 1 root root 4096 Nov 30 14:31 rescan
--w------- 1 root root 4096 Nov 30 14:31 reset
-r--r--r-- 1 root root 4096 Nov 30 14:31 resource
-rw------- 1 root root 1048576 Nov 30 14:31 resource0
lrwxrwxrwx 1 root root 0 Nov 30 10:15 subsystem -> ../../../../bus/pci
-r--r--r-- 1 root root 4096 Nov 30 14:31 subsystem_device
-r--r--r-- 1 root root 4096 Nov 30 14:31 subsystem_vendor
-rw-r--r-- 1 root root 4096 Nov 30 10:15 uevent
-r--r--r-- 1 root root 4096 Nov 30 10:15 vendor
Many of the files listed contain a single text value. The file ‘irq’ contains the text ‘16’, indicating that the device has been allocated IRQ line #16. Handy to know when you want to set up the interrupt handler.
File ‘resource0’ represents the previously mentioned memory block. Rather that having to know its physical memory address we can access it by its sysfs logical filename.
The basic idea from user space is to:
int fd = open("/sys/bus/pci/devices/0000:01:00.0/resource0", O_RDWR | O_SYNC);
unsigned char * map = (unsigned char *)(mmap(NULL, 16384, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0));
map[0] = 0x5a;
munmap(map, 16384);
This makes 16kiB of the total 1MiB available to the program as an array of unsigned char. The assignment map[0] = 0x5a; is writing to memory physically located on the Xilinx device.
Pretty neat huh?