1 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
3 %% paper.tex = template for Real Time Linux Workshop papers
5 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
6 \documentclass[10pt,a4paper]{article}
7 \usepackage[english]{babel}
10 \setlength{\paperheight}{297mm}
11 \setlength{\paperwidth}{210mm}
12 \setlength{\voffset}{-12mm}
13 \setlength{\topmargin}{0mm}
14 \setlength{\headsep}{8mm}
15 \setlength{\headheight}{10mm}
16 \setlength{\textheight}{235mm}
17 \setlength{\hoffset}{-4mm}
18 \setlength{\textwidth}{166mm}
19 \setlength{\oddsidemargin}{0mm}
20 \setlength{\evensidemargin}{0mm}
21 \setlength{\marginparwidth}{0mm}
22 \setlength{\marginparpush}{0mm}
23 \setlength{\columnsep}{6mm}
24 \setlength{\parindent}{6mm}
25 \setlength{\parskip}{2mm}
27 %% insert eps pictures
28 %% use as \epsin{epsfile}{width_in_mm}{label}{caption}
30 \newcounter{figcounter}
32 \refstepcounter{figcounter} \label{#3}
41 \parbox{7cm}{{\bf FIGURE \arabic{figcounter}:}\quad {\it #4 } } \\
46 %% use as \tabin{size_in_mm}{label}{caption}{table_data}
47 \newcounter{tabcounter}
49 \refstepcounter{tabcounter} \label{#2}
50 \[ \makebox[#1mm][c]{#4} \]
53 \parbox{7cm}{{\bf TABLE \arabic{tabcounter}:}\quad {\it #3 } } \\
57 \clubpenalty 10000 % prvni radek odstavce nebude sam na konci stranky
58 \widowpenalty 10000 % posl. radek odstavce nepujde na novou stranku
61 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
62 %% TITLE OF PAPER (REQUIRED)
63 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
64 COMEDI and UIO Drivers for PCI Multifunction\\Data Acquisition and Generic I/O Cards\\and Their QEMU Virtual Hardware Equivalents
68 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
70 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
71 {\bf Pavel P\'{i}\v{s}a }\\
72 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
73 %% AFFILIATION (REQUIRED)
74 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
75 Czech Technical University in Prague, Department of Control Engineering\\
76 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
77 %% STREET ADDRESS (REQUIRED)
78 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
79 %Technick\'{a} 2, 166 27 Praha 6, Czech Republic FIXME\\
80 Karlovo n\'{a}m\v{e}st\'{i} 13, 121 35 Praha 2, Czech Republic\\
81 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
83 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
85 pisa$@$cmp.felk.cvut.cz \\
86 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
88 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
89 {\bf Rostislav Lisov\'{y}}\\
90 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
91 %% AFFILIATION (REQUIRED)
92 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
93 Czech Technical University in Prague, Faculty of Electrical Engineering\\
94 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
95 %% STREET ADDRESS (REQUIRED)
96 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
97 %Technick\'{a} 2, 166 27 Praha 6, Czech Republic\\
98 Karlovo n\'{a}m\v{e}st\'{i} 13, 121 35 Praha 2, Czech Republic\\
99 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
101 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
111 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
112 %% ABSTRACT (REQUIRED)
113 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
115 The article describes implementation of UIO and Comedi drivers for Humusoft MF624 and MF614 data acquisition cards. Basic functions (D/A, A/D converters, digital inputs/outputs) of Humusoft MF624 card were implemented into the Qemu emulator as well which enable to experiment with drivers implementation without physical access to the cards and risk of data lost when drivers are developed and tested on same primary Linux kernel instance. The article can help newcomers in the area to gain knowledge required to implement support for other similar cards and hardware emulation of these cards. The matching real and virtual setup can be used in operating system courses for practical introduction to simple drivers implementation and helps with understanding internal computation world with real world computers interfacing.
122 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
123 %% SECTION (REQUIRED)
124 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
125 \section{Introduction}
126 When teaching development of Linux drivers one of the approaches is to explain kernel API and programming paradigms by creating driver which does not require any special hardware -- e.g. character driver which returns upper case ASCII text when receiving lower case. Although this approach can be useful, the issues associated with dealing with hardware should be practised as well.
128 The approach we took in this work eliminates the need of physical access to hardware whereas it provides full feature set of PCI device in form of virtual hardware. This was possible by implementing virtual PCI device into Qemu emulator.
130 The main reason of choosing DAQ cards for this project was easy interfacing from programmer's point of view and straightforward testing of proper function of the driver. This can be very helpful for beginners who are not familiar with hardware related topics.
132 \section{Humusoft MF614, MF624}
133 Humusoft MF614 and MF624 are data acquisition (DAQ) cards. Both of these cards use PCI interface to connect to the computer. The main features this cards provide are \textit{digital inputs}, \textit{digital outputs}, \textit{ADCs}, \textit{DACs}, \textit{timers}, \textit{encoder inputs}. Humusoft MF614 is predecessor of MF624 -- available functions are quite similar. The main difference is in driver programming -- MF614 has only 8-bit wide registers, whereas MF624 has ones 16- or 32-bit wide.
135 MF624 is available for purchase on manufacturer's web page. MF614 is no more produced.
138 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
140 Each UIO driver consists of two parts -- small kernel module (the need for it is mostly because of device-specific interrupt handling/disabling) and user-space driver logic (as shown in figure 1). The main advantage of this approach is that the most of the development happens in user-space, thus during prototyping the driver (or when using a bad one) the integrity and stability of the kernel will not be disrupted.
141 \epsin{img/uio}{80}{fig1:uio}{UIO driver structure}
143 \subsection*{Driver \texttt{uio\_pci\_generic}}
144 When dealing with any device compliant to PCI 2.3, it is also possible to use \texttt{uio\_pci\_generic} driver in kernel instead of programming a specific one. This driver makes all memory regions of the device available to user-space.
146 Binding to the device is done by writing Vendor and Device ID into \texttt{/sys/bus/pci/drivers/ uio\_pci\_generic/new\_id} file.
148 Interrupt handler uses Interrupt Disable bit in the PCI command register and Interrupt Status bit in the PCI status register. Because neither of MF614 or MF624 is PCI 2.3 compliant it is not possible to use this driver for them.
150 \subsection*{Implementing the Kernel Part}
151 In case of writing UIO driver for PCI device, initialization function of the module registers \texttt{struct pci\_driver} in standard way\footnote{For more information about PCI driver development see \cite{book1} available online at \textit{https://lwn.net/Kernel/LDD3/}}, where the probe function handles initialization of UIO-related structures. The main structure holding all data of particular UIO driver is \texttt{struct uio\_info}. Its simple initialization (including registration) is shown below:
154 1 | /* struct pci_dev *dev */
155 2 | struct uio_info *info;
156 3 | info = kzalloc(sizeof(struct uio_info),
159 6 | info->name = "mf624";
160 7 | info->version = "0.0.1";
162 9 | info->mem[0].name = "PCI chipset";
163 10 | info->mem[0].addr =
164 11 | pci_resource_start(dev, 0);
165 12 | info->mem[0].size =
166 13 | pci_resource_len(dev, 0);
167 14 | info->mem[0].memtype = UIO_MEM_PHYS;
168 15 | info->mem[0].internal_addr =
169 16 | pci_ioremap_bar(dev, 0);
171 18 | info->port[0].name =
172 19 | "Board programming registers";
173 20 | info->port[0].porttype = UIO_PORT_X86;
174 21 | info->port[0].start =
175 22 | pci_resource_start(dev, 1);
176 23 | info->port[0].size =
177 24 | pci_resource_len(dev, 1);
179 26 | uio_register_device(&dev->dev, info);
180 27 | pci_set_drvdata(dev, info);
182 Structure \texttt{uio\_mem} is used for enabling memory-mapped I/O regions, whereas structure \texttt{uio\_port} is used for I/O ports (for each of these structures there is statically allocated array with a size of 5 elements).
184 \subsection*{Interface to User-space}
185 Communication with kernel part of the UIO driver is possible through \texttt{/dev/uioX} file (where X is the number of instance of a driver). There are several syscalls possible to be used when interfacing with this file:
187 \item[\texttt{open()}] opens the device, returns file descriptor used for another syscalls.
188 \item[\texttt{read()}] blocks until an interrupt occurs (the value read is number of interrupts seen by the device).
189 \item[\texttt{mmap()}] is used to map memory of the device to user-space. The offset value passed to \texttt{mmap()} de\-ter\-mines the memory area of a device to map -- for \textit{n-th} area offset should be \textit{n*\texttt{sysconf( \_SC\_PAGESIZE)}}.
190 \item[\texttt{irqcontrol()}] is used for enabling (called with parameter set to \texttt{(int) 1}) or disabling (\texttt{(int) 0}) interrupts.
192 It is possible to define your own \texttt{mmap()}, \texttt{open()}, \texttt{release()} functions as an option. When there is need to use \texttt{irqcontrol()}, it is necessary to implement this function per device.
194 Information related to a particular driver instance can be found in \texttt{/sys/class/uio/uioX} directory. Most of the files are read-only. The subdirectory \texttt{maps} contains information about MMIO regions mapped by the driver, subdirectory \texttt{portio} is for I/O port regions.
196 When using UIO and \texttt{mmap()} with MF624 card (which has 32 or 128 bytes long memory regions) there is an issue with the return value of this syscall -- the pointer to the memory is page-size-aligned, so it is necessary to add low bits of physical address (page offset) of each memory region to it. Physical address can be obtained from \texttt{addr} file located in \texttt{/sys/class/uio/uioX/maps/mapX}. Region offset is equal to \texttt{addr \& (sysconf(\_SC\_PAGESIZE) - 1)}.
199 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
200 \section{Comedi Driver}
201 UIO driver is a versatile solution available mainly for uncommon devices. In our case of using DAQ card a special subsystem in Linux kernel designated for DAQ card drivers can be used. It is called Comedi (\textit{Linux control and measurement device interface}). It provides library functions for user- and kernel-space making development and usage of DAQ devices easier. It consists of three different parts.
203 \item[Comedi] is a part of Linux kernel. It consist of individual device drivers including Comedi driver providing basic set of functions used by device drivers.
204 \item[Comedilib] is a user-space library providing unified interface for another user-space application to devices supported by Comedi.
205 \item[Kcomedilib] is also a part of Linux kernel. It provides the same API as Comedilib, whereas this is used for real-time applications.
208 \subsection*{Implementing the Driver}
209 Each Comedi driver should be registered to the list of active Comedi drivers. This is done by invoking \texttt{comedi\_driver\_register()} function. The only parameter passed to this function is pointer to \texttt{struct comedi\_driver} structure. The most important fields of this structure are:
211 const char *driver_name; /* "my_driver" */
212 struct module *module; /* THIS_MODULE */
213 int (*attach) (struct comedi_device *,
214 struct comedi_devconfig *);
215 int (*detach) (struct comedi_device *);
217 Unlike the UIO or generic PCI driver, the main \textit{initialization function} is not \texttt{probe()} (of \texttt{struct pci\_driver}) but \texttt{attach()} (of \texttt{struct comedi\_driver}) which is invoked by Comedi subsystem.
219 The \texttt{attach()} function is responsible not only for common PCI device initialization but also for initialization of \texttt{struct comedi\_device} (which is accessible through a pointer passed to \texttt{attach()} function). The most important step is to allocate and initialize each \textit{subdevice} (in Comedis nomenclature \textit{subdevice} represents one particular function of the device -- e.g. ADC, digital out, etc.) of the DAQ card. Allocation is done by Comedi function \texttt{alloc\_subdevices(struct comedi\_device *dev, unsigned int num\_subdev)}, each \texttt{struct comedi\_subdevice} is then accessible in array called \texttt{subdevices} which is part of \texttt{struct comedi\_device}. Example of initialization of subdevice representing ADC:
221 1 | s = dev->subdevices + 0;
222 2 | s->type = COMEDI_SUBD_AI;
223 3 | s->subdev_flags = SDF_READABLE |
226 6 | s->maxdata = (1 << 14) - 1;
227 7 | s->range_table = &range_bipolar10;
228 8 | s->len_chanlist = 8;
229 9 | s->insn_read = mf624_ai_rinsn;
230 10 | s->insn_config = mf624_ai_cfg;
233 \subsection*{Interface to User-space}
234 After successful compilation and loading of particular Comedi driver, there should be \texttt{/dev/comediX} (where X is number of instance of a driver) file. For communication with this file Comedi library functions are used. For opening device -- \texttt{comedi\_open()}, for reading/writing ADCs/DACs -- \texttt{comedi\_data\_read()}, \texttt{comedi\_data\_write()} and for reading/writing digital inputs/outputs -- \texttt{comedi\_dio\_read()}, \texttt{comedi\_dio\_write()}.
236 There are already applications using Comedi API\footnote{For basic list of available applications see http://www.comedi.org/applications.html} -- thus in some cases there is no need for implementing user-space application from scratch.
237 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
238 \section{Qemu Virtual Hardware}
239 Qemu is an open-source processor emulator. Unlike common virtualization solutions it is able of emulating x86, x86-64, ARM and other widespread processor architectures. For the purposes of this work it was used for implementing virtual Humusoft MF624 DAQ card.
241 \subsection*{Implementation of Virtual PCI device}
242 When creating new virtual device in Qemu, main hook into Qemu device infrastructure is done by invoking \texttt{device\_init()} with parameter of pointer to initialization function with prototype of \texttt{static void (*)(void)}. For registering new PCI device, it is necessary to call \texttt{pci\_qdev\_register()} passing parameter of pointer to \texttt{PCIDeviceInfo}. The most important fields of this Qemu-specific data type are pointers to \textit{init} and \textit{exit} functions with prototype of \texttt{int (*)(PCIDevice *)}.
244 The PCI device specific initialization consists of:
246 \item Initializing configuration space of PCI device -- e.g. setting Vendor and Device IDs, device class, interrupt pin, etc.
247 \item Registration of I/O memory used by the device.
248 \item Creating a function (called when device gets allocated memory from virtual PCI controller) for mapping of physical memory to particular \textit{BARs} (Base Address Registers) of the PCI device.
251 The very basic (non-compilable) example:
255 1 | static CPUReadMemoryFunc * const mf624_BAR0_read[3] = { NULL, NULL, mf624_BAR0_read32 };
256 2 | static CPUWriteMemoryFunc * const mf624_BAR0_write[3] = { NULL, NULL, mf624_BAR0_write32 };
258 4 | static void mf624_map(PCIDevice *pci_dev, int region, pcibus_t addr, pcibus_t sz, int tp)
260 6 | mf624_state_t *s = DO_UPCAST(mf624_state_t, dev, pci_dev);
261 7 | cpu_register_physical_memory(addr + 0x0, BAR0_size, s->BAR0_mem_table_index);
264 10 | static int pci_mf624_init(PCIDevice *pci_dev)
266 12 | mf624_state_t *s = DO_UPCAST(mf624_state_t, dev, pci_dev); /* i.e. container_of() */
267 13 | uint8_t *pci_conf;
269 15 | pci_conf = s->dev.config;
270 16 | pci_config_set_vendor_id(pci_conf, PCI_VENDOR_ID_HUMUSOFT);
272 18 | pci_conf[PCI_INTERRUPT_PIN] = 0x1;
274 20 | s->BAR0_mem_table_index = cpu_register_io_memory(mf624_BAR0_read, mf624_BAR0_write,
275 21 | s, DEVICE_NATIVE_ENDIAN); /* returns unsigned int */
276 22 | pci_register_bar(&s->dev, 0, BAR0_size, PCI_BASE_ADDRESS_SPACE_MEMORY, mf624_map);
280 26 | static PCIDeviceInfo mf624_info = {
281 27 | .qdev.name = "mf624", .qdev.size = sizeof(mf624_state_t),
282 28 | .init = mf624_init, .exit = mf624_exit,
284 30 | static void reg_dev(void) { pci_qdev_register(&mf624_info); }
285 31 | device_init(reg_dev)
289 \subsection*{Usage of Virtual MF624}
290 When running any guest operating system in Qemu (with support for MF624 activated) the virtual MF624 device is available in the same way as if it was real hardware -- there are no issues with interfacing between guest operating system and virtual device. Interfacing between virtual hardware and \textit{real world} is handled by TCP/IP connection from MF624 part in Qemu to \textit{host} operating system. It is used for reading/setting output/input values (as shown in figure 2).
292 The most fundamental way of communication through this channel is by using \texttt{telnet} application. Example of real communication:
294 $ telnet localhost 55555
297 Connected to localhost.
298 Escape character is '^]'.
304 telnet> Connection closed.
307 As a much more easier way of interfacing, there is also graphical application created just for purposes of communication with virtual MF624 card (see figure 3). It was created using Qt4 graphical toolkit.
309 \epsin{img/qemu}{80}{fig2:qemu}{Qemu implementing virtual MF624 device}
312 \epsin{img/qt_gui}{70}{fig3:qemu2}{Graphical application used for interfacing between virtual MF624 and \textit{real world}}
315 The outcome of this work creates basic integrated tool for teaching PCI driver development (mostly) for GNU/Linux operating system. Its main advantage is the possibility to train driver development on \textit{real hardware} without the necessity of having an expensive DAQ device. The other advantage is a safe environment for driver prototyping -- where no mistake can damage host operating system.
317 All the information (including source code) related to topic covered in this article are publicly available on web page \texttt{rtime.felk.cvut.cz/ hw/index.php/Humusoft\_MF6xx}
321 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
322 %% REFERENCES (REQUIRED)
323 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
324 \begin{thebibliography}{9}%use this if you have <=9 bib refs
325 %\begin{thebibliography}{99}%use this if you have >9 bib refs
326 \bibitem {book1}{\it {Jonathan Corbet}, {Alessandro Rubini}, {Greg Kroah-Hartman}, {Linux Device Drivers, 3rd Edition}, {O'Reilly Media}, 2005}
327 \bibitem {paper1}{\it {Hans-J\"{u}rgen~Koch} http://www.kernel.org/doc/ htmldocs/uio-howto.html}
328 \bibitem {paper2}{\it {David Schleef}, {Frank Hess}, {Herman Bruyninckx} http://www.comedi.org/doc/}
329 \bibitem {paper3}{\it Fabrice Bellard et al. git://git.qemu.org/qemu.git}
330 \end{thebibliography}