[can-eth-gw.git] / doc / can-eth-gw.tex

%
% Typographic conventions (already applied)
% * Each \caption{} should end with a dot
% * "Linux traffic control" will be written with lowercase T and C
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\documentclass[11pt,abstract]{scrreprt}
\usepackage[utf8]{inputenc}
% \usepackage[T1]{fontenc}
\usepackage[right]{lineno}
\usepackage{fixltx2e}
\usepackage{graphicx}
\usepackage{longtable}
\usepackage{tabularx}
\usepackage{float}
\usepackage{wrapfig}
\usepackage{soul}
% \usepackage{t1enc}
\usepackage{textcomp}
\usepackage{marvosym}
\usepackage{wasysym}
\usepackage{latexsym}
\usepackage{amssymb}
\usepackage{hyperref}
\usepackage{multirow}
\usepackage{tikz}
\usepackage[draft]{fixme}
\usepackage{vertbars} 
\usepackage{fancyvrb}
\usepackage{appendix}
\usepackage{listings}
\usepackage{color}
\usepackage{caption}

%\usepackage{multicol}
\hypersetup{
        breaklinks = true, %allow links to break over lines
        pdffitwindow = true, %resize document window to fit document size
        colorlinks = true,
%       linkcolor = darkblue,
%       citecolor = darkblue,
%       urlcolor = darkblue,
        linkcolor = black,
        citecolor = black,
        urlcolor = black,
        plainpages=false,
        }
%\urlstyle{same}

\RecustomVerbatimEnvironment{Verbatim}{Verbatim}{frame=single,framesep=4pt}


\setlength{\columnsep}{8mm}
\providecommand{\alert}[1]{\textbf{#1}}

\title{Linux-Based CAN-Ethernet Gateway}
\author{R. Matějka, M. Sojka\\
Czech Technical University in Prague}
\providecommand{\version}{???}
\date{\today\\Version \version}

\newcommand{\superscript}[1]{\ensuremath{^{\mathrm{#1}}}}
\newcommand{\subscript}[1]{\ensuremath{_{\mathrm{#1}}}}
\renewcommand{\th}[0]{\superscript{th}}
\renewcommand{\st}[0]{\superscript{st}}
\newcommand{\nd}[0]{\superscript{nd}}
\newcommand{\rd}[0]{\superscript{rd}}

\newenvironment{todo}
{\begin{vertbar}\begin{itshape}}
    {\end{itshape}\end{vertbar}}
\pagestyle{headings}
\begin{document}
\lstset{language=sh,frame=single,basicstyle=\tt,commentstyle=\itshape\color{gray},columns=flexible}
%\linenumbers
%\onecolumn
\maketitle

% \begin{abstract}
% \end{abstract}

\tableofcontents

\chapter{Assignment}
\label{cha:assignment}

The goal is to implement CAN-Ethernet gateway based on Linux's AF\_CAN
subsystem with the following features:
\begin{enumerate}
\item Both user- and kernel-space implementations will be developed.
  User-space one will contain only the most basic functionality.
\item Ethernet side will use UDP datagrams to carry the CAN messages.
  Later it will be extended to support also TCP.
\item Initial version will route all CAN frames to Ethernet side and
  all received UDP datagrams to CAN messages. Later, filtering
  capabilities will be added.
\item UDP frames will contain timestamps of the time when the CAN
  message was received.
\item For kernel-based gateway, implement a user-space configuration
  tool, similar to \texttt{cangw} tool from \texttt{can-utils}.
\end{enumerate}

%% OVERVIEW %%
\chapter{Project overview}
\section{Gateway architecture}
\label{sec:arch}
The software architecture of canethgw is illustrated on figure \ref{fig:swarch}.
This section describes what can be seen on this figure.

The can-ethernet gateway (canethgw) is comprised of two kernel threads. 
One thread is serving the can to ethernet routing (can $\rightarrow$ eth) 
and the other thread is serving the oposite direction (eth $\rightarrow$ can).

The gateway uses berkley sockets as network interaces. There are two sockets
in the program. One socket is binded to can network and the other one to ethernet. These
two sockets are shared by kernel threads.

To get a better notion about canethgw operation a working cycle description is given.
The working cycle of the can $\rightarrow$ eth kernel thread is as follows. The thread is
waiting for incoming data on \emph{can} socket. When data are received, they are
put to udp packet and are sent to ethernet. There can be more than one recipient in ethernet 
network. The program holds list of recipients and sends copy of incoming data 
to all of them. This working cycle description can be analogically applied to eth $\rightarrow$ can
thread.

The gateway is configured over netlink with cegw tool. More on cegw is in
section \ref{sec:assoc}
\begin{center}
\includegraphics[scale=0.5]{img/swarch.pdf}
\captionof{figure}{Software architecture of canethgw.}\label{fig:swarch}
\end{center}
% \bibliography{can-eth-gw}
% \bibliographystyle{IEEEtran}

%% ASSOCIATED PROGRAMS %%
\section{Associated programs}
\label{sec:assoc}
The project consists not only from canethgw, but also from configuration
utility cegw, userspace implementation of the gateway, benchmarking program 
and other utils. These associated programs are presented below.

\begin{description}
\item[cegw]
Cegw is a configuration utility for canethgw. Its syntax is similar to 
cangw tool from can-utils. The difference from cangw is that every 
interface has to be defined with type-prefix. This can be explained well
 using an example. Figure \ref{fig:cegwex} displays commands to do a simple 
setup of canethgw. This setup sends all can frames received on 
vcan0 to 127.0.0.1:10502 and all can frames received from 127.0.0.1:10501 
to vcan0.
\begin{lstlisting}
# listen for incomping udp at 127.0.0.1 port 10501
cegw --listen udp@127.0.0.1:10501
# all from vcan0 route to 127.0.0.1 port 10502
cegw --add -s can@vcan0 -d udp@127.0.0.1:10502
# all from source address .. route to vcan0
cegw --add -s udp@127.0.0.1:10502 -d can@vcan0
\end{lstlisting}
\captionof{figure}{Simple setup using cegw.}\label{fig:cegwex}

\item[userspace implementation]
To measure a performance of canethgw, another implementation of
gateway was done in userspace. This implementation resembles
the kernel implementation but \emph{poll} is used instead of threads.
How to start a simple setup is in figure \ref{fig:userspace}.
\begin{lstlisting}
canethgw -s can@vcan0 -d udp@127.0.0.1:10502 -l udp@127.0.0.1:10501
\end{lstlisting}
\captionof{figure}{Command to run userspace canethgw.}\label{fig:userspace}

\item[cegwbench]
This is a benchmarking program. It sends can frames to defined interface and
listens on another interface for reception. Then it prints the transmission
duration to stdout. How to start simple cegwbench session is in figure
\ref{fig:runbench}. This program was used for benchmark described in
section \ref{sec:benchmark}.
\begin{lstlisting}
cegwbench -s udp@127.0.0.1:10501 -d can@vcan0 -n 100 -m oneattime
\end{lstlisting}
\captionof{figure}{cegwbench example.}\label{fig:runbench}

\item[cesend]
Sends a can frame over udp to localhost:10501. This is based on cansend 
from can-utils and the can frame specification string is the same.
\end{description}

%% DESIGN ISSUES %%
\chapter{Design issues}
This chapter encompasses problems which came up during development. 
This knowledge may be helpful in further research.

The original cangw was designed in modular way, so the first attempt to
design canethgw was done on cangw basis. The original cangw works with
sk\_buff structure and in softirq. However, there are some issues,
which complicate applicability of this approach for canethgw. These
problems are listed below:

\begin{description}
\item[sending udp packet in softirq]
The original cangw is receiving can frames in softirq. If one wanted 
to send such a can frame to ethernet with udp/ip packet, he have to 
avoid sleeping which is not easy. I have not found a way how to send a
udp packet using the protocol stack without sleeping. I also think
that bypassing protocol stack and reimplementing udp and ip is not a
good idea, because it will induce bugs.

\item[receiving udp packet]
The gateway have to hook somewhere to receive udp or tcp packets.
I have not found any convenient method to do this.
\end{description}

These problems led me to search for a different approach. The found solution
is based on berkley sockets. Even this approach has some issues, which cause
difficuties. These difficulties are listed below. For more complete 
information about this solution see section \ref{sec:arch}.

\begin{description}
\item[socket creation]
\label{itm:socket_creation}
It is not possible to create a \emph{can} socket in a netlink callback. Such an
attempt will cause rtnl\_lock deadlock. rtnl\_lock is being 
held when netlink callback
is processing, but the same lock is needed by register\_netdevice\_notifier
\cite{register_netdevice_notifier} function when \emph{can} socket is 
being created.
This problem can be solved by postponing the creation process.

\item[socket rebinding]
Once the socket is binded to a particular address it cannot be
changed. Rebinding is not possible. The overcome for this is to release the
old socket and create a new one.

\item[netlink feedback]
This problem is tightly linked to the socket creation issue described above.
If some operation cannot be done in netlink handler and have to be
postponed, the netlink handler can hardly return any feedback about an
operation result. That is why the configuration tool (cegw) doesn't return
information about an operation result.
\item[terminating thread]
The termination of kernel thread is not straightforward in this case, 
because the needs are slightly different than kernel api. The standard kernel
thread is working in a cycle waiting for kthread\_should\_stop(..) to return
true. This approach is not possible because the thread can be sleeping in
kernel\_sendmsg function and therefore not responding to kthread\_stop. Waking
using wake\_process won't work, because no incoming data will cause sleep
again. The socket shutdown is used for waking from receiving.
\end{description}

\chapter{Benchmark}
\label{sec:benchmark}
To evaluate canethgw a benchmark was conducted. The objective was
to compare performance of the gateway implementated in
userspace and in the kernel. 

The benchmark was designed as a set of four measurements. There is a separete
measurement for each of both directions, that is eth$\rightarrow$can and
can$\rightarrow$eth. These both directions are measured with userspace and
kernel implementation. In every of four measurements 100 can frames
were transceived.

The benchmark was performed on dedicated computer running very bare, 
single-purpose distribution to avoid external effects. The results show 
that kernel implementation is approximately 10\% faster. Figure \ref{fig:bench} 
shows transmission duration of every can frame sent.

Furher information, including benchmark source code, can be found in 
\emph{bench} folder in repository\cite{repo}.

\begin{center}
\includegraphics[scale=0.35]{img/bench.png}
\captionof{figure}{canethgw performance.}\label{fig:bench}
\end{center}

\appendix
\chapter{Repository contents}
Repository can be found at \cite{repo}. Below is a list of folders and 
their content.
\begin{center}
\begin{tabular}{l p{11cm}}
\textbf{bench/} &
Scripts which prepare and perform benchmark on dedicated computer are 
here. \\
\textbf{distro/} &
Linux distribution used for testing and benchmarking.
Busybox and kernel configuration files can be found here. \\
\textbf{doc/} &
Folder with documentation. \\
\textbf{kernel/} &
Can-ethernet gateway kernel implementation. \\
\textbf{test/} &
Contatins scripts used in debuging to test functionality. \\
\textbf{user/} &
Can-ethernet gateway userspace implementation. \\
\textbf{utils/cegw/} &
Configuration utility for kernel canethgw. \\
\textbf{utils/cegwbench/} &
Program used for benchmark. \\ 
\textbf{utils/cesend/} &
Sends can frame over udp. \\
\end{tabular}
\end{center}

\begin{thebibliography}{9}
\bibitem{repo} Project repository:
  \url{https://rtime.felk.cvut.cz/gitweb/can-eth-gw.git}
\bibitem{register_netdevice_notifier}
register\_netdevice\_notifier(..): \url{http://lxr.linux.no/linux+v3.5.3/net/core/dev.c#L1351}
\end{thebibliography}

\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: t
%%% ispell-local-dictionary: american
%%% End: