Ginger/trunk/RTCSA_SS/02Background.tex

\section{Background and Related Works}
\label{sec:bk_related}

This section describes background of Super-resolution technique, and the sensors
we are using in this work. The related works describes the works that using
thermal image to recognize activity.

\subsection{Background}
\paragraph{Super-Resolution Convolutional Neural Network}

C. Dong et al.~\cite{ChaoDong16} proposed a deep learning method for single image
super-resolution (SR). Their method learns how to directly map the low-resolution
image to high-resolution image. They show that the traditional sparse coding
based SR methods can be reformulated into a deep convolutional neural network.
SRCNN consists three operations, illustrated in Figure~\ref{fig:SRCNN_model}.
The first layer of SRCNN model extracts the patches from low-resolution image.
The second layer maps the patches from low-resolution to high-resolution. The
third layer will reconstruct the high-resolution image by the high-resolution
patches.

\begin{figure}[htb]
	\begin{center}
		\includegraphics[width=1\linewidth]{figures/SRCNN_model.pdf}
		\caption{Illustration of super-resolution convolutional neural network (SRCNN) model~\cite{ChaoDong16}.}
		\label{fig:SRCNN_model}
	\end{center}
\end{figure}


\subsection{Thermal cameras}
In this work, we use two different resolution thermal camera to play the role
of low-resolution and high-resolution camera. For low-resolution camera, we use
Grid-EYE thermal camera. Grid-EYE is a thermal camera that can output
$8 \times 8$ pixels thermal data with $2.5^\circ C$ accuracy and $0.25^\circ C$
resolution at $10$ fps. For the high-resolution one, we use Lepton 3. The
specification of them are shown in Table~\ref{table:specification of devices}.

\begin{table}[hb]
  \centering
  \footnotesize
  \begin{tabular}{|p{3cm}<{\centering}|p{2cm}<{\centering}|p{2.5cm}<{\centering}|}
    \hline
    Specification & Grid-EYE & Lepton 3\\
    \hline
    Resolution & 64 pixels (8x8) & 120x160\\
    \hline
    FOV-horizontal & $60^{\circ}$ & $49.5^{\circ}$\\
    \hline
    FOV-vertical & $60^{\circ}$ & $61.8^{\circ}$\\
    \hline
    Frame rate & 10Hz & 8.7Hz\\
    \hline
    Detect temperature range & $-20^{\circ}$C to $80^{\circ}$C & $0^{\circ}$C to $120^{\circ}$C\\
    \hline
    Output format & Absolute temperature (Celsius) & 14-bits value which is relative to camera temperature\\
    \hline
    Temperature accuracy & $\pm2.5^{\circ}$C & -\\
    \hline
    Temperature resolution & $0.25^{\circ}$C & -\\
    \hline
    Bits per pixel & 12 & 14\\
    \hline
    Data Rate & 7.68 kbps & 2.34 mbps\\
    \hline
  \end{tabular}
  \caption{Specification of Grid-EYE~\cite{grideye_datasheet} and Lepton 3~\cite{lepton_datasheet}.}
  \label{table:specification of devices}
\end{table}

\subsection{Related Works}
X. Chen et al.~\cite{7805509} proposed to use the visible camera as a guidance to super resolution the IR images. They proposed the IR-RBG multi-sensor imaging system. The approach bases on the fact that RGB channels have different correlation with IR images. To avoid wrong texture transfer, the method used cross correlation to find the proper matching between the region of IR image and RGB image. Next, the method applied sub-pixel estimation and guided filtering to remove the noise and discontinuities in IR image. At the last step of the iteration, they used truncated quadric model as the cost function to terminate the algorithm. After the iteration, the method fine tuned the output IR image with outlier detection to remove the black points which only appear at the edge of IR image. 
% \begin{figure}[ht]
%   \begin{center}
%     \includegraphics[width=0.7\linewidth]{figures/color_guided_SR.pdf}
%     \caption{Flow chart of IR-Color multi-sensor imaging system.}
%     \label{fig:color_guided_SR}
%   \end{center}
% \end{figure}

F. Almsari et al.~\cite{8713356} applied Generative Adversarial Network (GAN) to the super-resoltuion problem called {\it TIGAN}. TIGAN consists of two networks, generator and discriminator, which learn from each other with zero-sum games and try to find an equilibrium state. The generator was trained to transfer low-resolution thermal image into super-resolution thermal image domain. The generated image should be similar to its ground truth high-resolution image domain. While the discriminator was trained to discriminate between generated image and ground truth high-resolution image. Compared to SRCNN model, it preserves the high-frequency details and gave shaper textures.

% \begin{figure}[hb]
%   \begin{center}
%     \includegraphics[width=0.7\linewidth]{figures/GAN_SR.pdf}
%     \caption{Network architecture of TIGAN.}
%     \label{fig:GAN_SR}
%   \end{center}
% \end{figure}
% % background

The assumption on the resolution of thermal images in above works is at least $120 \times 160$. Compared to the $8 \times 8$ thermal images collected by Grid-EYE sensors, more thermal features are available. In addition, these works use RGB images as reference to train the model. In this work, RGB camera will not be used so as to protect privacy.