Models, Algorithms and Architectures for Video Analysis in Real-time

This thesis is meant to be the final report of three years of research in the context of the Doctoral Curriculum “Dottorato di Ricerca in Ingegneria dell’Informazione (XIV Ciclo)” on the topic of video analysis in real-time. High speed processing of videos is a key need for many fields; first, multi-media applications in which the videos are growing as relevance. Think for instance to the videos through the Internet web: standards as the MPEG-1, MPEG-2, MPEG-4 and the upcoming MPEG-7 are video codecs (COmpressor-DECompressorS) very frequently used to broadcast videos through the web. In fact, the bandwidth limitation of current web infrastructures prevents from the transmission of a huge video as it is. Compression before transmitting it and decompression to view it “at the other side” is more efficient since it allows less bandwidth consumption. In the MPEG standards (especially in the more recent ones) the main part of the codec algorithm is the shape coding of the objects that are moving in the scene: this is, indeed, a typical video analysis task. A second very large field of application of the video analysis is the pure information extraction from the video itself. The “level” of the information to be extracted characterizes the video analysis application. Those applications range from the shot detection (low level of information) to the object detection and tracking (medium level) to the scene understanding and modeling (high level). For example, the shot detection task is used to segment a video into
scenes, where a scene is a subsequence of the video (i.e. a sequence of
consecutive frames) with a homogeneous context. This is a very useful task for indexing videos and for context-based information retrieval from videos. The object detection and tracking from a sequence of images is probably the more spread field of video analysis applications. It is a key process for video-based traffic analysis and management systems, for video-surveillance and security systems, for target detection and pointing in military applications, and for many other applications. Therefore, the researches on video analysis reported in the literature are basically on this topic. Lastly, the scene understanding and modeling task uses the information from the lower levels to model the scene (and, typically, also the objects present in the scene) in order to understand the behaviour of the objects or to represent the scene with a higher level of description. All the above-mentioned applications typically require a real-time (or quasi real-time) execution and are characterized by a huge amount of data to be processed. For instance, the real-time processing of a video at the standard PAL (25 frames/sec) at a low resolution of 320x240 pixels. If we have color images (that is 3 channels for each pixel by using the RGB color space), each frame will require 320x240x3 bytes and it must be processed in 40 msec. With this
low resolution a simple transfer of data will require, indeed, a bandwidth of 5.49 MB/sec!!! Studying and, consequently, improving the performance and the efficiency of such systems is one of the main topic of the research described in this thesis. The study has focused both on the hardware and on the software point of view, trying to propose solutions that fit both with an embedded specialized system and with a general-purpose one. Besides improving the performance of video analysis applications, during this research new computational models and algorithms for video analysis has been analyzed and defined. In particular, this research has developed novel algorithms for motion detection and moving object segmentation from cluttered and hostile environments, such as outdoor scene in which the sudden changes of the light conditions, the frequent occlusions of moving objects by
means of buildings, poles, and so on, and the presence of shadows, are very limiting factors.
Moreover, this research covers also motion analysis in “high speed” videos, that is videos in which the objects are moving with a very high speed and in which the noise often renders the images almost unusable. This last topic is very promising and little research has been done (for now) on it by the computer vision community.