Camera motion estimation and object extraction using Multiple Consecutive Frames with ghost and noise removal for Pan-Tilt-Zoom Camera

Abstract

This research presents a new algorithm for extracting moving objects using Pan-Tilt- Zoom (PTZ) camera. Previously, system that uses moving camera faces some problems, including misalignment between current and template frames, appearance of unwanted objects (ghost), illumination changes, shadow and crowd. This research developed an algorithm to avoid misalignment images, remove ghost and noises. The proposed algorithm consists of six steps, which are camera motion estimation, object extraction, removing ghost, detecting shadow, refining shadow and noises elimination. This research proposed to apply camera motion estimation twice, which is between three consecutive frames. Keypoints of each image are detected using Speed-up Robust Features (SURF) detector, then produces homography matrix. The homography contains rotation and translation of one image from another image. It is used to warp previous frames with respect to the current frame. In object extraction, current frame is compared to both compensated previous frame 1 and compensated previous frame 2 using Wronskian Change Detector (WCD). Detecting changes using multiple frames produces ghost, which is actually moving objects in previous frames. Then, this research has developed a ghost removal technique, in which two output images of object extraction are compared each other, pixel by pixel. Then the existing method of shadow removal, Normalized Cross-Correlation (NCC) technique is applied to refine the output image. Some pixels may be misclassified as shadow pixels. Therefore, refinement of shadow detection is done so that the actual shadow is removed, while the false detected shadows are returned to be background entities. To remove other noises, a 3x3 noise filter has been created. The filter is used to scan the output image where the centre of the 3x3 window will look for white pixel. Number of white pixel in the whole window (filter) will be compared to the threshold. Finally, morphological operator is used to remove undesirable foreground pixels. The developed algorithm had been tested on seven image conditions; striped background, objects move slowly, camouflage object, small moving object, multiple moving objects, objects move towards the camera and shadow. The developed algorithm has successfully extracted moving object, removed ghost, removed shadow, noises and detected illumination changes. Based on the manual calculation and visual observations, this system has the highest average accuracy which is 95.13%, followed by single WCD 93.99%, Background Subtraction 93.42%, Luminance Ratio 92.92%, HSV Histogram 91.85% and Greyscale Histogram 91.47%.