It is probable that ghost would be produced when background model is established with pixel-based adaptive segmenter algorithm. According to the theory that adjacent pixel points have similar pixel values in spatial distribution in VIBE algorithm, the difference of foreground pixels was computed which are judged in the first time and the neighborhood pixel. Then the difference and adaptive threshold value were compared through weight to determine whether the pixel is foreground pixel point or ghost which will be judged as background pixel. At last the background models would be updated. Through twice judgments of the foreground pixels, the ghost would be removed quickly. The experiment results show that the improved algorithm removes the ghost faster than the original method.