Face Recognition Play A Vital Role In Variety Of Applications From Biometrics, Surveillance, Security, Identification To The Authentication. In This Paper We Design And Implement A Smart Security System For Restricted Area Where Access Is Limited To People Whose Faces Are Available In The Training Database. First We Are Going To Detect The Face By Detecting The Human Motion. Then Face Recognition Is Performed To Determine The Authority Of The Person To Enter The Sensitive Area. At The Same Time, We Track The Coordinate Of Detected Motion. Failing To Recognize The Face Finally Passes The Estimated Coordinate To Detect The Intruder Automatically. Experimental Results Demonstrate The Effectiveness Of Proposed Security System In Order To Restrict The Unauthorized Access And Enhanced Reliability By Use Of Face Recognition. Although This Reduces The Complexity Of Face Recognition, There Is Still A Concern Regarding The Real-time Protection Of Sensitive Portion. It Should Be Noted That This Problem Is Somewhat A Hard Task And Can Be Solved By Automatically Send Mail The Admin Person Attempting To Trespass The Sensitive Area.