Someone behind you memorizes that 4-digit or 6-digit code, and then they steal your handset when you're not paying attention. They can now unlock the phone.