There are some RPGMaker games that give a very spooky vibe. Try playing Ravenwood Horror.

Another good tactic is to make it so you know that one mistake and you're dead. Dead Space did a VERY good job at this, where I found the most scary part of the game is that if you don't pay attention, you can die with ease. People don't like dying in games.