Welcome to my favorite genre of game!

As they've already been mentioned before, I will second both La-Mulana (and though not mentioned, it's sequel), and FEZ.

The Witness didn't make me break out Pen and Paper, but it was definitely one I had to give my brain a break and come back to to look at things from a fresh angle.

Now for some previously unmentioned finds where your attention to detail will be rewarded. Not all will require notebooks. But there puzzles hidden in all of them.

Environmental Station Alpha is a Metroidvania.

Full Metal Furies is a side-scrolling beat-em up.

INSIDE is a side-scrolling platformer.

Inscryption is a deck-builder / puzzle room.