Although I do like The Deathly hallows, Half Blood Prince takes the cake. The tale of Tom Riddle's past, Slughorn, Felix Felicis, Draco's reluctance to join the dark side and Dumbledore's ultimate demise is very enthralling. Prisoner of Azkaban takes a sweet third place.
Prisoner of Askaban is my absolute favorite.
Prisoner of Azkaban was a more chill book. Without the impending threat of Voldemort, it explored the more fun side of Hogwarts. Sirius Black's plotline was one of the best parts of it.
I agree totally, it just seemed more magical.