ALttP was the most difficult for me simply because it was the first Zelda game I played, so I had no idea how things were done. I died so many times just getting through the Eastern Palace. I've since become an aLttP master though, and I can easily get through the whole game without dying.
I waited until I had completed most of the other Zelda games before I even bothered trying LoZ and AoL though, because I had heard they were very hard. With all that Zelda experience now under my belt, I beat the 1st quest of LoZ with relative ease. And though I haven't beaten AoL yet, I made it to the final dungeon fairly easily. (AoL is on hiatus though, and I haven't really put up a great effort to complete LoZ 2nd Quest yet).
With that said though, AoL would definitely be the hardest for a person that doesn't have alot of Zelda experience yet.