I don't think that's necessarily true. Phantom Hourglass is the first game I played and I think it's the worst in the series (Other than the CD-i ones). Also, don't most of the posts in this thread go against that idea?
What was the Zelda game that got you Interested in the series? Not your first but the one that made you a big fan. While I played Phantom Hourglass and the original Legend of Zelda first, Ocarina of Time got me hooked. I had enough experience by then that I didn't get as frustrated as I used to...