Spirit Tracks. Definitely.
Phantom Hourglass was an incredibly easy game, actually. The paths were very straighforward in the temples, and there were a few rough spots when traveling in the overworld, but for the most part, the game wasn't a challenge at all. Phantom Hourglass was really more on the frustrating side, having to sail everywhere, going really slowly, the Temple of the Ocean King, it really made me hate playing the game. Naturally, some of this spilled over into Spirit Tracks, but fortunately, not all of it.
Spirit Tracks was definitely the more difficult game. The bosses I had a little bit more trouble with, and the temples were more difficult, especially the latter parts of the Tower of Spirits. No light, very difficult and annoying phantom enemies, no set or obvious path of where to go, this part of the game along is more difficult than all of Phantom Hourglass.
Now, as said before, some of Phantom Hourglasses's mishaps did cross over to ST, train riding, slow pace, useless sidequests. But they only serve to make the game more annoying, not harder.