I can't say the Hyrule Castle one was the worst/hardest for me, but it WAS a huge pain. Some of the memories in the open world were hard to pinpoint the exact location, but once I did it was easy enough to find the spot. I think the hardest one of those would be the one in the woods. It just took forever of going through wooded areas to find that one.
Now, the one at the castle I knew was there just from the pic ... but with the castle in ruins now, it was very hard for me to find that spot. Took me a LONG time, but at least I knew where it was and just had to keep looking until I found it.