I agree with you. Due to Majora's Mask being a side game too, many of these questions will be left unanswered for fans to theorize about.
That said, Ocarina of Time is the connecting point of the timeline and nearly every game after alludes to it in one way or another.