Indeed. But would it be unreasonable to speculate that Link's mother was escaping, leaving Hyrule Town? Say, that there was a war going on during the time, and Link's father was killed in the war.
However, what if Link's father wasn't killed in a battle, per se, but in the Castle Town...