I do not think I would go as far as call it a neighboring country. It was a separate province in the Greater Hyrule autocracy that might have been a relatively independant but lacking in any real government since it was only a small village out in the eastern wilderness. That Hyrule either absorbed through size or need for defense by the people of Ordon. They were a similar people that spoke the same language as the Hylians. Many feudal countries and city states held on to outer regions like that, considering the villages a part of the kingdom, but left the people of those villages mostly alone. Sometimes sending a single governor to keep the people in line or maintain contact with the royal court.
They appear to be little more than just an outlying village that is for the most part left alone in the greater scheme of things as far as Hyrule is concerned. But they still pay a regular tribute to the Hylians, the ordon sword was this in the game, since they are subjects in the Hyrule kingdom. Hyrule did appear to have annexed the Gerudo Desert some time in their past and repurposed the Arbiter's Grounds into a prison, the Zora kingdom exists completely within Hyrule's borders and is a subject sate. The Goron cist could be considered one as well, but their home is much smaller and closely resembles another tiny village within the lands of Hyrule. Separating lands into smaller provinces is common for kingdoms such as this as well, It helps the King rule the greater kingdom more easily and leaves the outer provinces to be governed by others loyal to the king, I believe this is what Nintendo was trying to portray in TP.