No, they are not all adult men.
First they are not all men, we see both male and female travelers who turn out to be Yiga. And the most common (and very first) one we see is a young women probably in her 20's or 30's. So, not all of them are even original members, some are quite young...