Maybe the fish-men are reincarnations of soldiers that deserted the King when it started to flood. What I'm saying is this: the soldiers deserted, were killed in the flood, and were reincarnated as fish to pay the price of deserting them, or maybe the goddesses told them to stay in the castle...