I usually suck at stealth in games, so it seems strange that I managed to do this so easily when others have struggled.
The game basically shows you where you need to drop the bananas to lure them away with the collapsible platforms dotted around... shoot the support with an arrow to drop the bananas and just wait for them to notice.
If you do need to drop bananas yourself, remember to change the drop distance (Y while holding if I remember correctly) so that you don't have to get too close.
Also... keep an eye out for ladders... if you can get into the rafters you can move about more freely and work out their movement patterns from above... then it's just a matter of getting down without being seen.