I suppose a handful of things could constitute a good boss.
Something I personally like is dynamic battles, or battles with two or more phases. I appreciate the changing up of the potential monotony of the original phase to a whole new beast, so to speak, that requires a shift in strategy. It can also work well if it's only but a minor change made to the battle, so the structure of the fight is familiar, yet fresh.
When considering damage infliction (on the boss), hacking it with the sword until it stops moving should generally be avoided if the boss has a very telegraphic attack pattern. This design could work if the attack style of the boss could be easily be mixed and matched, allowing it to be unpredictable. An alternative method of damage infliction (which is very prevalent in the 3D games) is getting the boss to its critical point (usually with an item other than the sword) and slashing it from there. This works decently, so long as it doesn't constitute the entire battle. An easy remedy for this is adding a phase or two to the fight that shakes things up a bit (see above).
A boss, being true to its name, should be a challenging fight that actually requires effort from the player. Now, all bosses don't have to be put on a Dark Souls level, of course not. But some past bosses could have really done with some difficulty scaling. Hopefully future bosses will be an obvious upgrade in terms of challenge compared to said previous bosses. Back on point, difficulty can not only come from the amount of damage the boss deals to the player, but also the amount of thought required to figure out the boss' weakness. Personally, I think adding puzzle elements to bosses really benefits the challenge, and it's certainly a nice variation from simply using a newly acquired item on the boss. If implemented well, the results should be very satisfactory.
I could spend a considerable amount of time with the likes of themes and atmosphere, but I'll summarize it with this: "Good arena. Good music. Good design. Good boss."