First of all, I'd like to start with that I loved the DS and PSP and couldn't pick a favourite between those 2. Brilliant little machines.
Hardware-wise, the Vita, easily. I much prefer it to the 3DS in almost everyway; I found it more comfortable, controls are tighter and it doesn't need a circle pad pro, it's got quite a bit of kick to it, more than the 3DS, and the OLED really is very crisp and looks great (thanks for removing that Sony). The 3DS brings nothing new to the table really, it's just a juiced up DS with a pseudo-3D feature I don't like even when done
properly. Not to say the Vita isn't anything but a jumped-up PSP of course, but I feel it improved on the PSP a lot more than the 3DS did to the DS.
Software-wise, the 3DS hands down. It definitely has both quality and quantity when it comes to games over it's competitor. The Vita does have a nice collection of titles but I'm sorry, you don't have Pokemon. The game that will forever be Ninendo's trump card when it comes to handhelds and will 100% make me get it.
The 3DS library's gone from strength to strength, and looks like it's going to just keep getting better!
So overall, the 3DS, simply because it's got the games. I like the Vita more but I'm here for the games! I wanted to like the Vita and I do, but I feel it doesn't fill any void. Neither does the 3DS really, but it's got the games to back it up and to soldier on despite this. I loved the PSP but gotta go with the 3DS on this one. After the fail start, Ninty's really done a good job with it.