First of all, I'd like to start with that I loved the DS and PSP and couldn't pick a favourite between those 2. Brilliant little machines.
Hardware-wise, the Vita, easily. I much prefer it to the 3DS in almost everyway; I found it more comfortable, controls are tighter and it doesn't need a circle pad pro, it's got quite a bit of kick to it, more than the 3DS, and the OLED really is very crisp and looks great (thanks for removing that Sony). The 3DS brings nothing new to the table really, it's just a juiced up DS with a pseudo-3D feature I don't like even when done properly. Not to say the Vita isn't anything but a jumped-up PSP of course, but I feel it improved on the PSP a lot more than the 3DS did to the DS.
Software-wise, the 3DS hands down. It definitely has both quality and quantity when it comes to games over it's competitor. The Vita does have a nice collection of titles but I'm sorry, you don't have Pokemon. The game that will forever be Ninendo's trump card when it comes to handhelds and will 100% make me get it. The 3DS library's gone from strength to strength, and looks like it's going to just keep getting better!
So overall, the 3DS, simply because it's got the games. I like the Vita more but I'm here for the games! I wanted to like the Vita and I do, but I feel it doesn't fill any void. Neither does the 3DS really, but it's got the games to back it up and to soldier on despite this. I loved the PSP but gotta go with the 3DS on this one. After the fail start, Ninty's really done a good job with it.
The Vita only had two interesting games, now one with the other coming to consoles with an HD makeover. Looking at you Assassin's Creed: Liberation. So, all we have left is Uncharted... and a Vita is not worth it for one game. The 3DS on the other hand, well, has lot's of great titles with more to come. Can't say the same about the Vita. It may have great hardware, but it's a damn waste of it. Yeah, this went to 3DS so fast that Sony couldn't even announce one more good game in time to save it. Vita my ***, this thing is as good as dead.
I recently got a Vita after its recent price drop, and have owned a 3DS for a while. The two are totally different types of systems and can't be compared.
I didn't think much of the Vita until I actually got one, and was pleasantly surprised. It's a great piece of hardware, a lot more than what I was expecting. I've been using it for movies and TV shows as much as I have for gaming.
For games, the 3DS library obviously is amazing. The only game I physically have on Vita is Persona 4 Golden which alone is worth buying the system for imo. The rest of the games I want to play are all free with PlayStation Plus which is cool. (Uncharted Golden Abyss, Gravity Rush, Rayman Origins). The Vita has more home console type games while the 3DS is more like classic handheld systems.
Once the PS4 comes out, technically every PS4 game can be played on the Vita so that will solve the lack of games problem. I'm hoping the Vita will be more successful once the PS4 and the new PS Vita TV releases.
Hands down the 3DS. I know the PS Vita is more powerful and all, but I don't give a rat's *** about graphical power, I buy the systems for the games. It's the same reason I never got a PS3 or will never get a PS4, and it's the reason why I (almost) never bring out my 360 (I DO need it for 4-player Halo and CoD, and I WOULD like to pick up Bioshock Infinite at some point, but beyond that I no use for it.)
It's also why I haven't even considered getting a 3DS until recently. The system started with a very weak lineup, the only possible game I could want on it already being in my possession in its original form. Then some other games came out which look interesting and I might try out at some point, but for handheld Nintendo systems I don't consider them to really start until the Pokemon games come out. Then I automatically buy it and the system.
Definitely the 3DS. Yes, the Vita is powerful, and it has its perks if you like to watch movies and some of the other apps on it. But honestly I've probably spent about 35 hours on it in total at most. Gotta agree with what Turo said, the only 2 exclusive interesting games for it are Uncharted: Golden Abyss, and Assassin's Creed: Liberation, with Assassin's Creed getting a console port, and I wouldn't be surprised if Uncharted gets one for the PS3/PS4 soon enough as most other Sony handheld games have been ported to PS3.
3DS gives gamers what they want, games. And I have spent hundreds of hours combined on them just from the ones that have released this year, like Fire Emblem: Awakening, Animal Crossing: New Leaf, and Mario & Luigi: Dream Team just to name a few. On top of that, I still have games to look forward to with the 3DS, while with the Vita I can't really think of anything that I have looked forward to for about a year.
The Vita hands down, Hardware is what matters, plus it has several Ratchet and Clank games available for it, it has a OLED screen, and a ton of other features, also to anyone that says the Vita ripped off Nintendo, really?, if that is the case, then the DS ripped off a tablet computer.
Also, the Vita has been hacked multiple times, so if I get a older model I can use a emulator, and thus just opened up my game library to something that Nintendo doesn't even offer.
So, long story short, till the 3DS can offer something other than a mostly useless 3D screen, the Vita has the upper hand.
Which is why Wii U is doing so great, why GameCube did so great, and even why the Virtual Boy did so great! xD
Yes, that was a bit caustic, but the idea of hardware is irrelevant bites at me more than anything else. Hardware is what makes or breaks a console. It even makes and breaks a computer - literally - so that's saying something. But, hardware isn't the only thing. SUPPORT for the console matters a ton. And pricing, too.
The VB failed because it tried to be advanced with its hardware and the games sucked. And Gamecube? If you wanna list Wii U fine (even though it's a bit early), but the GameCube was ahead of its time in terms of hardware.