In XNA, penalty of trying to do anything on your own (load data formats in particular) is much higher than living with imperfections of the ones supported right off the shelf, at least that's what the documentation looks like, and what i heard from someone who bended the content pipeline dearly to support what he needs (runtime geometry generation) and whom i trust to know what he is doing. So i guess .X and .FBX are pretty much gonna be it. They both support skeletal animation, and i think also vertex morping. I'm not exactly sure what features exactly Blender is able to export into them, but i can look into it if you like.
Blender 2.5 is going to finally have good Collada support (current one doesn't have armatures and is broken in multitude of ways), and Autodesk provides a good tool for Collada <-> FBX conversion.
If you look around hobby development community, you'll see that Id software data formats are quite popular. On the one hand the MD2 and MD3 vertex morphing character animation formats, MD5 skeletal animation format, whose needs are obviously covered by the above, and on the other hand the BSP file format, which describes game levels.
One thing that speaks for BSP is that a good editor is available (GTKRadiant and derivatives). This is an efficient format which can cull hidden geometry. However, the runtime cost of building custom geometry in XNA is significant, and with the limits of these formats are quite modest at 64k vertices and 64k polygons in the whole game level, which is something you can just dump onto the graphics card nowadays and no performance drop will be noticeable. So you don't gain any benefit from supporting it directly.
I doubt there are any powerful, modern, available formats and tools for culling-optimized level design, whose performance benefits would translate into XNA.
A couple of links you might want to check out, if you haven't already:
3D World Studio content pipeline
Quake port
All unmaintained and out of date, no idea how optimized or not.
