Probably a little of both. I've never towed 5th wheel or gooseneck, so I don't know. Intuitively, I'd guess having the tongue weight directly over the axle vs 5 feet (or more) away make a big difference. That much of a moment arm I'd guess can create a bit of a tail wagging the dog situation. Its not always a problem, but when it is, you want to know what to do and not crap your pants. Wind is MUCH more of a problem in my motorhome for some reason.
Enclosed bumper hitch trailers have always felt more wind affected to me, especially if they aren't loaded right. The other thing with them is being passed or passing a semi or large RV they can move around quite a bit.
The large goosenecks I've pulled were always livestock trailers or flatbeds hauling a tractor and such so a little different but much more stable.
I prefer the ease of backing up a bumper hitch but the gooseneck is more maneuverable.
If you won't have much weight in the trailer I'd think you'd want it located closer to the axles but in front of the axles.