Much of this goes back to Archimedes and his principle, which states, "Any object, wholly or partially immersed in a fluid, is buoyed up by a force equal to the weight of the fluid displaced by the object"
Buoyancy results from the weight of water that is displaced.
From a swimmer's perspective (reflective of one's intrinsic buoyancy), there may be differences in buoyancy characteristics between gender, at least suggested in the data. Much of this does have to do with adiposity deposition location. Some data have women tend to have a center of buoyancy in the horizontal position that is more toward the hips. These considerations don't necessarily consider lung volume - which is higher in men compared to women.
So, when thinking about gender differences in diving and achieving horizontal trim, distribution of ballast weight must then be considered. I would also argue that BCD type (back inflate versus jacket inflate, even BP use) can influence this (but isn't necessarily influenced by gender). In many cases, ballast weight is often concentrated on the the weight belt and no where else, so that coveted horizontal position can't be achieved! This is the case where one size does not fit all! But, in resort diving situations, many divers do not necessarily take the time and effort to figure out how to trim out their kits to achieve horizontal buoyancy.
I'm still thinking this through - but figured I'd put in my 2psi out there.