Accurate segmentation of whole brain MR images including the cortex, white matter and subcortical structures is challenging due to inter-subject variability and the complex geometry of brain anatomy. However a precise solution would enable accurate, objective measurement of structure volumes for disease quantification. Our contribution is three-fold. First we construct an adaptive statistical atlas that combines structure specific relaxation and spatially varying adaptivity. Second we integrate an isotropic pairwise class-specific MRF model of label connectivity. Together these permit precise control over adaptivity, allowing many structures to be segmented simultaneously with superior accuracy. Third, we develop a framework combining the improved adaptive statistical atlas with a multi-atlas method which achieves simultaneous accurate segmentation of the cortex, ventricles, and sub-cortical structures in severely diseased brains, a feat not attained in . We test the proposed method on 46 brains including 28 diseased brain with Alzheimer's and 18 healthy brains. Our proposed method yields higher accuracy than state-of-the-art approaches on both healthy and diseased brains.