Attempts to correlate measures of intellectual ability with localized anatomical imaging features of the brain have yielded variable findings distributed across frontal, parietal, and temporal lobes. To better define the gray and white matter correlates of intellectual ability and the effects of sex and age, we analyzed the brains of 105 healthy individuals, ages 7-57 years, who had a Full Scale Intelligence Quotient (FSIQ) of 70 or higher. We examined associations of FSIQ with cortical thickness and with white matter volume throughout the cerebrum. Thinning of left ventromedial and right dorsolateral prefrontal cortices correlated significantly with FSIQ. Sex modified correlations of cortical thickness with FSIQ in the left inferior frontal, left cingulate, and right dorsomedial prefrontal cortices. Correlations of local white matter volumes with FSIQ varied by age, with adults showing inverse correlations of white matter volume with FSIQ in a large territory of right frontal white matter likely corresponding to fiber tracts of the superior corona radiata and superior longitudinal fasciculus. These findings corroborate the role of frontal and parietal association cortices and long association white matter fibers in higher intelligence and suggest ways in which the neuroanatomical correlates of higher intelligence may vary by sex and age.