Functional networks often guide our interpretation of spatial maps of brain-phenotype associations. However, methods for assessing enrichment of associations within networks of interest have varied in terms of both scientific rigor and underlying assumptions. While some approaches have relied on subjective interpretations, others have made unrealistic assumptions about the spatial structure of imaging data, leading to inflated false positive rates. We seek to address this gap in existing methodology by borrowing insight from a method widely used in genomics research for testing enrichment of associations between a set of genes and a phenotype of interest. We propose Network Enrichment Significance Testing (NEST), a flexible framework for testing the specificity of brain-phenotype associations to functional networks or other sub-regions of the brain. We apply NEST to study phenotype associations with structural and functional brain imaging data from a large-scale neurodevelopmental cohort study.