Phenotypic diversity and fidelity can be balanced by controlling stochastic molecular mechanisms. Epigenetic silencing is one that has a critical role in stress response. Here we show that in yeast, incomplete silencing increases stochastic noise in gene expression, probably owing to unstable chromatin structure. Telomere position effect is suggested as one mechanism. Expression diversity in a population achieved in this way may render a subset of cells to readily respond to various acute stresses. By contrast, strong silencing tends to suppress noisy expression of genes, in particular those involved in life cycle control. In this regime, chromatin may act as a noise filter for precisely regulated responses to environmental signals that induce huge phenotypic changes such as a cell fate transition. These results propose modulation of chromatin stability as an important determinant of environmental adaptation and cellular differentiation.