A framework for detecting nonlinear oscillatory activity in broadband time series is presented. First, a narrow-band oscillatory mode is extracted from a broadband background. Second, it is tested whether the extracted mode is significantly different from linearly filtered noise, modeled as a linear stochastic process possibly passed through a static nonlinear transformation. If a nonlinear oscillatory mode is positively detected, it can be further analyzed using nonlinear approaches such as phase synchronization analysis. For linear processes standard approaches, such as the coherence analysis, are more appropriate. The method is illustrated in a numerical example and applied to analyze experimentally obtained human electroencephalogram time series from a sleeping subject.