The main purpose of this paper is to describe formalisms, algorithms, and certain unique features of a system for optimization of intensity modulated radiotherapy (IMRT). The system is coupled to a commercial treatment planning system with an accurate dose calculation engine based on the kernel superposition algorithm. The system was designed for use for research as well as for routine clinical practice. It employs dose- and dose-volume-based objective functions. The system can optimize IMRT plans with multiple target volumes simultaneously. Each target volume may be assigned a different prescription dose with constraints on either underdosing, or overdosing, or both. For organs at risk more than one constraint may be applied. This feature allows simultaneous treatment of primary, regional disease and electively treated nodes. The system allows specification of constraints on logical combinations of anatomic structures, such as a region of overlap between the prostate planning target volume and rectum or the volume of lung excluding the tumor. The optimization may also be performed on plans which, in addition to intensity-modulated beams, include other modalities such as non-IMRT photon and electron beams and brachytherapy sources. The various features of the system are illustrated with one phantom example and two clinical examples: a brain stereotactic radiosurgery case and a nasopharynx case. In the cylindrical phantom example, the use of the system for overlap regions is demonstrated. The brain stereotactic radiosurgery example shows the improvement of IMRT plans over the conventional arcs based plan and the three-dimensional conformal plan with multiple fixed gantry angles and demonstrates the application of our system to cases where small grid sizes are important. The nasopharynx example shows the potential of IMRT to simultaneously treat large and boost fields. It also illustrates the power of IMRT to protect normal anatomic structures for highly complex situations and the efficiency in planning and delivery achievable with IMRT. The overall IMRT planning time is typically less than 2 h on a Sun Ultrasparc workstation, most of which is spent in repeated computation of dose distributions.