In this paper, optimum location of power system stabilizers in order to damp inter-area oscillations has been investigated by using modal series method. Nonlinear effects grow in dynamical behaviors of heavily loaded stressed power systems significantly. Hence linear based methods and techniques no longer present clear and exact response of the systems. In this article by defining new nonlinear participation factors and a nonlinear interaction index in modal series frame, location of PSS is determined and its effectiveness has been analyzed. For two various operating conditions, low and high stress cases proper site of the controller is determined by using both the conventional methods such as mode shapes, participation factors and residues in one side and the proposed nonlinear participation factors in modal series frame in other side. The obtained results are compared with each other and with nonlinear time domain simulations. Studies carried out on IEEE 50-generator test case system and selected machines to locate PSS by modal series method are validated by time domain simulations. Also nonlinear interaction index shows increasing of nonlinear interaction between fundamental modes of the system when PSS placement has been selected properly and its design has been done well. This shows a high influence of PSS on damping inter-area oscillations and improving system oscillatory stability. Nonlinear time domain simulation shows that improper selection of PSS location may result in poor performance of system and deterioration of oscillatory response of the system.