This paper presents a composite controller that combines nonlinear disturbance observer and second order sliding mode controller for attitude tracking of flexible spacecraft. First, a new nonsingular sliding surface is introduced. Then, a second order sliding mode attitude controller is designed to achieve high-precision tracking performance. An extended state observer is also developed to estimate the total disturbance torque consisting of environmental disturbances, system uncertainties and flexible vibrations. The estimated result is used as feed-forward compensation. Although unknown bounded disturbances, inertia uncertainties and the coupling effect of flexible modes are taken into account, the resulting control method offers robustness and finite time convergence of attitude maneuver errors. Finite-time stability for the closed-loop system is rigorously proved using the Lyapunov stability theory. Simulation results are presented to demonstrate the effectiveness and robustness of the proposed control scheme.