We study the position of compact operators in the space of all continuous linear operators and its subspaces in terms of ideals. One of our main results states that for Banach spaces $X$ and $Y$ the subspace of all compact operators $\mathcal K(X,Y)$ is an $M(r_1 r_2, s_1 s_2)$-ideal in the space of all continuous linear operators $\mathcal L(X,Y)$ whenever $\mathcal K(X,X)$ and $\mathcal K(Y,Y)$ are $M(r_1,s_1)$- and $M(r_2,s_2)$-ideals in $\mathcal L(X,X)$ and $\mathcal L(Y,Y)$, respectively, with $r_1+s_1/2>1$ and $r_2+s_2/2>1$. We also prove that the $M(r,s)$-ideal $\mathcal K(X,Y)$ in $\mathcal L(X,Y)$ is separably determined. Among others, our results complete and improve some well-known results on $M$-ideals.