Numerical investigation was made on NASA Rotor 37 and the result was compared with an experiment. Working conditions from blocking point to stall point were calculated to study the rules of boundary layer separation and the instability mechanism of transonic axial-flow compressor. The research discovers a radial vortex behind passage shock wave in suction side boundary layer

which facilitates boundary layer separation and converges part of airflow near suction side to blade tip. With working conditions go to stall point

blade tip passage is jammed by two pieces of blocking zone; one located in the front of blade pressure side is caused by the interference of clearance leakage vortex and passage shock

while the other located on suction side after 50% chord is caused by the interference of radial vortex and leakage flow. The two blocking zones' expansion arouses instability of the compressor.