Real-time surgical workflow recognition is a prerequisite for autonomous robot-assisted surgery. Existing approaches to this problem mainly focus on the output accuracy without considering the spurious phase changes issue, which is crucial for online applications in operation room. In this work, we employ a multi-module model to incorporate both spatial and temporal features for accurate inferences via a transformer-like architecture. Especially, we introduce a feedback loop at the output stage to enforce phase consistency. Through thorough experiments with Cholec80 dataset, our model is shown to be comparative with state of the arts in terms of accuracy, while consistently outperforming them in the output continuity aspect. In addition, we validate the benefits of using ResNeSt over the conventional ResNet for feature extraction in this field of research. However, we acknowledge the trade-off between inference accuracy and phase change amount, which needs further study.