A new type of motion saliency—that is, temporal context-aware motion saliency (TCAMS)—was proposed to detect the saliency of motion using its temporal context, presenting the semantic information of an event in a highly informative manner. Our definition differs from the typical definition of motion saliency. According to our definition of TCAMS, a novel detection method is proposed based on the interaction between the human visual and working memory systems, transferring visual intelligence to machine intelligence. Experimental evaluations demonstrated that the proposed TCAMS detection method exhibited accurate motion-saliency detection and good temporal context description performance. The benefits of the proposed TCAMS detection method were also comprehensively demonstrated using a wide range of applications, from low-level to high-level computer vision tasks—such as object detection, video synopsis, and action recognition—where the temporal context of motion was as essential as the motion itself. We showed that TCAMS could present an accurate motion state in a clear and informative manner.