Security assessment (SA) system is crucial to ensure the production safety of a smart factory with rapid development of artificial intelligence. In this article, we propose a novel SA framework. Different from the conventional static monitoring systems based on traditional sensing technologies, the proposed framework can automatically detect objects via visual sensing. We use a skeleton-based graph convolutional network to generate action vocabulary for the intermediate representations of action-to-action cooccurrence relations. These representations are encoded into the sequential interaction models to form the interaction representations. Integrating the states of molten steel levels as the reference labels, the sequential representations are fed into a recurrent neural network model with multilayer gated recurrent units (GRUs) to capture the key interactions leading to the accidents, in which an attention mechanism is used to reweight the actions and eliminate the invalid interactions. The predicted labels and the hidden states of the scenes are passing among multilayer GRUs. Finally, we optimize the global output to dynamically assess the security by calculating a joint objective function with a regularized cross-entropy loss. On the self-collected dataset from our partner Iron and Steel company and on-line video clips, the proposed framework performs better than the existing SA schemes.