Stochastic computing (SC) has attractive characteristics, compared with deterministic (or general binary) computing, such as smaller area of the implemented circuits, higher fault tolerance and so on. This study focuses on the transient fault tolerance of SC circuits with linear finite state machines (linear FSMs). To improve the transient fault tolerability of linear-FSM-based SC circuits, we propose a scheme for encoding the states of the FSM with stochastic numbers (SNs). Moreover, we discuss approximating state transition of the FSM so as to reduce the area overhead. The proposed SC circuits are modeled as Markov processes to clarify their behaviors when any transient fault occurs. Experimental results clarify the improvement in the fault tolerability of the SC circuits based on the proposed state encoding with SNs.