Road maintenance optimization through a discrete-time semi-Markov decision process

doi:10.1016/j.ress.2012.03.011

Reliability Engineering & System Safety

Volume 103, July 2012, Pages 110-119

https://doi.org/10.1016/j.ress.2012.03.011 Get rights and content

Abstract

Optimization models are necessary for efficient and cost-effective maintenance of a road network. In this regard, road deterioration is commonly modeled as a discrete-time Markov process such that an optimal maintenance policy can be obtained based on the Markov decision process, or as a renewal process such that an optimal maintenance policy can be obtained based on the renewal theory. However, the discrete-time Markov process cannot capture the real time at which the state transits while the renewal process considers only one state and one maintenance action. In this paper, road deterioration is modeled as a semi-Markov process in which the state transition has the Markov property and the holding time in each state is assumed to follow a discrete Weibull distribution. Based on this semi-Markov process, linear programming models are formulated for both infinite and finite planning horizons in order to derive optimal maintenance policies to minimize the life-cycle cost of a road network. A hypothetical road network is used to illustrate the application of the proposed optimization models. The results indicate that these linear programming models are practical for the maintenance of a road network having a large number of road segments and that they are convenient to incorporate various constraints on the decision process, for example, performance requirements and available budgets. Although the optimal maintenance policies obtained for the road network are randomized stationary policies, the extent of this randomness in decision making is limited. The maintenance actions are deterministic for most states and the randomness in selecting actions occurs only for a few states.

Introduction

It is important for a road agency to efficiently and cost-effectively maintain a road network under its jurisdiction at the required performance levels over the service life. In this regard, there is a need to develop a long-term maintenance optimization model to minimize the life-cycle costs of the road network based on the reasonable modeling of the life-cycle road deterioration. In this paper, we model the life-cycle deterioration of different road segments in the road network as a semi-Markov process and develop linear programming (LP) optimization models to minimize the life-cycle costs of the road network based on the semi-Markov decision process (SMDP). The deterioration model is proposed on the basis of the initial inputs of the deterioration process at the beginning of the planning horizon and will be continuously tested given the regular inspection data. If it is found from the real inspection data that the real deterioration process is quite different from the initial input deterioration process, the deterioration model needs to be updated and the optimization model needs to be recalculated according to the updated deterioration model.

In this paper, the road performance measured by road roughness is divided into discrete performance states. To model the road deterioration as a semi-Markov process, the holding time in each state is assumed to follow a discrete Weibull distribution [1], which is widely used in modeling the life-time of road and other infrastructure assets [2], [3], [4]. Four maintenance actions are standardized for all performance states. The transition probabilities are calculated in terms of the holding time in each state. LP optimization models are developed to minimize the expected average costs over an infinite planning horizon and a finite planning horizon, respectively. The reasons of applying LP optimization models are that they are practical for the maintenance of a road network having a large number of road segments and that they are convenient to incorporate various constraints (e.g. performance requirements and available budgets) on the decision process [5].

Without the loss of generality, we develop a hypothetical road network to illustrate the application of the proposed optimization models. The optimal maintenance policy obtained for the road network is a randomized stationary policy [6], i.e., it specifies a probability on selecting a maintenance action when a road segment stays in a particular state. However, the scope of randomness in decision making is usually limited, i.e., the maintenance actions are deterministic for most states and the randomness in selecting actions occurs only for a few states. Therefore, the optimal maintenance policy obtained is practical in the maintenance operation of a road network.

Section snippets

Literature review

Road maintenance optimization is a planning problem based on Operational Research methods [7] and the prediction of road life-cycle deterioration can be modeled by stochastic processes [8]. Commonly, the deterioration of a road segment can be modeled as a discrete-time Markov process such that an optimal maintenance policy can be obtained based on the Markov decision process [9], [10], [11]. A Markov process is a stochastic process in which the state at a future time point only depends on its

Roughness as a road performance measure

The performance of a road segment can be measured by structural condition and service condition. Roughness is a measure of road surface distortion that reflects the ability of the road to provide a comfortable ride to users [24]. Roughness is primarily associated with serviceability; however, it is also related to structural deficiencies and accelerated road deterioration. Roughness has a significant effect on vehicle operating costs, safety, comfort, and speed of travel, and therefore it is a

Preliminary of a discrete-time semi-Markov process

For a finite state space $S = {S_{1}, S_{2}, .. ., S_{m}}$ , a finite action set $A = {a_{1}, a_{2}, .. ., a_{l}}$ , and discrete time points 0, 1, …, t, (t+1), …, a one-step transition probability in a discrete-time Markov process is expressed as follows: $P^{(1)} (j | i, a) = \Pr (S_{t + 1} = j | S_{t} = i, A_{t} = a) \forall i, j \in S, a \in A, t = 0, 1, \dots$ where $P^{(1)} (j | i, a)$ =the probability that the next state is in j at time point (t+1) given that the current state is in i and that action a is taken at time point t. The one-step transition probability is independent of the current

Planning horizon, decision epoch and decision period

The planning horizon in the maintenance of a road network is the whole length of time for which an optimal maintenance policy is derived and implemented. The decision epoch is a time point at which a decision is made on which maintenance action to take. The decision period is the time interval between two successive decision epochs.

State space

As discussed previously, the performance of a road segment is measured by the RQI, which is derived from the IRI. Here, S, the state space of the performance of a

Parameters of discrete Weibull distribution

The parameters of the discrete Weibull distribution for road deterioration can be determined by the maximum-likelihood estimation method if there are sufficient observation data of the lifetimes of road segments in each state. If historical observation data are insufficient, expert opinions can be solicited to derive these parameters. A group of experts can be invited to answer questions pertaining to their beliefs about the likelihood of a road segment remaining in a given state for a certain

Objective function

The infinite-time LP model seeks an optimal randomized stationary policy that minimizes the expected average maintenance cost of the road network over an infinite time horizon: $Min \frac{L_{R} \sum_{i \in S} \sum_{a \in A} c (i, a) π (i, a)}{\sum_{i \in S} \sum_{a \in A} E [τ (i, a)] π (i, a)}$ where c(i, a)=the average unit maintenance cost associated with state–action pair (i, a) for a road segment; π(i, a)=the steady-state road segment distribution of state–action pair (i, a), $\sum_{i \in S} \sum_{a \in A} π (i, a) = 1$ ; $E [τ (i, a)]$ =the expected holding time of state–action pair (i, a); L_R

Optimal maintenance policy

By solving the infinite-time and finite-time LP models, λ(i, a), the steady-state intensity of each state–action pair in an infinite planning horizon, and π_n(i, a), the road segment distribution at time point t_n for each state–action pair in a finite planning horizon, can be obtained. In some cases, for a state i, λ(i, a) or π_n(i, a) may be nonzero for two or more maintenance actions. This means that the decision maker needs to randomly choose maintenance actions for a fraction of the road

Illustrative example

Real road performance data currently are not available to verify the optimization models proposed in this paper. In this section, a hypothetical road network is used to illustrate the application of the proposed optimization models. There are 100 road segments in the network. The length of each segment is 1 km. The total length of the road network is 100 km. The planning horizon of the finite-time LP model is 20 years.

Limitation and methodology for future improvement

The main limitation of this study is that there are no real data available to test the proposed optimization models and to verify some assumptions associated with these models. In particular, the real distribution of the holding time in each state cannot be derived from statistical analysis methods (e.g., regression analysis [31]) without real data. The use of a discrete Weibull distribution in this paper may not precisely represent the real road deterioration process although it is widely used

Conclusions

The maintenance of a road network is a complicated issue that requires efficient and practical optimization models. In this paper, both infinite and finite time LP models have been developed based on the semi-Markov decision process. The infinite-time LP model seeks an optimal randomized stationary policy that minimizes the expected average maintenance cost of the road network over an infinite time horizon. The objective of the finite-time LP model is to minimize the average maintenance cost of

Acknowledgment

This study is sponsored by the Public Policy Research Grant HKUST6004-PPR-10 of the Hong Kong Research Grant Council.

References (34)

S. Martorell et al.
Age-dependent reliability model considering effects of maintenance and working conditions
Reliability Engineering and System Safety
(1999)
J.M. Van Noortwijk et al.
The use of lifetime distributions in bridge maintenance and replacement modeling
Computers and Structures
(2004)
J. Worm et al.
Model based decision support for planning of road maintenance
Reliability Engineering and System Safety
(1996)
J.M. Van Noortwijk
Explicit formulas for the variance of discounted life-cycle cost
Reliability Engineering and System Safety
(2003)
J.M. Van Noortwijk et al.
Optimal inspection decisions for the block mats of the Eastern-Scheldt barrier
Reliability Engineering and System Safety
(1999)
L. Dieulle et al.
Sequential condition-based maintenance scheduling for a deteriorating system
European Journal of Operational Research
(2003)
A. Grall et al.
Asymptotic failure rate of a continuously monitored system
Reliability Engineering and System Safety
(2006)
C.T. Barker et al.
Optimal non-periodic inspection for a multivariate degradation model
Reliability Engineering and System Safety
(2009)
D. Chen et al.
Optimization for condition-based maintenance with semi-Markov decision process
Reliability Engineering and System Safety
(2005)
C.E. Love et al.
A discrete semi-Markov decision model to determine the optimal repair replacement policy under general repairs
European Journal of Operational Research
(2000)

Y. Li et al.

A steady-state solution for the optimal pavement resurfacing problem

Transportation Research Part A

(2002)

T. Nakagawa et al.

The discrete Weibull distribution

IEEE Transactions on Reliability

(1975)

Y. Kleiner

Scheduling inspection and renewal of large infrastructure assets

Journal of Infrastructure System

(2001)

X. Zhang et al.

Optimal performance-based building facility management

Computer-Aided Civil and Infrastructure Engineering

(2010)

M.L. Puterman

Markov decision processes: discrete stochastic dynamic programming

(2005)

N.D. Singpurwalla

Survival in dynamic environments

Statistical Science

(1995)

K. Golabi et al.

A statewide pavement management system

Interfaces

(1982)

Cited by (47)

Optimal intervention policy of emergency storage batteries for expressway transportation systems considering deterioration risk during lead time of replacement
2024, Reliability Engineering and System Safety
An emergency storage battery (ESB) is an important asset to ensure the reliability and safety of complex technological systems. Even if an asset owner decides to replace a deteriorated ESB in expressway transportation systems , a lead time (LT) from ordering replacement parts to delivery is inevitable because the replacement parts cannot be stockpiled. During the LT, the ESB continues to deteriorate, i.e., the risk of deterioration continuously increases. If the LT has a non-negligible length, it is necessary to find intervention policies that appropriately consider the risk during the LT. In this study, we propose a methodology to derive optimal intervention policies by explicitly considering deterioration risk during the LT in a stochastic control framework. Specifically, the geometric Brownian motion is statistically estimated as the deterioration process of the ESB, and an optimization model is proposed to find the optimal intervention policy. The proposed methodology is applied to a case study on an expressway transportation system. In the case study, a sensitivity analysis is also performed to obtain practical implications through a life cycle cost (LCC) comparison. Our findings show that the proposed methodology can reduce the LCC by 5.49% compared with when the LT is not considered.
Sustainability-informed management optimization of asphalt pavement considering risk evaluated by multiple performance indicators using deep neural networks
2023, Reliability Engineering and System Safety
Sustainability considerations throughout the entire pavement life-cycle in the decision-making process under uncertainty are needed to achieve optimal pavement management from the perspective of the economy, environment, and society. A novel sustainability-informed management optimization of asphalt pavement is presented in this study. First, a deep neural network (DNN) model is trained using the Long-Term Pavement Performance (LTPP) database to learn the nonlinear and complex relationships among multiple performance indicators of asphalt pavement (i.e., the international roughness index (IRI), rut depth, and alligator and transverse cracking) and their associated parameters (i.e., the climate, traffic, and pavement structure and properties). Based on the multiple time-dependent limit-state functions incorporating the uncertainties associated with these parameters, the DNN model prediction, and the IRI measurement, Monte Carlo simulation is conducted to estimate the system failure probability of asphalt pavement. Finally, a genetic algorithm-based tri-objective optimization is utilized to find the optimal maintenance and rehabilitation actions that reduce the extent of detrimental economic, environmental, and social consequences during the pavement's life-cycle. The capabilities of the proposed approach are illustrated using LTPP asphalt pavement sections in Pennsylvania and Florida, USA.
Joint maintenance planning of deteriorating co-located road and water infrastructures with interdependencies
2022, Reliability Engineering and System Safety
Due to co-location and spatial proximity, deteriorating water infrastructure (WI) and transportation infrastructure (TI) have complex interdependencies, including both physical dependency from road to pipe, and operational dependency from pipe to road. On one hand, traffic load is not only a dominant factor for the physical degradation of road but may also be an important contributing factor to the sudden failure of the pipe underneath the co-located road due to the extreme propagated stress. On the other hand, the water pipe breaks and repairs will also cause traffic blockage, lane closure, and repaving of road surface, which influence both road user costs and road maintenance costs. Most of the existing maintenance work mainly considered WI and TI separately but neglected their complex interdependencies. This study proposes a joint proactive maintenance planning framework for the co-located road and pipe under their degradation uncertainties by explicitly taking into account both their physical and operational dependencies. In particular, the competing failure modes of both the traffic-induced sudden failure and the corrosion-induced gradual failure are simultaneously considered for the pipe to capture the physical dependency. Road maintenance costs and road user costs in the presence of pipe maintenance disruption are also formulated for the road to capture the operational dependency. A case study is further provided to comprehensively evaluate the performance of the proposed joint maintenance planning strategy over several existing benchmark strategies and demonstrate the cost-saving benefit of the proposed work. The impacts of interdependencies on the proposed maintenance planning decisions are also thoroughly investigated.
A review on empirical methods of pavement performance modeling
2022, Construction and Building Materials
This paper conducts a comprehensive review of empirical methods of pavement performance modeling. The paper firstly analyzes performance measures used in existing pavement performance modeling. Then, based on whether pavement performance models can be automatically updated with newly observed data, this paper classifies the empirical methods into traditional modeling methods and adaptive modeling methods and discusses their main features, strengths, and weaknesses. It is found that, in the traditional modeling methods, comprehensive modeling methods typically can well incorporate both temporal and spatial characteristics of pavement performance but the interpretation capability of performance models based on machine learning techniques needs to be enhanced; the adaptive modeling methods describe the deterioration behavior of pavement performance in a more accurate way but need to improve the setting of updating conditions and the evaluation of updating effects. Finally, recommendations are made for future studies.
Multi-year planning for optimal navigation channel dredging and dredged material management
2022, Transportation Research Part E: Logistics and Transportation Review
Citation Excerpt :
Planning of maritime navigation channel dredging and roadway maintenance on network level have similar features and considerations, including infrastructure condition deterioration, maintenance cost subject to a budget, economic value of each road segment, asset bundling for easier management, etc. Plenty of research has proposed advanced methodologies for roadway maintenance planning problems, such as mathematical programming (e.g., mixed integer programming), dynamic programming, Markov chain, and heuristic algorithms (e.g., prioritization and meta-heuristic algorithms) (Golabi et al., 1982; Morcous and Lounis, 2005; Zhang and Gao, 2012; Gao and Zhang, 2013; Binhomaid and Hegazy, 2014; Ma et al., 2018). However, these methodologies cannot be directly applied in maritime channel maintenance planning problem that is studied in this paper.
Navigation channels are critical maritime infrastructure that supports economic and recreational activities and intermodal freight supply chains, impacting broad areas of the hinterland. Maintenance of the maritime infrastructure is vital to the operations of the transportation system. This paper studies a strategic-level maritime infrastructure asset management problem: multi-year planning of navigation channel maintenance dredging and dredged material management. A mathematical model called dredging planning optimization model (DPOM) is developed to optimize channel dredging planning and dredged material management in confined disposal facilities (CDFs), accounting for practical constraints and considerations, including channel linkage and dependency, channel bundling, CDF accessibility and capacity, shoaling and navigability deterioration over time, reimbursable costs, channel economic values, etc. The problem is formulated as a mixed integer nonlinear programming (MINLP) model, with the objective of maximizing the total economic-value-weighted average navigability under fixed budget. The model can be reformulated as an equivalent mixed integer programming (MIP) model, which can be solved by the CPLEX solver, using the branch and bound algorithm. A heuristic algorithm called the dynamic planning prioritization (DPP) algorithm, is proposed in order to solve large-scale problems due to the computational complexity. DPP incorporates a dynamic ranking criterion to overcome the challenge of simultaneously handling all of the practical constraints and considerations, as well as the impact of channel prioritization on future year decisions. Finally, a real-world case study is proposed to illustrate the model results and demonstrate the effectiveness of the MIP model and the DPP algorithm.
Asphalt pavement maintenance plans intelligent decision model based on reinforcement learning algorithm
2021, Construction and Building Materials
Making a proper maintenance plan for the pavement health is the key to maintain a good service level and bearing capacity. To cope with the increasing demand for pavement maintenance, based on the idea of reinforcement learning, an intelligent decision-making model for pavement maintenance plans based on proximal policy optimization algorithm was proposed in this paper. The decision model fully considers the comprehensive maintenance benefit-cost ratio during the whole life cycle of the road. To overcome the problems of the experience-led manual decision-making, it conducted the decision-making between pavement conditions and maintenance plans based on data mining technique. Besides, a method for constructing a reinforcement learning Environment module based on a deep artificial neural network was proposed, and a reward function is designed for road maintenance decisions. The model was applied to the highway maintenance decision in Jiangsu Province, and verified that the decision overall accuracy of the reinforcement learning model was 82.2%, which was an increase of 17.2% compared with the artificial neural network model.

View all citing articles on Scopus

View full text

Road maintenance optimization through a discrete-time semi-Markov decision process

Abstract

Introduction

Section snippets

Literature review

Roughness as a road performance measure

Preliminary of a discrete-time semi-Markov process

Planning horizon, decision epoch and decision period

State space

Parameters of discrete Weibull distribution

Objective function

Optimal maintenance policy

Illustrative example

Limitation and methodology for future improvement

Conclusions

Acknowledgment

Reliability Engineering and System Safety

Computers and Structures

Reliability Engineering and System Safety

Reliability Engineering and System Safety

Reliability Engineering and System Safety

European Journal of Operational Research

Reliability Engineering and System Safety

Reliability Engineering and System Safety

Reliability Engineering and System Safety

European Journal of Operational Research

Transportation Research Part A

The discrete Weibull distribution

IEEE Transactions on Reliability

Scheduling inspection and renewal of large infrastructure assets

Journal of Infrastructure System

Optimal performance-based building facility management

Computer-Aided Civil and Infrastructure Engineering

Markov decision processes: discrete stochastic dynamic programming

Survival in dynamic environments

Statistical Science

A statewide pavement management system

Interfaces