Leaflet Free Edge Detection for the Automatic Analysis of Prosthetic Heart Valve Opening and Closing Motion Patterns from High Speed Video Recordings

Abstract

Prosthetic heart valves (PHVs) are routinely used in clinical settings to replace defective native heart valves in patients suffering from valvular heart disease. Although PHV designs must be rigorously tested using cardiovascular testing equipment to ensure their optimal characteristics and safe operation, visual data obtained during simulations are typically assessed manually, a tedious and error-prone task. The valve orifice area over time, which informs on the opening and closing motion patterns, constitutes a key quality metric for PHV assessment. In addition to the very fast motion of the valve’s leaflets, a major issue lies in the orifice being partly occluded by the leaflets’ inner side or inaccurately depicted due to its transparency, which is not addressed in the literature. In this paper, we propose a novel orifice segmentation approach for automatic PHV quantitative performance analysis, based on the detection of the leaflet free edges to accurately extract the actual orifice area. Utilizing video frames recorded with a high speed digital camera during in vitro simulations, an initial estimation of the orifice area is first obtained via active contouring and then refined to capture the leaflet free edges via a curve extension scheme based on brightness and smoothness criteria. Evaluation on three different PHVs demonstrated the effectiveness of our approach to detect valve leaflet free edges and extract the actual orifice area, significantly outperforming a baseline algorithm both in terms of valve design evaluation metrics and computer vision evaluation metrics.

1 Introduction

1.1 Context

More than 5 million people are diagnosed with valvular heart disease each year in the United States alone [1]. Prosthetic heart valves (PHVs) are routinely used in clinical settings to replace defective native heart valves in patients. Compared to mechanical valves, bioprosthetic valves typically behave more similarly to native valves, but have shorter life cycles and are prone to calcification, an issue that can be accelerated by the valve’s design and consequent increased stress on the leaflets [2]. Researchers are constantly working on new designs for more reliable and durable biological PHVs. PHV designs must be rigorously tested using cardiovascular testing equipment, to assess their performance and ensure their optimal characteristics. In addition to flow and pressure measurements, it is also common to manually assess the performance of the PHVs from visual data (images/videos) that are collected during simulations. This manual assessment is a tedious and error-prone task, with potentially devastating consequences.

In this paper, we are interested in the automatic, accurate, and efficient quantitative performance analysis of biological PHVs from video footage acquired during in vitro testing, using computer vision methods. A quantitative analysis of videos also allows one to study additional quality parameters, such as the valves’ orifice area over time, their opening and closing motion patterns, and their leaflet kinematics. Figure 1 shows sample frames of the opening motion of a biological PHV, acquired during in vitro simulations. One difficulty, computer vision-wise, comes from the fast motion of the valve’s leaflets during the opening and closing phases, which results in blurry edges. However, the major issue that we address in this paper lies in the orifice area being partly occluded by the inner side of the leaflets or inaccurately depicted due to its transparency.

Fig. 1.

Example frames from a test video, showing the opening motion of a tricuspid prosthetic valve: valve closed (a), partly open (b), and completely open (c).

1.2 Related Works

High-speed digital cameras offer an interesting medium for the quantitative evaluation of PHVs. Although they require direct optical access, unlike imaging modalities that are typically used for in vivo assessments [3], they are capable of recording fast movement such as the valve’s leaflet opening and closing patterns with great detail. Earlier works in the literature utilizing high-speed cameras have typically relied on the addition of external mechanisms to help process the videos. Examples include particles added to the working fluid [4, 5], manually placed markers on the leaflets [5,6,7], and laser light points projection [8]. Advancements in the quality, availability, and affordability of high-speed digital cameras have allowed researchers to circumvent the need for these external mechanisms; recent works typically make use of computer vision and image processing methods, including digital kymography, thresholding, and deformable models, to focus exclusively on the valves’ original pixel intensities.

A few works [9,10,11] have reported using digital kymography to analyze valve opening and closing procedures. While enabling a fast analysis of PHV recordings, digital kymograms, which consist of image lines projected along a time axis, focus on a very local region of the valve only.

Thresholding is arguably the simplest image segmentation method. Three variations have been proposed in [12] to segment the orifice area of trileaflet PHVs and assess the leaflet fluttering behavior: user-defined thresholding, Otsu’s method, and finite mixture model-based thresholding. Thresholding typically fails due to the valve orifice not having a homogeneous gray-level representation.

Deformable models, such as active contours (snakes), have been utilized to address this issue of non-homogeneity of valve orifices. Wittenberg et al. [13] investigated valve movements via endoscopic high-speed recordings of native pig heart valves in an ex vivo (explanted heart) setting. They computed the orifice area over time using manually initialized snakes. Local constraints on three nodes, corresponding to the commissions between the valve leaflets, were also marked manually. Kondruweit et al. [11], in a similar setup, proposed a combination of digital kymograms calculated at different angles and snakes (based on [13]), to analyze the effective orifice area over time. Both kymogram lines and snakes were manually initialized. In an effort to automate the PHV assessment, Condurache et al. [14] proposed a method to segment the orifice area during in vitro mechanical simulations and compared automatic thresholding, similar to that of [12], with snakes. Their snake implementation included anchor-points-based attractors, automatically added to the frames by changing the intensity of strategic pixels, to better fit the leaflet boundaries. In a following paper [15], the authors tackled the automatic analysis of leaflet fluttering, but incorrectly assumed that the detected orifice boundaries were in line with leaflet borders.

1.3 Contributions

Of all the reviewed related works, [15] presented the most interesting computer vision-based approach for analyzing valve opening and closing motion patterns in an automated way. However, one of the difficulties that they failed to address was the fact that part of the orifice may be occluded by the valve’s leaflets, or inaccurately depicted due to visible inner valve regions, making their method incapable of fully detecting and tracking the actual free edges of the leaflets. Figure 2 illustrates this last issue, showing two troublesome cases where part of the inner side of the leaflets is visible. In this paper, we address these issues by developing an automated method for tracking free edges separately, and not simply extracting them from the boundaries of the segmented orifice area.

Our contributions are two-fold. From a theoretical viewpoint, we propose a 2D curve extension scheme based on brightness and smoothness criteria that allows us to recover the true shape of the curve when deformable models typically fail due to visible inner regions of the underlying object. From a practical viewpoint, we apply this curve extension scheme to the problem of detecting and tracking the free edges of PHV leaflets and propose a novel approach for the automatic analysis of prosthetic heart valve opening and closing motion patterns from high-speed video recordings.

The paper is structured as follows. Section 2 details our approach, Sect. 3 discusses its experimental evaluation, and Sect. 4 presents concluding remarks.

Fig. 2.

Example frames in which the orifice area is partially visually bordered by the inner side of the leaflets (blue arrows) instead of the free edges of the leaflets. (Color figure online)

Fig. 3.

Flow chart of the proposed method.

2 Proposed Method

Two main algorithms are proposed to extract information from the simulation videos that can be used for the quantitative assessment of PHV designs (Fig. 3):

• 1. segmentation of the orifice area, which utilizes active contours;

• 2. detection of the leaflet free edges, which allows us to refine the shape of the orifice and address cases for which the orifice is partially bordered by the inner side of the valve’s leaflets (Fig. 2).

The extracted information can be used to determine several quality parameters, such as how fast the valve opens and closes, how long the valve stays open and its maximum opening, which constitute crucial data for assessing PHV designs. Each algorithm is further detailed in the remainder of this section.

2.1 Orifice Area Segmentation

Figure 3 (top) presents an overview of the orifice area segmentation algorithm. First, pre-processing is carried out to determine the region of interest (ROI) inside the frames and obtain an initial approximation (mask) of the orifice area. The orifice area is then segmented either via active contouring, initialized using the mask, or via thresholding, depending on where the valve is in its opening and closing cycle (trigger). This first algorithm builds upon the method proposed in [15], but differs on several levels. Although the general idea of the pre-processing steps is similar, the way to select the final ROI and the construction of the mask differ. The trigger part was also absent from [15].

Fig. 4.

Pre-processing: original image (a), circular outline of the valve (ROI) (b), thinned opening area (c), centerlines (in green) and corresponding anchor points (in red, on circular outline) (d), and triangular initial mask $M_0$ for active contouring (e). (Color figure online)

Fig. 5.

Triggering active contouring: observation of the mean gray level value inside the maximal circle fitted inside the initial triangular mask, at different frames of the opening phase. The last circle on the right corresponds to Frame A.

Pre-processing. Knowing the actual radius of the imaged PHV, an ROI that corresponds to the inside of the valve’s circular outline is first determined via the circular Hough transform [16] applied on a Canny edge map of the frame (Fig. 4a–b). A frame showing a small valve opening (akin to a star shape) is preferred. Then, an initial mask $M_0$ for active contouring is determined from the anchor/commission points as follows. The opening area is segmented via thresholding, then dilated and thinned to remove any artifacts on the area borders (Fig. 4c). The linear Hough transform is then applied on the thinned segmented area to find the three centerlines, at approximately 120$^\circ$ from each other (Fig. 4d). $M_0$ (Fig. 4e) is derived as the triangle connecting three anchor points (red ‘x’ on the circular outline in Fig. 4d) corresponding to the intersection of the centerlines with the circular outline. The anchor points are found by extending the lines from their end point closest to the circular outline.

Trigger. The trigger part determines when to activate (and deactivate) active contouring, i.e. when the valve is open enough for the active contour to work properly. The maximal circle MaxC that can be fitted inside $M_0$ is used for observing the gray values of the orifice area. Figure 5 depicts how the orifice area is observed at different frames in the opening phase to determine the first frame to be segmented via active contouring (Frame A) as follows:

$$\begin{aligned} |\bar{u_0}(MaxC)_{Frame A} - \bar{u_0}(MaxC)_{Frame A-1} |> 0.1\bar{u_0}(MaxC)_{Frame A-1} \end{aligned}$$

(1)

where $u_0$ is the frame image and $\bar{u_0}(MaxC)_{Frame A}$ is the mean intensity of MaxC in Frame A. The orifice area in Frame A is used as a threshold value for deactivating active contouring at Frame B during the closing phase.

Active Contouring. An active contour is a parametric curve. The segmentation is carried out as a minimization of the energy of this curve, which is defined as a combination of internal and external energy terms. The external energy coming from the image data pushes the active contour to fit the data (here the boundaries of the visible orifice area), while the internal energy restricts its deformation to ensure its smoothness. We used the Chan-Vese model [17], which tries to segment the image based on intensities as opposed to edges, according to the following energy functional for curve C (2) and minimization problem (3):

$$\begin{aligned} \begin{aligned} F(c_1,c_2,C) = \mu L(C) + \nu A(in(C))&+ \lambda _1 \int _{in(C)} {|u_0(x,y) - c_1 |}^2 dxdy \\&\quad + \lambda _2 \int _{out(C)} {|u_0(x,y) - c_2 |}^2 dxdy \end{aligned} \end{aligned}$$

(2)

$$\begin{aligned} \begin{aligned} \inf _{c_1,c_2,C} F(c_1,c_2,C) \end{aligned} \end{aligned}$$

(3)

where $u_0$ is the frame image, $c_1$ and $c_2$ are respectively the averages inside (in) and outside (out) C, L and A are the length and area, and $\mu \ge$ 0, $\nu \ge$ 0, $\lambda _1$ > 0, and $\lambda _2$ > 0 are fixed parameters (typically $\lambda _1$ = $\lambda _2$ = 1, $\nu$ = 0, and $\mu$ is selected according to the dataset, here $\mu$ = 1). At Frame A, the active contour is initialized using the contour of $M_0$. The final contour in a given frame, evolved until equilibrium, is used as the initial approximation for the next frame. This procedure continues up to Frame B. The segmented orifice areas (OA) at each frame are obtained from the pixels inside the evolved curve C. They may be smaller than the actual orifice areas due to the positioning of the leaflets (Fig. 2), and will be enlarged via the leaflet free edge detection algorithm.

Thresholding. For the few frames before Frame A and after Frame B, global thresholding (typically 0.2 over a maximal value of 1) is used to obtain OA.

2.2 Leaflet Free Edge Detection

Figure 3 (bottom) presents an overview of the proposed leaflet free edge detection algorithm. The goal is to detect the leaflet free edges in every frame in the valve cycle, by extending the curve derived from the segmented orifice area OA (Sect. 2.1) that could have stopped short of the actual free edges, thus addressing the issue of the visible inner side of the leaflets bordering the orifice area. Starting from the border pixels of OA, the curve is expanded along normal lines using a combination of brightness and smoothness criteria, yielding the set of free edge pixels as output. Figure 6 illustrates the various steps.

Fig. 6.

Leaflet free edge detection: cropped region of initial frame (a), normal lines (blue) to each segmented orifice border pixels (red) and segmented orifice area (black) (b), expanded normal lines showing a jagged profile (white) when considering only the brightness criterion (c), and detected free edge pixels (red) when considering both brightness and smoothness criteria (d). Blue arrows in (d) point to key expansion locations; the actual free edges were successfully recovered. (Color figure online)

Extracting Border. The boundary pixels of OA, output by the first algorithm, are extracted to form the initial free edge border $FE = \{(x,y) \in u_0 \mid B_{OA}(x,y) = 1\}$, where B indicates the boundary. OAs are also superimposed on the original frames to change the corresponding intensities to 0 (Fig. 6b).

Computing Normal Lines. The next step is to compute a normal line to each border pixel. We find the orientation of the normal line at each border pixel $fe_i \in FE$ via the first derivative in the X and Y directions:

$$\begin{aligned} N_{orientation}(fe_i) = -dx/dy \end{aligned}$$

(4)

We then extend the normal lines locally from the border pixels inward and outward of OA, by 25 pixels (Fig. 6b, blue lines), thus obtaining two sets of pixels $NFE_{in}$ and $NFE_{out}$ for each border pixel $fe_i$. The length of the normal lines could also be determined dynamically as a percentage of the valve’s radius.

Brightness Profile. Free edge pixels tend to be the brightest locally. We thus compute the intensity profile along each normal line, find the maximal value in the profile, then expand the orifice by incorporating the normal line going from the border pixel to the location of the maximum (Fig. 6c):

$$\begin{aligned} \begin{aligned} EOA_{MaxB} = OA\ \cup ~&\{ nfe_1, ..., nfe_{max} \mid u_0(max) \\&=\max _{j \in NFE_{in}, k \in NFE_{out}}(u_0(j), u_0(k)) \} \end{aligned} \end{aligned}$$

(5)

where $nfe_1$ is the border pixel (first element of $NFE_{in}$ and $NFE_{out}$ for $fe_i$) and $nfe_{max}$ is the pixel with the maximal value in either $NFE_{in}$ or $NFE_{out}$. Morphological closing is then applied on $EOA_{MaxB}$ to fill any gap between the added normal lines. A first set of free edge pixels $FE_{MaxB} = \{(x,y) \in u_0 \mid B_{EOA_{MaxB}}(x,y) = 1\}$ is derived, where B indicates the boundary.

Smoothness. In some regions, especially close to the commission points, the pixel with the maximal brightness may not correspond to a free edge and cause a false detection (Fig. 6c, lower left region). The smoothness of the final contour obtained from EOA is thus also considered. A second set of free edge pixels $FE_{LocB}$ is found by selecting the local (instead of absolute) maximal brightness pixels in the normal line profiles that are closer to the segmented orifice area:

$$\begin{aligned} \begin{aligned} EOA_{LocB} = OA\ \cup ~&\{nfe_1, ..., nfe_{max} \mid u_0(max) = peak, \\&\qquad dist(max,1) = \min _{j \in Peaks}(dist(j,1)) \} \end{aligned} \end{aligned}$$

(6)

$$\begin{aligned} FE_{LocB} = \{(x,y) \in u_0 \mid B_{EOA_{LocB}}(x,y) = 1\} \end{aligned}$$

(7)

where Peaks represents the set of pixels along the normal lines $NFE_{in}$ and $NFE_{out}$ for which the intensity corresponds to a peak value.

Expanding Curve. For each original border pixel, the two sets $FE_{MaxB}$ and $FE_{LocB}$ are compared; the corresponding pixel in either $FE_{MaxB}$ or $FE_{LocB}$ providing the smoother final contour, i.e. the smallest distance with the neighboring border pixel, is selected as the final free edge pixel:

$$\begin{aligned} \begin{aligned} FE_{Final} =&\big \{ \{ fe_{i+1} \in FE_{MaxB}\ \mid \ dist(fe_{i+1}, fef_i)< dist(fe_{j+1}, fef_i) \}\\&\qquad \,\, \cup \{ fe_{j+1} \in FE_{LocB}\ \mid \ dist(fe_{j+1}, fef_i) < dist(fe_{i+1}, fef_i) \} \big \} \end{aligned} \end{aligned}$$

(8)

where $fef_i$ is already in $FE_{Final}$. The final expanded curve that represents the leaflet free edges is the collection of all final candidate free edge pixels (Fig. 6d).

3 Experimental Results

3.1 Experimental Setup and Video Dataset

As there are no public PHV video datasets available, we created our own with an experimental setup using tricuspid biological PHVs submitted to the Pulse Duplicator system from ViVitro Labs Inc., the world’s most widely cited and used in vitro cardiovascular hydrodynamic testing system [18]. The Pulse Duplicator simulates one half of the human heart and generates a pulsatile flow through the mounted PHV. Flow and pressure sensors, located on each side of the valve, measure and collect the flow data during each simulation.

Videos of the simulations were recorded with a Photron SA3 high speed digital camera with a 1000 fps frame rate, a 1/1000 s shutter speed, and a 300 frame test cycle, i.e. from the moment the valve opens until it closes completely, with resolutions between 400 $\times$ 400 and 1024 $\times$ 1024 pixels depending on the valve. Sample frames are shown in Fig. 1. The dataset comprises three videos taken from three different PHVs, for a total of 1745 frames. Ground truth data, defining the leaflet free edges, were obtained in a semi-automatic fashion.

3.2 Evaluation

We compare our proposed extension scheme for the detection of the leaflet free edges (Sect. 2.2) to the active contour-based orifice area segmentation (Sect. 2.1, based on the work in [15]), the latter acting as a baseline. We first compare the PHVs’ orifice curves, which constitute key data for PHV performance evaluation. Figure 7 illustrates the orifice area over time for the three PHVs. For our proposed method, the orifice area is computed from the region bordered by $FE_{Final}$. The results are in very good agreement with the ground truth, both in terms of trend and magnitude, and show a significant improvement over the baseline, which systematically falls short of the actual orifice area. Table 1, which shows the root-mean-square error (RMSE) values for the three valves computed from the differences with the ground truth, confirms the accuracy of our proposed method versus that of the baseline. The information extracted from our proposed method can be used, for instance, to determine how fast the valve opens and closes, how long its stays open, as well as its maximum opening, which constitute crucial data for PHV design assessment.

Fig. 7.

Comparison of the segmented orifice areas over time for the baseline and proposed approaches with the ground truth data (GT).

Table 1. RMSE values of the segmented orifice areas for the baseline and proposed approaches, both absolute (in pixels) and relative (to the maximal orifice area).

Table 2. Average Hausdorff distance $d_H$ (± standard deviation) in pixels between the ground truth and the orifice boundaries for the baseline and proposed approaches.

Although the orifice curves are relevant from a valve evaluation perspective, they do not allow to properly evaluate the orifice shape accuracy. Therefore, we also evaluate our approach with a boundary matching metric, i.e. the Hausdorff distance, defined between curves A and B as follows [19]:

$$\begin{aligned} d_H(A,B) = \max \big (\max _i\{ d(a_i,B) \}, \max _j \{ d(b_j,A) \} \big ) \end{aligned}$$

(9)

$$\begin{aligned} d(a_i,B) = \min _j ||b_j - a_i ||\end{aligned}$$

(10)

where $a_i$ is a point on A and $b_j$ a point on B. Table 2 shows the average $d_H$ over all frames between the ground truth and the orifice boundaries extracted by the baseline and the proposed approaches, for the three PHVs. The smaller distances obtained for our method confirm its ability to recover the shape of the actual orifice area more accurately then the baseline. On a frame-by-frame basis, the largest distances typically occur when the valve is almost closed.

Figure 8 shows typical examples of the orifice boundaries extracted by the baseline and proposed approaches at various moments in the cycle, along with $d_H$ values. The first three columns constitute good cases, in which our approach successfully tracked the leaflet free edges, whereas the baseline approach failed locally either due to the leaflet’s visible inner side (2nd and 3rd columns), or due to the presence of small attached pieces (see 1st column, left region). The last column constitutes a bad case in which our approach yielded small errors due to local brightness maxima, typically occurring near the commission points. A stricter smoothness criterion could address this issue.

Fig. 8.

Typical results and Hausdorff distances $d_H$, at different moments in the cycle, for the baseline and proposed approaches, along with the ground truth, for PHV A.

Fig. 9.

Comparison with attractors (yellow arrows): original (cropped) frame (a), results (red) of [15] without (b) and with (c) attractors and of our proposed method (d). Blue arrows indicate problematic areas, where attractors fail and our method succeeds. (Color figure online)

3.3 Comparison with Attractors

We also compare our free edge detection approach to that based on attractors [15]. In [15], artificial edges are added to the circular outline of the valve on both sides of the commission points to improve the active contour segmentation and detect the actual orifice area. However, this approach only works in cases where the free edges are located on the circular outline, which only covers a special case of valves and motion. Figure 9 shows that the method in [15], contrary to ours, is unable to detect the free edges when the valve’s motion is such that leaflet inner sides are visible and the free edges are away from the circular outline.

4 Conclusion

This paper presents an automatic approach for the quantitative performance analysis of bioprosthetic heart valves from high speed videos recorded during in vitro simulations of the opening and closing motions. The proposed approach addresses the issue of the valve orifice being partially bordered by the inner side of its leaflets, by tracking the actual leaflet free edges. An initial estimation of the orifice area is first obtained via active contouring and then refined via a curve extension scheme, allowing the method to successfully segment the orifice area for valve design evaluation purposes. Evaluation on videos of three valves demonstrated our approach’s effectiveness to detect the leaflet free edges and thus obtain the actual orifice area during a complete cycle. Our approach outperformed a baseline algorithm both in terms of valve design evaluation metrics (accuracy of the temporal orifice area curves) and computer vision evaluation metrics (accuracy of the orifice shape). Future works will explore a setup using several cameras, mitigating the occlusion of the valve orifice while providing more visual information on the leaflet motion pattern.