Keywords

1 Introduction

Connected filters, and more generally connective morphology [1, 2] are relative newcomers in the field of mathematical morphology. Rather than being based on adjunctions of erosions and dilations by structuring elements, they are based on a generalised notion of connectivity, and connectivity openings form the building blocks of operators. The key property of these image operators is that they do not consider the image as a collection of pixels, but more as a collection of connected structures, referred to as (quasi) flat-zones or connected components. They work by merging these connected structures and assigning new grey-levels or colours to them, based on certain criteria. This property means that they are strictly edge preserving: they cannot introduce new edges in an image. Their use ranges from simple image filtering, through detection and enhancement of specific structures to hierarchical segmentation.

As so often happens, the development of these filters originally stemmed from practical needs, and only later a rigorous mathematical framework was set up. Likewise, the initial algorithms were comparatively inefficient, but as the potential power of these filters became apparent, a great deal of effort was made into development of efficient algorithms. In this review I will first describe the history of these image operators, introducing the most important types of connected operators. After that, I will present the lattice-theoretical framework of connectivity classes, and several generalisations that have been proposed. This is followed by a discussion of the most important algorithms, both sequential and parallel, for efficient computation of these operators. Finally, I will discuss future perspectives.

2 A Brief History

The very first connected filter was published in the thesis of Klein [3], in which he presented reconstruction operators. The basic reconstruction operator is the reconstruction f from marker g. In the binary case, all connected components of f which have a non-empty intersection with g re preserved, all others are removed. The opening-by-reconstruction is obtained by choosing

$$\begin{aligned} g = f \ominus B \end{aligned}$$
(1)

with B some structuring element. This preserves all connected components of f into which some translate of B fits entirely. If B is a ball of radius r, this preserves only those connected components with an erosion width larger than 2r. An example is shown in Fig. 1. Quite clearly, the opening by reconstruction preserves shapes exactly, whereas the regular, structural opening does not.

Fig. 1.
figure 1

A comparison of openings: (a) original image of bacteria; (c) structural opening by \(11\,\times \,11\) square; (b) reconstruction of (a) by (b); (d) moment-of-inertia opening with \(\lambda =11^4/4\).

Full size image

Area openings were proposed in the early 1990s [4, 5] as filters which remove connected components with an area smaller than some threshold \(\lambda \) from images. After this, Breen and Jones [6] proposed attribute openings as a generalisation. In this case some increasing measure \(\mu (C)\) of each connected component is computed, and compared to some threshold. An example is shown in Fig. 1(d), where the measure is the trace of the moment of inertia, and the threshold is set to that of an \(11\times 11\) square. The latter is clearly more selective for elongated features.

The above filters are anti-extensive, i.e. the remove bright, or foreground structures. Extensive filters which remove dark, or background structures can be defined by duality, i.e., by inverting the image, applying the opening (or thinning) and inverting the result.

The grey-scale case can be obtained by threshold superposition [7]. The easiest way to visualise this process is through the concept variously known as component trees [8, 9], max-trees and min-trees [10], and opening trees [11]. An example of a max-tree of a simple grey scale image is shown in Fig. 2. It can be constructed by thresholding the grey level image at all threshold levels, and computing the connected components, called peak components, of every threshold set. It is easy to see that each peak component at grey level h is nested within a single peak component at a lower level. Therefore, the peak components can be organized into a tree structure called the max-tree, because the leaves are the local maxima. A min-tree is is simply a max-tree of the inverted image, and component tree may be either.

Fig. 2.
figure 2

A simple grey-scale image, its peak components and max-tree

Full size image

Once a max-tree has been computed filtering can be done by applying some decision function which determines which nodes of the tree to keep, and which to remove. In the simplest case, some increasing attribute, such as area, is computed for each node, and this is compared to some threshold. In this case we end up pruning those branches of the tree which contain nodes that are too small to meet the criterion. All pixels in the removed nodes are assigned the grey levels of the nearest surviving ancestor. In more complicated cases, the attributes are non-increasing, and removed and retained nodes along the root path may alternate. In this we need to make more complicated filtering rules to assign new grey levels [10, 12]. Although we add complexity, it does allow implementation of scale invariant shape filters [12, 13], which allow filtering of features such as blood vessels based on shape, rather than size [14, 15]. Non-increasing filtering also takes place in so-called vector-attribute filters [16, 17]. In this case each node of the tree contains a feature vector describing size, shape, or grey-level statistics. In the simplest case, we can then retain or remove nodes based on how closely they resemble some prototype vector, using distance measures like euclidean or Mahalanobis distances. Machine learning and clustering can also be used to distinguish different classes of nodes [18]. An interesting development is that of building a component tree of a component tree, and steering the filtering process from that second tree [19].

Several self-dual filters have been developed: levelings [20, 21] as self-dual versions of the reconstruction operators, and level-line tree [22,23,24] also called tree-of-shapes [25] methods in the case of attribute filters, which can be seen as combining a max-tree and min-tree in such a way that the extrema are the leaves.

All these tree structures allow more than just filtering. As these trees are multi-scale representations of the image, all kinds of multi-scale analysis is very readily performed using these trees, including computation of pattern spectra [26,27,28,29] and morphological profiles [30, 31]. There are also important relations between the various partition trees and hierarchical segmentation methods [32,33,34].

Component trees and their relatives require a total order of the pixel values in order to work, so adaptations are need for work in colour, vector and tensor images. Approaches include total pre-orders in component trees [35], merging of trees of shapes from different colour bands [36], and area scale spaces for colour images using local extremal values [37, 38]. However, the most common way to deal with colour, vector or tensor images in connective morphology is through hierarchies of quasi flat zones [32, 39,40,41]. In principle we start out at the flat-zones of the image, i.e. connected regions of maximal extent of constant colour. We then hierarchically merge these into larger components in increasing order of dissimilarity. The simplest approach only performs binary merges, and thus creates a binary partition trees [39]. Alternatively, we can allow mergers of multiple regions simultaneously, creating so-called alpha trees [40].

3 Connectivity Theory

In the discussion above, we implicitly assumed the usual definition of a connected component, as a maximal connected subset of some set X. We did not, however specify what we actually mean by “connected”. Serra [42, 43] introduced the general notion of connectivity classes. These are essentially families of all elements of the lattice under consideration that are connected. In the case of binary images that means the family of all connected sets. In all cases, a connectivity class \(\mathcal {C}\) must have the following three properties

  1. 1.

    \(\mathbf {0} \in \mathcal {C}\)

  2. 2.

    \(\mathcal {C}\) is sup-generating

  3. 3.

    For any set \(\{C_i\} \subset \mathcal {C}\), with i from some index set, \(\bigwedge _i C_i \ne \mathbf {0} \quad \Rightarrow \quad \bigvee _i C_i \in \mathcal {C}\)

This means that the least element of the lattice (\(\emptyset \) in the set theoretical case) is connected, that any element of the lattice can be generated by taking the supremum of elements of \(\mathcal {C}\), and that if the infimum of elements of \(\mathcal {C}\) is not \(\mathbf {0}\), their supremum is connected. In the set theoretical case this means that if the intersection of any number of connected sets is not empty, their union is connected.

It is easily verified that the standard notions of (path) connectivity in images adhere to these three axioms. Many others have been proposed in a framework often called second-generation connectivity, where the standard connectivity is modified either by applying operators like dilation, opening or closings to the image [42, 44, 45], or by using some second image in so-called mask-based second-generation connectivity [46]. These methods allow objects broken up by noise to be considered a single, connected entity, or objects linked by single pixels to be considered separate. This is done by considering whether pixels are connected at a given grey level in a modified image, rather than in the original. A further extension also assigns grey levels to the edges between pixels, allowing further freedom in deciding what is connected [47].

3.1 Beyond Connectivity

In [42] suggested that there are generalisations to connectivity that have been explored further by other researchers. In particular, the notion of hyperconnectivity has received much attention [48,49,50,51,52]. Initially, Serra changed the third axiom of connectivity classes, from merely requiring a non-empty intersection for a group of connected sets to have a connected union, to some stronger requirement called the overlap criterion. This might be that the intersection contains a ball of a certain diameter, leading to viscous hyperconnections [53, 54] It has been shown that this can lead to a family of filters that bridges the gap between connected filters and structural morphological filters [53]. It has since been shown that the overlap criterion is not required to define these hyperconnections, but that other axiomatics can be used [48, 52].

Other extensions of connectivity are partial connections [34], and attribute-space connections [55].

4 Algorithms

In support of all the theoretical developments, algorithms for fast computation of connected operators have been an important area of research [5, 9, 10, 40, 56, 57]. For a recent review see [58]. Many of the algorithms centre around component trees (min-tree, max-tree, tree-of-shapes) on the one hand, and partition trees on the other (binary partition trees and alpha trees, mainly). In the former category, there were two different approaches, i.e. building the tree from root to leaf, by recursive or non-recursive flood-filling [10, 59, 60], and the reverse approach, starting from the leaves joining up branches, and ultimately aggregating them into a single tree. These latter fall into two categories: (i) those using a priority queue [8], derived from the area opening algorithm of Vincent [5], and (ii) using Tarjan’s union-find [9, 61, 62] based on the area opening algorithm in [57]. All algorithms have their pros and cons as reviewed in [58].

An important aspect of efficient computation on modern compute architectures is the ability to use parallellism, available either through multi-core processors, or even the graphics processing unit (GPU). Connected filters are particularly troublesome for parallel implementation since the processing order of most algorithms is very much data driven, and the usual approach of dividing the data over different cores (with some degree of overlap) is hampered by the fact that we do not a priori know how much overlap is needed, because we do not know the location or borders of connected components. This problem was addressed in [63], in which it was shown that max-trees or min-trees of disjoint image regions could be merged into a complete tree, complete with the correct attribute information. The cost of merging is proportional to the number of grey levels, worst case, so the algorithm does not work efficiently beyond 16 bits per pixel. A solution has recently been presented by Moschini et al. [64], in which first a root-to-leaf flood-filling algorithm algorithm is used on a uniformly quantised approximation image, in which the number of grey levels equals the number of processors, after which a variant of Berger’s leaf-to-root algorithm is used to refine the tree. These algorithms have been used in various multi-scale analysis tasks in remote sensing [27, 65] and astronomical imaging [28, 66].

A limitation of both the above parallel algorithms is that they only work on shared-memory parallel computers. In practice this means that images up to a few gigapixel can be processed. This is because the entire max-tree must be stored in memory. In a recent development, a distributed memory algorithm suitable for use on clusters has been developed [67]. This does not build a single max-tree, but rather a forest of trees, each for one tile of the image, which can be stored on one node of the cluster. After building these trees, only the sub-tree connected to the tile boundaries are exchanged and merged hierarchically to correct the topology. Once these corrections have been made, filtering the individual trees yields the same result as filtering the entire max-tree. The results show a speed up of up to 103 on 128 cores, in the case of a 153 Gpixel image.

The above algorithms focus on regular path connectivity, but they have been extended to the case of second-generation connectivity, without loss of efficiency [46, 68], and even some hyperconnections [54, 69].

Other important algorithms concern building the binary partition tree [39], the sequential alpha tree algorithm [40], and a parallel algorithm for alpha trees [70], based on the same merge strategy as [63].

5 Conclusions and Perspectives

Connected filters are versatile tools for image filtering, feature extraction, object recognition, and segmentation. An attractive property of these methods is that they allow modelling of perceptual grouping through changing the notion of connectivity, and simultaneously allow modelling of object properties through the choice of attributes used. Finally, it is relatively straightforward to introduce machine learning into the framework, in the form of a decision function used to accept or reject connected components.

There are still many issues to be addressed. Algorithmically, we still not have a solution that would allow us to use the power of GPUs on these methods. Likewise, the new distributed algorithm for attribute filters only works up to 16-bits per pixel images, because it uses the same merging algorithm as in [63]. The same holds for [70]. Therefore, extreme-dynamic range versions of these algorithms are needed, capable of handling floating point values.

Integration of machine learning is also an area of research that is still in its infancy, and more work is needed here. This is not restricted to the decision function needed in attribute filters, but also in e.g. computation of connectivity masks, where currently only ad hoc approaches are used in determining the best connectivity mask to achieve a particular task. Whether it is possible to learn the best mask is an open question.