The Kolmogorov–Sinai entropy in the setting of fuzzy sets for image texture analysis and classification

doi:10.1016/j.patcog.2015.12.012

Pattern Recognition

Volume 53, May 2016, Pages 229-237

https://doi.org/10.1016/j.patcog.2015.12.012 Get rights and content

Highlights

•
Development of a new method for feature extraction using chaos and fuzzy sets.
•
Modeling Kolmogorov–Sinai (K–S) entropy with fuzzy logic is feasible.
•
Fuzzy K–S entropy is an effective feature for analysis and classification of textural images.

Abstract

The Kolmogorov–Sinai (K–S) entropy is used to quantify the average amount of uncertainty of a dynamical system through a sequence of observations. Sequence probabilities therefore play a central role for the computation of the entropy rate to determine if the dynamical system under study is deterministically non-chaotic, deterministically chaotic, or random. This paper extends the notion of the K–S entropy to measure the entropy rate of imprecise systems using sequence membership grades, in which the underlying deterministic paradigm is replaced with the degree of fuzziness. While constructing sequential probabilities for the calculation of the K–S entropy is difficult in practice, the estimate of the K–S entropy in the setting of fuzzy sets in an image is feasible and can be useful for modeling uncertainty of pixel distributions in images. The fuzzy K–S entropy is illustrated as an effective feature for image analysis and texture classification.

Introduction

Texture is an important feature to describe an attribute of an image, which is a phenomenon existing in natural scenes, physical and biological appearances, and art work. Mathematical methods such as measures of smoothness, coarseness, and spatial regularity of distributions of pixels are often used to quantify and classify the texture content of different types of images. Although texture is inherently present in images, it is easy to recognize but difficult to define [1]. This is because texture is subject to human perception, and therefore there is no single precise mathematical definition of texture [2]. In general, there are three main approaches to the analysis of texture: statistical, structural, and spectral [3]. Statistical approaches characterize textures as smooth, coarse, grainy, and so on. Structural techniques deal with the arrangement of image primitives, such as the description of texture based on regularly spaced parallel lines. Spectral techniques are based on properties of the frequency content of an image to detect global periodicity by taking into account high-energy, narrow peaks in the spectrum. It is known that texture analysis has a long and rich development in image processing and computer vision [4], [5], [6], [7] as there is a textbook devoted to the topic of image texture [8]. In fact, the extraction of texture features continues receiving considerable attention to both technical developments [9], [10], [11], [12], [13], [14] and applications [15], [16], [17], [18], [19], [20].

It has been pointed out that texture analysis is difficult because of the lack of effective methods for characterizing texture at different scales and in terms of spatial multiresolution; therefore, techniques such as wavelet transform and Gabor filter can be useful for texture analysis [6], [21]. Although there are many developments of methods and their applications for texture analysis, the challenge of the classification of texture in images still remains. This is because the properties of texture are ill-defined and complex in nature, being due to the high degree of local spatial variations of image intensity in orientation and scale. Such complexity makes it a burdensome task for current mathematical models to effectively discriminate various types of textural information.

As a matter of fact, information and uncertainty are a coupled entity that exists and needs to be addressed in many complex problems of pattern recognition [22]. It is the uncertainty in information, which makes real-life patterns be both predictable and unpredictable at once. For example, in performance art, we know the pianist will play all the notes that Beethoven wrote, but the performance seems like it can go anywhere spontaneously. This is known as “the uncertainty principle and pattern recognition” [23]. A major school of thought for quantifying the predictability or uncertainty in complex information processing is the theory of chaos and nonlinear dynamics. Chaos is the study of surprises, the nonlinear and the unpredictable; while traditional science deals with supposedly predictable phenomena. In spite of its wide applications in many fields of science and engineering, techniques developed for measuring nonlinear behavior in chaotic signals are mainly concerned with time-series data. Little effort has been spent on the formulation and extension of chaos analysis for quantifying the unpredictability of the nature of texture.

The study presented in this paper was motivated by the utilization of the theories of chaos and fuzzy sets for addressing dynamic uncertainty to characterize the spatial arrangement of the distribution of intensities in textural images. In fact, it has been pointed out that the theory of chaos and fuzzy logic are among the most interesting fields of mathematical research, and these two theories were applied to study the semantic dynamics of self-reference [24]. The rest of this paper is organized as follows. Section 2 briefly describes the concept of the Kolmogorov–Sinai (K–S) entropy. The notion of the fuzzy K–S entropy is introduced in Section 3. Section 4 presents the estimate of the fuzzy K–S entropy in images. Experimental results on different types of textural images and benchmark data are discussed in Section 5. Finally, Section 6 is the conclusion of the research findings.

Section snippets

K–S entropy

The theory of chaos and the notion of entropy measure in information theory have been found useful for solving various problems in complex systems [25], [26], [27], ranging from electromagnetic transmission of information to medicine and biology [28], [29], [30]. The three most well-known quantitative measures of chaos are the Lyapunov exponents, Kolmogorov–Sinai (K–S) entropy, and mutual information [31], [32]. This paper particularly focuses on the K–S entropy [33] to extend its notion to

K–S entropy of fuzzy sets

Let $X = {x}$ be a collection of points. A fuzzy set A in X is defined as a set of ordered pairs: $A = {(x, μ_{A} (x)) | x \in X}$ , where $μ_{A} (x)$ is called the fuzzy membership function for the fuzzy set A, which maps each element of X to a real number in the interval [0, 1] [38]. The entropy of the fuzzy set A, denoted by D(A), is a measure of the degree of its fuzziness, which has the following three properties [39]:

1.
D(A)=0 if A is a crisp (non-fuzzy) set, that is, if $μ_{A} (x) \in {0, 1} \forall x$ ;
2.
D(A) is maximum $\Leftrightarrow μ_{A} (x) = 0.5$ (most

K–S entropy of fuzzy information in images

This section develops a model for estimating the K–S entropy of fuzzy information in images. In fact, image properties are not random unless they are corrupted with noise. The spatial information or the appearance of various objects in many real images are inherently imprecise and non-random. A random process in an image is considered when the histogram of the image is used and treated as a probability density function that expresses the probable occurrence of a certain gray level in the image.

Analysis of typical textural images

Fig. 1 shows the gray-scale images of (a) “Lena” (512×512), (b) partial cancer cell (603×1819), (c) partial normal cell (678×1447), and (d) human abdominal organs (246×366). The cancer cell is of cell lines derived from a human head and neck squamous cell carcinoma (SCC-61). The normal cell is of mouse embryonic fibroblast (MEF) cells. Both cell images were produced by the combination of the focused-ion-beam and scanning-electron-microscope techniques. The organelles of the cancer and normal

Conclusion

A new concept for calculating the K–S entropy in the framework of the theory of fuzzy sets has been presented and applied to extract a useful feature for texture image analysis and classification, while the modeling of the K–S entropy for images is difficult to construct. The fuzzy K–S entropy of an image is a number that characterizes the nonlinear behavior of an ensemble of fuzzy membership grades of a partition of an image space. The extended framework of the K–S entropy can be useful for

Conflict of interest

The author declares that there are no conflicts of interest.

Acknowledgments

The author thanks the Handling Editor, Nicole Vincent, and the two anonymous reviewers for their constructive comments and suggestions, which helped improve the paper. The microscope (cell) images were provided by Kazuhisha Ichikawa of the Institute of Medical Science of the University of Tokyo. The CT (abdomen) image was provided by Taichiro Tsunoyama of the Teikyo University School of Medicine. Satoshi Haga of the University of Aizu assisted the author in partial coding and testing the

Tuan D. Pham received his Ph.D. degree in Civil Engineering in 1995 from the University of New South Wales, Sydney, Australia. He is a Professor of Biomedical Engineering at Linkoping University, Sweden. Prior to this current position, he held positions as a Professor and a Leader of the Aizu Research Cluster for Medical Engineering and Informatics at The University of Aizu, Japan; and Group Leader of Bioinformatics Research at The University of New South Wales, Canberra, Australia.

His current

References (67)

J. Mao et al.
Texture classification and segmentation using multiresolution simultaneous autoregressive models
Pattern Recognit.
(1992)
J. Zhang et al.
Brief review of invariant texture analysis methods
Pattern Recognit.
(2002)
E. de Ves et al.
A statistical model for magnitudes and angles of wavelet frame coefficients and its application to texture retrieval
Pattern Recognit.
(2014)
Z. Ji et al.
Detect foreground objects via adaptive fusing model in a hybrid feature space
Pattern Recognit.
(2014)
J. Xie et al.
Effective texture classification by texton encoding induced statistical features
Pattern Recognit.
(2015)
A. Hafiane et al.
Joint adaptive median binary patterns for texture classification
Pattern Recognit.
(2015)
T. Song et al.
Exploring space-frequency co-occurrences via local quantized patterns for texture representation
Pattern Recognit.
(2015)
S. Hegenbart et al.
A scale- and orientation-adaptive extension of local binary patterns for texture classification
Pattern Recognit.
(2015)
A. Depeursinge et al.
Three-dimensional solid texture analysis in biomedical imagingreview and opportunities
Med. Image Anal.
(2014)
I. Theodorakopoulos et al.
HEp-2 cells classification via sparse representation of textural features fused into dissimilarity space
Pattern Recognit.
(2014)

T.D. Pham

The butterfly effect in ER dynamics and ER-mitochondrial contacts

Chaos Solitons Fractals

(2014)

L.A. Zadeh

Fuzzy sets

Inf. Control

(1965)

A. De Luca et al.

A definition of a nonprobabilistic entropy in the setting of fuzzy sets theory

Inf. Control

(1972)

L.A. Zadeh

Fuzzy sets as a basis for a theory of possibility

Fuzzy Sets Syst.

(1978)

T. Tsunoyama et al.

Identification of intestinal wall abnormalities and ischemia by modeling spatial uncertainty in computed tomography imaging findings

Comput. Methods Programs Biomed.

(2014)

Y. Chen et al.

Entropy and regularity dimension in complexity analysis of cortical surface structure in early Alzheimer׳s disease and aging

J. Neurosci. Methods

(2013)

H. Kwakernaak

Fuzzy random variables–I. Definitions and theorems

Inf. Sci.

(1978)

H. Kwakernaak

Fuzzy random variables–II. Algorithms and examples for the discrete case

Inf. Sci.

(1979)

M.L. Puri

Fuzzy random variables

J. Math. Anal. Appl.

(1986)

N. Orlov et al.

WND-CHARMmulti-purpose image classification using compound image transforms

Pattern Recognit. Lett.

(2008)

D.A. Forsyth et al.

Computer Vision: a Modern Approach

(2003)

M. Nixon et al.

Feature Extraction and Image Processing for Computer Vision

(2012)

R.C. Gonzalez et al.

Digital Image Processing

(2010)

M. Petrou et al.

Image Processing: Dealing with Texture

(2006)

C. Scharfenberger et al.

Structure-guided statistical textural distinctiveness for salient region detection in natural images

IEEE Trans. Image Process.

(2015)

J.M. Guo et al.

Content-based image retrieval using features extracted from halftoning-based block truncation coding

IEEE Trans. Image Process.

(2015)

R. Parekh

Using texture analysis for medical diagnosis

IEEE MultiMed.

(2012)

E. Rexhepaj et al.

A texture based pattern recognition approach to distinguish melanoma from non-melanoma cells in histopathological tissue microarray sections

PLoS One

(2013)

J.S. Nelson et al.

Improved nuclear medicine uniformity assessment with noise texture analysis

J. Nucl. Med.

(2014)

M. Knop et al.

Tracking protein turnover and degradation by microscopyphoto-switchable versus time-encoded fluorescent proteins

Open Biol.

(2014)

T. Chang et al.

Texture analysis and classification with tree-structured wavelet transform

IEEE Trans. Image Process.

(1993)

G.J. Klir

Uncertainty and Information: Foundations of Generalized Information Theory

(2005)

R. Fritz, The uncertainty principle and pattern recognition, URL: 〈...

Cited by (28)

Renyi entropy analysis of a deep convolutional representation for texture recognition
2023, Applied Soft Computing
Despite the recent success of convolutional neural networks in computer vision in general, texture images still pose an important challenge to those models, especially when dealing with textures “in the wild”. In this context, deep learning models can benefit from the introduction of features long known to be useful for texture modeling. And that is the case of entropy, a measure of texture regularity that has played an important role in classical computer vision. Based on this observation, here we propose an alternative analysis over deep convolutional neural features based on entropy for texture representation and, particularly, texture classification. More precisely, we couple a module to the convolutional backbone that locally computes the Renyi entropy of the latent representation at multiple levels. The rationale for using Renyi entropy is essentially two-fold: (1) It connects the concept of entropy with multifractal theory, another well explored measure especially in domains of application where we seek some physical interpretation of the descriptors, e.g., in medicine, biology, and others; (2) It has an extra degree of freedom ( $α$ parameter) that can be fine-tuned. The main contribution of this study is the development of a strategy that can improve the performance of convolutional neural networks in texture recognition tasks, adding low computational cost. The effectiveness of our method is verified in texture classification of benchmark datasets, as well as in a practical task of plant species identification. Our method achieves a competitive accuracy of 84.5% in the classification of KTH-TIPS-2b and 80.9% in FMD, which are two challenging benchmark databases. Besides, an accuracy of 91.6% is obtained in the plant identification application, to our knowledge outperforming all the results previously published on this task. In both scenarios, the proposed descriptors outperform several approaches from the state-of-the-art, confirming the method potential as a rich and robust solution for texture analysis in general. The results also suggest that information-theoretical measures like entropy can be a reliable source of information to compose a precise and robust latent representation of texture images.
Automated detection of conduct disorder and attention deficit hyperactivity disorder using decomposition and nonlinear techniques with EEG signals
2021, Computer Methods and Programs in Biomedicine
Citation Excerpt :
Entropy measures the chaos within a system and is useful in measuring the ambiguity and variability in signals [33]. In this study, seven entropies such as approximate entropy [34], sample entropy [35], Kolmogorov-sinai entropy [36], Tsalli's entropy [37], fuzzy entropy [38], higher-order spectrum (HOS) entropy [39] and modified multi-scale entropy (MMSE) [40] were studied. FD is a popular feature used to distinguish signals.
Attention deficit hyperactivity disorder (ADHD) is often presented with conduct disorder (CD). There is currently no objective laboratory test or diagnostic method to discern between ADHD and CD, and diagnosis is further made difficult as ADHD is a common neuro-developmental disorder often presenting with other co-morbid difficulties; and in particular with conduct disorder which has a high degree of associated behavioural challenges. A novel automated system (AS) is proposed as a convenient supplementary tool to support clinicians in their diagnostic decisions. To the best of our knowledge, we are the first group to develop an automated classification system to classify ADHD, CD and ADHD+CD classes using brain signals.
The empirical mode decomposition (EMD) and discrete wavelet transform (DWT) methods were employed to decompose the electroencephalogram (EEG) signals. Autoregressive modelling coefficients and relative wavelet energy were then computed on the signals. Various nonlinear features were extracted from the decomposed coefficients. Adaptive synthetic sampling (ADASYN) was then employed to balance the dataset. The significant features were selected using sequential forward selection method. The highly discriminatory features were subsequently fed to an array of classifiers.
The highest accuracy of 97.88% was achieved with the K-Nearest Neighbour (KNN) classifier. The proposed system was developed using ten-fold validation strategy on EEG data from 123 children. To the best of our knowledge this is the first study to develop an AS for the classification of ADHD, CD and ADHD+CD classes using EEG signals.
Our AS can potentially be used as a web-based application with cloud system to aid the clinical diagnosis of ADHD and/or CD, thus supporting faster and accurate treatment for the children. It is important to note that testing with larger data is required before the AS can be employed for clinical applications.
Review-material degradation assessed by digital image processing: Fundamentals, progresses, and challenges
2020, Journal of Materials Science and Technology
Material degradation is accompanied by the changes in surface structure, morphology, and composition. These changes can be recorded by a variety of image acquisition devices that export digital images in grayscale or true color to a detector. Information regarding corrosion type and extent can be extracted with image processing methods. This paper provides a comprehensive review of material degradation assessed by digital image processing. Digital image processing systems used to assess material degradation are briefly reviewed, and the algorithms developed to process metallic materials degradation images are described. Physical and electrochemical methods that can be used to support digital image processing results are summarized, and future work that will augment the present methods of evaluating material degradation are discussed.
Distributed cooperative learning over time-varying random networks using a gossip-based communication protocol
2020, Fuzzy Sets and Systems
Motivated by applications of distributed estimation and distributed decision making in wireless sensor networks (WSNs) and unmanned aerial vehicle (UAV) networks, we study a distributed learning problem over time-varying undirected random networks. Using a gossip-based communication protocol, a novel distributed cooperative learning (DCL) algorithm, termed the gossip-based DCL (GBDCL) algorithm, is presented to solve the problem by training the raw data distributed and blocked throughout different nodes. Exploiting the robustness of the gossip-based protocol, each node is guaranteed to build the same learning model in theory against random disconnections and communication route variations in the network topology. It is proved that the GBDCL algorithm converges to the optimal consensus asymptotically. The correctness and effectiveness of the presented GBDCL algorithm are verified in the theoretical analysis and simulations.
Automated demarcation of the homogeneous domains of trace distribution within a rock mass based on GLCM and ISODATA
2020, International Journal of Rock Mechanics and Mining Sciences
Citation Excerpt :
The texture is an important feature to describe an attribute of an image, which is a phenomenon existing in natural scenes,9 physical and biological appearances,10 and artwork.11 Mathematical methods such as measures of smoothness, coarseness, and spatial regularity of pixels distributions are often used to quantify and classify the texture content of different types of images.12 Rock texture refers to the degree of crystallinity, grain size, and fabric (geometrical relationships) among the constituents of a rock, which is a major factor in determining the mechanical behaviour of rocks and in the prediction of performance of rock strength, rock cutting and rock drilling.13–15
In view of the irregular trace distribution of rock discontinuities, rock mass appears as both a statistical distribution and a texture distribution in the spatial image. This paper proposes a new method on statistical texture analysis for automated demarcating the homogeneous domains of trace distribution within a rock mass. Grey-Level Co-occurrence Matrix (GLCM) is used to quantify the statistical texture features of trace distribution. Relativity, Inverse Difference Moment and Entropy are screened from ten texture parameters of GLCM using robustness analysis and using principal components analysis. The reliability of three screened texture parameters is verified by comparing the Chebyshev polynomials fitting of three screened texture parameters with Normal distribution, Fisher distribution, and Exponent distribution using χ² testing. Automated demarcation of the homogeneous domains is implemented by means of classifying three texture parameters of Relativity, Inverse Difference Moment and Entropy in a moving window using the Iterative Self-Organizing Data Analysis Techniques Algorithm (ISODATA). The screening process of texture parameters and a case study indicates that texture parameters and automated demarcation method is so robust, reliable, and efficient that it could replace the traditional representation of the probability statistics in trace distribution and greatly save a lot of manual labor in a large-scale domain.
Computer-aided diagnosis of congestive heart failure using ECG signals – A review
2019, Physica Medica
Citation Excerpt :
The KS entropy is a parameter used to enumerate chaos, for solving problems in complex systems. It is employed to measure the ambiguity of a system, linked to a series of outcomes or observations of chaotic trajectories after m units of time [24]. Modified Multi Scale Entropy (MMSE)
The heart muscle pumps blood to vital organs, which is indispensable for human life. Congestive heart failure (CHF) is characterized by the inability of the heart to pump blood adequately throughout the body without an increase in intracardiac pressure. The symptoms include lung and peripheral congestion, leading to breathing difficulty and swollen limbs, dizziness from reduced delivery of blood to the brain, as well as arrhythmia. Coronary artery disease, myocardial infarction, and medical co-morbidities such as kidney disease, diabetes, and high blood pressure all take a toll on the heart and can impair myocardial function. CHF prevalence is growing worldwide. It afflicts millions of people globally, and is a leading cause of death. Hence, proper diagnosis, monitoring and management are imperative. The importance of an objective CHF diagnostic tool cannot be overemphasized. Standard diagnostic tests for CHF include chest X-ray, magnetic resonance imaging (MRI), nuclear imaging, echocardiography, and invasive angiography. However, these methods are costly, time-consuming, and they can be operator-dependent. Electrocardiography (ECG) is inexpensive and widely accessible, but ECG changes are typically not specific for CHF diagnosis. A properly designed computer-aided detection (CAD) system for CHF, based on the ECG, would potentially reduce subjectivity and provide quantitative assessment for informed decision-making. Herein, we review existing CAD for automatic CHF diagnosis, and highlight the development of an ECG-based CAD diagnostic system that employs deep learning algorithms to automatically detect CHF.

View all citing articles on Scopus

His current research interests include image processing, pattern recognition, fractals and chaos applied to biology and medicine. His research has been funded by the Australian Research Council, JSPS (Japan), academic institutions, and industry. He has served as an Area Editor, Associate Editor, and Editorial Board Member of several journals and book series including Pattern Recognition (Elsevier), Current Bioinformatics (Bentham), Recent Patents on Computer Science (Bentham), Proteomics Insights (open access journal, Libertas Academica Press), Book Series on Bioinformatics and Computational BioImaging (Artech House), International Journal of Computer Aided Engineering and Technology (Inderscience Publishers). Pham has served as chair and technical committee member of more than 30 international conferences in the fields of image processing, pattern recognition, and computational life sciences.

View full text

The Kolmogorov–Sinai entropy in the setting of fuzzy sets for image texture analysis and classification

Highlights

Abstract

Introduction

Section snippets

K–S entropy

K–S entropy of fuzzy sets

K–S entropy of fuzzy information in images

Analysis of typical textural images

Conclusion

Conflict of interest

Acknowledgments

Pattern Recognit.

Pattern Recognit.

Pattern Recognit.

Pattern Recognit.

Pattern Recognit.

Pattern Recognit.

Pattern Recognit.

Pattern Recognit.

Med. Image Anal.

Pattern Recognit.

Chaos Solitons Fractals

Inf. Control

Inf. Control

Fuzzy Sets Syst.

Comput. Methods Programs Biomed.

J. Neurosci. Methods

Inf. Sci.

Inf. Sci.

J. Math. Anal. Appl.

Pattern Recognit. Lett.

Computer Vision: a Modern Approach

Feature Extraction and Image Processing for Computer Vision

Digital Image Processing

Image Processing: Dealing with Texture

Structure-guided statistical textural distinctiveness for salient region detection in natural images

IEEE Trans. Image Process.

Content-based image retrieval using features extracted from halftoning-based block truncation coding

IEEE Trans. Image Process.

Using texture analysis for medical diagnosis

IEEE MultiMed.

A texture based pattern recognition approach to distinguish melanoma from non-melanoma cells in histopathological tissue microarray sections

PLoS One

Improved nuclear medicine uniformity assessment with noise texture analysis

J. Nucl. Med.

Tracking protein turnover and degradation by microscopyphoto-switchable versus time-encoded fluorescent proteins

Open Biol.

Texture analysis and classification with tree-structured wavelet transform

IEEE Trans. Image Process.

Uncertainty and Information: Foundations of Generalized Information Theory