A feature fusion sequence learning approach for quantitative analysis of tremor symptoms based on digital handwriting

Highlights

• An automated essential tremor assessment model based on a drawing task.

• First database of 3 types of tremor tasks in patients with essential tremors.

• The patient's diagnosis was independently scored by multiple neurologists.

• Digital ink sequences were analyzed using a hybrid model of CNN and transformer.

• The system incorporates both sequence features and kinematic handwriting features.

Abstract

Essential tremor and Parkinson's disease are common movement disorders, and early diagnosis and evaluation are critical to managing these diseases. Currently, laboratory tests are the only way to diagnose and assess tremor symptoms. Analysis of a patient's fine motor control, especially handwriting, is a powerful tool for disease assessment. However, traditional visual assessment methods by neurologists typically lead to biased diagnostic results due to some subjective factors. Therefore, it is necessary to automatically identify and quantify the captured motion events with the help of artificial intelligence in combination with the various dynamic attributes encapsulated in the digital ink features, such as pen pressure, stroke speed, handwriting variability, etc. In this paper, a novel Transformer deep-learning model is developed for sequence learning of electronic handwriting to effectively evaluate its potential in aiding the diagnosis of tremor symptoms. The one-dimensional convolution with an ingenious fusion attention mechanism is applied to the original pen sensor signal sequences and derived features are used as the embedding layer of the Transformer encoder part, and the global dynamic features are fused before the decision layer. Our proposed system performs excellent on private datasets and outperforms state-of-the-art methods on the PaHaW dataset.

Introduction

Essential tremor (ET) is a neurodegenerative disease that affects approximately 6.38 to 7.63 million people in the United States, close to 2.2% of the population (Louis, 2014). ET typically manifests as postural and intentional tremors involving the upper extremities at a frequency of 8–12 Hz (Louis, 2019), and ET patients have a more than 4-fold increased risk of Parkinson's disease (PD) (Benito-Leon, Louis, & Bermejo-Pareja, 2009). The office history and examination are the sole components required for diagnosis (Shanker, 2019). Neurologists observe patients performing several specific tasks with the naked eye and score each task according to the Clinical Rating Scale for Tremor (CRST) criteria (Fernandez et al., 2019). However, such a protocol depends on the clinical experience and expertise of the physician and a thorough review of the patient, and subjective bias leads to unsatisfactory results in the assessment of movement disorders (Kassubek, 2014). In addition, tremor symptoms can develop and change over time, and manual tracking of disease progression often takes up significant healthcare resources. Further, assessing the disease remains challenging for the elderly, those with poor cognitive abilities or limited mobility (Hu et al., 2011). Therefore, early diagnosis of the disease adapted to the needs of individual patients with movement disorders and accurate assessment of symptom severity is crucial for prognosis and treatment (Ossig & Reichmann, 2015). With this goal in mind, the interest in computer-assisted diagnosis has grown from all walks of life in society over the years (Ali et al., 2019, Parisi et al., 2018). There is an urgent need to provide physicians with objective, multi-modal, quantitative evaluation criteria to aid decision-making.

Neurological disorders that cause brain changes caused by ET and PD, such as neuronal loss, synaptic dysfunction, brain atrophy, etc., may lead to dysfunction of the motor system and its components. According to this view, the unique role of handwriting can be confidently assumed in the context of symptom assessment. Handwriting is a complex activity involving perceptual-motor and cognitive components, and its changes can be considered a promising biomarker for disease assessment (De Stefano et al., 2019, Faundez-Zanuy et al., 2020, Vessio, 2019). A growing body of knowledge has proved that PD and healthy individuals can be automatically distinguished through a simple and easy drawing task (Angelillo et al., 2019, Diaz et al., 2019, Diaz et al., 2021, Drotár et al., 2013, Drotár et al., 2014, Drotár et al., 2014, Drotár et al., 2016, Gupta and Bansal, 2019, Impedovo, 2019, Kamran et al., 2021, Moetesum et al., 2019, Naseer et al., 2020). However, no studies have collected enough handwriting data from ET patients and obtained reliable expert scores to support the construction of an automated quantitative tremor assessment system. Developing a handwriting-based decision support tool is desirable because it could provide a noninvasive, real-time, low-cost solution to optimize standard clinical assessments performed by human experts.

Following this research direction, an online (dynamic) system that uses a digital tablet combined with a stylus with high-precision sensing can be employed (Diaz, Moetesum, Siddiqi, & Vessio, 2021). Such a device can capture temporal-spatial variables during the writing process, as well as information about the pressure exerted by the pen tip on the writing surface, the tilt direction, and even the measurement of the rotation angle. In contrast to the offline (static) morphological characteristics of handwriting, dynamic handwriting analysis deals with the kinetic characteristics during the writing process and is more suitable for disease diagnosis (Angelillo et al., 2019, Diaz et al., 2021, Gupta and Bansal, 2019, Impedovo, 2019). Therefore, selecting the appropriate features is crucial for designing automatic diagnostic systems. Classical statistical feature classifiers focus on the original time series and are susceptible to high dimensionality burden, which leads to overfitting. For this reason, some kinetic features are derived from the raw data and can represent high-level information in handwriting, such as entropy features characterizing sequence complexity or signal-to-noise features (Drotár et al., 2014, Impedovo, 2019, Rosenblum et al., 2013). Although such high-level features can have the ability to aid the model to obtain global information about the handwriting, on the other hand, it also loses detailed local features since arbitrarily long sequences are compressed into single-valued features.

Another approach to obtain tremor information in digital handwriting is to use deep learning models to automatically sense differences in information. Some recent work based on convolutional neural networks (CNN) has exploited morphological features extracted from two-dimensional static images to automatically achieve the recognition of tremor (Diaz et al., 2019, Kamran et al., 2021, Moetesum et al., 2019, Naseer et al., 2020). While this approach can replace manually designed features, it only provides a holistic view of handwriting patterns, and dynamic changes in the temporal direction may also help enhance the learning effect. Kamran et al. (Kamran et al., 2021) constructed a recurrent neural network (RNN) model to learn the dynamic features of electronic handwriting sequences, but it also neglected to explore global information that would likely provide the network with meaningful additional information. The cutting-edge deep-learning model like Transformer, which has natural advantages in processing time series data without losing spatial–temporal information, have not been fully explored. Transformer has significant advantages in almost all domains (including computer vision), with advantages such as its self-attention mechanism for efficient acquisition of high-level features. Rather than compressing the raw data as many researchers have done, we would like to take advantage of the time series nature of the data and fully consider multi-scale channel and time orientation information, with the final decision layer considering fusing depth features and global single-value features to accommodate digital handwriting of variable length.

This paper reports the first attempt to present an automated quantitative assessment system based on Transformer model for the drawing task in movement disorders, as shown in Fig. 1. The system is based on a highly sensitive digital tablet that collects handwriting from a high-dimensional multisensory fusion. The method explores tremor patterns through a cascaded deep network. 1) Constructing multi-scale squeeze-and-excitation-ResNet (MSSE-ResNet) based on raw digital ink data and dynamic features, which serves as an embedding of Transformer model layer to provide meaningful high-level features for the encoder. 2) Further construct encoder with multi-headed self-attentive mechanism, and fuse global kinematic features of handwriting before the classification decision layer to output the severity of tremors. The main contributions of this paper are summarized as follows:

1) Relying on rigorous patient screening experiments (biochemistry, imaging, physical examination, medical history), independent scoring by the neurological expert committee, and a comprehensive consensus score for supervised learning, we established the first essential tremor patient composed of the drawing task tremor inspection task database.

2) We propose the first digital ink Transformer model based on sequence learning combined with a CNN deep feature extraction network for an automated quantification system of movement disorder diseases.

3) We designed complete validation experiments to demonstrate the optimal performance of the proposed method in the evaluation of ET tremor and early diagnosis of PD.