Elsevier

Neurocomputing

Volume 507, 1 October 2022, Pages 345-357
Neurocomputing

Forgery-free signature verification with stroke-aware cycle-consistent generative adversarial network

https://doi.org/10.1016/j.neucom.2022.08.017Get rights and content

Abstract

In recent years, the performance of handwritten signature verification (HSV) has been considerably improved by deep learning methods. However, deep HSV still faces significant challenges due to the lack of training data, especially for skilled forgeries. In this context, signature synthesis is a promising alternative to address the problem of insufficient data. Compared with offline modality, online signatures are more likely to produce natural duplicates by virtue of their dynamic information. Therefore, we propose a novel convolutional neural network model for offline HSV, called SigCNN, and utilize CycleGAN in style transfer fields to generate realistic offline signatures from online specimens and their duplicates. To compensate for the deficiency of vanilla CycleGAN in generating diverse stroke widths, we propose a new method, Stoke-cCycleGAN, to generate signatures at desired stroke width levels. By online signature duplication and online-to-offline conversion, our SigCNN model can be trained without requiring skilled forgeries. Experimental results showed that our SigCNN trained on generated signatures achieved competitive results on public datasets compared to existing methods. Code of Stroke-cCycleGAN is available at https://github.com/KAKAFEI123/Stroke-cCycleGAN.

Introduction

The handwritten signature is a widely accepted biometric trait for identity verification [1], as it can be acquired in a user-friendly and non-invasive manner. Handwritten signature verification (HSV) authenticates a given signature by comparing it with the reference signatures of its claimed authorship. This is a challenging task because handwritten signatures generally have large intra-writer variability and may be skillfully imitated by malicious forgers. In recent years, the performance of HSV has been considerably improved by deep learning models [2], [3], [4], [5], whereas it still faces significant challenges from limited real-world data and scarce skilled forgeries for training [6]. A large-scale signature dataset is not always available for training, because the distribution and commercial use of signatures may encounter legal problems such as privacy concerns. Besides, the collection of skilled forgeries requires the counterfeiters to carefully observe, practice and imitate the signing process of genuine signatures, thus leading to expensive manpower and time costs. In this situation, signature synthesis and augmentation become vital and have gained increasing research attention in recent years [7], [8], [9], [10], [11].

According to data acquisition methods, signatures can be divided into two categories [12]: 1) offline signatures, which are scanned images of signatures written on paper documents; and 2) online signatures, which are time series collected by electronic devices such as tablets. For online HSV, methods for i) duplicating signatures from real-world references [13], [14], [15], [16], [17], [18], [11] and ii) synthesizing signatures of new identities in the absence of real references [19], [20], [21], [22], [23], [24] have been explored. Among these methods, model-based strategies [16], [20], [21], [17], [18], [11] are mostly based on the kinematic theory of rapid human movements [25]. In particular, the sigma-lognormal model (ΣΛ model) [26] analytically decomposes the complex movement into a weighted sum of parameterized lognormal responses, from which signature trajectories can be reconstructed. Galbally et al. [20], [21] combined spectral analysis, geometric deformation and ΣΛ parameter variation to generate synthetic online signatures. Diaz et al. [16] proposed duplicating signatures from real data by perturbing the ΣΛ parameters, thereby improving the training of the HSV system. Ferrer et al. [17] generated skilled forgeries from genuine online signatures by forging both the trajectory and velocity based on the ΣΛ model. Lai et al. [11], [18] proposed a SynSig2Vec framework to duplicate signatures at various distortion levels by introducing various perturbations to the ΣΛ parameters, and then trained the Sig2Vec model in a forgery-free manner via a novel learning-by-synthesis strategy. Their method has achieved state-of-the-art results on DeepSignDB [3] (the largest online dataset so far), which proved the high quality of their ΣΛ-based duplicates.

In offline HSV, offline signature variants can be generated from offline real data (off-2-off) via geometrical/morphological transformations [27] or bio-inspired models [28]. In addition, methods generating realistic offline signatures from online specimens (on-2-off) have been developed to enlarge offline databases. In contrast to off-2-off methods, on-2-off methods can introduce natural variation to variants more easily, as they take dynamic features not available in offline modality into account during the duplication and synthesis process. Rabasse et al. [29] used a warping transform between two genuine online signatures from the same writer to generate duplicated data and converted them into static bitmap images for verification. Diaz-Cabrera et al. [30] combined several dynamic properties to generate enhanced offline signatures from real online data. In [31], they utilized a cognitive-inspired model to duplicate online signatures, and then converted the online samples to offline signatures using the ink model [32]. Ferrer et al. [17] proposed to duplicate realistic online and offline skilled forgeries from online specimens. Apart from the on-2-off methods based on online signatures or their duplicates [29], [30], [31], [17], there are also several methods converting fully synthetic signature trajectories to realistic offline signatures via the ink model [22], [23], [24].

Most abovementioned on-2-off methods utilized rule-based ink models, and few studies have used deep learning models to realize automatic on-2-off conversion. Different from hand-crafted models, deep learning models are learnable, and can be trained to get command of the on-2-off conversion by their powerful learning ability. Although Melo et al. [7] utilized a neural network to map online and offline signatures, they still required a special training set of paired online and offline signatures, which is not available for most existing datasets. In addition, mismatching between two versions of one signature due to imperfect data collection [33] may cause poor learning effects. Hence, this paper applies an unsupervised deep learning model, CycleGAN [34], to realize automatic on-2-off conversion without requiring paired training data. However, the stroke widths generated by the vanilla CycleGAN are uniform and in lack of diversity. This is not exactly consistent with real-world signatures written with various pens. Therefore, this paper introduces stroke width embeddings and proposes Stroke-cCycleGAN to generate offline signatures with diverse stroke widths.

The overall diagram for our system is shown in Fig. 1. Firstly, considering online signatures with dynamic information are more suitable for natural duplication, we duplicate the online signatures based on ΣΛ model, and then utilize the CycleGAN or Stroke-cCycleGAN to realize automatical on-2-off signature conversion. After that, sufficient realistic offline signatures are generated with natural duplicates. Secondly, we propose a novel SigCNN model to learn multi-scale and multi-level representations from signature images, and train the SigCNN on the generated realistic signatures in the training phase. Thirdly, in the enrollment phase, registered users enroll their genuine signatures as reference signatures, which are fed into the SigCNN for feature extraction and then stored as templates. Lastly, in the verification phase, the query signature is authenticated after being compared with the reference signatures of its stated identity. And our main contributions are as follows:

  • 1) We propose to use CycleGAN, an unsupervised image translation model, to generate realistic offline signatures from online specimens and their duplicates without requiring paired online and offline signatures. To the best of our knowledge, this is the first paper which leverages deep learning based style transfer [35] techniques for unpaired on-2-off signature conversion.

  • 2) The stroke widths generated by the vanilla CycleGAN are uniform, which is not exactly consistent with real-world signatures with different stroke widths. To increase the diversity of generated stroke widths, we introduce the stroke width as a conditional input and propose Stroke-cCycleGAN, which can generate offline signatures at desired stroke width levels.

  • 3) We propose a lightweight and effective SigCNN model for offline signature verification, which was trained on the generated signatures and achieved an obvious boost in the verification accuracy.

  • 4) By following the learning-by-synthesis strategy of the SynSig2Vec method [11], our proposed SigCNN model requires only genuine online signatures and their ΣΛ-based duplicates for training, i.e., no skilled forgeries are required, yet achieves new state-of-the-art performance under global threshold.

The remainder of this paper is organized as follows: Section 2 provides a detailed description of the proposed on-2-off signature generation and forgery-free signature verification system. Section 3 describes the experiments and presents analyses to validate the proposed method. Section 4 provides the concluding statements.

Section snippets

Forgery-free Offline Handwritten Signature Verification Methodology

The proposed forgery-free offline HSV methodology comprises three parts: 1) Duplicating online signatures based on ΣΛ model; 2) generating realistic offline signatures from online specimens and their duplicates; 3) training the SigCNN model for signature feature extraction based on on-2-off data in the training phase, and testing the SigCNN on real-world offline datasets in the verification phase.

Datasets and Implementation Details

Datasets and Preprocess. Three subsets from DeepSignDB, listed in Table 2, were adopted as online signatures for duplication. For each online genuine signature, we created 10 duplicates in low distortion and 10 duplicates in high distortion. In the G & D case, each sample subset SΩ comprises one genuine reference signature, |SP|=5 duplicates in low distortion, and |SN|=10 duplicates in high distortion. In the G & F case, SΩ is composed of 1+|SP|=6 genuine signatures and |SN|=10 forgeries for

Conclusion

In this paper, a novel automatic on-2-off method based on CycleGAN in the style transfer field is proposed to generate realistic offline signatures from online genuine and duplicated specimens. Unlike previous deep on-2-off methods, our approach uses the unpaired style transfer scheme without requiring paired online and offline datasets. To compensate for the deficiency of vanilla CycleGAN in generating diverse stroke widths, we introduce embedded stroke width as a conditional input and propose

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This research is supported in part by NSFC (Grant No.: 61936003) and GD-NSF (No.2017A030312006, No.2021A1515011870).

Jiajia Jiang is a master student in signal and information processing at the South China University of Technology, Guangzhou, China. She received a BS in electronics and information engineering from South China University of Technology, Guangzhou, China in 2020. Her research interests include machine learning and handwritten biometric recognition.

References (64)

  • Y. Chong et al.

    Style transfer for unsupervised domain-adaptive person re-identification

    Neurocomputing

    (2021)
  • C.-L. Liu et al.

    Handwritten digit recognition: investigation of normalization and feature extraction techniques

    Pattern Recognition

    (2004)
  • E.N. Zois et al.

    Offline signature verification and quality characterization using poset-oriented grid features

    Pattern Recognition

    (2016)
  • P. Maergner et al.

    Combining graph edit distance and triplet networks for offline signature verification

    Pattern Recognition Letters

    (2019)
  • Y. Guerbai et al.

    The effective use of the one-class SVM classifier for handwritten signature verification based on writer-independent parameters

    Pattern Recognition

    (2015)
  • M. Diaz et al.

    A perspective analysis of handwritten signature technology

    ACM Computing Surveys (CSUR)

    (2019)
  • R. Tolosana et al.

    DeepSign: Deep on-line signature verification

    IEEE Transactions on Biometrics, Behavior, and Identity Science

    (2021)
  • V.K. Melo et al.

    Deep learning approach to generate offline handwritten signatures based on online samples

    IET Biometrics

    (2019)
  • M.M. Yapici et al.

    Deep learning-based data augmentation method and signature verification system for offline handwritten signature

    Pattern Analysis and Applications

    (2021)
  • H. Li et al.

    AVN: An adversarial variation network model for handwritten signature verification

    IEEE Transactions on Multimedia

    (2021)
  • T.M. Maruyama et al.

    Intrapersonal parameter optimization for offline handwritten signature augmentation

    IEEE Transactions on Information Forensics and Security

    (2020)
  • S. Lai, L. Jin, Y. Zhu, Z. Li, L. Lin, SynSig2Vec: Forgery-free learning of dynamic signature representations by sigma...
  • J. Galbally et al.

    Improving the enrollment in dynamic signature verfication with synthetic samples

  • M.E. Munich et al.

    Visual identification by signature tracking

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2003)
  • M. Song et al.

    An immune clonal selection algorithm for synthetic signature generation

    Mathematical Problems in Engineering

    (2014)
  • M. Diaz et al.

    Towards an automatic on-line signature verifier using only one reference per signer

  • M.A. Ferrer et al.

    Generating off-line and on-line forgeries from on-line genuine signatures

  • S. Lai et al.

    SynSig2Vec: Learning representations from synthetic dynamic signatures for real-world verification

  • D.V. Popel, Signature analysis, verification and synthesis in pervasive environments, in: Image Pattern Recognition:...
  • M.A. Ferrer et al.

    Static signature synthesis: A neuromotor inspired approach for biometrics

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2014)
  • M.A. Ferrer et al.

    A behavioral handwriting model for static and dynamic signature synthesis

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2016)
  • M.A. Ferrer et al.

    Static and dynamic synthesis of Bengali and Devanagari signatures

    IEEE Transactions on Cybernetics

    (2017)
  • Cited by (9)

    • A new synthetic feature generation scheme based on artificial immune systems for robust offline signature verification

      2023, Expert Systems with Applications
      Citation Excerpt :

      Unfortunately, skilled forgeries were not considered in this work. Quite recently, Jiang et al. (2022) adopted a Cycle GAN to produce synthetic genuine and skilled forgeries in order to compensate the need of real skilled forgeries in the training of a SigCNN model. Next, the SigCNN was retrained on real signatures of each writer along with random forgeries to perform the verification task.

    View all citing articles on Scopus

    Jiajia Jiang is a master student in signal and information processing at the South China University of Technology, Guangzhou, China. She received a BS in electronics and information engineering from South China University of Technology, Guangzhou, China in 2020. Her research interests include machine learning and handwritten biometric recognition.

    Songxuan Lai received the Ph.D degree in electronics and information engineering from South China University of Technology in 2021. He is currently working as an artificial intelligence engineer at Huawei Cloud. His research interests include handwriting analysis and recognition, OCR systems, and machine learning.

    Lianwen Jin received the B.S. degree from the University of Science and Technology of China, Anhui, China, and the Ph.D. degree from the South China University of Technology, Guangzhou, China, in 1991 and 1996, respectively. He is currently a Professor with the School of Electronic and Information Engineering, South China University of Technology. He is the author of more than 100 scientific papers. Dr. Jin was a recipient of the award of New Century Excellent Talent Program of MOE in 2006 and the Guangdong Pearl River Distinguished Professor Award in 2011. His research interests include image processing, handwriting analysis and recognition, machine learning, cloud computing, and intelligent systems.

    Yecheng Zhu received the master’s degree in South China University of Technology in 2021. His research interests include machine learning, handwriting analysis and computer vision.

    Jiaxin Zhang is a master student in electronic and communication engineering at the South China University of Technology, Guangzhou, China. He received a BS in Optoelectronics Information Science and Engineering from South China University of Technology, Guangzhou, China in 2020. His research interests include machine learning, document analysis and recognition, and computer vision.

    Bangdong Chen is a master student in signal and information processing at the South China University of Technology, Guangzhou, China. He received a BS in electronics and information engineering from South China University of Technology, Guangzhou, China in 2020. His research interests include machine learning and computer vision.

    1

    S. Lai contributed to this work in his PhD.

    View full text