Skip to main content
Log in

Hybrid Hopfield Neural Network

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Hopfield and Tank have shown that a neural network can find solutions for complex optimization problems, although it can be trapped in a local minimum of the objective function returning a suboptimal solution. When the problem has constraints they can be added to the objective function as penalty terms using Lagrange multipliers. In this paper, we introduce an approach inspired by the work of Andrew, Chu, and Gee to implement a neural network to obtain solutions satisfying the linear equality constraints using the Moore–Penrose pseudo inverse matrix to construct a projection matrix to send any configuration to the subspace of configuration space that satisfies all the constraints. The objective function of the problem is modified to include Lagrange multipliers terms for the equations of constraints. Furthermore, we have found that such a condition makes the network converge to a set of stable states even if some diagonal elements of the weight matrix is negative. If after several steps the network does not converge to a stable state, we just solve the problem using simulated annealing that significantly outperforms hill climbing, feed-forward neural network and convolutional neural network. We use this technique to solve the NP-hard Light Up puzzle. Hopfield neural networks are widely used for pattern recognition and optimization tasks. However, the standard Hopfield network model uses non-negative weights between neurons, which can limit its performance in certain situations. By introducing negative weights, the network can potentially learn more complex and nuanced patterns, and exhibit improved convergence properties. Thus, the motivation for the article “Hybrid Hopfield Neural Network” is to explore the benefits of incorporating negative weights into Hopfield networks, and investigate their impact on the performance of the network.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data availability

We do not analyze or generate any datasets, because our work proceeds within a theoretical and mathematical approach - one can obtain the relevant materials from the references below.

Code Availability

Code for Light Up puzzle solving is available at https://github.com/carlacursino/lujl.git for review.

References

  1. Hopfield JJ. Neural networks and physical systems with emergent collective computational abilities. Proc Natl Acad Sci. 1982;79:2554–8.

    Article  MathSciNet  Google Scholar 

  2. Hopfield JJ. Neurons with graded response have collective computational properties like those of two-state neurons. Proc Natl Acad Sci. 1984;81(1084):3088–92.

    Article  Google Scholar 

  3. Hopfield JJ, Tank DW. Neural computation of decisions in optimization problems. Biol Cybern. 1985;52:141–52.

    Article  Google Scholar 

  4. Wilson GV, Pawley GS. On the stability of the TSP algorithm of Hopfield and Tank. Biol Cybern. 1988;58:63–70.

    Article  Google Scholar 

  5. Aiyer SVB, Niranjan M, Fallside F. A theoretical investigation into the performance of the Hopfield model. IEEE Trans Neural Netw. 1990;1:204–15.

    Article  Google Scholar 

  6. Andrew HG, Aiyer SVB, Prager RW. An analytical framework for optimizing neural networks. Neural Netw. 1993;6:79–97.

    Article  Google Scholar 

  7. Gee AH. Problem Solving with Optimization Networks, Ph.D. thesis, Queen’s College. Cambridge, UK: Cambridge University; 1993.

  8. Chu PC. A neural network for solving optimization problems with linear equality constraints. Proc IEEE Int Jt Conf Neural Netw. 1992;2:272–7.

    Google Scholar 

  9. Kate AS. Neural networks for combinatorial optimization: a review of more than a decade of research. J Comput. 1999;11(1):15–34.

    MathSciNet  Google Scholar 

  10. McPhail BP. The complexity of puzzles: NP-completeness results for Nurikabe and Minesweeper. Reed College, Undergraduate Thesis. 2003.

  11. Yen S-J, Chiu S-Y. A Simple and Rapid Lights-up Solver. In: International Conference on Technologies and Applications of Artificial Intelligence, 2010.

  12. Shrivastava Y, Dasgupta S, Reddy SM. Guaranteed convergence in a class of Hopfield networks. IEEE Trans Neural Netw. 1992;3(6):951–61.

    Article  Google Scholar 

  13. Emílio GOG, Salcedo-Sanz S, Angel MPB, Antônio PF. A Hybrid Hopfield Neural Network-Genetic Algorithm Approach for the Light Up Puzzle. In: IEEE Congress of Evolutionary Computation. 2007. pp. 1403–7.

  14. Banerjee S, Anindya R. Linear algebra and matrix analysis for statistics. CRC Press; 2014.

    Book  Google Scholar 

  15. Laub AJ. Matrix analysis for scientists and engineers. SIAM; 2005.

    Google Scholar 

  16. Potvin J-Y, Smith KA. Handbook of meta heuristic. Kluwer Academic Publishers; 2003. p. 429–55.

    Book  Google Scholar 

  17. Leondes CT. Implementation techniques: neural network systems techniques and applications. Academic Press; 1998.

    Google Scholar 

  18. Metropolis N, Rosenbluth A, Rosenbluth A, Teller A, Teller E. Equation of state calculations by fast computing machines. J Chem Phys. 1953;21:1087–92.

    Article  Google Scholar 

  19. Light Up puzzle available on https://www.minijuegos.com/juegos/jugar.php?id=3502.

  20. Light Up puzzle available at https://www.puzzle-light-up.com/.

  21. Sun L, Browning J, Perera R. Shedding some light on light-up with artificial intelligence, 2021. ArXiv Preprint ArXiv:2107.10429.

Download references

Funding

This study was funded by CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Higher Education Personnel Improvement Coordination) (Grant number 88887.624622/2021-00).

Author information

Authors and Affiliations

Authors

Contributions

CC and LAVD conceived of the presented idea, CC developed the theory, coded the solution, performed the computations and LAVD supervised the findings of this work.

Corresponding author

Correspondence to Carla Cursino.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Consent to Participate

This article does not contain any studies with individuals.

Consent for Publication

This article does not contain any studies with individuals.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix A Moore–Pensore Pseudo Inverse Matrix

There is a generalization for the inverse of a matrix. The Moore–Penrose matrix \(\varvec{A}^+\) exists for any matrix and is unique. See detailed theory in the linear algebra books of Laub [15] and Banerjee [14].

Let \(\varvec{A}\) be a matrix whose inverse is \(\varvec{A}^{-1}\) then

$$\begin{aligned} \varvec{A}^+=\varvec{A}^{-1} \end{aligned}$$
(A1)

If \(\varvec{A}\) does not have an inverse, either because the determinant is null or because it is not square, then the pseudo-inverse is defined as

$$\begin{aligned} \varvec{A}^+=(\varvec{A}^T\varvec{A})^{-1}\varvec{A}^T\qquad \text {or}\qquad \varvec{A}^+=\varvec{A}^T(\varvec{A}\varvec{A}^ T)^{-1} \end{aligned}$$
(A2)

If \(\det (\varvec{A}\varvec{A}^T)= 0\) and \(\det (\varvec{A}^T\varvec{A})= 0\) then

$$\begin{aligned} \varvec{A}^+=\lim _{\epsilon \rightarrow 0}(\varvec{A}^T\varvec{A}+\epsilon ^2 \varvec{I})^{-1}\varvec{A}^T= \lim _{\epsilon \rightarrow 0}\varvec{A}^T(\varvec{A}\varvec{A}^T+\epsilon ^2 \varvec{I})^{-1} \end{aligned}$$
(A3)

A.1 Properties

The Moore–Penrose matrix \({\varvec{A}}^+\) satisfies the following properties

$$\begin{aligned} {\varvec{A}}^+{\varvec{A}}{\varvec{A}}^+&={\varvec{A}}^+ \end{aligned}$$
(A4)
$$\begin{aligned} {\varvec{A}}{\varvec{A}}^+{\varvec{A}}&={\varvec{A}} \end{aligned}$$
(A5)
$$\begin{aligned} ({\varvec{A}}{\varvec{A}}^+)^T&={\varvec{A}}{\varvec{A}}^+ \end{aligned}$$
(A6)
$$\begin{aligned} ({\varvec{A}}^+{\varvec{A}})^T&={\varvec{A}}^+{\varvec{A}} \end{aligned}$$
(A7)
$$\begin{aligned} ({\varvec{A}}^+)^+&= {\varvec{A}} \end{aligned}$$
(A8)

A.2 Projection Matrices

Given a matrix \(\varvec{A}\), we have the projections matrices onto the four fundamental subspaces of linear algebra, see Banerjee pag. 273 [14]

  • \(\varvec{P}=\varvec{A}^+\varvec{A}\): Projection matrix onto the row space of \(\varvec{A}\).

  • \(\varvec{P}_{{{\mathcal {N}}}}=\varvec{I}-\varvec{A}^+\varvec{A}\): Projection matrix onto the null space of \(\varvec{A}\).

  • \(\varvec{Q}=\varvec{A}\varvec{A}^+\): Projection matrix onto the column space of \(\varvec{A}\).

  • \(\varvec{Q}_{{{\mathcal {N}}}}=\varvec{I}-\varvec{A}\varvec{A}^+\): Projection matrix onto the left null space of \(\varvec{A}\).

A.3 Particular Cases

The pseudo-inverse of a scalar \(\alpha\) is (scalar is a matrix \(1\times 1\))

$$\begin{aligned} \alpha ^+={\left\{ \begin{array}{ll} \alpha ^{-1} &{}\qquad \text {if}\qquad \alpha \ne 0\\ 0 &{}\qquad \text {if}\qquad \alpha =0 \end{array}\right. } \end{aligned}$$
(A9)

Pseudo-inverse of a vector (row or column)

$$\begin{aligned} \varvec{v}^+=\frac{\varvec{v}^T}{\varvec{v}\cdot \varvec{v}} \end{aligned}$$
(A10)

Pseudo-inverse of a non square matrix

$$\begin{aligned} A=\begin{pmatrix} 1 &{} 2 &{} 3\\ 2 &{} 1 &{} 3 \end{pmatrix}\qquad \therefore \qquad A^+=\begin{pmatrix} -{{4}\over {9}}&{}{{5}\over {9}}\\ {{5}\over {9}}&{}-{{4}\over {9}}\\ {{ 1 }\over {9}}&{}{{1}\over {9}} \end{pmatrix} \end{aligned}$$
(A11)

Pseudo-inverse of a singular matrix

$$\begin{aligned} \varvec{A}=\begin{pmatrix} 1 &{} 1\\ 1 &{} 1\end{pmatrix}\qquad \therefore \qquad \varvec{A}^+=\begin{pmatrix} \frac{1}{4} &{} \frac{1}{4}\\ \frac{1}{4} &{} \frac{1}{4}\end{pmatrix} \end{aligned}$$
(A12)

Modern programing languages, such as julia, python and matlab have implementation for the Moore–Penrose pseudo inverse matrix. In all of these programing languages the pseudo inverse is called with the routine pinv.

Appendix B Projection Matrix into Valid Subspace

In this appendix, we will give a simple geometric interpretation of the projection of vectors \(\varvec{v}\) onto the null space \({{\mathcal {N}}}\) of the matrix \(\varvec{A}\) obtained from the equation of constraints. The component of \(\varvec{v}\) in this subspace is denoted by \(\varvec{v}_{{{\mathcal {N}}}}\). There is also a subspace \({{\mathcal {R}}}\) of vectors orthogonal to this subspace which is the row space of \(\varvec{A}\). The component of \(\varvec{v}\) in this subspace is denoted by \(\varvec{v}_{{{\mathcal {R}}}}\). Consider the constraint equation

$$\begin{aligned} x+2y=4, \end{aligned}$$
(B13)

which can be put in matrix form

$$\begin{aligned} \begin{pmatrix} 1&2 \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix}= \begin{pmatrix} 4\end{pmatrix}. \end{aligned}$$
(B14)

In a more compact form, this can be rewritten as

$$\begin{aligned} {\varvec{A}\varvec{x}}=\varvec{b}, \end{aligned}$$
(B15)

where

$$\begin{aligned} {\varvec{A}}=\begin{pmatrix} 1&2 \end{pmatrix} \qquad \text {and}\qquad {\varvec{b}}=\begin{pmatrix} 4\end{pmatrix}. \end{aligned}$$
(B16)
Fig. 4
figure 4

The Illustration of the straight line representing the equation of constraint, projected vectors \(\varvec{v}_{{{\mathcal {N}}}}\), \(\varvec{v}_{{{\mathcal {R}}}}\), and the solutions \(\hat{\varvec{b}}\) and \(\hat{\varvec{x}}\)

Figure  4 shows the straight line which is the equation of constraint. One solution of this equation is

$$\begin{aligned} \hat{\varvec{b}}=\varvec{A}^+\varvec{b}, \end{aligned}$$
(B17)

where \(\varvec{A}^+\) is the Moore–Penrose pseudo inverse of \(\varvec{A}\). In our case,

$$\begin{aligned} \varvec{A}^+=\begin{pmatrix} 0.2 \\ 0.4 \end{pmatrix} \qquad \text {and}\qquad \hat{\varvec{b}}=\begin{pmatrix} 0.8 \\ 1.6 \end{pmatrix}. \end{aligned}$$
(B18)

To obtain other solution, construct the matrix \(\varvec{P}\) that project any vector \(\varvec{v}\) into the orthogonal subspace \({{\mathcal {R}}}\), given by

$$\begin{aligned} {\varvec{P}} = \varvec{A}^+\varvec{A} \qquad \text {in our case}\qquad \varvec{P} = \begin{pmatrix} 0.2 &{} 0.4\\ 0.4 &{} 0.8\end{pmatrix} \end{aligned}$$
(B19)

and the matrix \({\varvec{I}}-\varvec{P}\) that project any vector \(\varvec{v}\) into the subspace \({{\mathcal {N}}}\), given in by

$$\begin{aligned} \varvec{P}=\begin{pmatrix} 0.8 &{} -0.4\\ -0.4 &{} 0.2\end{pmatrix}. \end{aligned}$$
(B20)

For a particular case of \(\varvec{v}=\begin{pmatrix} 2 \\ 1 \end{pmatrix}\) we have

$$\begin{aligned} \varvec{v}_{{{\mathcal {R}}}}=\varvec{P}\varvec{v}=\begin{pmatrix} 0.8 \\ 1.6 \end{pmatrix} \end{aligned}$$
(B21)

and

$$\begin{aligned} \varvec{v}_{{{\mathcal {N}}}}=(\varvec{I}-\varvec{P})\varvec{v}=\begin{pmatrix} 1.2 \\ -0.6 \end{pmatrix}. \end{aligned}$$
(B22)

Then for this particular vector, the other solution satisfying the constraint is

$$\begin{aligned} \hat{\varvec{x}}=\hat{\varvec{b}}+\varvec{v}_{{{\mathcal {N}}}} \qquad \text {which in our case is}\qquad \hat{\varvec{x}}=\begin{pmatrix} 2 \\ 1 \end{pmatrix}. \end{aligned}$$
(B23)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cursino, C., Dias, L.A.V. Hybrid Hopfield Neural Network. SN COMPUT. SCI. 5, 232 (2024). https://doi.org/10.1007/s42979-023-02575-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-023-02575-6

Keywords

Navigation