Abstract
Hopfield and Tank have shown that a neural network can find solutions for complex optimization problems, although it can be trapped in a local minimum of the objective function returning a suboptimal solution. When the problem has constraints they can be added to the objective function as penalty terms using Lagrange multipliers. In this paper, we introduce an approach inspired by the work of Andrew, Chu, and Gee to implement a neural network to obtain solutions satisfying the linear equality constraints using the Moore–Penrose pseudo inverse matrix to construct a projection matrix to send any configuration to the subspace of configuration space that satisfies all the constraints. The objective function of the problem is modified to include Lagrange multipliers terms for the equations of constraints. Furthermore, we have found that such a condition makes the network converge to a set of stable states even if some diagonal elements of the weight matrix is negative. If after several steps the network does not converge to a stable state, we just solve the problem using simulated annealing that significantly outperforms hill climbing, feed-forward neural network and convolutional neural network. We use this technique to solve the NP-hard Light Up puzzle. Hopfield neural networks are widely used for pattern recognition and optimization tasks. However, the standard Hopfield network model uses non-negative weights between neurons, which can limit its performance in certain situations. By introducing negative weights, the network can potentially learn more complex and nuanced patterns, and exhibit improved convergence properties. Thus, the motivation for the article “Hybrid Hopfield Neural Network” is to explore the benefits of incorporating negative weights into Hopfield networks, and investigate their impact on the performance of the network.



Similar content being viewed by others
Data availability
We do not analyze or generate any datasets, because our work proceeds within a theoretical and mathematical approach - one can obtain the relevant materials from the references below.
Code Availability
Code for Light Up puzzle solving is available at https://github.com/carlacursino/lujl.git for review.
References
Hopfield JJ. Neural networks and physical systems with emergent collective computational abilities. Proc Natl Acad Sci. 1982;79:2554–8.
Hopfield JJ. Neurons with graded response have collective computational properties like those of two-state neurons. Proc Natl Acad Sci. 1984;81(1084):3088–92.
Hopfield JJ, Tank DW. Neural computation of decisions in optimization problems. Biol Cybern. 1985;52:141–52.
Wilson GV, Pawley GS. On the stability of the TSP algorithm of Hopfield and Tank. Biol Cybern. 1988;58:63–70.
Aiyer SVB, Niranjan M, Fallside F. A theoretical investigation into the performance of the Hopfield model. IEEE Trans Neural Netw. 1990;1:204–15.
Andrew HG, Aiyer SVB, Prager RW. An analytical framework for optimizing neural networks. Neural Netw. 1993;6:79–97.
Gee AH. Problem Solving with Optimization Networks, Ph.D. thesis, Queen’s College. Cambridge, UK: Cambridge University; 1993.
Chu PC. A neural network for solving optimization problems with linear equality constraints. Proc IEEE Int Jt Conf Neural Netw. 1992;2:272–7.
Kate AS. Neural networks for combinatorial optimization: a review of more than a decade of research. J Comput. 1999;11(1):15–34.
McPhail BP. The complexity of puzzles: NP-completeness results for Nurikabe and Minesweeper. Reed College, Undergraduate Thesis. 2003.
Yen S-J, Chiu S-Y. A Simple and Rapid Lights-up Solver. In: International Conference on Technologies and Applications of Artificial Intelligence, 2010.
Shrivastava Y, Dasgupta S, Reddy SM. Guaranteed convergence in a class of Hopfield networks. IEEE Trans Neural Netw. 1992;3(6):951–61.
Emílio GOG, Salcedo-Sanz S, Angel MPB, Antônio PF. A Hybrid Hopfield Neural Network-Genetic Algorithm Approach for the Light Up Puzzle. In: IEEE Congress of Evolutionary Computation. 2007. pp. 1403–7.
Banerjee S, Anindya R. Linear algebra and matrix analysis for statistics. CRC Press; 2014.
Laub AJ. Matrix analysis for scientists and engineers. SIAM; 2005.
Potvin J-Y, Smith KA. Handbook of meta heuristic. Kluwer Academic Publishers; 2003. p. 429–55.
Leondes CT. Implementation techniques: neural network systems techniques and applications. Academic Press; 1998.
Metropolis N, Rosenbluth A, Rosenbluth A, Teller A, Teller E. Equation of state calculations by fast computing machines. J Chem Phys. 1953;21:1087–92.
Light Up puzzle available on https://www.minijuegos.com/juegos/jugar.php?id=3502.
Light Up puzzle available at https://www.puzzle-light-up.com/.
Sun L, Browning J, Perera R. Shedding some light on light-up with artificial intelligence, 2021. ArXiv Preprint ArXiv:2107.10429.
Funding
This study was funded by CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Higher Education Personnel Improvement Coordination) (Grant number 88887.624622/2021-00).
Author information
Authors and Affiliations
Contributions
CC and LAVD conceived of the presented idea, CC developed the theory, coded the solution, performed the computations and LAVD supervised the findings of this work.
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare that they have no conflict of interest.
Ethical Approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Consent to Participate
This article does not contain any studies with individuals.
Consent for Publication
This article does not contain any studies with individuals.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix A Moore–Pensore Pseudo Inverse Matrix
There is a generalization for the inverse of a matrix. The Moore–Penrose matrix \(\varvec{A}^+\) exists for any matrix and is unique. See detailed theory in the linear algebra books of Laub [15] and Banerjee [14].
Let \(\varvec{A}\) be a matrix whose inverse is \(\varvec{A}^{-1}\) then
If \(\varvec{A}\) does not have an inverse, either because the determinant is null or because it is not square, then the pseudo-inverse is defined as
If \(\det (\varvec{A}\varvec{A}^T)= 0\) and \(\det (\varvec{A}^T\varvec{A})= 0\) then
A.1 Properties
The Moore–Penrose matrix \({\varvec{A}}^+\) satisfies the following properties
A.2 Projection Matrices
Given a matrix \(\varvec{A}\), we have the projections matrices onto the four fundamental subspaces of linear algebra, see Banerjee pag. 273 [14]
-
\(\varvec{P}=\varvec{A}^+\varvec{A}\): Projection matrix onto the row space of \(\varvec{A}\).
-
\(\varvec{P}_{{{\mathcal {N}}}}=\varvec{I}-\varvec{A}^+\varvec{A}\): Projection matrix onto the null space of \(\varvec{A}\).
-
\(\varvec{Q}=\varvec{A}\varvec{A}^+\): Projection matrix onto the column space of \(\varvec{A}\).
-
\(\varvec{Q}_{{{\mathcal {N}}}}=\varvec{I}-\varvec{A}\varvec{A}^+\): Projection matrix onto the left null space of \(\varvec{A}\).
A.3 Particular Cases
The pseudo-inverse of a scalar \(\alpha\) is (scalar is a matrix \(1\times 1\))
Pseudo-inverse of a vector (row or column)
Pseudo-inverse of a non square matrix
Pseudo-inverse of a singular matrix
Modern programing languages, such as julia, python and matlab have implementation for the Moore–Penrose pseudo inverse matrix. In all of these programing languages the pseudo inverse is called with the routine pinv.
Appendix B Projection Matrix into Valid Subspace
In this appendix, we will give a simple geometric interpretation of the projection of vectors \(\varvec{v}\) onto the null space \({{\mathcal {N}}}\) of the matrix \(\varvec{A}\) obtained from the equation of constraints. The component of \(\varvec{v}\) in this subspace is denoted by \(\varvec{v}_{{{\mathcal {N}}}}\). There is also a subspace \({{\mathcal {R}}}\) of vectors orthogonal to this subspace which is the row space of \(\varvec{A}\). The component of \(\varvec{v}\) in this subspace is denoted by \(\varvec{v}_{{{\mathcal {R}}}}\). Consider the constraint equation
which can be put in matrix form
In a more compact form, this can be rewritten as
where
Figure 4 shows the straight line which is the equation of constraint. One solution of this equation is
where \(\varvec{A}^+\) is the Moore–Penrose pseudo inverse of \(\varvec{A}\). In our case,
To obtain other solution, construct the matrix \(\varvec{P}\) that project any vector \(\varvec{v}\) into the orthogonal subspace \({{\mathcal {R}}}\), given by
and the matrix \({\varvec{I}}-\varvec{P}\) that project any vector \(\varvec{v}\) into the subspace \({{\mathcal {N}}}\), given in by
For a particular case of \(\varvec{v}=\begin{pmatrix} 2 \\ 1 \end{pmatrix}\) we have
and
Then for this particular vector, the other solution satisfying the constraint is
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Cursino, C., Dias, L.A.V. Hybrid Hopfield Neural Network. SN COMPUT. SCI. 5, 232 (2024). https://doi.org/10.1007/s42979-023-02575-6
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s42979-023-02575-6