Training Fully Connected Neural Networks is ∃ℝ-Complete

by Daniel Bertschinger, Christopher Hertrich, Paul Jungeblut, Tillmann Miltzow, and Simon Weber

Theory of Computing, Volume 22(6), pp. 1-48, 2026

Bibliography with links to cited articles

[1]   Mikkel Abrahamsen: Covering polygons is even harder. In Proc. 62nd FOCS, pp. 375–386. IEEE Comp. Soc., 2021. [doi:10.1109/FOCS52979.2021.00045]

[2]   Mikkel Abrahamsen, Anna Adamaszek, and Tillmann Miltzow: The art gallery problem is R-complete. J. ACM, 69(1):4:1–70, 2022. [doi:10.1145/3486220]

[3]   Mikkel Abrahamsen, Linda Kleist, and Tillmann Miltzow: Training neural networks is R-complete. In Proc. 34th Adv. Neural Info. Proc. Sys. (NeurIPS’21), pp. 18293–18306. Curran Assoc., 2021. Available at NeurIPS.

[4]   Mikkel Abrahamsen and Tillmann Miltzow: Dynamic toolbox for ERTINV, 2019. [arXiv:1912.08674]

[5]   Reyan Ahmed, Felice de Luca, Sabin Devkota, Stephen Kobourov, and Mingwei Li: Multicriteria scalable graph drawing via stochastic gradient descent, (SGD)2. IEEE Trans. Visualiz. & Computer Graphics, 28(6):2388–2399, 2022. Open-access link at the NSF PAR. [doi:10.1109/TVCG.2022.3155564]

[6]   Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee: Understanding deep neural networks with rectified linear units. In Int. Conf. Learning Representations (ICLR’18). ICLR, 2018. Available at OpenReview.

[7]   Egor Bakaev, Florestan Brunck, Christoph Hertrich, Jack Stade, and Amir Yehudayoff: Better neural network expressivity: Subdividing the simplex. In Proc. 58th STOC, pp. 500–507. ACM Press, 2026. [doi:10.1145/3798129.3800768]

[8]   Ainesh Bakshi, Rajesh Jayaram, and David P. Woodruff: Learning two layer rectified neural networks in polynomial time. In Proc. 32nd Ann. Conf. on Learning Theory (COLT’19), pp. 195–268. MLR Press, 2019. Available at PMLR.

[9]   Marie Louisa Tølbøll Berthelsen and Kristoffer Arnsfelt Hansen: On the computational complexity of decision problems about multi-player Nash equilibria. Theory Computing Sys., 66(3):519–545, 2022. [doi:10.1007/s00224-022-10080-1]

[10]   Daniel Bertschinger, Christoph Hertrich, Paul Jungeblut, Tillmann Miltzow, and Simon Weber: Training fully connected neural networks is R-complete. In Proc. 36th Adv. Neural Info. Proc. Sys. (NeurIPS’23), pp. 36222–36237. Curran Assoc., 2023. Available at NeurIPS.

[11]   Daniel Bienstock, Gonzalo Muñoz, and Sebastian Pokutta: Principled deep neural network training through linear programming. Discr. Optimization, 49(100795):1–37, 2023. [doi:10.1016/j.disopt.2023.100795]

[12]   Vittorio Bilò and Marios Mavronicolas: R-complete decision problems about (symmetric) Nash equilibria in (symmetric) multi-player games. ACM Trans. Econ. Comput., 9(3):14:1–25, 2021. [doi:10.1145/3456758]

[13]   Manon Blanc and Kristoffer Arnsfelt Hansen: Computational complexity of multi-player evolutionarily stable strategies. In Proc. 16th Comp. Sci. Symp. in Russia (CSR’21), pp. 1–17. Springer, 2021. [doi:10.1007/978-3-030-79416-3_1]

[14]   Lenore Blum, Mike Shub, and Steve Smale: On a theory of computation and complexity over the real numbers: NP-completeness, recursive functions and universal machines. Bull. AMS, 21(1):1–46, 1989. [doi:10.1090/S0273-0979-1989-15750-9]

[15]   Digvijay Boob, Santanu S. Dey, and Guanghui Lan: Complexity of training ReLU neural network. Discr. Optimization, 44(1):1–16, 2022. [doi:10.1016/j.disopt.2020.100620]

[16]   Cornelius Brand, Robert Ganian, and Mathis Rocton: New complexity-theoretic frontiers of tractability for neural network training. In Proc. 36th Adv. Neural Info. Proc. Sys. (NeurIPS’23), pp. 56456–56468. Curran Assoc., 2023. Available at NeurIPS.

[17]   Alon Brutzkus and Amir Globerson: Globally optimal gradient descent for a ConvNet with Gaussian inputs. In Proc. 34th Internat. Conf. Machine Learning (ICML’17), pp. 605–614. MLR Press, 2017. Available at PMLR.

[18]   Sébastien Bubeck and Mark Sellke: A universal law of robustness via isoperimetry. J. ACM, 70(2):10:1–18, 2023. [doi:10.1145/3578580]

[19]   Peter Bürgisser and Felipe Cucker: Exotic quantifiers, complexity classes, and complete problems. Found. Computational Math., 9(2):135–170, 2009. [doi:10.1007/s10208-007-9006-9]

[20]   John F. Canny: Some algebraic and geometric computations in PSPACE. In Proc. 20th STOC, pp. 460–467. ACM Press, 1988. [doi:10.1145/62212.62257]

[21]   Jean Cardinal, Stefan Felsner, Tillmann Miltzow, Casey Tompkins, and Birgit Vogtenhuber: Intersection graphs of rays and grounded segments. J. Graph Algor. & Appl., 22(2):273–295, 2018. [doi:10.7155/jgaa.00470]

[22]   Sitan Chen, Aravind Gollakota, Adam R. Klivans, and Raghu Meka: Hardness of noise-free learning for two-hidden-layer neural networks. In Proc. 35th Adv. Neural Info. Proc. Sys. (NeurIPS’22), pp. 10709–10724. Curran Assoc., 2022. Available at NeurIPS.

[23]   Sitan Chen, Adam R. Klivans, and Raghu Meka: Learning deep ReLU networks is fixed-parameter tractable. In Proc. 62nd FOCS, pp. 696–707. IEEE Comp. Soc., 2021. [doi:10.1109/FOCS52979.2021.00073]

[24]   Dmitry Chistikov, Stefan Kiefer, Ines Marusic, Mahsa Shirmohammadi, and James Worrell: On restricted nonnegative matrix factorization. In Proc. 43rd Internat. Colloq. on Automata, Languages, and Programming (ICALP’16), pp. 103:1–14. Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, 2016. [doi:10.4230/LIPIcs.ICALP.2016.103]

[25]   George Cybenko: Approximation by superpositions of a sigmoidal function. Math. of Control, Signals & Systems, 2(4):303–314, 1989. [doi:10.1007/BF02551274]

[26]   Julian D’Costa, Engel Lefaucheux, Eike Neumann, Joël Ouaknine, and James Worrel: On the complexity of the escape problem for linear dynamical systems over compact semialgebraic sets. In Proc. Internat. Symp. Math. Foundations of Comp. Sci. (MFCS’21), pp. 33:1–21. Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, 2021. [doi:10.4230/LIPIcs.MFCS.2021.33]

[27]   Pedro J. de Rezende, Cid C. de Souza, Stephan Friedrichs, Michael Hemmer, Alexander Kröller, and Davi C. Tozoni: Engineering art galleries. In Lasse Kliemann and Peter Sanders, editors, Algorithm Engineering: Selected Results and Surveys, volume 9220 of LNCS, pp. 379–417. Springer, 2016. [doi:10.1007/978-3-319-49487-6_12, arXiv:1410.8720]

[28]   Argyrios Deligkas, John Fearnley, Themistoklis Melissourgos, and Paul G. Spirakis: Approximating the existential theory of the reals. J. Comput. System Sci., 125:106–128, 2022. [doi:10.1016/j.jcss.2021.11.002]

[29]   Steffen Dereich and Sebastian Kassing: On minimal representations of shallow ReLU networks. Neural Networks, 148:121–128, 2022. [doi:10.1016/j.neunet.2022.01.006]

[30]   Santanu S. Dey, Guany Wang, and Yao Xie: Approximation algorithms for training one-node ReLU neural networks. IEEE Trans. Signal Processing, 68:6696–6706, 2020. [doi:10.1109/TSP.2020.3039360]

[31]   Ilias Diakonikolas, Surbhi Goel, Sushrut Karmalkar, Adam R. Klivans, and Mahdi Soltanolkotabi: Approximation schemes for ReLU regression. In Proc. 33rd Ann. Conf. on Learning Theory (COLT’20), pp. 1452–1485. MLR Press, 2020. Available at PMLR.

[32]   Michael G. Dobbins, Linda Kleist, Tillmann Miltzow, and Paweł Rzążewski: Completeness for the complexity class ∀∃R and area-universality. Discr. Comput. Geom., 70(1):154–188, 2023. [doi:10.1007/s00454-022-00381-0]

[33]   Ronen Eldan and Ohad Shamir: The power of depth for feedforward neural networks. In Proc. 29th Ann. Conf. on Learning Theory (COLT’16), pp. 907–940. Springer, 2016. Available at PMLR.

[34]   Jeff Erickson, Ivor van der Hoog, and Tillmann Miltzow: Smoothing the gap between NP and R. SIAM J. Comput., 53(6):102–138, 2024. [doi:10.1137/20M1385287]

[35]   Vincent Froese and Christoph Hertrich: Training neural networks is NP-hard in fixed dimension. In Proc. 36th Adv. Neural Info. Proc. Sys. (NeurIPS’23), pp. 44039–44049. Curran Assoc., 2023. Available at NeurIPS.

[36]   Vincent Froese, Christoph Hertrich, and Rolf Niedermeier: The computational complexity of ReLU network training parameterized by data dimensionality. J. Artif. Intell. Res., 74:1775–1790, 2022. [doi:10.1613/jair.1.13547]

[37]   Jugal Garg, Ruta Mehta, Vijay V. Vazirani, and Sadra Yazdanbod: R-completeness for decision versions of multi-player (symmetric) Nash equilibria. ACM Trans. Econ. Comput., 6(1):1:1–23, 2018. [doi:10.1145/3175494]

[38]   Xavier Glorot, Antoine Bordes, and Yoshua Bengio: Deep sparse rectifier neural networks. In Proc. 14th Internat. Conf. Artificial Intelligence and Statistics (AISTATS’11), pp. 315–323. MLR Press, 2011. Available at PMLR.

[39]   Surbhi Goel, Varun Kanade, Adam R. Klivans, and Justin Thaler: Reliably learning the ReLU in polynomial time. In Proc. 30th Ann. Conf. on Learning Theory (COLT’17), pp. 1004–1042. MLR Press, 2017. Available at PMLR.

[40]   Surbhi Goel and Adam R. Klivans: Learning neural networks with two nonlinear layers in polynomial time. In Proc. 32nd Ann. Conf. on Learning Theory (COLT’19), pp. 1470–1499. MLR Press, 2019. Available at PMLR.

[41]   Surbhi Goel, Adam R. Klivans, Pasin Manurangsi, and Daniel Reichman: Tight hardness results for training depth-2 ReLU networks. In Proc. 12th Innovations in Theoret. Comp. Sci. Conf. (ITCS’21), pp. 22:1–14. Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, 2021 (virtual conference). [doi:10.4230/LIPIcs.ITCS.2021.22]

[42]   Surbhi Goel, Adam R. Klivans, and Raghu Meka: Learning one convolutional layer with overlapping patches. In Proc. 35th Internat. Conf. Machine Learning (ICML’18), pp. 1783–1791. MLR Press, 2018. Available at PMLR.

[43]   Ian Goodfellow, Yoshua Bengio, and Aaron Courville: Deep Learning. MIT Press, 2016. Available at MIT.

[44]   Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J. Cree: Regularisation of neural networks by enforcing Lipschitz continuity. Machine Learning, 110(2):393–416, 2021. [doi:10.1007/s10994-020-05929-w]

[45]   Christian Haase, Christoph Hertrich, and Georg Loho: Lower bounds on the depth of integral ReLU neural networks via lattice polytopes. In Int. Conf. Learning Representations (ICLR’23). ICLR, 2023. Available at OpenReview.

[46]   Boris Hanin: Universal function approximation by deep neural nets with bounded width and ReLU activations. Mathematics, 7(10):992:1–9, 2019. [doi:10.3390/math7100992]

[47]   Boris Hanin and David Rolnick: Complexity of linear regions in deep networks. In Proc. 36th Internat. Conf. Machine Learning (ICML’19), pp. 2596–2604. MLR Press, 2019. Available at PMLR.

[48]   Boris Hanin and Mark Sellke: Approximating continuous functions by ReLU nets of minimal width, 2018. [arXiv:1710.11278]

[49]   Simon B. Hengeveld and Tillmann Miltzow: A practical algorithm with performance guarantees for the Art Gallery problem. Discr. Math. & Theor. Comput. Sci., 25(2):1–59, 2023. Preliminary version in SoCG’21. [doi:10.46298/DMTCS.9225]

[50]   Christoph Hertrich, Amitabh Basu, Marco Di Summa, and Martin Skutella: Towards lower bounds on the depth of ReLU neural networks. SIAM J. Discr. Math., 37(2):997–1029, 2023. [doi:10.1137/22M1489332]

[51]   Christoph Hertrich and Leon Sering: ReLU neural networks of polynomial size for exact maximum flow computation. Math. Programming, 210(1):377–406, 2025. [doi:10.1007/s10107-024-02096-x]

[52]   Kurt Hornik: Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2):251–257, 1991. [doi:10.1016/0893-6080(91)90009-T]

[53]   Joey Huchette, Gonzalo Muñoz, Tiago Serra, and Calvin Tsay: When deep learning meets polyhedral theory: A survey, 2023. [arXiv:2305.00241]

[54]   Paul Jungeblut: On the complexity of Lombardi graph drawing. In Graph Drawing & Network Visualization (GD’23), pp. 180–194. Springer, 2023. [doi:10.1007/978-3-031-49272-3_13]

[55]   Paul Jungeblut, Linda Kleist, and Tillmann Miltzow: The complexity of the Hausdorff distance. Discr. Comput. Geom., 71(1):177–213, 2024. [doi:10.1007/s00454-023-00562-5]

[56]   Ross Kang and Tobias Müller: Sphere and dot product representations of graphs. Discr. Comput. Geom., 47(3):548–569, 2012. [doi:10.1007/s00454-012-9394-8]

[57]   Sammy Khalife, Hongyu Cheng, and Amitabh Basu: Neural networks with linear threshold activations: Structure and algorithms. Math. Programming, 206:333–356, 2024. [doi:10.1007/s10107-023-02016-5]

[58]   Jan Kratochvíl and Jiří Matoušek: Intersection graphs of segments. J. Combin. Theory–B, 62(2):289–315, 1994. [doi:10.1006/jctb.1994.1071]

[59]   Shiyu Liang and Rayadurgam Srikant: Why deep neural networks for function approximation? In Int. Conf. Learning Representations (ICLR’17). ICLR, 2017. Available at OpenReview.

[60]   Anna Lubiw, Tillmann Miltzow, and Debajyoti Mondal: The complexity of drawing a graph in a polygonal region. J. Graph Algor. & Appl., 26(4):421–446, 2022. [doi:10.7155/jgaa.00602]

[61]   Jiří Matoušek: Intersection graphs of segments and R, 2014. [arXiv:1406.2636]

[62]   Colin McDiarmid and Tobias Müller: Integer realizations of disk and segment graphs. J. Combin. Theory–B, 103(1):114–143, 2013. [doi:10.1016/j.jctb.2012.09.004]

[63]   Tillmann Miltzow and Reinier F. Schmiermann: On classifying continuous constraint satisfaction problems. TheoretiCS, 3(10):1–54, 2024. Preliminary version in FOCS’22. [doi:10.46298/THEORETICS.24.10]

[64]   Nikolai E. Mnëv: The universality theorems on the classification problem of configuration varieties and convex polytopes varieties. In Oleg Y. Viro and Anatoly M. Vershik, editors, Topology and Geometry — Rohlin Seminar, volume 1346 of Lecture Notes in Mathematics, pp. 527–543. Springer, 1988. [doi:10.1007/BFb0082792]

[65]   Guido Montúfar, Yue Ren, and Leon Zhang: Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sums. SIAM J. Appl. Algebra Geom., 6(4):618–649, 2022. [doi:10.1137/21M1413699]

[66]   Guido F. Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio: On the number of linear regions of deep neural networks. In Proc. 27th Adv. Neural Info. Proc. Sys. (NIPS’14), pp. 2924–2932. Curran Assoc., 2014. Available at NeurIPS.

[67]   Anirbit Mukherjee and Amitabh Basu: Lower bounds over Boolean inputs for deep neural networks with ReLU gates, 2017. [arXiv:1711.03073]

[68]   Quynh Nguyen, Mahesh Chandra Mukkamala, and Matthias Hein: Neural networks should be wide enough to learn disconnected decision regions. In Proc. 35th Internat. Conf. Machine Learning (ICML’18), pp. 3740–3749. MLR Press, 2018. Available at PMLR.

[69]   Razvan Pascanu, Guido Montúfar, and Yoshua Bengio: On the number of inference regions of deep feed forward networks with piece-wise linear activations. In Int. Conf. Learning Representations (ICLR’14). ICLR, 2014. Available at OpenReview.

[70]   Grant O. Passmore and Paul B. Jackson: Combined decision techniques for the existential theory of the reals. In Internat. Conf. Intell. Computer Math. (CICM’09), pp. 122–137. Springer, 2009. [doi:10.1007/978-3-642-02614-0_14]

[71]   Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl Dickstein: On the expressive power of deep neural networks. In Proc. 34th Internat. Conf. Machine Learning (ICML’17), pp. 2847–2854. MLR Press, 2017. Available at PMLR.

[72]   Daniel Richardson: Some undecidable problems involving elementary functions of a real variable. J. Symbolic Logic, 33(4):514–520, 1969. [doi:10.2307/2271358]

[73]   Jürgen Richter-Gebert and Günter M. Ziegler: Realization spaces of 4-polytopes are universal. Bull. AMS, 32(4):403–412, 1995. [doi:10.1090/S0273-0979-1995-00604-X]

[74]   Itay Safran and Ohad Shamir: Depth-width tradeoffs in approximating natural functions with neural networks. In Proc. 34th Internat. Conf. Machine Learning (ICML’17), pp. 2979–2987. MLR Press, 2017. Available at PMLR.

[75]   Marcus Schaefer: Complexity of some geometric and topological problems. In Graph Drawing (GD’09), pp. 334–344. Springer, 2009. [doi:10.1007/978-3-642-11805-0_32]

[76]   Marcus Schaefer: Realizability of graphs and linkages. In János Pach, editor, Thirty Essays on Geometric Graph Theory, pp. 461–482. Springer, 2013. [doi:10.1007/978-1-4614-0110-0_24]

[77]   Marcus Schaefer: Complexity of geometric k-planarity for fixed k. J. Graph Algor. & Appl., 25(1):29–41, 2021. [doi:10.7155/jgaa.00548]

[78]   Marcus Schaefer: RAC-drawability is R-complete and related results. J. Graph Algor. & Appl., 27(9):803–841, 2023. [doi:10.7155/jgaa.00646]

[79]   Marcus Schaefer, Jean Cardinal, and Tillmann Miltzow: The existential theory of the reals as a complexity class: A compendium. In János Pach and Géza Tóth, editors, Courses in Discrete and Computational Geometry, volume 31 of Bolyai Society Mathematical Studies, pp. 167–313. Springer, Cham, 2026. [doi:10.1007/978-3-032-10503-5_5, arXiv:2407.18006]

[80]   Marcus Schaefer and Daniel Štefankovič: Fixed points, Nash equilibria, and the existential theory of the reals. Theory Computing Sys., 60(2):172–193, 2017. [doi:10.1007/s00224-015-9662-0]

[81]   Marcus Schaefer and Daniel Štefankovič: Beyond the existential theory of the reals. Theory Computing Sys., 68(2):195–226, 2024. [doi:10.1007/s00224-023-10151-x]

[82]   Marcus Schaefer and Daniel Štefankovič: The complexity of tensor rank. Theory Computing Sys., 62(5):1161–1174, 2018. [doi:10.1007/s00224-017-9800-y]

[83]   Thiago Serra, Christian Tjandraatmadja, and Srikumar Ramalingam: Bounding and counting linear regions of deep neural networks. In Proc. 35th Internat. Conf. Machine Learning (ICML’18), pp. 4558–4566. MLR Press, 2018. Available at PMLR.

[84]   Shai Shalev-Shwartz and Shai Ben-David: Understanding Machine Learning: From Theory to Algorithms. Cambridge Univ. Press, 2014. [doi:10.1017/CBO9781107298019]

[85]   Yaroslav Shitov: The complexity of positive semidefinite matrix factorization. SIAM J. Optim., 27(3):1898–1909, 2017. [doi:10.1137/16M1080616]

[86]   Peter W. Shor: Stretchability of pseudolines is NP-hard. In Appl. Geom. & Discr. Math., pp. 531–554. Amer. Math. Soc., 1991. [doi:10.1090/dimacs/004/41]

[87]   Jack Stade: The point-boundary art gallery problem is R-hard. In Proc. 41st Internat. Symp. Comput. Geom. (SoCG’25), pp. 74:1–23. Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, 2025. [doi:10.4230/LIPIcs.SoCG.2025.74, arXiv:2210.12817]

[88]   Moritz Stargalla, Christoph Hertrich, and Daniel Reichman: The computational complexity of counting linear regions in ReLU neural networks. In Proc. 38th Adv. Neural Info. Proc. Sys. (NeurIPS’25), pp. 125970–125988. Curran Assoc., 2025. Available at NeurIPS.

[89]   Matus Telgarsky: Benefits of depth in neural networks. In Proc. 29th Ann. Conf. on Learning Theory (COLT’16), pp. 1517–1539. Springer, 2016. Available at PMLR.

[90]   Leslie G. Valiant: A theory of the learnable. Comm. ACM, 27(11):1134–1142, 1984. [doi:10.1145/1968.1972]

[91]   Gal Vardi, Gilad Yehudai, and Ohad Shamir: On the optimal memorization power of ReLU neural networks. In Int. Conf. Learning Representations (ICLR’22). ICLR, 2022. Available at OpenReview.

[92]   Dmitry Yarotsky: Error bounds for approximations with deep ReLU networks. Neural Networks, 94:103–114, 2017. [doi:10.1016/j.neunet.2017.07.002]

[93]   Chulhee Yun, Suvrit Sra, and Ali Jadbabaie: Small ReLU networks are powerful memorizers: A tight analysis of memorization capacity. In Proc. 32nd Adv. Neural Info. Proc. Sys. (NeurIPS’19). Curran Assoc., 2019. Available at NeurIPS.

[94]   Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals: Understanding deep learning (still) requires rethinking generalization. Comm. ACM, 64(3):107–115, 2021. [doi:10.1145/3446776]

[95]   Liwen Zhang, Gregory Naitzat, and Lek-Heng Lim: Tropical geometry of deep neural networks. In Proc. 35th Internat. Conf. Machine Learning (ICML’18), pp. 5824–5832. MLR Press, 2018. Available at PMLR.

[96]   Xiao-Dong Zhang: Complexity of neural network learning in the real number model. In Workshop on Physics and Computation, pp. 146–150. IEEE Comp. Soc., 1992. [doi:10.1109/PHYCMP.1992.615511]