Numerical Mathematics Challenges in Machine Learning Workshop

Most machine learning (ML) methods are based on numerical mathematics (NM) concepts, from differential equation solvers over dense matrix factorizations to iterative linear system and eigen-solvers. As long as problems are of moderate size, NM routines can be invoked in a black-box fashion. However, for a growing number of real-world ML applications, this separation is insufficient and turns out to be a severe limit on further progress.

The increasing complexity of real-world ML problems must be met with layered approaches, where algorithms are long-running and reliable components rather than stand-alone tools tuned individually to each task at hand. Constructing and justifying dependable reductions requires at least some awareness about NM issues. With more and more basic learning problems being solved sufficiently well on the level of prototypes, to advance towards real-world practice the following key properties must be ensured: scalability, reliability, and numerical robustness. Unfortunately, these points are widely ignored by many ML researchers, preventing applicability of ML algorithms and code to complex problems and limiting the practical scope of ML as a whole.

Description and Motivation

Our workshop addresses the abovementioned concerns and limitations. By inviting numerical mathematics researchers with interest in *both* numerical methodology *and* real problems in applications close to machine learning, we will probe realistic routes out of the prototyping sandbox. Our aim is to strengthen dialog between NM and ML. While speakers will be encouraged to provide specific high-level examples of interest to ML and to point out accessible software, we will also initiate discussions about how to best bridge gaps between ML requirements and NM interfaces and terminology; the ultimate goal would be to figure out how at least some of NM's high standards of reliability might be transferred to ML problems.

The workshop will reinforce the community's awakening attention towards critical issues of numerical scalability and robustness in algorithm design and implementation. Further progress on most real-world ML problems is conditional on good numerical practices, understanding basic robustness and reliability issues, and a wider, more informed integration of good numerical software. As most real-world applications come with reliability and scalability requirements that are by and large ignored by most current ML methodology, the impact of pointing out tractable ways for improvement is substantial.

General Topics of Interest

A basic example for the NM-ML interface is the linear model (or Gaussian Markov random field), a major building block behind sparse estimation, Kalman smoothing, Gaussian process methods, variational approximate inference, classification, ranking, and point process estimation. Linear model computations reduce to solving large linear systems, eigenvector approximations, and matrix factorizations with low-rank updates. For very large problems, randomized or online algorithms become attractive, as do multi-level strategies. Additional examples include analyzing global properties of very large graphs arising in social, biological, or information transmissing networks, or robust filtering as a backbone for adaptive exploration and control.