Recent seminars

Europe/Lisbon
Online

Markus Reichstein

Markus Reichstein, MPI for Biogeochemistry
Integrating Machine Learning with System Modelling and Observations for a better understanding of the Earth System

The Earth is a complex dynamic networked system. Machine learning, i.e. derivation of computational models from data, has already made important contributions to predict and understand components of the Earth system, specifically in climate, remote sensing and environmental sciences. For instance, classifications of land cover types, prediction of land-atmosphere and ocean-atmosphere exchange, or detection of extreme events have greatly benefited from these approaches. Such data-driven information has already changed how Earth system models are evaluated and further developed. However, many studies have not yet sufficiently addressed and exploited dynamic aspects of systems, such as memory effects for prediction and effects of spatial context, e.g. for classification and change detection. In particular new developments in deep learning offer great potential to overcome these limitations. Yet, a key challenge and opportunity is to integrate (physical-biological) system modeling approaches with machine learning into hybrid modeling approaches, which combines physical consistency and machine learning versatility. A couple of examples are given with focus on the terrestrial biosphere, where the combination of system-based and machine-learning-based modelling helps our understanding of aspects of the Earth system.

Europe/Lisbon
Online

Tom Goldstein

Tom Goldstein, University of Maryland
Building (and breaking) neural networks that think fast and slow

Most neural networks are built to solve simple pattern matching tasks, a process that is often known as “fast” thinking. In this talk, I’ll use adversarial methods to explore the robustness of neural networks. I’ll also discuss whether vulnerabilities of AI systems that have been observed in academic labs can pose real security threats to industrial systems. Then, I’ll present methods for constructing neural networks that exhibit “slow” thinking abilities akin to human logical reasoning. Rather than learning simple pattern matching rules, these networks have the ability to synthesize algorithmic reasoning processes and solve difficult discrete search and planning problems that cannot be solved by conventional AI systems. Interestingly, these reasoning systems naturally exhibit error correction and robustness properties that make them more difficult to break than their fast thinking counterparts.

Additional file

document preview

Goldstein slides.pdf

Europe/Lisbon
Online

João Sacramento

João Sacramento, ETH Zürich
The least-control principle for learning at equilibrium

A large number of models of interest in both neuroscience and machine learning can be expressed as dynamical systems at equilibrium. This class of systems includes deep neural networks, equilibrium recurrent neural networks, and meta-learning. In this talk I will present a new principle for learning equilibria with a temporally - and spatially - local rule. Our principle casts learning as a least-control problem, where we first introduce an optimal controller to lead the system towards a solution state, and then define learning as reducing the amount of control needed to reach such a state. We show that incorporating learning signals within a dynamics as an optimal control enables transmitting activity-dependent credit assignment information, avoids storing intermediate states in memory, and does not rely on infinitesimal learning signals. In practice, our principle leads to strong performance matching that of leading gradient-based learning methods when applied to an array of benchmarking experiments. Our results shed light on how the brain might learn and offer new ways of approaching a broad class of machine learning problems.

Additional file

document preview

Sacramento slides.pdf

Europe/Lisbon
Online

Frederico Fiuza

Frederico Fiuza, SLAC
Accelerating the understanding of nonlinear dynamical systems using machine learning

The description of nonlinear, multi-scale dynamics is a common challenge in a wide range of physical systems and research fields — from weather forecast to controlled nuclear fusion. The development of reduced models that balance between accuracy and complexity is critical to advancing theoretical comprehension and enabling holistic computational descriptions of these problems. I will discuss how techniques from statistical and machine learning are offering new ways of inferring reduced physics models from the increasingly abundant data of nonlinear dynamics produced by experiments, observations, and simulations. In particular, I will focus on how sparse regression techniques can be used to infer interpretable plasma physics models (in the form of nonlinear partial differential equations) directly from the data of first-principles fully-kinetic simulations. The potential of this approach is demonstrated by recovering the fundamental hierarchy of plasma physics models based solely on particle-based simulation data of complex plasma dynamics. I will discuss how this data-driven methodology provides a promising tool to accelerate the development of reduced theoretical models of nonlinear dynamical systems and to design computationally efficient algorithms for multi-scale simulations.

Additional file

document preview

fiuza slides.pdf

Europe/Lisbon
Online

Robert Nowak

Robert Nowak, University of Wisconsin-Madison
The Neural Balance Theorem and its Consequences

Rectified Linear Units (ReLUs) are the most common activation function in deep neural networks. Weight decay is the most prevalent form of regularization used in deep learning. Together ReLUs and weight decay lead to an interesting effect known as “Neural Balance”: the norms of the input and output weights of each ReLU are automatically equalized at a global minima of the training objective. Neural Balance has a number of important consequences ranging from characterizations of the function spaces naturally associated to neural networks, their immunity to the curse of dimensionality, and to new and more effective architectures and training strategies.

This talk is based on joint work with Rahul Parhi and Liu Yang.