Michael Bowling (August 27, 2026; UCOMM 2-108)
What should models model in model-based RL?
Matthew Vandergrift (August 26, 2026; UCOMM 2-108)
Can we be non-i.i.d in the batch dimension?
In typical machine learning problems we often have treat vectors as our base statistical objects, and we process them in batches where we assume that each vector is independent of all others in the batch. This is a useful paradigm, and this assumption of independence is true, or at least true enough, for many problem instances. In this talk I want to explore what might happen in a setting where this is not true. We use offline batch regression between non i.i.d batches as a starting point for this problem. How can we handle such a setting, starting from a simple naive approach and proposing one maybe more elegant way to handle this setting.
Hriday Mehta (August 25, 2026; UCOMM 2-108)
Thinking about Memory in RL
An implication of the Big World Hypothesis is that an agent can never fully perceive the state of the world. To distinguish situations that produce identical observations, an agent therefore requires some form of internal state distilled from its history. This talk traces several approaches to incomplete perception across three decades, from belief states and Predictive State Representations (PSRs) to the recurrent hidden states commonly used in deep reinforcement learning, as well as early approaches to memory in RL. A common feature of these approaches is that memory is passive: it is a fixed function of the agent's past, computed on the agent's behalf rather than explicitly chosen by the agent. We will explore an alternative perspective in which memory itself becomes part of the agent's decision-making: the agent can choose what information to retain and when to retain it. Using a minimal gridworld in which an agent can control a single memory bit, we demonstrate that an agent can learn to reason about and deliberately use its own memory. This provides a simple setting for studying memory not merely as an architectural mechanism, but as an object of action and decision-making.
Haruto Tanaka (August 13, 2026; UCOMM 2-108)
Is momentum needed to optimize the final layer of networks in deep RL?
The de facto optimizer in modern deep reinforcement learning (RL) is Adam, which utilizes the gradient moments to stabilize the optimization process. While the benefits of gradient moments in stationary settings (e.g., supervised learning) are well known, it remains unclear whether retaining prior gradient information is entirely beneficial in deep RL. In this talk, I hypothesize that momentum aids feature learning but impedes learning the predictor (i.e., final layer of the network) in deep RL settings. I evaluate this hypothesis with deep RL benchmarks by replacing the predictor's optimizer with normalized SGD.
Jiamin He (August 12, 2026; UCOMM 2-108)
Learning to achieve goals in hindsight
Learning to achieve multiple (sub)goals is a long-standing challenge in reinforcement learning. In this talk, I will focus on the setting in which the agent has only limited memory and yet the space of goals is vast. Most existing algorithms fall short in this setting, which demands generalization across goals without the luxury of storing massive past experience. Instead, I'll discuss how the idea of learning to achieve goals in hindsight can be leveraged to improve learning efficiency.
Adrian Orenstein (August 11, 2026; UCOMM 2-108)
How can agents learn when decisions need more time?
Reinforcement learning (RL) agents spend the same amount of time making decisions regardless of the state's complexity or importance. And yet, the difficulty of decision-making varies from state to state. If more processing time improves the actions taken, and better actions earn greater future rewards, then agents could learn where to spend additional time. In this tea time talk, I will present some early results on how decision-making problems can be framed so that agents learn when to ask for more computation time.
Shivam Garg (August 6, 2026; UCOMM 2-108)
Adventures of a big agent in a stationary world: Modeling the problem
In this talk, I will discuss some reasonable ways, as I see them, for modeling a big agent and a world where this agent might act. Further, the small agent will make a cameo in the stationery [sic] world!
In particular, I will
To keep the exposition concise, we will consider simple, well-known examples from supervised learning, but against the backdrop of RL.
Dikshant (August 4, 2026; UCOMM 2-108)
What are Option Models?
Planning needs a model of the world, and the usual single-step model applied autoregressively is fragile, as per-step errors compound over long horizons making the prediction problem harder. In this talk, I trace one idea repeatedly rediscovered and renamed over the past 30 years -- β-models (Sutton, 1995), option models (Silver and Ciosek, 2012), γ-models (Janner et al., 2020), geometric horizon models (Thakoor et al., 2022), and jumpy world models (Farebrother et al., 2026) -- multi-step, “jumpy” models that predict where a behavior ends up rather than what happens at the next step. I will develop this through Rich’s mixture-of-timescales framing, show how the weighting over future timescales is effectively a termination condition, which turns a multi-step model into an option model, and briefly discuss how these models are learned and used for planning over temporally extended actions or options.
Paulius Sasnauskas (July 30, 2026; UCOMM 2-108)
Programming In-Context Learning
Humans are able to understand new concepts from only a few labelled examples, which is hardly feasible with supervised learning or RL. However, Transformers are able to some extent infer the underlying rule given few examples, and apply it to a given unlabeled example, also called in-context learning. In this talk, I argue that current Transformers are still not capable of generalization. I hypothesize that including programmable circuits with search could make the in-context computation more reliable.
Ahmed Khan (July 29, 2026; UCOMM 2-108)
Active Inference for Reinforcement Learning
Reinforcement learning (RL) is a framework for modelling an agent’s interaction with an environment, under the normative principle of reward maximisation. State-of-the-art RL algorithms can solve complex tasks in simultaneously continuous and high-dimensional environments. However, for reproducing goal-directed behaviour for the most difficult of tasks, three challenges remain: dealing with partial observability, balancing exploration and exploitation, and designing reward functions. In this talk, I show that the active inference framework, being a corollary of the free energy principle under which agents minimise surprise, parsimoniously addresses all three of these challenges. I will give a tour of the mathematics underlying active inference, describe an example implementation of this framework for solving RL problems, and then outline my own perspective on scaling such a regime to even more difficult tasks.
Kiran Deol (July 28, 2026; UCOMM 2-108)
Not All Capacity is Equal: Local Measures of Plasticity in Continual Learning
Plasticity loss describes the decline in a neural network’s ability to learn over the course of continual training. Many diagnostics characterize the problem through global or task-agnostic markers, overlooking how the current task interacts with the network’s parameterization. We work from the insight that plasticity depends not only on the network's overall capacity, but on said capacity being accessible and useful to the current task. We introduce UPC and GAP, two task-conditioned metrics derived from the singular value decomposition of the network’s output Jacobian. These metrics quantify the availability and task alignment of local output-changing parameter directions, often providing earlier indicators of plasticity loss without requiring training to task completion. Through controlled manipulations of the network’s available subspaces, we empirically show that networks with comparable - or even greater - overall capacity can exhibit substantially different plasticity, which our metrics correctly flag. We also use this framework to compare several plasticity interventions and find that adding random neurons is the most robust. Together, our results characterize plasticity in terms of usable, task-relevant capacity rather than aggregate capacity alone, while establishing random expansion as a strong baseline for future interventions.
Alex Ayoub (July 23, 2026; UCOMM 2-108)
Simple and Scalable Multitask Reinforcement Learning
A central challenge in multitask reinforcement learning is training a single policy that can identify the task it is currently facing and adapt its behavior through interaction, without being given an explicit task identifier. In this talk, I present a simple approach that leverages current progress in casual autoregressive function approximators that conditions each action on the complete interaction history. This allows the function approximator to infer latent task information and use it to select task-dependent actions.
Matthew Vandergrift (July 22, 2026; UCOMM 2-108)
RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy
One goal in reinforcement learning (RL) research is to understand general-purpose sequential decision-making, using benchmark simulators as a proxy for learning in deployment settings. When running experiments, however, the goal of achieving high performance in the simulator can mutate into focusing exclusively on solving the simulator. To achieve high scores, researchers may adopt solutions exclusively meant for solving simulators, rather than learning while the agent is deployed outside a simulator. Solving simulators is also worthy of investigation, but it is a fundamentally different RL research question. In this paper, we argue that RL researchers need to distinguish between two use cases of simulators: solving simulators and using simulators as a proxy for learning in deployment. We first discuss how these two use-cases are importantly different, in terms of constraints on how the agent can use the simulator, which algorithms are appropriate, and which evaluation metrics are appropriate. We then highlight several issues and misleading conclusions that can occur by not making the distinction between these two settings clear, supported with examples and simple experiments. This work is a call to the community to begin clearly distinguishing how they are using simulators in their work, hopefully sparking further discussion on which empirical practices work best in each setting.
Justin Stevens (July 21, 2026; UCOMM 2-108)
From Hints to Textbooks: Studying Scaffolds through Bayesian Methods and LLMs
Intelligent tutoring systems increasingly rely on AI-generated hints, but not all hints do the same job: some point out a feature of a problem, some state a governing principle, some walk a student to the answer, and some extend thinking toward future problems. This talk presents an analysis for classifying mathematical hints into these four types and for studying how hints sequence from one type to the next.
Using a corpus of Socratic hint chains spanning seven mathematics domains, we model hint-to-hint transitions with a Dirichlet prior and a multinomial likelihood via Bayesian conjugacy, yielding a transition matrix for each domain. Comparing these matrices using KL divergence reveals that counting & probability are consistent outliers. We apply the same machinery to Quarfoot's nine-part problem typology, recovering the expected structure in which “masterpieces” and “improvisations” sit opposite atomic “first notes” in terms of KL-divergence.
Finally, we will present notes on how future curricula and textbooks can evolve using ideas from intelligent tutoring systems, including hints and problem-type classifications. These ideas come from the author’s experience recently revising and finishing his Olympiad Number Theory book: numbertheoryguy.com/publications/olympiad-number-theory-book.
Alireza Masoumian (July 16, 2026; UCOMM 2-108)
Imitate through values not just behaviours!
I am going to talk about imitation learning and the advantage of assuming a realizable set of value functions rather than a realizable set of policies. We will see how this different assumption leads us to more interesting primal-dual algorithms instead of vanilla supervised learning (behavioural cloning), and see value class-based sample complexities in contrast to policy class-based guarantees. We’ll see some ideas in the area of saddle-point optimization and no-regret learning.
Rosanna Dovganyuk (July 15, 2026; UCOMM 2-108)
Imperfect Flows
Flow models are generative frameworks that incrementally transform random noise into complex data distributions. I explain how flow matching can be used as a technique to train Continuous Normalizing Flows (CNFs) by predicting the underlying velocity vector field. Within the flow matching framework, we investigate the different sources of inaccuracies or imperfections that arise during both the learning and sampling processes. We want to understand whether these imperfections can counterintuitively improve downstream performance. Finally, we ask how learning models of the world is useful and explore how flow models are currently integrated into reinforcement learning environments
Shuai Liu (July 14, 2026; UCOMM 2-108)
Can offline policy gradients avoid suffering from suboptimal actions that are rarely seen?
Offline policy-gradient methods often rely on importance sampling, causing their performance to deteriorate when the behavior policy assigns very small probability to certain actions, even when those actions are suboptimal. Ideally, an offline method should not pay heavily for actions that are both rare and unimportant. In this work-in-progress talk, I will explain the bottlenecks created by importance sampling and softmax policy gradients, introduce the intuition behind pessimistic approaches, and present a differentiable objective inspired by these ideas. I will then discuss why initializing from the behavior policy revives the difficulty and outline a possible approach toward a more practical method.
Levi Lelis (July 8, 2026; UCOMM 2-108)
Learning Neural Languages and Their Interpreters
Neural networks often break down on examples that differ from those seen in training. One reason for this brittleness lies deep within our own field: computation. Current neural models can be effective at perception tasks, such as recognizing images of dogs and cats, but many problems cannot be solved (even approximately) in a fixed number of computational steps. For example, when manipulating lists of integers, the number of steps needed can grow with the input, and a sufficiently large input will eventually cause a fixed-depth model to produce garbage. In this talk, I will present an architecture that can adapt its computation at test time to problems unseen during training. The model learns, end-to-end, a vocabulary of neural symbols, together with a neural interpreter that executes them. Because execution is differentiable, we can synthesize neural programs at test time by searching the space of programs with gradient descent. The learned language allows the model to generalize combinatorially to problems requiring more or fewer computational steps than those seen in training, and to novel compositions of learned primitives, where competing neural models fail.
Edan Meyer (July 7, 2026; UCOMM 2-108)
Why You Should Care About Connectivity
We want agents that can learn many things at once — many auxiliary predictions, many options, many goals — but it is not clear how a neural network should accommodate multiple outputs with distinct objectives. Sharing representations across objectives seems desirable, yet doing so naively creates problems. Starting from a small example, I show how network connectivity affects learning speed, and that this problem grows with the network itself: when connectivity is poorly configured, scaling up a network can hurt its ability to continually learn rather than help it.
Csaba Szepesvari (June 25, 2026; UCOMM 2-108)
Some last minute (as ever) thoughts about never-ending learning
I will make another attempt to formalize what we want from agents that need to learn in big worlds where the agent's prior knowledge is very limited. At a high level we want agents to get more skillful in face of evidence experienced, but stay indefinitely curious, and they should not stay nimble (age is just a number!). How do we formalize all of this in a meaningful way so that if we can recognize (test?) whether a particular learning algorithms satisfies our criteria. To get started, we introduce a number of simplifying assumptions: There will be no function approximation and no limit of how much memory the agents can use. The environment is a finite MDP, though nothing is known about the MDP. I allow the per step compute time spent by the agents to scale with the size of the environment. This is work in progress, and I am interested in starting a discussion about the ideas I will present. The approach we will take is to see whether we can formalize these requirements by putting constraints on the learning algorithm when they interact with "learnable" MDPs; the interesting question is how to formalize what is learnable, and how algorithms that behave well in learnable MDPs behave in the rest of MDPs.
Marlos Machado (June 24, 2026; UCOMM 2-108)
The Laplacian Representation as a Basis for Integration
Research is inherently reductionist: in each project, we isolate a small question and design a minimal system to instantiate the underlying idea. In reinforcement learning, although we ultimately want agents that learn representations, models, options, and plans, this approach typically leads us to focus on only a subset of these components. While this has clear advantages, it also leads to design choices tailored to specific use cases rather than to how components will work together. If we care about integration, and ultimately about agents with the core capabilities required for intelligence, we should instead focus on components that naturally compose. In this talk, I focus on one such component: the agent's learned representation, in particular the Laplacian representation. I will briefly discuss how my research group has been using it to advance several capabilities we believe are central to intelligence, and then, time permitting, try to highlight two of our recent results that support the idea of the Laplacian representation as a good inductive bias for reinforcement learning. First, I will show how the Laplacian representation can be used to model the emergence of diverse cell types across the hippocampal-entorhinal circuit. Second, I will discuss how it can serve as a substrate for behavioural foundation models and how, by allowing these models to continue learning after deployment, they can move beyond the linear span of their initial representations, enabling capabilities beyond those of existing approaches.
Khurram Javed (June 18, 2026; UCOMM 2-108)
Physical Atari: A Robust and Accessible Platform for Real-time Reinforcement Learning on Robots
I will present Physical Atari, a robotics platform for testing reinforcement learning algorithms on robots. In the first half of the talk, I will talk about the key ideas to make a robotics platform suitable for reinforcement learning, and in the second half, I will share some results that show the necessity of continual adaptation when applying reinforcement learning to physical systems.
Richard Sutton (June 17, 2026; UCOMM 2-108)
Rediscovering Reinforcement Learning
The 2026 tea time talks are coordinated and organized by Esraa Elelimy (elelimy AT ualberta DOT ca) and Seth Akins (slakins AT ualberta DOT ca).