open-problem

Learn a relation as a formula: not a polynomial with fixed monomials, not a network, but a short symbolic expression found by search. It would make learned relations readable and comparable, and it is slow and hard. Noted for later; nothing is implemented. The most promising route is distillation from an already trained factor (§5).

Sources: Schmidt & Lipson, Distilling free-form natural laws from experimental data, Science 2009; Mangan, Brunton, Proctor & Kutz, Inferring biological networks by sparse identification of nonlinear dynamics, IEEE TMBMC 2016 (implicit SINDy); Kaheman, Kutz & Brunton, SINDy-PI, Proc. R. Soc. A 2020; Cranmer, Interpretable Machine Learning for Science with PySR and SymbolicRegression.jl, arXiv:2305.01582, 2023; Cranmer et al., Discovering Symbolic Models from Deep Learning with Inductive Biases, NeurIPS 2020

Bibliography: Schmidt & Lipson 2009 · Mangan et al. 2016 · Kaheman et al. 2020 · Cranmer 2023 · Cranmer et al. 2020

Theory (CT-ML wiki): Statistical Game

1. The idea

Symbolic regression searches over expression trees for . The implicit version searches for with on the data, so that the formula is a relation with no privileged output, the way this family’s polynomials are (Algebraic Implicit Learners). The algebraic family is the special case of a fixed library of monomials, where fitting is a nullspace problem (Fitting is a Nullspace Problem); implicit SINDy is the same with any fixed library of candidate terms. Symbolic implicit learning drops the fixed library.

2. Why it would be worth it

  • Readable relations. The output is a law, , not a weight vector.
  • Model selection is native: a Pareto front of complexity against residual is a family of candidate relations, and the free energy is a principled way to choose among them (Open Problems in Algebraic Implicit Learning §4 is the same jump-between-models problem).
  • Distillation. A learned factor (a pair potential, an energy network) can be explained afterwards by a formula, as in Cranmer et al. 2020, and the formula then corrected by a learned residual. This is the route that avoids most of §3; see §5.

3. Why it is hard

  • Trivial solutions. fits every dataset, and so does for any that vanishes on it. Explicit regression avoids this by fixing ; implicit regression needs a normalisation. The nullspace formulation normalises the coefficient vector; Schmidt and Lipson match ratios of partial derivatives instead; SINDy-PI tries each library term as the left-hand side in turn.
  • Search cost. Genetic search over expression trees is slow, and the implicit version has no target column to guide it.
  • Real locus. A formula can vanish on the data and on extra spurious components, the same problem as Open Problems in Algebraic Implicit Learning §1.

4. What exists to build on

  • SymbolicRegression.jl (and PySR on top of it): evolutionary search, Pareto fronts, custom losses, dimensional constraints, templates that fix part of a formula. Explicit regression; an implicit loss is possible through its custom-loss interface.
  • DataDrivenDiffEq.jl (SciML): SINDy and an implicit optimiser for fixed libraries.
  • This family’s own nullspace fitting, which is implicit and exact for polynomial libraries.

A first experiment: recover the circle, then the robot arm’s kinematic relation, from samples, with an implicit loss in SymbolicRegression.jl, and compare with the polynomial nullspace fit.

5. Distillation: a learned factor as the teacher

The trivial solution of §3 is a problem of learning from samples alone: samples say only where vanishes. A trained factor says much more. Its field is defined everywhere, points towards the relation, and vanishes only on it; the solver projects any point onto the relation; an energy network (or a closed-form mixture) even has an energy . So the plan is in three steps: learn a factor, distil it into a formula, learn the residual.

5.1 Distillation losses that exclude

teacher signalloss for the formula why fails
projections of off-relation points (implicit_infer with ) and normals , , the gradient must have unit length on the relation
the signed distance near the relationa first-order distance estimate, scale-free
the field itself (any learner), the relation being the minima of off the relation, so a constant does not fit
the energy (energy networks, mixtures)as above, with values instead of gradients

The first is implicit-surface fitting with normals, a classical way to normalise; here the normals come free from the teacher. The third and fourth fit a density rather than a relation: cleaner as losses, but the target is the teacher’s smoothed log-density, which is not a short formula even for a circle (it has terms in and a smoothing width). Fitting the zero set and normals gives the short formula (, normalised).

Two refinements:

  • Avoid inheriting the teacher’s bias. The teacher’s relation sits slightly inside a curved one (the smoothing bias, Implicit Diffusion Learners §5). Fit the zero set to the data, and take only the normals, and the off-relation points, from the teacher.
  • Several branches. A multivalued relation is one formula ( is allowed), so the implicit version needs no branch labels, unlike explicit symbolic regression on “y given x”. implicit_roots supplies all branches for sampling targets.

5.2 The residual

Keep the formula and learn what it misses. The tool for this exists: a product of relations adds fields (Composing Diffusion Factors, ProductRelation), so

with a small diffusion model trained on the data with the symbolic factor fixed. Its denoising target is the part of the noise the formula does not explain, ; this is exact only at small noise levels (the composition obstruction T9 again), which are the levels the implicit learner reads. The residual must be penalised towards zero, or it absorbs everything and the formula becomes decoration; its size is then the honest measure of what the formula misses. Iterating (distil the residual too) is the AI-Feynman or sparse-regression pattern of peeling off one term at a time.

What one gets: a readable relation that is exact where the physics is known, a learned correction where it is not, and the free energy (or the residual’s norm) to decide whether a term is worth adding. The force-law tutorial already did the explicit half of this (symbolic regression on a learned pair force, as in Cranmer et al. 2020); this is the implicit version.

5.3 What to try first

  1. Teacher: the circle model of the training tutorial. Distil with the projection-and-normal loss in SymbolicRegression.jl (custom loss, its expression gradients), expect .
  2. A relation with a known part and an unknown part, e.g. the robot arm with a sagging second link: the formula should find rigid kinematics, the residual the sag.
  3. Check that the residual factor plus the formula answers queries better than either alone, and that the residual vanishes on data with no unknown part.

Related: Algebraic Implicit Learners, Fitting is a Nullspace Problem, Open Problems in Algebraic Implicit Learning, Geometric Deep Learning and Physical Laws, Composing Diffusion Factors, Implicit Diffusion Learners