implementation

NoisePredictor wraps a Lux model as and derives from it the two quantities the rest of the package needs: the score and the denoiser.

Sources: code: predictor.jl; noise blocks: Bao et al., One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale, ICML 2023 (UniDiffuser)

Bibliography: Bao et al. 2023

Theory (CT-ML wiki): Parametric Lens · Lens

1. LuxCore, not Lux

The prompt asks the package to “wrap some Lux.jl model”. It does — and Lux is not a dependency. Any Lux layer is a LuxCore.AbstractLuxLayer, and the wrapper only ever calls LuxCore.apply(model, input, ps, st). So Lux, Zygote, Optimisers and the rest stay out of the dependency tree, exactly as the parent package depends on LuxCore rather than Lux (Lux as a Parametric Lens §“What is kept unchanged”).

NoisePredictor <: LuxCore.AbstractLuxWrapperLayer, which is the LuxCore trick for “my parameters are my child’s parameters, with no extra nesting”. So

LuxCore.setup(rng, NoisePredictor(unet, VPSDE())) == LuxCore.setup(rng, unet)

and wrapping costs nothing. This is the one place where the project’s decision to mirror the LuxCore API name-for-name pays off directly rather than aspirationally.

2. Three quantities, one network

The second and third are derived, not learned, and that they are derived is the whole reason a noise predictor can serve as a prior rather than only as a sampler.

The score identity is exact for the perturbation kernel: with ,

so a perfect is a perfect score. The denoiser is Tweedie’s formula, and

Tweedie is exact, and the test suite proves it

For Gaussian data the closed-form posterior mean is , and denoise reproduces it to floating point, for every tested. Tweedie’s formula turns a noise predictor into an MMSE denoiser; that is the precise sense in which a diffusion model is a prior, and it is what the RED-Diff regulariser scores against.

3. The input adapter

Diffusion architectures disagree about how arrives: a second positional argument, a sinusoidal embedding, a concatenated channel, a NamedTuple. The wrapper takes an input function, defaulting to (x, t) -> (x, t), and knows nothing about time embeddings — that is the model’s business.

The cost is that a wrong input fails inside the user’s model with the user’s error message rather than here with a good one. There is no way to validate it without knowing the architecture, so it is not validated.

4. Implementation difficulties

4.1 Forward-only where it can be, any AD backend where it cannot

epsilon is the only place the network is evaluated forwards, and RED-Diff never needs a reverse pass through model. That is RED-Diff’s stop-gradient, promoted to an architectural property: sampling and RED-Diff work with no AD at all. reddiff.md §5.

The implicit learner is different. Its Newton steps want the input Jacobian, and its backward pass wants a parameter VJP. Both go through the ad field — any ADTypes object — and are supplied by package extensions, so the package still has no AD dependency of its own. epsilon itself calls the hook _apply(ad, …), which a backend may replace by a compiled forward pass (ad = AutoReactant()). See backends. Training by denoising score matching is ordinary Lux training (examples/circle_mlp.jl).

4.2 The 1/σ_t in score is a real singularity

score divides by , so it is unusable at and inaccurate for small even with the tmin floor. Nothing in this package calls score — the RED-Diff gradient is expressed in -space precisely to avoid it — so it exists for completeness and for anyone building a sampler on top. If you call it near , that is on you.

4.3 denoising_loss returns a scalar, so it cannot be a vector energy

Scalar and Multivariate Energy argues that a factor should expose its vector residual, because the implicit function theorem needs the Jacobian. denoising_loss collapses to sum(abs2, ε̂ - ε) immediately.

The vector residual is right there — — and returning it would let SquaredNorm do the scalarisation and would make the diffusion factor’s energy space rather than . It is not done because RED-Diff’s gradient is already in closed form and the Jacobian is exactly what the method refuses to compute; the vector form would be honest bookkeeping with nothing downstream to consume it. Recorded as a genuine inconsistency with the vault’s stated design, with a reason.

5. Running on a device

NoisePredictor(model, schedule; input, ad, device = (to = gpu_device(), back = cpu_device())) runs the network on an accelerator and leaves everything else where it is. The solvers’ state, the field’s noise draws, the small dense blocks and the Krylov vectors stay on the CPU. Every entry point that evaluates or differentiates the network (epsilon, energy, energy_columns, the input Jacobian and its blocks, the input VJP and JVP, the parameter VJP) moves its array arguments with to, calls a copy of the predictor without a device, and returns its result through back. Parameter cotangents are not moved back: they belong to the parameters, which the caller has put on the device.

Two decisions make this fast rather than merely correct.

  • Derivatives happen on the device. The first version wrapped the model in a layer that moved its input. Forward-mode dual numbers were then made on the CPU, converted, and moved in and out of every Hessian-vector product: 234 ms for one product over 4000 columns, where the device itself needs about 20. With the device at the predictor’s entry points the duals are created on the device, and the same product takes 21 ms.
  • No CPU-only constructions in the hooks. collect brings a GPU array home and a basis vector made with zeros lives on the CPU. The DifferentiationInterface extension copies with similar and moves its constants next to the input. On a device it builds Jacobian and Hessian blocks from one product per free coordinate instead of a chunked Jacobian.

The input function receives device arrays and must build its time features there. Eight numbers per column are made on the CPU and copied next to x. MLDataDevices’ gpu_device() also converts Float64 to Float32; a consumer GPU computes Float64 at 1/64 of the rate.

The tests run every hook on JLArrays, the reference GPU array type that runs on the CPU and forbids scalar indexing, so a CPU-only construction fails in CI without a GPU. The measured costs on CUDA are in energy_network §8.

6. A noise level per block of coordinates

NoisePredictor(model, schedule; input, blocks = [1:784, 785:794]) partitions the coordinates into blocks that may sit at different noise levels, as UniDiffuser does for image and text (Bao et al. 2023). Time is then a number (every block at one level), a vector with one time per block, or a matrix for a batch; input always receives the matrix and builds one time embedding per block. and act coordinate by coordinate: , score divides by per coordinate, denoise uses Tweedie’s formula per coordinate.

Why: with one level for all coordinates a model learns the joint score at that level, and a query fixes some coordinates while the network still expects them noised. A block held at a low level is a clean condition, and the network’s output on the other blocks is the conditional score . A block at level 1 is noise, and the output is the marginal score of the others. One network then serves the joint, every conditional and every marginal, which a relation queried in every direction needs (MNIST in Both Directions §6–7).

  • Training: block_times(rng, pred, B; joint, clean) draws each block’s time independently from (a fraction joint of the columns shares one time; a fraction clean of the rest holds one block at exactly , the condition a query that holds a block clean uses, which uniform times reach in about 1% of the columns), and denoising_gradient(pred, x₀, T, ε, ps, st; weights) noises each coordinate at its block’s level. weights weights the loss per coordinate, or per entry as a matrix; a small block (a label) needs a large weight, and a block held clean can be left out (weight 0): it is a condition there. On MNIST this training is what made label → image samples clean (MNIST in Both Directions §9).
  • The closed-form mixture stays exact. With a variance per coordinate, ; its Jacobian is times a symmetric matrix, so the JVP and the VJP are now separate formulas.
  • An energy network has and Jacobian : and , one Hessian-vector product each. The mixed derivative takes as before.
  • On a device the time matrix moves with , and is spread over the coordinates there; tested on JLArrays.

Related: schedule, reddiff, factor, backends, The VP-SDE, Lux as a Parametric Lens, Scalar and Multivariate Energy