NoisePredictorwraps a Lux model as and derives from it the two quantities the rest of the package needs: the score and the denoiser.
Sources: code:
predictor.jl; noise blocks: Bao et al., One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale, ICML 2023 (UniDiffuser)Bibliography: Bao et al. 2023
Theory (CT-ML wiki): Parametric Lens · Lens
1. LuxCore, not Lux
The prompt asks the package to “wrap some Lux.jl model”. It does — and Lux is not a
dependency. Any Lux layer is a LuxCore.AbstractLuxLayer, and the wrapper only ever calls
LuxCore.apply(model, input, ps, st). So Lux, Zygote, Optimisers and the rest stay out
of the dependency tree, exactly as the parent package depends on LuxCore rather than Lux
(Lux as a Parametric Lens §“What is kept unchanged”).
NoisePredictor <: LuxCore.AbstractLuxWrapperLayer, which is the LuxCore trick for
“my parameters are my child’s parameters, with no extra nesting”. So
LuxCore.setup(rng, NoisePredictor(unet, VPSDE())) == LuxCore.setup(rng, unet)and wrapping costs nothing. This is the one place where the project’s decision to mirror the LuxCore API name-for-name pays off directly rather than aspirationally.
2. Three quantities, one network
The second and third are derived, not learned, and that they are derived is the whole reason a noise predictor can serve as a prior rather than only as a sampler.
The score identity is exact for the perturbation kernel: with ,
so a perfect is a perfect score. The denoiser is Tweedie’s formula, and
Tweedie is exact, and the test suite proves it
For Gaussian data the closed-form posterior mean is , and
denoisereproduces it to floating point, for every tested. Tweedie’s formula turns a noise predictor into an MMSE denoiser; that is the precise sense in which a diffusion model is a prior, and it is what the RED-Diff regulariser scores against.
3. The input adapter
Diffusion architectures disagree about how arrives: a second positional argument, a
sinusoidal embedding, a concatenated channel, a NamedTuple. The wrapper takes an input
function, defaulting to (x, t) -> (x, t), and knows nothing about time embeddings —
that is the model’s business.
The cost is that a wrong input fails inside the user’s model with the user’s error message
rather than here with a good one. There is no way to validate it without knowing the
architecture, so it is not validated.
4. Implementation difficulties
4.1 Forward-only where it can be, any AD backend where it cannot
epsilon is the only place the network is evaluated forwards, and RED-Diff never needs a
reverse pass through model. That is RED-Diff’s stop-gradient, promoted to an architectural
property: sampling and RED-Diff work with no AD at all. reddiff.md §5.
The implicit learner is different. Its Newton steps want the input Jacobian, and its backward
pass wants a parameter VJP. Both go through the ad field — any ADTypes object — and are
supplied by package extensions, so the package still has no AD dependency of its own. epsilon
itself calls the hook _apply(ad, …), which a backend may replace by a compiled forward pass
(ad = AutoReactant()). See backends. Training by denoising score
matching is ordinary Lux training (examples/circle_mlp.jl).
4.2 The 1/σ_t in score is a real singularity
score divides by , so it is unusable at and inaccurate for small even
with the tmin floor. Nothing in this package calls score — the RED-Diff gradient is
expressed in -space precisely to avoid it — so it exists for completeness and
for anyone building a sampler on top. If you call it near , that is on you.
4.3 denoising_loss returns a scalar, so it cannot be a vector energy
Scalar and Multivariate Energy argues that a factor should expose its vector residual,
because the implicit function theorem needs the Jacobian. denoising_loss collapses to
sum(abs2, ε̂ - ε) immediately.
The vector residual is right there — — and returning it
would let SquaredNorm do the scalarisation and would make the diffusion factor’s energy
space rather than . It is not done because RED-Diff’s gradient is
already in closed form and the Jacobian is exactly what the method refuses to compute; the
vector form would be honest bookkeeping with nothing downstream to consume it. Recorded as
a genuine inconsistency with the vault’s stated design, with a reason.
5. Running on a device
NoisePredictor(model, schedule; input, ad, device = (to = gpu_device(), back = cpu_device()))
runs the network on an accelerator and leaves everything else where it is. The solvers’ state,
the field’s noise draws, the small dense blocks and the Krylov vectors stay on the CPU. Every
entry point that evaluates or differentiates the network (epsilon, energy,
energy_columns, the input Jacobian and its blocks, the input VJP and JVP, the parameter VJP)
moves its array arguments with to, calls a copy of the predictor without a device, and
returns its result through back. Parameter cotangents are not moved back: they belong to the
parameters, which the caller has put on the device.
Two decisions make this fast rather than merely correct.
- Derivatives happen on the device. The first version wrapped the model in a layer that moved its input. Forward-mode dual numbers were then made on the CPU, converted, and moved in and out of every Hessian-vector product: 234 ms for one product over 4000 columns, where the device itself needs about 20. With the device at the predictor’s entry points the duals are created on the device, and the same product takes 21 ms.
- No CPU-only constructions in the hooks.
collectbrings a GPU array home and a basis vector made withzeroslives on the CPU. The DifferentiationInterface extension copies withsimilarand moves its constants next to the input. On a device it builds Jacobian and Hessian blocks from one product per free coordinate instead of a chunked Jacobian.
The input function receives device arrays and must build its time features there. Eight
numbers per column are made on the CPU and copied next to x. MLDataDevices’ gpu_device()
also converts Float64 to Float32; a consumer GPU computes Float64 at 1/64 of the rate.
The tests run every hook on JLArrays, the reference GPU array type that runs on the CPU and forbids scalar indexing, so a CPU-only construction fails in CI without a GPU. The measured costs on CUDA are in energy_network §8.
6. A noise level per block of coordinates
NoisePredictor(model, schedule; input, blocks = [1:784, 785:794]) partitions the coordinates
into blocks that may sit at different noise levels, as UniDiffuser does for image and text
(Bao et al. 2023). Time is then a number (every block at one level), a vector with one time per
block, or a matrix for a batch; input always receives the matrix and
builds one time embedding per block. and act coordinate by coordinate:
, score divides by per coordinate,
denoise uses Tweedie’s formula per coordinate.
Why: with one level for all coordinates a model learns the joint score at that level, and a query fixes some coordinates while the network still expects them noised. A block held at a low level is a clean condition, and the network’s output on the other blocks is the conditional score . A block at level 1 is noise, and the output is the marginal score of the others. One network then serves the joint, every conditional and every marginal, which a relation queried in every direction needs (MNIST in Both Directions §6–7).
- Training:
block_times(rng, pred, B; joint, clean)draws each block’s time independently from (a fractionjointof the columns shares one time; a fractioncleanof the rest holds one block at exactly , the condition a query that holds a block clean uses, which uniform times reach in about 1% of the columns), anddenoising_gradient(pred, x₀, T, ε, ps, st; weights)noises each coordinate at its block’s level.weightsweights the loss per coordinate, or per entry as a matrix; a small block (a label) needs a large weight, and a block held clean can be left out (weight 0): it is a condition there. On MNIST this training is what made label → image samples clean (MNIST in Both Directions §9). - The closed-form mixture stays exact. With a variance per coordinate, ; its Jacobian is times a symmetric matrix, so the JVP and the VJP are now separate formulas.
- An energy network has and Jacobian : and , one Hessian-vector product each. The mixed derivative takes as before.
- On a device the time matrix moves with , and is spread over the coordinates there; tested on JLArrays.
Related: schedule, reddiff, factor, backends, The VP-SDE, Lux as a Parametric Lens, Scalar and Multivariate Energy