DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models

Motivation

Deploying flow-matching models across objectives such as prompt following, text rendering, and aesthetics exposes a fundamental trade-off. Reinforcement learning can improve beyond existing models but suffers from high-variance trajectory-level gradients and cross-task interference. On-policy distillation is stable and dense, but imitation-based: a shared student merely interpolates among specialized teachers rather than exceeding them.

• Beyond imitation. We derive a closed-form target that extends flow-based OPD from teacher imitation to reward extrapolation.
• Reward interpretation. We characterize conditions under which the teacher–reference contrast is reward-aligned, and introduce controlled reference degradation to strengthen it.
• Strong results. Across single- and multi-teacher settings, DreOPD achieves the best average performance while surpassing specialized teachers on most metrics.

Method

DreOPD converts trajectory-level implicit reward extrapolation into a per-state, closed-form velocity target, so that extrapolative post-training reduces to stable velocity regression on student rollouts.

1. Implicit reward extrapolation objective

Generalized OPD reinterprets teacher matching through the implicit reward log(π_T/π_ref) and introduces a scaling coefficient λ. At λ=1 the reference terms cancel and it reduces to reverse-KL teacher matching; the regime λ>1 (ExOPD) amplifies the teacher–reference contrast beyond imitation.

2. Closed-form extrapolated velocity target

Under shared-covariance Gaussian flow transitions, the conditional objective at each student-visited state is strongly convex in the velocity and admits a unique closed-form optimizer. This turns a trajectory-level contrast into a simple per-state target, avoiding likelihood-ratio gradients over the denoising path.

At λ=1 this recovers teacher imitation v_λ^⋆=v_T; for λ>1 it moves beyond the teacher along the teacher–reference direction. The displacement from the teacher makes the roles of extrapolation strength and reference explicit:

3. On-policy velocity regression

We train the student by regressing toward the (stop-gradient) target on its own current rollouts, giving dense and stable supervision with the extrapolative direction baked into the target.

4. Reward interpretation & degraded reference

At the distribution level, the optimum is a reward tilt of the reference, and the expected reward increases monotonically with λ. Since the step beyond the teacher scales with v_T−v_ref, a mildly degraded reference enlarges this reward-aligned contrast while preserving generative structure, strengthening extrapolation.

Results

We evaluate DreOPD with SD3.5-M at 512×512 across three text-to-image objectives: compositional correctness (GenEval), text rendering (OCR), and aesthetic quality (PickScore, ClipScore, HPSv2.1), plus out-of-domain Aesthetic and ImageReward.

Single-Teacher Distillation

Single-teacher distillation on three independent tasks. GenEval and OCR are reported by their respective students; PickScore, ClipScore, and HPSv2.1 by the aesthetics student. The Teacher row combines the in-domain scores of three task-specific teachers. Bold: best. Subscripts show absolute changes from the corresponding teacher (green = gain, red = drop). DeRef. = degraded reference.

Multi-Teacher Distillation

Single-task RL, multi-task RL, and multi-teacher OPD. Bold: best; Blue rows: Ours; Gray cells: in-domain reward. Average is the mean of min-max normalized scores across all metrics. Subscripts show absolute changes from the corresponding teacher. DreOPD reaches the best average (0.9939) and surpasses teachers on most metrics.

Ablation Studies

Impact of the extrapolation factor λ

Ablation on λ. In-domain: GenEval (GE), OCR, PickScore (Pick), ClipScore (Clip), HPSv2.1 (HPS); OOD: Aesthetic (Aes), ImageReward (IR). Bold: best. Extrapolation (λ>1) improves over teacher matching, peaking around λ=1.25.

Degraded reference construction

Ablation on the degraded reference. Bold: best. Mild velocity quantization (8-bit) yields the strongest, most structurally faithful teacher–reference contrast; overly aggressive degradation captures corruption artifacts rather than task-relevant differences.

Noise level & sampling

Additional Qualitative Results

Additional examples across the three objectives. Use the tabs to switch between tasks.

Conclusion

DreOPD translates trajectory-level implicit reward extrapolation into a closed-form velocity target, extending on-policy distillation from teacher imitation to teacher–reference extrapolation while retaining regression-based training. A mildly degraded reference strengthens a reward-aligned contrast without disrupting generative structure. Across single- and multi-teacher settings, DreOPD achieves the best average performance while surpassing teachers on most metrics, establishing reward extrapolation as an effective framework for consolidating specialized flow models.

BibTeX

@article{lin2026dreopd,
  title={DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models},
  author={Lin, Mingfeng and Cai, Chengfei and Xu, Lin and Wei, Yuxiang and Han, Liang},
  journal={arXiv preprint arXiv:2608.09233},
  year={2026}
}