PADS: Physics-Aware, Model-Free Diffusion Training for Robust Generative Intelligence in Optical Neural Networks
Article Information
Abstract
Integrating diffusion models into physical computing systems such as optical neural networks (ONNs) enables low-power, high-speed generative intelligence but faces two critical barriers: the simulation-to-reality (sim2real) gap and the prohibitive computational cost of gradient-based training with differentiable simulators. The iterative denoising of diffusion models (typically 1000 steps) accumulates errors from physical imperfections (aberrations, misalignments, quantization), causing severe performance degradation when the models are deployed on real hardware. To address this, we propose PADS (Physics-Aware Diffusion Training via Score Gradient Estimation), a model-free optimization framework that removes simulator dependency by formulating physical parameter learning as a black-box optimization problem and optimizing the parameters directly with Monte Carlo score function estimation. Validated on a diffractive optical system with realistic non-idealities, PADS achieves FID scores of 38.7 on Fashion-MNIST (FMNIST) and 74.3 on CIFAR-10, outperforming simulator-based training (SBT) by 46% and enhanced hybrid training (HBT+) by 19% on CIFAR-10. It also reduces GPU memory consumption by 99% (from 14.8 GB to 0.2 GB), which makes edge deployment feasible. This work advances physical generative intelligence by modeling compound sim2real errors, developing a zeroth-order, model-free training framework, and providing experimental validation on a diffractive optical system.
Graphical Abstract
Keywords
Data Availability Statement
Funding
Conflicts of Interest
AI Use Statement
Ethical Approval and Consent to Participate
References
- Kazerouni, A., Aghdam, E. K., Heidari, M., Azad, R., Fayyaz, M., Hacihaliloglu, I., & Merhof, D. (2023). Diffusion models in medical imaging: A comprehensive survey. Medical Image Analysis, 88, 102846.
[CrossRef] [Google Scholar] - Kim, S. W., Philion, J., Torralba, A., & Fidler, S. (2021, June). DriveGAN: Towards a controllable high-quality neural simulation. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 5816-5825). IEEE.
[CrossRef] [Google Scholar] - Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022, June). High-resolution image synthesis with latent diffusion models. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 10674-10685). IEEE.
[CrossRef] [Google Scholar] - Bond-Taylor, S., Leach, A., Long, Y., & Willcocks, C. G. (2022). Deep generative modelling: A comparative review of VAEs, GANs, normalizing flows, energy-based and autoregressive models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), 7327-7347.
[CrossRef] [Google Scholar] - Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., & Lakshminarayanan, B. (2021). Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57), 1-64. https://www.jmlr.org/papers/v22/19-1028.html
[Google Scholar] - Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33, 6840-6851.
[Google Scholar] - Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020). Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456.
[CrossRef] [Google Scholar] - Wright, L. G., Onodera, T., Stein, M. M., Wang, T., Schachter, D. T., Hu, Z., & McMahon, P. L. (2022). Deep physical neural networks trained with backpropagation. Nature, 601(7894), 549-555.
[CrossRef] [Google Scholar] - Lin, X., Rivenson, Y., Yardimci, N. T., Veli, M., Luo, Y., Jarrahi, M., & Ozcan, A. (2018). All-optical machine learning using diffractive deep neural networks. Science, 361(6406), 1004-1008.
[CrossRef] [Google Scholar] - Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017, September). Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 23-30). IEEE.
[CrossRef] [Google Scholar] - Xue, Z., Zhou, T., Xu, Z., Yu, S., Dai, Q., & Fang, L. (2024). Fully forward mode training for optical neural networks. Nature, 632(8024), 280-286.
[CrossRef] [Google Scholar] - Zhao, G., Shu, X., & Zhou, R. (2025). High-performance real-world optical computing trained by in situ gradient-based model-free optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(9), 7194-7205.
[CrossRef] [Google Scholar] - Song, J., Meng, C., & Ermon, S. (2020). Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502.
[CrossRef] [Google Scholar] - Xiao, Z., Kreis, K., & Vahdat, A. (2021). Tackling the generative learning trilemma with denoising diffusion GANs. arXiv preprint arXiv:2112.07804.
[CrossRef] [Google Scholar] - Oguz, I., Dinc, N. U., Yildirim, M., Ke, J., Yoo, I., Wang, Q., ... & Psaltis, D. (2024). Optical diffusion models for image generation. Advances in Neural Information Processing Systems, 37, 59150-59173.
[CrossRef] [Google Scholar] - Chen, S., Li, Y., Wang, Y., Chen, H., & Ozcan, A. (2025). Optical generative models. Nature, 644(8078), 903-911.
[CrossRef] [Google Scholar] - Wang, Y., Zhong, M., Liao, K., Sun, Q., Hu, X., & Gong, Q. (2026). Diffractive optical neural networks: Pathways toward intelligent and scalable photonic computing. Nanophotonics, 15(8), e70098.
[CrossRef] [Google Scholar] - Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4), 229-256.
[CrossRef] [Google Scholar] - Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., ... & Kavukcuoglu, K. (2016, June). Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning (pp. 1928-1937). PMLR. https://proceedings.mlr.press/v48/mniha16.html
[Google Scholar] - Wang, Z., Zoghi, M., Hutter, F., Matheson, D., & de Freitas, N. (2016). Bayesian optimization in a billion dimensions via random embeddings. Journal of Artificial Intelligence Research, 55, 361-387.
[CrossRef] [Google Scholar] - Mohamed, S., Rosca, M., Figurnov, M., & Mnih, A. (2020). Monte Carlo gradient estimation in machine learning. Journal of Machine Learning Research, 21(132), 1-62. https://www.jmlr.org/papers/v21/19-346.html
[Google Scholar] - Jang, E., Gu, S., & Poole, B. (2016). Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144.
[CrossRef] [Google Scholar] - Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
[CrossRef] [Google Scholar] - Matsushima, K., & Shimobaba, T. (2009). Band-limited angular spectrum method for numerical simulation of free-space propagation in far and near fields. Optics Express, 17(22), 19662-19673.
[CrossRef] [Google Scholar] - Xiao, H., Rasul, K., & Vollgraf, R. (2017). Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747.
[CrossRef] [Google Scholar] - Krizhevsky, A. (2009). Learning multiple layers of features from tiny images (Technical report). University of Toronto. Retrieved from https://cave.cs.toronto.edu/kriz/learning-features-2009-TR.pdf
[Google Scholar] - Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017). GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Advances in Neural Information Processing Systems, 30, 6626-6637.
[Google Scholar] - Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., & Aila, T. (2019). Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32.
[Google Scholar] - Nichol, A. Q., & Dhariwal, P. (2021, July). Improved denoising diffusion probabilistic models. In International Conference on Machine Learning (pp. 8162-8171). PMLR. https://proceedings.mlr.press/v139/nichol21a.html
[Google Scholar] - Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., & Aila, T. (2020). Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33, 12104-12114.
[Google Scholar] - Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35, 26565-26577.
[CrossRef] [Google Scholar] - Brühl, E., Buckup, T., & Motzkus, M. (2017). Minimization of $1/f^n$ phase noise in liquid crystal masks for reliable femtosecond pulse shaping. Optics Express, 25(19), 23376-23386.
[CrossRef] [Google Scholar] - Matsumoto, N., Itoh, H., Inoue, T., Otsu, T., & Toyoda, H. (2014). Stable and flexible multiple spot pattern generation using LCOS spatial light modulator. Optics Express, 22(20), 24722.
[CrossRef] [Google Scholar] - Salimans, T., & Ho, J. (2022). Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512.
[CrossRef] [Google Scholar] - Zhou, T., Lin, X., Wu, J., Chen, Y., Xie, H., Li, Y., Fan, J., Wu, H., Fang, L., & Dai, Q. (2021). Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit. Nature Photonics, 15(5), 367-373.
[CrossRef] [Google Scholar]
Cite This Article
TY - JOUR AU - Huang, Qibao AU - Zhao, Ai'an AU - Xu, Xinye PY - 2026 DA - 2026/11/20 TI - PADS: Physics-Aware, Model-Free Diffusion Training for Robust Generative Intelligence in Optical Neural Networks JO - Optical Wireless Communication T2 - Optical Wireless Communication JF - Optical Wireless Communication VL - 1 IS - 1 SP - 60 EP - 75 DO - 10.62762/OWC.2026.824441 UR - https://www.icck.org/article/abs/owc.2026.824441 KW - diffusion models KW - physical neural networks KW - model-free optimization KW - zeroth-order optimization KW - optical computing AB - Integrating diffusion models into physical computing systems such as optical neural networks (ONNs) enables low-power, high-speed generative intelligence but faces two critical barriers: the simulation-to-reality (sim2real) gap and the prohibitive computational cost of gradient-based training with differentiable simulators. The iterative denoising of diffusion models (typically 1000 steps) accumulates errors from physical imperfections (aberrations, misalignments, quantization), causing severe performance degradation when the models are deployed on real hardware. To address this, we propose PADS (Physics-Aware Diffusion Training via Score Gradient Estimation), a model-free optimization framework that removes simulator dependency by formulating physical parameter learning as a black-box optimization problem and optimizing the parameters directly with Monte Carlo score function estimation. Validated on a diffractive optical system with realistic non-idealities, PADS achieves FID scores of 38.7 on Fashion-MNIST (FMNIST) and 74.3 on CIFAR-10, outperforming simulator-based training (SBT) by 46% and enhanced hybrid training (HBT+) by 19% on CIFAR-10. It also reduces GPU memory consumption by 99% (from 14.8 GB to 0.2 GB), which makes edge deployment feasible. This work advances physical generative intelligence by modeling compound sim2real errors, developing a zeroth-order, model-free training framework, and providing experimental validation on a diffractive optical system. SN - 5 Articles Required PB - Institute of Central Computation and Knowledge LA - English ER -
@article{Huang2026PADS,
author = {Qibao Huang and Ai'an Zhao and Xinye Xu},
title = {PADS: Physics-Aware, Model-Free Diffusion Training for Robust Generative Intelligence in Optical Neural Networks},
journal = {Optical Wireless Communication},
year = {2026},
volume = {1},
number = {1},
pages = {60-75},
doi = {10.62762/OWC.2026.824441},
url = {https://www.icck.org/article/abs/owc.2026.824441},
abstract = {Integrating diffusion models into physical computing systems such as optical neural networks (ONNs) enables low-power, high-speed generative intelligence but faces two critical barriers: the simulation-to-reality (sim2real) gap and the prohibitive computational cost of gradient-based training with differentiable simulators. The iterative denoising of diffusion models (typically 1000 steps) accumulates errors from physical imperfections (aberrations, misalignments, quantization), causing severe performance degradation when the models are deployed on real hardware. To address this, we propose PADS (Physics-Aware Diffusion Training via Score Gradient Estimation), a model-free optimization framework that removes simulator dependency by formulating physical parameter learning as a black-box optimization problem and optimizing the parameters directly with Monte Carlo score function estimation. Validated on a diffractive optical system with realistic non-idealities, PADS achieves FID scores of 38.7 on Fashion-MNIST (FMNIST) and 74.3 on CIFAR-10, outperforming simulator-based training (SBT) by 46\% and enhanced hybrid training (HBT+) by 19\% on CIFAR-10. It also reduces GPU memory consumption by 99\% (from 14.8 GB to 0.2 GB), which makes edge deployment feasible. This work advances physical generative intelligence by modeling compound sim2real errors, developing a zeroth-order, model-free training framework, and providing experimental validation on a diffractive optical system.},
keywords = {diffusion models, physical neural networks, model-free optimization, zeroth-order optimization, optical computing},
issn = {5 Articles Required},
publisher = {Institute of Central Computation and Knowledge}
}
Article Metrics
Publisher's Note
ICCK stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and Permissions
Copyright © 2026 by the Author(s). Published by Institute of Central Computation and Knowledge. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
Portico