Hitchhiker's Guide to Super-Resolution: Introduction and Recent Advances

German Research Center for Artificial Intelligence
RPTU Kaiserslautern-Landau
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023


With the advent of Deep Learning (DL), Super-Resolution (SR) has also become a thriving research area. However, despite promising results, the field still faces challenges that require further research e.g., allowing flexible upsampling, more effective loss functions, and better evaluation metrics. We review the domain of SR in light of recent advances, and examine state-of-the-art models such as diffusion (DDPM) and transformer-based SR models. We present a critical discussion on contemporary strategies used in SR, and identify promising yet unexplored research directions. We complement previous surveys by incorporating the latest developments in the field such as uncertainty-driven losses, wavelet networks, neural architecture search, novel normalization methods, and the latests evaluation techniques. We also include several visualizations for the models and methods throughout each chapter in order to facilitate a global understanding of the trends in the field. This review is ultimately aimed at helping researchers to push the boundaries of DL applied to SR.


Super-Resolution (SR) is the process of enhancing Low-Resolution (LR) images to High-Resolution (HR). The applications range from natural images to highly advanced satellite, and medical imaging. Despite its long history, SR remains a challenging task in computer vision because it is notoriously ill-posed: several HR images can be valid for any given LR image due to many aspects like brightness and coloring. The fundamental uncertainties in the relation between LR and HR images pose a complex research task. Thanks to rapid advances in Deep Learning (DL), SR has made significant progress in recent years. Unfortunately, entry into this field is overwhelming because of the abundance of publications. It is knotty work to get an overview of the advantages and disadvantages between publications. This OpenAcess work unwinds some of the most representative advances in the crowded field of SR. It discusses common knowledge in image SR, standard procedures, and, most importantly, recent advances and future directions. New researchers will find this work as an easy starting point to get into the topic of image SR and more advanced researchers will get updated on current trends and can use this work as reference for known procedures, metrics and explanations.

The work commences with an introduction to the basic definitions and terminology. It then proceeds to discuss evaluation metrics for SR solutions. Along the way, we introduce various datasets that provide diverse data types, like 8K resolution images or video sequences. It further delves into a discussion of different learning objectives of Super-Resolution: Regression-based SR (also with uncertainty), Generative SR, and, more recently, Denoising Diffusion Probabilistic Models.


This work also dives into the world of various upsampling methods, a critical aspect of image SR. It covers the whys and hows of interpolation-based and learning-based upsampling techniques and also problems appearing with upsampling techniques (e.g., artifacts) in supplemental material. Also, our work explores the challenges and potential solutions of flexible upsampling for real-world scenarios with arbitrary scaling factors.

Moreover, it surveys additional learning strategies such as curriculum learning, enhanced predictions, learned degradation, network fusion, multi-task learning, and normalization techniques like the Adaptive Deviation Modulator (AdaDM).


This paper examines in detail different SR models and architectures and provides comprehensive explanations as well as unified visualizations. Architectures are categorized into different branches of image SR, such as: Pre-upsampling, Post-upsampling, Progressive Upsampling, and Iterative Up-and-Down Upsampling.


We discuss these primary categories: Simple Networks, Residual Networks, Recurrent-Based Networks, Lightweight and Wavelet-based Models. These various state-of-the-art architectures are examined with their pros and cons. Finally, you will get a scoop of Unsupervised SR and Neural Architecture Search for SR! At last, but not least, we will discuss the future of SR and the challenges that lie ahead. So, gather your party and embark on this thrilling adventure, delving into the diverse and intricate world of deep learning-based super-resolution networks and unlock its secrets! Spoiler-Warning: Big tables included.



  author={Moser, Brian B. and Raue, Federico and Frolov, Stanislav and Palacio, Sebastian and Hees, Jörn and Dengel, Andreas},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  title={Hitchhiker's Guide to Super-Resolution: Introduction and Recent Advances},