A05 | Image/Video Quality Assessment: From Test Databases to Similarity-Aware and Perceptual Dynamic Metrics

Prof. Dietmar Saupe, University of Konstanz
Email | Website

Dietmar Saupe

Prof. Andrés Bruhn, University of Stuttgart
Email | Website

Andrés Bruhn

Project A05  [completed]

Dr. Vlad Hosu, University of Konstanz – Email | Website

Oliver Wiedemann, University of Konstanz – Email | Website

The project addresses methods for automated visual quality assessment and their validation beyond mean opinion scores. We propose to enhance the methods by including similarity awareness and predicted eye movement sequences, quantifying the perceptual viewing experience, and to apply the metrics for quality-aware media processing. Moreover, we will set up and apply media databases that are diverse in content and authentic in the distortions, in contrast to current scientific data sets.

Research Questions

How can crowdsourcing be applied to help generating very large video media data bases for research applications in quality of multimedia?

What is the performance of state-of-the-art video quality assessment methods that were designed based on small training sets for such large and diversified media databases?

Quality assessment in such extremely large empirical studies requires crowdsourcing. How should that be organized to achieve sufficient reliability and efficiency?

Are machine learning techniques suitable to identify the best performing video quality assessment metrics for given media content?

What statistical/perceptual features should be extracted to express similarity for this task? 

How can one design new or hybrid strategies for video quality assessment based on the above?

Can we improve methods for image/video quality assessment by studying patterns of human visual attention and other perceptual aspects?

How can knowledge on human visual attention derived from eyetracking studies be incorporated into perceptual image/video quality assessment methods?

How can the quality assessment methods be applied in quality-aware media processing such as perceptual coding?

Fig. 1: Training Better Algorithms to Predict Subjective Quality Opinions.

Fig. 2: Saliency Driven Compression.

Publications

  1. M. Jenadeleh et al., “An Image Quality Dataset with Triplet Comparisons for Multi-dimensional Scaling,” 2024, IEEE. doi: 10.1109/qomex61742.2024.10598258.
  2. D. Saupe and S. Hviid del Pin, “National differences in image quality assessment: An investigation on three large-scale IQA datasets,” in 2024 16th International Conference on Quality of Multimedia Experience (QoMEX), IEEE, Ed., in 2024 16th International Conference on Quality of Multimedia Experience (QoMEX). IEEE, May 2024, pp. 214–220. doi: 10.1109/qomex61742.2024.10598250.
  3. S. Su et al., “Going the Extra Mile in Face Image Quality Assessment: A Novel Database and Model,” IEEE Transactions on Multimedia, vol. 26, pp. 2671–2685, 2024, doi: 10.1109/tmm.2023.3301276.
  4. M. Jenadeleh, R. Hamzaoui, U.-D. Reips, and D. Saupe, “Crowdsourced Estimation of Collective Just Noticeable Difference for Compressed Video with the Flicker Test and QUEST+,” IEEE Transactions on Circuits and Systems for Video Technology, pp. 1–1, May 2024, doi: 10.1109/tcsvt.2024.3402363.
  5. M. Jenadeleh, A. Heß, S. Hviid del Pin, E. Gamboa, M. Hirth, and D. Saupe, “Impact of feedback on crowdsourced visual quality assessment with paired comparisons,” in 2024 16th International Conference on Quality of Multimedia Experience (QoMEX), IEEE, Ed., in 2024 16th International Conference on Quality of Multimedia Experience (QoMEX). IEEE, May 2024, pp. 125–131. doi: 10.1109/qomex61742.2024.10598256.
  6. M. Testolina, V. Hosu, M. Jenadeleh, D. Lazzarotto, D. Saupe, and T. Ebrahimi, “JPEG AIC-3 Dataset: Towards Defining the High Quality to Nearly Visually Lossless Quality Range,” in 15th International Conference on Quality of Multimedia Experience (QoMEX), in 15th International Conference on Quality of Multimedia Experience (QoMEX). 2023, pp. 55–60. [Online]. Available: https://ieeexplore.ieee.org/document/10178554
  7. G. Chen, H. Lin, O. Wiedemann, and D. Saupe, “Localization of Just Noticeable Difference for Image Compression,” in 2023 15th International Conference on Quality of Multimedia Experience (QoMEX), in 2023 15th International Conference on Quality of Multimedia Experience (QoMEX). Jun. 2023, pp. 61–66. doi: 10.1109/QoMEX58391.2023.10178653.
  8. O. Wiedemann, V. Hosu, S. Su, and D. Saupe, “Konx: cross-resolution image quality assessment,” Quality and User Experience, vol. 8, no. 1, Art. no. 1, Aug. 2023, doi: 10.1007/s41233-023-00061-8.
  9. M. Jenadeleh, J. Zagermann, H. Reiterer, U.-D. Reips, R. Hamzaoui, and D. Saupe, “Relaxed forced choice improves performance of visual quality assessment methods,” in 2023 15th International Conference on Quality of Multimedia Experience (QoMEX), in 2023 15th International Conference on Quality of Multimedia Experience (QoMEX). 2023, pp. 37–42. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10178467
  10. X. Zhao et al., “CUDAS: Distortion-Aware Saliency Benchmark,” IEEE Access, vol. 11, pp. 58025–58036, Jun. 2023, doi: 10.1109/access.2023.3283344.
  11. F. Götz-Hahn, V. Hosu, and D. Saupe, “Critical Analysis on the Reproducibility of Visual Quality Assessment Using Deep Features,” PLoS ONE, vol. 17, no. 8, Art. no. 8, 2022, [Online]. Available: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0269715
  12. J. Lou, H. Lin, D. Marshall, D. Saupe, and H. Liu, “TranSalNet: Towards perceptually relevant visual saliency prediction,” Neurocomputing, vol. 494, pp. 455–467, 2022, [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0925231222004714
  13. H. Lin, H. Men, Y. Yan, J. Ren, and D. Saupe, “Crowdsourced Quality Assessment of Enhanced Underwater Images - a Pilot Study,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). IEEE, Sep. 2022, pp. 1–4. [Online]. Available: https://ieeexplore.ieee.org/document/9900904
  14. M. Zameshina et al., “Fairness in generative modeling: do it unsupervised!,” in Proceedings of the Genetic and Evolutionary Computation Conference Companion, in Proceedings of the Genetic and Evolutionary Computation Conference Companion. ACM, Jul. 2022, pp. 320–323. doi: 10.1145/3520304.3528992.
  15. H. Lin et al., “Large-Scale Crowdsourced Subjective Assessment of Picturewise Just Noticeable Difference,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 9, Art. no. 9, 2022, [Online]. Available: https://ieeexplore.ieee.org/document/9745537
  16. B. Roziere et al., “EvolGAN: Evolutionary Generative Adversarial Networks,” in Computer Vision -- ACCV 2020, in Computer Vision -- ACCV 2020. Cham: Springer International Publishing, Nov. 2021, pp. 679–694. [Online]. Available: https://openaccess.thecvf.com/content/ACCV2020/html/Roziere_EvolGAN_Evolutionary_Generative_Adversarial_Networks_ACCV_2020_paper.html
  17. F. Götz-Hahn, V. Hosu, H. Lin, and D. Saupe, “KonVid-150k : A Dataset for No-Reference Video Quality Assessment of Videos in-the-Wild,” IEEE Access, vol. 9, pp. 72139–72160, 2021, doi: 10.1109/ACCESS.2021.3077642.
  18. H. Lin, G. Chen, and F. W. Siebert, “Positional Encoding: Improving Class-Imbalanced Motorcycle Helmet use Classification,” in 2021 IEEE International Conference on Image Processing (ICIP), in 2021 IEEE International Conference on Image Processing (ICIP). 2021, pp. 1194–1198. [Online]. Available: https://ieeexplore.ieee.org/document/9506178
  19. H. Men, H. Lin, M. Jenadeleh, and D. Saupe, “Subjective Image Quality Assessment with Boosted Triplet Comparisons,” IEEE Access, vol. 9, pp. 138939–138975, 2021, [Online]. Available: https://ieeexplore.ieee.org/abstract/document/9559922
  20. S. Su, V. Hosu, H. Lin, Y. Zhang, and D. Saupe, “KonIQ++: Boosting No-Reference Image Quality Assessment in the Wild by Jointly Predicting Image Quality and Defects,” in 32nd British Machine Vision Conference, in 32nd British Machine Vision Conference. 2021, pp. 1–12. [Online]. Available: https://www.bmvc2021-virtualconference.com/assets/papers/0868.pdf
  21. B. Roziere et al., “Tarsier: Evolving Noise Injection in Super-Resolution GANs,” in 2020 25th International Conference on Pattern Recognition (ICPR), in 2020 25th International Conference on Pattern Recognition (ICPR). 2021, pp. 7028–7035. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/9413318
  22. T. Guha et al., “ATQAM/MAST’20: Joint Workshop on Aesthetic and Technical Quality Assessment of Multimedia and Media Analytics for Societal Trends,” in Proceedings of the 28th ACM International Conference on Multimedia, in Proceedings of the 28th ACM International Conference on Multimedia. New York, NY, USA: Association for Computing Machinery, 2020, pp. 4758–4760. doi: 10.1145/3394171.3421895.
  23. H. Lin, M. Jenadeleh, G. Chen, U.-D. Reips, R. Hamzaoui, and D. Saupe, “Subjective Assessment of Global Picture-Wise Just Noticeable Difference,” in Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), in Proceedings of the IEEE International Conference on Multimedia and Expo (ICME). 2020, pp. 1–6. [Online]. Available: https://ieeexplore.ieee.org/document/9106058
  24. H. Men, V. Hosu, H. Lin, A. Bruhn, and D. Saupe, “Visual Quality Assessment for Interpolated Slow-Motion Videos Based on a Novel Database,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). 2020, pp. 1–6. [Online]. Available: https://ieeexplore.ieee.org/document/9123096/authors#authors
  25. H. Lin, J. D. Deng, D. Albers, and F. W. Siebert, “Helmet Use Detection of Tracked Motorcycles Using CNN-Based Multi-Task Learning,” IEEE Access, vol. 8, pp. 162073–162084, 2020, [Online]. Available: https://ieeexplore.ieee.org/abstract/document/9184871
  26. B. Roziere et al., “Evolutionary Super-Resolution,” in Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, in Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion. New York, NY, USA: Association for Computing Machinery, 2020, pp. 151–152. doi: 10.1145/3377929.3389959.
  27. V. Hosu, H. Lin, T. Szirányi, and D. Saupe, “KonIQ-10k : An Ecologically Valid Database for Deep Learning of Blind Image Quality Assessment,” IEEE Transactions on Image Processing, vol. 29, pp. 4041–4056, 2020, [Online]. Available: https://ieeexplore.ieee.org/document/8968750
  28. O. Wiedemann and D. Saupe, “Gaze Data for Quality Assessment of Foveated Video,” in ACM Symposium on Eye Tracking Research and Applications, in ACM Symposium on Eye Tracking Research and Applications. New York, NY, USA: Association for Computing Machinery, 2020. doi: 10.1145/3379157.3391656.
  29. H. Lin et al., “SUR-FeatNet: Predicting the Satisfied User Ratio Curvefor Image Compression with Deep Feature Learning,” Quality and User Experience, vol. 5, no. 1, Art. no. 1, 2020, doi: 10.1007/s41233-020-00034-1.
  30. M. Jenadeleh, M. Pedersen, and D. Saupe, “Blind Quality Assessment of Iris Images Acquired in Visible Light for Biometric Recognition,” Sensors, vol. 20, no. 5, Art. no. 5, 2020, [Online]. Available: https://www.mdpi.com/1424-8220/20/5/1308
  31. M. Lan Ha, V. Hosu, and V. Blanz, “Color Composition Similarity and Its Application in Fine-grained Similarity,” in 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), in 2020 IEEE Winter Conference on Applications of Computer Vision (WACV). Piscataway, NJ: IEEE, 2020, pp. 2548–2557. [Online]. Available: https://ieeexplore.ieee.org/document/9093522
  32. O. Wiedemann, V. Hosu, H. Lin, and D. Saupe, “Foveated Video Coding for Real-Time Streaming Applications,” in 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX), in 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX). 2020, pp. 1–6. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/9123080
  33. H. Men, V. Hosu, H. Lin, A. Bruhn, and D. Saupe, “Subjective annotation for a frame interpolation benchmark using artefact amplification,” Quality and User Experience, vol. 5, no. 1, Art. no. 1, 2020, [Online]. Available: https://link.springer.com/article/10.1007%2Fs41233-020-00037-y
  34. X. Zhao, H. Lin, P. Guo, D. Saupe, and H. Liu, “Deep Learning VS. Traditional Algorithms for Saliency Prediction of Distorted Images,” in 2020 IEEE International Conference on Image Processing (ICIP), in 2020 IEEE International Conference on Image Processing (ICIP). 2020, pp. 156–160. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/9191203
  35. V. Hosu et al., “From Technical to Aesthetics Quality Assessment and Beyond: Challenges and Potential,” in Joint Workshop on Aesthetic and Technical Quality Assessment of Multimedia and Media Analytics for Societal Trends, in Joint Workshop on Aesthetic and Technical Quality Assessment of Multimedia and Media Analytics for Societal Trends. New York, NY, USA: Association for Computing Machinery, 2020, pp. 19–20. doi: 10.1145/3423268.3423589.
  36. H. Men, H. Lin, V. Hosu, D. Maurer, A. Bruhn, and D. Saupe, “Visual Quality Assessment for Motion Compensated Frame Interpolation,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2019, pp. 1–6. [Online]. Available: https://ieeexplore.ieee.org/document/8743221
  37. H. Lin, V. Hosu, and D. Saupe, “KADID-10k: A Large-scale Artificially Distorted IQA Database,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2019, pp. 1–3. [Online]. Available: https://ieeexplore.ieee.org/document/8743252
  38. C. Fan et al., “SUR-Net: Predicting the Satisfied User Ratio Curve for Image Compression with Deep Learning,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2019, pp. 1–6. [Online]. Available: https://ieeexplore.ieee.org/document/8743204
  39. V. Hosu, B. Goldlücke, and D. Saupe, “Effective Aesthetics Prediction with Multi-level Spatially Pooled Features,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9367–9375, 2019, [Online]. Available: https://ieeexplore.ieee.org/document/8953497
  40. V. Hosu, H. Lin, and D. Saupe, “Expertise Screening in Crowdsourcing Image Quality,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2018, pp. 276–281. [Online]. Available: https://ieeexplore.ieee.org/document/8463427
  41. H. Men, H. Lin, and D. Saupe, “Spatiotemporal Feature Combination Model for No-Reference Video Quality Assessment,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2018, pp. 1–3. [Online]. Available: https://ieeexplore.ieee.org/document/8463426
  42. D. Varga, D. Saupe, and T. Szirányi, “DeepRN: A Content Preserving Deep Architecture for Blind Image Quality Assessment,” in Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), in Proceedings of the IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2018, pp. 1–6. [Online]. Available: https://ieeexplore.ieee.org/document/8486528
  43. M. Jenadeleh, M. Pedersen, and D. Saupe, “Realtime Quality Assessment of Iris Biometrics Under Visible Light,” in Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPRW), CVPR Workshops, in Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPRW), CVPR Workshops. IEEE, 2018, pp. 443–452. [Online]. Available: https://ieeexplore.ieee.org/document/8575548
  44. S. Egger-Lampl et al., “Crowdsourcing Quality of Experience Experiments,” in Information Systems and Applications, incl. Internet/Web, and HCI, D. Archambault, H. Purchase, and T. Hossfeld, Eds., in Information Systems and Applications, incl. Internet/Web, and HCI. , Springer International Publishing, 2017, pp. 154–190.
  45. U. Gadiraju et al., “Crowdsourcing Versus the Laboratory: Towards Human-centered Experiments Using the Crowd,” in Information Systems and Applications, incl. Internet/Web, and HCI, D. Archambault, H. Purchase, and T. Hossfeld, Eds., in Information Systems and Applications, incl. Internet/Web, and HCI. , Springer International Publishing, 2017, pp. 6–26.
  46. V. Hosu et al., “The Konstanz natural video database (KoNViD-1k).,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2017, pp. 1–6. [Online]. Available: https://ieeexplore.ieee.org/document/7965673
  47. M. Spicker, F. Hahn, T. Lindemeier, D. Saupe, and O. Deussen, “Quantifying Visual Abstraction Quality for Stipple Drawings,” in Proceedings of the Symposium on Non-Photorealistic Animation and Rendering (NPAR), ACM, Ed., in Proceedings of the Symposium on Non-Photorealistic Animation and Rendering (NPAR). Association for Computing Machinery, 2017, pp. 8:1-8:10. doi: 10.1145/3092919.3092923.
  48. V. Hosu, F. Hahn, O. Wiedemann, S.-H. Jung, and D. Saupe, “Saliency-driven Image Coding Improves Overall Perceived JPEG Quality,” in Proceedings of the Picture Coding Symposium (PCS), in Proceedings of the Picture Coding Symposium (PCS). IEEE, 2016, pp. 1–5. [Online]. Available: https://www.uni-konstanz.de/mmsp/pubsys/publishedFiles/HoHaWi16.pdf
  49. I. Zingman, D. Saupe, O. A. B. Penatti, and K. Lambers, “Detection of Fragmented Rectangular Enclosures in Very High Resolution Remote Sensing Images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 54, no. 8, Art. no. 8, 2016, [Online]. Available: https://ieeexplore.ieee.org/document/7452408
  50. D. Saupe, F. Hahn, V. Hosu, I. Zingman, M. Rana, and S. Li, “Crowd Workers Proven Useful: A Comparative Study of Subjective Video Quality Assessment,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). 2016, pp. 1–2. [Online]. Available: https://www.uni-konstanz.de/mmsp/pubsys/publishedFiles/SaHaHo16.pdf
  51. V. Hosu, F. Hahn, I. Zingman, and D. Saupe, “Reported Attention as a Promising Alternative to Gaze in IQA Tasks,” in Proceedings of  the 5th ISCA/DEGA Workshop on Perceptual Quality of Systems (PQS 2016), in Proceedings of  the 5th ISCA/DEGA Workshop on Perceptual Quality of Systems (PQS 2016). 2016, pp. 117–121. [Online]. Available: https://www.isca-speech.org/archive/PQS_2016/abstracts/25.html

Project Group A

Models and Measures

A01 | Uncertainty Quantification and Analysis
A03 | Quantification of Visual Explainability
A07 | Visual Attention Modeling
A08 | Learning and Explaining Dimensionality Reduction
A09 | Scalability and Complexity in Immersive Analytics

 

Completed

A02 | Quantifying Visual Computing Systems
A04 | Quantitative Models for Visual Abstraction
A05 | Visual Quality Assessment
A06 | Computational Photography

 

Project Group B

Adaptive Algorithms

B01 | Adaptive Self-Consistent Visualization
B04 | Motion Estimation
B06 | Adaptive Algorithms for Graph Views and View Transitions

 

Completed

B02 | Adaptive Network Visualization
B05 | Large Scale Variational 3D Reconstruction
B07 | Computational Uncertainty Quantification

 

Project Group C

Interaction

C01 | Evaluating Mixed Reality Experiences
C05 | Human-Machine Interaction with Adaptive Multisensory Systems
C06 | User-Adaptive Mixed Reality
C07 | Optimization for Dynamic XR User Interfaces

 

Completed

C02 | Physiologically Based Interaction
C03 | Immersive Virtual Environments
C04 | Mobile Visualization and Interaction
Quantifying interactions with adaptable multisensory systems

 

Project Group D

Applications

D02 | Visual Analytics in Linguistics
D04 | Immersive Analytics for the Life Sciences
D05 | Visual Computing for ERP-based Brain Activity

 

Completed

D01 | Modeling of 3D Virtual Cities
D03 | Exploration and Analysis of Provenance Data

 

Central Services

INF | Collaboration Infrastructure
Ö | Public Relations

FOR SCIENTISTS

Projects
People
Publications
Graduate School
Equal Opportunity

FOR PUPILS

BOGY
Events for the Public

PRESS AND MEDIA

Press releases
Images

SOCIAL MEDIA

© SFB-TRR 161 | Quantitative Methods for Visual Computing | 2019.