A05 | Image/Video Quality Assessment: From Test Databases to Similarity-Aware and Perceptual Dynamic Metrics

Prof. Dietmar Saupe, University of Konstanz
Email | Website

Dietmar Saupe

Prof. Andrés Bruhn, University of Stuttgart
Email | Website

Andrés Bruhn

Dr. Vlad Hosu, University of Konstanz – Email | Website

Oliver Wiedemann, University of Konstanz – Email | Website

The project addresses methods for automated visual quality assessment and their validation beyond mean opinion scores. We propose to enhance the methods by including similarity awareness and predicted eye movement sequences, quantifying the perceptual viewing experience, and to apply the metrics for quality-aware media processing. Moreover, we will set up and apply media databases that are diverse in content and authentic in the distortions, in contrast to current scientific data sets.

Research Questions

How can crowdsourcing be applied to help generating very large video media data bases for research applications in quality of multimedia?

What is the performance of state-of-the-art video quality assessment methods that were designed based on small training sets for such large and diversified media databases?

Quality assessment in such extremely large empirical studies requires crowdsourcing. How should that be organized to achieve sufficient reliability and efficiency?

Are machine learning techniques suitable to identify the best performing video quality assessment metrics for given media content?

What statistical/perceptual features should be extracted to express similarity for this task? 

How can one design new or hybrid strategies for video quality assessment based on the above?

Can we improve methods for image/video quality assessment by studying patterns of human visual attention and other perceptual aspects?

How can knowledge on human visual attention derived from eyetracking studies be incorporated into perceptual image/video quality assessment methods?

How can the quality assessment methods be applied in quality-aware media processing such as perceptual coding?

Fig. 1: Training Better Algorithms to Predict Subjective Quality Opinions.

Fig. 2: Saliency Driven Compression.

Publications

  1. M. Testolina, V. Hosu, M. Jenadeleh, D. Lazzarotto, D. Saupe, and T. Ebrahimi, “JPEG AIC-3 Dataset: Towards Defining the High Quality to Nearly Visually Lossless Quality Range,” in 15th International Conference on Quality of Multimedia Experience (QoMEX), in 15th International Conference on Quality of Multimedia Experience (QoMEX). 2023, pp. 55–60. doi: 10.1109/QoMEX58391.2023.10178554.
  2. M. Jenadeleh, J. Zagermann, H. Reiterer, U.-D. Reips, R. Hamzaoui, and D. Saupe, “Relaxed forced choice improves performance of visual quality assessment methods,” in 2023 15th International Conference on Quality of Multimedia Experience (QoMEX), in 2023 15th International Conference on Quality of Multimedia Experience (QoMEX). 2023, pp. 37–42. doi: 10.1109/QoMEX58391.2023.10178467.
  3. S. Su et al., “Going the Extra Mile in Face Image Quality Assessment: A Novel Database and Model,” IEEE Transactions on Multimedia, vol. 26, pp. 2671–2685, 2023, doi: 10.1109/TMM.2023.3301276.
  4. O. Wiedemann, V. Hosu, S. Su, and D. Saupe, “Konx: cross-resolution image quality assessment,” Quality and User Experience, vol. 8, no. 8, Art. no. 8, Aug. 2023, doi: 10.1007/s41233-023-00061-8.
  5. X. Zhao et al., “CUDAS: Distortion-Aware Saliency Benchmark,” IEEE Access, vol. 11, pp. 58025–58036, 2023, doi: 10.1109/ACCESS.2023.3283344.
  6. G. Chen, H. Lin, O. Wiedemann, and D. Saupe, “Localization of Just Noticeable Difference for Image Compression,” in 2023 15th International Conference on Quality of Multimedia Experience (QoMEX), in 2023 15th International Conference on Quality of Multimedia Experience (QoMEX). 2023, pp. 61–66. doi: 10.1109/QoMEX58391.2023.10178653.
  7. H. Lin et al., “Large-Scale Crowdsourced Subjective Assessment of Picturewise Just Noticeable Difference,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 9, Art. no. 9, 2022, doi: 10.1109/TCSVT.2022.3163860.
  8. M. Zameshina et al., “Fairness in generative modeling: do it unsupervised!,” in Proceedings of the Genetic and Evolutionary Computation Conference Companion, in Proceedings of the Genetic and Evolutionary Computation Conference Companion. ACM, Jul. 2022, pp. 320--323. doi: 10.1145/3520304.3528992.
  9. H. Lin, H. Men, Y. Yan, J. Ren, and D. Saupe, “Crowdsourced Quality Assessment of Enhanced Underwater Images - a Pilot Study,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). IEEE, Sep. 2022, pp. 1--4. doi: 10.1109/QoMEX55416.2022.9900904.
  10. J. Lou, H. Lin, D. Marshall, D. Saupe, and H. Liu, “TranSalNet: Towards perceptually relevant visual saliency prediction,” Neurocomputing, vol. 494, pp. 455–467, 2022, doi: https://doi.org/10.1016/j.neucom.2022.04.080.
  11. F. Götz-Hahn, V. Hosu, and D. Saupe, “Critical Analysis on the Reproducibility of Visual Quality Assessment Using Deep Features,” PLoS ONE, vol. 17, no. 8, Art. no. 8, 2022, doi: 10.1371/journal.pone.0269715.
  12. F. Götz-Hahn, V. Hosu, H. Lin, and D. Saupe, “KonVid-150k : A Dataset for No-Reference Video Quality Assessment of Videos in-the-Wild,” IEEE Access, vol. 9, pp. 72139--72160, 2021, doi: 10.1109/ACCESS.2021.3077642.
  13. B. Roziere et al., “Tarsier: Evolving Noise Injection in Super-Resolution GANs,” in 2020 25th International Conference on Pattern Recognition (ICPR), in 2020 25th International Conference on Pattern Recognition (ICPR). 2021, pp. 7028–7035. doi: 10.1109/ICPR48806.2021.9413318.
  14. H. Men, H. Lin, M. Jenadeleh, and D. Saupe, “Subjective Image Quality Assessment with Boosted Triplet Comparisons,” IEEE Access, vol. 9, pp. 138939–138975, 2021, doi: 10.1109/ACCESS.2021.3118295.
  15. H. Lin, G. Chen, and F. W. Siebert, “Positional Encoding: Improving Class-Imbalanced Motorcycle Helmet use Classification,” in 2021 IEEE International Conference on Image Processing (ICIP), in 2021 IEEE International Conference on Image Processing (ICIP). 2021, pp. 1194–1198. doi: 10.1109/ICIP42928.2021.9506178.
  16. S. Su, V. Hosu, H. Lin, Y. Zhang, and D. Saupe, “KonIQ++: Boosting No-Reference Image Quality Assessment in the Wild by Jointly Predicting Image Quality and Defects,” in 32nd British Machine Vision Conference, in 32nd British Machine Vision Conference. 2021, pp. 1–12. [Online]. Available: https://www.bmvc2021-virtualconference.com/assets/papers/0868.pdf
  17. B. Roziere et al., “EvolGAN: Evolutionary Generative Adversarial Networks,” in Computer Vision -- ACCV 2020, in Computer Vision -- ACCV 2020. Cham: Springer International Publishing, Nov. 2021, pp. 679--694. doi: 10.1007/978-3-030-69538-5_41.
  18. O. Wiedemann, V. Hosu, H. Lin, and D. Saupe, “Foveated Video Coding for Real-Time Streaming Applications,” in 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX), in 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX). 2020, pp. 1–6. doi: 10.1109/QoMEX48832.2020.9123080.
  19. H. Lin, M. Jenadeleh, G. Chen, U. Reips, R. Hamzaoui, and D. Saupe, “Subjective Assessment of Global Picture-Wise Just Noticeable Difference,” in Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), in Proceedings of the IEEE International Conference on Multimedia and Expo (ICME). 2020, pp. 1–6. doi: 10.1109/ICMEW46912.2020.9106058.
  20. T. Guha et al., “ATQAM/MAST’20: Joint Workshop on Aesthetic and Technical Quality Assessment of Multimedia and Media Analytics for Societal Trends,” in Proceedings of the 28th ACM International Conference on Multimedia, in Proceedings of the 28th ACM International Conference on Multimedia. Seattle, WA, USA: Association for Computing Machinery, 2020, pp. 4758–4760. doi: 10.1145/3394171.3421895.
  21. B. Roziere et al., “Evolutionary Super-Resolution,” in Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, in Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion. Cancún, Mexico: Association for Computing Machinery, 2020, pp. 151–152. doi: 10.1145/3377929.3389959.
  22. H. Lin et al., “SUR-FeatNet: Predicting the Satisfied User Ratio Curvefor Image Compression with Deep Feature Learning,” Quality and User Experience, vol. 5, no. 1, Art. no. 1, 2020, doi: 10.1007/s41233-020-00034-1.
  23. X. Zhao, H. Lin, P. Guo, D. Saupe, and H. Liu, “Deep Learning VS. Traditional Algorithms for Saliency Prediction of Distorted Images,” in 2020 IEEE International Conference on Image Processing (ICIP), in 2020 IEEE International Conference on Image Processing (ICIP). 2020, pp. 156–160. doi: 10.1109/ICIP40778.2020.9191203.
  24. M. Jenadeleh, M. Pedersen, and D. Saupe, “Blind Quality Assessment of Iris Images Acquired in Visible Light for Biometric Recognition,” Sensors, vol. 20, no. 5, Art. no. 5, 2020, doi: 10.3390/s20051308.
  25. O. Wiedemann and D. Saupe, “Gaze Data for Quality Assessment of Foveated Video,” in ACM Symposium on Eye Tracking Research and Applications, in ACM Symposium on Eye Tracking Research and Applications. Stuttgart, Germany: Association for Computing Machinery, 2020. doi: 10.1145/3379157.3391656.
  26. V. Hosu, H. Lin, T. Sziranyi, and D. Saupe, “KonIQ-10k : An Ecologically Valid Database for Deep Learning of Blind Image Quality Assessment,” IEEE Transactions on Image Processing, vol. 29, pp. 4041--4056, 2020, doi: 10.1109/TIP.2020.2967829.
  27. V. Hosu et al., “From Technical to Aesthetics Quality Assessment and Beyond: Challenges and Potential,” in Joint Workshop on Aesthetic and Technical Quality Assessment of Multimedia and Media Analytics for Societal Trends, in Joint Workshop on Aesthetic and Technical Quality Assessment of Multimedia and Media Analytics for Societal Trends. Seattle, WA, USA: Association for Computing Machinery, 2020, pp. 19–20. doi: 10.1145/3423268.3423589.
  28. H. Men, V. Hosu, H. Lin, A. Bruhn, and D. Saupe, “Visual Quality Assessment for Interpolated Slow-Motion Videos Based on a Novel Database,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). 2020, pp. 1–6. doi: 10.1109/QoMEX48832.2020.9123096.
  29. M. Lan Ha, V. Hosu, and V. Blanz, “Color Composition Similarity and Its Application in Fine-grained Similarity,” in 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), in 2020 IEEE Winter Conference on Applications of Computer Vision (WACV). Piscataway, NJ: IEEE, 2020, pp. 2548--2557. doi: 10.1109/WACV45572.2020.9093522.
  30. H. Lin, J. D. Deng, D. Albers, and F. W. Siebert, “Helmet Use Detection of Tracked Motorcycles Using CNN-Based Multi-Task Learning,” IEEE Access, vol. 8, pp. 162073–162084, 2020, doi: 10.1109/ACCESS.2020.3021357.
  31. H. Men, V. Hosu, H. Lin, A. Bruhn, and D. Saupe, “Subjective annotation for a frame interpolation benchmark using artefact amplification,” Quality and User Experience, vol. 5, no. 1, Art. no. 1, 2020, doi: 10.1007/s41233-020-00037-y.
  32. C. Fan et al., “SUR-Net: Predicting the Satisfied User Ratio Curve for Image Compression with Deep Learning,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2019, pp. 1–6. doi: 10.1109/QoMEX.2019.8743204.
  33. V. Hosu, B. Goldlücke, and D. Saupe, “Effective Aesthetics Prediction with Multi-level Spatially Pooled Features,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9367–9375, 2019, doi: 10.1109/CVPR.2019.00960.
  34. H. Men, H. Lin, V. Hosu, D. Maurer, A. Bruhn, and D. Saupe, “Visual Quality Assessment for Motion Compensated Frame Interpolation,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2019, pp. 1–6. doi: 10.1109/QoMEX.2019.8743221.
  35. H. Lin, V. Hosu, and D. Saupe, “KADID-10k: A Large-scale Artificially Distorted IQA Database,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2019, pp. 1–3. doi: 10.1109/QoMEX.2019.8743252.
  36. V. Hosu, H. Lin, and D. Saupe, “Expertise Screening in Crowdsourcing Image Quality,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2018, pp. 276–281. doi: https://dx.doi.org/10.1109/QoMEX.2018.8463427.
  37. M. Jenadeleh, M. Pedersen, and D. Saupe, “Realtime Quality Assessment of Iris Biometrics Under Visible Light,” in Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPRW), CVPR Workshops, in Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPRW), CVPR Workshops. IEEE, 2018, pp. 443–452. doi: 10.1109/CVPRW.2018.00085.
  38. D. Varga, D. Saupe, and T. Szirányi, “DeepRN: A Content Preserving Deep Architecture for Blind Image Quality Assessment,” in Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), in Proceedings of the IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2018, pp. 1–6. doi: 10.1109/ICME.2018.8486528.
  39. H. Men, H. Lin, and D. Saupe, “Spatiotemporal Feature Combination Model for No-Reference Video Quality Assessment,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2018, pp. 1–3. doi: 10.1109/QoMEX.2018.8463426.
  40. V. Hosu et al., “The Konstanz natural video database (KoNViD-1k).,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2017, pp. 1–6. doi: 10.1109/QoMEX.2017.7965673.
  41. M. Spicker, F. Hahn, T. Lindemeier, D. Saupe, and O. Deussen, “Quantifying Visual Abstraction Quality for Stipple Drawings,” in Proceedings of the Symposium on Non-Photorealistic Animation and Rendering (NPAR), ACM, Ed., in Proceedings of the Symposium on Non-Photorealistic Animation and Rendering (NPAR). Association for Computing Machinery, 2017, pp. 8:1-8:10. [Online]. Available: https://doi.org/http://dx.doi.org/10.1145/3092919.3092923
  42. S. Egger-Lampl et al., “Crowdsourcing Quality of Experience Experiments,” in Information Systems and Applications, incl. Internet/Web, and HCI, vol. Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments. Dagstuhl Seminar 15481, Dagstuhl Castle, Germany, November 22 – 27, 2015, Revised Contributions, no. LNCS 10264, D. Archambault, H. Purchase, and T. Hossfeld, Eds., in Information Systems and Applications, incl. Internet/Web, and HCI, vol. Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments. Dagstuhl Seminar 15481, Dagstuhl Castle, Germany, November 22 – 27, 2015, Revised Contributions. , Springer International Publishing, 2017, pp. 154–190. doi: 10.1007/978-3-319-66435-4_7.
  43. U. Gadiraju et al., “Crowdsourcing Versus the Laboratory: Towards Human-centered Experiments Using the Crowd,” in Information Systems and Applications, incl. Internet/Web, and HCI, vol. Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments. Dagstuhl Seminar 15481, Dagstuhl Castle, Germany, November 22 – 27, 2015, Revised Contributions, no. LNCS 10264, D. Archambault, H. Purchase, and T. Hossfeld, Eds., in Information Systems and Applications, incl. Internet/Web, and HCI, vol. Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments. Dagstuhl Seminar 15481, Dagstuhl Castle, Germany, November 22 – 27, 2015, Revised Contributions. , Springer International Publishing, 2017, pp. 6–26. doi: 10.1007/978-3-319-66435-4_2.
  44. I. Zingman, D. Saupe, O. A. B. Penatti, and K. Lambers, “Detection of Fragmented Rectangular Enclosures in Very High Resolution Remote Sensing Images,” IEEE Transactions on Geoscience and Remote Sensing, vol. 54, no. 8, Art. no. 8, 2016, doi: 10.1109/TGRS.2016.2545919.
  45. D. Saupe, F. Hahn, V. Hosu, I. Zingman, M. Rana, and S. Li, “Crowd Workers Proven Useful: A Comparative Study of Subjective Video Quality Assessment,” in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX), in Proceedings of the International Conference on Quality of Multimedia Experience (QoMEX). 2016, pp. 1–2. [Online]. Available: https://www.uni-konstanz.de/mmsp/pubsys/publishedFiles/SaHaHo16.pdf
  46. V. Hosu, F. Hahn, I. Zingman, and D. Saupe, “Reported Attention as a Promising Alternative to Gaze in IQA Tasks,” in Proceedings of  the 5th ISCA/DEGA Workshop on Perceptual Quality of Systems (PQS 2016), in Proceedings of  the 5th ISCA/DEGA Workshop on Perceptual Quality of Systems (PQS 2016). 2016, pp. 117–121. doi: 10.21437/PQS.2016-25.
  47. V. Hosu, F. Hahn, O. Wiedemann, S.-H. Jung, and D. Saupe, “Saliency-driven Image Coding Improves Overall Perceived JPEG Quality,” in Proceedings of the Picture Coding Symposium (PCS), in Proceedings of the Picture Coding Symposium (PCS). IEEE, 2016, pp. 1–5. doi: 10.1109/PCS.2016.7906397.

Project Group A

Models and Measures

 

Completed

 

Project Group B

Adaptive Algorithms

 

Completed

 

Project Group C

Interaction

 

Completed

 

Project Group D

Applications

 

Completed