Project B05

B05 | Efficient Large Scale Variational 3D Reconstruction

Prof. Bastian Goldlücke, University of Konstanz
Email | Website

Prof. Andrés Bruhn, University of Stuttgart
Email | Website

[completed]

Dr. Ole Johannsen, University of Konstanz – Email | Website

About this project
Results

The central goal of the project is to research and develop high-performance variational methods for large scale 3D reconstruction problems, which are general and accurate while meeting computation time constraints imposed by visual computing applications. Key abilities will be that

a variety of possible sources of 3D information can be integrated,
accuracy can be locally improved at the cost of run-time, and
for a given allowed total run-time, global accuracy is maximized according to application-specific metrics.

Research Questions

What is the best geometric representation to implement locally adaptive geometry optimization?

Which variational models efficiently allow local refinement of the scene structure in a mathematically consistent manner?

Which models in shape reconstruction best translate to the adaptive shape representation?

How can the local refinement of surface properties (e.g., texture) be modeled and optimized—ideally together with the geometry?

What are natural, ideally convex priors on adaptive grids?

What are optimal local accuracy measures for different applications?

How can an ideal trade-off between accuracy and run-time be determined?

Results on synthetic data with semi-reflective statue of Warrior. Top images depict center views of corresponding input light fields. Bottom resulting geometry.

Fig. 1: Results on synthetic data with semi-reflective statue of Warrior. Top images depict center views of corresponding input light fields. Bottom resulting geometry.

Publications

F. Petersen, B. Goldluecke, C. Borgelt, and O. Deussen, “GenDR: A Generalized Differentiable Renderer,” in Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 3992–4001. doi: 10.1109/CVPR52688.2022.00397.
- BibTeX
- Link
BibTeX
@inproceedings{https://doi.org/10.48550/arxiv.2204.13845, affiliation = {Deussen, Oliver, Universität Konstanz}, author = {Petersen, Felix and Goldluecke, Bastian and Borgelt, Christian and Deussen, Oliver}, booktitle = {Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR)}, doi = {10.1109/CVPR52688.2022.00397}, orcid-numbers = {Deussen, Oliver/0000-0001-5803-2185}, pages = {3992-4001}, title = {GenDR: A Generalized Differentiable Renderer}, url = {https://doi.ieeecomputersociety.org/10.1109/CVPR52688.2022.00397}, year = 2022 }
Link
https://doi.ieeecomputersociety.org/10.1109/CVPR52688.2022.00397
F. Petersen, B. Goldluecke, O. Deussen, and H. Kuehne, “Style Agnostic 3D Reconstruction via Adversarial Style Transfer,” in 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), IEEE, Jan. 2022, pp. 2273–2282. [Online]. Available: http://dblp.uni-trier.de/db/conf/wacv/wacv2022.html#PetersenGDK22
- BibTeX
- Link
BibTeX
@inproceedings{conf/wacv/PetersenGDK22, affiliation = {Deussen, Oliver, Universität Konstanz}, author = {Petersen, Felix and Goldluecke, Bastian and Deussen, Oliver and Kuehne, Hilde}, booktitle = {2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, month = {01}, orcid-numbers = {Deussen, Oliver/0000-0001-5803-2185}, pages = {2273-2282}, publisher = {IEEE}, title = {Style Agnostic 3D Reconstruction via Adversarial Style Transfer}, url = {http://dblp.uni-trier.de/db/conf/wacv/wacv2022.html#PetersenGDK22}, year = 2022 }
Link
http://dblp.uni-trier.de/db/conf/wacv/wacv2022.html#PetersenGDK22
S. Giebenhain and B. Goldlücke, “AIR-Nets: An Attention-Based Framework for Locally Conditioned Implicit Representations,” in 2021 International Conference on 3D Vision (3DV), 2021, pp. 1054–1064. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/9665836
- BibTeX
- Link
BibTeX
@inproceedings{9665836, author = {Giebenhain, Simon and Goldlücke, Bastian}, booktitle = {2021 International Conference on 3D Vision (3DV)}, pages = {1054-1064}, title = {AIR-Nets: An Attention-Based Framework for Locally Conditioned Implicit Representations}, url = {https://ieeexplore.ieee.org/abstract/document/9665836}, year = 2021 }
Link
https://ieeexplore.ieee.org/abstract/document/9665836
K. Kurzhals et al., “Visual Analytics and Annotation of Pervasive Eye Tracking Video,” in Proceedings of the Symposium on Eye Tracking Research & Applications (ETRA), ACM, 2020, pp. 16:1–16:9. doi: 10.1145/3379155.3391326.
- BibTeX
- Link
BibTeX
@inproceedings{10.1145/3379155.3391326, affiliation = {Kurzhals, Kuno, Visualisierungsinstitut der Universität Stuttgart. Rodrigues, Nils, Visualisierungsinstitut der Universität Stuttgart. Koch, Maurice, Visualisierungsinstitut der Universität Stuttgart. Stoll, Michael, Institut für Visualisierung und Interaktive Systeme. Bruhn, Andrés, Institut für Visualisierung und Interaktive Systeme. Bulling, Andreas, Institut für Visualisierung und Interaktive Systeme. Weiskopf, Daniel, Visualisierungsinstitut der Universität Stuttgart}, author = {Kurzhals, Kuno and Rodrigues, Nils and Koch, Maurice and Stoll, Michael and Bruhn, Andrés and Bulling, Andreas and Weiskopf, Daniel}, booktitle = {Proceedings of the Symposium on Eye Tracking Research & Applications (ETRA)}, doi = {10.1145/3379155.3391326}, orcid-numbers = {Kurzhals, Kuno/0000-0003-4919-4582, Rodrigues, Nils/0000-0002-1485-8249, Koch, Maurice/0000-0003-0469-8971, Bruhn, Andrés/0000-0003-0423-7411, Bulling, Andreas/0000-0001-6317-7303, Weiskopf, Daniel/0000-0003-1174-1026}, pages = {16:1-16:9}, publisher = {ACM}, title = {Visual Analytics and Annotation of Pervasive Eye Tracking Video}, url = {https://doi.org/10.1145/3379155.3391326}, year = 2020 }
Link
https://doi.org/10.1145/3379155.3391326
V. Hosu, B. Goldlücke, and D. Saupe, “Effective Aesthetics Prediction with Multi-level Spatially Pooled Features,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9367–9375, 2019, [Online]. Available: https://ieeexplore.ieee.org/document/8953497
- BibTeX
- Link
BibTeX
@article{journals/corr/abs-1904-01382, affiliation = {Hosu, Vlad, Universität Konstanz}, author = {Hosu, Vlad and Goldlücke, Bastian and Saupe, Dietmar}, journal = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, pages = {9367-9375}, title = {Effective Aesthetics Prediction with Multi-level Spatially Pooled Features}, url = {https://ieeexplore.ieee.org/document/8953497}, year = 2019 }
Link
https://ieeexplore.ieee.org/document/8953497
N. Marniok and B. Goldluecke, “Real-time Variational Range Image Fusion and Visualization for Large-scale Scenes using GPU Hash Tables,” in Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), 2018, pp. 912–920. [Online]. Available: https://ieeexplore.ieee.org/document/8354209
- BibTeX
- Link
BibTeX
@inproceedings{marniok2018realtime, author = {Marniok, Nico and Goldluecke, Bastian}, booktitle = {Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV)}, pages = {912-920}, title = {Real-time Variational Range Image Fusion and Visualization for Large-scale Scenes using GPU Hash Tables}, url = {https://ieeexplore.ieee.org/document/8354209}, year = 2018 }
Link
https://ieeexplore.ieee.org/document/8354209
D. Maurer, N. Marniok, B. Goldluecke, and A. Bruhn, “Structure-from-motion-aware PatchMatch for Adaptive Optical Flow Estimation,” in Computer Vision – ECCV 2018. ECCV 2018. Lecture Notes in Computer Science, vol. 11212, V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, Eds., Springer International Publishing, 2018, pp. 575–592. doi: 10.1007/978-3-030-01237-3_35.
- BibTeX
- Link
BibTeX
@inbook{conf/eccv/MaurerMGB18, affiliation = {Bruhn, Andrés, Institut für Visualisierung und Interaktive Systeme}, author = {Maurer, Daniel and Marniok, Nico and Goldluecke, Bastian and Bruhn, Andrés}, booktitle = {Computer Vision – ECCV 2018. ECCV 2018. Lecture Notes in Computer Science}, doi = {10.1007/978-3-030-01237-3_35}, editor = {Ferrari, Vittorio and Hebert, Martial and Sminchisescu, Cristian and Weiss, Yair}, orcid-numbers = {Bruhn, Andrés/0000-0003-0423-7411}, pages = {575-592}, publisher = {Springer International Publishing}, title = {Structure-from-motion-aware PatchMatch for Adaptive Optical Flow Estimation}, url = {https://doi.org/10.1007/978-3-030-01237-3_35 }, volume = 11212, year = 2018 }
Link
https://doi.org/10.1007/978-3-030-01237-3_35
M. Stein et al., “Bring it to the Pitch: Combining Video and Movement Data to Enhance Team Sport Analysis,” in IEEE Transactions on Visualization and Computer Graphics, 2017, pp. 13–22. [Online]. Available: https://ieeexplore.ieee.org/document/8019849
- BibTeX
- Link
BibTeX
@inproceedings{stein2017bring, affiliation = {Keim, Daniel A., Universität Konstanz}, author = {Stein, M. and Janetzko, H. and Lamprecht, A. and Breitkreutz, T. and Zimmermann, P. and Goldlücke, Bastian and Schreck, Tobias and Andrienko, Gennady L. and Grossniklaus, M. and Keim, Daniel A.}, booktitle = {IEEE Transactions on Visualization and Computer Graphics}, number = 1, pages = {13-22}, title = {Bring it to the Pitch: Combining Video and Movement Data to Enhance Team Sport Analysis}, url = {https://ieeexplore.ieee.org/document/8019849}, volume = 24, year = 2017 }
Link
https://ieeexplore.ieee.org/document/8019849
N. Marniok, O. Johannsen, and B. Goldluecke, “An Efficient Octree Design for Local Variational Range Image Fusion,” in Pattern Recognition. GCPR 2017. Lecture Notes in Computer Science, vol. 10496, V. Roth and T. Vetter, Eds., in Lecture Notes in Computer Science, vol. 10496. , Springer International Publishing, 2017, pp. 401–412. doi: 10.1007/978-3-319-66709-6_32.
- BibTeX
- Link
BibTeX
@inbook{conf/dagm/MarniokJG17, author = {Marniok, Nico and Johannsen, Ole and Goldluecke, Bastian}, booktitle = {Pattern Recognition. GCPR 2017. Lecture Notes in Computer Science}, doi = {10.1007/978-3-319-66709-6_32}, editor = {Roth, Volker and Vetter, Thomas}, pages = {401-412}, publisher = {Springer International Publishing}, series = {Lecture Notes in Computer Science}, title = {An Efficient Octree Design for Local Variational Range Image Fusion}, url = {https://doi.org/10.1007/978-3-319-66709-6_32 }, volume = 10496, year = 2017 }
Link
https://doi.org/10.1007/978-3-319-66709-6_32
O. Johannsen et al., “A Taxonomy and Evaluation of Dense Light Field Depth Estimation Algorithms,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Workshops, IEEE, 2017, pp. 1795–1812. [Online]. Available: https://ieeexplore.ieee.org/document/8014960
- BibTeX
- Link
BibTeX
@inproceedings{conf/cvpr/JohannsenHGABBB17, author = {Johannsen, Ole and Honauer, Katrin and Goldluecke, Bastian and Alperovich, Anna and Battisti, Federica and Bok, Yunsu and Brizzi, Michele and Carli, Marco and Choe, Gyeongmin and Diebold, Maximilian and Gutsche, Marcel and Jeon, Hae-Gon and Kweon, In So and Park, Jaesik and Park, Jinsun and Schilling, Hendrik and Sheng, Hao and Si, Lipeng and Strecke, Michael and Sulc, Antonin and Tai, Yu-Wing and Wang, Qing and Wang, Ting-Chun and Wanner, Sven and Xiong, Zhang and Yu, Jingyi and Zhang, Shuo and Zhu, Hao}, booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Workshops}, pages = {1795-1812}, publisher = {IEEE}, title = {A Taxonomy and Evaluation of Dense Light Field Depth Estimation Algorithms}, url = {https://ieeexplore.ieee.org/document/8014960}, year = 2017 }
Link
https://ieeexplore.ieee.org/document/8014960
O. Johannsen, A. Sulc, N. Marniok, and B. Goldluecke, “Layered Scene Reconstruction from Multiple Light Field Camera Views,” in Computer Vision – ACCV 2016. ACCV 2016. Lecture Notes in Computer Science, vol. 10113, S.-H. Lai, V. Lepetit, K. Nishino, and Y. Sato, Eds., Springer International Publishing, 2016, pp. 3–18. doi: 10.1007/978-3-319-54187-7_1.
- BibTeX
- Link
BibTeX
@inbook{conf/accv/JohannsenSMG16, author = {Johannsen, Ole and Sulc, Antonin and Marniok, Nico and Goldluecke, Bastian}, booktitle = {Computer Vision – ACCV 2016. ACCV 2016. Lecture Notes in Computer Science}, doi = {10.1007/978-3-319-54187-7_1}, editor = {Lai, Shang-Hong and Lepetit, Vincent and Nishino, Ko and Sato, Yoichi}, pages = {3-18}, publisher = {Springer International Publishing}, title = {Layered Scene Reconstruction from Multiple Light Field Camera Views}, url = {https://doi.org/10.1007/978-3-319-54187-7_1}, volume = 10113, year = 2016 }
Link
https://doi.org/10.1007/978-3-319-54187-7_1