Project D01

Abstract

Within this paper we propose an end-to-end approach for classifying terrestrial images of building facades into five different utility classes (commercial, hybrid, residential, specialUse, underConstruction) by using Convolutional Neural Networks (CNNs). For our examples we use images provided by Google Street View. These images are automatically linked to a coarse city model, including the outlines of the buildings as well as their respective use classes. By these means an extensive dataset is available for training and evaluation of our Deep Learning pipeline. The paper describes the implemented end-to-end approach for classifying street-level images of building facades and discusses our experiments with various CNNs. In addition to the classification results, so-called Class Activation Maps (CAMs) are evaluated. These maps give further insights into decisive facade parts that are learned as features during the training process. Furthermore, they can be used for the generation of abstract presentations which facilitate the comprehension of semantic image content. The abstract representations are a result of the stippling method, an importance-based image rendering.

Abstract

The design of three-dimensional and four-dimensional Apps, running on the leading operating systems Android, iOS and Windows is the next challenge in Architecture, BIM, Civil Engineering, Digital Cultural Heritage (DCH) preservations and many more. Based on experiences developing Apps for archaeology and architecture, the paper introduces with general workflows for 3D data collection, using laser scanning, geometric computer vision and photogrammetry. The resulting point clouds have to be merged, using the most recent developments of laser scanning, computer vision, photogrammetry and statistical inference. 3D and 4D modeling is done using professional software from surveying and computer graphics, such as Leica’s Cyclone, Trimble’s SketchUp and Autodesk 3ds Max. The fourth dimension, time, is injected onto the 3D contemporary models using the texture of old photos. After homogenization of all 3D models in Autodesk 3ds Max these are exported to the game engine Unity to allow for the creation of the reference surface and finally the 3D urban model. The storyboard creates for the programmer an outline, which features and functions have to be fulfilled. Finally the Apps for Android, iOS and Windows are created and exported for the use on mobile devices.

Abstract

Durch die Digitalisierung der Photogrammetrie sind neue Auswertemethoden notwendig geworden, um das enorme Informationspotential der Bilder in allen Belangen auszuschöpfen. Dies erfordert auch ein Umdenken hinsichtlich der bisherigen Ansätze für die Erweiterung der Bündelblockausgleichung durch zusätzliche Parameter. Mittels exakt orientierten Bildern können dann die Methoden der dichten Bildzuordnung angewendet werden. Deren Punktwolken sind in 3D-CAD-Modelle zu überführen, die noch durch Bildtexturen angereichert werden können. Der folgende Beitrag beschreibt eine Neuauflage der Selbstkalibration, indem erstmalig eine exakte mathematische Begründung dafür gegeben wird und zwei Klassen von Parametersätzen eingeführt werden: Legendre- und Fourier-Parameter. Deren Leistungsfähigkeit ist anhand der Datensätze des DGPF-Kameratests über Vaihingen/Enz unter Beweis gestellt. Ferner wird ein Vergleich zu den über Jahrzehnte hinweg angewandten Parametersätzen von Brown, Ebner und Grün hergestellt. Der zweite Schwerpunkt demonstriert die Ableitung und Verarbeitung von dichten Punktwolken zu LoD3-Modellen, die durch eine Erweiterung der Methode des Semi-Global Matching mit der Software SURE erzeugt werden und mittels einer Fassadengrammatik die gewünschten Strukturinformationen liefern. Diese Modelle sind beispielsweise in Game Engines zu integrieren und u.a. in eindrucksvolle Augmented Reality Apps für mobile Geräte zu überführen.

Abstract

Recent years have shown a shift from pure geometric 3D city models to data with semantics. This is induced by new applications (e.g. Virtual/Augmented Reality) and also a requirement for concepts like Smart Cities. However, essential urban semantic data like building use categories is often not available. We present a first step in bridging this gap by proposing a pipeline to use crawled urban imagery and link it with ground truth cadastral data as an input for automatic building use classification. We aim to extract this city-relevant semantic information automatically from Street View (SV) imagery. Convolutional Neural Networks (CNNs) proved to be extremely successful for image interpretation, however, require a huge amount of training data. Main contribution of the paper is the automatic provision of such training datasets by linking semantic information as already available from databases provided from national mapping agencies or city administrations to the corresponding façade images extracted from SV. Finally, we present first investigations with a CNN and an alternative classifier as a proof of concept.

Abstract

While the generation of geometric 3D virtual models has become feasible to a great extent, the enrichment of the resulting urban building models with semantics remains an open research question in the field of geoinformation and geovisualisation. This additional information is not only valuable for applications like Building Information Modeling (BIM) but also offers possibilities to enhance the visual insight for humans when interacting with that kind of data. Depending on the application, presenting users the highest level of detail of building models is often neither the most informative nor feasible way. For example when using mobile apps, resources and display sizes are quite limited. A concrete use case is the imparting of building use types in urban scenes to users. Within our preliminary work, user studies helped to identify important features for the human ability to associate a building with its correct usage type. In this work we now embed this knowledge into building category-specific grammars to automatically modify the geometry of a building to align its visual appearance to its underlying use type. If the building category for a model is not known beforehand, we investigate its feature space and try to derive its use type from there. Within the context of this work, we developed a Virtual Reality (VR) framework that gives the user the possibility to switch between different building representation types while moving in the VR world, thus enabling us in the future to evaluate the potential and effect of the grammar-enhanced building model in an immersive environment.

Abstract

Virtual 3D cities are becoming increas- ingly important as a means of visually communicating diverse urban-related information. Since humans are the direct recipients of this information transfer, it is vital that the 3D city representations account for the humans' spatial cognition. Thus, our long-term goal is providing a model for the effective perception-aware visual communication of urban- or building-related semantic information via geometric 3D building representations which induce a maximum degree of perceptual insight in the user's mind. A first step towards this goal is to get a deeper understanding of a human's cognitive expe- rience of virtual 3D cities. In this context, the paper presents a user study on the human ability to perceive building categories, e.g.residential home, office building, building with shops etc., from geometric 3D building representations. The study reveals various dependencies between geometric properties of the 3D representations and the perceptibility of the building categories. Knowledge about which geometries are relevant, helpful or obstructive for perceiving a specific building category is derived. The importance and usability of such knowledge is demonstrated based on a perception-guided 3D building abstraction process. Zusammenfassung: Eine Studieüber die menschliche Wahrnehmung von Gebäudekategorien auf Basis unterschiedlicher 3D-Gebäuderepräsentationen. Virtuelle 3D-Städte werden zunehmend wichtig, um unterschiedlichste stadtrelevante Informationen visuell zu vermitteln. Da Menschen die direkten Empfänger dieses Informationstransfers sind, ist es unerlässlich, dass 3D-Stadtreprä-sentationen die räumliche Wahrnehmung von uns Menschen berücksichtigen. Unser längerfristiges Ziel ist es daher, ein Modell zur wahrnehmungsbe- wussten visuellen Kommunikation von städteoder gebäud espezifischen semantischen Informationen zu entwickeln, welchesüber geometrische 3D-Gebäuderepräsentationen dem Nutzer ein Maximum an Erkenntnisgewinn ermöglicht. Ein erster Schritt dorthin ist, sich ein besseres Verständnis der menschlichen Wahrnehmung von virtuellen 3D-Städten zu verschaffen. In diesem Zusammenhang präsentiert der Beitrag einen Nutzertestüber die menschliche Fähigkeit, Gebäudekategorien (z. B. Wohngebäude, Büros, Gebäude mit Läden usw.) anhand geometrischer 3D-Gebäuderepräsentationen zu erkennen. Die Studie zeigt zahlreiche Abhängigkeiten zwischen geometrischen Eigenschaften der 3D-Repräsentationen und der Wahrnehmbarkeit der Gebäudekategorien auf. Wissen darüber, welche geometrischen Eigenschaften relevant, hilfreich oder hinderlich sind, um eine bestimmte Gebäudekategorie zu erkennen, wird aus den Ergebnissen der Studie abgeleitet. Die Wichtigkeit und der Nutzen dieser Erkenntnisse werden anhand einer wahrnehmungsgesteuerten Abstraktion von 3D-Gebäudemodellen aufgezeigt.

Abstract

Virtual 3D cities are becoming increasingly important as a means of visually communicating diverse urban-related information. To get a deeper understanding of a human’s cognitive experience of virtual 3D cities, this paper presents a user study on the human ability to perceive building categories (e.g. residential home, office building, building with shops etc.) from geometric 3D building representations. The study reveals various dependencies between geometric properties of the 3D representations and the perceptibility of the building categories. Knowledge about which geometries are relevant, helpful or obstructive for perceiving a specific building category is derived. The importance and usability of such knowledge is demonstrated based on a perception-guided 3D building abstraction process.