This presentation is part of the 2017 3D Digital Documentation Summit.
Spherical Imaging and Virtual Environments for Conservation of Cultural Heritage Sites
Speaker 1: Yeah, thanks we’ve had some pretty great talk so far. And we have an update that Heritage Structures Lab is … The Art and Archeology Department, the Civil and Environmental Engineering Department, and we have some preliminary interests from the Architecture Department, as well as the Geo-Physics Department, so really hoping to make this new program at Princeton something that’s interdisciplinary. Because I think a lot of what people have talked about today shows how interdisciplinary these problems are, and how we need to have people coming at it with different research questions, and different solutions, in order to have great solutions for what we have for our research questions.
So today, I’m going to talk to you a little bit about spherical imaging and virtual reality for conservation.
So, currently there’s quite the dichotomy in the methods that are out there for doing documentation. You have 2D methods, such as digital mapping, where you can have an image of a building, and actually map out where, in using color regions, where different kinds of damage are. So you can have one color that means you have strong efflorescence somewhere. And now I’m talking about materials conservation for this, so if you have issues of delamination, of you have issues of sub-florescence. You can map those digitally on your images.
The other way that you could do a lot of documentation is 3D methods, which is what the focus of this summit has been. We’ve seen a lot of creative solutions for how people have used 3D documentation methods, such as LiDAR, or laser scanning, or photogrammetry, in order to document buildings. And, you can use those accompanied by BIM models in order to convey conservation challenges between conservators and building managers, or other people that are involved in a project.
Both of these sides … Both of these kinds of documentation, the 2D and 3D, both have their advantages and disadvantages. And, we heard from the Silman Group. We talked a lot about understanding the level of detail and understanding your project’s needs. And, that’s where I’m coming at for this presentation, is understanding what your project needs.
So, in 2D digital mapping, you have issues where it can’t always convey something that’s happening in 3D space. So, an example of a conservation challenge in 3D space is, say you have a wall, and you’re looking at an area of sub-florescence. You have a lot of salt coming out of your wall, and maybe some bricks are actually crushing. You might be losing some structural stability there. Understanding why that sub-florescence is occurring might not be obvious if you’re just looking at a 2D image.
This is a true example from Princeton’s campus. They had actually had a drain spout that terminated. It took salt water, so the water mixed with salts from deicing, and dumped it directly behind this wall, and they hadn’t actually terminated it in the sewer, and that led to these issues of sub-florescence. They continually treated the wall, but no one ever thought to treat the problem. No one ever looked on the other side of the wall and said, “Hey, why does this keep happening?”. So, that’s where the limits of 2D digital mapping really are. When you have those conservation issues that are not just on one side of a building, but are really in two different faces.
The advantage, though, of 2D digital mapping, again, something you’ve heard a lot about today is big data, and how do you deal with these large point clouds, and do you have the infrastructure? Do you have the need to deal with these large point clouds? 2D methods, if they are applicable, and you’ve decided with a team that this is the necessary level of detail for your project, there’s smaller data files. It’s faster to acquire them, and they are much more inexpensive.
$140,000 for maintenance of your LiDAR, was that what it was? That’s a lot. So, some 2D options, if you can afford them, can sometimes really save you in a project, and allow you to allot critical and necessary resources to other aspects of your project.
On the other side of things, you have photogrammetry, you have laser scanning, which you can use to make building information models. These are great for conveying 3D issues. They can allow you to show both sides of a wall, where maybe your symptoms of your problem, and your underlying problems actually are. It will allow you to have clearer communication between a building manager and a conservator, other people involved in your project.
The other reason that they’re great is that you can use them again. They’re recyclable. So, 3D models you can use again for education or tourism purposes. But, you need to know what the needs of your site are, and what the future goals are.
The downside that we’ve been talking about is that you have larger data files, more time, and more expensive. And so, it’s really critical for a project to understand what their resources are, what their needs are, and how to make those two actually line up in a feasible way, where you don’t have to say, “Oh my gosh, we have to write another grant, again.”, something that you can use within your own means.
So, this is actually where my research question formulated. It’s in between those two sets of 2D digital mapping … Photogrammetry’s a little … Much less expensive than laser scanning, and then you have laser scanning. But what’s really in the middle of these two kinds of approaches? This 2D and this 3D? In what efficient ways can we convey those 3D structural conservation issues that I was just talking about?
So, for didactic purposes, this is a documentation decision model that I tried to develop to make this issue a little bit more clear, if it’s not clear what I’m talking about yet. So, you can decide if your conservation problem is two dimensional or three dimensional. The two dimensional problems could be if you have a façade of a building and there are gypsum crusts forming, but the symptoms of that are obvious on the building and the cause isn’t on some other face. You can just use two dimensional methods if you aren’t going to use a 3D model later. So, if you talk to the building manager, and you say, “Hi. Are we going to be using a 3D model later?” “Will you use this for tourism or education purposes?” “Is it within your budget?” Understanding the needs of both sides of a project. So you can just use 2D digital mappings.
If they’re going to be using a 3D model later, that’s great. Then all the methods that we’ve seen so far are applicable. You can use photogrammetry or laser scanning to document problems that are both 2D and 3D. The gap in current methods that I’ve seen though, is what happens if you have a conservation problem that’s three dimensional but you’re not actually going to use a 3D model later. Is it efficient for your project to create a 3D model in order to just communicate between different groups on a project? Or are there ways that are more efficient that can actually fill this gap?
So, I just want to emphasize where I’m coming at for this project. A lot of what we’ve heard today is less than millimeter accuracy, and we’re talking about detailed models. Detailed and preliminary models, defined by Letellier, are scaled models. They require having accuracy and metric information. Where I’m coming at, is from the initial planning, the communication of the reference stages. It’s before you actually are going to be needing metric data. That’s why the method I’m going to propose now is actually appropriate if you don’t need metric data.
The method I want to tell you a little bit about today, and I’ve seen, I think, in one of the projects already, virtual tours are mentioned. Virtual tours take 360 spherical panoramas, and what you are seeing here is an example of a 360 panorama. Sorry if it makes your dizzy, if I’m going around too quickly. A 360 spherical panorama takes the 360 around you, as well as captures the floor and the ceiling of your surroundings, to provide a fully immersive, yet not scaled, environment. So, what a virtual tour does is it takes bunch of these spherical panoramas and it can stitch them together in order to allow your user to walk from, say, the front of this temple to the right side of this temple, and get an initial understanding of what this building is like, and maybe where you would need to set up some laser scanning stations. It can help in the planning stages.
The informational modeling aspect of this is kind of the new idea. Virtual tours have been used for tourism and education purposes. There’s a good example: The Mount Vernon Mansion put together a really nice virtual tour which is entirely online. The informational modeling side is what I think could really help conservators and building mangers to talk to each other. You can allow for interactive data visualization, where if I’m in the virtual environment, looking at this building, I can click on the column and get information about what material it is. When was it last replaced? Does it need to replaced now? And you can access databases, and pass conservation records in order to have the information you might need to make a good judgment call, and still be within the costs of whatever your project is.
We did this for two different kinds of case studies, because we wanted to see how our approach would work this virtual tour and information modeling. The first of those case studies was a material conservation challenge. What I mean by this is what I’ve described already. It’s … You have delamination of stones from drainage issues, or you have sub-florescence and efflorescence in your buildings. And the second, I don’t know if people are familiar … I wasn’t familiar until I went to graduate school for Civil Engineering was about Structural Health Monitoring. And, this essentially means understanding how the bridge or building is feeling at that current time, so we equate Structural Health Monitoring to doctors, and understanding the pain of you, understanding the pain of the building in order to make good diagnoses. Those are the two case studies I would like to present today.
According to Princeton University policy, I’m not allowed to tell you this building’s name. It has a fancy, rich donor. The building’s kind of fallen apart a couple times. He doesn’t like his name associated with it.
So, Case Study, Building Nine. On Building Nine, what you see here looks rather innocuous. You just have a calcium carbonate buildup, coming down this arch, and this calcium carbonate buildup is actually caused because of poor drainage on an elevated walkway above. So, you have a walkway here, and there’s an arch below it. The drain is at a high point, and the low point of this walkway is right above that arch, right where we see that calcium carbonate buildup.
But, it’s pretty innocuous, calcium carbonate. Maybe it doesn’t look the best, but this is actually what the arch was like seven years ago, when they had to replace it the first time, and that’s due to the fact that you don’t just have the mortar leaching through those joints on your elevated walkway. We’re in New England. We have snow. Deicing salts are on that walkway, deicing salts go down those same mortar joints, and those actually cause real problems in your structure. They can cause a buildup of internal stresses, which can lead to this strong damage, and you would have to replace the arch entirely, which is what they did seven years ago. This is really why I can’t give you the name of the building.
And so, this is a problem because they haven’t actually fixed the drain for this. All they did was replace the entire arch. So, they treated the symptoms, they didn’t actually treat the underlying problems, and now we’re starting to see the exact same process again. So, what my work is going to … I’m going to show you the results in a second … Is trying to understand how we can better communicate between conservators and building managers in order to avoid treating just the symptoms of a problem, and really get at the heart of what’s happening and what’s destroying your structure.
In order to do this, just setting up the initial, what we did is, we created a virtual tour environment. The virtual tour environment is the spherical panoramas that I showed you before, and those are captured using a Ricoh Theta S camera. Has anyone in the room used a Ricoh camera before? Super cheap, super fast, and they’re actually pretty great at capturing enough information about 3D space to give you an understanding about how things are connected. So, we used a Ricoh Theta S camera, combined with what’s called this Panotour Pro software. Used a lot for tourism, when they’re trying to link different areas and give a user a virtual experience of a site. But, we used it to give a conservator and a building manager a virtual experience of the site. So, I’m only going to show you one of the sets of buildings, but we applied it to four different buildings on the campus to see if it would facilitate communication better.
We also wanted to ensure though, that our interface could still fulfill the same tasks of digital mapping methods. So, if you’re a conservator, and you’ve been doing it for 40 years, you really love digital mapping. We want to make sure you can still do digital mapping inside this interface. So, you can create user defined polygons. You can do them better if you take the time. I’m not a detail person, so this is just a cursory mapping of the efflorescence here, and then a user can click inside the interface on this area that you’ve mapped out, and see what kind of damage there is. You can either use legends and/or this approach, but sometimes the interactivity is easier for a user to understand what’s happening, instead of just having a legend with all of these different kinds of damage. So a user in this spherical panorama can click and see what kinds of damage are in the area.
The extra part of what we’re doing for the virtual tour and information modeling is we’re using what are called “hot spots”. Hot spots essentially mean that when I click on a region in my spherical panorama, I can get access to a new kind of information. An example of this is, I’m in my 360 virtual environment, I click on my PDF, and I can get access to the documentation records for that architectural setup of the building. Where I am, and get access to metric information if I need to.
Other types of data that you can access, or you can have links to databases that show previous or future interventions. That way you can make, again, good judgment calls on what you should do for a building.
You can access to image galleries, that way you can see what past damage has looked like in order to avoid going back to the same thing, like they did with that arch, as well as you can have the experience of stepping through the wall. I can connect this spherical panorama where I am to what’s on the other side of the wall. That’s the bottom right corner. If you would have clicked on that, it brings you to the outside view of this wall, where you would see the drainage pipe. As a result, this is what the front end of our interface looks like.
This is what a user would be presented with, and there are multiple ways of navigating through this virtual tour. A conservator or building manager can use Google Maps to click on specific locations where you’ve had these spherical panoramas taken, in order to go to a specific building. You could use dropdown menus, again, to go to specific buildings, or … This is more like the education and tour usage, you could click on hot spots in the scene to virtually walk a path of connected panoramas. Not as useful for conservation, but I wanted to throw that in there since that’s what’s currently used for education.
Here’s a video that kind of demonstrates the problem I was telling you about for Building Nine. On the left hand side, you see the Google Maps, where we’re situated for our building, and then in front of you is a hot spot that you can click on in order to walk closer to the building. So, you see, we click on the hot spot and we’re brought into the next spherical panorama, and we see that damage that I was just talking about. And, we see that calcium carbonate buildup. We can click on it and get a better understanding of what the current state is, what past interventions have been. This is just a cursory text box and image that you can use to talk between a building manager and a conservator.
Of course, you’re not always going to want this cursory approach, so you also should be incorporating in your scenes image galleries, past conservation reports, as well as databases. For our databases, we just used simple Google sheets because we talked to Building Facilities, and they wouldn’t give us real data anyway. You can incorporate websites and MySQL databases just as easily, and any web based database, into these virtual tours.
Now we’ve gone above, and we can see … Small for this screen, but we can see an explanation of what the problem is. Here I’m saying that all the leaves and debris are in that low corner, and this is where our drain is, and it’s perfectly dry. It’s a high point, you’re not actually getting water there. So, this could be a means of communication between a conservator and a building manager.
I wanted to understand … So, I said we wanted an efficient means of communicating this 3D data, and I did a comparison of how long it would take me to actually make a virtual tour, how much it would cost me, and the data files for the virtual tour environment, photogrammetry, and laser scanning. So, I did a temple in Nepal, and it … Actually you can see a much shorter amount of time to take all of these spherical panoramas, and stitch them together. And it’s much cheaper … Photogrammetry’s still not that expensive. There are many tools out there for photogrammetry. The data file size is really impressive for how small it is. And that was … This 1.3 gigabytes is all the buildings I did at Princeton, not just that one, and the information that I embedded within them, so it’s a much smaller data file that you’re dealing with. If you’re trying to send it back and forth between two people, it’s a lot more manageable.
I’m going to quickly just run through the second part, which was the structural health monitoring. Structural health monitoring, like I said, is a structure’s nervous system, and so, there are sensors that are embedded within a structure that detect strain, so if a building, or a bridge, stretches or compresses, they can give you an understanding of what the bridge or building is feeling. That can help you diagnose if there are cracks or if you have some settlement issues in your building.
And so, this one, I was allowed to use the name. It’s Streicker Bridge, it’s on Princeton’s campus, by Christian Menn. It’s a deck-stiffened arch, and there’s a top view. There are four legs. This is where all the sensors are located within the bridge. Each of those black lines up there is a strain sensor that can give you information, again, about if there are cracks in the area, and how your bridge is responding to certain kinds of load.
In SHM … Structural Health Monitoring is referred to as SHM … Data visualization is again one of the main problems, and it’s a reason that sometimes people just disregard an SHM system that they have put into a structure. Because they can’t visualize the results, or they don’t understand what’s happening. So we wanted to try this out for … Try our method in the virtual tours and information modeling for this approach.
So, a very similar model, but instead, is your sensor network, the sensors I just showed you, 2D or 3D? I’m not going to show you the video for the sake of time, but it’s a very similar process where you can click on different areas of the bridge. We’ve embedded pictures of where the sensors are on the bridge itself. You can click on a temperature sensor and be brought to a temperature variation for the last week. You can click on a strain sensor and be brought to a database of the strains from the past five years, which is information we already have to have for Structural Health Monitoring. Now, it’s visualize in a way that and SHM practitioner and a building manager can easily communicate about where they are on their bridge and what they’re actually talking about.
And so, some future work that we’re doing is that we want to really work more on the data visualization for Structural Health Monitoring. The Civil Engineering Departments, they also want me to do some Civil Engineering Structural Health Monitoring aspects, so I’m working more on that. And then, how do we integrate the 3D scans and the photogrammetry models that we have with numerical modeling results? The type of numerical modeling that I do a lot of is called DEM, it’s not Digital Elevation Models. It’s the other DEM, Distinct Element Modeling. What that means is that we take a building and chop it up, brick by brick, and we put loads on it in order to see where cracks might form, and understand how the structure acts. Not just globally, but also locally.
These are some examples that we’ve done. We’ve looked at Amiens Cathedral, we have a cross section cut to understand what happens in wind. We looked at a rammed earth arch to understand where cracks form in historic rammed earth arches. But, something that we’ve now come across is that when we have our 3D numerical models, it’s not as intuitive when we’re trying to understand where cracks form, so you see that it collapses, but you don’t always get a good view of where exactly a crack formed, or if a crack formed and it was stable, but a different crack formed and then it fell, so we want to combine virtual reality viewing with the numerical modeling, so that way we can actually walk into the model itself and watch the cracks form. Trying to, again, communicate better the results of our numerical modeling with people who are interested in those results.
So, conclusions: VTIM, The Virtual Tours and Information Modeling, is just an initial and non-metric solution, but it allows you to communicate efficiently between people involved in a project.
Thank you very much for your time.
Speaker 2: Questions. Right up here in front.
Speaker 3: [inaudible 00:22:22] help build the interface that you’re using to embed all the information in [inaudible 00:22:18]
Speaker 1: The Panotour Software is the skeleton that you can use, and then you can either do it in the gooey, or you can create some templates in XML for all of the hot spots that you see there. So, it’s a customized interface that was built using a skeletal platform. Does that explain that?
Yes, Panotour Pro. It’s … I think it’s only $300 for an educational license, so not that expensive.
Speaker 4: I actually have two questions.
In Panotours, is it possible to insert any type of reference measurement in there to then make the whole thing somewhat scaled? Say you took like, you know, this brick is 12″ long, or whatever …
Speaker 1: So, what happens in the spherical viewing … I don’t know if you’re going to see it here, because I did a good job cutting the screen … In the spherical viewing environment, you might understand what’s happening in the direct center, but as you get to the fringes of those spherical panoramas, you can get some shift, and so we had thought about if there’s a good way of showing that metric and how it shifts as you shift your view in the spherical panorama, which is entirely possible … Just a nice geometry problem for you … But currently, as the software stands, and for what we did, metric information isn’t readily available.
Speaker 4: Second question is, do you have any thoughts on using like 360 video, same spherical stuff, but with video, as you walk through like [inaudible 00:23:55] on your site?
Speaker 1: Yeah, that was something … That was actually where we started with this project, was using … Because the Ricoh Theta camera that I used, you can take the 360 stills or you can take a video. And we had done the two stills and connected them through the software and then we had done a 360 video where I walked to show you the problem, and it’s just … It takes longer sometimes because if you’re on the ground and they have to actually walk up, it wasn’t as efficient in terms of time, as well as we didn’t feel that … When we were looking at it, it was harder to understand as I’m moving around. It’s sort of disorienting, and so that’s why we thought just connecting them, but you can try it. If you’re better at it than I am with the 360 video and being a little more still, you can definitely try to 360 video. Since the images were still, it made it a lot easier to convey what you’re talking about for a user.
Speaker 4: Okay, thank you.
Speaker 1: Thank you.
Documentation is pivotal for conservation, monitoring, and preventative maintenance of historic buildings and sites of cultural heritage. In the past 20 years, the technology associated with documenting historic sites has not only catalyzed the number of structures presented in 3D and virtual reality environments, but it has also vastly altered the way conservation of invaluable cultural heritage sites is carried out by archaeologists, conservationists, structural engineers, and all others involved in the work. Included in this technological revolution are advancements in digital photogrammetry, laser scanning, and LIDAR; however, a 3D model or virtual-reality environment on its own is not able to fully capture or convey pertinent information concerning a site of cultural heritage. It is imperative that a viewer of either a 3D model or virtual-reality environment be able to interact and gain a deeper understanding of a cultural heritage sites through appropriate connections to conservation or structural health monitoring results and data. The research presented here reflects a methodology and digital workflow which utilizes technologies in 360 degree spherical imaging as well as innovative tools for virtually experiencing the built-environment. The environment was captured utilizing the Ricoh Theta spherical imaging camera and processed utilizing the Kolor Panotour virtual environment software. The virtual environment was made interactive by adding responsive points of interest which allow for certain types of damage to be highlighted and explored for future conservation work. Exploration of the damages in these types of environments can include additional 2D images which accentuate aspects of structural damage, hyperlinks to papers associated with this work, videos illustrating other aspects of the site or area in question, as well as additional forms of metadata. The results of this research will be recommendations for how archaeologists, conservationists, structural engineers, and others associated with cultural heritage sites can document their sites and meaningfully convey metadata to appropriate audiences.
Rebecca Napolitano received her undergraduate degree from Connecticut College where she majored in Latin and Physics and researched the digital reconstruction of ancient Roman architecture. She has since continued these studies in the Heritage Structures Program at Princeton University which is a joint program between the departments of Civil and
Environmental Engineering and Art and Archaeology. Her main areas of interest as she pursues her Master’s and PhD at Princeton are historic structures, archaeological sites, photogrammetry, virtual reality experiences, 3D reconstructions, structural analysis, and conservation.
Branko Gliši? received his degrees in Civil Engineering and Theoretical Mathematics at University of Belgrade, Serbia, and Ph.D. at the EPFL, Switzerland. After eight-year long experience at SMARTEC SA, Switzerland, where he was involved in numerous research, development, and implementation engineering projects, he has been employed, first as an Assistant Professor, and currently as an Associate Professor at the Department of Civil and Environmental Engineering of Princeton University. His main areas of interest are heritage structures, monitoring methods, advanced sensors, data management and analysis, and smart structures. Prof. Gliši? is member of several professional associations and recipient of several awards.