Karsten Bruun Qvist
Introduction
A recent post on this blog looked at the possibility of deriving left-right stereo photos, or anaglyphs from image files that in addition to a normal 2D photo also contain its depth map, a type of image that will be called depth photos in this article. The previous post was motivated by the fact that more and more smartphones have the ability to make depth photos, and by a rapid development in depth map technology. The post showed a couple of examples where this method worked quite well, and it went on to suggest a number of reasons why this would be interesting to many stereo photographers. If you have not read the post, it might be worthwhile to do that now, as it provides background for what follows.
While the previous post gave examples, it did not provide direct comparison to any established, well know method of making stereo images. The goal of this post is to do exactly that.
Technique
The simple idea was to compare stereo images captured with a Fujifilm Real 3D W1 stereo camera with those derived from depth photos recorded with a Huawei P30 Pro smartphone, under different shooting scenarios.
Cameras
The Fujifilm Real 3D W1 was introduced in 2009, and has, together with its successor, the W3 from 2010 been considered a reference point for point and shoot stereo photography. The camera has two 10 megapixel sensors (1/2.3'', or 6.17 x 4.55 mm), and two zoom lenses (35-105 mm, in 35 mm equivalent focal length), and directly captures a left and a right image, with a stereo base of 77 mm. The camera outputs a special stereo image file format, mpo that essentially contains two jpg images.
The Huawei P30 Pro smartphone came on the market ten years later, in 2019. The sensor used for our purpose has 40 megapixels that are 'binned' into a 10 megapixel output. The size of this sensor is 1/1.7'', or 7.60 x 5.7 mm, meaning that physically it is 54% larger than the Fuji sensors.
Photos taken in the Aperture mode of the phone's native camera app are depth photos, containing both a normal 2D image and its corresponding depth map, when stored as jpg-files. Currently, most imaging software will ignore the depth map, but a couple of exceptions are Photopea.com, Stereophoto Maker, and of course, the native phone camera app. A 3x zoom range is available in this mode (27-81 mm, in 35 mm equivalent focal length).
Huawei does not provide any information on how depth maps are determined with the P30 Pro, but it is likely based on a combination of approaches. With three rear facing optical cameras, it is possible that two or three contribute. Phase detect, or dual pixel autofocus can also contribute to depth maps. The TOF (time of flight) sensor does not seem to contribute much as depth maps look essentially the same when it is covered. Most likely AI plays a major role in weaving together whatever information is used.
With depth map construction methods undocumented, it is likely that they vary considerably between smartphone manufacturers and models. If you try this method with a different phone, your results may differ from those obtained here.
If you have a smartphone that supports depth maps, you always have a stereo camera at hand, that only requires one exposure
Shooting
A number of different scenes were recorded with both cameras, mounted on a tripod, and with the lenses adjusted to about 35 mm focal length (35 mm equivalent focal length). Attempts were made to cover different types of subjects, and depth ranges.
The Fuji camera was used in aperture mode, with ISO set at 100 and aperture at f/8. The Huawei P30 Pro was used in its Aperture mode. Please note, that with other phones that can record depth maps, the appropriate camera mode may have a different name. You may find guidelines on what to look for in the previous post.
Processing
Both the mpo files from the Fuji camera and the jpg depth photos from the Huawei phone were processed in Masuji Suto's excellent and free Stereophoto Maker version 6.15.
Fuji mpo files were opened using the ‘File/Open Stereo Image’ command. Auto alignment was used to compensate for minor misalignments and differences between the two cameras, and color differences between the two images were eliminated with the ‘Adjust/Auto Color Adjustment’ command.
Huawei depth photos needs to be opened in a different way, using the ‘Edit/Depth map/Open Jpeg include depth map’ command. This launches a view with the 2D image to the left, and the depth map to the right. In cases where depth maps were predominantly very dark (close up shots), or bright a levels adjustment was used to balance the tonal distribution.
To convert to a stereo image, with left and right eye views, the ‘Edit/Depth map/Create 3D image form 2D + Depth map’ command was invoked. This opens a dialogue box were the Deviation (usually 3.0-4.5%) was chosen, and the convergence point can be set by pointing at a miniature version of the depth map. The value of Deviation you choose here seems to be the largest one possible between the two views, only realized if the depth map spans the entire tonal range. In most cases, the actual deviation will be smaller than the value set here.
With these choices made, the simulated left and right eye views were calculated and displayed.
With either camera, tones and colors were sometimes adjusted moderately using Levels, or Curves. Perspective correction and a slight crop was applied in some cases. Although additional processing in software such as Photoshop can be beneficial, a point was made of limiting editing to what is possible within Stereophoto Maker, for the purpose of this comparison.
B/W anaglyphs were chosen as the format for presenting the images, partly because it was assumed that readers who have made it so far would be in possession of red-cyan glasses, and also because the two cameras render color rather differently. Conversion to B/W was an attempt to avoid that from being a distraction.
As it involves no more effort than taking a normal 2D photo, you may consider taking all your smartphone photos as depth photos, so they have the potential to become stereo photos
Results
Basic image quality
Although the focus of this post is to compare the depth rendering capability of the two methods, it seems necessary to address the elephant in the room: how does the technical image quality of two very different cameras compare, all 3D considerations aside? Below we see crops of an image we will return to later, with the Fuji to the left.
If we recall that the Fuji Real 3D W1 was introduced 10 years before the Huawei P30 Pro, and that the latter has a 54% larger sensor, plus no doubt the benefit of many computational photography advances, it may not be too surprising that the Huawei yields better detail resolution, less noise, better dynamic range, micro contrast and color rendition. That said, I still find the Fuji adequate for the screen resolution I normally use (1920 x 1200).
Depth rendering
The first depth example is a photo captured at the main entrance of Frederiksberg Gardens, one of the major parks in Copenhagen. Below we see red-cyan anaglyphs of the two captures, with the Fuji one to the left. The convergence point is set at the front of the scene. The two images are, as intended, captured in rather similar ways, and there is substantial depth in both images. One may notice that one image appears slightly warmer in tone than the other does; this unexpectedly turned out to be difficult to prevent completely.
Inside the Frederiksberg Gardens, a bridge leading to an artificial island with a Chinese inspired teahouse offers a view with a smooth variation in depth. The Fuji image is to the left, and the convergence point is set on the grass, at the front of the scene.
The fact that only one exposure is needed also means that scenes with movement are handled just as well as stationary ones
In the next example, the point of convergence is set on the inscription under the statue, and as usual, the Fuji image is to the left. There is quite a bit of depth in both images, but in the Huawei image, there is also an abrupt shift in depth at the left edge of the statue (camera right) that does not work so well. This type of issue occasionally happens when converting depth photos to stereo images and more will be said about it in the Reflections section that follows.
In the next example, a small fountain in the Royal Danish Horticultural Society Garden, there is a bit more foreground interest. The convergence point is set at the front edge of the fountain basin, and again we have the Fuji version to the left.
We are fortunate that progress in both generation of depth maps, and in inpainting is very fast these years
Here is a closer view of the same scene, with a distance of 1.5 and 2 meter between statue and camera, and the convergence point set on the figure.
In the next example, again with the Fuji image is to the left, the convergence point is set on the registration plate of the car. Perhaps a bit more depth effect in the Fuji image, but also more ghosting towards the back, at least under my viewing conditions.
Our next example is illustrative of what happens when we move up close. Here, the distance between the camera and Mickey's nose, set as point of convergence, is about 50 cm. Again the Fuji image is to the left.
It probably should not surprise us that at such close distance the permissible depth is exceeded substantially with the Fuji camera, resulting in massive ghosting and close to breakdown of the image. The Huawei image manages much better, although we may notice some ghosting in the background, and a bit of artefacts on the left side of Mickey's nose (image right), and left thumb. However, they seem less evident with anaglyph glasses on, than off.
As this example illustrates, the depth photo to stereo photo method has an advantage at short distances, so one may ask, how close can you go? From the point of view of avoiding breakdown of the image, you can go very close. However, at very short distances, the depth of field is extremely limited, so unless you can accept some blur this becomes the limiting factor.
Considering the combination of convenience, image and depth rendering quality, better editability and new depth tuning possibilities, I would expect the depth photo to stereo photo approach to be of interest to many stereo photographers
Our final example is a shot of snowdrops, recorded at around 10 cm distance. As the Fuji was already unable to give a useful result at 50 cm, only the image obtained with the Huawei is shown.
This close the Huawei P30 Pro produces a depth map that is very dark so a levels adjustment was applied in Stereophoto Maker, in order to obtain a tonal distribution that gives better depth rendering. This, however, also amplifies random variations and errors in the depth map and may have contributed to the slight edge artefacts that can be seen in the anaglyph above. This of course is a minor complaint compared to the fact that the Fuji Real 3D W1 has nothing to offer at this distance, unless substantially modified with mirrors or prisms, or run as a 2D camera, cha-cha fashion.
Reflections
This section will provide some reflections on the results above and on the properties of the depth photo to stereo photo method in general.
Convenience
A strong point of the depth photo to stereo image method is that if you have a smartphone that supports depth maps, you always have a stereo camera at hand!
Making a depth photo is very simple; it only involves selecting the correct camera mode in the phones native camera app, and taking a single photo. This is certainly simpler than using the sequential (cha-cha method), or putting a mirror attachment on the phone (see previous post). As it involves no more effort than taking a normal 2D photo, you may even consider taking all your smartphone photos as depth photos - then they have the potential to become stereo photos, if desired. The fact that only one exposure is needed also means that scenes with movement are handled just as well as stationary ones. Futhermore, the basic processing in Stereophoto Maker is very easy, as described in the Technique section above.
Depth rendering
The point of showing a number of comparisons between anaglyphs obtained with the two methods is to allow you to make your own evaluation of the depth rendering. As for myself, I have been surprised to see how well the depth photo to stereo photo method works with the Huawei P30 Pro.
As mentioned, smartphone manufacturers likely make depth maps in different ways, so it is probable that it does not work as well with some phones. On the other hand, given that the Huawei P30 Pro was introduced in 2019, it is equally likely that it works better with some of the more recent phones. Immense progress that has been made the last few years in estimating depth maps from single 2D images using artificial intelligence, which is likely part of the technology smartphone manufacturers use.
The examples above clearly show that at short distances, the Huawei P30 Pro produces much better depth effects than the Fuji Real 3D W1. At a range beyond 5-8 m the results are a bit less clear cut, but after having converted over 500 Huawei images I see a tendency for the Fuji to render depth deeper into many scenes than the Huawei does. In some cases, you can get away with boosting depth rendering of the Huawei by increasing the Deviation variable used in the conversion, or by applying a Levels adjustment to the depth map, to expand the tonal variation at long distances. In other cases, this results in unpleasant artefacts.
Artefacts There are at least a couple of sources of artefacts with the depth photo to stereo photo method. Firstly, there are usually imperfections in the depth map; some of them will matter, others will not. If they do matter, it is often possible to reduce the problem by manually editing the depth map. There is a simple depth map editor built into Stereophoto maker that allows one to paint in corrections, but you would be well advised to first check if they make any difference to the viewing experience, as many do not ('Edit/Depth map/Correct depth map'). For edits that are complex, or require precision, it is preferable to move to a program with better painting tools, such as Photopea.com, or Photoshop.
The second problem stems from the fact that when creating left and right eye views, gaps arise that needs to be filled with something. To understand how this happens, consider a thought experiment. Imagine that you are looking at a simple scene, with one object in front of a distant background. Now, when you move your head a bit to the left, background that you did not see before becomes visible along the left edge of the object, it becomes disoccluded. When you move your head a bit to the right, background you did not see before becomes visible along the right edge of the object. When you ask software to generate left and right eye view from a single, ‘middle’ image the software initially has no information on what to put into these disoccluded areas, so it depends on a process called inpainting to ‘hallucinate’ content that it deems similar to the surroundings. Doing this well is easier if the deviation/displacement is small, the depth map has high quality, and if the background has little distinct detail. With some photos, the depth effect can therefore be taken further than with others.
Maybe we have one thing working for us: any issue due to disocclusion and unsuccessful inpainting will only be visible to one eye during viewing, as only one eye will see hallucinated content, in any given part of the image. Put in a different way, the areas that need inpainting in the left and right hand image are different. Our visual system has millions of years of training in making sense of imperfect input, and that may help pull off the trick, as long as the final image is synthesized in the mind, as it is when viewing stereo images. We are also fortunate that progress in both generation of depth maps, and in inpainting is very fast these years, so we can expect artefacts to become smaller and less frequent in the years to come.
On the other hand, it was noticed that when viewing on a 40'' 3D TV any artefacts present become more noticeable than when viewing on a computer monitor, or even smaller screen, which suggest that the method, at present may not be ready for big screen projection.
Increased editability
When it comes to editing traditional stereo pairs, global or whole image adjustments of e.g. brightness, contrast, color can easily be made inside Stereophoto Maker, or other software. However,as soon as we would like to apply changes only to certain parts of the image, things become complicated.
What if you want to clone something out? You would need to clone it out of two images, in a way that does not cause disturbance when viewing. What if you would like to dodge, or burn (lighten, or a darken) certain parts of the image? Again, you would need to apply the change to two photos, keeping the adjustments in sync. Therefore, while local editing is considered an essential aspect of high quality 2D photo editing, it soon becomes very tiring with stereo pairs, and does not seem to be used much.
With the depth image approach, you only need to apply edits to one image, which makes it no more complicated than editing any other 2D photo. For instance, if you open a depth photo in Photopea.com it loads the 2D photo and the depth map as separate layers, and you are ready to make all kinds of adjustments, global, or local. When done editing, you save out the edited image and depth map, and load them into Stereophoto Maker. Finally, you can apply all your editing skills!
'Tuning' depth after capture
As mentioned above it is possible to tune depth rendering, or ‘warp’ depth space by applying tonal adjustments such as Levels or Curves, both available in Stereophoto Maker to depth maps. In this way, we can enhance and manipulate depth rendering in a way that does not have an equivalent in traditional stereo photography. Yet, it does not stop with global, whole depth map adjustments. If we want something to stick out, or recede a bit more, nothing stops us from painting it a bit darker, or lighter locally in the depth map. In short, if we are willing to put in the work, it is possible to control depth in ways that are not possible in traditional stereo photography. A closer look at these possibilities will be the subject of a future post.
Closing thought
Considering the combination of:
Convenience
Image and depth rendering quality
Enhanced editability, and
Unique depth tuning possibilities
I would expect the depth photo to stereo photo approach to be of interest to many stereo photographers, in particular those who do not require that every stereo photo they take is suitable for big screen projection. I hope many will join the exploration of its possibilities and limitations. Do feel free to comment, and share your thoughts and experience!
Some helpful references
Jan Zimandl: Stereo Photo from Depth Photo: https://www.strv.com/blog/stereo-photo-from-depth-photo-engineering-ios
David Kuntz: Getting the right depth in 3D photography https://stereosite.com/taking-stereo-photos/getting-the-right-depth-in-3d-photography/
David Kuntz: Stereo Window Basics https://stereosite.com/taking-stereo-photos/stereo-window-basics/
Rebecca Sharpe : The Stereo Window Tutorial https://stereoscopy.blog/2020/06/14/the-stereo-window/
Software
Masuji Suto’s Stereophoto Maker (Win software): http://stereo.jpn.org/eng/stphmkr/ . Probably the World’s most used stereo photo editor. Reads jpg files with embedded depth maps and convert them to stereo images. Frequently updated. General tutorial: https://stereoscopy.blog/2019/09/08/how-to-use-stereo-photo-maker-basic-tutorial/ Another general tutorial: https://www.wikihow.com/Make-3D-Images-Using-StereoPhoto-Maker Specialized tutorial for converting 'depth images' to stereo images: http://stereo.jpn.org/kitkat/indexe.html
Photopea.com – excellent free browser based general photo editor. Not specialized for 3D, but reads 'depth photos' into separate layers for the photo and the depth map when opening the file
DEPTHY - A free web resource that is able to read some photos containing embedded depth maps (but not all), and convert them to various formats: http://depthy.stamina.pl/