Digigen & The TinMan

Hopefully this won't be information overload, but here's a little history.

Seadragon is the zooming technology that handles loading large amounts of visual information. It was originally Blaise's startup company and was acquired in 2006 by Gary Flake's 'Live Labs' division at Microsoft: an internet technology team focused on speeding up the transfer of technology from Microsoft Research to the internet.

Seadragon is impressive technology, but although it could quickly and smoothly arrange large numbers of images of virtually any size in 3D, the arrangements come up with by the Seadragon team were just things like arranging the photos in a sphere or on a flat plane. That certainly was impressive, but ultimately lacked meaning.

Photosynth is half Seadragon and half computer vision techniques used by Noah Snavely at the University of Washington. Noah's 2006 project 'Photo Tourism' (co-advised by Steve Seitz of University of Washington and Rick Szeliski of Microsoft Research) used existing computer vision techniques to perform minimal 3D reconstruction of batches of photos, but rather than focus on getting the 3D model generated by the computer vision algorithms to be of the highest quality (as everyone else in the computer vision research field and industry was doing), Noah used these algorithms simply to figure out the positions of the photos, relative to each other. Given those positions, Noah simply focused on presenting the input images, rather than focusing on the 3D model generated by the process. Noah now had a way to go to web photo achives and automatically spatially organize most of the photos of a location, but browsing all of those full quality photos at once would have meant a tremendously large download before you could see anything.

Seadragon provided a perfect way to load only as much of the photos as currently fit on the screen and Noah's Photo Tourism work provided spatial meaning. The two technologies were perfect compliments for each other to bring the concept to the web.

In 2008, Live Labs released a prototype version of Seadragon as an app for the iPhone. http://channel9.msdn.com/Blogs... (This app was recently removed from the iOS app store when Live Labs closed in November of 2010.)

Greg Pascale, a Live Labs summer intern, also released a Photosynth viewer for iPhone ( http://bit.ly/isynth ) after Photosynth launched on 2008 August 20th. Although this app was not officially released by Microsoft or the Photosynth team and did not implement Seadragon (and therefore does not load the full resolution photos) it still points to iOS friendliness from inside their ranks (Greg is now a Photosynth team member) and is a cool app.

When Photosynth launched as a public service in August of 2008, it moved out of Live Labs and became a subdivision of Live Search's 'Virtual Earth' team (known, since June of 2009 as Bing Maps). Bing has also proved to be very platform agnostic, friendly toward iPhone and iPad (among other mobile platforms), as their apps will attest.

Photosynth, as Blaise demoed it at TED in 2007 and as it was released in 2008, can be viewed as a growth of panoramic stitching, but it is something distinctly different than traditional panorama stitchers like the tools used to create Apple Quicktime VR panoramas, PtGui, AutoStitch, Photoshop's Photo Merge, or Microsoft Research's ICE (Image Composite Editor http://bit.ly/microsoftice ). Many people mistook the Photosynth app for Windows as a panorama stitcher, however Photosynth truly shines when the camera focuses on the same subject, but moves to many different points of view. By tracking individual portions of the images and watching how those portions move around each other from different points of view, Photosynth (and other photogrammetry applications) can estimate the relative positions of those portions of the scene as well as where the camera must have been, relative to the scene to see those keypoints in the arrangement that they are in each image.

Traditional panorama stitchers only stitch in 2D, meaning that they can line up different images that were taken from the same point of view, but cannot correctly stitch images when the camera lens has changed position while it turns, because when the camera lens changes position while it turns, foreground objects line up differently against the background. This means that the closer that the subject of your photography is to your camera, the more important it is (when shooting panoramas) for the lens to remain stationary in the air while the camera rotates around it. Parallax still occurs even with distant objects, however the more distant an object is the smaller the effect is because the object makes up a smaller portion of the image.

It turns out that a very high percentage of people who tried Photosynth actually shot panoramas, rather than Photosynths. Shooting a single panorama and uploading it as a synth does not really give you a good panorama or a good synth. To give people a better option (and because Microsoft already had a great panorama stitching in ICE, but no good way for average users to share the panoramas that ICE stitched), Photosynth's website began accepting traditional panoramas in March of 2010. In November of 2010, Microsoft Research also created a plugin for Photoshop for Windows to give people who preferred other panorama stitchers to ICE to upload their existing panoramas to the Photosynth website.

Somewhat confusingly, the 'Photosynth' app for iOS is actually a traditional panorama stitching program (more like mobile ICE than mobile Photosynth) even though the panoramas it creates are able to be uploaded to the Photosynth website. As I mentioned above, shooting panoramas is different than shooting photosynths. For panorama shooting tips for the mobile app, subscribe to http://youtube.com/masteringph...

The vision for the future is that all of the panoramas and photosynths on the Photosynth website will begin to link together. This was certainly part of the initial vision of Photo Tourism and Photosynth, however as of yet, photosynths and panoramas which overlap on the map have no connection on the Photosynth site. Each synth or pano is it's own island. This will be changing as Bing is gearing up their collaborative Read/Write World project (announced at O'Reilly Media's Where 2.0 2011 conference) which is aimed directly at linking together data from Photosynth, other panorama sites like 360 Cities, Bing Maps, etc. I'll be collecting more information at http://bit.ly/readwriteworld

The real innovation in Photosynth's mobile panorama app for iOS is the live tracking of the camera's video feed against the shots which have been captured so far. This paves the road for being able to hold up your phone in the future, point it at the world and having a future Bing Maps app be able to figure out exactly where you are standing (far more accurately than your GPS signal can provide) giving birth to the first generation of true augmented reality apps. This is a call back to the live tracking of the mobile video tracking against the indoor panorama that Blaise demoed last year at TED 2010. http://www.ted.com/talks/blais...

Seadragon is alive in that it was built into Microsoft Silverlight and now goes under the name of 'Deep Zoom'. http://www.microsoft.com/silve... There was also a version created in Javascript, which goes under the name of Seadragon AJAX and is used on http://zoom.it/

For much more history that you can research at your leisure, see the list of links in the left column of this spreadsheet of mine: http://docs.com/XFO

Cheers!
Nate Lawrence