To acquire panoramic video sequences, we have developed two types of Double-Mirror-Pyramid cameras that capture up to 360-degree fields of view at high-resolution. The first one, A Single View Double-Mirror-Pyramid Panoramic Camera, acquires a single sequence from one viewpoint, whereas the second, A Multiview Double-Mirror-Pyramid Panoramic Camera, provides multiple video sequences each taken from a different viewpoint, e.g. stereo sequences for 3D viewing. Both of these cameras belong to the family of pyramid cameras.
High-Resolution Double Pyramid Panoramic Cameras
High-resolution panoramic capture is highly desirable in many applications such as immersive virtual environments, tele-conferencing, surveillance, and robot navigation. In addition, a single viewpoint for all viewing directions, a large depth-of-field (omni-focus), and real-time acquisition are desired in some imaging applications (e.g. 3D reconstruction and rendering). The FOV of a conventional camera is limited by the size of its sensor and the focal length of its lens. For example, a typical 16mm lens with 2/3? CCD sensor has a 30 deg x 23 deg FOV. The number of pixels on the sensor (640 x 480 for NTSC camera) determines the resolution. The depth-of-field is limited and is determined by various imaging parameters such as aperture, focal length, and the scene location of the object.
Many approaches have been presented to achieve various subsets of these properties: wide FOV, high resolution, large depth-of-field, a single viewpoint, and real-time acquisition. Among these, mirror-pyramid (MP)-based camera systems offer a promising approach to capturing high-resolution, wide-FOV panoramas as they provide single-viewpoint images at video rate. Such systems use planar mirrors assembled in pyramid or prism shapes, and as many cameras as the number of mirror faces, each located and oriented to capture the part of the scene reflected off one of the flat mirror faces. Images from the individual cameras are concatenated to yield a 360-degree wide panoramic image. Compared to designs using parabolic or hyperbolic mirrors, flat mirrors are easier to design and produce, and they introduce minimal optical aberrations.
We have developed a double-mirror-pyramid design that doubles the size of the visual field of the single-pyramid based systems. With this prototype, we have developed methods for optimally choosing the parameters of MP-based camera systems, e.g., camera placement, pyramid geometry, sensor usage, and uniformity of image resolution, and how the resultant image quality can be evaluated.
Overview of panoramic imaging
The existing methods of capturing panoramas fall into one of the two categories: dioptric methods, where only refractive elements (lenses) are employed, and catadioptric methods, where a combination of reflective and refractive components is used. Typical dioptric systems include: the camera cluster method where multiple cameras point in different directions to cover a wide FOV; the fisheye method where a single camera acquires a wide FOV image through a fisheye lens; and the rotating camera method where a conventional camera pans to generate mosaics, or a camera with a non-frontal, tilted sensor pans around its viewpoint to acquire panoramic omni-focused images. The catadioptric methods include: sensors in which a single camera captures the scene as reflected off a single curved mirror, or sensors in which multiple cameras image the scene as reflected off the planar mirror surfaces.
The dioptric camera clusters are capable of capturing high-resolution panoramas at video rate. However, the cameras in these clusters due to physical constraints, which makes it difficult or even impossible to mosaic individual images to form a true panoramic view, while apparent continuity across images may be achieved by ad hoc image blending. The sensors with fisheye lens are able to deliver large FOV images at video rate, but suffer from low resolution, irreversible distortion for close-by objects, and non-unique viewpoints for different portions of the FOV. The rotating cameras deliver high-resolution wide FOV via panning, as well as omni-focus when used in conjunction with non-frontal imaging, but they have limited vertical FOV. Furthermore, because they sequentially capture different parts of the FOV, moving objects may be imaged incorrectly. typically do not share a unique viewpoint
The catadioptric sensors that use a parabolic- or a hyperbolic-mirror to map an omni-directional view onto a single sensor are able to achieve a single viewpoint at video rate, but the resolution of the acquired image is limited to that of the sensor used and varies significantly with the viewing direction across the visual fields. Analogous to the dioptric case, this resolution problem can be alleviated partially by replacing the simultaneous imaging of the entire FOV with panning and sequential imaging of its parts, followed by mosaicing the images, at the expense of video rate. Another category of the catadioptric sensors employs a number of planar mirrors assembled in the shape of right mirror-pyramids, together with as many cameras as the number of pyramid faces. Each of these cameras, capturing the part of the scene reflected off one of the faces, is located and oriented strategically such that the mirror images of their viewpoints are co-located at a single point inside the pyramid. Effectively, this creates a virtual camera that captures wide-FOV, high-resolution panorama at video rate.
Proposed Double-Mirror-Pyramid Camera
The main challenge in constructing a panoramic camera from multiple sensors is to co-locate the entrance pupils of the multiple cameras so that adjacent cameras cover contiguous FOV without obstructing the view of other cameras or their own. Nalwa first used a right mirror pyramid (MP) formed from planar mirrors for this purpose. He reported an implementation using a 4-sided right pyramid and 4 cameras. The pyramid stands on its horizontal base. Each triangular face forms a 45-degree angle with the base. The cameras are positioned in the horizontal plane that contains the pyramid’s vertex such that the entrance pupil of each camera is equidistant from the vertex and the mirror images of the entrance pupils coincide at a common point, C, on the axis of the pyramid. The cameras are pointed vertically downward at the pyramid faces such that the virtual optical axes of the cameras are all contained in a plane parallel to the pyramid base, effectively viewing the world horizontally outward from the common virtual viewpoint C.
The vertical dimension of the panoramic FOV in each of the aforementioned cases is the same as that of each of the cameras used only their horizontal FOVs are concatenated to obtain a wider, panoramic view. We have developed a panoramic design that uses a dual mirror-pyramid (DMP), formed by joining two mirror-pyramids such that their bases coincide (Fig. 2), together with two layers of camera clusters. Such a DMP-based design thus doubles the vertical FOV while preserving the ability to acquire panoramic high-resolution images from an apparent single viewpoint at video rate.
- H. Hua and N. Ahuja, A High Resolution Panoramic Camera, IEEE Conf. On Computer Vision and Pattern Recognition (CVPR), December 11-13, 2001, Hawaii, I-960-967.
- K.-H. Tan, H. Hua and N. Ahuja, Multiview Panoramic Cameras Using Mirror Pyramids, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 26, No. 7, July 2004, 941-946.
- K.-H. Tan, H. Hua and N. Ahuja, Multiview Mirror Pyramid-based Panoramic Cameras, Proceedings of the IEEE Workshop on Omnidirectional Vision (Omnivis), June 2002, Copenhagen, Denmark, 87-93.