A Multilinear (Tensor) Framework for Computer Vision and Graphics

Tuesday, March 02, 2004
6:00 PM
Free and open to the public

Natural images are the consequence of multiple factors related to scene structure, illumination, and imaging. Multilinear algebra, the algebra of higher-order tensors, offers a potent mathematical framework for explicitly dealing with the multifactor nature of image formation. I will present a multilinear model that computes (nonlinear) manifold representations of image ensembles in which the multiple constituent factors (or modes) are disentangled and analyzed explicitly. This nonlinear model is computed via a tensor decomposition, known as the N-mode SVD, which is an extension to tensors of the conventional matrix singular value decomposition (SVD). I will demonstrate the potency of our model in the context of facial image ensembles, where the relevant factors include different facial geometries, expressions, lighting conditions, and viewpoints.

When applied to the difficult problem of automated face recognition, our multilinear representation, called TensorFaces, yields significantly improved recognition rates relative to eigenfaces, the standard, linear, principle components analysis (PCA) approach. Our multilinear framework is also valuable in the context of image synthesis in computer graphics. I will present TensorTextures, a novel, multilinear approach to image-based rendering. Given a sparse set of sample images of a texture, TensorTextures learns the interaction between viewpoint, illumination, and geometry that determines surface appearance, including complex details due to surface mesostructure, such as self-occlusion and self-shadowing. The TensorTextures algorithm provides a parsimonious, explicitly multi-factor approximation to the bidirectional texture function (BTF). Time permitting, I will also summarize the application of tensor analysis to human motion capture data, leading to decompositions that we call "human motion signatures", which are useful in human action recognition and graphical animation synthesis.

x x


M. Alex O. Vasilescu

Research Scientist
New York University