The problem of decomposing an image into its different intrinsic layers, such as shading, reflectance, and shape components, is one of the fundamental problems of computer vision. We believe that many computer vision tasks will benefit from having access to such separations. As an example, object detection should benefit from factoring out shadow and lighting effects, or image segmentation of materials should be improved by working on reflectance properties alone. In this project we aim to both decompose images into their intrinsic layers, and also show that this results to empirical improvements on different tasks. 

Our approach is to model the physical process of light interacting with materials. We model material properties, shape, and lighting conditions as separate sources that result in the observed pixel color and brightness. In "Recovering Intrinsic Images with a Global Sparsity Prior on Reflectance" we extend the Retinex theory by including models of material reflectances. Our conditional random field model encourages solutions in which the image is explained by a sparse set of reflectance types. This corresponds to the intuition that there are only few materials in an image. In the figure we show some example outputs of this model

Intrinsic image decomposition problems arise in many different contexts and we study several different variants that arise when some  scene modalities may be known. For example when using multiple depth cameras for full body scanning, we are provided with a good 3D estimate of the scene, or given a video we know that light varies smoothly. Our approach is to model the physical process using techniques from computer graphics together with probabilistic inference to reason about the unknown sources that constituted the scene.