Joint Learning Over Visual and Geometric Data

Many challenges remain in applying machine learning to domains where obtaining massive annotated data is difficult. We discuss approaches that aim to reduce supervision load for learning algorithms in the visual and geometric domains by leveraging correlations among data as well as among learning tasks -- what we call joint learning. The basic notion is that inference problems do not occur in isolation but rather in a "social context" that can be exploited to provide self-supervision by enforcing consistency, thus improving performance and increasing sample efficiency. An example is voting mechanisms where multiple "experts" must collaborate on predicting a particular outcome, such as an object detection. This is especially challenging across different modalities, such as when mixing point clouds with image data, or geometry with language data. Another example is the use of cross-task consistency constraints, as in the case of inferring depth and normals from an image, which are obviously correlated. Even at the level of latent representations, joint learning can avoid blind-spots of any one individual representation and better adapt to data particularities. The talk will present a number of examples of joint learning, including the above as well as 3D object pose estimation and spatio-temporal data aggregation.