Discriminative cosine embedding in machine learning

This application claims the benefit of U.S. Provisional Application No. 62/761,144, filed Mar. 12, 2018, which is incorporated herein by reference in its entirety.

This invention was made with government support under contract 20131JCXK005, awarded by the U.S. Department of Justice and contract W911 NF16C0040, awarded by the U.S. Army. The government has certain rights in the invention.

Recent developments in deep neural networks have addressed a wide array of components. Considerable effort has been directed to developing deeper structures and more effective non-linearities. Apart from structural developments, thee have also been efforts directed to combating over-fitting and obtaining better gradients. Recent studies have recognized the importance of stronger loss functions. A robust loss function which encourages highly discriminate feature learning is a direct way to provide a network with more informed gradients towards the ultimate supervised task. A fully connected layer coupled with the cross-entropy loss and the Softmax layer, which is referred to collectively as the Softmax loss herein, is arguably the most prevalent prior art loss function. The Softmax loss has proved to be versatile and is able to provide reasonably good gradients owing to the well-behaved cross-entropy loss. Some efforts have been directed to modifying the Softmax loss to increase discrimination in terms of larger angular margins, or to normalizing the features input to Softmax, thereby solving a non-convex problem.

A different way to obtain highly discriminative features involves minimizing alternate loss functions or augmenting the Softmax with supplementary losses. Contrastive loss and the Triplet loss replace the Softmax loss with losses which focus on learning a discriminative embedding while trying to minimize intra-class variation in the learned features. This is accomplished by sampling training pairs or triplet sets which leads to expensive hard-sample mining in large-scale applications. Center loss on the other hand, is an approach which ignores hard-sample mining while only trying to minimize intra-class variation along with the Softmax loss. Another approach, which was proposed to work on random batches, is the Multi-batch estimator. Multi-batch is an example of the metric learning approach which, for a batch size of in, utilizes all m2-m pairs for a better estimate of the gradient.

The aforementioned works all operate in the l 2 or Euclidean space. Although l 2 embeddings perform well in many applications, the performance gain they provide is limited in situations wherein the number of samples per class is high. In such situations, forcing all samples from a class to converge toward each other in the l 2 sense is too difficult a task because it requires that the network converge not only in angle but also the norm of the features.

Further, during testing in typical supervised classification such as face recognition, the most common metric is the cosine distance which ignores the norm. This creates a disconnect between training and testing because the network learns a behavior (to constrain the norm as well) that is ignored during testing. Such a framework is inefficient.

Recently, congenerous cosine (“COCO”), a form of cosine loss, was proposed for person recognition. COCO minimizes intra-class variation center and maximizes inter-class variation of samples with the centers of other classes as opposed to the global batch center, which significantly raises computational complexity. The approach uses hard normalization and is similar to recent other studies, all of which formulate a non-convex constraint.

The Copernican Loss (L C ) disclosed herein has two important properties. First, L C is designed to augment the standard Softmax loss while explicitly minimizing intra-class variation and simultaneously maximizing inter-class variation. Second, L C operates using the cosine distance and thereby directly affects angles leading to a cosine embedding, which removes the disconnect between training and testing. This improves efficiency because more of the model complexity is utilized to learn a more discriminative embedding rather than learning to constrain the norm.

Copernican loss does not require hard sample mining or data augmentation of any kind and can be effectively minimized using stochastic gradient descent (SGD) on random mini-batches. L C only needs to maintain a center for each class called the “Planet center” and computes the mean of the mini-batch called the “Sun center”. To minimize intra-class variation, it minimizes the cosine distance of the samples to their corresponding Planet centers. To discriminate between the samples within a minibatch, Copernican loss maximizes the cosine distance of the samples away from the mean of the mini-batch (the Sun center). This eliminates the need to compute pair-wise gradients such as the Multi-batch while providing the similarly discriminative gradients in a more efficient manner.

FIG. 1 shows a sample batch in 2d space comprising 8 samples, represented by hollow circles, in 3 classes, represented by solid circles. The solid dark circle in the center represents the global batch center (i.e., the “Sun”).

FIG. 2 shows an example of an ambiguous configuration.

FIG. 3 is a graph showing the norm of the difference between the MSE gradient and the cosine similarity gradient as dimensionality increases along with the angle between the two

Copernican loss minimizes the sum of planet loss and sun loss. Planet loss minimizes the intra-class variation and pushes the samples towards the class centers under the angular or cosine metric. Sun loss maximizes the inter-class variation by moving all samples away from the global batch center. An example of this is shown in FIG. 1 , which shows a sample batch in of 2D features in 2D space comprising eight samples in three classes. The dotted black lines denote the gradient direction for the sun loss and the solid thin lines denote the gradient direction from planet loss.

The Need for Simultaneous Discrimination and Invariance.

Learning robust features is a key problem in supervised learning. Robustness, as used herein, refers to two specific properties of a useful feature. 1) Invariance to intra-class nuisance transformations while being 2) discriminative to inter-class transformations. Although loss functions exist which implicitly optimize for this criterion, such as the Softmax loss and negative log likelihood, the features learned using pure forms of the loss functions are not robust enough for harder classification tasks. Thereby, explicit simultaneous maximization of intra-class similarity and inter-class discrimination is critical. In contrast to some known loss functions. Copernican loss explicitly optimizes for both objectives (invariance and discrimination) between all samples in a mini-batch. Although, constrastive embedding and Triplet loss both advocate simultaneous optimization of discrimination and invariance, they search for useful gradients through sample mining.

One way to mitigate the need for mining is to discriminate all samples belonging to different classes away from each other. Copernican loss does this by moving the samples away from the global batch mean, which efficiently provides discriminative gradients without sample mining.

The Need for a Cosine Embedding.

Classification in machine learning, at its fundamental level, is typically conducted using the inner-product. The inner-product between the weight vector ω and the sample x is a product of their norms and the cosine of the angle between the two vectors. The ideal classification vector ω would provide a high inner-product with a sample from the correct class and a low inner-product for samples from impostor classes. Thus it is desirable ω T x correct >ω T x imposter ⇒∥x correct ∥ 2 cos(θ correct )>∥x imposter ∥ 2 cos(θ imposter ). The classification decision therefore, ultimately rests on the norm of the sample x and the cosine angle of the weight vector. In this light, there are two ways of increasing discrimination. 1) Increase the norm of the correct class; and/or 2) decrease the cosine angle between the weight vector and the sample. For binary class problems, increasing the norm of one class over another may be feasible, however for multi-class problems this approach would not be effective.

Maximizing the norm of samples from one particular class over all others, would hinder correct classification of other classes. Thereby, one approach to increase discrimination that can be applied to multi-class problems is to maximize the angle (or equivalently the cosine distance) between classes. This reasoning also applies to the Softmax loss function which is arguably the most commonly used loss in supervised deep learning. Minimizing the intra-class cosine distance while simultaneously maximizing the inter-class cosine distance is optimized by the Copernican loss disclosed herein.

Limitations on an l 2 embedding. There have been multiple loss functions proposed that learn an l 2 embedding such as the Center loss, Triplet loss and Multi-batch. These losses explicitly minimize the l 2 distance between samples from the same class. For sample features x 1 and x 2 from the same class, ∥x 1 −x 2 ∥ 2 2 =∥x 1 ∥ 2 2 +∥x 2 ∥ 2 2 −2∥x 1 ∥ 2 ∥x 2 ∥ 2 cos(θ), where θ is the angle between the two samples. Minimizing this quantity requires both minimizing the difference between norm of the features x 1 and x 2 and also minimizing the cosine distance between the two. There are two downsides to this approach.

The first downside is that, during testing and extraction of a similarity score (e.g. for open set face or object feature extraction), only the cosine distance is taken into account. This implies that explicitly constraining the norm is inefficient for the loss function layer from the perspective of matching because the model needs to learn sub-tasks (i.e., constraining the norm) that is not required during testing. Successful approaches, such as batch normalization, do not require the model or the weights themselves to perform normalization; it is instead performed explicitly through the normalization operation. This allows the model complexity to be used to focus on the angles between the samples.

The second downside is that, for tasks with a large number of samples per class (such as typical object recognition), having a large set of samples per class all converge to the same point in the Euclidean sense proves to be a difficult challenge for a deep network. It is much easier to simply have the angle of the sample features converge as opposed to convergence in the l 2 sense. With this approach, the network does not need to constrain norms, and model complexity is better utilized in creating a larger angular margin resulting in better performance during testing.

This is likely the reason that none of the previous works on learning l 2 embeddings report results on object recognition datasets with a large number of samples per class. Examples include datasets such as CIFARlO and CIFARlOO which have fewer classes (10 and 100 respectively) but many more samples per class (6000 and 600 images per class respectively).

The focus of those works is mainly face recognition which is characterized by a large number of classes (e.g. above 10,000 for CASIA-WebFace) with relatively few samples per class (average of about 50 samples per class for CASIA-WebFace).

Copernican Loss—

For a batch of size m, Copernican loss, denoted by L C , is defined as the sum of three loses as follows:

Here, L P is the Planet loss minimizing intra-class cosine variation, L S is the Sun loss, maximizing inter-class cosine variation and L Soft is the Softmax loss. β is the margin for the Sun loss, s is the global center for the particular batch (i.e., the “sun”) and the p's are the class centers (i.e., the “planets”).

In the ideal case, s and p's would represent the class centers and the global center of the entire dataset. However, computing these quantities over the entire dataset would be very expensive, especially for large-scale applications. To get around this problem, the global center (sun s) of the entire dataset can be approximated with the global center of each mini-batch. Therefore, for every batch,

The computation of the class centers (planets p) also face a similar issue due to scale. However, class centers cannot be effectively estimated using a single batch due to high variance, especially in the early stages of training. One way to obtain a better estimate is to maintain a class center and update it with every batch in the following sense,

Here, it is assumed that there are n instances of class i in the minibatch and α is the update weight factor and is usually set to a small value (say 0.05). Compared to direct computation of class centers over the mini-batch set, this update provides more robustness to sample perturbation while simultaneously addressing the problem of scalability in estimating the centers. Copernican loss is simple to implement and, unlike Triplet loss, it does not require hard sampling mining which would have increased the computation complexity of the overall loss. Further, computation of discriminative gradients only requires m gradient computations compared to m(m−1) for Multi-batch.

Resolving an Ambiguous Configuration.

An example of an ambiguous configuration is shown in FIG. 2 . Here, four samples are mapped into the feature space during early training when classes are mixed. The algorithm needs to pick pairs of samples from different classes to take a gradient step towards better discrimination. Picking the Alice-Carol pair would not help the configuration as they have sufficient separation (>1) and neither will Bob-Alice pairs. Only Carol-Bob pairs will provide useful updates by moving Bob away. Center loss would worsen things since it will push the two Alice's closer to Bob. Triplet loss would need to sample Carol-Bob, which is expensive to determine by hard-sample mining and unlikely to determine if no sample mining is performed. Multi-batch solves this problem by considering all pairs in the minibatch. Copernican loss is guaranteed to provide discriminative gradients for all samples without the need for sample mining or m(m−2) gradient computations (only m are needed), because it moves samples away from the sun center (i.e., the black dot in FIG. 2 ). To bring the Alice pair closer, planet loss provides a gradient which will be effective since Bob has moved out of the way due to the sun loss.

Optimization—The gradients of Copernican loss are straight forward. Since the sun center and planet centers are updated during the forward pass, only gradients with respect to the input x i are required to be derived which are as follows:

∂ L P ∂ x i = - 1 m ⁢ ∑ i m 1  x i  2 ⁢ ( p y i  p y i  2 - cos ⁡ ( x i , p y i ) ⁢ x i  x i  2 ) ( 3 ) ∂ L S ∂ x i = { 1 m ⁢ ∑ i m 1  x i  2 ⁢ ( s  s  2 - cos ⁡ ( x i , s ) ⁢ x i  x i  2 ) ( 4 )

where

∂ L S ∂ x i

becomes 0 only if cos(x i , s)<β. It is interesting to note that the gradient direction of the cosine similarity is different from the mean squared error (MSE) gradient by a scaled version of the sample feature. This scaling factor is the cosine angle between the sample feature and the target vector. FIG. 3 shows the norm of the difference between the MSE gradient and the cosine similarity gradient as dimensionality increases along with the angle between the two. For all dimensions above 10, the angle between the cosine similarity gradient and MSE gradient is about 20°. It is interesting to note that a gradient direction differing by 20° can result in significantly improved performance. As the sample features converged to the planets or class centers, the MSE gradient direction converges to the cosine similarity gradient direction, since cos(x i , p y i )→1. This fact, coupled with superior performance of the cosine similarity, leads to the speculation that during the initial stages of training, when the features are not informative of the class structure, MSE or l 2 embedding gradients are noisier. Cosine embeddings obtain better gradients by ignoring this noise because it only looks at angular margins which are not affected by the norm of the vectors.

A method of improving the discrimination capabilities of derp neural networks, known as Copernican loss, has been presented wherein feature vectors are adjusted to minimize intra-class variation and maximize inter-class variation by learning a discriminative cosine embedding. Copernican loss discriminates between all samples from different classes within a batch of size m with just m gradient computations.