Transformer-based multi-scale pedestrian re-identification method

This application claims the priority of Chinese Patent Application No. 202211404764.8, filed on Nov. 10, 2022, which is hereby incorporated by reference in its entirety.

The present disclosure relates to pedestrian re-identification technology, specifically a Transformer-based multi-scale pedestrian re-identification method, which belongs to the field of computer vision.

Pedestrian re-identification aims to correlate pedestrian images captured by different cameras to realize cross-camera and cross-scenario pedestrian recognition and retrieval, which is widely used in the field of intelligent monitoring. In recent years, the pedestrian re-identification method based on the deep convolutional network has achieved excellent results, but CNN is limited by the fact that it can only process adjacent features each time and it is easy to lose the fine-grained features of pedestrians in the process of downsampling, resulting in a decline recognition rate. Meanwhile, the Transformer has shown excellent modeling capabilities for both spatial and sequential data. Compared to CNN, the Transformer eliminates downsampling operations, allowing for the preservation of more fine-grained features.

Most Transformer-based pedestrian re-identification methods typically segment the entire image into image sequences and feed them into the Transformer network, and then use the global token to represent the pedestrian features. However, directly segmenting the image ignores some structural information and slows down the convergence speed of the Transformer network. Dividing a single-scale pedestrian feature into a sequence of features as input to the Transformer network disregards the multi-scale representation of pedestrian features. In the network output, the rich local pedestrian features are still not fully utilized, yet these features provide crucial fine-grained information for pedestrian re-identification.

Based on the above, the present invention proposes a Transformer-based multi-scale pedestrian re-identification method, which utilizes rich information contained in image features of multiple scales to optimize the feature extraction of pedestrian images.

The purpose of the present invention is to address the shortcomings of existing techniques by proposing a multi-scale pedestrian re-identification method based on Transformer. By constructing a feature cascading module, while retaining the low-dimensional detailed features of pedestrians, the support of high-dimensional features is introduced, and pedestrian features of multiple scales are obtained, which alleviates the requirements of the Transformer network for large training data sets, and can help the model quickly converge and improve performance. At the same time, this method divides the pedestrian features of multiple scales to construct a multi-scale feature sequence and inputs it into the same Transformer network after adding coding information representing different scales, so as to guide the model to pay attention to the pedestrian information at different scales. Furthermore, the local feature multi-scale fusion module is constructed to make full use of the multi-scale pedestrian fine-grained local features output from the network to construct a multi-scale feature set for each pedestrian, which guides the model to mine more robust pedestrian feature representations from local to global and shallow to deep.

The technical solutions adopted by the present invention to solve its technical problems are as follows:

Step (1): Introduce the feature cascading branch in the residual network ResNet50, and perform feature fusion on the feature maps of different scales in adjacent layers. Step (2): Segment the feature maps of different scales fused in step (1), and then add a learnable [SCALE_TOKEN] to each segmented feature group to represent different scales. Flatten the subgraphs divided by different scales, and add [POS_TOKEN] representing the position and the global feature [CLS_TOKEN] as the input of the transformer. Step (3): Construct a pedestrian feature extraction network based on standard Transformer, and input the feature subgraph vector obtained in step 2 into the network to obtain pedestrian features. Step (4): Construct the local feature multi-scale fusion module, slice the features obtained in step (3), take the [CLS_TOKEN] vector as the global feature, and at the same time input the local features with different scales into the local feature multi-scale fusion module and re-slice them to obtain the final features. Step (5): Use the [CLS_TOKEN] vector and the multi-scale fusion vector obtained in step (4) to train according to the training strategy to obtain the final ReID model.

Further, the specific implementation process of the step (1) is as follows:

Step 1-1: Utilize ResNet50 pre-trained on ImageNet as the backbone network, retain the first pooling stage and the first three stages of the backbone network, while removing the final stage, spatial down-sampling operations, global average pooling layer, and fully connected layer. Step 1-2: Obtain the feature information of a total of 4 stages retained in step 1-1 from ResNet50, and construct a multi-scale feature cascade module, and obtain pedestrian feature information through pairwise fusion between adjacent layers. First, upsample and perform 1×1 convolution on the features obtained from Stage 1, increasing the size of the feature maps to twice their original size. Then, perform feature fusion between the features from Stage 1 and Stage 2, Stage 2 and Stage 3, and Stage 3 and Stage 4. As shown in formula (1) (2):

F sc =Contact(UpSample( F s ), F b )  (1)

F agg =conv 1×1 ( F sc )  (2)

F s , F b respectively represent two features that need to be fused, UpSample represents the upsampling operation, Contact is the vector connection operation, F agg the obtained fusion feature.

Further, the specific implementation process of the step (2) is as follows:

Step 2-1: For the obtained fusion feature F agg , divide it according to the sub-feature map size ps,generate a feature map sequence x p ={F agg i |i=1,2, . . . N}, N the number of splits, and then add a learnable [SCALE_TOKEN] to x p . [SCALE TOKEN] has the same dimension size as x p , as shown in formula (3):

x p k =x p +E scale   (3)

x p k is the sequence of feature maps generated at the k scale; E scale is [SCALE_TOKEN], indicating the learnable feature scale.

In summary, add [CLS_TOKEN] and [POS_TOKEN] to the feature processing of the three scales obtained in step 1-2 to obtain feature Z, as shown in formula (4):

Z=[x cls ;x p 1 ;x p 2 ;x p 3 ]+E pos   (4)

x cls is the global feature vector of [CLS_TOKEN]; x p 1 , x p 2 , and x p 3 are feature sequences of three scales respectively; E pos is [POS_TOKEN], indicating the spatial position.

Further, the specific implementation process of the step (3) is as follows:

Step 3-1: In order to fully utilize the multi-scale information, construct a multi-scale feature extraction model for pedestrians based on Transformer. The model is mainly composed of multi-layer stacked Transformer blocks. A single Transformer layer is composed of multi-head attention mechanism MSA, layer normalization LN and multi-layer perceptron MLP. A single Transformer block can be formulated as (5) (6):

Z′ l =MSA ( LN ( Z l-1 ))+ Z l-1 l= 1 . . . L   (5)

Z l =MLP ( LN ( Z′ l ))+ Z′ l l= 1 . . . L   (6)

Z l represents the output feature of the l-th layer Transformer block, Z′ l is the intermediate result in the Transformer block, and L is the total number of layers. For the multi-scale feature Z generated by each image, as the input of the Transformer network, the network output result is the output feature of the last layer.

Further, the specific implementation process of the step (4) is as follows:

Step 4-1: After obtaining the output feature Z L of the Transformer model, divide the feature Z L to obtain four sets of feature vectors, that is, the global feature f cls and three features f scale_1 , f scale_2 and f scale_3 of different scales; the features of different scales are double Linear interpolation, and then perform feature fusion through 1*1 convolution to obtain the final local feature f final , then, according to the pedestrian structure, f final can be divided into four local features f 1 , f 2 , f 3 , f 4 .

Further, the specific implementation process of the step (5) is as follows:

Step 5-1: Use the labeled data in the pedestrian re-identification dataset as supervision information, and use ID loss and difficult triplet loss to train the network for each training batch; ID loss uses cross-entropy loss to train the network, and the formula is as follows:

L id =Σ i=1 N −q i log( p i )  (7)

N is the number of pedestrian categories, q i is the supervised label, and p i is the predicted label.

Difficult triplet loss randomly samples P identities, and extracts K instances from each share to form a mini batch of size P*K; selects each picture x a in the batch as an anchor point in turn, and selects the farthest positive sample picture x p and the nearest negative sample picture x n in the batch to form a triplet to train the network and enhance the generalization ability of the network. The formula is:

k represents the number of output feature groups.

Step 5-3: When the model is stable, get the final ReID model, input the image to be queried and the test set image into the final ReID model for feature extraction, compare whether the features of the query image and the test set image belong to the same category, and output pedestrian images of the same type.

The beneficial effects of the present invention are as follows:

The present invention introduces a Transformer-based multi-scale pedestrian re-identification method. By utilizing a ResNet feature concatenation module, the method collects pedestrian features at different scales, ensuring that low-dimensional fine-grained details are preserved while incorporating high-dimensional semantic features. This enables the model to learn more robust pedestrian representations. Moreover, the multi-scale information guides the model to pay attention to pedestrian features at different scales. The model explores pedestrian latent information from global to local perspectives and from shallow to deep layers. Furthermore, to fully leverage the fine-grained local features of pedestrians, the present invention incorporates a local feature multi-scale fusion module. This module integrates and splits the information from different scales provided by the network output, allowing the model to focus on the feature information of different parts and scales of the pedestrian. The results demonstrate that this approach extracts features with enhanced robustness, effectively improving the model's generalization ability.

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 represents the overall implementation process flowchart of the present invention;

FIG. 2 illustrates the schematic diagram of the ResNet50 feature concatenation module in the present invention;

FIG. 3 depicts the schematic diagram of the feature segmentation process in the present invention;

FIG. 4 illustrates the schematic diagram of the Transformer network in the present invention;

FIG. 5 presents the schematic diagram of the multi-scale local feature fusion module in the present invention;

FIG. 6 showing the overall structural diagram of the present invention.

The present invention will be further described below in conjunction with the accompanying drawings.

Referring to FIG. 1 , which illustrates the overall implementation process flowchart of the present invention, the invention involves constructing a feature concatenation module based on ResNet to extract features from images at different scales and perform pairwise concatenation. This process allows for the preservation of fine-grained details while incorporating support from higher-level features. Additionally, the [SCALE_TOKEN] is introduced to each patch, indicating the scale of the feature information within the transformer network. In addition, by fully leveraging the multi-scale local features of the network, the present invention explores the latent information between pedestrians across different parts and scales, effectively enhancing the model's generalization ability. As shown in FIG. 6 , the present invention comprises the following steps:

Step (1): As shown in FIG. 2 , introduce the feature cascading branch in the residual network ResNet50, and perform feature fusion on the feature maps of different scales in adjacent layers. Step (2): As shown in FIG. 3 , segment the feature maps of different scales fused in step (1), and then add a learnable [SCALE_TOKEN] to each segmented feature group to represent different scales. Flatten the subgraphs divided by different scales, and add [POS_TOKEN] representing the position and the global feature [CLS_TOKEN] as the input of the transformer. Step (3): As shown in FIG. 4 , construct a pedestrian feature extraction network based on standard Transformer, and input the feature subgraph vector obtained in step 2 into the network to obtain pedestrian features. Step (4): As shown in FIG. 5 , construct the local feature multi-scale fusion module, slice the features obtained in step (3), take the [CLS_TOKEN] vector as the global feature, and at the same time input the local features with different scales into the local feature multi-scale fusion module and re-slice them to obtain the final features. Step (5): Use the [CLS_TOKEN] vector and the multi-scale fusion vector obtained in step (4) to train according to the training strategy to obtain the final ReID model.

Further, the specific implementation process of the step (1) is as follows:

Step 1-1: Utilize ResNet50 pre-trained on ImageNet as the backbone network, retain the first pooling stage and the first three stages of the backbone network, while removing the final stage, spatial down-sampling operations, global average pooling layer, and fully connected layer. Step 1-2: Obtain the feature information of a total of 4 stages retained in step 1-1 from ResNet50, and construct a multi-scale feature cascade module, and obtain pedestrian feature information through pairwise fusion between adjacent layers. First, upsample and perform 1×1 convolution on the features obtained from Stage 1, increasing the size of the feature maps to twice their original size. Then, perform feature fusion between the features from Stage 1 and Stage 2, Stage 2 and Stage 3, and Stage 3 and Stage 4. As shown in formula (1) (2):

F sc =Contact(UpSample( F s ), F b )  (1)

F agg =conv 1×1 ( F sc )  (2)

F s , F b respectively represent two features that need to be fused, UpSample represents the upsampling operation, Contact is the vector connection operation, F agg is the obtained fusion feature.

Further, the specific implementation process of the step (2) is as follows:

Step 2-1: For the obtained fusion feature F agg , divide it according to the sub-feature map size ps, generate a feature map sequence x p ={f agg i |i=1,2, . . . N} is the number of splits, and then add a learnable [SCALE_TOKEN] to x p . [SCALE TOKEN] has the same dimension size as x p , as shown in formula (3):

x p k =x p +E scale   (3)

x p k is the sequence of feature maps generated at the k-th scale; E scale is [SCALE_TOKEN], indicating the learnable feature scale.

In summary, add [CLS_TOKEN] and [POS_TOKEN] to the feature processing of the three scales obtained in step 1-2 to obtain feature Z, as shown in formula (4):

Z=[x cls ;x p 1 ;x p 2 ;x p 3 ]+E pos   (4)

x cls is the global feature vector of [CLS_TOKEN]; x p 1 , x p 2 , and x p 3 are feature sequences of three scales respectively; E pos is [POS_TOKEN], indicating the spatial position.

Further, the specific implementation process of the step (3) is as follows:

Step 3-1: In order to fully utilize the multi-scale information, construct a multi-scale feature extraction model for pedestrians based on Transformer. The model is mainly composed of multi-layer stacked Transformer blocks. A single Transformer layer is composed of multi-head attention mechanism MSA, layer normalization LN and multi-layer perceptron MLP. A single Transformer block can be formulated as (5) (6):

Z′ l =MSA ( LN ( Z l-1 ))+ Z l-1 l= 1 . . . L   (5)

Z l =MLP ( LN ( Z′ l ))+ Z′ l l= 1 . . . L   (6)

Z l represents the output feature of the l-th layer Transformer block, Z′ 1 is the intermediate result in the Transformer block, and L is the total number of layers. For the multi-scale feature Z generated by each image, as the input of the Transformer network, the network output result is the output feature of the last layer.

Further, the specific implementation process of the step (4) is as follows:

Step 4-1: After obtaining the output feature Z L of the Transformer model, divide the feature Z L to obtain four sets of feature vectors, that is, the global feature f cls and three features f scale_1 , f scale_2 and f scale_3 of different scales; the features of different scales are double Linear interpolation, and then perform feature fusion through 1*1 convolution to obtain the final local feature f final ; then, according to the pedestrian structure, f final can be divided into four local features f 1 , f 2 , f 3 , f 4 .

Further, the specific implementation process of the step (5) is as follows:

Step 5-1: Use the labeled data in the pedestrian re-identification dataset as supervision information, and use ID loss and difficult triplet loss to train the network for each training batch; ID loss uses cross-entropy loss to train the network, and the formula is as follows:

L id =Σ i=1 N −q i log( p i )  (7)

N is the number of pedestrian categories, q i is the supervised label, and p i is the predicted label.

Difficult triplet loss randomly samples P identities, and extracts K instances from each share to form a mini batch of size P*K; selects each picture x a in the batch as an anchor point in turn, and selects the farthest positive sample picture x p and the nearest negative sample picture x n in the batch to form a triplet to train the network and enhance the generalization ability of the network. The formula is:

k represents the number of output feature groups.

Step 5-3: When the model is stable, get the final ReID model, input the image to be queried and the test set image into the final ReID model for feature extraction, compare whether the features of the query image and the test set image belong to the same category, and output pedestrian images of the same type.