2024 Pedestrian Attribute Recognition and Person Retrieval Challenge at WACV

Challenge description

Overview

The Task: The challenge will use an extension of the UPAR Dataset [1,2], which consists of images of pedestrians annotated for 40 binary attributes. For deployment and long-term use of machine-learning algorithms in a surveillance context, the algorithms must be robust to domain gaps that occur when the environment changes. This challenge aims to spotlight the problem of domain gaps in a real-world surveillance context and highlight the challenges and limitations of existing methods to provide a direction for future research.

The Dataset: Analogous to the first UPAR challenge, the challenge will use an extension of the UPAR dataset [1,2]. The challenge dataset consists of the harmonization of three public datasets (PA100K [3], PETA [4], and Market1501-Attributes [5]) for training and validation, and a novel test set. 40 binary attributes have been unified between those for which we provide additional annotations. This dataset enables the investigation of PAR methods' generalization ability under different attribute distributions, viewpoints, varying illumination, and low resolution.

The Tracks: This challenge is split into two tracks associated with semantic pedestrian attributes, such as gender or clothing information: Pedestrian Attribute Recognition (PAR) and attribute-based person retrieval. Both tracks build on the same data sources but will have different evaluation criteria. Each track evaluates how robust a given method is to domain shifts. In contrast to last year’s challenge, multiple domains will be available for training and validation. The test data for the final challenge ranking originates from a different and previously unseen domain.

• Track 1: Pedestrian Attribute Recognition: The task is to train an attribute classifier that accurately predicts persons’ semantic attributes, such as age or clothing information, under domain shifts.

• Track 2: Attribute-based Person Retrieval: Attribute-based person retrieval aims to find persons in a huge database of images called gallery that match a specific attribute description. The goal of this track is to develop an approach that takes binary attribute queries and gallery images as input and ranks the images according to their similarity to the query.

The Phases: Each track will be composed of two phases, i.e., the development and test phases. During the development phase, public training data will be released, and participants must submit their predictions concerning a validation set. At the test (final) phase, participants will need to submit their results for the test data, which will be released just a few days before the end of the challenge. As we progress into the test phase, validation annotations will become available together with the test images for the final submission.

At the end of the challenge, participants will be ranked using the test data. It is important to note that this competition involves submitting results and code. Therefore, participants will be required to share their code and trained models after the end of the challenge (with detailed instructions) so that the organizers can reproduce the results submitted at the test phase in a code verification stage. At the end of the challenge, top-ranked methods that pass the code verification stage will be considered valid submissions and compete for any prize that may be offered.

Important Dates

Tentative Schedule already available.

Dataset

Detailed information about the dataset can be found here.

How to enter the competition

We provide a starting kit that helps you get the data and annotations needed to participate in the challenge. You can find it here. Please follow the instructions.

The competition will be run on the Codabench. First, register on Codabench in the following links to submit results during the development and test phase of the challenge. Then, pick a track (or all tracks) to follow and train on the respective training splits. The validation and testing data will remain the same across all challenges.

By submitting result files to the Codabench challenges following the format provided in the starting kit and complying with the challenge rules, the submission will be listed on the leaderboard and ranked.

• Track 1: Pedestrian Attribute Recognition (competition link): Train on predefined data and evaluate generalization properties.

• Track 2: Attribute-based Person Retrieval (competition link): Train on predefined data and evaluate generalization properties.

The participants will need to register through the platform, where they can access the data and submit their predictions on the validation and test data (i.e., development and test phases) and obtain real-time feedback on the leaderboard. The development and test phases will open/close automatically based on the defined schedule.

Starting Kit

The starting kit includes a download script that downloads the sub-datasets, the annotations for the development phase, and submission templates for both tracks. The submission templates for both tracks include random predictions and show the expected submission format. More details can be found in the following “Making a submission” section. Participants are required to make submissions using the defined templates, by changing the random predictions/rankings by the ones obtained by their models. Note, the evaluation script will verify the consistency of submitted files and may invalidate the submission in case of any inconsistency.

Warning: the maximum number of submissions per participant at the test stage will be set to 3. Participants are not allowed to create multiple accounts to make additional submissions. The organizers may disqualify suspicious submissions that do not follow this rule.

Making a submission

Training data consists of image files and associated attribute annotations. There is one file for named “train.csv” for both tasks. The file contains one row per training image and 41 columns. The first column specifies the image path; the others include binary attribute annotations. The value 1 indicates the presence of an attribute, and 0 stands for the absence, respectively. 

• Track 1: Extract attribute predictions from the trained model for all images in "val.csv" ("test.csv" during the testing phase) and store the results in a file named "predictions.csv". Submission files should have the same format as the annotation file. I.e., they contain as many rows as there are images in "val.csv"/"test.csv", and image paths and respective attribute predictions are stored in the columns in the same order as in the annotation files. It is important to keep the order of images as specified in the "val.csv"/"test.csv" files. We recommend simply concatenating your attribute predictions to the "val.csv"/"test.csv" files to avoid any issues. Finally, zip the submission files and upload the zip file to the Codabench platform.

• Track 2: We provide two types of evaluation files for the attribute-based retrieval task: files that contain the attribute queries (“val/test_queries.csv”) and files that include the gallery images (“val/test_imgs.csv”). Query files include one query per row. In each row, a unique combination of the 40 binary attributes is given that represents the attribute set that should be searched. The gallery files are simply a list of images that should be sorted according to their similarity to the queries. Submission files should contain one row per query with the ranking positions of each gallery image separated by commas. The following table provides an example. Please note that the first row and column are just for visualization purposes and should not be included in the submission files.

For each query, the ranking positions are assigned to the four gallery images. It is important to keep the order of gallery images as defined in the "val/test_imgs.csv" files. Concerning Query 1, image a.png ranks first based on the algorithm, followed by images b.png, c.png, and d.png. Regarding the second query, the method considers b.png the most similar, followed by c.png, a.png, and d.png. Finally, for submission, create a file "rankings.csv" and zip it.

Then, sign in on Codabench -> go to our challenge webpage (and associated track) on Codabench -> go on the "My Submissions" tab -> then select your "the_filename_you_want.zip" file and -> submit. 

Warning: the last step ("submit") may take some minutes on Track 2 (e.g., >10min) with status "Running" due to the amount of computation and available Codabench resources (just wait). If everything goes fine, you will see the obtained results on the leaderboard ("Results" tab).

Note, Codabench will only keep the last valid submission on the leaderboard. This helps participants to receive real-time feedback on the submitted files. Participants are responsible to upload the file they believe will rank them in a better position as a last and valid submission.

Evaluation Metric

Different evaluation metrics are used for the two tracks:

• Track 1: Harmonic mean from mA and instance-based F1

• Track 2: mADM [7]

The calculation works as follows. First, the metrics are computed separately for each of the datasets included in the validation set and then averaged across the splits.

Basic Rules

According to the Terms and Conditions of the Challenge, the following rules apply.

• Models cannot be trained on any additional real or synthetic data except ImageNet. Pre-training with COCO (or any other dataset) is not allowed.

• Any use of the test images is prohibited and will result in disqualification. The test data may not be used in any way, not even unsupervised, semi-supervised, or for domain adaptation.

• Validation data and released labels in the testing phase can be used to validate the method's performance as well as for training the model.

• Participants may train only one model.  Ensembles are not allowed. This model has to be used to compute predictions for the entire validation/test set. The participants are not allowed to use different approaches/models/hyper-parameter sets/etc. for different subsets of the validation/test data. 

• The maximum number of submissions per participant at the test stage will be set to 3. Participants are not allowed to create multiple accounts to make additional submissions. The organizers may disqualify suspicious submissions that do not follow this rule.

• In order to win the challenge, top-ranked participants' scores must improve the baseline performance provided by the challenge organizers.

• The performances on test data will be verified after the end of the challenge during a code verification stage. Only submissions that pass the code verification will be considered in the final list of winning methods.

• To be part of the final ranking, the participants will be asked to fill out a survey (fact sheet) where detailed and technical information about the developed approach is provided.

Final Evaluation and Ranking

Important dates regarding code submission and fact sheets are defined in the schedule.

• Code verification: After the end of the test phase, participants are required to share with the organizers the source code used to generate the submitted results, with detailed and complete instructions (and requirements) so that the results can be reproduced locally (preferably using docker). Note that only solutions that pass the code verification stage are eligible to be announced in the final list of winning solutions. Participants are required to share both training and prediction code with pre-trained models. Participants are requested to share with the organizers a link to a code repository with the required instructions. This information must be detailed inside the fact sheets (detailed next).

  • Ideally, the instructions to reproduce the code should contain:
    1) how to structure the data (at train and test stage).
    2) how to run any preprocessing script, if needed.
    3) how to extract or load the input features, if needed.
    4) how to run the docker used to run the code and to install any required libraries, if possible/needed.
    5) how to run the script to perform the training.
    6) how to run the script to perform the predictions, that will generate the output format of the challenge. The script must be able to generate predictions for any input images (Task 1) or query input image combinations (Task 2) specified in a text file (formats analogous to those provided for testing).

• Fact sheets: In addition to the source code, participants are required to share with the organizers a detailed scientific and technical description of the proposed approach using the template of the fact sheets provided by the organizers. Latex template of the fact sheets can be downloaded here.

Sharing the requested information with the organizers: Send the compressed project of your fact sheet (in .zip format), i.e., the generated PDF, .tex, .bib, and any additional files to <upar.challenge@gmail.com>, and put in the Subject of the E-mail "WACV 2024 UPAR Challenge / Fact Sheets and Code repository"

IMPORTANT NOTE: we encourage participants to provide detailed and complete instructions so that the organizers can easily reproduce the results. If we face any problem during code verification, we may need to contact the authors, and this can take time, and the release of the list of winners may be delayed.

Challenge Results (test phase)

We are happy to announce the winning solution of the WACV 2024 Pedestrian Attribute Recognition and Person Retrieval Challenge. The team had its code verified at the code verification stage. The associated fact sheets and link to code repository are available here (will be available soon). The organizers would like to thank all the participants for making this challenge a success.

• Pedestrian Attribute Recognition (PAR) Challenge (Track 1)
  • 1st place: hdcolab - Team Leader: Doanh C. Bui (School of Electrical Engineering, Korea University). Team members: Hung Ba Ngo (Graduate School of Data Science, Chonnam National University), Thinh V. Le (University of Information Technology, Vietnam National University).
  • 2nd place: DS - Team Leader: Jonghu Jeong (Deeping Source Inc.). Team members: Minyong Cho, Seungkwan Lee, Hoyong Jeong, Philipp Benz (All members are from Deeping Source Inc.)

Associated Workshop

Check the associated Real-World Surveillance: Applications and Challenges Workshop

[1]  Cormier, Mickael; Specker, Andreas; Jacques, Julio C. S. et al. (2023): UPAR Challenge: Pedestrian Attribute Recognition and Attribute-based Person Retrieval – Dataset, Design, and Results – link

[2] Specker, Andreas; Cormier, Mickael; Beyerer, Jürgen (2023): UPAR: Unified Pedestrian Attribute Recognition and Person Retrieval – link

[3] Liu, Xihui; Zhao, Haiyu; Tian, Maoqing; Sheng, Lu; Shao, Jing; Yi, Shuai et al. (2017): HydraPlus-Net: Attentive Deep Features for Pedestrian Analysis. In: 2017 IEEE International Conference on Computer Vision. ICCV 2017 : proceedings : 22 - 29 October 2017, Venice, Italy. Unter Mitarbeit von Katsushi Ikeuchi. 2017 IEEE International Conference on Computer Vision (ICCV). Venice, 10/22/2017 - 10/29/2017. Institute of Electrical and Electronics Engineers. Piscataway, NJ: IEEE (IEEE Xplore Digital Library), S. 350–359.

[4] Deng, Yubin; Luo, Ping; Loy, Chen Change; Tang, Xiaoou (2014): Pedestrian Attribute Recognition At Far Distance. In: Kien A. Hua (Hg.): Proceedings of the 22nd ACM international conference on Multimedia. the ACM International Conference. Orlando, Florida, USA, 03.11.2014 - 07.11.2014. New York, NY: ACM, S. 789–792.

[5] Lin, Yutian; Zheng, Liang; Zheng, Zhedong; Wu, Yu; Hu, Zhilan; Yan, Chenggang; Yang, Yi (2019): Improving Person Re-identification by Attribute and Identity Learning. In: Pattern Recognition 95, S. 151–161. DOI: 10.1016/j.patcog.2019.06.006.

[6] Jia, J., Huang, H., Chen, X., & Huang, K. (2021). Rethinking of pedestrian attribute recognition: A reliable evaluation under zero-shot pedestrian identity setting. arXiv preprint arXiv:2107.03576.

[7] Specker, A., & Beyerer, J. (2023, July). Balanced Pedestrian Attribute Recognition for Improved Attribute-based Person Retrieval. In 2023 IEEE 13th International Conference on Pattern Recognition Systems (ICPRS) (pp. 1-7). IEEE.