2016 Looking at People ECCV Challenge - Track 1: First Impressions (first round)

The evaluation consists of computing the mean accuracy among the video tested for the 5 traits between the predicted continuous values and the continuous ground truth values.

If you want to test your method locally to see your results before submitting them, you can download the evaluation script here.

The file evaluateTraits.py perform the evaluation between the predictions produced by the participants methods and the ground-truth.

Once participants train their models, each video sample should output 5 continuous prediction values between the range [0, 1] corresponding to each trait. Then, participants may want to use the function in the script generatePredFile.py to generate their output predictions into a CSV file, which must have the same format as the ground-truth CSV file and is stored at:

>>> ./output/pred/predictions.csv

You can find an example of this file here. Thus, the predictions file that participants generate for the submission with their computed scores must have the same video names and number of entries in the same order as the example predictions file provided. So participants may want to use the example file as a reference of the ground-truth file for generating their predictions file in the proper format for submission, by replacing it to:

>>> ./data/gt/validation_gt.csv

Then, you can test your results using the script evaluateTraits.py. It reads the CSV files for the predictions and ground-truth and performs the evaluation. The resulting output is a list of 5 elements, which refer to the mean accuracies among all videos for each Big Five personality traits.

For each trait and a given video, the accuracy is computed simply as 1 minus the absolute distance among the predicted value and the ground truth value. The mean accuracy among all Big Five traits is also computed.

Note that the predictions should be in the same order as the ground-truth, so the first column of the CSV files should be identical both for the predictions and the ground truth.

Taking the last point into account, the main function of the script evaluateTraits.py can consider the different ground-truth files for the different data subsets as:

>>> gt = np.asarray(readScores("./data/gt/training_gt.csv"))

>>> gt = np.asarray(readScores("./data/gt/validation_gt.csv"))

>>> gt = np.asarray(readScores("./data/gt/test_gt.csv"))

For submitting the results, you just have to provide the predictions.csv file, that can be generated with the above script generatePredFile.py, and compress it into a ZIP file ready to upload as a submission.