Human-centric Spatio-Temporal Video Grounding

Important Date

Time zone: Beijing, UTC+8

Task Rules

1. The results should be stored in results.json, with the following format:
   {
    'video_id': {
    'st_frame': st,
    'ed_frame': ed,
    'bbox': {
     'st': [x1, y1, x2, y2],
     'st+1':[x1, y1, x2, y2],
     ...
     'ed':[x1, y1, x2, y2],
     }
    },
   }
2. Using extra training data are not allowded. But the usage of pretrain model has no limitations.
3. You have 10 submission chances in total.
4. The evaluation process can takes times. And a failed submission will not cause the reduction of submission chances.

Task Metric

1. \(vIoU\): \(vIoU = \frac{1}{\left | S_u \right |} \sum_{t \in S_i} IoU(Box^t, Box^{t'})\), where \(S_i\) is the set of frames in the intersection of selected and ground truth tube, \(S_u\) is the set of frames in the union of selected and ground truth tube, \(Box^t\) and \(Box^{t'}\) are predicted bounding box and ground truth bounding box of frame \(t\). vIoU can directly reflect the accuracy of the prediction results spatiotemporally.
2. \(vIoU@R\) stands for the percentage of samples whose \(vIoU\) is larger than \(R\)
3. \(mvIoU\) stands for mean value of \(vIoU\).

Dataset

Human-centric Spatio-Temporal Video Grounding(HC-STVG), which only focuses human in the videos. We provide 16k annotation-video paris with different movie scenes. Specifically, we annotated the description statement and all the trajectories of the corresponding person (a series of Bounding Boxes). It's worth noting that all of our clips will include multiple people to increase the challenge of video comprehension.

Data Statistics

Dataset	Total	Trainval	Test	Video_len
HCVG	16685	12000	4665	20s

Image Example



Organizers

Home page

Zongheng Tang
Beihang University
tzhhhh123@buaa.edu.cn
Home page

Si Liu
Beihang University
liusi@buaa.edu.cn
Home page

Luoqi Liu
Qihoo/360
liuluoqi@360.cn
Home page

Yafeng Deng
Qihoo/360
dengyafeng@360.cn

Citation

@article{tang2020human,
title={Human-centric Spatio-Temporal Video Grounding With Visual Transformers},
author={Tang, Zongheng and Liao, Yue and Liu, Si and Li, Guanbin and Jin, Xiaojie and Jiang, Hongxu and Yu, Qian and Xu, Dong},
journal={arXiv preprint arXiv:2011.05049},
year={2020}
}