We offer a benchmark suite together with an evaluation server, such that authors can upload their results and get a ranking regarding the different tasks (pixel-level semantic labeling). If you would like to submit your results, please register, login, and follow the instructions on our submission page.
The first SkyScapes task involves predicting a per-pixel semantic labeling of the image without considering higher-level object instance or boundary information or multi-class lane-markings.
To assess performance, we rely on the standard Jaccard Index, commonly known as the PASCAL VOC intersection-over-union metric IoU = TP ⁄ (TP+FP+FN) [1], where TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively, determined over the whole test set. Owing to the two semantic granularities, i.e. classes and categories, we report two separate mean performance scores: IoUcategory and IoUclass. In either case, pixels labeled as void do not contribute to the score.
It is well-known that the global IoU measure is biased toward object instances that cover a large image area. In street scenes with their strong scale variation this can be problematic. Specifically for traffic participants, which are the key classes in our scenario, we aim to evaluate how well the individual instances in the scene are represented in the labeling. To address this, we additionally evaluate the semantic labeling using an instance-level intersection-over-union metric iIoU = iTP ⁄ (iTP+FP+iFN). Again iTP, FP, and iFN denote the numbers of true positive, false positive, and false negative pixels, respectively. However, in contrast to the standard IoU measure, iTP and iFN are computed by weighting the contribution of each pixel by the ratio of the class’ average instance size to the size of the respective ground truth instance. It is important to note here that unlike the instance-level task below, we assume that the methods only yield a standard per-pixel semantic class labeling as output. Therefore, the false positive pixels are not associated with any instance and thus do not require normalization. The final scores, iIoUcategory and iIoUclass, are obtained as the means for the two semantic granularities. The same policy is applied to other benchmarks.
Detailed results including performances regarding individual classes and categories can be found here.
Use the buttons in the first row to hide columns or to export the visible data to various formats. Use the widgets in the second row to filter the table by selecting values of interest (multiple selections possible). Click the numeric columns for sorting.
| name | fine | coarse | 16-bit | depth | video | sub up | backbone | IoU class | iIoU class | dice class | recall class | precision class | IoU Category | iIoU Category | runtime [s] | paper | code | hidden |
Detailed results including performances regarding individual classes and categories can be found here.
Use the buttons in the first row to hide columns or to export the visible data to various formats. Use the widgets in the second row to filter the table by selecting values of interest (multiple selections possible). Click the numeric columns for sorting.
| name | fine | coarse | 16-bit | depth | video | sub up | backbone | IoU class | iIoU class | dice class | recall class | precision class | IoU Category | iIoU Category | runtime [s] | paper | code | hidden |
Detailed results including performances regarding individual classes and categories can be found here.
Use the buttons in the first row to hide columns or to export the visible data to various formats. Use the widgets in the second row to filter the table by selecting values of interest (multiple selections possible). Click the numeric columns for sorting.
| name | fine | coarse | 16-bit | depth | video | sub up | backbone | IoU class | dice class | recall class | precision class | runtime [s] | paper | code | hidden |
Detailed results including performances regarding individual classes and categories can be found here.
Use the buttons in the first row to hide columns or to export the visible data to various formats. Use the widgets in the second row to filter the table by selecting values of interest (multiple selections possible). Click the numeric columns for sorting.
| name | fine | coarse | 16-bit | depth | video | sub up | backbone | IoU class | dice class | recall class | precision class | IoU Category | runtime [s] | paper | code | hidden |
Detailed results including performances regarding individual classes and categories can be found here.
Use the buttons in the first row to hide columns or to export the visible data to various formats. Use the widgets in the second row to filter the table by selecting values of interest (multiple selections possible). Click the numeric columns for sorting.
| name | fine | coarse | 16-bit | depth | video | sub up | backbone | IoU class | dice class | recall class | precision class | IoU Category | runtime [s] | paper | code | hidden |
Detailed results including performances regarding individual classes and categories can be found here.
Use the buttons in the first row to hide columns or to export the visible data to various formats. Use the widgets in the second row to filter the table by selecting values of interest (multiple selections possible). Click the numeric columns for sorting.
| name | fine | coarse | 16-bit | depth | video | sub up | backbone | IoU class | dice class | recall class | precision class | runtime [s] | paper | code | hidden |
Detailed results including performances regarding individual classes and categories can be found here.
Use the buttons in the first row to hide columns or to export the visible data to various formats. Use the widgets in the second row to filter the table by selecting values of interest (multiple selections possible). Click the numeric columns for sorting.
| name | fine | coarse | 16-bit | depth | video | sub up | backbone | IoU class | dice class | recall class | precision class | runtime [s] | paper | code | hidden |
In addition to the previously introduced measures, we report additional meta information for each method, such as timings or the kind of information each algorithm is using, e.g. depth data or multiple video frames. Please refer to the result tables for further details.
[1] M. Everingham, A. S. M. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, “The Pascal Visual Object Classes challenge: A retrospective,” IJCV, vol. 111, iss. 1, 2014.