This repository contains the source code of our paper Structured Knowledge Distillation for Dense Prediction. It is an extension of our paper Structured Knowledge Distillation for Semantic Segmentation (accepted for publication in CVPR'19, oral).
We have update a more stable version of training the GAN part in the master branch.
If you want to transfer our pair-wise distilaltion and pixel-wise distillation in your own work or you want to use our trained models in the conference version, you can checkout to the old branck 'cvpr_19'.
Demo video for the student net (ESPNet) on Camvid
After distillation with mIoU 65.1:
Before distillation with mIoU 57.8:
This repository is organized as:
- libs This directory contains the inplaceABNSync modes.
- dataset This directory contains the dataloader for different datasets.
- network This directory contains a model zoo for network models.
- utils This directory contains api for calculating the distillation loss.
We apply the distillation method to training the PSPNet. We used the dataset splits (train/val/test) provided here. We trained the models at a resolution of 512x512. Pi: Pixel-wise distillation PA: Pair-wise distillation HO: holistic distillation
| Model | Average |
|---|---|
| baseline | 69.10 |
| +Pi | 70.51 |
| +Pi+Pa | 71.78 |
| +Pi+Pa+Ho | 74.08 |
Pretrain models for three tasks can be found here:
| Task | Dataset | Network | Method | Evaluation Metric | Link |
|---|---|---|---|---|---|
| Semantic Segmentation | Cityscapes | ResNet18 | Baseline | miou: 69.10 | - |
| Semantic Segmentation | Cityscapes | ResNet18 | + our distillation | miou: 75.3 | link |
| Object Detection | COCO | FCOS-MV2-C128 | Baseline | mAP: 30.9 | - |
| Object Detection | COCO | FCOS-MV2-C128 | + our distillation | mAP: 34.0 | link |
| Depth estimation | nyudv2 | VNL | baseline | rel: 13.5 | - |
| Depth estimation | nyudv2 | VNL | + our distillation | rel: 13.0 | link |
Note: Other chcekpoints can be obtained by email: yifan.liu04@adelaide.edu.au if needed.
python3.5
pytorch0.4.1
ninja
numpy
cv2
Pillow
We recommend to use Anaconda.
We have tested our code on Ubuntu 16.04.
Some parts of InPlace-ABN have a native CUDA implementation, which must be compiled with the following commands:
cd libs
sh build.sh
python build.pyThe build.sh script assumes that the nvcc compiler is available in the current system search path.
The CUDA kernels are compiled for sm_50, sm_52 and sm_61 by default.
To change this (e.g. if you are using a Kepler GPU), please edit the CUDA_GENCODE variable in build.sh.
- download the Cityscape dataset
- sh run_test.sh [you should change the data-dir to your own]. By using our distilled student model, which can be gotten in [ckpt], an mIoU of 73.05 is achieved on the Cityscape test set, and 75.3 on validation set.
| Model | Average | roda | sidewalk | building | wall | fence | pole | trafficlight | trafficsign | vegetation | terrain | sky | person | rider | car | truck | bus | train | motorcycle | bicycle |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| IoU | 73.05 | 97.57 | 78.80 | 91.42 | 50.76 | 50.88 | 60.77 | 67.93 | 73.18 | 92.49 | 70.36 | 94.56 | 82.81 | 61.64 | 94.89 | 60.14 | 66.62 | 59.93 | 61.50 | 71.71 |
Note: Depth estimation task and object detection task can be test through the original projects of VNL and FCOS using our checkpoints.
Download the pre-trained teacher weight:
If you want to reproduce the ablation study in our paper, please modify is_pi_use/is_pa_use/is_ho_use in the run_train_eval.sh. sh run_train_eval.sh
If you want to test your method on the cityscape test set, please modify the data-dir and resume-from path to your own, then run the test.sh and submit your results to www.cityscapes-dataset.net/submit/ sh test.sh
For academic use, this project is licensed under the 2-clause BSD License - see the LICENSE file for details. For commercial use, please contact Yifan Liu and Chunhua Shen.