segmentation and classification docs updated

doc update

diff --git a/README.md b/README.md
index 9050c67e29cba6aa437cfacc4ce1db5dee6c3d24..97e0c1faafce20f0e5a87105f318cdc39d5ff402 100644 (file)
--- a/README.md
+++ b/README.md
@@ -34,7 +34,7 @@ This code also includes tools for **Quantization Aware Training** that can outpu
 - [**Multi Task Estimation**](docs/Multi_Task_Learning.md)<br>
 - Object Detection - coming soon..<br>
 - Object Keypoint Estimation - coming soon..<br>
-- [**Quantization Aware Training**](docs/Quantization.md)<br>
+- [**Quantization**](docs/Quantization.md)<br>
 
 Above are some of the examples are currently available. Click on each of the links above to go into the full description of the example.
 

diff --git a/docs/Image_Classification.md b/docs/Image_Classification.md
index bac12dd5deb0824231bdd2c0120b0b57941e970f..c972a907cdbdab9aca019fbe7f13ceb5f0fe5414 100644 (file)
--- a/docs/Image_Classification.md
+++ b/docs/Image_Classification.md
@@ -1,57 +1,57 @@
 # Training for Image Classification
  
- Image Classification is a fundamental task in Deep Learning and Computer Vision. Here we show couple of examples of training CNN / Deep Learning models for Image Classification. For this example, we  use MobileNetV2 as the model for training, but other models can also be used.
+Image Classification is a fundamental task in Deep Learning and Computer Vision. Here we show couple of examples of training CNN / Deep Learning models for Image Classification. For this example, we  use MobileNetV2 as the model for training, but other models can also be used.
 
- Commonly used Traning/Validation commands are listed in the file [run_classification.sh](../run_classification.sh). Uncommend one and run the file to start the run. 
+Commonly used Traning/Validation commands are listed in the file [run_classification.sh](../run_classification.sh). Uncommend one and run the file to start the run. 
 
- ## Cifar Dataset 
- Cifar10 and Cifar100 are popular Datasets used for training CNNs. Since these datasets are small, the training can be finished in a a short time and can give an indication of how good a particular CNN model is. The images in these datasets are small (32x32).
+## Cifar Dataset 
+Cifar10 and Cifar100 are popular Datasets used for training CNNs. Since these datasets are small, the training can be finished in a a short time and can give an indication of how good a particular CNN model is. The images in these datasets are small (32x32).
 
 ### Cifar100 Dataset
- * Since the dataset is small, the training script itself can download the dataset before training.
+Since the dataset is small, the training script itself can download the dataset before training.
 
- * Training can be started by the following command:<br>
-    ```
-    python ./scripts/train_classification_main.py --dataset_name cifar100_classification --model_name mobilenetv2_tv_x1 --data_path ./data/datasets/cifar100_classification --img_resize 32 --img_crop 32 --rand_scale 0.5 1.0
-   ```
+Training can be started by the following command:<br>
+```
+python ./scripts/train_classification_main.py --dataset_name cifar100_classification --model_name mobilenetv2_tv_x1 --data_path ./data/datasets/cifar100_classification --img_resize 32 --img_crop 32 --rand_scale 0.5 1.0
+```
  
- * In the script, note that there are some special settings for the cifar datasets. The most important one is the 'strides' settings. Since the input images in Cifar are small we do not want to have as many strides as in a large sized image. See the argument args.model_config.strides being set in the script.
+In the script, note that there are some special settings for the cifar datasets. The most important one is the 'strides' settings. Since the input images in Cifar are small we do not want to have as many strides as in a large sized image. See the argument args.model_config.strides being set in the script.
 
- * During the training, **validation** accuracy will also be printed. But if you want to explicitly check the accuracy again with **validation** set, it can be done:<br>
-    ```
-    python ./scripts/train_classification_main.py --phase validation --dataset_name cifar100_classification --model_name mobilenetv2_tv_x1 --data_path ./data/datasets/cifar100_classification --img_resize 32 --img_crop 32
-   ```
+During the training, **validation** accuracy will also be printed. But if you want to explicitly check the accuracy again with **validation** set, it can be done:<br>
+```
+python ./scripts/train_classification_main.py --phase validation --dataset_name cifar100_classification --model_name mobilenetv2_tv_x1 --data_path ./data/datasets/cifar100_classification --img_resize 32 --img_crop 32
+```
 
- ### Cifar10 Dataset
-  * Training can be started by the following command:<br>
-    ```
-    python ./scripts/train_classification_main.py --dataset_name cifar10_classification --model_name mobilenetv2_tv_x1 --data_path ./data/datasets/cifar10_classification --img_resize 32 --img_crop 32 --rand_scale 0.5 1.0
-    ```
+### Cifar10 Dataset
+ Training can be started by the following command:<br>
+```
+python ./scripts/train_classification_main.py --dataset_name cifar10_classification --model_name mobilenetv2_tv_x1 --data_path ./data/datasets/cifar10_classification --img_resize 32 --img_crop 32 --rand_scale 0.5 1.0
+```
 
-  ## ImageNet Dataset
-  * It is difficult to reproduce the accuracies reported in the original papers for certain models (especially true for MobileNetV1 and MobileNetV2 models) due to the need for careful hyper-parameter tuning. In our examples, we have incorporated hyper-parameters required to train high accuracy classification models.
+## ImageNet Dataset
+It is difficult to reproduce the accuracies reported in the original papers for certain models (especially true for MobileNetV1 and MobileNetV2 models) due to the need for careful hyper-parameter tuning. In our examples, we have incorporated hyper-parameters required to train high accuracy classification models.
 
-  * Important note: ImageNet dataset is huge and download may take long time. Attempt this only if you have a good internet connection. Also the training takes a long time. In our case, using four GTX 1080 Ti, it takes nearly two days to train.
+Important note: ImageNet dataset is huge and download may take long time. Attempt this only if you have a good internet connection. Also the training takes a long time. In our case, using four GTX 1080 Ti, it takes nearly two days to train.
 
-  * Training can be started by the following command:<br>
-    ```
-    python ./scripts/train_classification_main.py --dataset_name imagenet_classification --model_name mobilenetv2_tv_x1 --data_path ./data/datasets/imagenet_classification
-    ```
+Training can be started by the following command:<br>
+```
+python ./scripts/train_classification_main.py --dataset_name imagenet_classification --model_name mobilenetv2_tv_x1 --data_path ./data/datasets/imagenet_classification
+```
 
-* Training with ResNet50:<br>
-    ```
-    python ./scripts/train_classification_main.py --dataset_name imagenet_classification --model_name resnet50_x1 --data_path ./data/datasets/imagenet_classification
-  ```
+Training with ResNet50:<br>
+```
+python ./scripts/train_classification_main.py --dataset_name imagenet_classification --model_name resnet50_x1 --data_path ./data/datasets/imagenet_classification
+```
   
-  * After the training, the **validation** accuracy using (make sure that  args.dataset_name and args.pretrained are correctly set)<br>
-    ```
-    python ./scripts/train_classification_main.py --phase validation --dataset_name imagenet_classification --model_name mobilenetv2_tv_x1 --data_path ./data/datasets/imagenet_classification --pretrained <checkpoint_path>
-    ```
+After the training, the **validation** accuracy using (make sure that  args.dataset_name and args.pretrained are correctly set)<br>
+```
+python ./scripts/train_classification_main.py --phase validation --dataset_name imagenet_classification --model_name mobilenetv2_tv_x1 --data_path ./data/datasets/imagenet_classification --pretrained <checkpoint_path>
+```
 
 ## ImageNet or any other classification dataset - manual download
-  * In this case, the images of the dataset is assumed to be arranges in folders. 'train'  and 'validation' are two separate folders and underneath that, each class should have a different folder.
+In this case, the images of the dataset is assumed to be arranges in folders. 'train'  and 'validation' are two separate folders and underneath that, each class should have a different folder.
 
-  * Assume that that folder './data/datasets/image_folder_classification' has the  the classification dataset. This folder should contain folders and images as follows: 
+Assume that that folder './data/datasets/image_folder_classification' has the  the classification dataset. This folder should contain folders and images as follows: 
   image_folder_classification<br>
   &nbsp;&nbsp;train<br>
   &nbsp;&nbsp;&nbsp;&nbsp;class1<br>
@@ -68,50 +68,51 @@
   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;image files here<br>
   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;....<br>
 
-* Note 'class1', 'class2' etc are examples and they stand for the names of the classes that we are trying to classify.
-
-* Here we use ImageNEt dataset as an example, but it could ne any other image classification dataset arranged in folders. 
-
-* The download links for ImageNet are given in ../modules/pytorch_jacinto_ai/vision/datasets/imagenet.py. 
-    ```
-    cd ./data/datasets/image_folder_classification
-    wget http://www.image-net.org/challenges/LSVRC/2012/nnoupb/ILSVRC2012_img_train.tar
-    wget http://www.image-net.org/challenges/LSVRC/2012/nnoupb/ILSVRC2012_img_val.tar
-    mkdir train
-    mkdir validation
-    tar -C train -xvf ILSVRC2012_img_train.tar
-    tar -C validation -xvf ILSVRC2012_img_val.tar
-    ```
-* After downloading and extracting, use this script to arrange the validation folder into folders of classes: 
-    ```
-    cd validation
-    wget https://github.com/soumith/imagenetloader.torch/blob/master/valprep.sh
-    ./valprep.sh
-    rm ./valprep.sh
-    ```
-
-* Training with **MobileNetV2** model can be started by the following command from the base folder of the repository:<br>
-    ```
-    python ./scripts/train_classification_main.py --dataset_name image_folder_classification --model_name mobilenetv2_tv_x1 --data_path ./data/datasets/image_folder_classification
-    ```
-
-* Training with **ResNet50** model:<br>
-    ```
-    python ./scripts/train_classification_main.py --dataset_name image_folder_classification --model_name resnet50_x1 --data_path ./data/datasets/image_folder_classification
-    ```
-
-* Training with **RegNet800MF model and BGR image input transform**:<br>
-    ```python ./scripts/train_classification_main.py --dataset_name image_folder_classification --model_name regnetx800mf_x1 --data_path ./data/datasets/image_folder_classification --input_channel_reverse True --image_mean 103.53 116.28 123.675 --image_scale 0.017429 0.017507 0.017125
-    ```
-
-* If the dataset is in a different location, it can be specified by the --data_path option, but dataset_name must be *image_folder_classification* for folder based classification.
+Note 'class1', 'class2' etc are examples and they stand for the names of the classes that we are trying to classify.
+
+Here we use ImageNEt dataset as an example, but it could ne any other image classification dataset arranged in folders. 
+
+The download links for ImageNet are given in ../modules/pytorch_jacinto_ai/vision/datasets/imagenet.py. 
+```
+cd ./data/datasets/image_folder_classification
+wget http://www.image-net.org/challenges/LSVRC/2012/nnoupb/ILSVRC2012_img_train.tar
+wget http://www.image-net.org/challenges/LSVRC/2012/nnoupb/ILSVRC2012_img_val.tar
+mkdir train
+mkdir validation
+tar -C train -xvf ILSVRC2012_img_train.tar
+tar -C validation -xvf ILSVRC2012_img_val.tar
+```
+
+After downloading and extracting, use this script to arrange the validation folder into folders of classes: 
+```
+cd validation
+wget https://github.com/soumith/imagenetloader.torch/blob/master/valprep.sh
+./valprep.sh
+rm ./valprep.sh
+```
+
+Training with **MobileNetV2** model can be started by the following command from the base folder of the repository:<br>
+```
+python ./scripts/train_classification_main.py --dataset_name image_folder_classification --model_name mobilenetv2_tv_x1 --data_path ./data/datasets/image_folder_classification
+```
+
+Training with **ResNet50** model:<br>
+```
+python ./scripts/train_classification_main.py --dataset_name image_folder_classification --model_name resnet50_x1 --data_path ./data/datasets/image_folder_classification
+```
+
+Training with **RegNet800MF model and BGR image input transform**:<br>
+```python ./scripts/train_classification_main.py --dataset_name image_folder_classification --model_name regnetx800mf_x1 --data_path ./data/datasets/image_folder_classification --input_channel_reverse True --image_mean 103.53 116.28 123.675 --image_scale 0.017429 0.017507 0.017125
+```
+
+If the dataset is in a different location, it can be specified by the --data_path option, but dataset_name must be *image_folder_classification* for folder based classification.
 
 
 ## Results
 
 ### ImageNet (ILSVRC2012) Classification (1000 class)
 
-* ImageNet classification results are as follows:
+ImageNet classification results are as follows:
 
 |Dataset  |Mode Name          |Resize Resolution|Crop Resolution|Complexity (GigaMACS)|Top1 Accuracy% |Model Configuration Name|
 |---------|----------         |-----------      |----------     |--------             |--------       |------------------------|
@@ -123,11 +124,12 @@
 |ImageNet |MobileNetV1[1]     |256x256          |224x224        |0.569                |70.60          |                        |
 |ImageNet |MobileNetV2[2]     |256x256          |224x224        |0.300                |72.00          |                        |
 |ImageNet |ResNet50[3]        |256x256          |224x224        |4.087                |76.15          |                        |
+|ImageNet |**RegNet4800MF**[4]|256x256          |224x224        |0.400                |**72.6**       |                        |
 |ImageNet |**RegNetX800MF**[4]|256x256          |224x224        |0.800                |**75.2**       |                        |
 |ImageNet |RegNetX1.6F[4]     |256x256          |224x224        |1.6                  |**77.0**       |                        |
 
 
-Notes:
+#### Notes
 - As can be seen from the table, the models included in this repository provide a good Accuracy/Complexity tradeoff. 
 - However, the Complexity (in GigaMACS) does not always indicate the speed of inference on an embedded device. We have to also consider the fact that regular convolutions and Grouped convolutions are typically more efficient in utilizing the available compute resources (as they have more compute per data trasnsfer) compared to Depthwise convolutions.
 - Hence, although the MobileNetV2 based models may have less GigaMACS compared to the RegNetX models, the RegNetX based models may not be slower in practice. Overall, RegNetX based models are highly recommend as they strike a good balance between Complexity (GigaMACS), Compute Efficiency (more compute per data transfer) and easiness of Quantization.

diff --git a/docs/Semantic_Segmentation.md b/docs/Semantic_Segmentation.md
index c18b462e8191883cc2f496b7cb68c11a1f975b58..7203feba3f7d45cd41dca694d74f057689bc536b 100644 (file)
--- a/docs/Semantic_Segmentation.md
+++ b/docs/Semantic_Segmentation.md
@@ -1,5 +1,4 @@
 # Semantic Segmentation
-
 Semantic segmentation assigns a class to each pixel of the image. It is useful for tasks such as lane detection, road segmentation etc. 
 
 Commonly used Training/Validation commands are listed in the file [run_segmentation.sh](../run_segmentation.sh). Uncommend one line and run the file to start the run. 
@@ -7,142 +6,168 @@ Commonly used Training/Validation commands are listed in the file [run_segmentat
 ## Model Configurations
 A set of common example model configurations are defined for all pixel to pixel tasks. These models can support multiple inputs (for example image and optical flow) as well as support multiple decoders for multi-task prediction (for example semantic segmentation + depth estimation + motion segmentation). 
 
-Whether to use multiple inputs or how many decoders to use are fully configurable. This framework is also flexible to add different model architectures or backbone networks. Some of the model configurations are currently available are:<br>
-* **deeplabv3lite_mobilenetv2_tv**: (default) This model is mostly similar to the DeepLabV3+ model [[6]] using MobileNetV2 backbone. The difference with DeepLabV3+ is that we removed the convolutions after the shortcut and kep one set of depthwise separable convolutions to generate the prediction. The ASPP module that we used is a lite-weight variant with depthwise separable convolutions (DWASPP). We found that this reduces complexity without sacrificing accuracy. Due to this we call this model DeepLabV3+(Lite) or simply  DeepLabV3Lite. (Note: The suffix "_tv" is used to indicate that our backbone model is from torchvision)<br> 
-* **fpnlite_pixel2pixel_aspp_mobilenetv2_tv**: This is similar to Feature Pyramid Network [[3]], but adapted for pixel2pixel tasks. We stop the decoder at a stride of 4 and then upsample to the final resolution from there. We also use DWASPP module to improve the receptive field. We call this model FPNPixel2Pixel.
-* **fpnlite_pixel2pixel_aspp_mobilenetv2_tv_fd**: This is also FPN, but with a larger encoder stride(64). This is a low complexity model (using Fast Downsampling Strategy [8]) that can be used with higher resolutions.
-* **fpnlite_pixel2pixel_aspp_resnet50**: Feature Pyramid Network (FPN) based pixel2pixel using ResNet50 backbone with DWASPP.
+Whether to use multiple inputs or how many decoders to use are fully configurable. This framework is also flexible to add different model architectures or backbone networks. Some of the model configurations are currently available are:
 
-## Datasets: Cityscapes Dataset 
+These are some of the supported models that use MobileNetV2 backbone.
+
+**deeplabv3lite_mobilenetv2_tv**: (default) This model is mostly similar to the DeepLabV3+ model [[7]] using MobileNetV2 backbone. The difference with DeepLabV3+ is that we removed the convolutions after the shortcut and kep one set of depthwise separable convolutions to generate the prediction. The ASPP module that we used is a lite-weight variant with depthwise separable convolutions (DWASPP). We found that this reduces complexity without sacrificing accuracy. Due to this we call this model DeepLabV3+(Lite) or simply  DeepLabV3Lite. (Note: The suffix "_tv" is used to indicate that our backbone model is from torchvision)
+
+**unetlite_pixel2pixel_aspp_mobilenetv2_tv**: UNet [6] based pixel2pixel model.
+
+**fpnlite_pixel2pixel_aspp_mobilenetv2_tv**: This is similar to Feature Pyramid Network [[4]], but adapted for pixel2pixel tasks. We stop the decoder at a stride of 4 and then upsample to the final resolution from there. We also use DWASPP module to improve the receptive field. We call this model FPNPixel2Pixel.
+
+**fpnlite_pixel2pixel_aspp_mobilenetv2_tv_fd**: This is also FPN, but with a larger encoder stride(64). This is a low complexity model (using Fast Downsampling Strategy [12]) that can be used with higher resolutions.
+
+
+**We find that RegNetX models strike a good balance between accuracy, model complexity, speed of inference on device and easiness of quantization.**  For RegNetX based pixel2pixel models, the same group size used in the encoder is used in the decoder part as well. Following are some of the RegNetX based models that are supported.
+
+**deeplabv3lite_regnetx800mf**: RegNetX-800MF based DeepLabV3Plus model.
+
+**unetlite_pixel2pixel_aspp_regnetx800mf**: RegNetX-800MF based UNet model.
+
+**fpnlite_pixel2pixel_aspp_regnetx400mf**: RegNetX-400MF based FPN model.
+
+**fpnlite_pixel2pixel_aspp_regnetx800mf** RegNetX-800MF based FPN model.
+
+**fpnlite_pixel2pixel_aspp_regnetx1p6gf** RegNetX-1.6GF based FPN model.
+
+**fpnlite_pixel2pixel_aspp_regnetx3p2gf** RegNetX-3.2GF based FPN model.
 
-* Download the Cityscapes dataset [[1]] from https://www.cityscapes-dataset.com/. You will need need to register before the data can be downloaded. Unzip the data into the folder ./data/datasets/cityscapes/data. This folder should contain leftImg8bit and gtFine folders of cityscapes. 
+
+## Datasets: Cityscapes Dataset 
+Download the Cityscapes dataset [[2]] from https://www.cityscapes-dataset.com/. You will need need to register before the data can be downloaded. Unzip the data into the folder ./data/datasets/cityscapes/data. This folder should contain leftImg8bit and gtFine folders of cityscapes. 
 
 ## Datasets: VOC Dataset 
-* The PASCAL VOC dataset [[2]] can be downloaded using the following:<br>
-    ```
-    mkdir ./data/datasets/voc
-    cd /data/datasets/voc
-    wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
-    wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar
-    wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar
-    ```
-* Extact the dataset files into ./data/datasets/voc/VOCdevkit
-    ```
-    tar -xvf VOCtrainval_11-May-2012.tar
-    tar -xvf VOCtrainval_06-Nov-2007.tar
-    tar -xvf VOCtest_06-Nov-2007.tar
-    ```
-* Download Extra annotations: Download the augumented annotations as explained here: https://github.com/DrSleep/tensorflow-deeplab-resnet. For this, using a browser, download the zip file SegmentationClassAug.zip from: https://www.dropbox.com/s/oeu149j8qtbs1x0/SegmentationClassAug.zip?dl=0
-* Unzip SegmentationClassAug.zip and place the images in the folder ./data/datasets/voc/VOCdevkit/VOC2012/SegmentationClassAug
-* Create a list of those images in the ImageSets folder using the following:<br>
-    ```
-    cd VOCdevkit/VOC2012/ImageSets/Segmentation
-    ls -1 SegmentationClassAug | sed s/.png// > trainaug.txt
-    wget http://home.bharathh.info/pubs/codes/SBD/train_noval.txt  
-    mv train_noval.txt trainaug_noval.txt 
-    ```
+The PASCAL VOC dataset [[1]] can be downloaded using the following:<br>
+```
+mkdir ./data/datasets/voc
+cd /data/datasets/voc
+wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
+wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar
+wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar
+```
+
+Extact the dataset files into ./data/datasets/voc/VOCdevkit
+```
+tar -xvf VOCtrainval_11-May-2012.tar
+tar -xvf VOCtrainval_06-Nov-2007.tar
+tar -xvf VOCtest_06-Nov-2007.tar
+```
+
+Download Extra annotations: Download the augumented annotations as explained here: https://github.com/DrSleep/tensorflow-deeplab-resnet. For this, using a browser, download the zip file SegmentationClassAug.zip from: https://www.dropbox.com/s/oeu149j8qtbs1x0/SegmentationClassAug.zip?dl=0
+
+Unzip SegmentationClassAug.zip and place the images in the folder ./data/datasets/voc/VOCdevkit/VOC2012/SegmentationClassAug
+
+Create a list of those images in the ImageSets folder using the following:<br>
+```
+cd VOCdevkit/VOC2012/ImageSets/Segmentation
+ls -1 SegmentationClassAug | sed s/.png// > trainaug.txt
+wget http://home.bharathh.info/pubs/codes/SBD/train_noval.txt  
+mv train_noval.txt trainaug_noval.txt 
+```
 
 ## Training
-* These examples use two gpus because we use slightly higher accuracy when we restricted the number of GPUs used. 
-
-* **Cityscapes Segmentation Training** with MobileNetV2 backbone and DeeplabV3Lite decoder can be done as follows:<br>
-    ```
-    python ./scripts/train_segmentation_main.py --model_name deeplabv3lite_mobilenetv2_tv --dataset_name cityscapes_segmentation --data_path ./data/datasets/cityscapes/data --img_resize 384 768 --output_size 1024 2048 --pretrained https://download.pytorch.org/models/mobilenet_v2-b0353104.pth --gpus 0 1
-  ```
-
-* Cityscapes Segmentation Training with **RegNet800MF backbone and FPN decoder** can be done as follows:<br>
-    ```
-    python ./scripts/train_segmentation_main.py --dataset_name cityscapes_segmentation --model_name fpnlite_pixel2pixel_aspp_regnetx800mf --data_path ./data/datasets/cityscapes/data --img_resize 384 768 --output_size 1024 2048 --gpus 0 1 --pretrained https://dl.fbaipublicfiles.com/pycls/dds_baselines/160906036/RegNetX-800MF_dds_8gpu.pyth
-  ```
-
-  * It is possible to use a **different image size**. For example, we trained for 1536x768 resolution by the following. (We used a smaller crop size compared to the image resize resolution to reduce GPU memory usage). <br>
-    ```
-    python ./scripts/train_segmentation_main.py --model_name deeplabv3lite_mobilenetv2_tv --dataset_name cityscapes_segmentation --data_path ./data/datasets/cityscapes/data --img_resize 768 1536 --rand_crop 512 1024 --output_size 1024 2048 --pretrained https://download.pytorch.org/models/mobilenet_v2-b0353104.pth --gpus 0 1
-    ```
-
-* Train **FPNPixel2Pixel model at 1536x768 resolution** (use 1024x512 crop to reduce memory usage):<br>
-    ```
-    python ./scripts/train_segmentation_main.py --model_name fpnlite_pixel2pixel_aspp_mobilenetv2_tv --dataset_name cityscapes_segmentation --data_path ./data/datasets/cityscapes/data --img_resize 768 1536 --rand_crop 512 1024 --output_size 1024 2048 --pretrained https://download.pytorch.org/models/mobilenet_v2-b0353104.pth --gpus 0 1
-    ```
+
+These examples use two gpus because we use slightly higher accuracy when we restricted the number of GPUs used. 
+
+**Cityscapes Segmentation Training** with MobileNetV2 backbone and DeeplabV3Lite decoder can be done as follows:<br>
+```
+python ./scripts/train_segmentation_main.py --model_name deeplabv3lite_mobilenetv2_tv --dataset_name cityscapes_segmentation --data_path ./data/datasets/cityscapes/data --img_resize 384 768 --output_size 1024 2048 --pretrained https://download.pytorch.org/models/mobilenet_v2-b0353104.pth --gpus 0 1
+```
+
+Cityscapes Segmentation Training with **RegNet800MF backbone and FPN decoder** can be done as follows:<br>
+```
+python ./scripts/train_segmentation_main.py --dataset_name cityscapes_segmentation --model_name fpnlite_pixel2pixel_aspp_regnetx800mf --data_path ./data/datasets/cityscapes/data --img_resize 384 768 --output_size 1024 2048 --gpus 0 1 --pretrained https://dl.fbaipublicfiles.com/pycls/dds_baselines/160906036/RegNetX-800MF_dds_8gpu.pyth
+```
+
+It is possible to use a **different image size**. For example, we trained for 1536x768 resolution by the following. (We used a smaller crop size compared to the image resize resolution to reduce GPU memory usage). <br>
+```
+python ./scripts/train_segmentation_main.py --model_name deeplabv3lite_mobilenetv2_tv --dataset_name cityscapes_segmentation --data_path ./data/datasets/cityscapes/data --img_resize 768 1536 --rand_crop 512 1024 --output_size 1024 2048 --pretrained https://download.pytorch.org/models/mobilenet_v2-b0353104.pth --gpus 0 1
+```
+
+Train **FPNPixel2Pixel model at 1536x768 resolution** (use 1024x512 crop to reduce memory usage):<br>
+```
+python ./scripts/train_segmentation_main.py --model_name fpnlite_pixel2pixel_aspp_mobilenetv2_tv --dataset_name cityscapes_segmentation --data_path ./data/datasets/cityscapes/data --img_resize 768 1536 --rand_crop 512 1024 --output_size 1024 2048 --pretrained https://download.pytorch.org/models/mobilenet_v2-b0353104.pth --gpus 0 1
+```
  
-* **VOC Segmentation Training** can be done as follows:<br>
-    ```
-    python ./scripts/train_segmentation_main.py --model_name deeplabv3lite_mobilenetv2_tv --dataset_name voc_segmentation --data_path ./data/datasets/voc --img_resize 512 512 --output_size 512 512 --pretrained https://download.pytorch.org/models/mobilenet_v2-b0353104.pth --gpus 0 1
-  ```
+**VOC Segmentation Training** can be done as follows:<br>
+```
+python ./scripts/train_segmentation_main.py --model_name deeplabv3lite_mobilenetv2_tv --dataset_name voc_segmentation --data_path ./data/datasets/voc --img_resize 512 512 --output_size 512 512 --pretrained https://download.pytorch.org/models/mobilenet_v2-b0353104.pth --gpus 0 1
+```
 
 ## Validation
- * During the training, **validation** accuracy will also be printed. But to explicitly check the accuracy again with **validation** set, it can be done as follows (fill in the path to the pretrained model):<br>
-    ```
-    python ./scripts/train_segmentation_main.py --phase validation --model_name deeplabv3lite_mobilenetv2_tv --dataset_name cityscapes_segmentation --data_path ./data/datasets/cityscapes/data --img_resize 384 768 --output_size 1024 2048 --gpus 0 1 --pretrained ?????
-    ```
+During the training, **validation** accuracy will also be printed. But to explicitly check the accuracy again with **validation** set, it can be done as follows (fill in the path to the pretrained model):<br>
+```
+python ./scripts/train_segmentation_main.py --phase validation --model_name deeplabv3lite_mobilenetv2_tv --dataset_name cityscapes_segmentation --data_path ./data/datasets/cityscapes/data --img_resize 384 768 --output_size 1024 2048 --gpus 0 1 --pretrained ?????
+```
+
 
 ## Inference
 Inference can be done as follows (fill in the path to the pretrained model):<br>
-    ```
-    python ./scripts/infer_segmentation_main.py --phase validation --model_name deeplabv3lite_mobilenetv2_tv --dataset_name cityscapes_segmentation_measure --data_path ./data/datasets/cityscapes/data --img_resize 384 768 --output_size 1024 2048 --gpus 0 1 --pretrained ?????
-    ```
-          
+```
+python ./scripts/infer_segmentation_main.py --phase validation --model_name deeplabv3lite_mobilenetv2_tv --dataset_name cityscapes_segmentation_measure --data_path ./data/datasets/cityscapes/data --img_resize 384 768 --output_size 1024 2048 --gpus 0 1 --pretrained ?????
+```
+
+ 
 ## Results
 
-### Cityscapes Segmentation
-
-|Dataset    |Mode Architecture             |Backbone Model |Backbone Stride|Resolution |Complexity (GigaMACS)|MeanIoU%  |Model Configuration Name                      |
-|---------  |----------                    |-----------    |-------------- |-----------|--------             |----------|----------------------------------------      |
-|Cityscapes |FPNLitePixel2Pixel with DWASPP|FD-MobileNetV2 |64             |768x384    |0.99                 |62.43     |fpnlite_pixel2pixel_aspp_mobilenetv2_tv_fd    |
-|Cityscapes |UNetLite with DWASPP          |MobileNetV2    |32             |768x384    |**2.20**             |**68.94** |**unetlite_pixel2pixel_aspp_mobilenetv2_tv**  |
-|Cityscapes |DeepLabV3Lite with DWASPP     |MobileNetV2    |16             |768x384    |**3.54**             |**69.13** |**deeplabv3lite_mobilenetv2_tv**              |
-|Cityscapes |FPNLitePixel2Pixel            |MobileNetV2    |32(\*2\*2)     |768x384    |3.66                 |70.30     |fpnlite_pixel2pixel_mobilenetv2_tv            |
-|Cityscapes |FPNLitePixel2Pixel with DWASPP|MobileNetV2    |32             |768x384    |3.84                 |70.39     |fpnlite_pixel2pixel_aspp_mobilenetv2_tv       |
-|Cityscapes |FPNLitePixel2Pixel            |FD-MobileNetV2 |64(\*2\*2)     |1536x768   |3.85                 |69.82     |fpnlite_pixel2pixel_mobilenetv2_tv_fd         |
-|Cityscapes |FPNLitePixel2Pixel with DWASPP|FD-MobileNetV2 |64             |1536x768   |**3.96**             |**71.28** |**fpnlite_pixel2pixel_aspp_mobilenetv2_tv_fd**|
-|Cityscapes |FPNLitePixel2Pixel with DWASPP|FD-MobileNetV2 |64             |2048x1024  |7.03                 |72.67     |fpnlite_pixel2pixel_aspp_mobilenetv2_tv_fd    |
-|Cityscapes |DeepLabV3Lite with DWASPP     |MobileNetV2    |16             |1536x768   |14.48                |73.59     |deeplabv3lite_mobilenetv2_tv                  |
-|Cityscapes |FPNLitePixel2Pixel with DWASPP|MobileNetV2    |32             |1536x768   |**15.37**            |**74.98** |**fpnlite_pixel2pixel_aspp_mobilenetv2_tv**   |
-|Cityscapes |FPNLitePixel2Pixel with DWASPP|FD-ResNet50    |64             |1536x768   |30.91                |-         |fpnlite_pixel2pixel_aspp_resnet50_fd          |
-|Cityscapes |FPNLitePixel2Pixel with DWASPP|ResNet50       |32             |1536x768   |114.42               |-         |fpnlite_pixel2pixel_aspp_resnet50             |
-|.
-|Cityscapes |DeepLabV3Lite GroupedConvASPP |RegNet800MF [9]|32             |768x384    |**11.19**            |**68.44** |**deeplav3lite_pixel2pixel_aspp_regnetx800mf**|
-|Cityscapes |FPNLite GroupedConvASPP       |RegNet800MF [9]|32             |768x384    |**7.29**              |**70.22** |**fpnlite_pixel2pixel_aspp_regnetx800mf**     |
-|Cityscapes |UNetLite GroupedConvASPP      |RegNet800MF [9]|32             |768x384    |**6.09**             |**69.93** |**unetlite_pixel2pixel_aspp_regnetx800mf**    |
-
-
-For comparison, here we list a few models from the literature:
-
-|Dataset    |Mode Architecture             |Backbone Model |Backbone Stride|Resolution |Complexity (GigaMACS)|MeanIoU%  |Model Configuration Name                     |
-|---------  |----------                    |-----------    |-------------- |-----------|--------             |----------|------------------------|
-|Cityscapes |ERFNet [4]                    |-              |-              |1024x512   |27.705               |69.7      |-                       |
-|Cityscapes |SwiftNetMNV2 [5]              |MobileNetV2    |-              |2048x1024  |41.0                 |75.3      |-                       |
-|Cityscapes |DeepLabV3Plus [6,7]           |MobileNetV2    |16             |           |21.27                |70.71     |-                       |
-|Cityscapes |DeepLabV3Plus [6,7]           |Xception65     |16             |           |418.64               |78.79     |-                       |
-
-Notes:
-- The suffix **'Lite'** in the model names indicates complexity optimizations in the Decoder part of the model - especially the use of DepthWise Separable Convolutions instead of regular convolutions.
-- (\*2\*2) in the above table represents two additional Depthwise Separable Convolutions with strides (at the end of the backbone encoder). 
-- FD-MobileNetV2 Backbone uses a stride of 64 (this is used in some rows of the above table) and is achieved by Fast Downsampling Strategy [8]
-- As can be seen from the table, the models included in this repository provide a good Accuracy/Complexity tradeoff. 
-- However, the Complexity (in GigaMACS) does not always indicate the speed of inference on an embedded device. We have to also consider the fact that regular convolutions and Grouped convolutions are typically more efficient in utilizing the available compute resources (as they have more compute per data trasnsfer) compared to Depthwise convolutions.
-- Hence, although the MobileNetV2 based models may have less GigaMACS compared to the RegNetX models, the RegNetX based models may not be slower in practice. Overall, RegNetX based models are highly recommend as they strike a good balance between Complexity (GigaMACS), Compute Efficiency (more compute per data transfer) and easiness of Quantization.
+Dataset    |Mode Name                     |Input Size |GigaMACs  |Accuracy%      |Model Name       |
+|----------|------------------------------|-----------|----------|---------------|-----------------|
+|Cityscapes|RegNetX800MF+FPNLite          |768x384    |**8.84**  |**72.01**      |fpnlite_pixel2pixel_aspp_regnetx800mf  |
+|Cityscapes|RegNetX1.6GF+FPNLite          |1024x512   |**24.29** |**75.84**      |fpnlite_pixel2pixel_aspp_regnetx1p6gf  |
+|Cityscapes|RegNetX3.2GF+FPNLite          |1024x512   |**49.40** |**77.24**      |fpnlite_pixel2pixel_aspp_regnetx3p2gf  |
+|Cityscapes|RegNetX3.2FF+FPNLite          |1536x768   |**111.16**|**78.90**      |fpnlite_pixel2pixel_aspp_regnetx3p2gf  |
+|-
+|Cityscapes|RegNetX400MF+FPNLite          |768x384    |**6.09**  |**68.03**      |fpnlite_pixel2pixel_aspp_regnetx400mf  |
+|Cityscapes|RegNetX400MF+FPNLite          |1536x768   |**24.37** |**73.96**      |fpnlite_pixel2pixel_aspp_regnetx400mf  |
+|-
+|Cityscapes|MobileNetV2S16+DeepLabV3Lite  |768x384    |**3.54**  |**69.13**      |deeplabv3lite_mobilenetv2_tv           |
+|Cityscapes|MobileNetV2+UNetLite          |768x384    |**2.20**  |**68.94**      |unetlite_pixel2pixel_aspp_mobilenetv2_tv |
+|Cityscapes|MobileNetV2+FPNLite           |768x384    |**3.84**  |**70.39**      |fpnlite_pixel2pixel_aspp_mobilenetv2_tv |
+|Cityscapes|MobileNetV2+FPNLite           |1536x768   |**15.07** |**74.61**      |fpnlite_pixel2pixel_aspp_mobilenetv2_tv |
+|-
+|Cityscapes|FD-MobileNetV2+FPNLite        |1536x768   |**3.96**  |**71.28**      |fpnlite_pixel2pixel_aspp_mobilenetv2_tv_fd |
+|Cityscapes|FD-MobileNetV2+FPNLite        |2048x1024  |**7.03**  |**72.67**      |fpnlite_pixel2pixel_aspp_mobilenetv2_tv_fd |
+|-
+|Cityscapes|MobileNetV2+DeepLab[10]       |769x769    |21.27     |70.71          |                 |
+|Cityscapes|MobileNetV3+DeepLab[10]       |769x769    |15.95     |72.41          |                 |
+|Cityscapes|Xception65+DeepLab[10]        |769x769    |418.64    |78.79          |                 |
+|Cityscapes|ERFNet[8]                     |1024x512   |27.705    |69.7           |                 |
+|Cityscapes|MobileNetV2+SwiftNet[9]       |2048x1024  |41.0      |75.3           |                 |
+|Cityscapes|ResNet50+FCN[3][11]           |1040x520   |285.4     |71.6           |                 |
+|Cityscapes|ResNet50+DeepLabV3[5][11]     |1040x520   |337.5     |73.5           |                 |
+
+
+#### Notes
+- The suffix 'Lite' in the name of models such as DeepLabV3Lite, FPNLite & UNetLite indicates the use of Depthwise convolutions or Grouped convolutions. If the feature extractor (encoder) uses Depthwise Convolutions, then Depthwise convolutions are used throughout such models, even in the neck and decoder. If the feature extractor (encoder) uses grouped convolutions as in the case of RegNetX, then grouped convolutions (with the same group size as that of the feature extractor) are used even in the neck and decoder.<br>
+- GigaMACS: Complexity in Giga Multiply-Accumulations (lower is better). This is an important metric to watch out for when selecting models for embedded inference.<br>
+- Accuracy%: Original Floating Point Validation Accuracy% obtained after training.<br>
+- Overall, RegNetX based models are highly recommend as they strike a good balance between Complexity (GigaMACS), speed of inference on device and easiness of Quantization and we recommend them.<br>
 
 
 ## References
-[1]The Cityscapes Dataset for Semantic Urban Scene Understanding, Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, Bernt Schiele, CVPR 2016, https://www.cityscapes-dataset.com/
-
-[2] The PASCAL Visual Object Classes (VOC) Challenge
-Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J. and Zisserman, A.
+[1] The PASCAL Visual Object Classes (VOC) Challenge, Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J. and Zisserman, A.
 International Journal of Computer Vision, 88(2), 303-338, 2010, http://host.robots.ox.ac.uk/pascal/VOC/
 
-[3] Feature Pyramid Networks for Object Detection, Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie, CVPR 2017
+[2] The Cityscapes Dataset for Semantic Urban Scene Understanding, Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, Bernt Schiele, https://arxiv.org/abs/1604.01685
+
+[3] Fully Convolutional Networks for Semantic Segmentation, Jonathan Long, Evan Shelhamer, Trevor Darrell, https://arxiv.org/abs/1411.4038
+
+[4] Feature Pyramid Networks for Object Detection Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie, https://arxiv.org/abs/1612.03144
 
-[4] ERFNet: Efficient Residual Factorized ConvNet for Real-time Semantic Segmentation, E. Romera, J. M. Alvarez, L. M. Bergasa and R. Arroyo, Transactions on Intelligent Transportation Systems (T-ITS), 2017
+[5] Rethinking Atrous Convolution for Semantic Image Segmentation, Liang-Chieh Chen, George Papandreou, Florian Schroff, Hartwig Adam, https://arxiv.org/abs/1706.05587
 
-[5] In Defense of Pre-trained ImageNet Architectures for Real-time Semantic Segmentation of Road-driving Images, Marin Orsic, Ivan Kreso, Petra Bevandic, Sinisa Segvic, CVPR 2019.
+[6] U-Net: Convolutional Networks for Biomedical Image Segmentation, Olaf Ronneberger, Philipp Fischer, Thomas Brox, https://arxiv.org/abs/1505.04597
 
-[6] Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation, Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam, CVPR 2018
+[7] Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation, Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, Hartwig Adam, https://arxiv.org/abs/1802.02611
 
-[7] Tensorflow/Deeplab Model Zoo, https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md
+[8] E. Romera, J. M. Alvarez, L. M. Bergasa, and R. Arroyo. Erfnet: Efficient residual factorized convnet for real-time semantic segmentation. IEEE Transactions
+on Intelligent Transportation Systems, 19(1):263272, 2018.
 
-[8] FD-MobileNet: Improved MobileNet with a Fast Downsampling Strategy, Zheng Qin, Zhaoning Zhang, Xiaotao Chen, Yuxing Peng - https://arxiv.org/abs/1802.03750)
+[9] In Defense of Pre-trained ImageNet Architectures for Real-time Semantic Segmentation of Road-driving Images, Marin Oršić*, Ivan Krešo*, Siniša Šegvić, Petra Bevandić (* denotes equal contribution), CVPR, 2019.
 
-[9] Designing Network Design Spaces, Ilija Radosavovic Raj Prateek Kosaraju Ross Girshick Kaiming He Piotr Dollar´, Facebook AI Research (FAIR), https://arxiv.org/pdf/2003.13678.pdf, https://github.com/facebookresearch/pycls
+[10] Tensorflow DeepLab ModelZoo, https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md
 
+[11] Torchvision & Torchvision Models, https://github.com/pytorch/vision/tree/master/torchvision, https://pytorch.org/docs/stable/torchvision/models.html
 
+[12] FD-MobileNet: Improved MobileNet with a Fast Downsampling Strategy, Zheng Qin, Zhaoning Zhang, Xiaotao Chen, Yuxing Peng - https://arxiv.org/abs/1802.03750)