release commit

diff --git a/README.md b/README.md
index 510711c0e358218f67304c720b0c3d6675eaee7e..b4b321e70a1cf507a7a9b6adad7e2341c646f5e9 100644 (file)
--- a/README.md
+++ b/README.md
@@ -1,16 +1,15 @@
 # Jacinto-AI-DevKit (PyTorch)
 
 # Jacinto-AI-DevKit (PyTorch)
 

-### Deep Learning Models / Training / Calibration & Quantization - Using PyTorch<br>
-Internal URL: https://bitbucket.itg.ti.com/projects/jacinto-ai-devkit/repos/pytorch-jacinto-ai-devkit<br>
-External URL: https://git.ti.com/jacinto-ai-devkit/pytorch-jacinto-ai-devkit<br>
+Note: If you have not visited our landing page in github, please do so: [https://github.com/TexasInstruments/jacinto-ai-devkit](https://github.com/TexasInstruments/jacinto-ai-devkit)

 
 

-We provide a set of low complexity deep learning examples and models for low power embedded systems. Low power embedded systems often requires balancing of complexity and accuracy. This is a tough task and requires significant amount of expertise and experimentation. We call this process **complexity optimization**. In addition we would like to bridge the gap between Deep Learning training frameworks and real-time embedded inference by providing ready to use examples and enable **ease of use**.
+### Deep Learning Models / Training / Calibration & Quantization - Using PyTorch<br>
+This code provides a set of low complexity deep learning examples and models for low power embedded systems. Low power embedded systems often requires balancing of complexity and accuracy. This is a tough task and requires significant amount of expertise and experimentation. We call this process **complexity optimization**. In addition we would like to bridge the gap between Deep Learning training frameworks and real-time embedded inference by providing ready to use examples and enable **ease of use**. Scripts for training, validation, complexity analysis are also provided. 

 
 

-We have added several complexity optimized Deep Learning examples for commonly used vision tasks. We provide training scripts, accuracy, complexity and in some cases, the trained models as well. Our expectation is that these Deep Learning examples and models will find application in a variety of problems, and you will be able to build upon the **building blocks** that we have provided. 
+This code also includes tools for **Post Training Calibration and Trained Quantization (a.k.a Quantization Aware Training)** that can output an 8-bit Quantization friendly model - these tools can be used to improve the quantized accuracy and bring it near floating point accuracy. For more details, please refer to the section on [Quantization](docs/Quantization.md).

 
 

-We also have a **Calibration tool for Quantization** that can output an 8-bit Quantization friendly model using a few calibration images - this tool can be used to improve the quantized accuracy and bring it near floating point accuracy. This tools adjusts weights and biases and also collects the ranges of activations to make the model quantization friendly. For more details, please refer to the section on Quantization.
+Our expectation is that these Deep Learning examples, models and tools will find application in a variety of problems, and the users will be able to build upon the **building blocks** that we have provided. 

 
 

-**Several of these models have been verified to work on [TI's Jacinto Automotive Processors](http://www.ti.com/processors/automotive-processors/tdax-adas-socs/overview.html).** 
+**Several of these models have been verified to work on [TI's Jacinto7 Automotive Processors](http://www.ti.com/processors/automotive-processors/tdax-adas-socs/overview.html).** This code is primarily intended for learning and research. 

 
 ## Installation Instructions
 - These instructions are for installation on **Ubuntu 18.04**. 
 
 ## Installation Instructions
 - These instructions are for installation on **Ubuntu 18.04**. 


@@ -35,19 +34,18 @@ The following examples are currently available. Click on each of the links below
     - [Motion Segmentation](docs/Motion_Segmentation.md)
     - [**Multi Task Estimation**](docs/Multi_Task_Learning.md)
 - Object Detection
     - [Motion Segmentation](docs/Motion_Segmentation.md)
     - [**Multi Task Estimation**](docs/Multi_Task_Learning.md)
 - Object Detection

-    - [**Object Detection**](docs/Object_Detection.md)
-    - [Object Keypoint Estimation](docs/Keypoint_Estimation.md)
- - [**Quantization**](docs/Quantization.md)<br>
-
-
-We have written down some of the common training and validation commands in the shell scripts (.sh files) provided in the root folder.
+    - Object Detection - coming soon..
+    - Object Keypoint Estimation - coming soon..
+ - Quantization
+     - [**Post Training Calibration For Quantization**](docs/Quantization.md)<br>
+     - [**Quantization Aware Training**](docs/Quantization.md)<br>

 
 

-## Model Zoo
-Sample models are uploaded in our [modelzoo](./data/modelzoo). Some of our scripts use the pretrained models from this modelzoo.
+Some of the common training and validation commands are provided in shell scripts (.sh files) in the root folder.

 
 ## Additional Information
 For information on other similar devkits, please visit:<br> 
 
 ## Additional Information
 For information on other similar devkits, please visit:<br> 

-http://git.ti.com/jacinto-ai-devkit
+- [https://github.com/TexasInstruments/jacinto-ai-devkit](https://github.com/TexasInstruments/jacinto-ai-devkit)<br> AND
+- [https://git.ti.com/jacinto-ai-devkit](https://git.ti.com/jacinto-ai-devkit)<br>

 
 ## Acknowledgements
 
 
 ## Acknowledgements
 

diff --git a/docs/Quantization.md b/docs/Quantization.md
index 14c48810c2eb0a933fa63c176b8d0d0728497589..ff07b768fb6c4cef113c5b922e095efa8be5abd3 100644 (file)
--- a/docs/Quantization.md
+++ b/docs/Quantization.md
@@ -1,43 +1,52 @@
 # Quantization
 
 # Quantization
 

-As we know Quantization is the process of converting floating point data & operations to fixed point (integer). CNNs can be quantized to 8-bits integer data/operations without significant accuracy loss. This includes quantization of weights, feature maps and all operations (including convolution of weights). **We use power-of-2, symmetric quantization for both weights and activations**.
+Quantization is the process of converting floating point data & operations to fixed point (integer). CNNs can be quantized to 8-bits integer data/operations without significant accuracy loss. This includes quantization of weights, feature maps and all operations (including convolution of weights). The quantization style used in this code is **power-of-2, symmetric quantization for both weights and activations**.

 
 

-There are two primary methods of quantization - Post Training Quantization and Trained Quantization. 
+In order to make the activations quantization friendly, it is important to clip them during Calibration or Trained Quantization. PACT2 activation function has been developed to clip the activations to a power-of-two value. PACT2 can be used in the place of commonly used activation functions such as ReLU. 
+<p float="left"> <img src="quantization/pact2_activation.png" width="640" hspace="5"/> </p>
+This code also contains two wrapper modules called QuantCalibrateModule and QuantTrainModule that will handle such tasks as replacing ReLUs with PACT2, quantizing weights, quantizing activations etc - so the user does not need to do any of these things - the only thing that is required is to wrap the user's module in these wapper modules as explained below.

 
 

-## Post Training Calibration & Quantization
+There are two primary methods of quantization - Post Training Calibration For Quantization and Trained Quantization (a.k.a Quantization Aware Training). 
+
+## Post Training Calibration For Quantization

 
 Post Training Calibration & Quantization can take a model trained in floating point and with a few steps convert it to a model that is friendly for quantized inference. Compared to the alternative (Trained Quantization), the advantages of this method are:
 - Calibration is fast - a typical calibration finishes in a few minutes.  
 - Ground truth is not required - just input images are sufficient.
 - Loss function or backward (back-propagation) are not required. 
 
 
 Post Training Calibration & Quantization can take a model trained in floating point and with a few steps convert it to a model that is friendly for quantized inference. Compared to the alternative (Trained Quantization), the advantages of this method are:
 - Calibration is fast - a typical calibration finishes in a few minutes.  
 - Ground truth is not required - just input images are sufficient.
 - Loss function or backward (back-propagation) are not required. 
 

-Thus, this is the preferred method of quantization from an ease of use point of view.  As explained earlier, in this method, the training happens entirely in floating point. The inference (possibly in an embedded device) happens in fixed point. In between training and fixed point inference, the model goes through the step called Calibration with some sample images. The Calibration happens in PC and Quantized Inference happens in the embedded device. Calibration basically tries to make the quantized output similar to the floating point output - by choosing appropriate activation ranges, weights and biases. The step by step process is as follows:
+Thus, this is the preferred method of quantization from an ease of use point of view.  As explained earlier, in this method, the training happens entirely in floating point. The inference (possibly in an embedded device) happens in fixed point. In between training and fixed point inference, the model goes through the step called Calibration with some sample images. The Calibration happens in PC and Quantized Inference happens in the embedded device. Calibration basically tries to make the quantized output similar to the floating point output - by choosing appropriate activation ranges, weights and biases. 

 
 

-#### Model preparation:
-- Replace all the ReLU, ReLU6 layers in the model by PACT2. Insert PACT2 after Convolution+BatchNorm if a ReLU is missing after that.  Insert PACT2 anywhere else required - where activation range clipping and range collection is required. For example it can ne after the Fully Connected Layer. We use forward post hooks of PyTorch nn.Modules to call these extra activation functions. Thus we are able to add these extra activations without disturbing the loading of existing pre-trained weights.
-- Clip the weights to an appropriate range if the weight range is very high.
-
-#### Forward iterations:
-- For each iteration perform a forward in floating point using the original weights and biases. During this pass PACT2 layers will collect output ranges using histogram and running average.
-- In addition, perform Convolution+BatchNorm merging and quantization of the resulting weights. These quantized and de-quantized weights are used in a forward pass. Ranges collected by PACT2 is used for activation quantization (and de-quantization) to generate quantized output.
-- The floating point output and quantized output are compared using statistic measures. Using such statistic measures, we can adjust the weights and biases of Convolutions and Batch Normalization layers - so that the quantized output becomes closer to the floating point output.
-- Within a few iterations, we could get reasonable quantization accuracy for several models that we tried this method on.
+A block diagram of Post Training Calibration is shown below:
+<p float="left"> <img src="quantization/bias_calibration.png" width="640" hspace="5"/> </p>

 
 Depending on how the activation range is collected and Quantization is done, we have a few variants of this basic scheme.  
 - Simple Calib: Calibration includes PACT2 for activation clipping, running average and range collection. In this method we use min-max for activation range collection (no histogram).
 - **Advanced Calib**: Calibration includes PACT2 with histogram based ranges, Weight clipping, Bias correction. 
 
 Depending on how the activation range is collected and Quantization is done, we have a few variants of this basic scheme.  
 - Simple Calib: Calibration includes PACT2 for activation clipping, running average and range collection. In this method we use min-max for activation range collection (no histogram).
 - **Advanced Calib**: Calibration includes PACT2 with histogram based ranges, Weight clipping, Bias correction. 

-- Advanced DW Calib: Calibration includes Per-Channel Quantization of Weights for Depthwise layers, PACT2 with histogram based ranges, Weight clipping, Bias correction. One of the earliest papers that clearly explained the benefits of Per-Channel Quantization for weights only (while the activations are quantized as Per-Tensor) is [6] 
+- Advanced DW Calib: Calibration includes Per-Channel Quantization of Weights for Depthwise layers, PACT2 with histogram based ranges, Weight clipping, Bias correction. One of the earliest papers that clearly explained the benefits of Per-Channel Quantization for weights only (while the activations are quantized as Per-Tensor) is [[6]]

 - Advanced Per-Chan Calib: Calibration includes Per-Channel Quantization for all layers, PACT2 with histogram based ranges, Weight clipping, Bias correction.
 
 Out of these methods, **Advanced Calib** is our recommended Calibration method as of now, as it has the best trade-off between the Accuracy and the features required during fixed point inference. All the Calibration scripts that we have in this page uses "Advanced Calib" by default. Other Calibration methods described here are for information only. 
 
 - Advanced Per-Chan Calib: Calibration includes Per-Channel Quantization for all layers, PACT2 with histogram based ranges, Weight clipping, Bias correction.
 
 Out of these methods, **Advanced Calib** is our recommended Calibration method as of now, as it has the best trade-off between the Accuracy and the features required during fixed point inference. All the Calibration scripts that we have in this page uses "Advanced Calib" by default. Other Calibration methods described here are for information only. 
 

-In order to do Calibration easily we have a developed a wrapper module called QuantCalibrateModule, which is located in pytorch_jacinto_ai.xnn.quantize.QuantCalibrateModule. We make use of a kind of Parametric Activation called **PACT2** in order to store the calibrated ranges of activations. PACT2 is a improved form of PACT [1]. **PACT2 uses power of 2 activation ranges** for activation clipping. PACT2 can learn ranges very quickly (using a statistic method) without back propagation - this feature makes it quite attractive for Calibration. Our wrapper module replaces all the ReLUs in the model with PACT2. It also inserts PACT2 in other places where activation ranges need to be collected.  Statistical range clipping in PACT2 improves the Quantized Accuracy over simple min-max range clipping. 
+#### How to use  QuantCalibrateModule
+- The section briefly explains how to make use of our helper/wrapper module to do the calibration of your model. For further details, please see pytorch_jacinto_ai.engine.train_classification.py and pytorch_jacinto_ai.engine.train_pixel2pixel.py. The step by step process is as follows:
+
+###### Model preparation:
+- Replace all the ReLU, ReLU6 layers in the model by PACT2. Insert PACT2 after Convolution+BatchNorm if a ReLU is missing after that.  Insert PACT2 anywhere else required - where activation range clipping and range collection is required. For example it can ne after the Fully Connected Layer. We use forward post hooks of PyTorch nn.Modules to call these extra activation functions. Thus we are able to add these extra activations without disturbing the loading of existing pre-trained weights.
+- Clip the weights to an appropriate range if the weight range is very high.
+- Note that this Model preparation is automatically done by QuantCalibrateModule
+
+###### Forward iterations:
+- For each iteration perform a forward in floating point using the original weights and biases. During this pass PACT2 layers will collect output ranges using histogram and running average.
+- In addition, perform Convolution+BatchNorm merging and quantization of the resulting weights. These quantized and de-quantized weights are used in a forward pass. Ranges collected by PACT2 is used for activation quantization (and de-quantization) to generate quantized output.
+- The floating point output and quantized output are compared using statistic measures. Using such statistic measures, we can adjust the weights and biases of Convolutions and Batch Normalization layers - so that the quantized output becomes closer to the floating point output.
+- Within a few iterations, we could get reasonable quantization accuracy for several models that we tried this method on.

 
 

-As explained, our method of **Calibration does not need ground truth, loss function or back propagation.** However in our script, we make use of ground truth to measure the loss/accuracy even in the Calibration stage - although that is not necessary. 
+In order to do Calibration easily we have a developed a wrapper module called QuantCalibrateModule, which is located in pytorch_jacinto_ai.xnn.quantize.QuantCalibrateModule. We make use of a kind of Parametric Activation called **PACT2** in order to store the calibrated ranges of activations. PACT2 is a improved form of PACT [[1]]. **PACT2 uses power of 2 activation ranges** for activation clipping. PACT2 can learn ranges very quickly (using a statistic method) without back propagation - this feature makes it quite attractive for Calibration. Our wrapper module replaces all the ReLUs in the model with PACT2. It also inserts PACT2 in other places where activation ranges need to be collected.  Statistical range clipping in PACT2 improves the Quantized Accuracy over simple min-max range clipping. 

 
 

-#### How to use  QuantCalibrateModule
-The section briefly explains how to make use of our helper/wrapper module to do the calibration of your model. For further details, please see pytorch_jacinto_ai.engine.train_classification.py and pytorch_jacinto_ai.engine.train_pixel2pixel.py.
+As explained, the method of **Calibration does not need ground truth, loss function or back propagation.** However in the calibration script, we make use of ground truth to measure the loss/accuracy even in the Calibration stage - although that is not necessary. 

 
 

+###### Sample Code Snippet:

 ```
 # create your model here:
 model = ...
 ```
 # create your model here:
 model = ...


@@ -67,36 +76,38 @@ torch.onnx.export(model.module, dummy_input, os.path.join(save_path,'model.onnx'
 
 Few examples of calibration are provided below. These commands are also listed in the file **run_quantization.sh** for convenience.<br>
 
 
 Few examples of calibration are provided below. These commands are also listed in the file **run_quantization.sh** for convenience.<br>
 

-#### Calibration of ImageNet Classification MobileNetV2 model 
+- Calibration of ImageNet Classification MobileNetV2 model 

 ```
 python ./scripts/train_classification_main.py --phase calibration --dataset_name image_folder_classification --model_name mobilenetv2_tv_x1 --data_path ./data/datasets/image_folder_classification --pretrained https://download.pytorch.org/models/mobilenet_v2-b0353104.pth --batch_size 64 --quantize True --epochs 1 --epoch_size 100
 ```
 
 ```
 python ./scripts/train_classification_main.py --phase calibration --dataset_name image_folder_classification --model_name mobilenetv2_tv_x1 --data_path ./data/datasets/image_folder_classification --pretrained https://download.pytorch.org/models/mobilenet_v2-b0353104.pth --batch_size 64 --quantize True --epochs 1 --epoch_size 100
 ```
 

-#### Calibration of ImageNet Classification ResNet50 model 
+- Calibration of ImageNet Classification ResNet50 model 

 ```
 python ./scripts/train_classification_main.py --phase calibration --dataset_name image_folder_classification --model_name mobilenetv2_tv_x1 --data_path ./data/datasets/image_folder_classification --pretrained https://download.pytorch.org/models/resnet50-19c8e357.pth --batch_size 64 --quantize True --epochs 1 --epoch_size 100
 ```
 
 ```
 python ./scripts/train_classification_main.py --phase calibration --dataset_name image_folder_classification --model_name mobilenetv2_tv_x1 --data_path ./data/datasets/image_folder_classification --pretrained https://download.pytorch.org/models/resnet50-19c8e357.pth --batch_size 64 --quantize True --epochs 1 --epoch_size 100
 ```
 

-#### Calibration of Cityscapes Semantic Segmentation model 
+- Calibration of Cityscapes Semantic Segmentation model 

 ```
 python ./scripts/train_segmentation_main.py --phase calibration --dataset_name cityscapes_segmentation --model_name deeplabv3lite_mobilenetv2_tv --data_path ./data/datasets/cityscapes/data --img_resize 384 768 --output_size 1024 2048 --gpus 0 1 
 --pretrained ./data/modelzoo/semantic_segmentation/cityscapes/deeplabv3lite-mobilenetv2/cityscapes_segmentation_deeplabv3lite-mobilenetv2_2019-06-26-08-59-32.pth 
 --batch_size 12 --quantize True --epochs 1 --epoch_size 100
 ```
 
 ```
 python ./scripts/train_segmentation_main.py --phase calibration --dataset_name cityscapes_segmentation --model_name deeplabv3lite_mobilenetv2_tv --data_path ./data/datasets/cityscapes/data --img_resize 384 768 --output_size 1024 2048 --gpus 0 1 
 --pretrained ./data/modelzoo/semantic_segmentation/cityscapes/deeplabv3lite-mobilenetv2/cityscapes_segmentation_deeplabv3lite-mobilenetv2_2019-06-26-08-59-32.pth 
 --batch_size 12 --quantize True --epochs 1 --epoch_size 100
 ```
 

-## Trained Quantization
-
+## Trained Quantization (a.k.a Quantization Aware Training)

 As explained in the previous section, Calibration is our preferred method of making a model quantization friendly. However, in exceptional cases, it is possible that the drop in accuracy during calibration is more than acceptable. In this case, Trained Quantization can be used. 
 
 As explained in the previous section, Calibration is our preferred method of making a model quantization friendly. However, in exceptional cases, it is possible that the drop in accuracy during calibration is more than acceptable. In this case, Trained Quantization can be used. 
 

-Unlike Calibration, Trained Quantization involves ground truth, loss function and back propagation. The most popular method of trained quantization is [4]. It takes care of merging Convolution layers with the adjascent Batch Normalization layers (on-the-fly) during the quantized training (if this merging is not correctly done, quantized training may not improve the accuracy). In addition, we use Straight-Through Estimation (STE) [2,3] to improve the gradient flow in back-propagation. Also, the statistical range clipping in PACT2 improves the Quantized Accuracy over simple min-max range clipping. 
+Unlike Calibration, Trained Quantization involves ground truth, loss function and back propagation. The most popular method of trained quantization is [[4]]. It takes care of merging Convolution layers with the adjascent Batch Normalization layers (on-the-fly) during the quantized training (if this merging is not correctly done, quantized training may not improve the accuracy). In addition, we use Straight-Through Estimation (STE) [[2,3]] to improve the gradient flow in back-propagation. Also, the statistical range clipping in PACT2 improves the Quantized Accuracy over simple min-max range clipping. 
+
+Note: Instead of STE and statistical ranges for PACT2, we also tried out approximate gradients for scale and trained quantization thresholds proposed in [[5]] (We did not use the gradient nomralization and log-domain training mentioned in the paper). We found that method to be able to learn the clipping thresholds for initial few epochs, but became unstable after a few epochs and loss became high. Compared to that learned thresholds method, our statistical PACT2 ranges/thresholds combined with STE is simple and stable. 

 
 

-Note: Instead of STE and statistical ranges for PACT2, we also tried out approximate gradients for scale and trained quantization thresholds proposed in [5] (We did not use the gradient nomralization and log-domain training mentioned in the paper). We found that method to be able to learn the clipping thresholds for initial few epochs, but became unstable after a few epochs and loss became high. Compared to that learned thresholds method, our statistical PACT2 ranges/thresholds combined with STE is simple and stable. 
+A block diagram of Trained Quantization is shown below:
+<p float="left"> <img src="quantization/trained_quant_ste.png" width="640" hspace="5"/> </p>

 
 In order to enable quantized training, we have developed the wrapper class pytorch_jacinto_ai.xnn.quantize.QuantTrainModule. The usage of this module can be seen in pytorch_jacinto_ai.engine.train_classification.py and pytorch_jacinto_ai.engine.train_pixel2pixel.py. 
 ```
 model = pytorch_jacinto_ai.xnn.quantize.QuantTrainModule(model, dummy_input=dummy_input)
 ```
 
 In order to enable quantized training, we have developed the wrapper class pytorch_jacinto_ai.xnn.quantize.QuantTrainModule. The usage of this module can be seen in pytorch_jacinto_ai.engine.train_classification.py and pytorch_jacinto_ai.engine.train_pixel2pixel.py. 
 ```
 model = pytorch_jacinto_ai.xnn.quantize.QuantTrainModule(model, dummy_input=dummy_input)
 ```

-The resultant model can then be used for training as usual and it will take care of quantization constraints during the training forward and backward passes.
+The Model Preparation steps used for Calibration applies for Trained Quantization as well. - Note that this Model preparation is automatically done by QuantTrainModule. The resultant model can then be used for training as usual and it will take care of quantization constraints during the training forward and backward passes.

 
 One word of caution is that our current implementation of Trained Quantization is a bit slow. The reason for this slowdown is that our implementation is using the top-level python layer of PyTorch and not the underlying C++ layer. But with PyTorch natively supporting the functionality required for quantization under the hood - we hope that this speed issue can be resolved in a future update. 
 
 
 One word of caution is that our current implementation of Trained Quantization is a bit slow. The reason for this slowdown is that our implementation is using the top-level python layer of PyTorch and not the underlying C++ layer. But with PyTorch natively supporting the functionality required for quantization under the hood - we hope that this speed issue can be resolved in a future update. 
 


@@ -115,7 +126,7 @@ python ./scripts/train_segmentation_main.py --dataset_name cityscapes_segmentati
 - This may not be such a problem as calibration and quantization may not take as much time as the original training. 
 - If your calibration/training crashes with insufficient GPU memory, reduce the batch size and try again.
 - The original training (without quantization) can use Multi-GPU as usual and we do not have any restrictions on that.
 - This may not be such a problem as calibration and quantization may not take as much time as the original training. 
 - If your calibration/training crashes with insufficient GPU memory, reduce the batch size and try again.
 - The original training (without quantization) can use Multi-GPU as usual and we do not have any restrictions on that.

-- Tools for Calibration and Trained Quantization have started appearing in mainstream Deep Learning training frameworks [7,8]. Using the tools natively provided by these frameworks may be faster compared to an implementation in the Python layer of these frameworks (like we have done) - but they may not be mature currently. 
+- Tools for Calibration and Trained Quantization have started appearing in mainstream Deep Learning training frameworks [[7,8]]. Using the tools natively provided by these frameworks may be faster compared to an implementation in the Python layer of these frameworks (like we have done) - but they may not be mature currently. 

 - In order that the activation range estimation is correct, **the same module should not be re-used multiple times within the module**. Unfortunately, in the torchvision ResNet models, the ReLU module in the BasicBlock and BottleneckBlock are re-used multiple times. We have corrected this by defining separate ReLU modules. This change is minor and **does not** affect the loading of existing pretrained weights. See the [our modified ResNet model definition here](./modules/pytorch_jacinto_ai/vision/models/resnet.py).
 - Use Modules instead of functions as much as possible (we make use of modules to decide whether to do activation range clipping or not). For example use torch.nn.AdaptiveAvgPool2d() instead of torch.nn.functional.adaptive_avg_pool2d(), torch.nn.Flatten() instead of torch.nn.functional.flatten() etc. If you are using functions in your model and is giving poor quantized accuracy, then consider replacing those functions by the corresponding modules.
 
 - In order that the activation range estimation is correct, **the same module should not be re-used multiple times within the module**. Unfortunately, in the torchvision ResNet models, the ReLU module in the BasicBlock and BottleneckBlock are re-used multiple times. We have corrected this by defining separate ReLU modules. This change is minor and **does not** affect the loading of existing pretrained weights. See the [our modified ResNet model definition here](./modules/pytorch_jacinto_ai/vision/models/resnet.py).
 - Use Modules instead of functions as much as possible (we make use of modules to decide whether to do activation range clipping or not). For example use torch.nn.AdaptiveAvgPool2d() instead of torch.nn.functional.adaptive_avg_pool2d(), torch.nn.Flatten() instead of torch.nn.functional.flatten() etc. If you are using functions in your model and is giving poor quantized accuracy, then consider replacing those functions by the corresponding modules.
 

diff --git a/docs/Semantic_Segmentation.md b/docs/Semantic_Segmentation.md
index 320c878ae65643b77d607bd18b5ee0469f7ad293..a341c860df8830271bf14525827bd2db727ca1be 100644 (file)
--- a/docs/Semantic_Segmentation.md
+++ b/docs/Semantic_Segmentation.md
@@ -2,52 +2,27 @@
 
 Semantic segmentation assigns a class to each pixel of the image. It is useful for tasks such as lane detection, road segmentation etc. 
 
 
 Semantic segmentation assigns a class to each pixel of the image. It is useful for tasks such as lane detection, road segmentation etc. 
 

-Commonly used Training/Validation commands are listed in the file [run_segmentation.sh](../run_segmentation.sh). Un-commend one and run the file to start the run. 
+Commonly used Training/Validation commands are listed in the file [run_segmentation.sh](../run_segmentation.sh). Uncommend one line and run the file to start the run. 

 
 

-## Models
-We have defined a set of example models for all pixel to pixel tasks. 
+## Model Configurations
+A set of common example model configurations are defined for all pixel to pixel tasks. These models can support multiple inputs (for example image and optical flow) as well as support multiple decoders for multi-task prediction (for example semantic segmentation + depth estimation + motion segmentation). 

 
 

-These models can support multiple inputs (for example image and optical flow) as well as support multiple decoders for multi-task prediction (for example semantic segmentation + depth estimation + motion segmentation). 
+Whether to use multiple inputs or how many decoders to use are fully configurable. This framework is also flexible to add different model architectures or backbone networks. The following model configurations are currently available:<br>

 
 

-Whether to use multiple inputs or how many decoders to use are fully configurable. Our framework is also flexible to add different model architectures or backbone networks if you wish to do. 
+* **deeplabv3lite_mobilenetv2_tv**: (default) This model is mostly similar to the DeepLabV3+ model [[6]] using MobileNetV2 backbone. The difference with DeepLabV3+ is that we removed the convolutions after the shortcut and kep one set of depthwise separable convolutions to generate the prediction. The ASPP module that we used is a lite-weight variant with depthwise separable convolutions (DWASPP). We found that this reduces complexity without sacrificing accuracy. Due to this we call this model DeepLabV3+(Lite) or simply  DeepLabV3Lite. (Note: The suffix "_tv" is used to indicate that our backbone model is from torchvision)<br> 

 
 

-The models that we support of the shelf are:<br>
+* **fpn_pixel2pixel_aspp_mobilenetv2_tv**: This is similar to Feature Pyramid Network [[3]], but adapted for pixel2pixel tasks. We stop the decoder at a stride of 4 and then upsample to the final resolution from there. We also use DWASPP module to improve the receptive field. We call this model FPNPixel2Pixel. 

 
 

-* **deeplabv3lite_mobilenetv2_tv**: (default) This model is mostly similar to the DeepLabV3+ model [3] using MobileNetV2 backbone. The difference with DeepLabV3+ is that we removed the convolutions after the shortcut and kep one set of depthwise separable convolutions to generate the prediction. The ASPP module that we used is a lite-weight variant with depthwise separable convolutions (DWASPP). We found that this reduces complexity without sacrificing accuracy. Due to this we call this model DeepLabV3+(Lite) or simply  DeepLabV3Lite. (Note: The suffix "_tv" is used to indicate that our backbone model is from torchvision)<br> 
-
-* **fpn_pixel2pixel_aspp_mobilenetv2_tv**: This is similar to Feature Pyramid Network [4], but adapted for pixel2pixel tasks. We stop the decoder at a stride of 4 and then upsample to the final resolution from there. We also use DWASPP module to improve the receptive field. We call this model FPNPixel2Pixel. 
+* **fpn_pixel2pixel_aspp_mobilenetv2_tv_es64**: This is also FPN, but with a larger encoder stride(64). This is a low complexity model that can be used with higher resolutions.

 
 * **fpn_pixel2pixel_aspp_resnet50**: Feature Pyramid Network (FPN) based pixel2pixel using ResNet50 backbone with DWASPP.
 
 
 * **fpn_pixel2pixel_aspp_resnet50**: Feature Pyramid Network (FPN) based pixel2pixel using ResNet50 backbone with DWASPP.
 

-## Datasets: Cityscapes Dataset [1]
-
-* Download the cityscapes dataset from https://www.cityscapes-dataset.com/. You will need need to register before the data can be downloaded. Unzip the data into the folder ./data/datasets/cityscapes/data. This folder should contain leftImg8bit and gtFine folders of cityscapes. 
-
-* These examples use two gpus because we use slightly higher accuracy when we restricted the number of GPUs used. 
-
-* Training can be done as follows:<br>
-    ```
-    python ./scripts/train_segmentation_main.py --dataset_name cityscapes_segmentation --data_path ./data/datasets/cityscapes/data --img_resize 384 768 --output_size 1024 2048 --gpus 0 1
-  ```
-
- * During the training, **validation** accuracy will also be printed. But if you want to explicitly check the accuracy again with **validation** set, it can be done:<br>
-    ```
-    python ./scripts/train_segmentation_main.py --evaluate True --dataset_name cityscapes_segmentation --data_path ./data/datasets/cityscapes/data --img_resize 384 768 --output_size 1024 2048 --gpus 0 1
-    ```
-  
-  * It is possible to use a different image size. For example, we trained for 1536x768 resolution by the following. (We used a smaller crop size compared to the image resize resolution to reduce GPU memory usage). <br>
-    ```
-    python ./scripts/train_segmentation_main.py --dataset_name cityscapes_segmentation --data_path ./data/datasets/cityscapes/data --img_resize 768 1536 --rand_crop 512 1024 --output_size 1024 2048 --gpus 0 1
-    ```
+## Datasets: Cityscapes Dataset 

 
 

-* Train FPNPixel2Pixel model at 1536x768 resolution (use 1024x512 crop to reduce memory usage):<br>
-    ```
-    python ./scripts/train_segmentation_main.py --model_name fpn_pixel2pixel_aspp_mobilenetv2_tv --dataset_name cityscapes_segmentation --data_path ./data/datasets/cityscapes/data --img_resize 768 1536 --rand_crop 512 1024 --output_size 1024 2048 --gpus 0 1
-    ```
+* Download the Cityscapes dataset [[1]] from https://www.cityscapes-dataset.com/. You will need need to register before the data can be downloaded. Unzip the data into the folder ./data/datasets/cityscapes/data. This folder should contain leftImg8bit and gtFine folders of cityscapes. 

 
 

-## Datasets: VOC Dataset [2]
-### Download the data
-* The dataset can be downloaded using the following:<br>
+## Datasets: VOC Dataset 
+* The PASCAL VOC dataset [[2]] can be downloaded using the following:<br>

     ```
     mkdir ./data/datasets/voc
     cd /data/datasets/voc
     ```
     mkdir ./data/datasets/voc
     cd /data/datasets/voc


@@ -55,7 +30,6 @@ The models that we support of the shelf are:<br>
     wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar
     wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar
     ```
     wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar
     wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar
     ```

-### Extract the data.

 * Extact the dataset files into ./data/datasets/voc/VOCdevkit
     ```
     tar -xvf VOCtrainval_11-May-2012.tar
 * Extact the dataset files into ./data/datasets/voc/VOCdevkit
     ```
     tar -xvf VOCtrainval_11-May-2012.tar


@@ -70,28 +44,60 @@ The models that we support of the shelf are:<br>
     ls -1 SegmentationClassAug | sed s/.png// > trainaug.txt
     ```
 
     ls -1 SegmentationClassAug | sed s/.png// > trainaug.txt
     ```
 

-* training can be done as follows from the base folder of the repository:<br>
+## Training
+* These examples use two gpus because we use slightly higher accuracy when we restricted the number of GPUs used. 
+
+* **Cityscapes Segmentation Training** can be done as follows:<br>
+    ```
+    python ./scripts/train_segmentation_main.py --model_name deeplabv3lite_mobilenetv2_tv --dataset_name cityscapes_segmentation --data_path ./data/datasets/cityscapes/data --img_resize 384 768 --output_size 1024 2048 --pretrained https://download.pytorch.org/models/mobilenet_v2-b0353104.pth --gpus 0 1
+  ```
+  
+  * It is possible to use a different image size. For example, we trained for 1536x768 resolution by the following. (We used a smaller crop size compared to the image resize resolution to reduce GPU memory usage). <br>
+    ```
+    python ./scripts/train_segmentation_main.py --model_name deeplabv3lite_mobilenetv2_tv --dataset_name cityscapes_segmentation --data_path ./data/datasets/cityscapes/data --img_resize 768 1536 --rand_crop 512 1024 --output_size 1024 2048 --pretrained https://download.pytorch.org/models/mobilenet_v2-b0353104.pth --gpus 0 1
+    ```
+
+* Train FPNPixel2Pixel model at 1536x768 resolution (use 1024x512 crop to reduce memory usage):<br>

     ```
     ```

-    python ./scripts/train_segmentation_main.py --dataset_name voc_segmentation --data_path ./data/datasets/voc --img_resize 512 512 --output_size 512 512 --gpus 0 1
+    python ./scripts/train_segmentation_main.py --model_name fpn_pixel2pixel_aspp_mobilenetv2_tv --dataset_name cityscapes_segmentation --data_path ./data/datasets/cityscapes/data --img_resize 768 1536 --rand_crop 512 1024 --output_size 1024 2048 --pretrained https://download.pytorch.org/models/mobilenet_v2-b0353104.pth --gpus 0 1
+    ```
+ 
+* **VOC Segmentation Training** can be done as follows:<br>
+    ```
+    python ./scripts/train_segmentation_main.py --model_name deeplabv3lite_mobilenetv2_tv --dataset_name voc_segmentation --data_path ./data/datasets/voc --img_resize 512 512 --output_size 512 512 --pretrained https://download.pytorch.org/models/mobilenet_v2-b0353104.pth --gpus 0 1

   ```
 
   ```
 

+## Validation
+ * During the training, **validation** accuracy will also be printed. But to explicitly check the accuracy again with **validation** set, it can be done as follows (fill in the path to the pretrained model):<br>
+    ```
+    python ./scripts/train_segmentation_main.py --evaluate True --model_name deeplabv3lite_mobilenetv2_tv --dataset_name cityscapes_segmentation --data_path ./data/datasets/cityscapes/data --img_resize 384 768 --output_size 1024 2048 --gpus 0 1 --pretrained ?????
+    ```
+
+## Inference
+Inference can be done as follows (fill in the path to the pretrained model):<br>
+    ```
+    python ./scripts/infer_segmentation_main.py --evaluate True --model_name deeplabv3lite_mobilenetv2_tv --dataset_name cityscapes_segmentation_measure --data_path ./data/datasets/cityscapes/data --img_resize 384 768 --output_size 1024 2048 --gpus 0 1 --pretrained ?????
+    ```
+          

 ## Results
 
 ### Cityscapes Segmentation
 
 ## Results
 
 ### Cityscapes Segmentation
 

-|Dataset    |Mode Architecture|Backbone Model|Model Name                         |Backbone Stride|Resolution |Complexity (GigaMACS)|MeanIoU%  |
-|---------  |----------       |-----------   |---------------------------------  |-------------- |-----------|--------             |----------|
-|Cityscapes |DeepLabV3Lite    |MobileNetV2   |deeplabv3lite_mobilenetv2_tv       |16             |768x384    |3.54                 |**69.13** |
-|Cityscapes |FPNPixel2Pixel   |MobileNetV2   |fpn_pixel2pixel_mobilenetv2_tv     |32             |768x384    |3.84                 |**70.39** |
-|Cityscapes |FPNPixel2Pixel   |MobileNetV2   |fpn_pixel2pixel_mobilenetv2_tv_es64|64             |1536x768   |3.96                 |**71.28** |
-|Cityscapes |FPNPixel2Pixel   |MobileNetV2   |fpn_pixel2pixel_mobilenetv2_tv_es64|64             |2048x1024  |7.03                 |          |
-|Cityscapes |DeepLabV3Lite    |MobileNetV2   |deeplabv3lite_mobilenetv2_tv       |16             |1536x768   |14.48                |**73.59** |
-|Cityscapes |FPNPixel2Pixel   |MobileNetV2   |fpn_pixel2pixel_mobilenetv2_tv     |32             |1536x768   |15.37                |**74.98** |
-|-
-|Cityscapes |ERFNet[5]        |              |                                   |               |1024x512   |27.705               |69.7      |
-|Cityscapes |SwiftNetMNV2[6]  |MobileNetV2   |                                   |               |2048x1024  |41.0                 |75.3      |
-|Cityscapes |DeepLabV3Plus[3] |MobileNetV2   |                                   |16             |           |21.27                |70.71     |
-|Cityscapes |DeepLabV3Plus[3] |Xception65    |                                   |16             |           |418.64               |78.79     |
+|Dataset    |Mode Architecture   |Backbone Model|Backbone Stride|Resolution |Complexity (GigaMACS)|MeanIoU%  |Model Configuration Name           |
+|---------  |----------          |-----------   |-------------- |-----------|--------             |----------|---------------------------------  |
+|Cityscapes |DeepLabV3Lite       |MobileNetV2   |16             |768x384    |3.54                 |**69.13** |deeplabv3lite_mobilenetv2_tv       |
+|Cityscapes |FPNPixel2Pixel      |MobileNetV2   |32             |768x384    |3.84                 |**70.39** |fpn_pixel2pixel_mobilenetv2_tv     |
+|Cityscapes |FPNPixel2Pixel      |MobileNetV2   |64             |1536x768   |3.96                 |**71.28** |fpn_pixel2pixel_mobilenetv2_tv_es64|
+|Cityscapes |FPNPixel2Pixel      |MobileNetV2   |64             |2048x1024  |7.03                 |          |fpn_pixel2pixel_mobilenetv2_tv_es64|
+|Cityscapes |DeepLabV3Lite       |MobileNetV2   |16             |1536x768   |14.48                |**73.59** |deeplabv3lite_mobilenetv2_tv       |
+|Cityscapes |FPNPixel2Pixel      |MobileNetV2   |32             |1536x768   |15.37                |**74.98** |fpn_pixel2pixel_mobilenetv2_tv     |
+
+|Dataset    |Mode Architecture   |Backbone Model|Backbone Stride|Resolution |Complexity (GigaMACS)|MeanIoU%  |Model Configuration Name           |
+|---------  |----------          |-----------   |-------------- |-----------|--------             |----------|---------------------------------  |
+|Cityscapes |ERFNet[[4]]         |              |               |1024x512   |27.705               |69.7      |N/A                                |
+|Cityscapes |SwiftNetMNV2[[5]]   |MobileNetV2   |               |2048x1024  |41.0                 |75.3      |N/A                                |
+|Cityscapes |DeepLabV3Plus[[6,7]]|MobileNetV2   |16             |           |21.27                |70.71     |N/A                                |
+|Cityscapes |DeepLabV3Plus[[6,7]]|Xception65    |16             |           |418.64               |78.79     |N/A                                |

 
 
 ## References
 
 
 ## References


@@ -101,12 +107,14 @@ The models that we support of the shelf are:<br>
 Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J. and Zisserman, A.
 International Journal of Computer Vision, 88(2), 303-338, 2010, http://host.robots.ox.ac.uk/pascal/VOC/
 
 Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J. and Zisserman, A.
 International Journal of Computer Vision, 88(2), 303-338, 2010, http://host.robots.ox.ac.uk/pascal/VOC/
 

-[3] Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation, Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam, CVPR 2018, https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md
+[3] Feature Pyramid Networks for Object Detection, Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie, CVPR 2017
+
+[4] ERFNet: Efficient Residual Factorized ConvNet for Real-time Semantic Segmentation, E. Romera, J. M. Alvarez, L. M. Bergasa and R. Arroyo, Transactions on Intelligent Transportation Systems (T-ITS), 2017

 
 

-[4] Feature Pyramid Networks for Object Detection, Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie, CVPR 2017
+[5] In Defense of Pre-trained ImageNet Architectures for Real-time Semantic Segmentation of Road-driving Images, Marin Orsic, Ivan Kreso, Petra Bevandic, Sinisa Segvic, CVPR 2019.

 
 

-[5] ERFNet: Efficient Residual Factorized ConvNet for Real-time Semantic Segmentation, E. Romera, J. M. Alvarez, L. M. Bergasa and R. Arroyo, Transactions on Intelligent Transportation Systems (T-ITS), 2017
+[6] Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation, Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam, CVPR 2018

 
 

-[6] In Defense of Pre-trained ImageNet Architectures for Real-time Semantic Segmentation of Road-driving Images, Marin Orsic, Ivan Kreso, Petra Bevandic, Sinisa Segvic, CVPR 2019.
+[7] Tensorflow/Deeplab Model Zoo, https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md

 
 
 
 

diff --git a/docs/quantization/bias_calibration.png b/docs/quantization/bias_calibration.png
new file mode 100755 (executable)
index 0000000..2dd290f
Binary files /dev/null and b/docs/quantization/bias_calibration.png differ

diff --git a/docs/quantization/pact2_activation.png b/docs/quantization/pact2_activation.png
new file mode 100755 (executable)
index 0000000..6a5468b
Binary files /dev/null and b/docs/quantization/pact2_activation.png differ

diff --git a/docs/quantization/trained_quant_ste.png b/docs/quantization/trained_quant_ste.png
new file mode 100755 (executable)
index 0000000..f42b952
Binary files /dev/null and b/docs/quantization/trained_quant_ste.png differ

diff --git a/modules/pytorch_jacinto_ai/vision/models/pixel2pixel/pixel2pixelnet.py b/modules/pytorch_jacinto_ai/vision/models/pixel2pixel/pixel2pixelnet.py
index b420347db92c1e26823beb2b25725356483efa86..fbec903270e03dc9223f46c43250ef8b2749ca17 100644 (file)
--- a/modules/pytorch_jacinto_ai/vision/models/pixel2pixel/pixel2pixelnet.py
+++ b/modules/pytorch_jacinto_ai/vision/models/pixel2pixel/pixel2pixelnet.py
@@ -47,7 +47,8 @@ class Pixel2PixelNet(torch.nn.Module):
         self.output_channels = model_config.output_channels
         self.num_decoders = len(model_config.output_channels) if (model_config.num_decoders is None) else model_config.num_decoders
         self.split_outputs = model_config.split_outputs
         self.output_channels = model_config.output_channels
         self.num_decoders = len(model_config.output_channels) if (model_config.num_decoders is None) else model_config.num_decoders
         self.split_outputs = model_config.split_outputs

-        self.multi_task = xnn.layers.MultiTask(self.num_decoders, model_config.multi_task_type, model_config.output_type) if model_config.multi_task else None
+        self.multi_task = xnn.layers.MultiTask(num_splits=self.num_decoders, multi_task_type=model_config.multi_task_type, output_type=model_config.output_type,
+                                               multi_task_factors=model_config.multi_task_factors) if model_config.multi_task else None

 
         #if model_config.freeze_encoder:
             #xnn.utils.freeze_bn(self.encoder)
 
         #if model_config.freeze_encoder:
             #xnn.utils.freeze_bn(self.encoder)

diff --git a/modules/pytorch_jacinto_ai/xnn/layers/multi_task.py b/modules/pytorch_jacinto_ai/xnn/layers/multi_task.py
index 7c9f8e63ac43bcc45a9f17c0bab71090fb101b2e..d3c1975fd3a90af81559dd6ede450968dac5e435 100644 (file)
--- a/modules/pytorch_jacinto_ai/xnn/layers/multi_task.py
+++ b/modules/pytorch_jacinto_ai/xnn/layers/multi_task.py
@@ -9,7 +9,7 @@ class MultiTask(torch.nn.Module):
     '''
     multi_task_type: "grad_norm" "pseudo_grad_norm" "naive_grad_norm" "dwa" "dwa_gradnorm" "uncertainty"
     '''
     '''
     multi_task_type: "grad_norm" "pseudo_grad_norm" "naive_grad_norm" "dwa" "dwa_gradnorm" "uncertainty"
     '''

-    def __init__(self, num_splits = 1, multi_task_type=None, output_type=None):
+    def __init__(self, num_splits = 1, multi_task_type=None, output_type=None, multi_task_factors = None):

         super().__init__()
 
         ################################
         super().__init__()
 
         ################################


@@ -19,9 +19,11 @@ class MultiTask(torch.nn.Module):
         self.num_splits = num_splits
         self.losses_short = None
         self.losses_long = None
         self.num_splits = num_splits
         self.losses_short = None
         self.losses_long = None

+

         # self.loss_scales = torch.nn.Parameter(torch.ones(num_splits, device='cuda:0'))
         # self.loss_scales = torch.ones(num_splits, device='cuda:0', dtype=torch.float32) #requires_grad=True
         # self.loss_scales = torch.nn.Parameter(torch.ones(num_splits, device='cuda:0'))
         # self.loss_scales = torch.ones(num_splits, device='cuda:0', dtype=torch.float32) #requires_grad=True

-        self.loss_scales = torch.tensor([0.2, 0.2, 2.6], device='cuda:0', dtype=torch.float32)
+        self.loss_scales = torch.ones(num_splits, device='cuda:0', dtype=torch.float32) if multi_task_factors is None else \
+                            torch.tensor(multi_task_factors, device='cuda:0', dtype=torch.float32)

         self.loss_offsets = torch.zeros(num_splits, device='cuda:0', dtype=torch.float32) #None
         self.dy_norms_smooth = None
         self.register_backward_hook(self.backward_hook)
         self.loss_offsets = torch.zeros(num_splits, device='cuda:0', dtype=torch.float32) #None
         self.dy_norms_smooth = None
         self.register_backward_hook(self.backward_hook)