EvalNorm: Estimating Batch Normalization Statistics for Evaluation
Batch normalization (BN) has been very effective for deep learning and is widely used. However, when training with small minibatches, models using BN exhibit a significant degradation in performance. In this paper we study this peculiar behavior of BN to gain a better understanding of the problem, and identify a potential cause based on a statistical insight. We propose `EvalNorm' to address the issue by estimating corrected normalization statistics to use for BN during evaluation. EvalNorm supports online estimation of the corrected statistics while the model is being trained, and it does not affect the training scheme of the model. As a result, an added advantage of EvalNorm is that it can be used with existing pre-trained models allowing them to benefit from our method. EvalNorm yields large gains for models trained with smaller batches. Our experiments show that EvalNorm performs 6.18% (absolute) better than vanilla BN for a batchsize of 2 on ImageNet validation set and from 1.5 to 7.0 points (absolute) gain on the COCO object detection benchmark across a variety of setups.
NurtureToken New!

Token crowdsale for this paper ends in

Buy Nurture Tokens

Authors

Are you an author of this paper? Check the Twitter handle we have for you is correct.

Saurabh Singh (add twitter)
Abhinav Shrivastava (add twitter)
Ask The Authors

Ask the authors of this paper a question or leave a comment.

Read it. Rate it.
#1. Which part of the paper did you read?

#2. The paper contains new data or analyses that is openly accessible?
#3. The conclusion is supported by the data and analyses?
#4. The conclusion is of scientific interest?
#5. The result is likely to lead to future research?

Github
User:
None (add)
Repo:
None (add)
Stargazers:
0
Forks:
0
Open Issues:
0
Network:
0
Subscribers:
0
Language:
None
Youtube
Link:
None (add)
Views:
0
Likes:
0
Dislikes:
0
Favorites:
0
Comments:
0
Other
Sample Sizes (N=):
Inserted:
Words Total:
Words Unique:
Source:
Abstract:
None
04/14/19 06:03PM
6,484
1,971
Tweets
tamakoji: EvalNorm: Estimating Batch Normalization Statistics for Evaluation https://t.co/swJ8VEPr2N
arxiv_cscv: EvalNorm: Estimating Batch Normalization Statistics for Evaluation https://t.co/qrX5o8NY3x
arxiv_cscv: EvalNorm: Estimating Batch Normalization Statistics for Evaluation https://t.co/qrX5o8wmEX
arxivml: "EvalNorm: Estimating Batch Normalization Statistics for Evaluation", Saurabh Singh, Abhinav Shrivastava https://t.co/9R5zESWm74
arxiv_cs_LG: EvalNorm: Estimating Batch Normalization Statistics for Evaluation. Saurabh Singh and Abhinav Shrivastava https://t.co/NXGSNY3jjT
BrundageBot: EvalNorm: Estimating Batch Normalization Statistics for Evaluation. Saurabh Singh and Abhinav Shrivastava https://t.co/l4jqld0DcH
Images
Related