DARTS+: Improved Differentiable Architecture Search with Early Stopping
Recently, there has been a growing interest in automating the process of neural architecture design, and the Differentiable Architecture Search (DARTS) method makes the process available within a few GPU days. In particular, a hyper-network called one-shot model is introduced, over which the architecture can be searched continuously with gradient descent. However, the performance of DARTS is often observed to collapse when the number of search epochs becomes large. Meanwhile, lots of "skip-connects" are found in the selected architectures. In this paper, we claim that the cause of the collapse is that there exist cooperation and competition in the bi-level optimization in DARTS, where the architecture parameters and model weights are updated alternatively. Therefore, we propose a simple and effective algorithm, named "DARTS+", to avoid the collapse and improve the original DARTS, by "early stopping" the search procedure when meeting a certain criterion. We demonstrate that the proposed early stopping criterion is effective in avoiding the collapse issue. We also conduct experiments on benchmark datasets and show the effectiveness of our DARTS+ algorithm, where DARTS+ achieves $2.32\%$ test error on CIFAR10, $14.87\%$ on CIFAR100, and $23.7\%$ on ImageNet. We further remark that the idea of "early stopping" is implicitly included in some existing DARTS variants by manually setting a small number of search epochs, while we give an explicit criterion for "early stopping".
NurtureToken New!

Token crowdsale for this paper ends in

Buy Nurture Tokens

Authors

Are you an author of this paper? Check the Twitter handle we have for you is correct.

Hanwen Liang (add twitter)
Shifeng Zhang (add twitter)
Jiacheng Sun (edit)
Xingqiu He (add twitter)
Weiran Huang (add twitter)
Kechen Zhuang (edit)
Zhenguo Li (add twitter)
Ask The Authors

Ask the authors of this paper a question or leave a comment.

Read it. Rate it.
#1. Which part of the paper did you read?

#2. The paper contains new data or analyses that is openly accessible?
#3. The conclusion is supported by the data and analyses?
#4. The conclusion is of scientific interest?
#5. The result is likely to lead to future research?

Github
User:
None (add)
Repo:
None (add)
Stargazers:
0
Forks:
0
Open Issues:
0
Network:
0
Subscribers:
0
Language:
None
Youtube
Link:
None (add)
Views:
0
Likes:
0
Dislikes:
0
Favorites:
0
Comments:
0
Other
Sample Sizes (N=):
Inserted:
Words Total:
Words Unique:
Source:
Abstract:
None
09/15/19 06:04PM
6,389
1,869
Tweets
arxiv_cs_LG: DARTS+: Improved Differentiable Architecture Search with Early Stopping. Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li https://t.co/xVowP49Ov5
ohnabe: DARTS+ skip connection 連発で、collapse が起こるのを、early stopping の criteria に、セル内のskip connect 数を入れて、Dart より改善したらしい。https://t.co/thooMJ93Sr
Memoirs: DARTS+: Improved Differentiable Architecture Search with Early Stopping. https://t.co/LoAlvYDGiP
arxiv_cscv: DARTS+: Improved Differentiable Architecture Search with Early Stopping https://t.co/q23tgmrS9F
arxiv_in_review: #AAAI2020 DARTS+: Improved Differentiable Architecture Search with Early Stopping. (arXiv:1909.06035v1 [cs\.CV]) https://t.co/L9vgFkb8oy
arxiv_cscv: DARTS+: Improved Differentiable Architecture Search with Early Stopping https://t.co/q23tgmahi7
arxivml: "DARTS+: Improved Differentiable Architecture Search with Early Stopping", Hanwen Liang, Shifeng Zhang, Jiacheng Su… https://t.co/nu5mYdUn5N
arxiv_cs_LG: DARTS+: Improved Differentiable Architecture Search with Early Stopping. Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li https://t.co/xVowP49Ov5
BrundageBot: DARTS+: Improved Differentiable Architecture Search with Early Stopping. Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li https://t.co/GA8jSf91CR
Images
Related