What's Hidden in a Randomly Weighted Neural Network?
Training a neural network is synonymous with learning the values of the weights. In contrast, we demonstrate that randomly weighted neural networks contain subnetworks which achieve impressive performance without ever training the weight values. Hidden in a randomly weighted Wide ResNet-50 we show that there is a subnetwork (with random weights) that is smaller than, but matches the performance of a ResNet-34 trained on ImageNet. Not only do these "untrained subnetworks" exist, but we provide an algorithm to effectively find them. We empirically show that as randomly weighted neural networks with fixed weights grow wider and deeper, an "untrained subnetwork" approaches a network with learned weights in accuracy.
NurtureToken New!

Token crowdsale for this paper ends in

Buy Nurture Tokens

Authors

Are you an author of this paper? Check the Twitter handle we have for you is correct.

Vivek Ramanujan (add twitter)
Mitchell Wortsman (add twitter)
Aniruddha Kembhavi (add twitter)
Ali Farhadi (add twitter)
Mohammad Rastegari (add twitter)
Ask The Authors

Ask the authors of this paper a question or leave a comment.

Read it. Rate it.
#1. Which part of the paper did you read?

#2. The paper contains new data or analyses that is openly accessible?
#3. The conclusion is supported by the data and analyses?
#4. The conclusion is of scientific interest?
#5. The result is likely to lead to future research?

Github
User:
None (add)
Repo:
None (add)
Stargazers:
0
Forks:
0
Open Issues:
0
Network:
0
Subscribers:
0
Language:
None
Youtube
Link:
None (add)
Views:
0
Likes:
0
Dislikes:
0
Favorites:
0
Comments:
0
Other
Sample Sizes (N=):
Inserted:
Words Total:
Words Unique:
Source:
Abstract:
None
12/01/19 06:01PM
0
0
Tweets
morastegari: Thanks to @labs_henry for making a video describing our new paper on What's hidden in a randomly weighted neural network? https://t.co/aiYf0xPrmg https://t.co/4dx5pC7Xno @RamanujanVivek @Mitchnw
angsuman: What's Hidden in a Randomly Weighted Neural Network? https://t.co/lOMeUEGYAU
DanilBaibak: What’s hidden in a randomly weighted Neural Network? Guys found a hidden subnetwork that matches the performance of a ResNet-34, but smaller than it. Hmm, what can be found in the transformers? #NeuralNetworks #DeepLearning https://t.co/2BT2g1cToC https://t.co/WNDf2sSxxF
yavarhusayn: https://t.co/UwX5zUPprw
RamanujanVivek: Code release for "What's hidden in a randomly weighted neural network?" Code: https://t.co/qlMGSWZHWh Arxiv: https://t.co/iLykdP7U3B Discussion thread below https://t.co/ejUiIbxkD0
tanimutomo: これまじ??? https://t.co/cWcdpAtwts
JasOberoiTweets: Papers like these make my morning commute worthwhile! #MachineLearning #NeuralNetworks https://t.co/HyN2jwLhyd https://t.co/3krwMFg36C
reddit_ml: [R] What's Hidden in a Randomly Weighted Neural Network? https://t.co/9XhDY5fs6B
rezamahmooudi: مقاله ای خیلی خوبی از اقای دکتر علی فرهادی و آقای دکتر محمد رستگاری و همکارانشان توصیه میکنم حتما مطالعه کنید What's Hidden in a Randomly Weighted Neural Network? https://t.co/GUggLlD8D1 #DeepLearning #MachineLearning #artifi https://t.co/1XiqAWEJhR
kchonyc: very interesting, but also not so interesting bc (1) isn't finding a subset of a net eqiv. (almost) to training the net? (2) you sample more, you increase your chance. https://t.co/uzsMyKpBIl
hardmaru: What's Hidden in a Randomly Weighted Neural Network? “Hidden in a randomly weighted Wide ResNet-50 we show that there is a subnetwork (with random weights) that is smaller than, but matches the performance of a ResNet-34 trained on ImageNet.” 😮 https://t.co/Z4MWcXEzR8 https://t.co/4Otdlz9oct
iam_aashusingh: Interesting paper! https://t.co/988kQE82Hx
StephenPiment: What's Hidden in a Randomly Weighted Neural Network? https://t.co/sPul0reA5g
tmasada: [1911.13299] What's Hidden in a Randomly Weighted Neural Network? https://t.co/z4gDVwC5Wz
Mitchnw: @RamanujanVivek @anikembhavi @morastegari Alternate title: Randomly weighted neural networks. What do they contain? Do they contain things? Lets find out. https://t.co/szj5c2oohG
Mitchnw: What's hidden in an overparameterized neural network with random weights? If the distribution is properly scaled (e.g. Kaiming Normal), then it contains a subnetwork which achieves high accuracy without ever modifying the values of the weights... https://t.co/szj5c2oohG (/n) https://t.co/RcTcgYGY9J
arxivml: "What's Hidden in a Randomly Weighted Neural Network?", Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali… https://t.co/3nkeV9PDUa
Memoirs: What's Hidden in a Randomly Weighted Neural Network?. https://t.co/uJyJ69Jkrh
HirokatuKataoka: RT @arxiv_cscv: What's Hidden in a Randomly Weighted Neural Network? https://t.co/bIaTEHUF6l
arxiv_cscv: What's Hidden in a Randomly Weighted Neural Network? https://t.co/bIaTEHUF6l
hiro6391: ランダムニューラルネットの一部をうまく選ぶと学習したモデルに近い精度が出る…? "What's Hidden in a Randomly Weighted Neural Network?" https://t.co/YS9D0l5TMS
arxiv_cs_LG: What's Hidden in a Randomly Weighted Neural Network?. Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari https://t.co/kfvHiAnAp0
Images