A method of simultaneously optimizing both the structure of neural networks
and the connection weights in a single training loop can reduce the enormous
computational cost of neural architecture search. We focus on the probabilistic
model-based dynamic neural network structure optimization that considers the
probability distribution of structure parameters and simultaneously optimizes
both the distribution parameters and connection weights based on gradient
methods. Since the existing algorithm searches for the structures that only
minimize the training loss, this method might find overly complicated
structures. In this paper, we propose the introduction of a penalty term to
control the model complexity of obtained structures. We formulate a penalty
term using the number of weights or units and derive its analytical natural
gradient. The proposed method minimizes the objective function injected the
penalty term based on the stochastic gradient descent. We apply the proposed
method in the unit selection of a fully-connected neural...

more |
pdf
| html
None.

BrundageBot:
Controlling Model Complexity in Probabilistic Model-Based Dynamic Optimization of Neural Network Structures. Shota Saito and Shinichi Shirakawa https://t.co/w8GhTz0xwF

arxivml:
"Controlling Model Complexity in Probabilistic Model-Based Dynamic Optimization of Neural Network Structures",
Shot…
https://t.co/98Zfkk0Isu

arxiv_cs_LG:
Controlling Model Complexity in Probabilistic Model-Based Dynamic Optimization of Neural Network Structures. Shota Saito and Shinichi Shirakawa https://t.co/duGnpTx6qx

StatsPapers:
Controlling Model Complexity in Probabilistic Model-Based Dynamic Optimization of Neural Network Structures. https://t.co/TmEmENNMF3

None.

None.

Sample Sizes : None.

Authors: 2

Total Words: 0

Unqiue Words: 0

The performance of deep neural networks, such as Deep Belief Networks formed
by Restricted Boltzmann Machines (RBMs), strongly depends on their training,
which is the process of adjusting their parameters. This process can be posed
as an optimization problem over n dimensions. However, typical networks contain
tens of thousands of parameters, making this a High-Dimensional Problem (HDP).
Although different optimization methods have been employed for this goal, the
use of most of the Evolutionary Algorithms (EAs) becomes prohibitive due to
their inability to deal with HDPs. For instance, the Covariance Matrix
Adaptation Evolutionary Strategy (CMA-ES) which is regarded as one of the most
effective EAs, exhibits the enormous disadvantage of requiring $O(n^2)$ memory
and operations, making it unpractical for problems with more than a few hundred
variables. In this paper, we introduce a novel EA that requires $O(n)$
operations and memory, but delivers competitive solutions for the training
stage of RBMs with over one million variables,...

more |
pdf
| html
None.

BrundageBot:
An Evolutionary Algorithm of Linear complexity: Application to Training of Deep Neural Networks. S. Ivvan Valdez and Alfonso Rojas-Domínguez https://t.co/RqxcqIGZdQ

arxivml:
"An Evolutionary Algorithm of Linear complexity: Application to Training of Deep Neural Networks",
S． Ivvan Valdez,…
https://t.co/rWRlkRoSeO

None.

None.

Sample Sizes : None.

Authors: 2

Total Words: 0

Unqiue Words: 0

Designing evolutionary algorithms capable of uncovering highly evolvable
representations is an open challenge; such evolvability is important because it
accelerates evolution and enables fast adaptation to changing circumstances.
This paper introduces evolvability ES, an evolutionary algorithm designed to
explicitly and efficiently optimize for evolvability, i.e. the ability to
further adapt. The insight is that it is possible to derive a novel objective
in the spirit of natural evolution strategies that maximizes the diversity of
behaviors exhibited when an individual is subject to random mutations, and that
efficiently scales with computation. Experiments in 2-D and 3-D locomotion
tasks highlight the potential of evolvability ES to generate solutions with
tens of thousands of parameters that can quickly be adapted to solve different
tasks and that can productively seed further evolution. We further highlight a
connection between evolvability and a recent and popular gradient-based
meta-learning algorithm called MAML; results...

more |
pdf
| html
None.

BrundageBot:
Evolvability ES: Scalable and Direct Optimization of Evolvability. Alexander Gajewski, Jeff Clune, Kenneth O. Stanley, and Joel Lehman https://t.co/lIHq6g4Q7x

arxivml:
"Evolvability ES: Scalable and Direct Optimization of Evolvability",
Alexander Gajewski, Jeff Clune, Kenneth O． Sta…
https://t.co/fWgiUxyPoY

None.

None.

Sample Sizes : None.

Authors: 4

Total Words: 0

Unqiue Words: 0

We present a new algorithm for finding compact neural networks encoding
reinforcement learning (RL) policies. To do it, we leverage in the novel RL
setting the theory of pointer networks and ENAS-type algorithms for
combinatorial optimization of RL policies as well as recent evolution
strategies (ES) optimization methods, and propose to define the combinatorial
search space to be the the set of different edge-partitionings (colorings) into
same-weight classes. For several RL tasks, we manage to learn colorings
translating to effective policies parameterized by as few as 17 weight
parameters, providing 6x compression over state-of-the-art compact policies
based on Toeplitz matrices. We believe that our work is one of the first
attempts to propose a rigorous approach to training structured neural network
architectures for RL problems that are of interest especially in mobile
robotics with limited storage and computational resources.

more |
pdf
| html
None.

arxivml:
"Reinforcement Learning with Chromatic Networks",
Xingyou Song, Krzysztof Choromanski, Jack Parker-Holder, Yunhao T…
https://t.co/LNuMuaeCrz

SciFi:
Reinforcement Learning with Chromatic Networks. https://t.co/Vnh8rtyBtV

None.

None.

Sample Sizes : None.

Authors: 9

Total Words: 0

Unqiue Words: 0

The mammalian olfactory system learns rapidly from very few examples,
presented in unpredictable online sequences, and then recognizes these learned
odors under conditions of substantial interference without exhibiting
catastrophic forgetting. We have developed a brain-mimetic algorithm that
replicates these properties, provided that sensory inputs adhere to a common
statistical structure. However, in natural, unregulated environments, this
constraint cannot be assured. We here present a series of signal conditioning
steps, inspired by the mammalian olfactory system, that transform diverse
sensory inputs into a regularized statistical structure to which the learning
network can be tuned. This pre-processing enables a single instantiated network
to be applied to widely diverse classification tasks and datasets - here
including gas sensor data, remote sensing from spectral characteristics, and
multi-label hierarchical identification of wild species - without adjusting
network hyperparameters.

more |
pdf
| html
None.

arxivml:
"Signal Conditioning for Learning in the Wild",
Ayon Borthakur, Thomas A． Cleland
https://t.co/V9sKO8oKmt

BioPapers:
Signal Conditioning for Learning in the Wild. https://t.co/E7QCvEku5o

None.

None.

Sample Sizes : None.

Authors: 2

Total Words: 8752

Unqiue Words: 2751

Assert is a website where the best academic papers on arXiv (computer science, math, physics), bioRxiv (biology), BITSS (reproducibility), EarthArXiv (earth science), engrXiv (engineering), LawArXiv (law), PsyArXiv (psychology), SocArXiv (social science), and SportRxiv (sport research) bubble to the top each day.

Papers are scored (in real-time) based on how verifiable they are (as determined by their Github repos) and how interesting they are (based on Twitter).

To see top papers, follow us on twitter @assertpub_ (arXiv), @assert_pub (bioRxiv), and @assertpub_dev (everything else).

To see beautiful figures extracted from papers, follow us on Instagram.

*Tracking 158,360 papers.*

Sort results based on if they are interesting or reproducible.

Interesting

Reproducible