Why does #compneuro need new learning methods? ANN models are usually trained with Gradient Descent (GD), which violates biological realities like Dale’s law and log-normal weights. Here we describe a superior learning algorithm for comp neuro: Exponentiated Gradients (EG)! 1/12 #neuroscience 🧪 - ThreadSky

repromancer.bsky.social • 121 days ago

Why does #compneuro need new learning methods? ANN models are usually trained with Gradient Descent (GD), which violates biological realities like Dale’s law and log-normal weights. Here we describe a superior learning algorithm for comp neuro: Exponentiated Gradients (EG)! 1/12 #neuroscience 🧪

Comments

matttkk.bsky.social•119 days ago

Dale’s “law” isn’t a law, btw - neurons have been documented releasing GABA, glycine and glutamate at the same terminals

repromancer.bsky.social•118 days ago

Yes, though I'm only aware of co-packaging of GABA with glu in lateral habenula (e.g. Kim et al. 2022), and I don't think it has been found in the cortex? I like these results as I think they indicate that a mixed weight solution isn't evolutionarily far away, suggesting a purpose of the EI split.

haesemeyerlab.bsky.social•121 days ago

This is neat! I assume that like GD, EG will still rely on non-local information to train a specific connection?

repromancer.bsky.social•121 days ago

Thanks! Yes, both rely on gradients of a loss function with respect to a specific connection being available.

repromancer.bsky.social•121 days ago

EG respects neural biology better than GD 🧠 Unlike GD, which often flips synaptic weights between excitatory and inhibitory, EG adheres to Dale’s law, keeping weights positive and aligned with biology. 2/12 #NeuroAI

repromancer.bsky.social•121 days ago

Log-normal synaptic weights: EG gets it right! 🌱 Trained with EG, networks naturally develop log-normal weight distributions observed in biological brains. This weight distribution stability contrasts with GD, which drifts from log-normal forms post-training. 3/12

repromancer.bsky.social•121 days ago

Resilience to synaptic pruning EG-trained networks maintain accuracy better than GD when synapses are pruned—a critical process in real neural networks during development and sleep. 4/12 #LearningAlgorithms #Neuroplasticity

repromancer.bsky.social•121 days ago

Faster learning, less retraining 🏃 When synapses are pruned and then relearning is required, EG networks adapt more quickly and efficiently than GD networks, showing better resilience and learning stability. 5/12 #MachineLearning

repromancer.bsky.social•121 days ago

Learning from noisy inputs. In tasks requiring selective focus, EG-trained networks ignore irrelevant signals more effectively than GD-trained ones, excelling at biologically relevant sensorimotor control. 6/12 #AI #Neuroscience

repromancer.bsky.social•121 days ago

Sparse and relevant inputs. EG outperforms GD in scenarios with many irrelevant signals by weighting relevant inputs more heavily. This leads to faster, smoother learning and aligns better with how neurons handle background noise. 7/12 #SparseCoding

neurograce.bsky.social•121 days ago

This requires initializing neurons to have all positive or all negative output weights before training, right?

repromancer.bsky.social•121 days ago

Not for EG, but yes you would need that for Dale's law. We didn't initialise with E/I seperation in the paper to keep the experiments as straightforward and close to "standard" ANNs as possible. You totally can though (and we have) and EG will maintain the E/I separation.

wattenberg.bsky.social•121 days ago

Is EG something like gradient descent with respect to a Riemannian metric of the form diag(1 / |x_i|)?

repromancer.bsky.social•120 days ago

Yes, if you linearise the update, so that is an interpretation for small learning rates.

wattenberg.bsky.social•120 days ago

Neat! Speculatively, is there any sense in which the "difficulty" of changing a biological synaptic weight depends on the proportional change rather than absolute change? If so, would that give an intuition for why this metric is natural?

repromancer.bsky.social•120 days ago

So, I don’t think we know biophysically why the changes are proportional in the brain, but this is indeed an intuition behind why EG matches the experimental result we quote here: https://bsky.app/profile/repromancer.bsky.social/post/3l7ln4hojvr2z

And this metric interpretation is also a way to see why the weights don’t cross zero!

Comments

Posting Rules

Reply