1/ Okay, one thing that has been revealed to me from the replies to this is that many people don't know (or refuse to recognize) the following fact: The unts in ANN are actually not a terrible approximation of how real neurons work! A tiny 🧵. 🧠📈 #NeuroAI #MLSky - ThreadSky

tyrellturing.bsky.social • 72 days ago

1/ Okay, one thing that has been revealed to me from the replies to this is that many people don't know (or refuse to recognize) the following fact:

The unts in ANN are actually not a terrible approximation of how real neurons work!

A tiny 🧵.

🧠📈 #NeuroAI #MLSky

Reposted from Blake Richards

Why does anyone have any issue with this?

I've seen people suggesting it's problematic, that neuroscientists won't like it, and so on.

But, I literally don't see why this is problematic...

Comments

jeffreybowers.bsky.social•72 days ago

By adopting rate based coding researchers ignore the dimension of time in learning and computation. Eg spike-time dependent plasticity. Or adaptively modifying time constants and delay lines that can can improve performance and help explain neural diversity. https://scholar.google.com/citations?view_op=view_citation&hl=en&user=KADNEXcAAAAJ&sortby=pubdate&citation_for_view=KADNEXcAAAAJ:XvxMoLDsR5gC

tyrellturing.bsky.social•71 days ago

Absolutely, this is a topic I myself have studied:

https://www.nature.com/articles/s42003-021-02437-y

And that's why it's important to clarify that it's an *approximation*, one adopted because it makes more math possible.

jeffreybowers.bsky.social•71 days ago

The question comes down to whether spike timing important. Without spikes difficult to understand STP, or learning of temporal parameters, or represent similarity/alignment in the domain of time. But if timing not so important than may well be a reasonable approximation.

tonyzador.bsky.social•72 days ago

100%👍

suryaganguli.bsky.social•72 days ago

There is also this one: https://www.sciencedirect.com/science/article/pii/S0896627321005018

tmoldwin.bsky.social•72 days ago

Everyone loves citing @davidbeniaguev.bsky.social 's paper but not mine, SAD.

jbarbosa.org•72 days ago

Could you share it here please?

tmoldwin.bsky.social•72 days ago

in the top level thread

nosrat.bsky.social•72 days ago

Your's is a nice reference for the exact opposite question! ;)
(Can we approximate ANN by brain?).

tmoldwin.bsky.social•72 days ago

I wouldn't say it's the opposite question, if there is an equivalence it should be bidirectional.

nosrat.bsky.social•72 days ago

Yes, maybe bidirectional is a better description. Anyway, thanks for sharing this paper!

alxfed.bsky.social•72 days ago

_Nothing_ is linear in a _live_ neuron, no matter how loud the people who can only multiply matrices and think that they understand real phenomena will shout. Relax, go back to your k̶n̶i̶t̶t̶i̶n̶g̶ number joggling, that doesn't reflect anything real. Leave humans alone.

tyrellturing.bsky.social•71 days ago

lol... The integration of currents in the RC-circuit provided by the membrane is linear. Learn some biophysics man...

alxfed.bsky.social•71 days ago

You are calling the accumulation of ions on one side of a membrane an 'RC-circuit'? Yes, that, indeed, is very funny. And it is about the same - you consider your _metaphors_ of phenomena to be real. They are not! By the way 'RC-circuit' charging is linear in one an only point t = 0. Ha-ha-ha!

tyrellturing.bsky.social•71 days ago

It's an equivalent circuit, of course. (Didn't think I had to explain that.) This is one of the most well-established results in neuroscience:

https://neurotext.library.stonybrook.edu/C3/C3_3/C3_3.html#:~:text=The%20simplest%20equivalent%20circuit%20for,single%20capacitor%20(Cm).

And the derivative of the membrane potential is a linear function of the current at *every* time-step.

alxfed.bsky.social•71 days ago

I'm not continuing this conversation. Thank you for your participation.

dominicstandage.bsky.social•72 days ago

Yeah, but they don’t leak, so they have no meaningful temporal dynamics. And their ‘synapses’ have no temporal dynamics either. Same holds for rate models, ie they leak too. And then there are exc/inh interactions, which also matter

dlevenstein.bsky.social•72 days ago

It’s quite easy to make ANN units leak, and add E/I interactions.

I think people confuse the specific way ANNs tend to be formulated in ML applications with ANNs as a general modeling approach?

dominicstandage.bsky.social•72 days ago

Terminology def gets in the way. Can you sent me a citation? Something like Wilson-Cowan E/I population nodes, trained by backprop?

dlevenstein.bsky.social•71 days ago

Here’s one way to E/I in ANNs (follow-up work has used recurrent as well as feedforward DANNs): https://openreview.net/forum?id=eU776ZYxEpz

And this review includes leaky units (Eqn 26), as well as another way to E/I (Eqn 35): https://pmc.ncbi.nlm.nih.gov/articles/PMC11576090/

dlevenstein.bsky.social•71 days ago

…and in my pinned thread we have ANN units with adaptation, which is formulated exactly how you would in an adapting Wilson-Cowan model (e.g. https://www.nature.com/articles/s41467-019-10327-5, one of my favorite neuron-like properties to study in ANNs 😉)

huwjarvis.net•72 days ago

📌

tjanicek.bsky.social•72 days ago

📌

hakwan.bsky.social•72 days ago

i'm w/ you here, Blake. qualified simplifications are often necessary to make things useful.

but do u know of ANN models that don't assume the neurons are all identical, but add a simple diversity/randomness factor?

i was thinking of things like this - https://symposium.cshlp.org/content/83/45.full.pdf?utm_source=chatgpt.com

fleurzeldenrust.bsky.social•72 days ago

https://www.pnas.org/doi/10.1073/pnas.2311885121

fleurzeldenrust.bsky.social•72 days ago

https://pubmed.ncbi.nlm.nih.gov/34608134/

fleurzeldenrust.bsky.social•72 days ago

Many interesting studies on heterogeneity…admittedly not in classical feed forward rate networks

willjharrison.bsky.social•71 days ago

This is cool. Thanks for sharing.

dlbarack.bsky.social•72 days ago

I don't see them as simplifications at all. ANNs are just constructed model systems with quite different realizations. Nonetheless, because they are samples from the same universality class, we can study either brains or ANNs to learn about the class. Think of them like model organisms

hakwan.bsky.social•72 days ago

yes, sorry i was simplifying the distinction between simplications and models. 😅

tyrellturing.bsky.social•71 days ago

I sure do! 🙂

Excuse me while I engage in a bit of self-promotion 😅:

https://www.biorxiv.org/content/10.1101/2024.08.07.606541v1.abstract

tomnotgeorge.bsky.social•71 days ago

and there's this nice work by @neuralreckoning.bsky.social ! https://www.nature.com/articles/s41467-021-26022-3

neuralreckoning.bsky.social•71 days ago

Thanks Tom!

neuralreckoning.bsky.social•72 days ago

Speaking for myself I use rate based abstractions in some of my work and it captures some things we're interested in. But I don't like dogmatism that rate based models capture everything of interest. Unproven and a priori unlikely. Likely that lots of relevant spike timing stuff happens at pop level

tyrellturing.bsky.social•72 days ago

As you know from our discussions of spike timing, I agree with you! 🙂 I would never claim rate-based models capture everything of interest.

But, I'm always confused by people who suggest rate-based units are "nothing like" real neurons, cause I think it's an obviously false statement.

neuralreckoning.bsky.social•72 days ago

I'm happy to go along with that! 🙂

gunnarblohm.bsky.social•72 days ago

I would love to see some research directly comparing ANN vs SNN properties... Especially how different neuron types / cortical layers properties compare to ANN unit properties. It seems totally unclear what the potential (mis-)match is and where the abstraction holds vs breaks

idoai.bsky.social•71 days ago

Here's a preprint that might answer your question:
https://www.biorxiv.org/content/10.1101/2024.12.17.628883v1

We explored how different neuronal properties across neuron types, cortical layers, and species translate into different functional complexities, using ANNs as I/O complexity measurements.

virati.bsky.social•72 days ago

A model that perfectly captures how an unbroken system transforms inputs into outputs does not make that model a good "approximation" of that system, *especially* when we're very often interested (and funding research to billions) that system when it's broken.

More here: https://medium.com/@virati/failure-modes-and-models-3d84020982dd

arnauya.bsky.social•72 days ago

also here!!!
Beniaguev, D., Segev, I., & London, M. (2021). Single cortical neurons as deep artificial neural networks. Neuron, 109(17), 2727-2739.

nsaphra.bsky.social•72 days ago

🕯
🕯 🕯
🕯 🕯

🕯 @ilennaj.bsky.social 🕯

🕯 🕯
🕯 🕯
🕯

jbimaknee.bsky.social•72 days ago

I don't have a problem with the first part. The second part is what is problematic, because it betrays a bias of the ML field.

Yes, the brain is a giant neural network, but *not* at all in the way NeurIPS thinks of neural networks. The brain is not a giant ANN of generic fully connected layers.

greybe.bsky.social•72 days ago

@jbimaknee.bsky.social That moment when you realize your ANN is just a glorified game of connect-the-dots, while your brain is out here multitasking like a pro with a cocktail of layers and feedback loops 🍹 #NeuroAI #DeepLearningReality

howard.fm•72 days ago

Nor is nearly any modern deep learning net though...

guyi.bsky.social•71 days ago

Yes but the macro structures in modern networks are still nothing like macro structures in the brain.

polavieja.bsky.social•72 days ago

Yes, a beautiful simple model of a neuron. It is actually quite interesting
how well the M-Pitts or DL abstraction has survived later models, as you argue. I would think, however, that its relevance for different neuroscientists depends on what they want to explain.

dlevenstein.bsky.social•72 days ago

I see neurons are units is your new brain is a computer 😉

tyrellturing.bsky.social•72 days ago

2/ First, let's start with the obvious: real neurons integrate their inputs. If you start from the basic principles of the relationship between voltage and current, and you know synapses induce currents when they receive neurotransmitter, then this is obvious.

tyrellturing.bsky.social•72 days ago

3/ Second, even though the true Hodgkin & Huxley model is far more complex, it's not an unreasonable approximation to model neurons as having a spike "threshold", with spikes occuring when threshold is reached.

tyrellturing.bsky.social•72 days ago

4/ That's why leaky-integrate-and-fire models are considered decent approximations of real neurons, and have been a major tool in the computational neuroscience tool belt for decades.

https://compneuro.neuromatch.io/tutorials/W2D3_BiologicalNeuronModels/student/W2D3_Tutorial1.html

tyrellturing.bsky.social•72 days ago

5/ We can make this simpler, and treat the spike generation as a Poisson process, following linear integration and a non-linear activation function. These "linear-non-linear" models approximate neurons quite well! See, e.g., this work from @jpillowtime.bsky.social :

https://direct.mit.edu/neco/article-abstract/29/12/3260/8316/Capturing-the-Dynamical-Repertoire-of-Single

tyrellturing.bsky.social•72 days ago

6/ But, spikes can be frustratingly hard to analyze formally. So, that's why early-on connectionists relied instead on "rate-based" models of neurons, where we integrate and use a nonlinearity to calculate a rate-of-fire. It's not as accurate, but still a reasonable approximation.

tyrellturing.bsky.social•72 days ago

7/ Of course, real neurons are more complex than single compartments! For example, @yiotapoirazi.bsky.social showed very nicely that accurately capturing real neurons requires multiple stages of linear integration followed by a non-linearity.

https://www.cell.com/AJHG/fulltext/S0896-6273(03)00149-1

neurograce.bsky.social•72 days ago

You can even derive a rate equation from the LIF model equations! So rate models aren't just post hoc approximations

mirceasci.bsky.social•72 days ago

Some argue that bursting encodes as much of the signal as single spikes, right? LIF doesn't have bursting. Also, depending on which part of the phase space a HH neuron is at a given time, an input signal can change from excitatory to inhibitory, which LIF can't. Are we simplifying too much with LIF?

tyrellturing.bsky.social•72 days ago

Well, no - because I don't think there's a *single* level we need to sit at.

We should do both more detailed models and more abstract models. Each has their use!

mirceasci.bsky.social•71 days ago

true, I know.. just wanted to leave that comment up there in case some bypasser was wondering what are the differences

tmoldwin.bsky.social•72 days ago

This isn't Mastodon so I can't make a joke about tooting my own horn, but...

https://www.frontiersin.org/journals/computational-neuroscience/articles/10.3389/fncom.2020.00033/full

rory.bio•70 days ago

But synaptic currents must be propagated by active conductances which do weird stuff like flip the input's sign (PIR), or only propagate inputs when state is in a goldilocks zone (Gidon et al 2020), etc.

Gating and sign inversion are powerful effects, surely they matter...

https://bsky.app/profile/tyrellturing.bsky.social/post/3ldh4plptb42a

tyrellturing.bsky.social•70 days ago

Sometimes they matter. It really depends on how far you are from the reversal potential.

For glutamatergic synapses, it probably doesn't matter much, because the reversal is very far from where the neuron usually is.

For GABAergic synapses, though, yes, it surely matters!

rory.bio•70 days ago

I'm talking about the voltage-gated conductances that carry signal from the synapse down to the soma/axon, e.g. T-Type Calcium. They allow an inhibitory synaptic current to effectively excite the axon.

tyrellturing.bsky.social•70 days ago

Yes, of course, there are many additional complexities this way as well.

But, as the data I presented in the thread shows, these complexities are more about the precise behaviour of the neuron.

The broad, rate-based, IO function is reasonably well described using a linear integration step.

mountaingeek76.bsky.social•72 days ago

It's obvious that ANNs are approximations of biological sets of neurons. Personally I think the more interesting question is whether Transformer/Mamba slices are ~equivalent to cortical columns.

maharrit.bsky.social•71 days ago

"The unts in ANN are actually not a terrible approximation of how real neurons work!"

This is a very weak defense of the analogy. You're being convinced by the marketing term 'neural net'.

ANN nodes are also not very good approximations of real neurons.

tyrellturing.bsky.social•71 days ago

1. I haven’t been convinced by “marketing”, I’ve been convinced by scientific theory and evidence, as outlined in the thread you clearly didn’t read.

2. It was never “marketing”. ANNs were originally invented as models of the brain, so they were named after the very thing they were trying to model.

maharrit.bsky.social•71 days ago

My apologies for using 'marketing'. So much of the vocabulary around artificial intelligence is aspirational. Neural networks were definitely labeled that being an attempt to model networks of neurons.

maharrit.bsky.social•71 days ago

As to not reading the thread, I read it several times but maybe I didn't understand your point.

maharrit.bsky.social•71 days ago

Your thread makes a claim that neural networks (the math) are a 'not terrible' model for networks of neurons (the biology).

maharrit.bsky.social•71 days ago

It may very well be a 'not terrible' model at the cellular level (I am not a biologist), but it doesn't seem to be a -good- model at larger scales, C. elegans and larger nets.

dylanmuir.bsky.social•72 days ago

Sure, non-spiking models capture some of the dynamics of cortical neurons. So why don’t we already have working simulations of brains? Because the *architecture* is crucial — that’s what’s missing from the NeuroAI ANN approach (and Ilya’s slide).

dylanmuir.bsky.social•72 days ago

Every five years, someone’s newly amazed that learning the structure of visual scenes gives you something that looks like receptive fields 🙄

PCA is also not a functional model for V1.

dylanmuir.bsky.social•72 days ago

Fitting a universal function approximator to data from V1 is not going to capture the mechanistic architecture of V1.

Comments

Posting Rules

Reply