Thanks Kevin! And thanks to my collaborators @alexkoulakov.bsky.social Sergey Shuvaev and Divyansha Lachi ! (S & D -- have you joined the party here yet?) - ThreadSky

tonyzador.bsky.social • 90 days ago

Thanks Kevin!

And thanks to my collaborators @alexkoulakov.bsky.social
Sergey Shuvaev and Divyansha Lachi !

(S & D -- have you joined the party here yet?)

Reposted from Kevin Mitchell

A wonderful paper from @tonyzador.bsky.social and colleagues: Encoding innate ability through a genomic bottleneck www.pnas.org/doi/10.1073/...

Comments

turrigiano.bsky.social•90 days ago

very interesting and relevant to things I have been thinking about (innate behaviors with learned components)

nurikolan.bsky.social•89 days ago

Hi, interesting work. Did you try limiting the precision of the "compressed" network? Number of params is a very crude proxy for the actual information capacity.

See e.g. https://aclanthology.org/2024.acl-long.713/ and https://doi.org/10.1162/tacl_a_00489

and very similar work by Gaier & Ha 2019 http://arxiv.org/abs/1906.04358

tonyzador.bsky.social•89 days ago

I guess the question you're raising is whether it would be harder to limit the precision of the compressed network?

Interesting question...i suspect not, bcs i think we are finding very different solutions whereas reducing precision finds slightly perturbed solutions (i think)

nurikolan.bsky.social•89 days ago

Sorry, missed the G&H ref!

Re: precision I meant that there's a chance that the "compressed" network is in practice not a bottleneck if it can store the same/approx. solution as the larger net in high precision weights.
Would be interesting to see how it does when quantized (either at test/train)

tonyzador.bsky.social•89 days ago

we kinda know that it's doing better than quantization bcs we can compress some networks x1000-fold vs 16-32-fold w/quant

also i think quant is finding a solution very near W* (original trained network) whereas our compression goes far from W*

tonyzador.bsky.social•89 days ago

Thanks for the ref to your work--looks very interesting and relevant!

Apologies for not having been aware (or cited it)...our first draft of this paper was posted in 2020 and it took us 4 years to finally publish

(we do cite Gaier & Ha, 2019--ref #38).

david-g-clark.bsky.social•90 days ago

Cool!
In your framework, does random connectivity have long/maximal genomic description length (a la Kolmogorov)? My guess would've been that one of the important 'rules' in wiring a nervous system, or pretrained ANN, is to wire (somewhat) randomly (e.g., in the olfactory system).

tonyzador.bsky.social•90 days ago

that said, olfactory wiring turns out to be less random than previously believed

https://www.cell.com/cell/fulltext/S0092-8674(22)01257-0

tonyzador.bsky.social•90 days ago

I definitely believe randomness plays a role. The genome helps shape the distributions of wiring diagrams!

We explore this in a model with @tyrellturing.bsky.social et al

https://www.biorxiv.org/content/10.1101/2024.08.07.606541v1

david-g-clark.bsky.social•90 days ago

Makes sense!
Does it then make more sense to have the g-network output parameters of a prob. distr. from which the connectome is sampled? In the current formulation, it seems like it would be forced to memorize a particular realization of random noise, giving overly pessimistic compression lengths?

tonyzador.bsky.social•90 days ago

Yes the g-network formulation is purely deterministic

The cell type version has stochasticity

Working on a version that actually follows a developmental program

tyrellturing.bsky.social•90 days ago

Yup, it would be crazy if randomness doesn't play a role, in fact!

arula-ratnakar.bsky.social•90 days ago

I know Dscam isoforms are specific to each neuron & allow neurons to avoid autapses (don’t connect to neuron w/ same isoform) and instead form synapses mostly onto “non self” neurons— do you think families of neurons w/ similar isoforms are more mutually repulsive to synapse formation in the family?

tonyzador.bsky.social•90 days ago

sadly i think the mechanism for wiring up cell types is not the same in mammals as in flies

arula-ratnakar.bsky.social•90 days ago

Ah, alright, that answers that I guess. It’s a really interesting problem!

tyrellturing.bsky.social•90 days ago

I would guess it depends on the cell type?

arula-ratnakar.bsky.social•90 days ago

That makes sense. I’m wondering if cell type A more likely to connect to cell type B is because of larger difference in “self marker” isoforms between A and B, and if cell types likely to connect to the same cell type have larger variability in “self markers” within the cell type

neuroai.bsky.social•90 days ago

Very cool. I’ve recently been making some evolutionary models, replacing hill climbing with homeostasis/mutation in an ecology w dynamic resources. Hope to get at the evolution of learning/memory with a non-maximization & more bio pov. Excited about your paper! would be fun to chat about all these.

tonyzador.bsky.social•90 days ago

would love to chat!

trackingactions.bsky.social•90 days ago

Ah this is brilliant work! I’ll immediately add it into my white paper on adaptive intelligence 🙏🏼

mameister4.bsky.social•90 days ago

"The feedback in our algorithm that guides the gradient from each generation can be viewed as a form of Lamarckian evolution, and is, as such, biologically unrealistic." Examples exist in unicellular organisms, and one can at least imagine mechanisms in metazoans: https://markusmeister.com/2019/04/19/irresponsible-musings-on-lamarckian-evolution/

mishaahrens.bsky.social•87 days ago

This is so cool!!

Speaking of genomic bottlenecks, Beamrider is the first computer game I ever played, on my father’s MSX computer

Comments

Posting Rules

Reply