Excited to share "Dense Associative Memory through the Lens of Random Features" accepted to #neurips2024🎉
DenseAMs need new weights for each stored pattern–hurting scalability. Kernel methods let us add memories without adding weights!
Distributed memory for DenseAMs, unlocked🔓
DenseAMs need new weights for each stored pattern–hurting scalability. Kernel methods let us add memories without adding weights!
Distributed memory for DenseAMs, unlocked🔓
Comments
In traditional Memory representations of DenseAMs (MrDAM) one row in the weight matrix stores one pattern. In our new Distributed representation (DrDAM) patterns are entangled via superposition, “distributed” across all dims of a featurized memory vector
1️⃣A similarity func between stored patterns & noisy input
2️⃣A rapidly growing separation func (e.g., exponential)
Together, they reveal kernels (e.g., RBF) that can be approximated via the kernel trick & random features (Rahimi&Recht, 2007)
1️⃣Far from memories
2️⃣“Spiky” (i.e., low temperature/high beta)
We need more random features Y to reconstruct highly occluded/correlated data!
Want to learn more?
📜Paper https://arxiv.org/abs/2410.24153
💾Code https://github.com/bhoov/distributed_DAM
👨🏫NeurIPS Page: https://neurips.cc/virtual/2024/poster/96886
🎥SlidesLive (use Chrome) https://recorder-v3.slideslive.com/#/share?share=97038&s=3c7ede33-220c-4c28-a72e-86835e88c4db