NATTEN just added fused support for self-cross attention!
so you can attend to local neighbourhood and registers or text condition.
it lets you reduce partial attention results (e.g. logsumexp provided by xformers APIs) into its LSE.
https://github.com/SHI-Labs/NATTEN/pull/182
so you can attend to local neighbourhood and registers or text condition.
it lets you reduce partial attention results (e.g. logsumexp provided by xformers APIs) into its LSE.
https://github.com/SHI-Labs/NATTEN/pull/182
Comments