I am reading 'attention is not all you need' https://arxiv.org/abs/2103.03404
I had read this paper in the past but felt the need to refresh my memory and look at self attention with mildly critical lens. Afaik, this paper talks about attention networks without surrounding structures like MLP, skip connections etc and its behaviour.
Reading “Low-Resource” Text Classification: A Parameter-Free Classification Method with Compressors
https://aclanthology.org/2023.findings-acl.426/
Not very familiar with information theory so it's a nice read, and also just a very clever solution
actually doing some testing using it for digit classification on MNIST just for fun. Might share code/results if people are interested
For sure, sounds interesting !