Yep.

I'd still call that memory. It's not the present; arguably for a (post-training) LLM the present totally consists of choosing probabilities for the next token, and there is no notion of future. That's really just a choice of interpretation, though.

During training they definitely can learn and remember things (or at least "learn" and "remember"). Sometimes despite our best efforts, because we don't really want them to know a real, non-celebrity person's information. Training ends before the consumer uses the thing though, and it's kind of like we're running a coma patient after that.