Not sure about JS, but from some cursory experimentation there don't seem to be any limitations on what can be done with the custom CSS. It seems to just load whatever CSS code you put there into a style block at the top of the finalized page source, so anything goes really. I wouldn't be surprised to see limitations on this in the future since this could obviously lead to some annoyances from making the entire page invisible to overlaying a seizure gif onto everything.
As far as documentation I don't know of anything official, but /m/kbinStyles seems like your best bet for asking questions, alongside using inspect element to identify the elements you want to style while consulting a CSS reference like w3schools. Also iirc, browser extensions like Stylus let you create custom themes for sites while previewing them on the fly which would be pretty helpful here. As long as you're not trying to do anything crazy complex you could probably learn what you need to style a magazine in an afternoon!