Well, metallic sodium liberates hydrogen real fast on contact with water, which I guess is tantamount to the same thing.

Yes. But not to the same level as just dropping a brick of pure sodium in a bathtub. In a battery like this there is not pure lithium/sodium/whatever just sloshing around inside. The sodium is tied up being chemically bonded with whatever the anode and cathode materials are. Only a minority of the available sodium is actually free in the form of ions carrying the charge from cathode to anode.

Just as with lithium-ion chemistry batteries, it is vital that the cells remain sealed from the outside because the materials inside will indeed react with air, water, and the water in the air. Exposing the innards will cause a rapid exothermic reaction, i.e. it will get very hot and optionally go off bang.