The size of the internal wire, solder connection, strain relief, and especially the cable shield size are all factors.
The shield is most critical if you look at the length of wire an miniscule power of any instrument without a powered preamp. Even with a built in preamp the output impedance will be high from most circuits.
Think of it this way, high impedance is another way of saying there is a voltage signal but not much current is able to flow to or from the device. If you try to pull or push too much current the signal will disappear. When there is not much current flowing, the signal is much more susceptible to other signals and noise crossing the wire.
Most 3.5mm audio connectors have poor shielding, strain relief, and the actual connection points where the wires are soldered are terrible. With the way they are constructed, the solder connection must be done very quickly to avoid damaging the thin plastic insulation between the rings that make up the tip terminal. With the larger quarter inch connector, there is a lot more heat mass in the actual terminals and there is enough room to make solder terminals with heat isolation. This helps to match the terminal with a larger wire gage so that both surfaces can evenly wet with solder with a properly set iron temperature. In theory this leads to a far more robust connection.
Most 3.5mm cables are unshielded. This is fine for the low impedance (high current flow) of an amplifier output stage, but it is totally insufficient for the high impedance input of an instrument.
This is why instrument cables generally cost so much more too. You're buying more copper, an engineered cable that has more that just wires in an extruded plastic sleeve, and the connectors are special purpose, beefier, and more engineered for a specialty task.
