You are not using the proper tools for your need.
To do HA here, you need to add something like KeepAlived. Install it on all of your manager nodes. Once configured, it will create a VIP that will move from one manager to the other according to which one is up and marked as the favorite one. You then point your dns clients to that VIP instead of the actual IP of any of your manager.
HAProxy can do some load balancing as well (TCP or HTTP) but here, Keepalived is what you need.