High-speed scale-out load balancing is a Mission Impossible. You can get the correct abstraction at the wrong cost or another layer of indirection (to paraphrase the authors of Fastly load balancing solution).

However, once every third blue moon you might get a team of smart engineers focused on optimal solutions to real-life problems. The result: a layer of misdirection, a combination of hardware ECMP and server-level traffic redirection. Enjoy!