A while ago I wrote about ATAoE and why I think a layer-2-only TFTP-like protocol shouldn’t be used these days. As always, the answer to that black-and-white opinion (and I’m full of them) is “it depends” – ATAoE works great if you do it right.

A reader who wanted to stay anonymous for obvious reasons provided great feedback in a comment to my ATAoE post. Focus on the first paragraph and its message: keep the storage network totally separate and build it from scratch.

One of things that people who have not deployed ATAoE think is that it's intended to be just plugged into any old network port on any old switch you have lying around the building. That's far from the typical case - an isolated SAN attached only to provide storage. You don't have authentication on your SAS extenders, so why have it on ATAoE?

We've adopted it over iSCSI (initially running in tandem with iSCSI, but after a few months the benefits were very clearly in favour of ATAoE) for all of our (oil/gas geological) bulk storage uses for 2 years now. You could probably call it Big Data, at over 3000 disks total and 5 Petabytes of data per site.

An unplanned side effect of this is that we have now abandoned all the Microsoft storage and use Debian and FreeBSD, giving us a lot less trouble overall, which was quite a surprise here.

At the time you wrote your blog post, we were just commissioning the systems, so I read it with great interest and I worried. I have to say that in practice, we haven't hit any of the worries you raised. The speeds over iSCSI on the same architecture were a big plus and it was far simpler to set up and in use is totally transparent and hasn't given a single problem down to the protocol choice, which is more than we can say for the iSCSI half of the project, which was disappointing in comparison.

Note that we don't use Coraid, so this is purely a comment on Debian-based ATAoE itself.

The same reader provided more details in a follow-up comment, concluding with the typical vendor response to ATAoE deployment plans:

The vendor-free approach has been extremely flexible though I can see some major problems for the salesmen due to this - they were very hostile and the scare stories were getting tedious. They have gone strangely quiet now :-)

Would I recommend ATAoE based on this feedback? No, but it does prove that you can get many things to work great if you do proper planning and design (and this is the point where I have to mention that I can help you if needed).