TL&DR: No. You can move on.

NANOG87 summary by John Kristoff prompted me to look at NANOG87 presentations, and one of them discussed ChatGPT and Network Engineering (video). I couldn’t resist the clickbait ;)

Like most using ChatGPT for something articles we’re seeing these days, the presentation is a bit too positive for my taste. After all, it’s all fine and dandy to claim ChatGPT generates working router configurations and related Jinja2 templates if you know what the correct configurations should look like and can confidently say “and this is where it made a mistake” afterwards.

Could someone who does not know the technology or device configuration details use ChatGPT to create a device configuration? Of course they could, but I wouldn’t¹. It’s even worse than pasting random configurations found in blog posts² or StackOverflow into your production gear.

Could you use it to generate configurations for a box you never worked with before? Please do, you might greatly improve your troubleshooting skills ;)

Anyway, after claiming that…

ChatGPT is aware of vendor specific syntax and can generate configurations

… which is technically true unless you claim it can generate working or correct configurations, the presentation moved on to solving troubleshooting issues based on Reddit questions, resulting in:

ChatGPT can potentially help with simple routing troubleshooting, but not with this problem.

Fair enough. On the next question (STP between Juniper and Fortinet) the conclusion was:

ChatGPT understands STP better than many networking professionals I’ve interviewed over the years.

Let’s focus on the understand part of the conclusion. While Sabine Hossenfelder recently argued that chatbots partly understand what they chat about that turned out to be another clickbait – the way I understood her video, she was effectively saying “we don’t have a good definition of what understand means, so whatever.” In the ChatGPT-versus-STP case, I’d say the correct conclusion would be:

ChatGPT can process more source materials and has better regurgitating skills than many networking professionals.

Unfortunately, the ChatGPT understands STP conclusion lead the authors to a follow-up conclusion:

ChatGPT might result in the obsolescence of many subreddits and Stack Overflow

Well, no. The only reason ChatGPT can make a statement about STP (or any other technology) that sounds about right is because it processed source material talking about those topics. Someone had to generate those materials, and if we assume ChatGPT could generate the results it does from vendor documentation, then it’s fair to ask the next question: do we use Stack Overflow just because we’re too lazy to read vendor documentation?

Sadly, as anyone with decent googling skills who had to turn to Stack Overflow can testify, the reason we’re using Stack Overflow as much as we do is because (networking) vendor product documentation focuses on what vendors want you to know: how to configure their product. It often lacks usage guidelines, vendor interoperability, undocumented quirks, gotchas, unintended use cases… not to mention the discussions of how things work. If ChatGPT generates a great answer to a question you’d usually ask on Stack Overflow, it might be because it’s processed numerous related answers humans wrote on Stack Overflow ;) Does that make Stack Overflow obsolete? There seems to be a circular dependency somewhere in this paragraph.

Another benefit of finding a “solution” on Stack Overflow³ is voting and feedback, including gems like “this seems great but it didn’t work for me” or “while your solution works, that’s not how things should be done.” Not exactly something I expect to get from a large language model. After all, the caveats represent a tiny percentage of its training materials.

Tim Bray nailed it in a recent blog post:

On the Internet, the truth is paywalled and the bullshit is free. And as just discussed, one of the problems with LLMs is that they’re expensive. Another is that they’re being built by capitalists. Given the choice between expensive quality ingredients and free bullshit, guess which they’ll pick?

Does that make ChatGPT useless? Of course not. I love using it to create introductory paragraphs for my technology resources pages (example), obviously with full attribution – it’s only fair to tell you it wasn’t me drastically improving my writing skills. I would also love to use something like ChatGPT as a friendlier search engine that would understand (here we go again) at least some of my intent⁴, offer some generic guidelines, and point me to the most appropriate source material. According to Tim Bray, Bing’s headed in that direction, but it seems like the Bing AI extensions work only with Microsoft Edge (or it’s yet again me being too stupid to use it), so whatever.