I must be a good prompt engineer – every time I ask ChatGPT something really simple it spews out nonsense. This time I asked it to build a small network with four routers:

I have a network with four Cisco routers (A,B,C,D). They are connected as follow: A-B, B-C, A-D, D-C. Each router has a loopback interface. Create router configurations that will result in A being able to reach loopback interfaces of all other routers.

Here’s what I got back¹:

Here’s an example configuration for the four routers that should allow Router A to reach the loopback interfaces of all other routers:

Maybe it’s just me, but I would expect an expert to ask a question or two first, like “are there any particular IP address you’d like to use, or is it all the same to you”. Ignoring that bedside manners detail, let’s inspect the configuration:

interface FastEthernet0/0
 ip address 192.168.1.1 255.255.255.0
 duplex auto
 speed auto
!
interface Loopback0
 ip address 10.0.0.1 255.255.255.0
!
router ospf 1
 network 192.168.1.0 0.0.0.255 area 0
 network 10.0.0.0 0.0.0.255 area 0
!
ip route 0.0.0.0 0.0.0.0 192.168.1.2

interface FastEthernet0/0
 ip address 192.168.1.2 255.255.255.0
 duplex auto
 speed auto
!
interface FastEthernet0/1
 ip address 192.168.2.1 255.255.255.0
 duplex auto
 speed auto
!
router ospf 1
 network 192.168.1.0 0.0.0.255 area 0
 network 192.168.2.0 0.0.0.255 area 0
!
ip route 0.0.0.0 0.0.0.0 192.168.2.2

interface FastEthernet0/0
 ip address 192.168.2.2 255.255.255.0
 duplex auto
 speed auto
!
interface Loopback0
 ip address 10.0.0.3 255.255.255.0
!
router ospf 1
 network 192.168.2.0 0.0.0.255 area 0
 network 10.0.0.0 0.0.0.255 area 0
!
ip route 0.0.0.0 0.0.0.0 192.168.2.1

interface FastEthernet0/0
 ip address 192.168.3.1 255.255.255.0
 duplex auto
 speed auto
!
interface Loopback0
 ip address 10.0.0.2 255.255.255.0
!
router ospf 1
 network 192.168.3.0 0.0.0.255 area 0
 network 10.0.0.0 0.0.0.255 area 0
!
ip route 0.0.0.0 0.0.0.0 192.168.3.2

ChatGPT also explained what it did, starting with:

In this configuration, Router A is connected to Router B via FastEthernet0/0 and to Router D via FastEthernet0/1. It has a loopback interface with IP address 10.0.0.1. […]

Figure out everything that’s wrong with the above configurations before moving on ;)

How about:

• While ChatGPT mentioned the A ➔ D link, it’s missing from the configuration of router A.
• Router D has a single interface that doesn’t match anything else (because A ➔ D link is missing).
• Router B has no loopback interface, so it’s impossible to reach it 🤔
• Loopback interfaces have overlapping subnets (they are configured as /24s not as /32s)

Then we have the chain of unnecessary static default routes: A ➔ B, B ➔ C, C ➔ B, D ➔ A?. Lovely ;)

But Wait It Will Get Better!

Of course, it will. GPT-4 does a much better job (thanks again to Andrew Yourtchenko for running the query for me). Here’s what it generated for Router A:

It’s a major improvement compared to what ChatGPT generated and pretty close to a perfect answer:

• All the required links are there
• Loopbacks have the expected subnet masks
• P2P links use smaller subnets
• OSPF is configured correctly, although not optimally.

Even what ChatGPT generated would be helpful if you take it with a grain of salt² as a starting point for a research project that would eventually result in a running network, but that’s not how the AI cheerleaders are selling the GPT wonders.

Based on all I’ve seen so far I cannot trust anything ChatGPT generates without validating it, which means it could be an interesting absent-minded assistant³, but I would still have to know what I’m doing.⁴

Ignoring the ontological dilemmas⁵, I don’t believe it’s possible to give reliably correct answers without understanding what you’re talking about. Large language models do a fantastic job extracting the meaning from a free-form query and generating a reply, but just because you consumed all the books in the world doesn’t mean that you know or understand what they’re all about. Don’t believe me? Read a graduate textbook about quantum physics and tell me how that worked out.

One could also argue about the mysterious properties of emergent behavior – once a model is large and complex enough, it might exhibit unexpected behavior, like seemingly understanding things. Does that remind you of how they told us OpenFlow would solve all networking problems?

Takeaway: As of March 2023, generic large language model is probably not the best tool to design networks and build network configurations. Let’s see what the next week’s model brings ;)