One of my friends reviewing the material of my AWS Networking webinar sent me this remark:

I'm always interested in hearing more about how AWS network works under the hood – it’s difficult to gain that knowledge.

As always, it’s almost impossible to find out the behind-the-scenes details, and whatever Amazon is telling you at their re:Invent conference should be taken with a truckload of salt… but it’s relatively easy to figure out a lot of things just by observing them and performing controlled experiments.

As in any (scientific) research:

• Figure out the problem domain;
• Observe what’s going on;
• Form a question;
• Form a hypothesis;
• Create and conduct as simple an experiment as possible to validate or refute the hypothesis;
• Analyze the data and draw a conclusion;
• Lather, rinse, repeat ;))

I talked about this approach in Learning How to Use New Tools part of Getting Started module of Building Network Automation Solutions online course; here’s a simple AWS networking example.

Fact: A VM (EC2 instance) running in a Virtual Private Cloud (VPC) could have a public and a private (intra-VPC) IP address. This should trigger the curiosity of any decent networking engineer (“I wonder how that’s done”)

Observation: Start a Linux VM in a public subnet of a VPC and log into the VM. Execute ifconfig or ip addr to see all IP addresses configured on VM interfaces. You would notice the VM private IP address, but not the public one.

Question: That’s funny… I wonder where the public IP address is…

Hypothesis: The public IP address is present on the Internet gateway which does NAT.

Now stop reading and figure out an experiment that would validate this hypothesis.

Here’s what I did:

• Create two VMs in a VPC subnet;
• Ping between their private IP addresses – it works;
• Ping between their public IP addresses – it works;
• Remove the default route pointing to the Internet gateway;
• Repeat the tests

Result: After removing the default route to the Internet gateway, the VMs cannot reach the public IP address of other VMs. Internet gateway is therefore somehow involved in this process.

Hypothesis: Internet gateway (whatever it happens to be and wherever it happens to be located within AWS) performs NAT between private and public VM addresses.

Question: I wonder what happens when a VM in a VPC pings the public I address of another VM in the same subnet.

Hypothesis: If the Internet gateway performs NAT between private and public IP addresses of a VM, then it should also NAT the source IP address of outgoing traffic. Pings received on the second VM should look like they’re coming from the public IP address of the first VM.

Experiment: Trivial; left as an exercise.

Got the idea? Now figure out how packet forwarding works and how you can influence it with routing tables configured on the VMs ;)

Want to know more?

I documented everything I discovered while experimenting with AWS in my new AWS Networking Deep Dive webinar. We covered regions, availability zones, VPCs, subnets, interfaces and addressing in the first live session.

The second live session – starting with sample deployment scenarios and then moving on to network security – is in just two days… and all you need to attend it is a current ipSpace.net subscription.