Introduction

In Previous blog, we covered Static routing on NSX-T Tier-0 router with Active-Standby mode. In this blog, we will cover static routing on NSX-T Tier-0 router with Active-Active Mode. We leverage Active-standby mode when stateful service ( DHCP, NAT,Bridge,etc)  requires but it does not provide ECMP feature.

Active-Active mode support ECMP feature but does not support stateful service till NSX version 4.0 .  NSX-T version 4.1 introduced Stateful with Active-Active deployment.

Note : We will cover Stateful HA with Active-Active deployment in another blog.

Topology Overview

Refer below topology diagram to get high level overview of infrastructure.

We have Admin PC (172.20.10.80) which is connected to VYOS Router ( Gateway for VLAN backed network) andwill be considered as underlay Infrastructure . NSX-T Datacenter have Tier-0 router deployed in Active-Active mode. We have NSX-T edge cluster which are hosting Tier-0 and Tier-1 logical router where 2 Edge Nodes are deployed in VM form factor.

Web server web01 (172.16.10.10) is connected to NSX logical segment Web segment with subnet 172.16.10.0/24 and attached with Tier-1 gateway. Gateway for this logical segment is configured at Tier-1 router.

NSX-T datacenter and Physical Infrastructure ( VYOS Router) is connected  with VLAN 100 and we will configure static route to allow connectivity between Admin PC  (172.20.10.80) and Web server (172.16.10.10).

Note : At VYOS router we will configure 2 static routes which will be pointing to both interfaces on Tier-0 router. VYOS router will perform load balancing for NSX-T overlay subnet.

After Static routing in place, Traffic flow from Web server to Admin PC will follow below path.

Web Server >> Tier-1  (Segment gateway) >> Tier-0 Gateway >>  VYOS router >> Admin PC

Figure 1: Topology Overview

In this Topology, below objects are already placed.

1. NSX-T Manager (4.0.1)
2. Transport Nodes (ESXI) are prepared for NSX-T
3. NSX-T edge cluster with two Nodes are deployed.
4. Tier-0 and Tier-1 routers are deployed.
5. Overlay segment (Web) is configured.

Pre-Validation

We will capture NSX-T configuration and connectivity between Web server and admin VM via ping.

NSX-T Configuration

Please refer below screenshots for NSX-T configuration like Segments, Router, etc.

Figure 2: NSX-T Segments

NSX-T Tier Gateway

Please refer below screenshot for NSX-T Tier-0 and Tier-1 gateway.

Figure 3 : Tier-0 Gateway (Active-Active)

Figure 4 : Tier-1 Gateway

Routes on NSX-T Edge Gateway

SSH to both NSX-T edge gateway and login to Tier-0 SR VRF to validate routes. Verify that route for NSX-T overlay subnet (172.16.0.0/16) is not present.

Figure 5 : Edge Node A Routing Table

Figure 6 : Edge Node B Routing Table

VM connectivity

We tried to ping Admin PC from Web  . Currently Connectivity is not in place, so we are getting Destination Host Unreachable. Traffic from Web server is getting dropped at tier-0 gateway and traffic from Admin PC is getting dropped at VYOS router as both routers doesn’t have exchanged routing information.

Figure 7 : Ping from Web server to Admin PC

Note : 100.64.0.0 belongs to Auto plumb subnet which get created while attaching Tier 1 router to Tier-0

Figure 8: Ping from Web server to VLAN gateway.

Configuration

To apply static configuration at Tier-0 router , Follow below steps.

Login to NSX-T Manager >> Networking >> Tier-0 Gateway >> Click on 3 vertical point and select edit.

Step 1 :  Apply Static route on Tier-0 router to apply North end routing.

Expand Routing section on Tier-0 router and click on Static route >> Set.

Figure 9 : Click on Static route

Click on Add Static Route and provide below information to apply static route. In this Topology, we will apply Default route towards Physical router.

Name: Default route

Network: 0.0.0.0/0

Next Hop : 192.168.100.1 ( VLAN SVI )

Figure 10 : Provide routing information

Click on Set and apply provide next HOP information as per below screenshot

Figure 11 : Apply Next Hop IP

Step 2 : Apply Routing on Physical Router ( Southend Routing) .

Login to VYOS router and apply Static routes for Overlay Network (172.16.10.0/24) towards both interfaces of NSX-T Tier-0 router.Note : You might have another underlay router like Cisco CSR . Please validate vendor configuration before applying changes.

Figure 12 : Static Route on VYOS Router.

This configuration will add 2 route on VYOS Router and will be available for routing load balancing.

Validation

Routes on VYOS Router

SSH to VYOS router and validate that both routes are available for NSX Overlay network (172.16.0.0/16).

Figure 13 : Validate 172.16.0.0/16 route is available on VYOS Router

Ping Test from Web server (172.16.10.10) PC to Admin PC(172.20.10.80)  

Figure 14 : Ping Test from Web server

Ping and traceroute from Admin PC towards Web Server VM.

Figure 15 : Web Server response from Admin PC

Figure 16 : Traceroute from Admin server to Web server.

Note : In above traceroute , we performed traceroute twice and VYOS router sent packet to both Tier-0 Interfaces ( Load balancing in static route).

In our scenario, we added static route on VYOS router without any IP SLA or BFD. In case of any failure of next HOP, there is no mechanism to remove static route which will create blackhole situation if any edge node goes down.

In case of any edge node failure , VYOS router will continue sending packet to down Edge nodes which will create intermittent ping failure or outage. This can be mitigated by using IPSLA  at VYOS router .

Edge Failover Testing

Now , We will test failover scenario when one Active edge node fails. To simulate, We are going to power Off Active Edge Node A and will ping Admin PC from Web server to monitor traffic failover.

Login to Vcenter server, and power off Sa-nsxEdge-01 VM to simulate Edge Node failure.

Figure 17: Power Off Edge Node VM

Once you click on Power Off , system will prompt you to confirm. Click on  Yes and validate that VM is powered off.

Validate NSX-T Edge status

Login to NSX Manager >> Networking >> Tier-0 Gateway >> Click on Active Active to identify status. Sa-nsxEdge-01 edge node will be reflecting as Unknown.

Figure 18 : NSX-T Edge status

Validate ping test

We will perform ping test from Admin PC (172.20.10.80) and web server (172.16.10.11) and vice versa.

Figure 19 : Ping from Admin PC to Web server

Figure 20: Ping test from web server to admin PC

In above ping test , we observed intermittent packet drop which were expected due to static route issue at VYOS router. To mitigate this issue, we manually removed static route which was pointed to Edge node A interface (192.168.10.11).

After removing static route , we got response without any DROP or outage.

Ping output after static route removal

Figure 21: Ping from admin PC to web server

Figure 22: Ping from web server to admin PC

Conclusion

In this blog , We covered NSX Tier-0 Deployment in Active Active mode and enable Static route. In later blogs, We will cover scenario with Active-Active Mode and dynamic routing (OSPF).