In Part 1 of this blog, we discussed at a high-level how Diamanti helps solve many shortcomings found in most of the common Container Network Interface (CNI) plugins available for Kubernetes. In this blog, we’ll deep dive further into Diamanti’s unique Layer 2 (L2) and Layer 3 (L3) networking models.

L2 networking model with hardware acceleration and VLAN-based segmentation

Diamanti accelerates networking with its Ultima I/O offload cards which virtualize networking and storage at the hardware layer, freeing up the host from networking and storage functions.  By offloading storage and network traffic, Diamanti Ultima SmartNIC drives 95% host utilization, making more processing power available for actual applications. This improves overall efficiency and reduces the server footprint, which in turn can provide 80% total cost of ownership (TCO) savings.

Diamanti provides two separate networking planes for storage and pod traffic which are physically isolated from the host network. Diamanti’s Ultima SmartNIC comes with a 40G (4x10G QFSP) interface, out of which 20G is reserved for Diamanti storage traffic and 20G is reserved for pod data traffic.

Diamanti Ultima SmartNIC bypasses the host networking stack and uses SR-IOV virtual functions to plumb a VNIC (Virtual Network Interface) directly to a pod’s network namespace on its local eth0 interface. IT administrators can choose to plumb more than one VNIC to a pod. 

Diamanti L2 networking brings traditional and familiar networking approaches to containers, making it very simple for organizations to move a workload to a container architecture. Each pod can be part of the corporate network and can be accessed directly within the corporate network. Diamanti allows users to create multiple Diamanti networks by assigning an IP to each VNIC, essentially the pod, out of a pool of IPs. From Diamanti’s perspective, these IP addresses are ranges of reserved IP addresses that are part of your corporate network.  

Diamanti L2 networking provides many unique advantages:

VLAN-based segmentation

Diamanti’s hardware-defined network uses VLAN-based segmentation for isolating the traffic between different tenant pods. Each Diamanti node is connected to a TOR (Top of Rack) switch, and pods on different nodes communicate with each other via this TOR switch.  As in traditional networking configuration, the network architect can carve out VLANs and IP address ranges, and configure them on the switch to which the Diamanti cluster connects to. Network architects have the option to configure those VLANs to be private within the local network of the cluster or exposed to the whole corporate network, and even though not recommended, it can be theoretically exposed to the public internet. This fits very well with traditional network architecture.

Traffic Isolation

Diamanti nodes have physically separate interfaces for management, storage, and container traffic. In other words, management traffic (i.e. Kubernetes API, Diamanti API, WebUI, and administrative shell access) and storage traffic are segregated from container traffic for performance and security reasons. 

Pods are first-class citizens in the corporate network

L2 networks allow pods to be part of the corporate network and can be configured to directly communicate with the rest of the network. Each pod has its own unique MAC address allowing customers to use traditional network inspection at the TOR switch and track individual packets back to the originating pod. Endpoints inside the cluster gain full network “citizenship”.

Quality of Service (QoS)

Diamanti provides guaranteed SLAs based on hardware-level QoS for each virtual interface. This eliminates the noisy neighbor problem. Each endpoint can optionally be annotated to use a specific performance tier that sets minimum guarantees and maximum allowed throughput (similar to Kubernetes Requests and Limits) for network and storage I/O.

Ingress is optional

With the Diamanti platform, even though Ingress controllers are recommended, applications can be architected without going through an Ingress controller. This broadens the protocol support (non-HTTP protocols) and reduces hops and latency. 

Static endpoint for consistent network identity

Diamanti provides the flexibility to create a static endpoint which makes sure that a pod’s IP remains the same even if a pod is restarted elsewhere. This provides an easy way to expose the pod to the outside world without worrying about the dynamic nature of Kubernetes.

L3 networking model with software-defined networking and VXLAN-based isolation

In addition to the unique L2 network offerings, Diamanti also supports L3 overlay networking for pods.  An overlay network can be created on top of the Diamanti L2 network. Diamanti’s overlay network implementation is based on Open vSwitch(OVS) and can coexist with L2 networking. Pods are assigned software-defined IP addresses in the overlay network range, and this is routed to the outside world via the OVS endpoint. 

This approach works well for the pod-to-pod communication within the cluster where pods are not limited on the number of underlying VNICs. Also, this approach might be more familiar to developers and architects who have used other SDN CNIs. It allows for a more seamless transition from other Kubernetes platforms to Diamanti. 

L3 Networking model with OpenShift SDN – Software-defined, VXLAN-based isolation

The Diamanti platform also supports running Red Hat OpenShift. When running OpenShift on top of Diamanti, the OpenShift SDN plugin is used for networking instead of the regular Diamanti networking CNI. OpenShift SDN works quite similar to the way Diamanti L3 overlay networking CNI, but there are some differences. 

• In the case of OpenShift SDN, the Ultima SmartNIC storage card works in complete bypass mode. One virtual function is created for both 10G network interfaces which are bonded together to a single network interface. 
• The host interface is completely unused in this case, and OpenShift uses the bonded 20G interface as its primary interface for both control and data traffic. 

Diamanti CSI is still supported in a similar fashion as the regular Diamanti offering and will continue to serve traffic from Ultima storage card for storage services.  

Considerations for choosing L2 vs L3 networking

Both L2 and L3 network models are being widely used in the industry and each comes with its own pros and cons. Let’s explore a few scenarios where each could be useful.

An L2 networking model is suitable for enterprises looking for: 

• Performance and low latencies
• QoS and guaranteed SLAs
• Easy and direct access to applications from outside the cluster
• A networking model that fits easily in existing traditional IT infrastructure
• Better network visibility of traffic flowing in and out of the pods

An L3 model is suitable for enterprises looking for: 

• Access applications running inside the pods within the cluster only
• Simple solutions where performance, SLAs and QoS are not a concern
• An easy way to achieve network isolation and segmentation based on VXLAN and don’t have control over switch configuration and VLAN
• Ability to run more pods in a cluster than allowed by the L2 model (64 per node)
• A way to scale across bigger clusters with a limited pool of IP addresses available for pods 

Best of Both Worlds

Diamanti understands that the selection of networking model is not driven as a prerequisite to cluster provisioning but is a much more dynamic requirement driven by the application intent. Integrating the two L2 and L3 networking functionalities in the CNI plugin and having the endpoint provisioning tied to the application configuration is the most obvious way to solve real-world customer scenarios. Additionally, the Diamanti CNI plugin allows you to tie both these network models to multi-homing, thereby allowing pods to have both L3 and L2 endpoints at the same time. This way some privileged pods can act as gateway to the cluster with one or more legs on the L2 network and other legs on the L3 network. This gives system administrators the flexibility to expose some pods to be part of the corporate network while isolating the rest of the pods behind the private L2 network or L3 overlay network. Diamanti’s CNI plugin provides the simplicity, security and efficiency required by enterprises to successfully deploy and manage cloud-native applications at scale.