[Cilium] BGP Control Plane & ClusterMesh

들어가며

이번 포스트에서는 BGP를 통한 라우팅과 kind, cluster-mesh를 통한 멀티 클러스터 환경에서의 Cilium 동작을 살펴보겠습니다.

실습 환경 구성

이번 실습에서는 지난 실습과 마찬가지로 k8s-w0를 별도의 네트워크에 배치하고 router를 통해 k8s-w0와 k8s-ctr/w1 노드간 통신을 확인합니다. 하지만 이번 실습에서는 frr을 설치하여 BGP 라우팅을 통해 통신을 확인해보겠습니다.
- 기본 배포 가상 머신 : k8s-ctr, k8s-w1, k8s-w0, router (frr 라우팅)
- router : router : 192.168.10.0/24 ↔ 192.168.20.0/24 대역 라우팅 역할, k8s 에 join 되지 않은 서버이며, BGP 동작을 위해 frr 툴이 설치되어있습니다.
- k8s-w0 : k8s-ctr/w1 노드와 다른 네트워크 대역에 배치됩니다.
- 실습 동작에 필요한 static routing이 설저된 상태로 배포 됩니다.

실습환경 배포 파일

Vagrantfile : 가상머신 정의, 부팅 시 초기 프로비저닝 설정을 포함하는 Vagrantfile입니다.

# Variables
K8SV = '1.33.2-1.1' # Kubernetes Version : apt list -a kubelet , ex) 1.32.5-1.1
CONTAINERDV = '1.7.27-1' # Containerd Version : apt list -a containerd.io , ex) 1.6.33-1
CILIUMV = '1.18.0' # Cilium CNI Version : https://github.com/cilium/cilium/tags
N = 1 # max number of worker nodes
  
# Base Image  https://portal.cloud.hashicorp.com/vagrant/discover/bento/ubuntu-24.04
BOX_IMAGE = "bento/ubuntu-24.04"
BOX_VERSION = "202508.03.0"
  
Vagrant.configure("2") do |config|
  #-ControlPlane Node
  config.vm.define "k8s-ctr" do |subconfig|
    subconfig.vm.box = BOX_IMAGE
  
    subconfig.vm.box_version = BOX_VERSION
    subconfig.vm.provider "virtualbox" do |vb|
      vb.customize ["modifyvm", :id, "--groups", "/Cilium-Lab"]
      vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
      vb.name = "k8s-ctr"
      vb.cpus = 2
      vb.memory = 2560
      vb.linked_clone = true
    end
    subconfig.vm.host_name = "k8s-ctr"
    subconfig.vm.network "private_network", ip: "192.168.10.100"
    subconfig.vm.network "forwarded_port", guest: 22, host: 60000, auto_correct: true, id: "ssh"
    subconfig.vm.synced_folder "./", "/vagrant", disabled: true
    subconfig.vm.provision "shell", path: "https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/5w/init_cfg.sh", args: [ K8SV, CONTAINERDV ]
    subconfig.vm.provision "shell", path: "https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/5w/k8s-ctr.sh", args: [ N, CILIUMV, K8SV ]
    subconfig.vm.provision "shell", path: "https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/5w/route-add1.sh"
  end
  
  #-Worker Nodes Subnet1
  (1..N).each do |i|
    config.vm.define "k8s-w#{i}" do |subconfig|
      subconfig.vm.box = BOX_IMAGE
      subconfig.vm.box_version = BOX_VERSION
      subconfig.vm.provider "virtualbox" do |vb|
        vb.customize ["modifyvm", :id, "--groups", "/Cilium-Lab"]
        vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
        vb.name = "k8s-w#{i}"
        vb.cpus = 2
        vb.memory = 1536
        vb.linked_clone = true
      end
      subconfig.vm.host_name = "k8s-w#{i}"
      subconfig.vm.network "private_network", ip: "192.168.10.10#{i}"
      subconfig.vm.network "forwarded_port", guest: 22, host: "6000#{i}", auto_correct: true, id: "ssh"
      subconfig.vm.synced_folder "./", "/vagrant", disabled: true
      subconfig.vm.provision "shell", path: "https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/5w/init_cfg.sh", args: [ K8SV, CONTAINERDV]
      subconfig.vm.provision "shell", path: "https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/5w/k8s-w.sh"
      subconfig.vm.provision "shell", path: "https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/5w/route-add1.sh"
    end
  end
  
  #-Router Node
  config.vm.define "router" do |subconfig|
    subconfig.vm.box = BOX_IMAGE
    subconfig.vm.box_version = BOX_VERSION
    subconfig.vm.provider "virtualbox" do |vb|
      vb.customize ["modifyvm", :id, "--groups", "/Cilium-Lab"]
      vb.name = "router"
      vb.cpus = 1
      vb.memory = 768
      vb.linked_clone = true
    end
    subconfig.vm.host_name = "router"
    subconfig.vm.network "private_network", ip: "192.168.10.200"
    subconfig.vm.network "forwarded_port", guest: 22, host: 60009, auto_correct: true, id: "ssh"
    subconfig.vm.network "private_network", ip: "192.168.20.200", auto_config: false
    subconfig.vm.synced_folder "./", "/vagrant", disabled: true
    subconfig.vm.provision "shell", path: "https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/5w/router.sh"
  end
  
  #-Worker Nodes Subnet2
  config.vm.define "k8s-w0" do |subconfig|
    subconfig.vm.box = BOX_IMAGE
    subconfig.vm.box_version = BOX_VERSION
    subconfig.vm.provider "virtualbox" do |vb|
      vb.customize ["modifyvm", :id, "--groups", "/Cilium-Lab"]
      vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
      vb.name = "k8s-w0"
      vb.cpus = 2
      vb.memory = 1536
      vb.linked_clone = true
    end
    subconfig.vm.host_name = "k8s-w0"
    subconfig.vm.network "private_network", ip: "192.168.20.100"
    subconfig.vm.network "forwarded_port", guest: 22, host: 60010, auto_correct: true, id: "ssh"
    subconfig.vm.synced_folder "./", "/vagrant", disabled: true
    subconfig.vm.provision "shell", path: "https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/5w/init_cfg.sh", args: [ K8SV, CONTAINERDV]
    subconfig.vm.provision "shell", path: "https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/5w/k8s-w.sh"
    subconfig.vm.provision "shell", path: "https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/5w/route-add2.sh"
  end
  
end

init_cfg.sh : args 참고하여 초기 설정을 수행하는 스크립트입니다.

#!/usr/bin/env bash
  
echo ">>>> Initial Config Start <<<<"
  
echo "[TASK 1] Setting Profile & Bashrc"
echo 'alias vi=vim' >> /etc/profile
echo "sudo su -" >> /home/vagrant/.bashrc
ln -sf /usr/share/zoneinfo/Asia/Seoul /etc/localtime # Change Timezone
  
  
echo "[TASK 2] Disable AppArmor"
systemctl stop ufw && systemctl disable ufw >/dev/null 2>&1
systemctl stop apparmor && systemctl disable apparmor >/dev/null 2>&1
  
  
echo "[TASK 3] Disable and turn off SWAP"
swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab
  
  
echo "[TASK 4] Install Packages"
apt update -qq >/dev/null 2>&1
apt-get install apt-transport-https ca-certificates curl gpg -y -qq >/dev/null 2>&1
  
# Download the public signing key for the Kubernetes package repositories.
mkdir -p -m 755 /etc/apt/keyrings
K8SMMV=$(echo $1 | sed -En 's/^([0-9]+\.[0-9]+)\..*/\1/p')
curl -fsSL https://pkgs.k8s.io/core:/stable:/v$K8SMMV/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v$K8SMMV/deb/ /" >> /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
  
# packets traversing the bridge are processed by iptables for filtering
echo 1 > /proc/sys/net/ipv4/ip_forward
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.d/k8s.conf
  
# enable br_netfilter for iptables 
modprobe br_netfilter
modprobe overlay
echo "br_netfilter" >> /etc/modules-load.d/k8s.conf
echo "overlay" >> /etc/modules-load.d/k8s.conf
  
  
echo "[TASK 5] Install Kubernetes components (kubeadm, kubelet and kubectl)"
# Update the apt package index, install kubelet, kubeadm and kubectl, and pin their version
apt update >/dev/null 2>&1
  
# apt list -a kubelet ; apt list -a containerd.io
apt-get install -y kubelet=$1 kubectl=$1 kubeadm=$1 containerd.io=$2 >/dev/null 2>&1
apt-mark hold kubelet kubeadm kubectl >/dev/null 2>&1
  
# containerd configure to default and cgroup managed by systemd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
  
# avoid WARN&ERRO(default endpoints) when crictl run  
cat <<EOF > /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
EOF
  
# ready to install for k8s 
systemctl restart containerd && systemctl enable containerd
systemctl enable --now kubelet
  
  
echo "[TASK 6] Install Packages & Helm"
export DEBIAN_FRONTEND=noninteractive
apt-get install -y bridge-utils sshpass net-tools conntrack ngrep tcpdump ipset arping wireguard jq yq tree bash-completion unzip kubecolor termshark >/dev/null 2>&1
curl -s https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash >/dev/null 2>&1
  
  
echo ">>>> Initial Config End <<<<"

k8s-ctr.sh : kubeadm init를 통하여 kubernetes controlplane 노드를 설정하고 Cilium CNI 설치, 편리성 설정(k, kc)하는 스크립트입니다. local-path-storageclass와 metrics-server도 설치합니다.

#!/usr/bin/env bash
  
echo ">>>> K8S Controlplane config Start <<<<"
  
echo "[TASK 1] Initial Kubernetes"
curl --silent -o /root/kubeadm-init-ctr-config.yaml https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/kubeadm-init-ctr-config.yaml
K8SMMV=$(echo $3 | sed -En 's/^([0-9]+\.[0-9]+\.[0-9]+).*/\1/p')
sed -i "s/K8S_VERSION_PLACEHOLDER/v${K8SMMV}/g" /root/kubeadm-init-ctr-config.yaml
kubeadm init --config="/root/kubeadm-init-ctr-config.yaml"  >/dev/null 2>&1
  
  
echo "[TASK 2] Setting kube config file"
mkdir -p /root/.kube
cp -i /etc/kubernetes/admin.conf /root/.kube/config
chown $(id -u):$(id -g) /root/.kube/config
  
  
echo "[TASK 3] Source the completion"
echo 'source <(kubectl completion bash)' >> /etc/profile
echo 'source <(kubeadm completion bash)' >> /etc/profile
  
  
echo "[TASK 4] Alias kubectl to k"
echo 'alias k=kubectl' >> /etc/profile
echo 'alias kc=kubecolor' >> /etc/profile
echo 'complete -F __start_kubectl k' >> /etc/profile
  
  
echo "[TASK 5] Install Kubectx & Kubens"
git clone https://github.com/ahmetb/kubectx /opt/kubectx >/dev/null 2>&1
ln -s /opt/kubectx/kubens /usr/local/bin/kubens
ln -s /opt/kubectx/kubectx /usr/local/bin/kubectx
  
  
echo "[TASK 6] Install Kubeps & Setting PS1"
git clone https://github.com/jonmosco/kube-ps1.git /root/kube-ps1 >/dev/null 2>&1
cat <<"EOT" >> /root/.bash_profile
source /root/kube-ps1/kube-ps1.sh
KUBE_PS1_SYMBOL_ENABLE=true
function get_cluster_short() {
  echo "$1" | cut -d . -f1
}
KUBE_PS1_CLUSTER_FUNCTION=get_cluster_short
KUBE_PS1_SUFFIX=') '
PS1='$(kube_ps1)'$PS1
EOT
kubectl config rename-context "kubernetes-admin@kubernetes" "HomeLab" >/dev/null 2>&1
  
  
echo "[TASK 7] Install Cilium CNI"
NODEIP=$(ip -4 addr show eth1 | grep -oP '(?<=inet\s)\d+(\.\d+){3}')
helm repo add cilium https://helm.cilium.io/ >/dev/null 2>&1
helm repo update >/dev/null 2>&1
helm install cilium cilium/cilium --version $2 --namespace kube-system \
--set k8sServiceHost=192.168.10.100 --set k8sServicePort=6443 \
--set ipam.mode="cluster-pool" --set ipam.operator.clusterPoolIPv4PodCIDRList={"172.20.0.0/16"} --set ipv4NativeRoutingCIDR=172.20.0.0/16 \
--set routingMode=native --set autoDirectNodeRoutes=false --set bgpControlPlane.enabled=true \
--set kubeProxyReplacement=true --set bpf.masquerade=true --set installNoConntrackIptablesRules=true \
--set endpointHealthChecking.enabled=false --set healthChecking=false \
--set hubble.enabled=true --set hubble.relay.enabled=true --set hubble.ui.enabled=true \
--set hubble.ui.service.type=NodePort --set hubble.ui.service.nodePort=30003 \
--set prometheus.enabled=true --set operator.prometheus.enabled=true --set hubble.metrics.enableOpenMetrics=true \
--set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_namespace\,source_workload\,destination_ip\,destination_namespace\,destination_workload\,traffic_direction}" \
--set operator.replicas=1 --set debug.enabled=true >/dev/null 2>&1
  
  
echo "[TASK 8] Install Cilium / Hubble CLI"
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz >/dev/null 2>&1
tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz
  
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
HUBBLE_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then HUBBLE_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz >/dev/null 2>&1
tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin
rm hubble-linux-${HUBBLE_ARCH}.tar.gz
  
  
echo "[TASK 9] Remove node taint"
kubectl taint nodes k8s-ctr node-role.kubernetes.io/control-plane-
  
  
echo "[TASK 10] local DNS with hosts file"
echo "192.168.10.100 k8s-ctr" >> /etc/hosts
echo "192.168.10.200 router" >> /etc/hosts
echo "192.168.20.100 k8s-w0" >> /etc/hosts
for (( i=1; i<=$1; i++  )); do echo "192.168.10.10$i k8s-w$i" >> /etc/hosts; done
  
  
echo "[TASK 11] Dynamically provisioning persistent local storage with Kubernetes"
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.31/deploy/local-path-storage.yaml >/dev/null 2>&1
kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' >/dev/null 2>&1
  
  
# echo "[TASK 12] Install Prometheus & Grafana"
# kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.18.0/examples/kubernetes/addons/prometheus/monitoring-example.yaml >/dev/null 2>&1
# kubectl patch svc -n cilium-monitoring prometheus -p '{"spec": {"type": "NodePort", "ports": [{"port": 9090, "targetPort": 9090, "nodePort": 30001}]}}' >/dev/null 2>&1
# kubectl patch svc -n cilium-monitoring grafana -p '{"spec": {"type": "NodePort", "ports": [{"port": 3000, "targetPort": 3000, "nodePort": 30002}]}}' >/dev/null 2>&1
  
# echo "[TASK 12] Install Prometheus Stack"
# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts  >/dev/null 2>&1
# cat <<EOT > monitor-values.yaml
# prometheus:
#   prometheusSpec:
#     scrapeInterval: "15s"
#     evaluationInterval: "15s"
#   service:
#     type: NodePort
#     nodePort: 30001
  
# grafana:
#   defaultDashboardsTimezone: Asia/Seoul
#   adminPassword: prom-operator
#   service:
#     type: NodePort
#     nodePort: 30002
  
# alertmanager:
#   enabled: false
# defaultRules:
#   create: false
# prometheus-windows-exporter:
#   prometheus:
#     monitor:
#       enabled: false
# EOT
# helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 75.15.1 \
#   -f monitor-values.yaml --create-namespace --namespace monitoring  >/dev/null 2>&1
  
  
echo "[TASK 13] Install Metrics-server"
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/  >/dev/null 2>&1
helm upgrade --install metrics-server metrics-server/metrics-server --set 'args[0]=--kubelet-insecure-tls' -n kube-system  >/dev/null 2>&1
  
  
echo "[TASK 14] Install k9s"
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
wget https://github.com/derailed/k9s/releases/latest/download/k9s_linux_${CLI_ARCH}.deb -O /tmp/k9s_linux_${CLI_ARCH}.deb  >/dev/null 2>&1
apt install /tmp/k9s_linux_${CLI_ARCH}.deb  >/dev/null 2>&1
  
  
echo ">>>> K8S Controlplane Config End <<<<"

kubeadm-init-ctr-config.yaml

apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
bootstrapTokens:
- token: "123456.1234567890123456"
  ttl: "0s"
  usages:
  - signing
  - authentication
localAPIEndpoint:
  advertiseAddress: "192.168.10.100"
nodeRegistration:
  kubeletExtraArgs:
    - name: node-ip
      value: "192.168.10.100"
  criSocket: "unix:///run/containerd/containerd.sock"
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: "K8S_VERSION_PLACEHOLDER"
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/16"

k8s-w.sh : kubernetes worker 노드 설정, kubeadm join 등을 수행하는 스크립트입니다.

#!/usr/bin/env bash
  
echo ">>>> K8S Node config Start <<<<"
  
  
echo "[TASK 1] K8S Controlplane Join"
curl --silent -o /root/kubeadm-join-worker-config.yaml https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/2w/kubeadm-join-worker-config.yaml
NODEIP=$(ip -4 addr show eth1 | grep -oP '(?<=inet\s)\d+(\.\d+){3}')
sed -i "s/NODE_IP_PLACEHOLDER/${NODEIP}/g" /root/kubeadm-join-worker-config.yaml
kubeadm join --config="/root/kubeadm-join-worker-config.yaml" > /dev/null 2>&1
  
  
echo ">>>> K8S Node config End <<<<"

kubeadm-join-worker-config.yaml

apiVersion: kubeadm.k8s.io/v1beta4
kind: JoinConfiguration
discovery:
  bootstrapToken:
    token: "123456.1234567890123456"
    apiServerEndpoint: "192.168.10.100:6443"
    unsafeSkipCAVerification: true
nodeRegistration:
  criSocket: "unix:///run/containerd/containerd.sock"
  kubeletExtraArgs:
    - name: node-ip
      value: "NODE_IP_PLACEHOLDER"

route-add1.sh : k8s node 들이 내부망과 통신을 위한 route 설정 스크립트입니다.

#!/usr/bin/env bash
  
echo ">>>> Route Add Config Start <<<<"
  
chmod 600 /etc/netplan/01-netcfg.yaml
chmod 600 /etc/netplan/50-vagrant.yaml
  
cat <<EOT>> /etc/netplan/50-vagrant.yaml
      routes:
      - to: 192.168.20.0/24
        via: 192.168.10.200
      # - to: 172.20.0.0/16
      #   via: 192.168.10.200
EOT
  
netplan apply
  
echo ">>>> Route Add Config End <<<<"

route-add2.sh : k8s node 들이 내부망과 통신을 위한 route 설정 스크립트입니다.

#!/usr/bin/env bash
  
echo ">>>> Route Add Config Start <<<<"
  
chmod 600 /etc/netplan/01-netcfg.yaml
chmod 600 /etc/netplan/50-vagrant.yaml
  
cat <<EOT>> /etc/netplan/50-vagrant.yaml
      routes:
      - to: 192.168.10.0/24
        via: 192.168.20.200
      # - to: 172.20.0.0/16
      #   via: 192.168.20.200
EOT
  
netplan apply
  
echo ">>>> Route Add Config End <<<<"

router.sh : router(frr - BGP) 역할, 간단 웹 서버 역할

#!/usr/bin/env bash
  
echo ">>>> Initial Config Start <<<<"
  
  
echo "[TASK 0] Setting eth2"
chmod 600 /etc/netplan/01-netcfg.yaml
chmod 600 /etc/netplan/50-vagrant.yaml
  
cat << EOT >> /etc/netplan/50-vagrant.yaml
    eth2:
      addresses:
      - 192.168.20.200/24
EOT
  
netplan apply
  
  
echo "[TASK 1] Setting Profile & Bashrc"
echo 'alias vi=vim' >> /etc/profile
echo "sudo su -" >> /home/vagrant/.bashrc
ln -sf /usr/share/zoneinfo/Asia/Seoul /etc/localtime
  
  
echo "[TASK 2] Disable AppArmor"
systemctl stop ufw && systemctl disable ufw >/dev/null 2>&1
systemctl stop apparmor && systemctl disable apparmor >/dev/null 2>&1
  
  
echo "[TASK 3] Add Kernel setting - IP Forwarding"
sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/g' /etc/sysctl.conf
sysctl -p >/dev/null 2>&1
  
  
echo "[TASK 4] Setting Dummy Interface"
modprobe dummy
ip link add loop1 type dummy
ip link set loop1 up
ip addr add 10.10.1.200/24 dev loop1
  
ip link add loop2 type dummy
ip link set loop2 up
ip addr add 10.10.2.200/24 dev loop2
  
  
echo "[TASK 5] Install Packages"
export DEBIAN_FRONTEND=noninteractive
apt update -qq >/dev/null 2>&1
apt-get install net-tools jq yq tree ngrep tcpdump arping termshark -y -qq >/dev/null 2>&1
  
  
echo "[TASK 6] Install Apache"
apt install apache2 -y >/dev/null 2>&1
echo -e "<h1>Web Server : $(hostname)</h1>" > /var/www/html/index.html
  
  
echo "[TASK 7] Configure FRR"
apt install frr -y >/dev/null 2>&1
sed -i "s/^bgpd=no/bgpd=yes/g" /etc/frr/daemons
  
NODEIP=$(ip -4 addr show eth1 | grep -oP '(?<=inet\s)\d+(\.\d+){3}')
cat << EOF >> /etc/frr/frr.conf
!
router bgp 65000
  bgp router-id $NODEIP
  bgp graceful-restart
  no bgp ebgp-requires-policy
  bgp bestpath as-path multipath-relax
  maximum-paths 4
  network 10.10.1.0/24
EOF
  
  
systemctl daemon-reexec >/dev/null 2>&1
systemctl restart frr >/dev/null 2>&1
systemctl enable frr >/dev/null 2>&1
  
  
echo ">>>> Initial Config End <<<<"

실습환경 배포

  $ vagrant up
  # =>     k8s-w0: >>>> Initial Config End <<<<
  #    ==> k8s-w0: Running provisioner: shell...
  #        k8s-w0: Running: /var/folders/7k/qy6rsdds57z3tmyn9_7hhd8r0000gn/T/vagrant-shell20250815-16129-ssrxpm.sh
  #        k8s-w0: >>>> K8S Node config Start <<<<
  #        k8s-w0: [TASK 1] K8S Controlplane Join
  #        k8s-w0: >>>> K8S Node config End <<<<
  #    ==> k8s-w0: Running provisioner: shell...
  #        k8s-w0: Running: /var/folders/7k/qy6rsdds57z3tmyn9_7hhd8r0000gn/T/vagrant-shell20250815-16129-iotl22.sh
  #        k8s-w0: >>>> Route Add Config Start <<<<
  #        k8s-w0: >>>> Route Add Config End <<<<

기본정보 확인

# cilium 상태 확인 : bgp-control-plane 미리 활성화.
$ kubectl get cm -n kube-system cilium-config -o json | jq
$ cilium status 
$ cilium config view | grep -i bgp
# => bgp-router-id-allocation-ip-pool
#    bgp-router-id-allocation-mode                     default
#    bgp-secrets-namespace                             kube-system
#    enable-bgp-control-plane                          true
#    enable-bgp-control-plane-status-report            true

#
$ kubectl exec -n kube-system -c cilium-agent -it ds/cilium -- cilium-dbg status --verbose
$ kubectl exec -n kube-system -c cilium-agent -it ds/cilium -- cilium-dbg metrics list 

# monitor
$ kubectl exec -n kube-system -c cilium-agent -it ds/cilium -- cilium-dbg monitor
$ kubectl exec -n kube-system -c cilium-agent -it ds/cilium -- cilium-dbg monitor -v
$ kubectl exec -n kube-system -c cilium-agent -it ds/cilium -- cilium-dbg monitor -v -v
...

네트워크 정보 확인 : `autoDirectNodeRoutes=false`

# router 네트워크 인터페이스 정보 확인
$ sshpass -p 'vagrant' ssh vagrant@router ip -br -c -4 addr

# k8s node 네트워크 인터페이스 정보 확인
$ ip -c -4 addr show dev eth1
# => 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
#        altname enp0s9
#        inet <span style="color: green;">192.168.10.100/24</span> brd 192.168.10.255 scope global eth1
#           valid_lft forever preferred_lft forever
$ for i in w1 w0 ; do echo ">> node : k8s-$i <<"; sshpass -p 'vagrant' ssh vagrant@k8s-$i ip -c -4 addr show dev eth1; echo; done
# => 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
#        altname enp0s9
#        inet <span style="color: green;">192.168.10.101/24</span> brd 192.168.10.255 scope global eth1
#           valid_lft forever preferred_lft forever
#    >> node : k8s-w0 <<
#    3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
#        altname enp0s9
#        inet <span style="color: green;">192.168.20.100/24</span> brd 192.168.20.255 scope global eth1
#           valid_lft forever preferred_lft forever

# 라우팅 정보 확인
$ sshpass -p 'vagrant' ssh vagrant@router ip -c route
# => ...
#    192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.200
#    192.168.20.0/24 dev eth2 proto kernel scope link src 192.168.20.200
$ ip -c route | grep static
# => 192.168.20.0/24 via 192.168.10.200 dev eth1 proto static

## 노드별 PodCIDR 라우팅이 없습니다!
$ ip -c route
# => default via 10.0.2.2 dev eth0 proto dhcp src 10.0.2.15 metric 100
#    10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 100
#    10.0.2.2 dev eth0 proto dhcp scope link src 10.0.2.15 metric 100
#    10.0.2.3 dev eth0 proto dhcp scope link src 10.0.2.15 metric 100
#    172.20.0.0/24 via 172.20.0.222 dev cilium_host proto kernel src 172.20.0.222
#    172.20.0.222 dev cilium_host proto kernel scope link
#    192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.100
#    192.168.20.0/24 via 192.168.10.200 dev eth1 proto static
$ for i in w1 w0 ; do echo ">> node : k8s-$i <<"; sshpass -p 'vagrant' ssh vagrant@k8s-$i ip -c route; echo; done
# => >> node : k8s-w1 <<
#    default via 10.0.2.2 dev eth0 proto dhcp src 10.0.2.15 metric 100
#    10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 100
#    10.0.2.2 dev eth0 proto dhcp scope link src 10.0.2.15 metric 100
#    10.0.2.3 dev eth0 proto dhcp scope link src 10.0.2.15 metric 100
#    172.20.1.0/24 via 172.20.1.44 dev cilium_host proto kernel src 172.20.1.44
#    172.20.1.44 dev cilium_host proto kernel scope link
#    192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.101
#    192.168.20.0/24 via 192.168.10.200 dev eth1 proto static
#    
#    >> node : k8s-w0 <<
#    default via 10.0.2.2 dev eth0 proto dhcp src 10.0.2.15 metric 100
#    10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 100
#    10.0.2.2 dev eth0 proto dhcp scope link src 10.0.2.15 metric 100
#    10.0.2.3 dev eth0 proto dhcp scope link src 10.0.2.15 metric 100
#    172.20.2.0/24 via 172.20.2.246 dev cilium_host proto kernel src 172.20.2.246
#    172.20.2.246 dev cilium_host proto kernel scope link
#    192.168.10.0/24 via 192.168.20.200 dev eth1 proto static
#    192.168.20.0/24 dev eth1 proto kernel scope link src 192.168.20.100

# 통신 확인
$ ping -c 1 192.168.20.100  # k8s-w0 eth1
# => PING 192.168.20.100 (192.168.20.100) 56(84) bytes of data.
#    64 bytes from 192.168.20.100: icmp_seq=1 ttl=63 time=2.46 ms
#    
#    --- 192.168.20.100 ping statistics ---
#    1 packets transmitted, 1 received, 0% packet loss, time 0ms
#    rtt min/avg/max/mdev = 2.461/2.461/2.461/0.000 ms

현재노드간 통신은 가능하지만 pod 간 통신은 불가능합니다.
autoDirectNodeRoutes=false을 통해 자동으로 같은 L2 네트워크에 있는 노드들간의 PodCIDR 라우팅하는 기능이 꺼져있어서 라우팅 룰이 없기때문입니다.

샘플 애플리케이션 배포 및 통신 문제 확인

샘플 애플리케이션 배포

# 샘플 애플리케이션 배포
$ cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webpod
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webpod
  template:
    metadata:
      labels:
        app: webpod
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - sample-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: webpod
        image: traefik/whoami
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: webpod
  labels:
    app: webpod
spec:
  selector:
    app: webpod
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: ClusterIP
EOF
# => deployment.apps/webpod created
#    service/webpod created

# k8s-ctr 노드에 curl-pod 파드 배포
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: curl-pod
  labels:
    app: curl
spec:
  nodeName: k8s-ctr
  containers:
  - name: curl
    image: nicolaka/netshoot
    command: ["tail"]
    args: ["-f", "/dev/null"]
  terminationGracePeriodSeconds: 0
EOF
# => pod/curl-pod created

통신 문제 확인 : 노드 내의 파드간에만 통신이 되는 중입니다.

# 배포 확인
$ kubectl get deploy,svc,ep webpod -owide
# => NAME                     READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES           SELECTOR
#    deployment.apps/webpod   3/3     3            3           102s   webpod       traefik/whoami   app=webpod
#    
#    NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE    SELECTOR
#    service/webpod   ClusterIP   10.96.28.4   <none>        80/TCP    102s   app=webpod
#    
#    NAME               ENDPOINTS                                      AGE
#    endpoints/webpod   172.20.0.72:80,172.20.1.7:80,172.20.2.190:80   102s
$ kubectl get endpointslices -l app=webpod
# => NAME           ADDRESSTYPE   PORTS   ENDPOINTS                             AGE
#    webpod-4pxps   IPv4          80      172.20.0.72,172.20.1.7,172.20.2.190   2m23s
$ kubectl get ciliumendpoints # IP 확인
# => NAME                      SECURITY IDENTITY   ENDPOINT STATE   IPV4           IPV6
#    curl-pod                  47416               ready            172.20.0.187
#    <span style="color: green;">webpod-697b545f57-j6b7t</span>   9975                ready            <span style="color: green;">172.20.0.72</span>
#    webpod-697b545f57-n7fk7   9975                ready            172.20.2.190
#    webpod-697b545f57-rl4d2   9975                ready            172.20.1.7

# 통신 문제 확인 : 노드 내의 파드들 끼리만 통신되는 중!
$ kubectl exec -it curl-pod -- curl -s --connect-timeout 1 webpod | grep Hostname
# => Hostname: webpod-697b545f57-j6b7t
# <span style="color: green;">👉 k8s-ctr에서 실행중이어서 같은 k8s-ctr에 있는 파드와는 통신이 됩니다.</span>
$ kubectl exec -it curl-pod -- curl -s --connect-timeout 1 webpod | grep Hostname
# <span style="color: green;">👉 다시 실행했을때는 Kubernetes Service에 의해 다른 노드의 파드로 로드밸런싱되어 통신이 실패합니다.</span>
# => command terminated with exit code 28
$ kubectl exec -it curl-pod -- sh -c 'while true; do curl -s --connect-timeout 1 webpod | grep Hostname; echo "---" ; sleep 1; done'
# => Hostname: webpod-697b545f57-j6b7t
#    ---
#    ---
#    Hostname: webpod-697b545f57-j6b7t
#    ---
#    ---
#    Hostname: webpod-697b545f57-j6b7t
#    ...
# <span style="color: green;">👉 여러번 실행해도 k8s-ctr 노드에 있는 파드만 통신이 되는것을 확인할 수 있습니다.</span>

# cilium-dbg, map
$ kubectl exec -n kube-system ds/cilium -- cilium-dbg ip list
# => IP                  IDENTITY                                                                            SOURCE
#    ...
#    <span style="color: green;">172.20.1.7/32</span>       k8s:app=webpod                                                                      custom-resource
#    ...
#    <span style="color: green;">172.20.2.190/32</span>     k8s:app=webpod                                                                      custom-resource
#    ...
#    192.168.10.101/32   reserved:remote-node
#    192.168.20.100/32   reserved:remote-node
# <span style="color: green;">👉 cilium도 다른 노드의 파드 IP를 알고 있지만, 라우팅이 되지 않아서 통신이 되지 않습니다.</span>
$ kubectl exec -n kube-system ds/cilium -- cilium-dbg endpoint list
$ kubectl exec -n kube-system ds/cilium -- cilium-dbg service list
$ kubectl exec -n kube-system ds/cilium -- cilium-dbg bpf lb list
$ kubectl exec -n kube-system ds/cilium -- cilium-dbg bpf nat list
$ kubectl exec -n kube-system ds/cilium -- cilium-dbg map list | grep -v '0             0'
$ kubectl exec -n kube-system ds/cilium -- cilium-dbg map get cilium_lb4_services_v2
$ kubectl exec -n kube-system ds/cilium -- cilium-dbg map get cilium_lb4_backends_v3
$ kubectl exec -n kube-system ds/cilium -- cilium-dbg map get cilium_lb4_reverse_nat
$ kubectl exec -n kube-system ds/cilium -- cilium-dbg map get cilium_ipcache_v2