Expunge vm not release in-memory dhcp record in vpc redundant virtual route correctly

[hulucc]

Environment

cloudstack 4.19.1.2

Problem

In-memory record never released before next reboot of virtual router, finally run out of ip address, new vm can not get ip from dhcp.

Reproduce

1. Create vpc with redundant router and vm
2. After vm success boot up, expunge it.
3. Check `/var/lib/misc/dnsmasq.leases`, the vm record still there.

Research

1. When expunging vm, cloudstack will call shell command like dhcp_release eth3 172.29.101.250 02:02:00:d4:00:4f try to release in-memory dhcp record in dnsmasq in virtual router vm
2. the dhcp_release command will actually do nothing, because dnsmasq will check the server address in config is matching the request address. check the dnsmasq source code below

// dnsmasq v2.85 src\rfc2131.c:1046
    case DHCPRELEASE:
      if (!(context = narrow_context(context, mess->ciaddr, tagif_netid)) ||
	  !(opt = option_find(mess, sz, OPTION_SERVER_IDENTIFIER, INADDRSZ)) ||
	  option_addr(opt).s_addr != server_id(context, override, fallback).s_addr) <<<<<<<<<<
	return 0;

3. The address will not match because dhcp_release is requesting the primary address, and dnsmasq is listening the secondary address.

// /etc/dnsmasq.d/cloud.conf
dhcp-hostsfile=/etc/dhcphosts.txt
listen-address=127.0.0.1,172.29.101.1           <<<<<<<<<<<<<<<
dhcp-range=set:interface-eth3-1,172.29.101.1,static
dhcp-option=tag:interface-eth3-1,15,cs2cloud.internal
dhcp-option=tag:interface-eth3-1,6,172.29.101.1,10.1.2.146
dhcp-option=tag:interface-eth3-1,3,172.29.101.1
dhcp-option=eth3,26,1450
dhcp-option=tag:interface-eth3-1,1,255.255.255.0

// tcpdump view of packet emit by dhcp_release
08:23:24.200332 lo    In  IP (tos 0x0, ttl 64, id 61594, offset 0, flags [DF], proto UDP (17), length 576)
    172.29.101.245.57053 > 172.29.101.245.67: [bad udp cksum 0x2663 -> 0xd1aa!] BOOTP/DHCP, Request from 02:02:00:d4:00:51, length 548, Flags [none] (0x0000)
          Client-IP 172.29.101.49
          Client-Ethernet-Address 02:02:00:d4:00:51
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Release
            Server-ID (54), length 4: 172.29.101.245
            END (255), length 0
            PAD (0), length 0, occurs 298

// ip addr show
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 02:02:00:d4:00:32 brd ff:ff:ff:ff:ff:ff
    altname enp0s10
    altname ens10
    inet 172.29.101.245/24 brd 172.29.101.255 scope global eth3
       valid_lft forever preferred_lft forever
    inet 172.29.101.1/24 brd 172.29.101.255 scope global secondary eth3
       valid_lft forever preferred_lft forever

Workaround

Add both primary and secondary address to cloud.conf make it works.

dhcp-hostsfile=/etc/dhcphosts.txt
listen-address=127.0.0.1,172.29.101.1,172.29.101.245

Or patch /opt/cloud/bin/cs/CsDhcp.py in virtual router

            if self.cl.is_redundant():
                listen_address.append(gateway)
                listen_address.append(ip)                            # <<<<< add this
            else:
                listen_address.append(ip)

[weizhouapache]

@hulucc
thanks for the reporting and sharing of workaround

[DaanHoogland]

thanks @hulucc , I created PR with your workaround, and we will test this for regressions.

see #10183