Error when using Shim (at least) 15.8 and DHCP Option 67 with non-Microsoft DHCP server

[MarkusSpier]

When attempting to boot Shim version 15.8 via DHCP and Option 67, an error occurs. In conjunction with a non-Microsoft DHCP server, the path is set incorrectly.

For example, if you set the path in Option 67 to: <some_directory>\shim_x64.efi so that it should look for grub2 in a subfolder, an error occurs. Grub2 is incorrectly requested directly from the root directory without the subfolder.

The error seems to occur because, with Option 67 set, the filename from the header is still used. If this is empty, subsequent requests are always made without the subfolder!

Since Microsoft DHCP apparently also sets the header by default when setting Option 67, this error is not observed here. However, if you use, for example, IPFire DHCP and only set Option 67 (here "option bootfile-name"), the header (here "filename") remains empty.

Actually, Shim should use the path set in Option 67 and not the one from the header in this scenario!

This error does not occur in an older version (15.4). It must have been introduced between 15.4 and 15.8.

Is this error known and will it be fixed in the next version?

Example in Microsoft DHCP:

Example in IPFire DHCP:

If you set here only the option 67, the error occurs. If you set also the header (filename) it works (like in this image)

[dennis-tseng99]

As you know, DHCP OFFER or ACK option 66 and 67 will provide the exact shim.efi location. To make sure IPFire DHCP server can generate the correct option 67, would you please capture and show the content of option 67 by using Wireshark(for example) when sending OFFER or ACK message out ?

Typically grub.efi located in the same directory where shim.efi resides.
Would you please list files under subfolder directory ?

[MarkusSpier]

@dennis-tseng99

The files in the subfolder are only the shim and the grub as you can see:

Here is wireshark:

The file that is requested is located in the subfolder, and in IPFire configured only as "option bootfile-name" WITH the subfolder. As soon as you also configure the "filename" the same as the "option bootfile-name", the request will be send including the subfolder.

Many of our customers with NON Microsoft DHCP servers have this problem after they updated to the new shim version. So we think, this is a common thing to NOT configure the filename option. But on Microsoft DHCP serves, as they set this option as default (as I mentiont in my first post) it works all fine.

[dennis-tseng99]

In your Wireshark, the Read Request(line 109) of TFTP reveals that the option 67 in OFFER message sent from IPFire server has successfully received and offered the subfolder \bblefi-x64\shim-x64.efi information. But after that, the firmware provides an incorrect virtual file system to the PXE. Hence, shim fallbacks to DEFAULT_LOADER(root directory) which is included in built time. Anyway, did you change the firmware version between shim-15.4 and 15.8 ? Is grub2_x64.efi accessible ?

[MarkusSpier]

No, the firmware is the same. Also, many of our customers run into this problem after updating to the new shim version and they using the same environment as before. Same hardware and so on.
If you use the old shim version, there is no problem. Also, if you set the "filname" in IP-Fire, there is no problem.

So, we think that this is a bug in shim and not a problem with firmware as "all" firmware seems to behaves like this. This is the case in VMs and on hardware.

[MarkusSpier]

@dennis-tseng99 Perhaps, I can clarify this by provide you this snippets.
So, as you can see here, the option 67 is set properly to the right filename BUT, the Bootfilename field is empty:

This is the offer, but it is the same behavior in ACK, the bootfilename field does not contain any value.

When the bootfilename is empty, as in the screenshot shown, the grub is requested in the wrong directory. It should be requested in the directory given in the option 67. When we set the bootfilename explicitly in the header with the same value as option 67, shim will request the grub correctly.

Can you confirm that this is a bug in the shim or by design?

[dennis-tseng99]

@MarkusSpier Some DHCP clients can work with opt-67 correctly, but some legal clients do NOT conform the DHCP standard who only read boot file name(BOOTP header) and skip option 67, specific in grub2:

https://github.com/rhboot/grub2/commit/93289dc67c7e213b21df0eb09afea5e3b00ad7df#diff-93cd75cf8712d66c15ae5885f7cac5dae3531aaef211a17ef9885eb443ecdb0b

For debug purpose, you could print ImagePath and PathName in read_image() to make sure shim can get the correct opt-67 to get grub.

[baramundi-jpovolni]

@dennis-tseng99 Hi Dennis,

I'm a colleague of Markus, and I appreciate you taking the time to respond to our issue posts, we truly value your input. I'd like to clarify the problem again, as there seems to be some misunderstanding about the exact issue with Shim.

Until recently, we used an older version of Shim (version 15.4), which worked perfectly. However, after updating to Shim version 15.8, we encountered the following issue:
Our customers use a variety of DHCP servers, which leads to differences in how headers and options are constructed. For example, in first image bellow, you can see how the DHCP ACK is built by "IPFire DHCP". The image includes a Wireshark screenshot of the ACK and a partial screenshot of a TFTP request for the grub_x64.efi file. The Problem is the following:

• The Shim loads and starts successfully on the client side.
• It then attempts to fetch the revocations.efi file via TFTP. This step fails because the file doesn't exist, but that's acceptable since the file is optional.
• The issue occurs during the final step when fetching the grub_x64.efi file (as shown in the green highlighted request). The source file path is incorrectly set to "grub_x64.efi". The source file path should be "\bblefi-x64\grub_x64.efi", matching the folder structure specified in the DHCP ACK's Option 67. The Shim should recognize its root folder and use it consistently for subsequent TFTP requests to fetch both revocations.efi and grub_x64.efi.

The problem lies in how Shim determines the its root folder path. Instead of using the value from Option 67, Shim relies solely on the "Boot file name" field in the DHCP header. This behavior works with Microsoft's DHCP server, where the "Boot file name" field is automatically set to match Option 67. However, with other DHCP servers, like IPFire, the "Boot file name" field is not automatically synchronized with Option 67. In such cases, the field must be explicitly set, as Markus described in his original post.

After explicitly setting the "Boot file name" field value, as Markus pointed out in the first post, everything works as expected, and Shim constructs the paths for both revocations.efi and grub_x64.efi correctly. Which can be seen in the image bellow.

Our question to the community is: Is this behavior by design, or was something overlooked during the implementation of Shim's root folder parsing, making this a potential bug? We find it quite strange that Shim does not use the Option 67 value to determine the correct path for subsequent files but instead relies on the "Boot file name" field from the header. It would be helpful if someone could provide an explanation for why this approach was chosen, assuming it was done intentionally.

[dennis-tseng99]

@MarkusSpier Thanks for your so detail information, but I cannot open your private wireshark image. Actually, shim just makes use of network protocol implemented in UEFI firmware via gnu-efi, and it doesn't change the DHCP/PXE network protocol at all. Rather, it is the firmware changes the parsing logic. Just remind that the "boot file name" field in the header is mandatory; while opt-67 is an optional TLV. Anyway, let's find out what happen to the firmware first.

[xmawaba]

Hi @dennis-tseng99,
i am also a colleague of the two above. Thanks for helping us out here.
First of all i also had problems opening the images last week, but today it worked. Maybe some issue with github?

In the image below (hopefully you can open it), there is the ip fire configuration on the left side and the boot screen on the right side.
In the ip fire configuration the boot filename header is set to a path which doesnt exist and opt 67 to the correct path. In the boot screen on the right one can see that the shim is downloaded using the correct path from opt 67 but the grub is trying to be downloaded from the not existing path which was set in the boot file name.

So the fields which are used are differing. Our assumption was that the path to download shim is created by the firmware (which works correctly), but the path which is created to downloaded the grub is created by shim (which uses the wrong one). Is that incorrect?

Thanks and best regards.

[dennis-tseng99]

@xmawaba Sorry for the late response. I compared two versions of related files - netboot.c in shim-15.8 and shim-15.4, and they are identical. BTW, if you want to clarify whether bootfilename or opt-67 is provided by the firmware for grub2_x64, please print the variables full_path, dir, and name in parseDhcp4() function, where full_path = dir + name. Currently, I can see both versions make use of the dir value from bootp header.

UINT8 *dir = pkt_v4->BootpBootFile;

[xmawaba]

Hi @dennis-tseng99 ,
thanks for your reply. First of all the "working" version was not 15.4 but the quite old 0.8 which was not correclty stated above by us. But I did some further investigation with the following result:

The change of behaviour was caused by a mixture of the set DEFAULT_BOOTLOADER value and the new backslash support in the newer shim versions which led to the directory being prepended to the DEFAULT_BOOTLOADER when backslashes (instead of slashes) are being used in the boot file name header. This i wouldn't consider a problem in shim, but with how it was used.

Nevertheless: Although shim always used the boot file name header, wouldn't it be a good idea to use opt 67 instead of the header, when set? As also the PXE specification is suggesting this "bootfile (128) * Can be overloaded if using Opt 67" [http://docs.smoe.org/misc/pxe-2.1-spec.pdf](url).

I run a test with a little test code which shows that the opt 67 would be correctly set when the header is not used/set incorrectly:

Debug-Log:

Testcode:

 static void PrintOpt(EFI_PXE_BASE_CODE_DHCPV4_PACKET* pkt_v4){
	UINTN type = 0;
	UINTN i = 0;
	while((type = pkt_v4->DhcpOptions[i]) != 255){
		UINT8 optLen = pkt_v4->DhcpOptions[i+1];

		if (type == 67){
			CHAR8* bootFileName = AllocateZeroPool(optLen + 1);
			CopyMem(bootFileName, &pkt_v4->DhcpOptions[i+2], optLen);
			bootFileName[optLen] = '\0';
			console_print(L"DEBUG: Opt 67 value: %a\n", bootFileName);
			FreePool(bootFileName);
			break;
		}

		i += optLen + 2;
	}
}