Assuming that the person pulling the disk doesn't have the passphrase, what's the difference?

Assuming things are configured correctly, the TPM would only provide the key to unlock the drive if the correct OS has been booted. Attempting to boot from a live usb to have a look at the data wouldn't work, since the TPM wouldn't present the key required.

You can see https://wiki.archlinux.org/title/Trusted_Platform_Module#PCR... for an example of doing so