Presumably it displays a mesage to the effect that it can’t open the file because it’s been saved by a newer version of Excel then?
Yes and no.
If for example the file was saved in Office 2003 as an XLS then the encryption is RC4 which can be broken extremely quickly.
Pretending you email me the file and I save it in Office 2007. By default the encryption will remain in RC4 but I can change that manually be amending the Crypto Provider. I would be given a warning similar to the one below (I couldn’t find the right message) but instead of saying a “Minor loss of fidelity” it would warn me that the file could not be opened in earlier versions.
This is the Crypto Provider screen which can upgrade or downgrade the cryptography in an XLS (or XLSX) file. Most users would never understand or encounter this.
The latest versions of Excel have a number of formats and Excel is smart enough to recognise that in a 97-2003 Workbook that the encryption needs to be RC4 and the user would have to manually change the Crypto Provider to work otherwise.
XLS has always been a very flexible format but you’re correct about the specification, it’s extremely complicated.
For most users saving in the latest versions of Excel, with a secure password, they have no fear about the security unlike in olden times of Office. The default option is suitably secure even as AES128, 100,000 iterations at SHA1.