Detecting DOSBox from Inside DOSBox

Disassembling MOUNT.COM reveals byte sequences like FE 38 00 00, and DOSBox has placed a dedicated FE /7 callback opcode there.

• On regular x86, FE /7 is an invalid opcode, causing #UD.
• In DOSBox, this byte sequence is interpreted as a callback call, so it effectively behaves like a 4-byte NOP.

This property can be used to create a method for distinguishing DOSBox. After installing an INT 06h exception handler, check whether the bytes near the current IP are FE 38 00 00 when the exception occurs.

• If they match, the environment is not DOSBox, so skip past the IP and return normally.
• If no exception occurs and the code continues on, it's likely a DOSBox-family environment.

Initially the author tested on 86Box, but it behaved unexpectedly, and the cause was a bug carried over from PCem. It treated any non-zero ModR/M opcode modifier as FE /1, causing FE /2, FE /4, and FE /7 to all execute like DEC. This issue has already been fixed in the upstream repository.

The author considers this technique a far more intrinsic way to identify DOSBox than simply checking BIOS strings, and notes that while more common DOS environments (NTVDM, Win9x DOS Prompt, DOSEMU, etc.) are actually easier to distinguish, this case was the trickiest one.