I know this is an older post but i am havign an ntvdm issue too.
I am trying to install some applications, i have installed them under windows xp dozens of times before with no issues at all on the same pc, i have never needed to adjust the properties or run them in compatability mode (even when doing so they still do not run)
I do not receive any error messages and there is nothing recorded in the log files, quite simply i double click the installer from the CD, the hour glass appears for a few seconds and the installer just drops out of the system, ntvdm appears for a few seconds in task manager then dissapears.
if i try to install the same application on a virtual machine version of windows running on the same pc there is no issue and it installs and loads straight away.
There are no virus' or malware, i have chjecked with adaare, malware bytses and spybot, i use avira anti virus and comodo firewall which i have used with this application before and even when they have been uninstalled have the same problems.