MDaemon 9.6.2/9.6.1 DEP configurations

A very common set up instruction-

MDaemon won’t start (or crashes) when Windows 2003 SP1 or Windows XP SP2 is installed.

The recent release of Microsoft server SP1 for Windows 2003 and SP2 for Windows XP has DEP, which conflicts with various components of MDaemon (all versions). The problem is that the MDaemon.exe (or MDSpamD.exe or CFengine.exe, for example) crashes and/or gets dumped from memory and restarted.

DEP is a new mechanism recently introduced to try and detect software that appears to use memory in what Microsoft deems to be a  ‘suspicious’ way. DEP does require newer hardware and needs a special type of CPU to function. The DEP detects our product as a threat and dumps any process that is using it. That’s why MDaemon/CFEngine/MDSpamD/etc. will get unloaded (or crashed purposely by Windows to force it to exit).

Configuring DEP to exclude MDaemon and its related services from its memory monitoring will completely solve the issue.

These are the services to exclude (and their default directory paths – your installation may vary):

\MDaemon\app\MDaemon.exe
\MDaemon\app\CFEngine.exe
\MDaemon\SpamAssassin\MDSpamD.exe
\MDaemon\WorldClient\WorldClient.exe
\MDaemon\WebAdmin\WebAdmin.exe
\MDaemon\SecurityPlus\ScanningProcess.exe (will only be present if you have installed SecurityPlus)

Here are some steps that explain how to disable DEP or to add MDaemon’s processes to the exemption list.

Data Execution Prevention (DEP)

Service pack 2(XP) and Service Pack 1(2003) contain a new security technology that prevents the execution of data as a program. The purpose is to make it impossible for malware like viruses and Trojans to exploit program errors and other security holes, for example buffer overflows.

Not all processors support hardware DEP. Currently only the AMD Athlon 64, AMD Sempron (mobile), AMD Opteron, and the Intel Itanium server processor support it. But at least for some Windows components there is a software implementation of DEP that can be enabled on all computers running Service Pack 2.

End users who are logged on as administrators can manually configure DEP between the OptIn and OptOut policies using the Data Execution Prevention tab inside the System Properties dialog box. The following procedure describes how to manually configure DEP on the computer:

Click Start, click Control Panel, and then double-click System.
Click the Advanced tab. Then, under Performance, click Settings.
Click the Data Execution Prevention tab.
Click Turn off hardware DEP (software DEP enabled) to select the Opt-in policy.
Click Hardware and software DEP enabled for all programs except to select the OptOut policy.
Click Add and add the applications that you do not want to use DEP with.
If you cannot boot or if DEP itself prevents you from using the dialog described above or if you want to disable DEP entirely, please read the chapter Boot or shutdown problems after Service Pack 2 installation above, subchapter ‘Data Execution Prevention prevents complete booting or shutting down’.

Here the information from Microsoft’s KB article on setting up the exemption.

Hardware-enforced DEP

Hardware-enforced DEP marks all memory locations in a process as non-executable unless the location explicitly contains executable code. A class of attacks exists that tries to insert and run code from non-executable memory locations. DEP helps prevent these attacks by intercepting them and raising an exception.

Hardware-enforced DEP relies on processor hardware to mark memory with an attribute that indicates that code should not be executed from that memory. DEP functions on a per-virtual memory page basis, and DEP typically changes a bit in the page table entry (PTE) to mark the memory page.

Processor architecture determines how DEP is implemented in hardware and how DEP marks the virtual memory page. However, processors that support hardware-enforced DEP can raise an exception when code is executed from a page that is marked with the appropriate attribute set.

Advanced Micro Devices (AMD) and Intel have defined and shipped Windows-compatible architectures that are compatible with DEP.

Beginning with Windows XP SP2, the 32-bit version of Windows uses one of the following:
The no-execute page-protection (NX) processor feature as defined by AMD.
The Execute Disable Bit (XD) feature as defined by Intel.
To use these processor features, the processor must be running in Physical Address Extension (PAE) mode. However, Windows will automatically enable PAE mode to support DEP. Users do not have to separately enable PAE by using the /PAE boot switch.

Software-enforced DEP

An additional set of Data Execution Prevention security checks have been added to Windows XP SP2. These checks, known as software-enforced DEP, are designed to block malicious code that takes advantage of exception-handling mechanisms in Windows. Software-enforced DEP runs on any processor that can run Windows XP SP2. By default, software-enforced DEP helps protect only limited system binaries, regardless of the hardware-enforced DEP capabilities of the processor.

Benefits

The primary benefit of DEP is that it helps prevent code execution from data pages, such as the default heap pages, various stack pages, and memory pool pages. Typically, code is not executed from the default heap and the stack. Hardware-enforced DEP detects code that is running from these locations and raises an exception when execution occurs. If the exception is unhandled, the process will be stopped. Execution of code from protected memory in kernel mode causes a Stop error.

DEP can help block a class of security intrusions. Specifically, DEP can help block a malicious program in which a virus or other type of attack has injected a process with additional code and then tries to run the injected code. On a system with DEP, execution of the injected code causes an exception. Software-enforced DEP can help block programs that take advantage of exception-handling mechanisms in Windows.

System-wide configuration of DEP

DEP configuration for the system is controlled through switches in the Boot.ini file. If you are logged on as an administrator, you can now easily configure DEP settings by using the System dialog box in Control Panel.

Windows supports four system-wide configurations for both hardware-enforced and software-enforced DEP.

OptIn – This setting is the default configuration. On systems with processors that can implement hardware-enforced DEP, DEP is enabled by default for limited system binaries and programs that ‘opt-in.’ With this option, only Windows system binaries are covered by DEP by default.
OptOut – DEP is enabled by default for all processes. You can manually create a list of specific programs that do not have DEP applied by using the System dialog box in Control Panel. Information technology (IT) professionals can use the Application Compatibility Toolkit to ‘opt-out’ one or more programs from DEP protection. System compatibility fixes, or shims, for DEP do take effect.
AlwaysOn – This setting provides full DEP coverage for the whole system. All processes always run with DEP applied. The exceptions list to exempt specific programs from DEP protection is not available. System compatibility fixes for DEP do not take effect. Programs that have been opted-out by using the Application Compatibility Toolkit run with DEP applied.
AlwaysOff – This setting does not provide any DEP coverage for any part of the system, regardless of hardware DEP support. The processor does not run in PAE mode unless the /PAE option is present in the Boot.ini file.
Hardware-enforced and software-enforced DEP are configured in the same manner. If the system-wide DEP policy is set to OptIn, the same Windows core binaries and programs will be protected by both hardware-enforced and software-enforced DEP. If the system cannot use hardware-enforced DEP, the Windows core binaries and programs will be protected only by software-enforced DEP.

Similarly, if the system-wide DEP policy is set to OptOut, programs that have been exempted from DEP protection will be exempted from both hardware-enforced and software-enforced DEP.

The Boot.ini file settings are as follows:
/noexecute=policy_level
Note policy_level is defined as AlwaysOn, AlwaysOff, OptIn, or OptOut.

Existing /noexecute settings in the Boot.ini file are not changed when Windows XP SP2 is installed. These settings are also not changed if a Windows operating system image is moved across computers with or without hardware-enforced DEP support.

During installation of Windows XP SP2, the OptIn policy level is enabled by default unless a different policy level is specified in an unattended installation. If the /noexecute=policy_level setting is not present in the Boot.ini file for a version of Windows that supports DEP, the behavior is the same as if the /noexecute=OptIn setting was included.

If you are logged on as an administrator, you can manually configure DEP to switch between the OptIn and OptOut policies by using the Data Execution Prevention tab in System Properties. The following procedure describes how to manually configure DEP on the computer:
Click Start, click Run, type sysdm.cpl, and then click OK.
On the Advanced tab, under Performance, click Settings.
On the Data Execution Prevention tab, use one of the following procedures:
Click Turn on DEP for essential Windows programs and services only to select the OptIn policy.
Click Turn on DEP for all programs and services except those I select to select the OptOut policy, and then click Add to add the programs that you do not want to use the DEP feature.
Click OK two times.
IT professionals can control system-wide DEP configuration by using a variety of methods. The Boot.ini file can be modified directly with scripting mechanisms or with the Bootcfg.exe tool that is included in Windows XP SP2.

For unattended installations of Windows XP SP2, you can use the Unattend.txt file to pre-populate a specific DEP configuration. You can use the OSLoadOptionsVar entry in the [Data] section of the Unattend.txt file to specify a system-wide DEP configuration.

Per-program DEP configuration

For the purposes of program compatibility, you can selectively disable DEP for individual 32-bit programs when DEP is set to the OptOut policy level. To do this, use the Data Execution Prevention tab in System Properties to selectively disable DEP for a program.

For IT professionals, a new program compatibility fix that is named DisableNX is included with Windows XP SP2. The DisableNX compatibility fix disables Data Execution Prevention for the program that the fix is applied to.