Clearing up an AD Lightweight Directory Service error on vCenter Server systems

Recently I was onsite with a customer helping them deploy a new vSphere 5.1 environment to host a new Exchange 2010 system. As part of the deployment, we setup Alan Renouf’s vCheck 6 script and started working through the process of setting it up to run as a scheduled task. As we were manually running the task we noticed that the output showed errors every minute for the AD Web Services and AD Lightweight Directory Services (ADAM).

We found the log entries in the AD Web Services log.

A little digging uncovered that the event 1209 error is reported when there is a problem with the port numbers in the registry for AD Web Services LDAP access (389/636).
http://blogs.technet.com/b/askds/archive/2010/04/09/friday-mail-sack-while-the-ned-s-away-edition.aspx#adws

On inspection of the registry key, the “Port SSL” type is incorrect and the data is missing. According to the Technet blog post, the value type should be “REG_DWORD” and the default data is 636.

I deleted the existing incorrect value and created a new value with the REG_DWORD type and the value data of 636 decimal.

Upon checking the Windows event logs, I could see that the AD Web Services was already using the corrected value, so no service restart was required.

The next log entry displayed the VCMSDS instance and LDAP/LDAPS (SSL) ports it is configured to use.

After this vCenter system was fixed, we checked all of the other vCenter servers onsite and found that their vCenter 4.1 server they were using for non-production also had the same error. That vCenter server was running on a Windows 2003 server and we did have to stop and restart the AD Web Services service to load the corrected SSL port value and resolve the error.

Thanks to Alan Renouf and the vCheck contributors at Virtu-Al.net for grabbing and displaying this error.

VMware vSphere 4.0 Update 2 is released

This evening VMware released Update 2 for ESX/ESXi 4, vCenter Management Server 4, vCenter Update Manager 4 and VMware Data Recovery.
A quick scan of the ESX 4 Update 2 release notes shows expanded support for FT on Intel i3/i5 Clarkdale, Xeon 34xx Clarkdale and Xeon 56xxx processors. Support for IOMMU on AMD Opteron 61xx and 41xx processors. Guest OS support for Ubuntu 10.04 and improvements to esxtop and resxtop to include NFS performance statistics Reads/s, Writes/s, MBRead/s, MBWrtn/s, cmd/s and gavg/s latency. Included in the resolved issues is a change in the way the Snapshot Manager “Delete All” operation works. In previous versions the snapshot farthest away from the base disk was committed to its immediate parent, then that parent would be committed to its parent until the last remaining snapshot is committed to the base. The release notes report that this operation will now start with the snapshot closest to the base disk and work toward the farthest. This should reduce the amount of disk space required during the “delete all/commit” operation and reduce the amount of data that is repeatedly committed. I think this is a great change. I have seen customers run out of space in datastores when the failed to keep track of active snapshots and didn’t understand the “delete all/commit” process.

The vCenter Management Server 4 Update 2 release notes list support for guest customization of:

◦Windows XP Professional SP2 (x64) serviced by Windows Server 2003 SP2
◦SLES 11 (x32 and x64)
◦SLES 10 SP3 (x32 and x64)
◦RHEL 5.5 Server Platform (x32 and x64)
◦RHEL 5.4 Server Platform (x32 and x64)
◦RHEL 4.8 Server Platform (x32 and 64)
◦Debian 5.0 (x32 and x64)
◦Debian 5.0 R1 (x32 and x64)
◦Debian 5.0 R2 (x32 and x64)

Among the resolved items, there is an update JRE (1.5.0_22) and number of fixed related to Host Profiles, support for vSwitch portgroup named longer than 50 characters, advanced settings to allow the use vDS connections as additional HA heartbeat networks, the addision of a parameter in vpxd.cfg to set a greater timeout value for VMotion operations involving VMs with swap files on local datastores, among many others. In the known issues section is astatement that while USB controllers can be added to VMs, attaching USB devices is not supported and that vSphere Web Access is experimentally supported.

The vCenter Update Manager 4 Update 2 release notes list improvement of operations in low bandwidth, high latency and slow networks, including a reference to KB 1017253 detailing the configuration of extended timeout values for ESX, vCenter and Update Manager Update 2.
The compatability matrix shows that Update Manager 4 Update 2 is only compatible with vCenter Management Server 4 Update 2.

VMware Data Recovery Update 2 includes the following new items:

The following enhancements have been made for this release of Data Recovery.

•File Level Restore (FLR) is now available for use with Linux.
•Each vCenter Server instance supports up to ten Data Recovery backup appliances.
•The vSphere Client plug-in supports fast switching among Data Recovery backup appliances.
•Miscellaneous vSphere Client Plug-In user interface enhancements including:
◦The means to name backup jobs during their creation.
◦Additional information about the current status of destination disks including the disk’s health and the degree of space savings provided by the deduplication store’s optimizations.
◦Information about the datastore from which virtual disks are backed up.

The support for up to 10 Data Recovery appliances per vCenter will allow up to 1000 jobs (100 per appliance x10), this is a significant increase in backup capacity.

The build numbers for the various items are:

ESX 4.0 Update 2 Build 261974
ESXi 4.0 Update 2 Installable Build 261974
ESXi 4.0 Update 2 Embedded Build 261974
VMware Tools Build 261974
vCenter Server 4.0 Update 2 Build 258672
vCenter Update Manager 4.0 Update 2 Build 264019

vSphere 4 Update 2 components can be downloaded here.

Licensing VMware ESX 4, ESXi and vCenter 4 Video KB article

This week VMware posted KB article 1010839 on licensing ESX 4, ESXi 4 and vCenter 4. I get many questions in class about the new license assignment process for vSphere. This KB article has a nice video demonstration and very concise text direction for assigning licenses.

New VMware network technical papers published

Network Segmentation in Virtualized Environments

As virtualization becomes the standard infrastructure for server deployments, a growing number of organizations want to consolidate servers that belong to different trust zones. The demand is increasing for information to help network security professionals understand and mitigate the risks associated with this practice. This paper provides detailed descriptions of three different virtualized trust zone configurations and identifies best practice approaches that enable secure deployment.

DMZ Virtualization Using VMware vSphere 4 and the Cisco Nexus 1000V Virtual Switch

This paper tackles the subject of DMZ security and virtualization. It covers a number of DMZ security requirements and scenarios, presenting how vSphere users can implement the Cisco Nexus 1000V virtual switch in a DMZ.

VMware Security Advisory 2009-0008

VMware has released security advisory VMSA-2009-0008. The advisory is for a vulnerability in an MIT Kerberos 5 package in the service console. The advisory explains:

An input validation flaw in the asn1_decode_generaltime function in MIT Kerberos 5 before 1.6.4 allows remote attackers to cause a denial of service or possibly execute arbitrary code via vectors involving an invalid DER encoding that triggers a free of an uninitialized pointer. A remote attacker could use this flaw to crash a network service using the MIT Kerberos library, such as kadmind or krb5kdc, by causing it to dereference or free an uninitialized pointer or, possibly, execute arbitrary code with the privileges of the user running the service.
NOTE: ESX by default is unaffected by this issue, the daemons kadmind and krb5kdc are not installed in ESX.

The advisory goes on to state that all currently supported version of ESX (not ESXi) are affected.
For ESX 3.5 the patch: ESX 3.5.0 ESX350-200906407-SG
md5sum: 6b8079430b0958abbf77e944a677ac6b
KB Article: VMware ESX 3.5, Patch ESX350-200906407-BG: Updates krb5-libs and pam_krb5

For ESX 2.5.5, 3.0.2, 3.0.3 and 4.0 patches are pending.

You can subscribe to VMware Security announcments here: http://lists.vmware.com/mailman/listinfo/security-announce