Upgrading vCenter Server to 5.5 in my lab – part 3

Upgrade vCenter Inventory Service

This is the third post in a series of articles chronicling the process of upgrading my personal vSphere lab system from 5.1 to 5.5.

Before you get started with the upgrade, you should read the following KB article:

Upgrading to vCenter Inventory Service 5.5 on a Microsoft Windows platform (2058272)

Information Required for vCenter Inventory Service Installation or Upgrade

Upgrade Procedure:

Mount the vSphere 5.5 installation media.
In the left pane, under Custom Install, click vCenter Inventory Service and then click Install.
inventory-service55-01
Note: If any of the prerequisites are not met, they are listed in the right pane under Prerequisites.

Select the appropriate language from the dropdown and click OK.
inventory-service55-02

In the welcome screen, click Next.
inventory-service55-04
Note: If a previous version of vCenter Inventory Service is installed, the welcome screen displays the message:
An earlier version of vCenter Inventory Service is already installed on this computer and will be upgraded to vCenter Inventory Service 5.5.
If the distinguished name for any existing SSL certificate is not unique, the old SSL certificate will be replaced with a new certificate.

Review the End User License Agreement. If you agree, select the I accept the terms in the license agreement option and click Next.

Select one of these options when presented with a database re-initialization warning:

    • Keep my existing database – Select this option to retain your existing data in the vCenter Inventory Service.  (I selected this to keep my existing vCenter inventory data.)
    • Replace my existing database – Select this option to replace the existing database. You must acknowledge that a backup of the Inventory Service database and SSL certificates has been performed.

In the dialog that appears, check if the Fully Qualified Domain Name (FQDN) of the host where vCenter Inventory Service is displayed. This field is auto-populated.
inventory-service55-06

Notes from the KB article:

    • If this field is not auto-populated, enter the FQDN manually.
    • If there is an issue with DNS resolution, you see the error:
      The Fully Qualified Domain name could not be resolved using DnsQuery API..
      If you see this error, resolve the DNS resolution issue before proceeding.

Click Next.

Optionally, provide an alternative TCP port number for the vCenter Inventory Service and click Next.
inventory-service55-07
Note: Changing the default ports is recommended only if you have an unchangeable port conflict in the same system.

Select the appropriate JVM memory size, depending on the environmental requirements. Click Next.
inventory-service55-08

Enter the SSO lookup Service URL and the SSO administrator username and password. You may be prompted to install the Lookup Service Certificate. Click Next.
inventory-service55-09

Notes from the KB article:

When prompted, verify the fingerprint of the certificate and click Yes to proceed.
inventory-service55-10
In the Ready to Install screen, click Install.
inventory-service55-11

Installing…
When the installation completes, click Finish.
inventory-service55-13
Installed programs showing the upgraded vCenter Inventory Service.
inventory-service55-14

Next up – the main event. Upgrading the vCenter Server to 5.5.

 

Upgrading vCenter Server to 5.5 in my lab – part 2

Upgrading vSphere Web Client

This is the second post in a series of articles chronicling the process of upgrading my personal vSphere lab system from 5.1 to 5.5.

Before you get started with the upgrade, you should read the following KB article:

Upgrading to vSphere Web Client 5.5 on a Microsoft Windows platform (2058265)

Information Required for the vSphere Web Client Installation

Upgrade procedure:

Mount the vSphere 5.5 installation media. (As before, I prefer to use “Run as Administrator.”)

In the left pane, under Custom Install, click vSphere Web Client and then click Install.
webclient55-01

Note: If any of the prerequisites are not met, they are listed in the right pane under Prerequisites.

Select the appropriate language from the dropdown and click OK.
webclient55-02

Note: The installer window may disappear for a lengthy period of time, but is still running in the background.

In the welcome screen, click Next.
webclient55-03

If a previous version of the vSphere Web Client is installed, the welcome screen displays this message:
An earlier version of vSphere Web Client is already installed on this computer and will be upgraded to vSphere Web Client 5.5

Review the End User License Agreement. If you agree, select the I accept the terms in the license agreement option and click Next.

Optionally, provide an alternative TCP port number for the vSphere Web Client Service and click Next.
webclient55-04

Note: Changing the default ports is recommended only if you have an unchangeable port conflict in the same system.

Enter the SSO Lookup Service URL and SSO administrator username and password. You may be prompted to install the Lookup Service Certificate. Click Next.
webclient55-05

Notes from the KB article:

When prompted, verify the fingerprint of the certificate and click Yes.
webclient55-06

In the Ready to Install screen, click Install.
webclient55-07

Installing…

When the installation completes, click Finish
webclient55-08

Success!
webclient55-09

The vSphere Web Client has been upgraded to 5.5.

Next up is the upgrade of the vCenter Inventory Service.

Upgrading vCenter Server to 5.5 in my lab – part 1

Upgrading vCenter Server 5.1 update 1 to 5.5 in my lab
Part 1

This past summer I had a student in one of my vSphere 5.1 Install, Configure and Manage courses who had attempted an upgrade from an earlier version of vSphere to 5.1 with disastrous results. This fellow was hopping mad and was not impressed by me asking him if he had read the upgrade guides or knowledgebase articles covering the proper sequence and compatibility issues he had encountered.

So with him in mind, I am documenting the process I followed to upgrade my lab vCenter Server and related components to vCenter Server 5.5.

My vCenter system is split between two windows 2008 R2 VMs. The vCenter server, SSO, Inventory service, ESXi Dump collector and vSphere Syslog Collector are on “vc5.mylab.local,” while the vSphere Web Client and vCenter Update Manager are installed on “app-01.mylab.local.” I have another Windows 2008 R2 VM running MS SQL Server 2008 R2 that handles the vCenter and Update Manager databases.

Here are the beginning software versions on my Windows 2008 R2 vCenter VM
vc55upgrade-01

In order to avoid the difficulties my former student encountered with a similar upgrade, check out the following before getting started with the upgrade:

VMware Product Interoperability Matrixes

vSphere Upgrade Guide

Best Practices for vCenter Server Upgrades

Required Information for Installing or Upgrading vCenter Single Sign-On, Inventory Service, vCenter Server and the vSphere Web Client  — Print this topic and use it as a worksheet for the upgrade and save it for later so you have the answers for the next upgrade!

Update sequence for vSphere 5.5 and its compatible VMware products (2057795)

Upgrade the various products in numerical order.

vc55upgrade-02

From the KB article:
Before you update the vCenter Server, disable vCenter Server from vCloud Director. Also ensure that you stop or disable other VMware services so that they do not communicate with vCenter Server during the update process.

In this article I will be running through the upgrade to vCenter Server 5.5 and its related products. The VMware KB article Methods of upgrading to vCenter Server 5.5 (2053130) describes the requirements for upgrading to vCenter 5.5. I will be following the order specified in the section “Upgrading components separately for vCenter server 5.5”:

  1. Upgrade vCenter Single Sign-On.
    For more information, see Upgrading to vCenter Single Sign-On 5.5 on a Microsoft Windows platform (2058249).
  2. Upgrade vSphere Web Client.
    For more information, see Upgrading to vSphere Web Client 5.5 on a Microsoft Windows platform (2058265).
  3. Upgrade vCenter Inventory Service.
    For more information, see Upgrading to vCenter Inventory Service 5.5 on a Microsoft Windows platform (2058272).
  4. Upgrade vCenter Server.
    For more information, see Upgrading to vCenter Server 5.5 on a Microsoft Windows platform (2058275).
  5. Upgrade vCenter Update Manager.
    For more information, see Upgrading to vSphere Update Manager 5.5 on a Microsoft Windows platform (2058423).

 

Upgrading vCenter Single Sign-On

Read the KB article below:
Upgrading to vCenter Single Sign-On 5.5 on a Microsoft Windows platform (2058249)

Information Required for vCenter Single Sign-On Installation

Follow the steps outlined in this KB article to back-up the existing 5.1 SSO database:

Backing up and restoring the vCenter Single Sign-On (SSO) 5.1 configuration (2034928) (You will need the “old” SSO admin password for this.) You won’t need the password for the upgrade.

You need to make sure that forward and reverse DNS lookups return the correct information about your SSO server and the vCenter server. See DNS Requirements for vSphere.

Additionally, you need to check the SSL certificate subject name and the registry of the VM running Single Sign-on. In my case SSO and vCenter server are installed on the same VM.

The registry key to check is referenced in this KB article:
Upgrade from vSphere 5.1 to vSphere 5.5 rolls back after importing Lookup Service data (2060511)

Check the HKEY_LOCAL_MACHINE\Software\VMware, Inc.\VMware Infrastructure\SSOServer\FQDNIp registry key value:

  • If the registry key value is an IP address, this issue might affect your system.
  • If the registry key value is set to the FQDN value you see in the certificate, your system is       not affected by this issue.

The installer will perform a prerequisite check, but it will not stop you from proceeding if the values do not match.

For example:

vc55upgrade-04

If there is a mismatch, you need to change the registry value to match the “Subject” name used for the SSL certificate.

vc55upgrade-03

 

Upgrade procedure:

Mount the vSphere 5.5 installation media.

Note: I prefer to browse the install media and launch the “autorun.exe” with the “Run as Administrator.”
If you are prompted by User Account Control, click Yes.

In the left pane, under Custom Install, click Single Sign-On Install and then click Install.
vc55upgrade-07

Note: If any of the prerequisites are not met, they are listed in the right pane under Prerequisites.

In the welcome screen, click Next.
vc55upgrade-09

If a previous version of vCenter Single Sign-On (SSO) is installed, you see this message in the welcome screen:
An earlier version of vCenter Single Sign-On is already installed on this computer and will be upgraded to vCenter Single Sign-On 5.5.0.

Review the End User License Agreement. If you agree, select the I accept the terms in the license agreement option and then click Next.

Review the Prerequisites check screen and then click Next.
vc55upgrade-11

The installer notifies that a previous version is detected and that Users, Groups, Solution Users, and Lookup Service artifacts will be migrated.

Click Next.
vc55upgrade-12

Select a deployment mode and click Next.
vc55upgrade-13

vCenter Single Sign-On for your first vCenter Server – Select this option to deploy your first SSO server. This server becomes the first SSO server in a new vSphere authentication domain.

After you select this option:
Provide a password for the SSO administrator user and click Next.
vc55upgrade-14

Note from the KB article: This dialog shows information related to a domain by the name vsphere.local.  This is not a domain that is auto-detected within the existing environment, but a net new domain used      internally by vSphere. The administrator@vsphere.local account performs the same function as the admin@System-Domain account in previous versions of vSphere. For more information about the administrator@vsphere.local account,  see the vSphere Software Components section of the vCenter Server and Host Management Guide.

Provide a site name and click Next.
vc55upgrade-15

Note: The site name is used in environments where there are SSO servers in multiple sites. Ensure to select this name carefully because it cannot be changed in the vSphere Web Client after the installation  completes. “Default-First-Site” is the default site name.

Optionally, provide an alternative installation location and then click Next. (I had previously installed SSO to the “D:” drive on my VM.)
vc55upgrade-18

Notes:
The installation requires 2 GB of disk space to be available.
In the vCenter Single Sign-On Information screen, click Install.
vc55upgrade-19

Installing…

vc55upgrade-20

When the installation completes, click Finish.

vc55upgrade-21

vCenter SSO 5.5 upgraded!
vc55upgrade-22

The first component is now upgraded. Next up is upgrading the vSphere Web Client to 5.5.

Configure vSphere 5.1 for remote debug logging

Recently I have been working with customers on designs for new vSphere 5.1 installs and upgrades. As part of the design, I have been specifying the installation and configuration of the vSphere ESXi Dump Collector service on their Windows vCenter Server. The ESXi dump collector service allows the collection of the diagnostic dump information generated when an ESXi host has a critical fault and generates a “purple diagnostic screen.”

This post is a walk through of installing and configuring the ESXi Dump Collector service on vCenter and configuring an ESXi host to use it.

The Windows Server 2008 R2 VMs I use for vCenter are configured with additional drives for installing applications and storing data. In this example from my virtual lab, I have a “d:\” drive for applications and data.

Install the vSphere ESXi Dump Collector

The installer for the dump collector in included on the vCenter installer ISO image. I mount the ISO image to the Windows 2008 R2 VM where I have installed vCenter server.

Launch “autorun.exe” as an administrator.

From the VMware vCenter Installer, select “VMware vSphere ESXi Dump Collector”. Then click “Install” to begin the installation.

After the installer starts, select “English” as the language.

On the Welcome… page, click “Next >.”

On the End User Patent Agreement page, click “Next >.”
On the End User License Agreement page, select “I accept…”; click “Next >.”
On the Destination Folder page, click the “Change…” button beside “vSphere ESXi Dump Collector repository directory:”
On the Change Current Destination Folder page, change the “Folder name:” value to “d:\…”. Click “OK.”
Back on the Destination Folder page, observe that the path has been updated and click “Next >”

On the Setup Type page, select “VMware vCenter Server installation”, then click “Next >.”

On the VMware vCenter Server Information page, enter the appropriate information for connecting to vCenter. Click “Next >” to continue.

If you are using the default self-signed SSL certificate for vCenter, you will receive a message with the SHA1 thumbprint value for the vCenter server’s certificate.  Click “Yes” to trust that the certificate for connecting to the vCenter server.

You can verify the thumbprint by looking at the certificate properties on your vCenter server.  Notice that the thumbprint from the installer matches the thumbprint on the vCenter server’s certificate.

On the vSphere ESXi Dump Collector Port Settings page, click “Next >” to accept the default value of UDP port 6500.

On the vSphere ESXi Dump Collector Identification page, select the FQDN of the vCenter server and click “Next >.”

On the Ready to Install page, click “Install.”

After the installer has completed, click “Finish” on the Installer Completed page.

You can view the configured settings with the vSphere Client by selecting VMware ESXi Dump Collector from the Administration page.

You can also view the configuration with the vSphere Web Client by selecting the vCenter server, then browsing to the “Manage” tab and selecting “ESXi Dump Collector” under “Settings.”

Configuring an ESXi host to transmit the core dump over the network to the dump collector service.

Now that we have installed the dump collector service, we need to configure the ESXi hosts to send their diagnostic dump files to the vCenter server.
I set this up through the ESXi console. You will notice that I am logged in a “root” as I had not configured the ESXi host to use Active Directory authentication yet. Any user account that has the “administrator” role on the ESXi host can configure these settings.

First, checked the current coredump network configuration:

~ # esxcli system coredump network get
   Enabled: false
Host VNic:
Network Server IP:
Network Server Port: 0

Next, I confirmed the name of the vmkernel connection I planned to use: “vmk0” with the old “esxcfg-vmknic -l” command

Then, I configured the system to send coredumps over the network through the “vmk0” vmkernel port to my vCenter server’s IPv4 address at port 6500:

~ # esxcli system coredump network set –interface-name vmk0 –server-ipv4 10.0.0.51 –server-port 6500

You have to enter the interface name and server IPv4 address. The port is optional if you are using the default of 6500.

Then, I enabled the ESXi host to use the dump collector service:
~ # esxcli system coredump network set –enable true

I verified that the settings were correctly configured:
~ # esxcli system coredump network get
   Enabled: true
Host VNic: vmk0
Network Server IP: 10.0.0.51
Network Server Port: 6500

I checked to see if the server was running:
~ # esxcli system coredump network check
Verified the configured netdump server is running
~ #

Here is a screenshot of the process:

FYI, by default, the diagnostic dump file (core dump) is stored on a local disk partition of the ESXi host. You can find the local partition from the local ESXi console (if it is enabled) with the following command:

# esxcli system coredump partition get

I have highlighted the command in the figure below:

More information about managing the ESXi core dump disk partition is in the online documentation here.

Now that the vCenter server has the dump collector service installed and the ESXi host is configured to use it, I had to try it out!

Using the vsish tool and specific setting that Eric Sloof or NTPRO.NL described in his post “Lets create some Kernel Panic using vsish“, I crashed the ESXi host. As you can see in the screenshots, I was rewarded with a purple screen and success with transmitting the dump over the network to my vCenter server!

The “CrashME” PSOD

Here is the coredump file that was transmitted. Success!

The coredump file on the vCenter server in the repository

For more information check out these KB articles:

ESXi Network Dump Collector in VMware vSphere 5.x

Configuring the Network Dump Collector service in vSphere 5.x

vSphere page updated

I finally spent a little time and have updated the vSphere page to vSphere 5.1. I intend to build on the Knowledgebase article section as items come up with customers or from the classroom.

Clearing up an AD Lightweight Directory Service error on vCenter Server systems

Recently I was onsite with a customer helping them deploy a new vSphere 5.1 environment to host a new Exchange 2010 system. As part of the deployment, we setup Alan Renouf’s vCheck 6 script and started working through the process of setting it up to run as a scheduled task. As we were manually running the task we noticed that the output showed errors every minute for the AD Web Services and AD Lightweight Directory Services (ADAM).

We found the log entries in the AD Web Services log.

A little digging uncovered that the event 1209 error is reported when there is a problem with the port numbers in the registry for AD Web Services LDAP access (389/636).
http://blogs.technet.com/b/askds/archive/2010/04/09/friday-mail-sack-while-the-ned-s-away-edition.aspx#adws

On inspection of the registry key, the “Port SSL” type is incorrect and the data is missing. According to the Technet blog post, the value type should be “REG_DWORD” and the default data is 636.

I deleted the existing incorrect value and created a new value with the REG_DWORD type and the value data of 636 decimal.

Upon checking the Windows event logs, I could see that the AD Web Services was already using the corrected value, so no service restart was required.

The next log entry displayed the VCMSDS instance and LDAP/LDAPS (SSL) ports it is configured to use.

After this vCenter system was fixed, we checked all of the other vCenter servers onsite and found that their vCenter 4.1 server they were using for non-production also had the same error. That vCenter server was running on a Windows 2003 server and we did have to stop and restart the AD Web Services service to load the corrected SSL port value and resolve the error.

Thanks to Alan Renouf and the vCheck contributors at Virtu-Al.net for grabbing and displaying this error.

VMware vSphere 4.0 Update 2 is released

This evening VMware released Update 2 for ESX/ESXi 4, vCenter Management Server 4, vCenter Update Manager 4 and VMware Data Recovery.
A quick scan of the ESX 4 Update 2 release notes shows expanded support for FT on Intel i3/i5 Clarkdale, Xeon 34xx Clarkdale and Xeon 56xxx processors. Support for IOMMU on AMD Opteron 61xx and 41xx processors. Guest OS support for Ubuntu 10.04 and improvements to esxtop and resxtop to include NFS performance statistics Reads/s, Writes/s, MBRead/s, MBWrtn/s, cmd/s and gavg/s latency. Included in the resolved issues is a change in the way the Snapshot Manager “Delete All” operation works. In previous versions the snapshot farthest away from the base disk was committed to its immediate parent, then that parent would be committed to its parent until the last remaining snapshot is committed to the base. The release notes report that this operation will now start with the snapshot closest to the base disk and work toward the farthest. This should reduce the amount of disk space required during the “delete all/commit” operation and reduce the amount of data that is repeatedly committed. I think this is a great change. I have seen customers run out of space in datastores when the failed to keep track of active snapshots and didn’t understand the “delete all/commit” process.

The vCenter Management Server 4 Update 2 release notes list support for guest customization of:

◦Windows XP Professional SP2 (x64) serviced by Windows Server 2003 SP2
◦SLES 11 (x32 and x64)
◦SLES 10 SP3 (x32 and x64)
◦RHEL 5.5 Server Platform (x32 and x64)
◦RHEL 5.4 Server Platform (x32 and x64)
◦RHEL 4.8 Server Platform (x32 and 64)
◦Debian 5.0 (x32 and x64)
◦Debian 5.0 R1 (x32 and x64)
◦Debian 5.0 R2 (x32 and x64)

Among the resolved items, there is an update JRE (1.5.0_22) and number of fixed related to Host Profiles, support for vSwitch portgroup named longer than 50 characters, advanced settings to allow the use vDS connections as additional HA heartbeat networks, the addision of a parameter in vpxd.cfg to set a greater timeout value for VMotion operations involving VMs with swap files on local datastores, among many others. In the known issues section is astatement that while USB controllers can be added to VMs, attaching USB devices is not supported and that vSphere Web Access is experimentally supported.

The vCenter Update Manager 4 Update 2 release notes list improvement of operations in low bandwidth, high latency and slow networks, including a reference to KB 1017253 detailing the configuration of extended timeout values for ESX, vCenter and Update Manager Update 2.
The compatability matrix shows that Update Manager 4 Update 2 is only compatible with vCenter Management Server 4 Update 2.

VMware Data Recovery Update 2 includes the following new items:

The following enhancements have been made for this release of Data Recovery.

•File Level Restore (FLR) is now available for use with Linux.
•Each vCenter Server instance supports up to ten Data Recovery backup appliances.
•The vSphere Client plug-in supports fast switching among Data Recovery backup appliances.
•Miscellaneous vSphere Client Plug-In user interface enhancements including:
◦The means to name backup jobs during their creation.
◦Additional information about the current status of destination disks including the disk’s health and the degree of space savings provided by the deduplication store’s optimizations.
◦Information about the datastore from which virtual disks are backed up.

The support for up to 10 Data Recovery appliances per vCenter will allow up to 1000 jobs (100 per appliance x10), this is a significant increase in backup capacity.

The build numbers for the various items are:

ESX 4.0 Update 2 Build 261974
ESXi 4.0 Update 2 Installable Build 261974
ESXi 4.0 Update 2 Embedded Build 261974
VMware Tools Build 261974
vCenter Server 4.0 Update 2 Build 258672
vCenter Update Manager 4.0 Update 2 Build 264019

vSphere 4 Update 2 components can be downloaded here.

Licensing VMware ESX 4, ESXi and vCenter 4 Video KB article

This week VMware posted KB article 1010839 on licensing ESX 4, ESXi 4 and vCenter 4. I get many questions in class about the new license assignment process for vSphere. This KB article has a nice video demonstration and very concise text direction for assigning licenses.

vCenter 2.5 Update 5 released

VMware released vCenter 2.5 Update 5. The release notes state:

VirtualCenter 2.5 Update 5 includes significant performance and scalability improvements to VMware HA. Use VirtualCenter 2.5 Update 5 for environments with more than 35 virtual machines per host in an HA cluster.
For information on the ESX Server host settings required for this scalability improvement, see ESX Server host settings required for environments with up to 80 virtual machines per host in an HA Cluster (KB 1012002).

KB 1012002 states that with vCenter 2.5 update 5 an ESX host in an HA cluster can support up to 80 VM’s. The article continues with the specific ESX settings that are needed. The “RunningVCpuLimit” needs to be set to 192, the Service Console memory needs to be raised to 512MB and the Host Agent (hostd) memory settings in /etc/vmware/hostd/config.xml need to be increased. Note that the ESX host will need to be restarted after changing the Service Consle memory allocation.

In addition to the HA change, the release adds new http connection timeout settings:

A new advanced setting entry vpxd.httpClientIdleTimeout can be used to configure the timeout value for an idle HTTP connection. The default value for this entry is 15 minutes (900 seconds), ensuring that the VirtualCenter Server closes the idle HTTP connection after the connection has been idle for 15 minutes. If a firewall session timeout value is set to less than 15 minutes, the value for vpxd.httpClientIdleTimeout should be changed to be smaller than the firewall’s timeout value.

No updates to the vCenter Enterprise Converter or Update Manager plug-ins have been made.