Long live VMFS3! VAAI enabled storage, ESXi 5.x and above, thin VMDK files = Not able to recover unused (null) space.

Basically we still need a VMFS3 datastore as a "AUX" to shrink disks.

This is interesting! Basically, logically, in plain English.. what happens is since the VAAI datamover used is not at the ESXi layer, the storage doesn't know what is on the VMDK and _has-to_ copy everything. There is no chance for ESXi layer to figure out which blocks to drop!

The conditions - ESXi 5.x onwards (VMFS5) + VAAI capable/enabled storage, Thin Provisioned VMs

"...When the source filesystem uses a different  blocksize from the destination filesystem, the legacy datamover (FSDM) is used. When the blocksizes of source and destination are equal, the new datamover (FS3DM) is used. FS3DM decides if it will use VAAI or just the software component. In either case, null blocks are not reclaimed"

Thanks to Boon Hong​ for highlighting this.

http://kb.vmware.com/kb/2004155

VMware vSphere Snapshots (draft-WIP)

This post aims to condense and place into a single page important information with regards to snapshots, svmotion (snapshots are used), cloning (snapshots used there too!) and some general issues  and questions which I've encountered in my working environment. (quiescing errors, during Avamar backup, during cloning of "hardened" windows GOS)

I started out looking for supporting articles but ended up going in and out of KBs and losing track of what belongs to what, where belongs to where. Hence this post. It's mostly my notes of what I think will be useful and important while troughing through the maze of KB articles.

Start here (Understanding how Snapshots work on different versions of ESX/ESXi)
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1015180

• Quiesce: If the  flag is 1 or true, and the virtual machine is powered on when the snapshot is taken, VMware Tools is used to quiesce the file system in the virtual machine. Quiescing a file system is a process of bringing the on-disk data of a physical or virtual computer into a state suitable for backups. This process might include such operations as flushing dirty buffers from the operating system's in-memory cache to disk, or other higher-level application-specific tasks.
  
  Note: Quiescing indicates pausing or altering the state of running processes on a computer, particularly those that might modify information stored on disk during a backup, to guarantee a consistent and usable backup. Quiescing is not necessary for memory snapshots; it is used primarily for backups.
• If the virtual disk is larger than 2TB in size, the redo log file is of  --sesparse.vmdk format.
• .vmsd
  The .vmsd file is a database of the virtual machine's snapshot information and the primary source of information for the snapshot manager. The file contains line entries which define the relationships between snapshots as well as the child disks for each snapshot.
• Snapshot.vmsnThe .vmsn file includes the current configuration and optionally the active state of the virtual machine.
• The above files will be placed in the working directory by default in ESX/ESX 3.x and 4.x.
• In ESXi 5.x and later snapshots descriptor and delta VMDK files will be stored in the same location as the virtual disks (which can be in a different directory to the working directory). 
• When removing a snapshot, the snapshot entity in the snapshot manager is removed before the changes are made to the child disks. The snapshot manager does not contain any snapshot entries while the virtual machine continues to run from the child disk. 
•  During a snapshot removal, if the child disks are large in size, the operation may take a long time. This can result in a timeout error message from either VirtualCenter or the VMware Infrastructure Client.

The child disk

The child disk, which is created with a snapshot, is a sparse disk. Sparse disks employ the copy-on-write (COW) mechanism, in which the virtual disk contains no data in places, until copied there by a write. This optimization saves storage space. The grain is the unit of measure in which the sparse disk uses the copy-on-write mechanism. Each grain is a block of sectors containing virtual disk data. The default size is 128 sectors or 64KB

The disk chain

Generally, when you create a snapshot for the first time, the first child disk is created from the parent disk. Successive snapshots generate new child disks from the last child disk on the chain. The relationship can change if you have multiple branches in the snapshot chain.

This diagram is an example of a snapshot chain. Each square represents a block of data or a grain as described above:

• Reverting virtual machines to a snapshot causes all settings configured in the guest operating system since that snapshot to be reverted. The configuration which is reverted includes, but is not limited to, previous IP addresses, DNS names, UUIDs, guest OS patch versions, etc.

When performing Storage vMotion
http://blogs.vmware.com/vsphere/2011/09/storage-vmotion-storage-drs-virtual-machine-snapshots.html
"It should also be noted that if you do a Storage vMotion of a VM with snapshots and the VM has the workingDir parameter set, theworkingDir setting will be removed from the .vmx & the .vmsn snapshot data file will be moved to the home folder of the VM on the destination datastore. You do get a warning in the migration wizard about this"

"Therefore, if you use the snapshot.redoNotWithParent = "TRUE" parameter, you should refrain from doing Storage vMotion operations."

This happens regardless even if you set the parameters above - in other words, try as best as possible to avoid putting the snapshot files on a datastore away from the parent -flat file disks if all the datastores involved are backing an SDRS cluster...

Troubleshooting http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1031200
Disable selective VSS writers for troubleshooting

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=5962168
Using custom "pre-freeze" and "post-thaw" scripts.

Covers SYNC and LGTO_SYNC drivers, not VSS.

This article details why the VM may become unresponsive and seem "hung" during a snapshot process.

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1007696
Details VSS  troubleshooting. This article also includes the services that need to be running on the GOS., Issues with quiescing.

When performing cloning on vSphere v5.x on a VM with snapshots
This is what's been observed: Base disk + snapshot will be copied over to the destination VM merging the snapshot(s) into a single VMDK at destination.

When you've run out of space on the datastore and snapshots cannot be deleted
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004545
This post details the steps to take with a command line tool provided you already have another datastore with sufficient space or have been able to increase the space on the same datastore that had run out of space.

There is a limit on how many open vmdk files an ESXi host can address depending on the VMFS version. 
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004424
This article is very interesting technically. Covers all versions of ESXi till date. There are changes to the HEAP size between version updates. Useful. Here's the table of limits reproduced:

Version/build	Default heap amount	Default allowed open VMDK storage per host	Minimum heap amount	Maximum heap amount	Maximum heap value	Maximum open VMDK storage per host
ESXi/ESX 3.5/4.0	16 MB	4 TB	N/A	N/A	N/A	N/A
ESXi/ESX 4.1	80 MB	8 TB	N/A	128 MB	128	32 TB
ESXi 5.0 Update 2 (914586) and earlier	80 MB	8 TB	N/A	256 MB	255	25 TB
ESXi 5.0 Patch 5 (1024429) and later	256 MB	60 TB	256 MB	640 MB	255	60 TB
ESXi 5.1 Patch 1 (914609) and earlier	80 MB	8 TB	N/A	256 MB	255	25 TB
ESXi 5.1 Update 1 (1065491) and later	256 MB	60 TB	256 MB	640 MB	255	60 TB

Disks (VMDK) larger than 2TB (for ESXi 5.5 with VMFS5 only. If using NFS, backend must be on file system that has large file support like EXT4. Extending disks beyond 2TB also requires the use of the Web Client or vCLI)

http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2058287

Changes in virtual machine snapshots for VMDKs larger than 2 TB:

• Snapshots taken on VMDKs larger than 2 TB are now in Space Efficient Virtual Disk (SESPARSE) format. No user interaction is required. The redo logs will be automatically created as SESPARSE instead of VMFSSPARSE (delta) when the base flat VMDK is larger than 2 TB.
• Extending a base flat disk on VMFSSPARSE or SESPARSE is not supported.
• The VMFSSPARSE format does not have the ability to support 2 TB or more.
• VMFSSPARSE and SESPARSE formats cannot co-exist in the same VMDK. In a virtual machine, both types of snapshot can co-exist, but not in the same disk chain. For example, when a snapshot is taken for a virtual machine with two virtual disks attached, one smaller than 2 TB and one larger than 2 TB, the smaller disk snapshot will be VMFSSPARSE the larger disk snapshot will be SESPARSE.
• Linked clones will be SESPARSE if the parent disk is larger than 2 TB.

What else can cause snapshots consolidation to fail?
Main reference article in spanish:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2046576
1. Locks (files are locked)
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=10051
2. Temporary loss of communication between vCenter and ESXi hosts during confirmation - this does not mean that the ESXi hosts are shown to be disconnected from vCenter. To "restore" connectivity restart management agents from the host. (My note from field experience - there is a chance that during the restart of the management agents, your host may really get disconnected from vCenter AND if your cluster is EVC enabled, you will have to shutdown all the running VMs on that host in order for that host to rejoin the EVC cluster - so beware!)
3. A snapshot configuration file with extension .vmsd in the VM home directory may interfere. Rename, move or delete that file.
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1003490

Heartbleed remediation for vCenter (build 1750787), ESXi (build 1746018), Web Client Integration plug-in (build 1750778), vSphere C# client (build 1746248)

Glad to report the vCenter update went without a hitch on my home lab. As aways YMMV.

Updating to vCenter 5.5.0u1a - install in sequence following custom install. No reboot required. All other components remain the same as 5.5.0u1

Versions of updated 5.5.0u1a vCenter SSO, Inventory Service, Web Client and vCenter Server.

VMware Update Manager will be restarted during installation.

Web Client Integration Plugin will still have the same name as 5.5.0u1 but the build/version has been updated

vSphere Client updated to build 1746248. Not sure if it's only my home NAS that's slow but it looked like before updating, the stats and info page for ESXi hosts would not display properly.

vSphere Client not displaying ESXi stats properly (before updating; could also be caused by my storage backend)

Can I perform a P2V conversion on an Active Directory domain controller? (Note!! Use at your own risk!)

Updated 2014-Apr: 

Summary; you can do it. Just make sure all the FSMO roles are on the DC that is being P2Ved. Also Microsoft test-case is based on only ONE DC. The ramifications of performing this action on a mult-DC environment is not clear.

In essence, this P2V test case is based on SCVMM and not VMware Converter. There are multiple steps involved. SCVMM will use VSS to take a snapshot of the current state of the AD and simultaneously create a VM (on MS platform) and start cloning.

VMware Converter process:

From the horses' mouth, this is how it coordinates with the ESXi layer, the source VM and the destination target VM.

1. Authenticate the Source Machine. (I take this as logging on to the source machine)

2. Get the Source VM information.

3. Install the Agent on the Source Computer.

4. Create a new Destination VM.

5. Call the VSS program to Clone or Snapshot the guest machine internally.

6. Copy the cloned info to the destination machine.

7. Uninstall the agent from the Source Machine.

"We do not invoke any other thing which will cause the Source Machine to hamper."

Original post 17 Feb 14

NOTE - Use these only at your OWN RISK. I cannot be held responsible for any issues that may arise through applying any of the following. It is generally a well known "no-no" to P2V and V2V a DC that is pre-2012.

(From Microsoft Support - Advisory only - Further details, if any, will be updated as more information becomes available)

Can I perform a P2V conversion on an Active Directory domain controller?

Yes. You can perform an offline P2V conversion on a domain controller. Performing the conversion offline helps avoid potential Active Directory USN rollback issues during the process.

http://technet.microsoft.com/en-us/library/dd221390.aspx

Recommendations:

Offline P2V:

The impact to the original is when you perform P2V, the source DC will restart into the Windows Preinstallation Environment. It is the recommended solution if you need to P2V multiple domain controllers. 

Online P2V:

SCVMM Online P2V will not impact original Physical environment, which has been double confirmed with System Center team.  But it will cause USN rollback problem for the virtual environment if you P2V multiple domain controllers. However, if you only P2V one DC with FSMO roles, it will not cause any problem.

If you P2V only one DC with FSMO role using Online P2V. Please perform the following steps on the converted DC in virtual machine:

1.         Clean up metadata for DCs no longer exist

Clean up server metadata

http://technet.microsoft.com/en-us/library/cc736378(v=ws.10).aspx

2.         Please disable initial synchronization when you start the virtual machine for the first time:

How to disable initial synchronization

On the PDC, go to the following registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters

Add the following Value:

Value name:  Repl Perform Initial Synchronizations Value type:  REG_DWORD Value data: 0  =============================

vmware vi fastpath unable to add server

"Error: You don't have permission to execute this command" - Append sudo to command line
"Error: Failed to add users" - Make sure your username parameter has a double "\". Note the red "\" . It is NOT a typo.

Eg: sudo vifp addserver vcenter.addomain.com --authpolicy adauth --username addomain\\adusername

vSphere Management Assistant (vMA) 5.1 hostname not sticking

This is a frustrating problem. Don't understand why in my company's production systems, the standard VMware installation instructions don't work.

I had no issues on my home lab. (hair pulling)

You configure the new hostname using the instructions from the vMA user guide, configuration is done through the web interface on port 5480 and/or directly from the VM console, and/or from a SSH session.

The moment you reboot the system, it reverts back to localhost.localdomain. It does not matter if you initiate the reboot from the web interface on port 5480 or from a shell session.

Anyhow, to fix this problem, create an "A" record on the AD DNS server, then from shell run "sudo -i" (this will drop you into a root session (text color will change to red), then run Suse's network configuration utility "system-config-network-tui"

Reference: http://communities.vmware.com/message/2159792?tstart=0

Edit (4/4/2013):
Same thing happened to my home vMA.

HOWTO Fix vCenter 4 search not working

First, reset Web Service in vCenter.

Then if it still doesn't work, on the vSphere clien (not verified nor tested)t:

1. Click Plug-in -> manage Plug-ins
2. Right click Hardware Status plugin and select Disable
3. Close and re-open vSphere client.
4. Click Plug-in -> manage Plug-ins
5. Right click Hardware Status plugin and select Enable

(Solution from one of my colleagues. Am not sure if this step is correct.. What does "Hardware Status" plugin have to do with search?)

If steps for client are wrong, welcome corrections.

Storage vMotion of a virtual vCenter (Yes, a Virtual Machine)

Yes, it can be done. We've done it. :-D

Also reference to post by Iwan Rahabok:
Can you storage vMotion the vCenter VM itself?

VMware/vSphere - CPU READY and CPU USAGE put simply

I was asked this question by my colleagues and after answering it with the official VMware explanation, they still didn't quite get it. (Yeah, actually if I look at it without the necessary background info, I'd probably not get it either...)

The following visualization helped put it simply:

CPU USAGE and CPU READY - What is it?

CPU Ready = % of time there is work to be done for VMs, but no physical CPU available to do it on (all host CPUs are busy serving other VMs). One rule of thumb that I heard is that below 5% Ready is normal; anything between 5% and 10%, best keep an eye on the VM and the host. Over 10% (for extended periods) you should be planning on taking some action.

-           

-          CPU Usage = raw, absolute amount of CPU used by corresponding VM at the given moment.

References:

http://vmfaq.com/entry/42/

The amount of time a virtual machine waits in the queue in a ready-to-run state before it can be scheduled on a CPU is known as ready time.

The higher the ready time is, the slower the virtual machine is performing. The ready time should preferably be as low as possible. Virtual machines that are allocated multiple cpus or have high timer interrupts are more frequently seen with high ready time values. 

To remove unused plug-in from vCenter use Managed Object Reference

Found at http://vcenterservername/mob

Logon with vSphere credentials;

1. Click on content, then
2. Extension manager
3. Find the plug-in which needs to be removed; for example, look for extensionList["VirtualCenter"], the parameter you need is just VirtualCenter
4. Click UnregisterExtension, in the VALUE field, enter the name of the plug-in you wish (in this example it's VirtualCenter
5. Click on Invoke Method (to remove plug-in

vCenter and SQL 2008 SP1 on Windows 2008 R2

1.  vCenter won't install on the Web Edition.
2. You can install SQL2008 on 2008 R2, just have to disable all services until SP1 has been applied.
3. When creating the ODBC connections, use "%systemdrive%\Windows\SysWoW64\Odbcad32.exe" and not the ODBC from Administrative Tools section.

BEFORE, installing vCenter Server, do:

1.       Start -> Administrative Tools -> Server Manager.

2.       From the left pane, select “Features”

3.       On the right hand pane, click “Add Features”

4.       Expand .NET Framework 3.5.1 Features

5.       Select only .NET Framework 3.5.1 (do not select WCF Activation as it will install IIS which conflicts with Apache Tomcat, which of course vCenter will install).

vCenter Server should now install without any issues.