Insomniac Ramblings is now Midus Renaissance.

This is not a technical post.

However I feel that this will be useful for people with problems sleeping.

As I'm no longer having sleep issues, I thought maybe it's time to update the name of this blog.

The "solution" I found in the end for this disorder was through the holistic path.

In short, I finally consulted with a Tradition Chinese doctor and "resetting" my internal flow has allowed me to sleep well at night after years of battling.

I think Insomnia is prevalent among many in our times - it detracts from the overall quality of life and can inadvertently, insidiously affect other aspects of one's life.

As this is an update and not a technical post (yet) I'd like to leave the topic open and hoping maybe someone else can be helped.

Looking forward to robust discussions (if any)

Advantages of using VMware PVSCSI interface vs LSI SAS and it's caveats

Updated (again) 1330hrs:
Appended some other interesting information from the discussion resulting from that Facebook post.
Thanks guys!

LSI SAS by defaults supports only queue depth of 25. (needs further confirmation) vs PVSCSI.

Original Post:-

While there are host OS (HOS) and guest OS (GOS) optimizations that will increase performance, there are caveats to note.

My recommendation would be to follow VMwares' best practice (gleaned from various forum posts and blogs - not sure if there are any such official articles/KBs) and do not configure your OS disk/partition with PVSCSI especially in a production environment where you may have a few other VMware administrators.

However, for a controlled test environment like home labs, by all means try it. All my home lab VMs are running PVSCSI on OS disks too. ;)

The details of why "don't do that" follow:

This is a reply to a post on Facebook's VMUG ASEAN to a question on how to configure PVSCSI replacement interface.

(Don't know if this hotlink to the post on VMUG  ASEAN will work. If anyone knows a sure-fire way to link Facebook posts let me know in the comments below :D )

Here's my 2 cents.  I did some deep dive research on PVSCSI and there are caveats. Some OS may have issues with it. Particularly VMware View. For PVSCSI to work, VMtools has to be installed and functional. There may be some situations where when you update or lose the VMtools you might lose connectivity to the disks connected using the PVSCSI device. I had considered using PVSCSI as the OS boot interface (after switching the vNIC using the article Lalit Sharma mentioned. However, if you get into a situation where you need to boot the OS (Windows in this case, Linux I don't have enough experience) to repair the OS, you will have to reconfigure the interface back to LSI or the default Windows boot media won't be able to access the OS disk. So take these things into consideration. Anyhow for my home lab, everything is on PVSCSI. Just it may not be wise in production environment especially if you have other vSphere admins that may not be as familiar.

Appends:-

Roshan Jha: Posted a recent VMware blog article (which I did not see earlier). 
It's VSAN related but relevant.

Which vSCSI controller should I choose for performance?  - Mark Achtemichuk

Kasim Hansia: "LSI only supports 32 queue depth and PVSCSI queue depth default values are 64 (device) and 254 (adapter). You can increase PVSCSI queue depths to 256 (device) and 1024 (adapter) inside a Windows or Linux Virtual Machine. "

Tan Wee Kiong - thanks for the correction of the initial assumption and the following KB article:

"Large-scale workloads with intensive I/O patterns might require queue depths significantly greater than Paravirtual SCSI default values (2053145)"

"The large-scale workloads with intensive I/O patterns require adapter queue depths greater than the Paravirtual SCSI (PVSCSI) default values. Current PVSCSI queue depth default values are 64 (for device) and 254 (for adapter). You can increase PVSCSI queue depths to 256 (for device) and 1024 (for adapter) inside a Windows virtual machine or Linux Virtual Machine."

Note that the article has made a distinction between a "device" and the "adapter".

Disabling many AD user accounts on Windows Server 2003 without powershell

This may or may not help you but it's for my future reference.

My source was from dumping using MAP (Microsoft Assessment and Planning) toolkit using report "ActiveDevicesUsageTracker"

My AD wasn't using the default OU structure
Usable output = "Username" column = samID

Retrieve User-DN on Windows Server 2003
With the samID above, for each name

dsquery user -samid

Disable AD user accounts on Windows Server 2003
dsmod user user-DN -disabled yes

References (just got the important bits):
http://technet.microsoft.com/en-sg/library/cc781527(v=ws.10).aspx
https://kb.bluecoat.com/index?page=content&id=KB4548

Not related but I needed to get the AD group membership of those disabled AD accounts for clean up purposes.

Retrieve by AD user object AD group membership:
dsget user "" -memberof -expand 

Reference:
http://social.technet.microsoft.com/wiki/contents/articles/2195.active-directory-dsquery-commands.aspx

VMware vSphere Snapshots (draft-WIP)

This post aims to condense and place into a single page important information with regards to snapshots, svmotion (snapshots are used), cloning (snapshots used there too!) and some general issues  and questions which I've encountered in my working environment. (quiescing errors, during Avamar backup, during cloning of "hardened" windows GOS)

I started out looking for supporting articles but ended up going in and out of KBs and losing track of what belongs to what, where belongs to where. Hence this post. It's mostly my notes of what I think will be useful and important while troughing through the maze of KB articles.

Start here (Understanding how Snapshots work on different versions of ESX/ESXi)
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1015180

• Quiesce: If the  flag is 1 or true, and the virtual machine is powered on when the snapshot is taken, VMware Tools is used to quiesce the file system in the virtual machine. Quiescing a file system is a process of bringing the on-disk data of a physical or virtual computer into a state suitable for backups. This process might include such operations as flushing dirty buffers from the operating system's in-memory cache to disk, or other higher-level application-specific tasks.
  
  Note: Quiescing indicates pausing or altering the state of running processes on a computer, particularly those that might modify information stored on disk during a backup, to guarantee a consistent and usable backup. Quiescing is not necessary for memory snapshots; it is used primarily for backups.
• If the virtual disk is larger than 2TB in size, the redo log file is of  --sesparse.vmdk format.
• .vmsd
  The .vmsd file is a database of the virtual machine's snapshot information and the primary source of information for the snapshot manager. The file contains line entries which define the relationships between snapshots as well as the child disks for each snapshot.
• Snapshot.vmsnThe .vmsn file includes the current configuration and optionally the active state of the virtual machine.
• The above files will be placed in the working directory by default in ESX/ESX 3.x and 4.x.
• In ESXi 5.x and later snapshots descriptor and delta VMDK files will be stored in the same location as the virtual disks (which can be in a different directory to the working directory). 
• When removing a snapshot, the snapshot entity in the snapshot manager is removed before the changes are made to the child disks. The snapshot manager does not contain any snapshot entries while the virtual machine continues to run from the child disk. 
•  During a snapshot removal, if the child disks are large in size, the operation may take a long time. This can result in a timeout error message from either VirtualCenter or the VMware Infrastructure Client.

The child disk

The child disk, which is created with a snapshot, is a sparse disk. Sparse disks employ the copy-on-write (COW) mechanism, in which the virtual disk contains no data in places, until copied there by a write. This optimization saves storage space. The grain is the unit of measure in which the sparse disk uses the copy-on-write mechanism. Each grain is a block of sectors containing virtual disk data. The default size is 128 sectors or 64KB

The disk chain

Generally, when you create a snapshot for the first time, the first child disk is created from the parent disk. Successive snapshots generate new child disks from the last child disk on the chain. The relationship can change if you have multiple branches in the snapshot chain.

This diagram is an example of a snapshot chain. Each square represents a block of data or a grain as described above:

• Reverting virtual machines to a snapshot causes all settings configured in the guest operating system since that snapshot to be reverted. The configuration which is reverted includes, but is not limited to, previous IP addresses, DNS names, UUIDs, guest OS patch versions, etc.

When performing Storage vMotion
http://blogs.vmware.com/vsphere/2011/09/storage-vmotion-storage-drs-virtual-machine-snapshots.html
"It should also be noted that if you do a Storage vMotion of a VM with snapshots and the VM has the workingDir parameter set, theworkingDir setting will be removed from the .vmx & the .vmsn snapshot data file will be moved to the home folder of the VM on the destination datastore. You do get a warning in the migration wizard about this"

"Therefore, if you use the snapshot.redoNotWithParent = "TRUE" parameter, you should refrain from doing Storage vMotion operations."

This happens regardless even if you set the parameters above - in other words, try as best as possible to avoid putting the snapshot files on a datastore away from the parent -flat file disks if all the datastores involved are backing an SDRS cluster...

Troubleshooting http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1031200
Disable selective VSS writers for troubleshooting

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=5962168
Using custom "pre-freeze" and "post-thaw" scripts.

Covers SYNC and LGTO_SYNC drivers, not VSS.

This article details why the VM may become unresponsive and seem "hung" during a snapshot process.

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1007696
Details VSS  troubleshooting. This article also includes the services that need to be running on the GOS., Issues with quiescing.

When performing cloning on vSphere v5.x on a VM with snapshots
This is what's been observed: Base disk + snapshot will be copied over to the destination VM merging the snapshot(s) into a single VMDK at destination.

When you've run out of space on the datastore and snapshots cannot be deleted
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004545
This post details the steps to take with a command line tool provided you already have another datastore with sufficient space or have been able to increase the space on the same datastore that had run out of space.

There is a limit on how many open vmdk files an ESXi host can address depending on the VMFS version. 
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004424
This article is very interesting technically. Covers all versions of ESXi till date. There are changes to the HEAP size between version updates. Useful. Here's the table of limits reproduced:

Version/build	Default heap amount	Default allowed open VMDK storage per host	Minimum heap amount	Maximum heap amount	Maximum heap value	Maximum open VMDK storage per host
ESXi/ESX 3.5/4.0	16 MB	4 TB	N/A	N/A	N/A	N/A
ESXi/ESX 4.1	80 MB	8 TB	N/A	128 MB	128	32 TB
ESXi 5.0 Update 2 (914586) and earlier	80 MB	8 TB	N/A	256 MB	255	25 TB
ESXi 5.0 Patch 5 (1024429) and later	256 MB	60 TB	256 MB	640 MB	255	60 TB
ESXi 5.1 Patch 1 (914609) and earlier	80 MB	8 TB	N/A	256 MB	255	25 TB
ESXi 5.1 Update 1 (1065491) and later	256 MB	60 TB	256 MB	640 MB	255	60 TB

Disks (VMDK) larger than 2TB (for ESXi 5.5 with VMFS5 only. If using NFS, backend must be on file system that has large file support like EXT4. Extending disks beyond 2TB also requires the use of the Web Client or vCLI)

http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2058287

Changes in virtual machine snapshots for VMDKs larger than 2 TB:

• Snapshots taken on VMDKs larger than 2 TB are now in Space Efficient Virtual Disk (SESPARSE) format. No user interaction is required. The redo logs will be automatically created as SESPARSE instead of VMFSSPARSE (delta) when the base flat VMDK is larger than 2 TB.
• Extending a base flat disk on VMFSSPARSE or SESPARSE is not supported.
• The VMFSSPARSE format does not have the ability to support 2 TB or more.
• VMFSSPARSE and SESPARSE formats cannot co-exist in the same VMDK. In a virtual machine, both types of snapshot can co-exist, but not in the same disk chain. For example, when a snapshot is taken for a virtual machine with two virtual disks attached, one smaller than 2 TB and one larger than 2 TB, the smaller disk snapshot will be VMFSSPARSE the larger disk snapshot will be SESPARSE.
• Linked clones will be SESPARSE if the parent disk is larger than 2 TB.

What else can cause snapshots consolidation to fail?
Main reference article in spanish:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2046576
1. Locks (files are locked)
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=10051
2. Temporary loss of communication between vCenter and ESXi hosts during confirmation - this does not mean that the ESXi hosts are shown to be disconnected from vCenter. To "restore" connectivity restart management agents from the host. (My note from field experience - there is a chance that during the restart of the management agents, your host may really get disconnected from vCenter AND if your cluster is EVC enabled, you will have to shutdown all the running VMs on that host in order for that host to rejoin the EVC cluster - so beware!)
3. A snapshot configuration file with extension .vmsd in the VM home directory may interfere. Rename, move or delete that file.
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1003490

Things to look out for when using VMware PVSCSI

Well, since VMXNET3 is optimum, why not PVSCSI?

Rolling out to a production environment we have to make sure we know the possible caveats and limitations so that the stakeholders can be informed and operations have the correct information for deployments.

Following is a summary of things to look out for based on URL here:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010398

• The VMware PVSCSI adapter driver is also compatible with the Windows Storport storage driver
• PVSCSI adapters are not suited for DAS environments.
• Cannot be used as a boot disk for Red Hat Enterprise Linux (RHEL) 5 (32 and 64 bit) and all update releases
• Hot-adding a PVSCSI adapter is only supported for those versions that support booting from a PVSCSI adapter.
• Hot add or hot remove requires a bus rescan from within the guest.
• Disks with snapshots might not experience performance gains when used on Paravirtual SCSI adapters if memory on the ESX host is over committed.
• Do not use PVSCSI on a virtual machine running Windows with spanned volumes. Data may become inaccessible to the guest operating system.
• If you upgrade from RHEL 5 to an unsupported kernel, you might not be able to access data on the virtual machine's PVSCSI disks. You can run vmware-config-tools.pl with the kernel-version parameter to regain access.
• If a virtual machine uses PVSCSI, it cannot be part of a Microsoft Cluster Server (MSCS) cluster.

I remember seeing somewhere some other considerations for View deployments and will update this post once there is more information.

Have a great day ahead!

A timeout (30000 milliseconds) was reached while waiting for a transaction response from the (name-of-service) service (WIP)

Post is purely to reference bookmarks during troubleshooting (Work-In-Progress)

General consensus so far looks like it's related to some third party application issues.

In my environment, eventlogs reported multiple user registry leaks.

My guess it has to do with the server being under load or some hardware (possibly disk?) related issues combined with the software combination and communications between tiers.

http://www.eversity.nl/blog/2012/08/a-timeout-30000-milliseconds-was-reached-while-waiting-for-a-transaction-response-from-the-name-of-service-service/

http://social.technet.microsoft.com/Forums/windowsserver/en-US/77a521a1-e3d5-4ab2-9c0f-be4a5498ce1c/windows-server-2008-sp2-stops-responding-as-multiple-services-timeout?forum=winserverTS

This thread has a bit more variations to the issue:
http://social.technet.microsoft.com/Forums/en-US/2c4b8121-da1c-4c11-b11d-2dff099ba245/windows-server-2008-r2-sp1-rds-hang-and-stop-responding-new-rdp-session-already-connected-session?forum=winserverTS


Official articles on how to change timeout. May not solve the root cause.
http://support.microsoft.com/kb/922918 (EventID: 7011)
http://social.technet.microsoft.com/wiki/contents/articles/1458.event-id-7011-basic-service-operations.aspx

Other (may be related)
http://technet.microsoft.com/en-us/library/cc756342%28v=ws.10%29.aspx (EventID: 7009)

Affecting VMtools:
http://ambitech.blogspot.sg/2013/01/esxi-5x-error-messages-in-windows-vm.html
Based on article comments, recommendation of not "updating OS patches + VMtools installation together" is not conclusive. (as in my environment, we patch first, then update tools)

Normal client OS with SSD:
http://www.bradymoritz.com/a-timeout-was-reached-30000-milliseconds-while-waiting-for-the
Seem to point to NOT disk issues. Hmmm...

How-To fix RDP connection issue with error "The Local Security Authority cannot be contacted"

Can be caused by:

1. User must change password on next logon and RDC is set to use only Network Level Authentication. Affects "workgroup" computers or computers on another domain (compared to the one you're logging in from). 
2. Missing language pack 

References:

http://blog.mnewton.com/articles/Solution-RDP-The-Local-Security-Authority-cannot-be-contacted/

On how to disable NLA (assuming you can get access to your remote server using the suggested methods:

http://blog.mnewton.com/articles/Solution-RDP-The-Local-Security-Authority-cannot-be-contacted/

Heartbleed remediation for vCenter (build 1750787), ESXi (build 1746018), Web Client Integration plug-in (build 1750778), vSphere C# client (build 1746248)

Glad to report the vCenter update went without a hitch on my home lab. As aways YMMV.

Updating to vCenter 5.5.0u1a - install in sequence following custom install. No reboot required. All other components remain the same as 5.5.0u1

Versions of updated 5.5.0u1a vCenter SSO, Inventory Service, Web Client and vCenter Server.

VMware Update Manager will be restarted during installation.

Web Client Integration Plugin will still have the same name as 5.5.0u1 but the build/version has been updated

vSphere Client updated to build 1746248. Not sure if it's only my home NAS that's slow but it looked like before updating, the stats and info page for ESXi hosts would not display properly.

vSphere Client not displaying ESXi stats properly (before updating; could also be caused by my storage backend)

How to change Office 2013 keys

For 32 bit Windows:

cscript "C:\Program Files\Microsoft Office\Office15\OSPP.VBS" /inpkey:yourkeygoeshere 

For 64 bit Windows (assuming you are using 32 bit Office):

cscript "C:\Program Files (x86)\Microsoft Office\Office15\OSPP.VBS" /inpkey:yourkeygoeshere

Gather information on a domain user (including last logon time, password status, NTFS group membership) without having to use ADUC

From command prompt type:

net user /do logonname

output will be similar to:

C:\Users\username>net user /do usernameexample
The request will be processed at a domain controller for domain thdm.local

User name                    usernameexample
Full Name                    FirstnameLastname
Comment                      
User's comment
Country code                 000 (System Default)
Account active               Yes
Account expires              Never

Password last set            1/1/2014 01:26:20 AM
Password expires             2/1/2014 01:26:20 AM
Password changeable          2/2/2014 01:26:20 AM
Password required            Yes
User may change password     Yes

Workstations allowed         All
Logon script                 logon.cmd
User profile
Home directory
Last logon                   2/4/2014 7:51:17 AM

Logon hours allowed          All

Local Group Memberships
Global Group memberships     *ACCESS-EVERYTHING

The command completed successfully.

Can I perform a P2V conversion on an Active Directory domain controller? (Note!! Use at your own risk!)

Updated 2014-Apr: 

Summary; you can do it. Just make sure all the FSMO roles are on the DC that is being P2Ved. Also Microsoft test-case is based on only ONE DC. The ramifications of performing this action on a mult-DC environment is not clear.

In essence, this P2V test case is based on SCVMM and not VMware Converter. There are multiple steps involved. SCVMM will use VSS to take a snapshot of the current state of the AD and simultaneously create a VM (on MS platform) and start cloning.

VMware Converter process:

From the horses' mouth, this is how it coordinates with the ESXi layer, the source VM and the destination target VM.

1. Authenticate the Source Machine. (I take this as logging on to the source machine)

2. Get the Source VM information.

3. Install the Agent on the Source Computer.

4. Create a new Destination VM.

5. Call the VSS program to Clone or Snapshot the guest machine internally.

6. Copy the cloned info to the destination machine.

7. Uninstall the agent from the Source Machine.

"We do not invoke any other thing which will cause the Source Machine to hamper."

Original post 17 Feb 14

NOTE - Use these only at your OWN RISK. I cannot be held responsible for any issues that may arise through applying any of the following. It is generally a well known "no-no" to P2V and V2V a DC that is pre-2012.

(From Microsoft Support - Advisory only - Further details, if any, will be updated as more information becomes available)

Can I perform a P2V conversion on an Active Directory domain controller?

Yes. You can perform an offline P2V conversion on a domain controller. Performing the conversion offline helps avoid potential Active Directory USN rollback issues during the process.

http://technet.microsoft.com/en-us/library/dd221390.aspx

Recommendations:

Offline P2V:

The impact to the original is when you perform P2V, the source DC will restart into the Windows Preinstallation Environment. It is the recommended solution if you need to P2V multiple domain controllers. 

Online P2V:

SCVMM Online P2V will not impact original Physical environment, which has been double confirmed with System Center team.  But it will cause USN rollback problem for the virtual environment if you P2V multiple domain controllers. However, if you only P2V one DC with FSMO roles, it will not cause any problem.

If you P2V only one DC with FSMO role using Online P2V. Please perform the following steps on the converted DC in virtual machine:

1.         Clean up metadata for DCs no longer exist

Clean up server metadata

http://technet.microsoft.com/en-us/library/cc736378(v=ws.10).aspx

2.         Please disable initial synchronization when you start the virtual machine for the first time:

How to disable initial synchronization

On the PDC, go to the following registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters

Add the following Value:

Value name:  Repl Perform Initial Synchronizations Value type:  REG_DWORD Value data: 0  =============================

Status check of AD RID pool with email (Powershell)

Import-Module activedirectory

$RIDinfo=dcdiag /test:ridmanager /v | find "Available RID"

send-mailmessage -to demo@somewhere.com -from someone@somewhere.com -subject $RIDinfo -smtpserver 10.10.10.10

Excellent politically correct email on handling different sites with different administrative functions.

Probably a good Idea to start passing things though , but I’ll log this one for you.  I’m not sure a fools guide may be any good to you guys.  While I was trying to install the printer on James’s Computer yesterday I was blocked by a lack of admin rights on his computer rather than technical know how, so even if you know what you’re doing I think you may get stopped at the last hurdle.  This sort of thing may be better aimed at your team as they hold all the rights to your system.  I’ve copied in and others so we can work out what we can do going forward as we are very willing to provide you with as much assistance as needed.  I feel even if we can’t get rights to your computers it would be useful to know for the future so we are all aware of who is supporting each element and in-turn shortening the amount of time it takes to fix each of your problems