VMware Site Recovery Manager - Error while executing 'DiscoverLuns' command.

During a VMware Site Recovery Manager implementation, I ran into this pesky error.  Not really an SRM error specifically (more of an issue with the SRA) but a show stopper non the less for SRM. 

Environment:
VMware Site Recovery 4.1 running on vSphere 4.1 with NetApp on the backend.  All NFS.  Using the 1.4.3 NetApp SRA.

Issue:
While configuring the Array Manager in the SRM setup, you receive an error after adding in the SAN.  “Error while executing ‘discoverLuns’ command”.

Workaround:

After searching around, I was able to find a knowledgebase article on NetApp’s site.  A NOW account is required.  (Here)

It seems as though the SRA requires an iGroup with a VMWARE OS type.  Even in environment that are ALL NFS, the SRA wants to see an iGroup with a VMware type.  The way around the bug is to just create a dummy group to satisfy the SRA.

For those that don’t care to login to a knowledgebase (Read my Rant here), here is the way around your issue.

How To:

Create a dummy initiator group of ostype vmware.

Example:
igroup create -i -t vmware VMware

iSCSI or FCP license is not required to create an initiator group. The initiator group does not require that initiators are added.

Select any string for the name of the iGroup. In the example above, the name of the iGroup is VMware. An FCP dummy iGroup may also be created instead of iSCSI. However, FCP iGroup cannot be created on a MultiStore vFiler.

Hopefully this will be fixed in future versions of the SRA.

Click Here to Continue Reading >>

My Virtual Self on Life Support

Things have finally appeared to settle down for me online.  For a while, it just seemed as though the entire internet was under attack.  And it was REALLY beginning to impact MY LIFE!

If you are a regular reader of the blog, you may have noticed it go DOWN for a while early May.  Google’s blogging platform Blogger, which hosts this blog, as down for a good two or 3 days.  Initially, it was out for a couple of hours due to a maintenance update but that quickly escalated to an actual data loss and another 2 day restore period where the site was in read only mode.  No posting or comments allowed.  This was just a couple months after Google’s well publicized Gmail outage.

Speaking of email, my journey to the Microsoft cloud has also not been a smooth one.  From the beginning, there had been numerous issues outside of my control and the recent multiday outage was just another example of how frustrating it is when the cloud services go down.  BTW: Google, Microsoft and Yahoo have all suffered outages recently.  For me, the Microsoft one is a bit more unforgivable since they are positioned for Enterprises.

That said, these events have definitely not stopped my march toward the cloud though.  I am finding myself migrating more and more of my utilities and workflow to the cloud.  Password management has been the latest with my move toward LastPass.  (Of course, one month after moving all my passwords up to LastPass, they suffered a small possible security breach! )

Of course with the blogger outage, I saw numerous posts and tutorials pop up with instructions on how to migrate from Blogger to WordPress online.  Moving from one cloud provider to another seems like a false security.  From my perspective, it could have just as easily been WordPress down for days. 

For now, things are getting back to normal – I think I’ll take a break and fire up the PlayStation. – DOH!

Click Here to Continue Reading >>

Troubleshooting paused Hyper-V Virtual Machines.

After months and months of stable usage, your Hyper-V Virtual Machines all begin to go into a PAUSED state!

What is going on?  After clicking resume, they seem to run a few seconds before entering a paused state again. 

The most common reason for this is a lack of disk space on your cluster drive.  Your cluster drive is typically mounted in C:\ClusterStorage on all of your Hyper-V hosts in the cluster.  This is where all Virtual Machine files including expanding VHDs and snapshots are typically located.  This is also where you need to free up some space to resolve the pause condition.  Try deleting any non critical VMs and removing unnecessary snapshots. 

After freeing up some space on your cluster drive, your Virtual Machines should be happy to resume their network duties.  

^{[A quick word about snapshots]}
Just like in VMware, unattended snapshots can run anyone’s day.   Snapshots are meant to be quick temporary point in time restore points to allow an administrator to quickly back out changes made to a system at that particular time.  Very useful for program changes, service packs and patches that occasionally don’t go as planned.  Once the testing period is over, snapshots should be deleted or committed.  This holds true for any hypervisor you happen to be running.

Click Here to Continue Reading >>

Unable to add ESXI host to VirtualCenter

Adding a host to vCenter is usually a pretty trivial thing.  Click. Done. Usually. 

Sometimes (especially after a repair or move from one vCenter to another), you can get a “Failed to configure the VIM account on the Host” error.  This is usually a quick one to fix by deleting the local VPXUSER account on the local host.

To access the local accounts on an ESXi host, launch the Virtual Infrastructure client and point it to the LOCAL host rather than the vCenter instance.  Once you connect with root, you will see a Local Users & Groups tab.  Deleting the VPXUser account is a mere right click away.  Re-Adding the Host to vCenter will recreate this account and establish the appropriate relationships between host and management server.

Click Here to Continue Reading >>

XenDesktop Agent not registered in Citrix Delivery Services Console

There are lots of reasons that a XenDesktop desktop machine would appear as Not Registered in the Citrix Delivery Services Console.  This particular Virtual Machine was being provisioned by Citrix Provisioning Services.  As the desktops boot up, the Citrix Virtual Desktop Agent (VDA) service communicates with the Citrix Desktop Delivery Controller (DDC) and registers itself to notify the controllers that it is ready to accept user requests. 

Failures in the registration typically occur due to a break in communication (no valid IP, no DHCP, no Network connection, etc.) but a lack of access to the Active Directory will also prevent a successful registration.

Pulling up the console for this particular desktop and attempting to log in presents me with the familiar domain trust issue that is common with provisioned machines.  Jumping over to the Provisioning Console allowed me to recreate the Domain Object in AD for that machine. 

Of course, once you change the Domain Object for a XenDesktop Machine, be sure to REASSIGN the Object to the Published Virtual Desktop in XD Delivery Console.   Reboot and you should be good.

Click Here to Continue Reading >>

ESXi Disk Full.

While working in my Lab, I was rebuilding an ESXi host and began to run into a strange issue.  I was unable to save any configuration changes to the host from within vCenter.  The server was a brand new installation yet was complaining about having no access to write to the disk.

Symptom:

After restarting the host, I would be able to make configuration changes for 3 – 5 minutes from when the machine started before beginning to experience updating errors.  The error seemed to indicate that the hard disk was full and it was unable to write configuration changes to /etc/vmware/esx.conf.

My particular lab runs a non standard ^{*(non supported)} motherboard (SuperMicro X8SIL) that seems to have some issues with the VMware implementation of CIM.  The Common Information Model is used to gather information about actual sensors in the hardware.  Since my board did not play nice with VMware, it flooded the log files and thus after a couple of minutes from boot up, the disk would fill.  A quick fix was to disable the Sensor Dashboard in advanced settings.

Workaround:

After restarting the host and BEFORE the disk filled, quickly change the Advanced Host value UserVars.CIMEnabled from 1 to 0.  This corrected the issue immediately.  Of course, I sacrificed the ability to view the board sensors from within Virtual Center.

Click Here to Continue Reading >>

Terminal Server license server - NOT ACTIVATED!

Written by Aaron Silber:

Just ran into an issue where all of a sudden people could not log into the TS Servers any more. When I looked at the license server, I noticed that all of a sudden my TS License server (running on a Windows 2008 Domain Controller) which had been working fine and without issues, all of a sudden was reporting that it was not activated.

Weird I thought, but ok I guess I’ll just reactivate it, sounds simple enough right? Not so fast, attempting to reactivate produced this lovely message:

 
^{The Activate Server Wizard encountered an internal error from the license server : Message Number: 0xc0110011}

Luckily a quick check on the Microsoft KB turned up a hotfix to correct this issue, something to do with corrupt certificate and/or a certificate that is part of the digital certificate chain expires. The Microsoft hotfix to resolve this issue can be downloaded from this KB article: http://support.microsoft.com/kb/983385

Aaron

Click Here to Continue Reading >>

Installing VMware Site Recovery Manager : Please wait while setup is initializing…

Does this screen look familiar?  Double clicking the SRM installation executable and you will be presented with this familiar initialization bar.  The issue is that it will NEVER complete if you are running on Windows 2008 R2.  After a bit of playing around, right clicking and launching ‘As Administrator"’ allowed the installation to continue.

As a test, I then tried to uninstall the application from the Windows 2008 R2 Control panel and the same issue presented itself.  Not sure how to ‘Launch as Administrator’ from the Control Panel but if you launch the original setup executable again, you will have the option to uninstall.

How about if you need to add parameters to the setup (as is the case for many to one installations), just make sure you launch from a ADMINISTRATOR:CMD window.  From there you can launch with <SRM Installer.exe> /V"CUSTOM_SETUP=1" for instance.

Click Here to Continue Reading >>

Hyper-V Networking

If you are familiar with VMware’s style of networking, with their port groups, vSwitches and native teaming, then Hyper-V’s networking might be a little intimidating.  Not that you can’t accomplish everything you need with Hyper-V networking, you just need to know where to find everything.
The differences between networking within VMware and Microsoft seem to boil down to approach.  VMware made the decision to handle most of the main networking functions from within the hypervisor or it’s own management software while Microsoft elected to rely on third party players and their individual software.  This distinction can lead someone who is familiar with VMware to see the Microsoft solution as convoluted or confusing while Microsoft loyalists will underscore the potential performance benefits and wider compatibility.  Whatever your take, here are some of the basics of Microsoft Hyper-V networking.
Teaming
All the teaming in VMware is handled by vSphere.  NICs on a physical host are just added to vSwitches and *poof*, Teamed. :)
 
On Hyper-V, you would leverage the Network Card manufacturer's native Windows tools to accomplish NIC teaming.
Depending on the manufacturer; the software features, functionality and of course Management interface will vary.
In this example, The HP Network Configuration Utility is provided to create teams across both HP ONboard NICs and the additional quad port card.  Make sure you are running the latest management software and drivers and read the release notes for any known Hyper-V issues.
For HP, this is also where you could create additional VLAN configurations (under the VLAN[802.1Q]) button.

Once the NICs are teamed, you will see an additional NIC presented to your Windows OS.  This is typical for Windows teaming software.  This new ‘Teamed’ NIC is what you will reference in Hyper-V.  Unfortunately, the network connections page on your hosts can get a bit crammed and confusing.  I do wish Hyper-V or the Teaming Software would hide connections that are part of a team to make the interface a bit easier to digest.

At this point, with the Teamed NIC created, you can now assign it to a ‘vSwitch’ in Hyper-V.

So in Hyper-V, you get the same net effect as long as you know where to look.  While VMware admins might find these steps cumbersome, if you have been building out physical Windows servers, most of this should seem pretty familiar.

Click Here to Continue Reading >>

EMC World 2011–View on vBlock notes

 
Joel Ramirez is currently in sunny Las Vegas at EMC World 2011.  Despite the ever increasing temptation of Blackjack tables and Showgirls, he managed to send me some of his thoughts on the sessions he attended.  I know he loves to share with the community but I am pretty sure it’s just a ploy to hang out in the Blogger’s Lounge! 

In the VMware View for vBlock session this morning, Hari Kannan, a VCE technologist for EMC, talked about the challenges in designing Virtual Desktop solutions. I believe the approach that EMC favors is a practical one. The components of the vBlock architecture will make sense to a datacenter administrator:

• backend storage, VMax or VNX
• the computing platform, Cisco UCS
• the VDI solution, VMware View on VMware vSphere

The idea behind rolling out desktops to thousands of users is kept simple by the EMC VCE team. Categorize users into three distinct profiles based on their I/O and usage tendencies (i.e. do they use a lot of network storage or code locally or use a lot of Adobe Flash?).       

These end up being one of three types of users: task users, knowledge users, and power users. They categorize servers into three distinct profiles based upon the core and memory resources that are required: gold, silver, and bronze. Profiling the users help size out the UCS solution requirements while profiling the servers helps with quick deployment of future servers.

As any datacenter administrator will attest to, procuring resources and deploying a new server can be exceptionally time-consuming. Approaching these tasks from a familiar perspective is the job of the Cisco UIM and VMware View interfaces. The UIM manages the server profiles and executes the deployment of a new server to the available UCS resources in a transparent manner to the admin.

The View Manager has the vAAI capability to offload the copying of the gold image to the storage array, where the greater resources of the intelligent storage platform can easily manage the copy of thousands of desktop images, releasing the virtual environment’s CPU and memory resources to focus on processing production I/O from the end user community.

My take on Joel’s notes are EMC’s and VMware’s increased synergy between hypervisor and storage.  I keep seeing the two companies build more and more of a competitive advantage with each other for these types of solutions.  Can you run mixtures of XenDesktop, Hyper-V and other parts in your solution? Of course!  The abstraction of the various layers within the stack allow for competitive swaps and is great for finding the RIGHT technology for your particular company. 

Just know it might not be considered a vBlock though. 

Click Here to Continue Reading >>

Unsolvable Hyper-V / SCVMM issue : Error 2912 coupled with a VMRC error.

Hyper-V is definitely growing on me the more and more I use it at client sites but occasionally something will come along that makes me yearn for the good old days of ESX. ;)  This was one of them.  Two identical physical servers, both with Windows 2008 R2 installed running the Hyper-V role.  I was configuring System Center Virtual Machine Manager to manage them and was consistently receiving this error whilst trying to add Host 1.  Host 2 added without issue.  This process is similar to adding Hosts to vCenter.  Host 1 for some odd reason would just not add.  Network connectivity was fine since I could ping back and forth, upwards, downwards and sideways without issue.  I ran through pretty much every troubleshooting idea I could find on the internet (Rebooting, Uninstalling Roles, Clearing out Certificates, Resetting WinRM, Turning off firewalls and antivirus, etc..) but could not get this thing to add correctly to SCVMM. 

Unfortunately, error 2912 seems to be a catch all error for SCVMM providing no real concrete errors other than something went very wrong.
After a great deal of failed troubleshooting, I ended up rebuilding the entire server fresh and proceeded without issue.  I just don’t know what went wrong and that irks me.  I can only really assume that it was a failed Windows Update or incorrect patch sequence that went awry on one and not the other.

On a more positive note though, I did run across a great tool to assist in tracing communications between Hyper-V hosts and SCVMM in VMMTrace.  Check it out.

Click Here to Continue Reading >>

Missing Active Directory option in Citrix Provisioning Services

I ran across a pretty puzzling situation earlier last week that I thought I would post about.  I was implementing a fairly straightforward XenDesktop POC and had set up some Windows 7 VMs which were pulling their images from a Citrix Provisioning Services server.  The VMs were booting up correctly and PXE’ing their OS but when attempting to log in, I was receiving a pretty familiar ‘Trust Relationship’ error message.  I have written before about trust issues with Citrix Provisioning Services and almost always, it is due to the AD object not being created from the PVS console.  Easy fix.  Head to the PVS console, right click and create the object.

If you look at the first screen grab below though, I didn’t have a menu for Active Directory functions!  I thought I was going a bit crazy.

After a bit of Googling around, I discovered that when running the PVS console remotely, (not from the PVS server directly), the Active Directory functions are hidden for security purposes.  Unfortunately, this was not my issue.  I was running the console locally but still not able to access the AD functionality.  I tried a few things without luck before determining it was a database corruption.  Fortunately, I was at the beginning of the PVS configuration so a quick rebuild of the database resolved the issue without much data loss.

Click Here to Continue Reading >>