On a sunny day in februari suddenly all hosts in one cluster give the folllowing error: Error: Cannot synchronize host s02esxhyp000003 Cannot complete login due to an incorrect user name or password. Removing the host from the cluster and re-add did not help, neither did restarting the services: service vmware-vpxa restart service mgmt-vmware restart. Every action i do in the cluster gives me the following error "vim.fault.Invalidlogin": Finally good old mister Google came up with the following solution. 1. Disconnect the host from vcenter (do not remove) 2. Run from the ESX console "userdel vpxuser", or connect to the host using the viclient and Delete the vpxuser 3. Reconnect the host in vCenter, it will first give an error about a wrong password and then starts the Add Host Wizard. I came across a nice knowwie when i was troubleshooting this error. The next time you add a host to a cluster watch the "Recent Tasks" that appear in a viclient connected directly to that host. So indeed start another viclient and connect directly to the host, then go back to the viclient connected to vcenter and add the host to a cluster (preferably one with resource pools).

After waiting for 18 agonizing days i finally received the results of my VCAP-DCA exam.

I am very glad to say i passed and can now officially be called a

"VMware Certified Advanced Professional - Datacenter Administrator"

Admit it!; it is a mouth full

 

Next up is the VCAP-DCD exam, i scheduled the Datacenter Design exam for friday the 30th December.

 

 

 

On wednesday september 7^th ITQ organized a “vSphere 5 What's new” session. The session was held in the Van der Valk hotel en Breukelen and was attended by approximately 40 people.

In this session Paul Geerlings introduced some new features like the “monster VMs”, Improvements in networking and changes to the available ESX versions. Then Dennis introduced us to some awesome new storage features. His talk was mainly about storage DRS and how it can help you simplify the administration of datastores and improve performance at the same time.

After the break I started out by talking about some new technologies which were introduced at the VMworld in Las Vegas: VXLAN, AppBlast and Octopus.

Then I went on and introduced the audience to the brand new vCenter Server Appliance which is a virtual appliance based on SLES 11. This appliance gives us the ability to quickly deploy vCenter without going through the hassle of installing an OS and a DB. It even delivers all the component needed to start using Vmware Auto deploy.

After a demo of the vCSA and Auto deploy (which went rather well considering it was a demo) I quickly introduced some new features in VMware SRM 5 namely: Host based replication and more flexible startup sequence possibilities.

At the end of the session Paul had the honor to explain the new licensing which basically comes down to this: Each sockets license you buy has no restriction on number of cores but comes with a vRAM entitlement. The entitlement is added to the vRAM pool so it can be used anywhere in your vSphere environment instead of being tied to one socket. If you run out of vRAM entitlement you have to purchase additional sockets licenses.

Thanks again to everybody who attended this session. I hope you had as much fun being there as I did.

By the way: SRM 5, View 5 and Workstation 8 are available for download as of today.

 

Today i received the following email from vmware about a change to the announced new licensing model for vSphere 5.

On the one hand i think its good that they listen to the community but on the other hand they can ask themselves "Did we think this new licensing model through enough before announcing it?"

On July 12, 2011, VMware announced our new Cloud Infrastructure Suite. The launch featured vSphere 5, the newest version of our flagship product.

As many of you know, as part of this announcement, we introduced changes to the vSphere licensing model in order to align costs with the benefits of virtualization rather than with the physical attributes of individual servers. While our goal was to provide a licensing model based on consumption and value rather than physical components and capacity, we strived to make the new model as non-disruptive as possible.

These changes generated much debate in the blogosphere, in conversations with our partners and customers, and across VMware communities. Some of the discussion had to do with confusion around the changes. We have been watching the blog commentaries carefully, and we have been listening to the partner and customer conversations very intently. A great deal of feedback was provided that examined the impact of the new licensing model on every possible use case and scenario, and equally importantly, reflected our partners’ and customers’ intense passion for VMware.

Our success depends on the active involvement of our channel partners. We are a company built on partner and customer goodwill, and we’ve taken your feedback in earnest. Our primary objective is to do right by our customers, so we are announcing three changes to the vSphere 5 licensing model that address the most recurring areas of your feedback.

Finally, we introduced the vSphere Desktop edition to address vSphere licensing in a desktop environment. The vSphere Desktop edition does not have any vRAM entitlements, and allows customers to purchase vSphere for VDI use case on per user basis. Our price books are being updated and will be distributed shortly.

One of the new features of VMware vSphere is booting the ESXi hosts from PXE. This feature is called Auto-Deploy and gives you the ability to use stateless ESXi servers.

Below is a step-by-step guide to get you started. It assumes you use the vCenter Virtual Appliance. of course auto deploy also works with any other DHCP/TFTP server and auto-deploy itself can be installed on a Windows vCenter server as well.

1. Open the vCenter administration webpage (https://<vcenter>:5480), Go to the services tab and make sure the ESXi services are started. If not click “Start ESXi Services”.
2. Configure DHCPd in /etc/dhcpd.conf. Configure at least the following items.
   1. DNS / GW
   2. Subnet
   3. scope or reservations
   4. next-server
   5. filename: undinonly.kpxe.vmw-hardwired
   6. uncomment or add the keyword autoritative;
   7. My dhcpd.conflooks like this:

option domain-name "vmtopia.lab";
default-lease-time 600;
max-lease-time 7200;
autoritative;

log-facility local7; 
 allow booting;
 subnet 192.168.16.0 netmask 255.255.255.0 {
   range 192.168.16.150 192.168.16.160;
   next-server 192.168.16.100;
   filename "/tftpboot/undionly.kpxe.vmw-hardwired";
 } 

1. bind dhcp to network interface by editing the /etc/sysconfig/dhcpd file
   Add or change this line: DHCPD_INTERFACE = “ANY”
2. Start DHCP: service dhcpd start
3. Make sure the boot files are in /tftpboot. The VCVA comes with the right files.
4. Start atftpd: service atftpd start
5. Open file /tftpboot/tramp to check if hostname is used. If no DNS available replace vcenter hostname for IP address
6. If you want to make sure your hosts can access tftpd even if DNS fails or when hostname is not in DNS you have to edit /etc/hosts.allow. At the bottom of the file change ALL : KNOWN to ALL : ALL
7. Poweron an ESX server. Check if it loads gPXE. As the information screen shows, it won't boot because there is no autodeploy rule associated with that host. The host will keep rebooting so power it off for now
8. The next step is to create autodeploy rules which associate the hosts to a boot image, host profile and a cluster or folder. The only way to create those rules is by using PowerCLI. So install the vSphere5 PowerCLI on a windows box, start it and login to vCenter using the  connect-viserver commandlet
9. Now we have to load an ESXi depot. The default ESXi depot can be downloaded from the Vmware site <<LINK>>. Now load the depot using the following powershell command:
   Add-esxsoftwaredepot <path to depot file>
10. In the default depot there are two image profiles. An image profile is basically a collection of VIB files. A VIB file can contain the base system, Vmware tools, drivers etc. We can show the profiles available in the depot by using:
    get-esximageprofile
11. To make a new ESX server boot from one of the profiles we have to create a rule. A rule has three properties:
    1. Rule name: Just give a descriptive name here like “imageprofilerule”
    2. Item: this is the object you want to associate with a host or group of hosts. In this case it will be an image profile (ESXi-5.0.0-381646-standard)
    3. Pattern: The pattern defines to which hosts this rule will apply. There are a lot of different variables you can put into the pattern like: Vendor, Model, MAC, Asset Tag and IP range. For now we will just use the IP range.
12. Create a new rule:
    new-deployrule -name imageprofilerule -item ESXi-5.0.0-381646-standard -pattern “ipv4=192.16.0.100-192.168.0.200”
13. After creating the rule, the rule has to be added to the active rule set:
    add-deployrule imageprofilerule
    To show the current rule set use: get-deployrule
14. Now autodeploy knows which image to boot. The host will be added to vCenter automatically but we have to tell autodeploy where to put the host (which datacenter or cluster). So let's add another rule:
    new-deployrule -name clusterrule -item cluster1 -pattern “Model=”BL 460c G6”
    Note that i used the server model in the matching pattern here so there will be only one type of machines in the cluster.
15. And add it to the ruleset: add-deployrule clusterrule
16. Now poweron your ESX server and watch the magic happen :)
17. Ater your first ESX server is started you can configure it, create a hostprofile from it and create a rule to make sure all your servers will use that hostprofile. I will discuss host profiles in another post.
    new-deployrule -name profilerule -item <hostprofilename> -ALLservers
    add-deployrule profilerule
    If you don't want all your servers to use the same profile you can also use a pattern here instead  of “allservers”
18. Now poweron all 10000 ESX servers, don't tell your boss you're ready installing all servers, leave the office pretending you need to be in the datacenter and enjoy you afternoon in the pub :)

cheers!

 

Today i was invited by VMware to participate in the VCP5 beta exam program.

The official exam code is:

VCP511: VMware Certified Professional on vSphere 5 (Private Beta - Invite Only)

It is only available for the official RCBETA 5 program members.

I decided i am going to take a shot at it on friday july 22nd. the beta exam periode only runs from july 12 to the 24th so there is not much time to prepare.

 

VMware just launched vSphere 5. Release date is not known yet but I expect it to be availeble very soon.

The vSphere 5 suite consists of:

1. ESX 5
2. vCenter 5
3. SRM 5
4. vCenter operations 1.0
5. vShield security
6. vCloud Director 1.5

FeatureHighlights:

• "Monster VM" capabilities: Create VM with 1TB of RAM, 32 virtual CPU's and capable of 1000000 IO per second!
• New HA capabilities
• Storage DRS and Policy driven storage
• Sateless ESX server deployment through autodeploy appliance (PXE boot)
• 3D graphics for Windows Aero
• vCenter server Virtual Appliance based on SLES Linux (Windows installable is still there)
• Client connected USB devices and USB 3 support
• SRM 5 will offer hypervisor based replication to replace array replication where needed
• vcloud director will provide linked clones
• And there will be an iPad vCloud director app!
• A little more on this page: http://www.vmware.com/products/vsphere/upgrade-center/overview.html#whatsnew

Licensing:

• ESX will be licensed based on vRAM pooled across the entire environment. Limitations on number of cores or sockets are lifted, you just pay for the amount of memory your VMs use.
• The Advanced version will disapear. Customers running the advanced version will get a free upgrade to the enterprise version.
• When you purchase vSphere 5 you are entitled to the free use of SUSE Linux Enterprise Server for VMware

Last Friday I took a shot at the VMware VCAP-DCA exam.
I kinda knew what to expect, but still it was a toughy!

Its all labs, no multiple choice questions.
You get 4 hours, which in the end isn't a lot, its not extremely difficult but it just is a lot.
There are PDFs at hand but you hardly have time to check them.
There was one time when i could have used the Hardening guide but alas it was nowhere to be found
I think i flunked it, big time, because i ran outta time after only 70% of the questions.
I will get the result within 10 business days, i'll post again when i receive the bad news

Tip:

Immediately skip the questions you do not know.
When there is time left try to do them using the PDFs.

 

13-072011 - Yep, i flunked it.

The number of point wasnot all that bad considering i missed about a third of the questions due to a lack of time.

 

Should you do that? And by routing I really mean routing in the sense that an IP router is moving the traffic from one subnet to another. Your first response would hopefully be: “That is not a best practice, Is it?”. Of course it is a best practice to have your servers on the same sub-net as your storage. In fact to me this has always been so obvious I never tried it otherwise.

Until a week ago. A customer asked me to investigate their storage performance troubles. It didn't take long before I figured out all their NFS and iSCSI traffic was passing a router. OK, it was the routing engine of their core switches which was supposedly very fast and low latency etc. But it turned out the routing added at least 25msec(!!!) of latency to the storage I/O. When storage and router were busy the latency spiked to above 100msec. After we took the routing out of the equation the latency dropped to an average of 3msec.

Lesson learned: never route your storage traffic.

Even if you would increase the routing capacity by 500% you will run out of capacity very soon. This is because a lot of the network traffic is likely to pass the router 3 times. For example: You copy a file from server A in sub-net one to server B in sub-net two. So server A reads the data from storage, this traffic flows through the router. Then the file is copied to server B and because this server is in another sub-net the data passes the router again. Then server B writes the data to the disk so it has to pass the router once again! Needless to say that in an expanding environment the router utilization will reach its limits very soon.

 

 

If you have deployed large numbers of ESX server you are probably familiar with scripted installations, the UDA appliance or things like Rapid Deployment. But did you know VMware also delivers an auto deployment appliance? It is called VMware Auto Deploy and it is slightly different compared to other deployment mechanisms: VMware Auto Deploy does not install a copy of ESX to harddisk but loads ESXi into memory using PXE. ESXi is than started from memory en configured using auto deploy plugins and host profiles. This means you end up with a stateless server which is fully configured every time you boot it. This also means that you don't have to create scripts to install or configure ESX.

I use this appliance in my home lab to quickly deploy any number of (virtual) ESXi servers I need. Of course the Vmware auto deploy server becomes a single point of failure so I doubt if it is usable outside a lab environment for now.

Vmware auto deploy comes as a virtual appliances which is actually the same appliance as the vMA but with some extra's. You can find the appliance at Vmware Labs. You can find administation manual here.

 

To help you setup the appliance quickly, here are the steps I followed to get the thing up and running:

1: Deploy the appliance Make sure to configure the networking correctly

2: Make sure the appliance is the only DHCP server in your lan or lookup the administrator guide on how to use an external DHCP server

3: Logon to the appliance using the vi-admin account

3: Enable the DHCP server (if you are not using external server) using this command:

sudo deploy-cmd dhcpconfig

4: Add your vCenter server as a target:

sudo vifp addserver <vcenter ip>

5: Now you have to create a new boot profile or edit the default profile. A boot profile determines, among other things, to which vCenter server the ESXi server is added, which host profile to use an in which datacenter and cluster the server will be placed. Add vCenter to the default boot profile:

deploy-cmd updateprofile --name default --vcenter 192.168.0.250

6: Boot the first ESXi server. Make sure it is using PXE.

7: Configure the server and create a host profile

8: Edit the host profile as nessecary. Make sure the profile sets the administrator password

9: Assign the host profile to the boot profile:

deploy-cmd updateprofile --name default --hostprofile autodeploy_profile

10: Additionally you can add vCenter location where the server will be added (datacenter / cluster):

deploy-cmd updateprofile –name –hostfolder=/<datacenter>/<cluster>

11: Start another server to check the results. If everything works correctly start the next server, and the next, and the next.....