Results for category "Windows Server 2012 R2"

HP BL460c Gen8/ Eumlex LOM/ VM Network Disconnect

You may be already aware of the issue related with HP Bl460c Gen8 servers using FlexFabric.

http://www.hyper-v.nu/archives/mvaneijk/2013/11/vnics-and-vms-loose-connectivity-at-random-on-windows-server-2012-r2/

I see that lot many are impacted due to this issue – However its still not clear where the issue is. The only details I see on this issue is on the blog from Marc van Eijk.

As per my observation, the issue happens only for the VMs which generates network load. And at a random point, the VM will get disconnected from network though the vNic stands connected. At this stage, you have the below options.

  • Disconnect the vNic from the Virtual Siwtich and then reconnect
  • Change the VLAN ID and then revert back to the orgnial VLAN ID
  • Reboot the VM
  • Fail-over the VM to another node (Least preferred)

Failover some times cause the VM to get stuck at saving state. And if that’s the case, migration will never happen and we may have trouble to get out of this crisis.

Get Stats Failed - Event 76

Get Stats Failed – Event 76

 

On the event viewer, you can refer for Event 1d 76. And the workaround for this issue is to start moving one more VM from the same server which will solve the dead lock for the problematic VM and move successfully to the next node. This worked for me all time.

 

Expanding a CSV Volume

Recently, I had a challenge with free space on one of the CSV Volume. The immediate idea is to expand the volume.  Is that viable?

Yes. Here is how I did.

Expand the volume on storage side first.

As CSV is a cluster resource, The Owner of the resource will have the privilege to perform most of the tasks. So identify the owner of this CSV Disk from fail over cluster. It should be definitely one among the servers in the group. Perform the next steps from the Owner node.

Once the volume is successfully expanded on storage side, we should be seeing the change in disk management like below. The added space will be marked as unallocated space.

 

Expanding a CSV Volume

Verifying DISK on Disk Management

 

The increase in capacity will be visible on the Failover Cluster Manager. However, the usable space will still be the old until we expand.

Expanding CSV Volume

Verifying DISK capacity fro Failover Cluster Manager

 

The next step is to extend the volume from Disk Management. Please note – this option will be grayed out if not logged into the Owner node.

Expand CSV Volume

Extend Volume

 

Go ahead with extending.

Extend Volume - Size Available

Extend Volume – Size Available

 

Expanding CSV Volume

Disk Expanded

 

Great ! This change should be reflected even in Fail-over Cluster manager as well as SCVMM.

Expand CSV Volume

Verifying after Expanding CSV

 

 

Expand CSV VOlume

After Disk Expansion – Change reflected on SCVMM

 

I had VMs running on the CSV Volume and all was safe ! However, I did the testing only for expansion. Shrinking the disk even throws a warning that data may erased and hence, couldn’t try. Will test it shortly :D.

 

Creating a Gen2 VM Template for SCVMM 2012 R2

In my previous post, I did mentioned about creating templates – how ever it was for Gen1 servers. With Windows Server 2012 R2, We have a new generation of VM – AKA Gen2.

Today I would like to give a brief on how to create a Gen2 Template. Usually, I follow the traditional method of building a VM on the Hyper-V host and prepare it for becoming a template. As we did the template creation for Gen1 VM, The whole process is divided into three stages.

 

The first stage is to prepare a VM for the template.

  1. Create a Gen 2 VM – I will name it as Golden Image
  2. Windows Patch Updates
  3. Install HyperV Integration Component
  4. Antivirus/Agents or any of the custom software required on all servers deployed using this image
  5. Enable RDP
  6. Disable Firewall
  7. Set Administrator password blank
  8. Export the VM – This is required if we need to update template on a later stage
  9. SysPrep and Shutdown

The second stage is to configure the per-requisites on Hyper-V for template / deployment using template.

  1. Run As account for Local Administrator – Create if not created already.
  2. Run As account to be used with ADJoin which has the rights to Join Computers on AD – Create if not created already.
  3. Gust OS Profile – Create only if not already created.
  4. Create Gen2 Hardware Profiles

In this second stage, Run As Account and Gust OS profile can be the same which was created for a Gen1 VM. How ever, we need to have a Gen2 Hardware profile created. As you know, the architecture itself is different between a Gen1 and Gen2 VM. Hence, Its important to have a Hardware Profile created for Gen2 Server. As all other steps are similar, I will only walk through this now.

Navigate to SCVMM -> Library -> Profiles -> Hardware Profiles

Create a new Hardware Profile

On the General Page, Enter the Profile Name

Make sure Generation is selected as Gen 2.

Gen2 Hardware Profile 1

Configure the Hardware components as per your requirement.

On the Firmware page, Enable Secure Boot should be checked (By default, its checked. Don’t uncheck it).

Gen2 Hardware Profile - Secure Boot

And that’s it. We have the Hardware profile for Gen2 VM. Now we are ready for stage 3 which is the template creation.

 

In the Third Stage, I assume that we have a Gen2 VM which is prepared with SysPrep.

Navigate to SCVMM -> VMs and Services -> All Hosts

Identify the Golden Image VM which we created and performed SYSPREP.

Right Click on the VM and select Properties.

Ensure that the Operating System and Integration Service is properly updated in SCVMM.

If Operating System is displayed as unknown, Select the right Operating system and Click on OK.

Template VM- Properties

Few experts has mentioned that we need to enable the check mark for  “Contains the Operating system for the virtual machine”. Logically its correct and hence I too did that. Don’t ask my if its mandatory.

Template VM- Disk Properties

Thats it. We are ready to create the template.

Right Click on the VM and Select Create -> Create VM Template

Read the warning and click on OK.

Template Name

On the Hardware Configuration, Just click next.

On the Operating System configuration, Select the appropriate Gust OS Profile.

Create VM - GUST OS PROFILE

Select the VMM Library Server

Select the path where you need to store this on Library. I will usually put under the Templates folders inside. You can even create a folder structure as you wish.

Review the summary and click on Finish.

Once the template is created, create a new VM.

Template Selection

Select the template and click on OK.

Click next to proceed.

Create VM

On the Configure Hardware Profile, Select the Gen2 Hardware profile we just created.

Create VM - hardware profile

On the Gust OS Profile, Select the appropriate GUST OS Profile.

Create VM - GUST OS PROFILE

And proceed with the next steps what we did for a Gen1 VM creation.

 

ODX along with SCVMM 2012 R2 – Fast File Copy

The next step after ODX is to make use of it in the real scenarios. In my viewpoint, Machine deployment will be the best consumer who can showcase a significant performance improvement using ODX. The traditional deployment using SCVMM make use of BITS over Network. And I see that for a deployment, which takes 10 minutes – more than half of the time will be used for moving the VHDX file to the destination HyperV Server.

With VMM 2012 R2, we optimized the speed for VM deployment from VMM library by leveraging Offloaded Data Transfer (ODX).  Many large environments use SAN storage.  ODX is a feature introduced on Windows Server 2012 which automatically orchestrates and optimizes the use of SAN storage by using tokenization on reads and writes without using buffers.  With this capability, copying the VM can offloaded to the SAN device which decreases the server cpu utilization and decreases network bandwidth consumption which provides faster VM deployments. To use ODX, SAN storage must support this feature.

Thansk Keiko Harada for making this post.

I just tested out ODX along with SCVMM 2012 r2. I am well impressed as the performance gain is clearly visible. The VM Deployment from templates only require few minutes compared to the time consumed before.

The current environment where I tested ODX contain Windows Server 2012 R2 Hyper-V Cluster along with SCVMM 2012 R2. Hyper-V Servers and SCVMM Server got disk allocated from an ODX capable SAN. SCVMM make use of the SAN Volume for library.

For ODX to work with SCVMM – Here is the Prerequisites

  • Only for new VM deployment from the VMM library
  • RAA (run as account) on source and target
    • VMM and the hosts (source and target) must be on the trusted domain with no firewall among the environments.
  • VMM R2 agent
  • VMM R2 Server
  • Windows Server 2012 and above
  • ODX supported SAN storage

I got all other prerequisites except the RAA.

And for that reason Fast File copy was not functional. While trying to deploy a VM from template, On the HOST Selection page, This was getting clearly evident.

The message inside “Deployment and transfer explanation” says – Creating Virtual Machines using fast file copy requires the host hostname to have an associated Run As Account.

Fast File Copy Require assosiated run as account

In my case, the Run As Account was not configured initially. I created the Hyper-V Cluster from SCVMM but later – while tried to configure the Run As Account for Host Management, that option was disabled.

HOST Access - Run As Account Disabled

Some search on Technet forum pointed me to this page where we can configure the Run As Account through Shell.

$YourCluster = Get-SCVMHostCluster -Name YOUR-CLUSTER-NAME

$YourRunAs = Get-SCRunAsAccount -Name “YOURRUNASACCOUNT”

Set-SCVmHostCluster -VMHostCluster $YourCluster -VMHostManagementCredential $YourRunAs

After executing this, Run As Account got displayed in the Host properties.

Host Access - Run As Account Assigned

Now we are all set with the prerequisites for Fast File Copy.

Lets try it.

Now VM Creation is using FAST FILE COPY 😀

Deploying using Fast File Copy

See the total time took for completing the file copy. Just 50 Seconds.

fast file copy - start time and end time

The same template when deployed with out Fast File transfer took more than 4 minutes.

Deploying without Fast File transfer

Thus the total time took for deploying a VM came down drastically – Now its just 3 Minutes.

Create VM - Total Time

Enjoy !

 

ODX – Why to wait ? Make use of it…

I heard about ODX in Windows Server 2012 Boot Camp which happened in Sep 2012 @ Dubai. Though I was impressed and a bit excited on these new technology, I realized that our existing storage don’t support ODX. And in late 2013, I got a chance to work with a storage which supports ODX. I searched a bit in Google to see how we can implement ODX.

All the documents says ODX is enabled by default. What next ? I need to enable and feel it. And at last I did it. 😀

Lets me brief those steps. My experiments was using Windows Server 2012 R2 Hyper-V and Windows Server 2012 R2 with SCVMM 2012 R2 both having SAN Disks from ODX capable SAN. I am planning few more tests which I will update on the following posts.

1) Verify that storage supports ODX

The best place to check this is Windows Server Catalog .

Navigate to Hardware Testing Status and search for the specific storage mode which needs to be checked.

Confirm that ODX is supported. Windows Server Catalog is a site hosted by Microsoft. If Microsoft has tested and certified, it should work well.

If your storage is not listed here, the next step is to contact the Storage vendor directly to confirm on ODX. I forsee a chance where a specific device may be supporting ODX, how ever if Microsoft has not yet tested – it will not be listed in the server catalog.

ODX - Compatibility

 

2) Storage Firmware is the second item to be checked. In my case, this particular storage will support ODX only if the Storage Firmware is above a certain level. Double check with the storage vendor. Its also recommended to make the server firmware and drivers updated.

3) On the Hyper-V Server and SCVMM Server, Verify the status of ODX

fltmc instances

Now, we need to check the supported features for the filter.

Registry Value - SupportFeatures

SupportedFeatures 3 means ODX is supported.

This is by default enabled. However, Just to confirm that no one has disabled it by mistake – check this too.

Get-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name “FilterSupportedFeaturesMode”

ODX Enabled

FilterSupportedFeaturesMode 0 Means ODX is enabled.

4) What next ? Nothing other than testing ODX. 😀

The first test is by copying a 33 GB folder from a CSV Volume to another ODX enabled SAN volume on a remote server.

As ODX is enabled, we should not see the traffic passing over network. To montor the network traffic, I am using the simple performance monitor inside Task Manager.

odx test 2

 

As I am connected over RDP, a slight traffic is expected on the management interface. However, the current data transfer speed on on the Copying job is 266 MB/s. And the traffic (Send) currently over the management network interface is 312 Kbps. The graph for the network interface traffic will give you an idea on the data transfer in the last 60 seconds. The maximum on the graph is 11 MB/s.

The next test is to know the behavior while copying the same file from the ODX enabled CSV Volume to a non ODX enabled remote share. We should see the traffic over network here.

ODX Test - Copying to ODX Disabled remote share

The data transfer on management interface is now 943 Mbps which is approximately 117 MBps. The speed on the copying job is 113 MBps which is almost matching. Thus we conclude that the data-transfer is happening over network.

To double check if the traffic we are seeing on the network is from the copying job, I just paused the copying job. See the difference on the traffic.

odx test - Pausing copy

Just after the copying job is paused, see the network traffic dipping to almost null.

And finally, I resumed the copying job which made the traffic to shoot up and hit the maximum.

odx test 6

# Reference – http://technet.microsoft.com/en-us/library/jj200627.aspx

 

 

RSS is limited to 4 queues

After I fixed the Processor overlapping issue using Set-NetAdapterRss, I noticed a new event.

HP FlexFabric 10Gb 2-port 554FLB Adapter #5 : RSS is limited to 4 queues. Enable Advanced Mode in the PXE BIOS to use up to 16 queues. This may require a firmware update.

 

Event 49

Event 49

To Fix this, We need to enabled Advanced Mode in PXE BIOS as mentioned in the event.

So we need to reboot the server. On the POST Screen, You will get an option eo enable PXE Bios. On this HP Blade, its CTRL + P.

PRESS CTRL P

Ctrl + P will open up PXE Select Utility. Advanced mode support will be disabled by default. You need to enable it.

ADVANCED MODE SUPPORT

The Personality should be selected as NIC.

Personality Selection

Save the setting first and then select continue.

From the next screen, You can just exist. And after the reboot, no more events are generated.

 

 

The processor sets overlap when LBFO is configured with sum-queue mode.

This event started just after I created the HyperV virtual switch.

 

The processor sets overlap when LBFO is configured with sum-queue mode.

The processor sets overlap when LBFO is configured with sum-queue mode.

 

As I couldn’t find much information on this event  on Blogs/Forums, I opened up a case with Microsoft for a quicker resolution. And the resolution which I got is to configure the HyperV Data Interfaces to use exclusive set of processors with respect to RSS configuration.

Set-NetAdapterRss -name vm-nic-1 -BaseProcessorNumber 4 -MaxProcessorNumber 7

Set-NetAdapterRss -name vm-nic-2 -BaseProcessorNumber 8 -MaxProcessorNumber 12

This event stopped after the next reboot, However, I got a new event – Event ID 49 – HP FlexFabric 10Gb 2-port 554FLB Adapter #5 : RSS is limited to 4 queues. Enable Advanced Mode in the PXE BIOS to use up to 16 queues. This may require a firmware update. I will write about this in my next post.

To my understanding, You can allocate a subset of logical processors to each HyperV Data interface. I am trying to understand more details on this. Based on a recent Technet Blog, VMQ will get enabled by default while we create a vSwitch. Once VMQ is enabled, RSS will be disabled on those interfaces. I will update this post once I get more details. If you can get this clarified, Please comment 😀 .

 

Update – Microsoft has published a KB article on this recently. Have a look on that too.

 

A port on the virtual switch has the same MAC as one of the underlying team members on Team Nic

On my Windows 2012 R2 HyperV Cluster, The event 16945 was getting logged frequently.

MAC CONFLICT

MAC CONFLICT

The current setup has two network interfaces teamed and this teamed interface is used for creating the Virtual Switch.  The virtual switch, the teamed inteface and one of the team member are having the same MAC Address which is the base of this event.

 

ADAPTER AND MAC DETAILS

ADAPTER AND MAC DETAILS

In my observation, This event will go off if we unchecked the option “Allow management operating system to share this network adapter” from Virtual Switch Manager. By doing so will remove the virtual interface – vEthernet and have the  configuration directly on Virtual Switch.

 

VIRTUAL SWITCH MANAGER

Its recommended to have dedicated interfaces for Management, HyperV Data, LiveMigrationa and CSV. If you have a dedicated interface for Management, this you can go ahead with this option.

 

 

Adding a node to existing Windows Server 2012 R2 Hyper-V Cluster

I started working with Windows Server 2012 R2 cluster few weeks back. At the initial time, due to the lack of time – I started with a three node cluster and later made it as a four node cluster and now adding one more node. I thought of making a post on how to safely add a node to an existing production cluster.

I personally don’t encourage changing the Tyre while driving the car :D, however sometime the situation may demand it.

Before start

  • Ensure that the new server is on same patch level with other cluster nodes
  • Ensure that the server is on the same Firmware level with other cluster nodes
  • Ensure that Fail-over Cluster feature is installed
  • Ensure that Hyper-V role is installed and the Virtual Switch is configured with same name and settings as of other cluster nodes
  • Ideally, All cluster nodes should be identical – So I assume the new server is also having the same hardware as other cluster members
  • Ensure the Multipath software has installed as of other cluster nodes
  • Ensure that the Network interfaces are configured properly to meet the requirement for Hyper-V
  • Ensure that NICs dedicated for Live Migration, CSV, Heartbeat etc are able to ping to the respective IPs in other cluster nodes
  • Add an additional SAN disk of smaller size to be used along with validation test
  • Ensure SAN disks are visible in Disk Management of the new server. Just check, Don’t try to make it online.

    Check if DISKS are visible in Disk Management

    Check if DISKS are visible in Disk Management

 

I would definitely go through the validation tests every time I create/add nodes to the cluster. This will give us the confidence that everything is configured and working as expected before the activity starts. Since we are trying to add an additional node to the existing cluster, We should understand that validation test may interrupt the VMs hosted on other nodes as the CSV disks may go offline as part of the test. For this reason, I did assigned one more temporary disk and made it as a CSV so that we can select only this specific disk while performing the validation.

Go to Failover Cluster Manager – Nodes

Right click on Nodes and Select “Add Nodes”

Add node

Add node

 

 

Click next on the “Before you begin” page after reading :D.

On the “Select Server” page, Key In the server name and click on Add. Ideally, adding the node shouldn’t take more than ten seconds.

Adding the new node

Adding the new node

 

On the “Validation Warning” Screen, We need to choose the right option specific to running validation test. As I mentioned earlier, I prefer to run all test with a spare disk form storage which don’t have any production data. So select Yes.

Validation Warning

Validation Warning

 

Clicking Next will Switch on the Validation Configuration Wizard

The account used for running validation wizard should be a local administrator on all nodes.

 

Validation Wizard - Before Begin

Validation Wizard – Before Begin

The next screen is “Testing Options”. I prefer to run all tests. If you are re-running after identifying some issues, Its Ok to choose only that failed module by select “Run only tests I select”.

 

Testing Options

Testing Options

The next screen is important. We need to choose the right SAN disk which will be used for validation test. I already added a 1 TB disk and made it as a CSV disk – Name CSV-4. Chose the right disk and click on next.

Select the Storage which will be used for Validation test

Select the Storage which will be used for Validation test

 

Please read the warning at the bottom –  TO AVOID ROLE FAILURES, IT IS RECOMMENDED THAT ALL ROLES USING CLUSTER SHARED VOLUMES BE STOPPED BEFORE THE STORAGE IS VALIDATED.

In the Confirmation page, Have a look on the specific tests which are going to perform on each modules and click on NEXT

ValidationWizard-Confirmation

 

Click Next to start the Validation tests.

Validation Test in Progress

Validation Test in Progress

 

Once done, You can review the result by clicking “View Report”.

If any of the test got failed, the Wizard assumes that something is not correct, hence shouldn’t proceed with the next step with the blessing of wizard. However, Once you click on finish, You will go back to “Add Node Wizard”  where you will get Option to Run Validation test again or proceed without Validation test.

 

Validation Wizard - Result Summary

Validation Wizard – Result Summary

In my case, I am using NICs with trunking for HyperV Data traffic,  So while creating the Virtual switch – I haven’t assigned any IP to this interface which the validation wizard claims that its not good. As I am sure that this error is not going to make an issue, I am proceeding with the next Option.

 

ValidationWarning-ProceedwithoutChecking

In the next screen, Review the node to be added and click on NEXT.

Wait for the successful confirmation and that’s it.

Cheers

Shaba

 

Windows Server 2012 R2 – NIC Teaming options

With Windows Server 2012, The new age of Windows teaming evolved. Very soon it was widely adapted – especially teaming along with Hyper-V. The key advantage which I see on using Windows Server teaming is the ease of configuration and the different teaming modes. And Switch Independent along with Hyper-V port was the recommended teaming option for Hyper-V.

All went well. Now we got Windows Server 2012 R2. As usual, I was playing around with Windows Server 2012 R2 and Hyper-V in the last few weeks. And the servers which I created was having Windows teaming – Switch Independent – HyperV Port. But later, I noticed one new load balancing algorithm named “Dynamic”. I couldn’t see much details around apart from few MVPs had touched on the overview. Off late, I came across Microsoft guide. Just a snip from this guide.

 

 Algorithms for load distribution

Outbound traffic can be distributed among the available links in many ways. One rule that guides any distribution algorithm is to try to keep all packets associated with a single flow (TCP-stream) on a single network adapter. This rule minimizes performance degradation caused by reassembling out-of-order TCP segments.

NIC teaming in Windows Server 2012 R2 supports the following traffic load distribution algorithms:

  • Hyper-V switch port. Since VMs have independent MAC addresses, the VM’s MAC address or the port it’s connected to on the Hyper-V switch can be the basis for dividing traffic. There is an advantage in using this scheme in virtualization. Because the adjacent switch always sees a particular MAC address on one and only one connected port, the switch will distribute the ingress load (the traffic from the switch to the host) on multiple links based on the destination MAC (VM MAC) address. This is particularly useful when Virtual Machine Queues (VMQs) are used as a queue can be placed on the specific NIC where the traffic is expected to arrive. However, if the host has only a few VMs, this mode may not be granular enough to get a well-balanced distribution. This mode will also always limit a single VM (i.e., the traffic from a single switch port) to the bandwidth available on a single interface. Windows Server 2012 R2 uses the Hyper-V Switch Port as the identifier rather than the source MAC address as, in some instances, a VM may be using more than one MAC address on a switch port.
  • Address Hashing. This algorithm creates a hash based on address components of the packet and then assigns packets that have that hash value to one of the available adapters. Usually this mechanism alone is sufficient to create a reasonable balance across the available adapters.

The components that can be specified, using PowerShell, as inputs to the hashing function include the following:

  •   Source and destination TCP ports and source and destination IP addresses (this is used by the user interface when “Address Hash” is selected)
  • Source and destination IP addresses only
  • Source and destination MAC addresses only

The TCP ports hash creates the most granular distribution of traffic streams resulting in smaller streams that can be independently moved between members. However, it cannot be used for traffic that is not TCP or UDP-based or where the TCP and UDP ports are hidden from the stack, such as IPsec-protected traffic. In these cases, the hash automatically falls back to the IP address hash or, if the traffic is not IP traffic, to the MAC address hash.

  • Dynamic. This algorithm takes the best aspects of each of the other two modes and combines them into a single mode.
    • Outbound loads are distributed based on a hash of the TCP Ports and IP addresses.  Dynamic mode also rebalances loads in real time so that a given outbound flow may move back and forth between team members.
    • Inbound loads are distributed as though the Hyper-V port mode was in use.   See Section 3.4 for more details.

The outbound loads in this mode are dynamically balanced based on the concept of flowlets.  Just as human speech has natural breaks at the ends of words and sentences, TCP flows (TCP communication streams) also have naturally occurring breaks.  The portion of a TCP flow between two such breaks is referred to as a flowlet.  When the dynamic mode algorithm detects that a flowlet boundary has been encountered, i.e., a break of sufficient length has occurred in the TCP flow, the algorithm will opportunistically rebalance the flow to another team member if appropriate.  The algorithm may also periodically rebalance flows that do not contain any flowlets if circumstances require it.    As a result the affinity between TCP flow and team member can change at any time as the dynamic balancing algorithm works to balance the workload of the team members.

In short, here is the best combination to be used in a real world scenario.

Switch Independent configuration / Address Hash distribution

This mode is best used for:

  • Active/Standby mode teams with just 2 team members; and
  • Teaming in a VM.

Switch Independent configuration / Hyper-V Port distribution

Will be best for Windows Server 2012 HyperV servers

Switch Independent configuration / Hyper-V Port distribution

This mode is best used for teaming in both native and Hyper-V environments- Obviously Windows Server 2012 R2 HyperV Infrastructure.

Exceptions

  • Teaming is being performed in a VM,
  • Switch dependent teaming (e.g., LACP) is required by policy, or
  • Operation of a two-member Active/Standby team is required by policy.

 

For anyone who is on HyperV implementation with Window Server 202 R2, Its worth reading this entire guide.

All credits to the author of this guide.