We use Citrix Presentation Server for a number of applications, and lately we have had a significant increase in issues with one set of our Citrix servers. We have 3 main sets of Citrix servers and the problems have only been happening on one set.
One of the sets doesn’t have this error, but wouldn’t because they are physical servers. They have been in production a long time, and we have plans to virtualize them.
The second set doesn’t get the errors, but it is fewer servers and fewer users.
The third set:
- is virtual
- runs on 2008 R2 Hyper-V
- has more servers (6 as opposed to 4 or 5 for the other two)
- supports more users and more users per server (averages around 20 users per server during business hours)
Around November, we started upgrading our hosts from 2008 to 2008 R2. The problems have been getting progressively worse peaking in the last 2 months. Our last 2008 host was converted in March.
After some event log review, we were able to correlate some of the issues to the following error in the event log:
Event Type: Warning
Event Source: netvsc
Event Category: None
Event ID: 5
Date: 4/19/2010
Time: 3:49:53 PM
User: N/A
Computer: <ServerNameChangedToProtectTheGuilty>
Description:
The miniport ‘Microsoft Virtual Machine Bus Network Adapter #4’ hung.For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 00 00 02 00 52 00 ……R.
0008: 00 00 00 00 05 00 00 80 …….
0010: 00 00 00 00 00 00 00 00 ……..
0018: 00 00 00 00 00 00 00 00 ……..
0020: 00 00 00 00 00 00 00 00 ……..
and right behind that would be this message:
Event Type: Information
Event Source: netvsc
Event Category: None
Event ID: 4
Date: 4/19/2010
Time: 3:49:53 PM
User: N/A
Computer: <ServerNameChangedToProtectTheGuilty>
Description:
The miniport ‘Microsoft Virtual Machine Bus Network Adapter #4’ reset.For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 00 00 00 00 02 00 52 00 ……R.
0008: 00 00 00 00 04 00 00 40 …….@
0010: 00 00 00 00 00 00 00 00 ……..
0018: 00 00 00 00 00 00 00 00 ……..
0020: 00 00 00 00 00 00 00 00 ……..
After doing a bit of searching and getting a lot of nothing, and doing some on site troubleshooting without much luck, I finally broke down and called Microsoft. I spent a day e-mailing back and forth with someone who was suggesting that I try all the things that I had already tried, so I contacted our TAM and had the case escalated.
The technician then informed me that there was an internal hotfix that had not been fully tested yet, that related to my issue. It seems that in 2008 R2 Hyper-V guests running Server 2003, the network adapter will hang and then reset under heavy load. The hotfix has to be applied to the host and then the integration services on the guest have to be updated. In my environment, when I updated, I had to remove the integration services from the guest before the updated NIC driver would install. I reported this behavior to the technician I was working with, but he said that he couldn’t reproduce that particular problem and that he had no issues updating his test environment.
It is my understanding that the hotfix will be released under KB981836. When you install this, it changes the integration services version from 6.1.7600.16385 to 6.1.7600.20683. You can see this if you look at the driver version on the guest NIC.
We are having this issue too on a Dell R710 server running 2008 R2 with 2003 R2 VMs.. any resolution yet? Did MS say when this hotfix might be available, or is there some way I can get it from you without having to spend hours on the phone with them too? Thanks!
The hotfix seems to have fixed the problem for us. There are some related issues that seem to be fixed by changing the C-state in the BIOS and/or configuring the host for “High Performance”. E-mail me if you have questions about that.
We have just started to get this issue also on a Dell R710 server running 2008 R2 with 2003 R2 VMs.
I have downloaded the hotfix and when running it a message box pops up with the line “This hotfix is not applicable to your version”
I am running 2008 R2 Ent.
Are you applying the hotfix to the host or the guest? (Just verifying) The hotfix applies to the Host, and then you have to update integration components on the guest.
I had those problems, too. But not with a 2003 guest system. I had problems with a 2008 R2 guest system.
I installed the hotfix on my Windows 7 and Windows Server 2008 R2 guest system. Previously I suddenly encountered problems with the RRAS (I could not establish a PPPoE connection, Remote Desktop sometimes does not work, loosing RDP connection when trying to establish PPPoE connection, and so on). On the Windows 7 guest client I encountered bluescreens which pointed to the network driver. Looks like they are gone, too.
After installing the hotfix on the server I could see a different NIC driver version and I don’t see any log entries anymore.
Were you running x64 or x86 of the hyper-v? My Clients Dell R710 is running Win Server Enterprise 2008 R2 and I’m getting the same error on a 2003 host plus I cannot install the hotfix. Tried applying to the host. The guest is running 2003 so cannot run the hostfix as its an .msu file
Matt
I am running x64. The way you apply the hotfix, is to run the update on the host, reboot and then update the Integration Services on the guest. After you do that, it is possible that you will still need to manually update the device drive for the NIC in the VM. If you go and look and the device driver version is not 6.1.7600.20683 or higher, then the update has not been applied. Hope this helps.
Michael
From Matt:
Looks like I’ve found the answer – there is a x64 version
http://support.microsoft.com/kb/2223005
Thanks
Matt
What a mess. We too discovered this problem when we P2Vd 6 Citrix servers with 2003.
This problem continues to this day! With no less than 6 hotfixes installed, we still see netvsc adapter reset issues. Now it happens a lot on 2008 R2 guests. Some in the Microsoft forums are saying it continues even after SP1 is installed.
We’ve reduced the frequency of the errors but suffice it to say this has made my department regret going to Hyper V.
@Jeff – I actually have pretty good luck with the machines that we run on Hyper-V. I do think there are some issues if you don’t keep the integration components updated on your guests.