Welcome back, it’s about time to dive into my least favorite subject - System Center Virtual Machine Manager. For the same reason as everyone else, I’ve been asked to do some work in preparation for getting rid of our VMware environment. Part of that effort is installing an up-to-date SCVMM instance. I’m gonna stipulate up front that I am not a fan of SCVMM. I am also aware that I don’t have any better choice at the moment - we’re not installing Nutanix, we’re not migrating to Proxmox, and we’re not doing any other option that you could probably name.
Given these contstraints, the only real thing we can do is get good. To that end, I intend this page to contain information not contained in the docs that should be - it’s either not there, or glossed over. Some of these changes are things that by all rights should be done automatically either by the setup routine or when the agent installs on a hyper-v host. Many of these things were things that required a support case or an incredibly annoying amount of digging to get right.
Patch fails to install
The docs do an OK job of getting you through the initial install, but I found very inconsistent patching behavior.
- Home lab: worked perfectly by doing nothing special
- Work: failed miserably with no useful feedback or errors quietly during patching, leaving a 1603 error in the logs
I went through an exhaustive troubleshooting process only to come to the conclusion that the patch process was running a custom action calling a dll to run a function that fails to read or change configuration on a service. None of that made sense but given that at this point it’s narrowed down to “Microsoft code behaving badly” and not normal permission issues or anything like that I had to throw up my hands and open a support case
As near as I can tell, what fixed it was opening an elevated command/powershell prompt and just.. starting the patch from there by typing its name. It doesn’t make any sense at all to me as just opening it normally by double clicking on the patch file asked for elevation - but what I gathered from the conversation is that sometimes this just happens. It must be cheaper to answer support cases about it than it is to spend the dev time troubleshooting it.
Starting a scan for patches fails
After installing a WSUS server and configuring it into the SCVMM fabric, I wanted to try out the process of scanning and patching a windows cluster. Naturally, this basic task did not go well.
Starting a compliance scan failed with the dreaded HRESULT 0x80338111 - Unknown error. Very helpful. Time to debug!
Microsoft does have excellent documentation on how to get a useful set of debug logs.
After following the steps while reproducing the error, I ended up with a 435KB log file to parse through. To help narrow down this mountain of data down to something useful, I searched the file for the command that actually runs, visible from the Jobs panel - Start-SCComplianceScan. Eventually I come across this message:
[0]0C98.15EC::2024/08/23-12:09:19.124222600 [Microsoft-VirtualMachineManager-Debug]{“Component”:4,“TraceFlag”:4,“FileName”:“WsmanAPIWrapper.cs”,“FileLine”:3748,“Message”:“Exception [System.Runtime.InteropServices.COMException (0x80338111): The WinRM client sent a request to the remote WS-Management service and was notified that the request size exceeded the configured MaxEnvelopeSize quota. \r\n at WSManAutomation.IWSManSession.Invoke(String actionUri, Object resourceUri, String parameters, Int32 flags)\r\n at Microsoft.Carmine.WSManWrappers.MyIWSManSession.Invoke(String actionUri, Object resourceUri, String parameters, Int32 flags)\r\n at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.Invoke(String actionUri, WSManUri targetUri, Hashtable parameters, Type returnType, Boolean isCarmineMethod, Boolean forceResponseCast)] while retrieving underlying WMI error to throw. Got string "<f:WSManFault xmlns:f="http://schemas.microsoft.com/wbem/wsman/1/wsmanfault" Code="2150859025" Machine="scvmap.ad.lab.internal"><f:Message>The WinRM client sent a request to the remote WS-Management service and was notified that the request size exceeded the configured MaxEnvelopeSize quota. </f:Message></f:WSManFault>"”,“TaskID”:"{00000000-0000-0000-0000-000000000000}",“TaskName”:"",“meta”:{“provider”:“Microsoft-VirtualMachineManager-Debug”,“event”:“2v0”,“time”:“2024-08-23T19:09:19.1242226Z”,“cpu”:0,“pid”:3224,“tid”:5612,“channel”:“Microsoft-VirtualMachineManager-Debug/Debug”,“level”:“Verbose “,“keywords”:“0x8000000000000000”,“activity”:“0CE58704-8D14-43FC-AF0E-EC39F8D94D10”}}
Clear as mud right? But the answer is in there. We know SCVMM is attempting a WSMAN operation and here’s a csharp file that looks related to that "FileName":"WsmanAPIWrapper.cs". And here is the actual problem: The WinRM client sent a request to the remote WS-Management service and was notified that the request size exceeded the configured MaxEnvelopeSize quota..
Well that’s something actionable, because when we google MaxEnvelopeSize we see other similar questions. It looks like the default value for Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WSMAN\Client -> maxEnvelopeSize is set to 0x800 / 2048 by default on the hypervisors, but on the SCVMM application server something had changed this default.
If we increase this value to 0x2000 / 8192 and restart the windows remote management service on the target machine, the command to kick off a scan succeeds.
I’m sure there will be more things as I get used to the product, I’ll update this page as I find them.