VMware VirtualCenter Server Service not starting

I've been fighting with this issue for a while now and haven't been able to find a solution - even with checking the posts on this forum.

I am running VMware vCenter Server 5.1.0.32743 on a Windows 2008R2 physical server (not on a domain). Whenever I make a hardware change (ie, adding a SCSI card) the VirtualCenter Server service (and also the VirtualCenter Management Webservices service) refuses to start.

When I remove the SCSI card the issue still persists and the only way for me to resolve so far is to restore the server from backup - obviously not ideal and I need to connect tape drives to this server.

I get the standard event 1000 in event viewer also.

I have tried changing the service startup types so that SSO service comes up first (as suggested in one of the KB articles) but this has no effect.

The back end of the vpxd-xx.log file shows this:

 

2012-11-22T12:35:51.780Z [00564 info 'vpxdvpxdMoOptionManager'] [OptionManagerMo] Invoking callbacks for key log.level, pre commit

2012-11-22T12:35:51.780Z [00564 info 'vpxdvpxdMoOptionManager'] [OptionManagerMo] Invoking callbacks for key log.level, pre commit

2012-11-22T12:35:51.780Z [00564 info 'vpxdvpxdMoOptionManager'] [OptionManagerMo] No change to log.level

2012-11-22T12:35:51.780Z [00564 info 'vpxdvpxdMain'] [VpxdMain] Setting OpenSSL verify locations CAFile=C:\ProgramData\VMware\SSL\ca_certificates.crt CAPath=C:\ProgramData\VMware\SSL

2012-11-22T12:35:51.780Z [00564 info 'Default'] Creating SSL Contexts

2012-11-22T12:35:51.858Z [00564 info 'utilvpxdDbParallelLoader'] Num rows: 6, num CPUs: 4, threads: 1, step: 6

2012-11-22T12:35:51.921Z [00564 info 'Default'] [VpxdResourcePoolHostMirror::SetRoot] root moref vim.ResourcePool:resgroup-8 in mirror for host vim.HostSystem:host-10

2012-11-22T12:35:51.921Z [00564 info 'Default'] [VpxdResourcePoolHostMirror::SetRoot] root moref vim.ResourcePool:resgroup-8 in mirror for host vim.HostSystem:host-13

2012-11-22T12:35:51.921Z [00564 info 'Default'] [VpxdResourcePoolHostMirror::SetRoot] root moref vim.ResourcePool:resgroup-53 in mirror for host vim.HostSystem:host-54

2012-11-22T12:35:53.028Z [00564 warning 'VpxProfiler'] Vpxd::ServerApp::Init [VpxdInvtHost::Init2(gDB)] took 1108 ms

2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::SetDesiredDasProtectState] desired protection state for VM vm-241 'unprotected' -> 'protected'

2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::UpdateActualDasProtectStateLocked] actual protection state for VM vm-241 'n/a' -> 'protected'

2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::SetDesiredDasProtectState] desired protection state for VM vm-61 'unprotected' -> 'protected'

2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::UpdateActualDasProtectStateLocked] actual protection state for VM vm-61 'n/a' -> 'protected'

2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::SetDesiredDasProtectState] desired protection state for VM vm-59 'unprotected' -> 'protected'

2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::UpdateActualDasProtectStateLocked] actual protection state for VM vm-59 'n/a' -> 'protected'

2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::SetDesiredDasProtectState] desired protection state for VM vm-60 'unprotected' -> 'protected'

2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::UpdateActualDasProtectStateLocked] actual protection state for VM vm-60 'n/a' -> 'protected'

2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::SetDesiredDasProtectState] desired protection state for VM vm-57 'unprotected' -> 'protected'

2012-11-22T12:35:53.059Z [00564 info 'vmdasVm'] [VmMo::UpdateActualDasProtectStateLocked] actual protection state for VM vm-57 'n/a' -> 'protected'

2012-11-22T12:35:53.106Z [00564 info 'dbdbPortgroup'] [VpxdInvtDVPortGroup::PreLoadDvpgConfig] loaded [0] dvpg config objects

2012-11-22T12:35:53.106Z [00564 info 'authvpxdMoSessionManager'] [SSO][SessionManagerMo::Init] Admin URI set to: https://10.105.110.110:7444/sso-adminserver/sdk

2012-11-22T12:35:53.106Z [00564 info 'authvpxdMoSessionManager'] [SSO][SessionManagerMo::Init] Downloading STS Root certificates ...

2012-11-22T12:35:53.122Z [00496 info 'Default'] Thread attached

2012-11-22T12:35:53.153Z [00564 error 'vpxdvpxdMain'] [Vpxd::ServerApp::Init] Init failed: Unexpected exception

--> Backtrace:

--> backtrace[00] rip 000000018018977a

--> backtrace[01] rip 0000000180100c98

--> backtrace[02] rip 0000000180101fae

--> backtrace[03] rip 000000018008aeab

--> backtrace[04] rip 0000000000514971

--> backtrace[05] rip 00000000004b1298

--> backtrace[06] rip 00000000004b16c9

--> backtrace[07] rip 0000000000420fae

--> backtrace[08] rip 000000014040bfb8

--> backtrace[09] rip 000000013f300078

--> backtrace[10] rip 000000013f30016a

--> backtrace[11] rip 000000013f300279

--> backtrace[12] rip 000000013f300609

--> backtrace[13] rip 000000013f642903

--> backtrace[14] rip 000000013fdee4b9

--> backtrace[15] rip 000000013fde835c

--> backtrace[16] rip 0000000140008a3b

--> backtrace[17] rip 000007fefecda82d

--> backtrace[18] rip 000000007764652d

--> backtrace[19] rip 00000000779dc521

-->

2012-11-22T12:35:53.153Z [00564 warning 'VpxProfiler'] ServerApp::Init [TotalTime] took 4353 ms

2012-11-22T12:35:53.153Z [00564 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...

2012-11-22T12:35:53.153Z [00564 info 'vpxdvpxdSupportManager'] Wrote uptime information

2012-11-22T12:36:00.813Z [03108 warning 'VpxProfiler' opID=SWI-eac14804] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2012-11-22T12:36:12.825Z [03108 warning 'VpxProfiler' opID=SWI-63f9e37] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2012-11-22T12:36:24.837Z [03108 warning 'VpxProfiler' opID=SWI-b155aed1] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2012-11-22T12:36:36.849Z [03108 warning 'VpxProfiler' opID=SWI-7a9c1975] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2012-11-22T12:36:48.861Z [03108 warning 'VpxProfiler' opID=SWI-a2d6fb25] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms

2012-11-22T12:36:49.126Z [00564 info 'Default'] Forcing shutdown of VMware VirtualCenter now

 

Interestingly (and this is where I cannot find anything online relating to this) discover-is from C:\Program Files\VMware\Infrastructure\SSOServer\utils\logs shows:

 

2012-11-21 16:15:07,110 - DDNBGF5J,,,,Executing action: 'discover-is'

2012-11-21 16:15:07,111 - DDNBGF5J,,,,Discovering identity sources

2012-11-21 16:15:08,022 - DDNBGF5J,,,,ERROR: Could not access HTTP invoker remote service at [https://10.105.110.110:7444/ims/CommandServer]; nested exception is org.apache.commons.httpclient.HttpException: Did not receive successful HTTP response: status code = 404, status message = [Not Found]

org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [https://10.105.110.110:7444/ims/CommandServer]; nested exception is org.apache.commons.httpclient.HttpException: Did not receive successful HTTP response: status code = 404, status message = [Not Found]

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:212)

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:145)

     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

     at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

     at $Proxy0.executeCommand(Unknown Source)

     at com.rsa.command.CacheableHttpInvokerTarget.executeCommand(HttpInvokerCommandTarget.java:261)

     at com.rsa.command.DelegatingCommandTarget.executeCommand(DelegatingCommandTarget.java:66)

     at com.rsa.command.TargetableCommand.execute(TargetableCommand.java:270)

     at com.rsa.authn.LoginCommand.execute(LoginCommand.java:612)

     at com.rsa.authn.AuthenticatedTargetImpl.login(AuthenticatedTargetImpl.java:161)

     at com.rsa.command.ConnectionFactory$ConnectionImpl.connect(ConnectionFactory.java:754)

     at com.rsa.command.ConnectionFactory.connect(ConnectionFactory.java:542)

     at com.rsa.riat.tools.internal.Utils.login(Utils.java:1050)

     at com.rsa.riat.tools.is.DiscoverIdentitySources.execute(DiscoverIdentitySources.java:81)

     at com.rsa.riat.tools.ConfigureRIATCmd.execute(ConfigureRIATCmd.java:192)

     at com.rsa.riat.tools.ConfigureRIATCmd.main(ConfigureRIATCmd.java:517)

Caused by: org.apache.commons.httpclient.HttpException: Did not receive successful HTTP response: status code = 404, status message = [Not Found]

     at org.springframework.remoting.httpinvoker.CommonsHttpInvokerRequestExecutor.validateResponse(CommonsHttpInvokerRequestExecutor.java:214)

     at org.springframework.remoting.httpinvoker.CommonsHttpInvokerRequestExecutor.doExecuteRequest(CommonsHttpInvokerRequestExecutor.java:131)

     at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:136)

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:192)

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:174)

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:142)

     ... 14 more

2012-11-21 17:20:55,979 - DDNBGF5J,,,,Executing action: 'discover-is'

2012-11-21 17:20:55,979 - DDNBGF5J,,,,Discovering identity sources

2012-11-21 17:20:56,884 - DDNBGF5J,,,,ERROR: Could not access HTTP invoker remote service at [https://10.105.110.110:7444/ims/CommandServer]; nested exception is org.apache.commons.httpclient.HttpException: Did not receive successful HTTP response: status code = 404, status message = [Not Found]

org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [https://10.105.110.110:7444/ims/CommandServer]; nested exception is org.apache.commons.httpclient.HttpException: Did not receive successful HTTP response: status code = 404, status message = [Not Found]

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:212)

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:145)

     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

     at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

     at $Proxy0.executeCommand(Unknown Source)

     at com.rsa.command.CacheableHttpInvokerTarget.executeCommand(HttpInvokerCommandTarget.java:261)

     at com.rsa.command.DelegatingCommandTarget.executeCommand(DelegatingCommandTarget.java:66)

     at com.rsa.command.TargetableCommand.execute(TargetableCommand.java:270)

     at com.rsa.authn.LoginCommand.execute(LoginCommand.java:612)

     at com.rsa.authn.AuthenticatedTargetImpl.login(AuthenticatedTargetImpl.java:161)

     at com.rsa.command.ConnectionFactory$ConnectionImpl.connect(ConnectionFactory.java:754)

     at com.rsa.command.ConnectionFactory.connect(ConnectionFactory.java:542)

     at com.rsa.riat.tools.internal.Utils.login(Utils.java:1050)

     at com.rsa.riat.tools.is.DiscoverIdentitySources.execute(DiscoverIdentitySources.java:81)

     at com.rsa.riat.tools.ConfigureRIATCmd.execute(ConfigureRIATCmd.java:192)

     at com.rsa.riat.tools.ConfigureRIATCmd.main(ConfigureRIATCmd.java:517)

Caused by: org.apache.commons.httpclient.HttpException: Did not receive successful HTTP response: status code = 404, status message = [Not Found]

     at org.springframework.remoting.httpinvoker.CommonsHttpInvokerRequestExecutor.validateResponse(CommonsHttpInvokerRequestExecutor.java:214)

     at org.springframework.remoting.httpinvoker.CommonsHttpInvokerRequestExecutor.doExecuteRequest(CommonsHttpInvokerRequestExecutor.java:131)

     at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:136)

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:192)

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:174)

     at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:142)

     ... 14 more

 

If I try to go to that commandserver url in the log above in a browser, it also get a 404:

The only thing similar I can find online is here:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2036170

Very similar exception error but nothing similar in my discover-is.log file.

From that kb link above I have run the following command:

C:\Program  Files\VMware\Infrastructure\SSOServer\utils>ssocli.cmd configure-riat  -a discover-is -u admin -p <masterPassword>

And I get the following output which confirms the discover-is log:

Netstat shows that port 7444 is opened by Java and nothing else. The other potential problem port (443) is not in use by anything else that it shouldnt be.

I suspect this may be the cause of the problem but I have no idea where to go next. Any suggestions would be greatly appreiciated.

Many thanks,

Matt