Hello,
we have been using vCenter server (5.0 and 5.1) on Windows Server 2003 for a long time. Now, with vCenter Server 5.5 there is a need for Windows Server 2008 64-bit and accompanying database requirements. We decided to test vCenter linux appliance, and get rid of Microsoft stuff. So far we've tried vCenter Server 5.1 and 5.5 appliance on 5.0, 5.1 and 5.5 vSphere hypervisor with no luck. The problem is always the same: vpxd is interrupted because of segmentation fault, and this fault occurs in libvmomi.so. I've tried suggested network settings (correct IP, FQDN, and correct forward/backl resolving, etc.) and everything possible to be found on the internet.
The problem arises on fresh start, and running installation wizard:
Failed to execute '/usr/sbin/vpxd_servicecfg 'db' 'write' 'embedded' '' '' '' '' CENSORED':
/usr/sbin/vpxd_commonutils: line 910: 5898 Segmentation fault (core dumped) $VPXD "$@"
/usr/sbin/vpxd_commonutils: line 910: 5907 Segmentation fault (core dumped) $VPXD "$@"
/usr/sbin/vpxd_commonutils: line 910: 6276 Segmentation fault (core dumped) $VPXD "$@"
VC_DB_SCHEMA_VERSION=
VC_DB_SCHEMA_INITIALIZED=1
VC_CFG_RESULT=407(Error: Failed to initialize schema.)
This leads to segmentation fault, and in system log we can find:
vpxd[4422] general protection ip:7f3146a9d5df sp:7fff38f1bd88 error:0 in libvmomi.so
[7f31467b5000+41f000]
This clearly states problem in libvmomi.so, which is segfaulting somewhere. I did strace of vpxd:
futex(0x7f9247442f90, FUTEX_WAKE_PRIVATE, 2147483647) = 0
brk(0) = 0x7f925274f000
brk(0x7f9252770000) = 0x7f9252770000
futex(0x7f924e594018, FUTEX_WAKE_PRIVATE, 2147483647) = 0
open("/usr/lib/locale/locale-archive", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/locale.alias", O_RDONLY) = 3
fstat(3, {st_dev=makedev(8, 3), st_ino=339728, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=2512, st_atime=2013/10/17-12:08:59, st_mtime=2013/03/18-21:30:01, st_ctime=2013/09/07-01:07:16}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f924e7ae000
read(3, "# Locale name alias data base.\n#"..., 4096) = 2512
read(3, "", 4096) = 0
close(3) = 0
munmap(0x7f924e7ae000, 4096) = 0
open("/usr/lib/locale/en_US.UTF-8/LC_CTYPE", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_CTYPE", O_RDONLY) = 3
fstat(3, {st_dev=makedev(8, 3), st_ino=402782, st_mode=S_IFREG|0644, st_nlink=176, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=512, st_size=256324, st_atime=2013/10/17-12:09:34, st_mtime=2013/03/18-21:30:39, st_ctime=2013/09/07-01:07:16}) = 0
mmap(NULL, 256324, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f924e5f5000
close(3) = 0
open("/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 3
fstat(3, {st_dev=makedev(8, 3), st_ino=402784, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=56, st_size=26050, st_atime=2013/10/17-12:08:59, st_mtime=2013/09/07-01:07:28, st_ctime=2013/09/07-01:07:28}) = 0
mmap(NULL, 26050, PROT_READ, MAP_SHARED, 3, 0) = 0x7f924e5ee000
close(3) = 0
futex(0x7f9246897f80, FUTEX_WAKE_PRIVATE, 2147483647) = 0
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9244b83000
brk(0x7f9252793000) = 0x7f9252793000
brk(0x7f92527b4000) = 0x7f92527b4000
brk(0x7f92527d5000) = 0x7f92527d5000
brk(0x7f92527f6000) = 0x7f92527f6000
brk(0x7f9252817000) = 0x7f9252817000
brk(0x7f9252838000) = 0x7f9252838000
brk(0x7f9252859000) = 0x7f9252859000
brk(0x7f925287a000) = 0x7f925287a000
brk(0x7f925289c000) = 0x7f925289c000
brk(0x7f92528bd000) = 0x7f92528bd000
brk(0x7f92528de000) = 0x7f92528de000
brk(0x7f92528ff000) = 0x7f92528ff000
brk(0x7f9252920000) = 0x7f9252920000
brk(0x7f9252941000) = 0x7f9252941000
brk(0x7f9252962000) = 0x7f9252962000
brk(0x7f9252986000) = 0x7f9252986000
brk(0x7f92529a8000) = 0x7f92529a8000
brk(0x7f92529c9000) = 0x7f92529c9000
brk(0x7f92529ea000) = 0x7f92529ea000
brk(0x7f9252a0b000) = 0x7f9252a0b000
brk(0x7f9252a2c000) = 0x7f9252a2c000
brk(0x7f9252a4d000) = 0x7f9252a4d000
brk(0x7f9252a6e000) = 0x7f9252a6e000
brk(0x7f9252a8f000) = 0x7f9252a8f000
brk(0x7f9252ab0000) = 0x7f9252ab0000
brk(0x7f9252ad1000) = 0x7f9252ad1000
brk(0x7f9252af2000) = 0x7f9252af2000
brk(0x7f9252b13000) = 0x7f9252b13000
brk(0x7f9252b34000) = 0x7f9252b34000
brk(0x7f9252b55000) = 0x7f9252b55000
brk(0x7f9252b76000) = 0x7f9252b76000
brk(0x7f9252b97000) = 0x7f9252b97000
brk(0x7f9252bb8000) = 0x7f9252bb8000
brk(0x7f9252bd9000) = 0x7f9252bd9000
brk(0x7f9252bfa000) = 0x7f9252bfa000
brk(0x7f9252c1b000) = 0x7f9252c1b000
brk(0x7f9252c3c000) = 0x7f9252c3c000
brk(0x7f9252c5d000) = 0x7f9252c5d000
brk(0x7f9252c7e000) = 0x7f9252c7e000
brk(0x7f9252c9f000) = 0x7f9252c9f000
brk(0x7f9252cc0000) = 0x7f9252cc0000
brk(0x7f9252ce1000) = 0x7f9252ce1000
brk(0x7f9252d02000) = 0x7f9252d02000
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV (core dumped) +++
And also did GDB and backtrace of vpxd:
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff27a25df in Vmomi::DynamicData::DynamicData() () from /usr/lib/vmware-vpx/libvmomi.so
(gdb) backtrace
#0 0x00007ffff27a25df in Vmomi::DynamicData::DynamicData() () from /usr/lib/vmware-vpx/libvmomi.so
#1 0x00007ffff5889e92 in Vim::HistoricalInterval::HistoricalInterval() () from /usr/lib/vmware-vpx/libtypes.so
#2 0x00005555574f2834 in ?? ()
#3 0x000055555833a3d6 in ?? ()
#4 0x00007ffff7dc95d8 in ?? () from /usr/lib/vmware-vpx/libtypes.so
#5 0x000055555833a320 in ?? ()
#6 0x0000000000000000 in ?? ()
(gdb)
So at the end, there is some problem in Vmomi::DynamicData::DynamicData() in libvmomi.so
Is there any resolution for this error in vCenter server appliance ? Any other operating system/appliance/anything runs on host machine for months without a problem. I did hardware checks of server hardware and tried everything I could to diagnose the problem. One thing never works on any host I have: vpxd, it segfaults on libvmomi.so
Thank you for any possible hint.
Best regards,
Pavel Komurka