dbus daemon fails with undefined symbol: XML_SetHashSalt-CFANZ编程社区

Environment

Red Hat Enterprise Linux (RHEL) 7 and later

Issue

dbus is timing out and processes are failing to connect

Raw

Jan 1 01:00:00 hostname NetworkManager[1534]: <warn>  [546.45] ifcfg-rh: dbus: couldn't initialize system bus: Could not connect: Connection refused

dbus is showing symbol lookup errors

Raw

Traceback (most recent call last):
File "/usr/bin/firewall-cmd", line 31, in <module>
from firewall.client import FirewallClient, FirewallClientIPSetSettings, \
File "/usr/lib/python3.6/site-packages/firewall/client.py", line 28, in <module>
import dbus.mainloop.glib
File "/usr/lib64/python3.6/site-packages/dbus/__init__.py", line 101, in <module>
from dbus._dbus import Bus, SystemBus, SessionBus, StarterBus
File "/usr/lib64/python3.6/site-packages/dbus/_dbus.py", line 39, in <module>
from dbus.bus import BusConnection
File "/usr/lib64/python3.6/site-packages/dbus/bus.py", line 39, in <module>
from dbus.connection import Connection
File "/usr/lib64/python3.6/site-packages/dbus/connection.py", line 37, in <module>
from dbus.proxies import ProxyObject
File "/usr/lib64/python3.6/site-packages/dbus/proxies.py", line 34, in <module>
from dbus._expat_introspect_parser import process_introspection_data
File "/usr/lib64/python3.6/site-packages/dbus/_expat_introspect_parser.py", line 26, in <module>
from xml.parsers.expat import ParserCreate
File "/usr/lib64/python3.6/xml/parsers/expat.py", line 4, in <module>
from pyexpat import *
ImportError: /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so: undefined symbol: XML_SetHashSalt

Jan 1 01:00:00 hostname dbus-daemon: /usr/bin/dbus-daemon: symbol lookup error: /usr/bin/dbus-daemon: undefined symbol: XML_SetHashSalt

Resolution

The issue is due to dbus linking against an incompatible library named libexpat.so.1 and taking precedence over the library shipped by Red Hat in /usr/lib64.
To fix this issue, there are 2 alternatives.

Solution 1: Remove any path to 3rd party libraries from `/etc/ld.so.conf.d/`

Boot the system with the RHEL DVD into Rescue environment

Once booted, enter the chroot:

Raw
```
# chroot /mnt/sysimage
```

Remove the paths to 3rd party libraries from /etc/ld.so.conf.d

In the example below (and rest of the document), a file named /etc/ld.so.conf.d/oracle.conf contains a path to 3rd party libraries, including libexpat.so.1

Raw

# cat /etc/ld.so.conf.d/oracle.conf
/soft/oracle/product/client/12.1/lib

# ls -1 /soft/oracle/product/client/12.1/lib/libexpat.so*
/soft/oracle/product/client/12.1/lib/libexpat.so
/soft/oracle/product/client/12.1/lib/libexpat.so.1
/soft/oracle/product/client/12.1/lib/libexpat.so.1.5.2

# ls /usr/lib64/libexpat.so*
/usr/lib64/libexpat.so.1
/usr/lib64/libexpat.so.1.6.0

In the example above, the incompatible library libexpat.so.1.5.2 takes precedence over system library libexpat.so.1.6.0.

Raw

# rm /etc/ld.so.conf.d/oracle.conf

Rebuild the ld cache

Raw
```
# ldconfig
```

Verify that dbus now links against the expected library in /usr/lib64

Raw

# ldd /usr/bin/dbus-daemon | grep libexpat
    libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f5000c0c000)

Reboot
Update the 3rd party software scripts to set LD_LIBRARY_PATH if needed

If your 3rd party application doesn't start anymore, you may need to fix its startup script.

If you know which scripts are used to start the 3rd party application, edit the scripts to add the following statement so that the 3rd party libraries take precedence:

Raw
```
export LD_LIBRARY_PATH="/soft/oracle/product/client/12.1/lib"
```
If your 3rd party software vendor didn't explain how to use the LD_LIBRARY_PATH alternative to start their application, contact the vendor.

Solution 2: Protect system services from 3rd party libraries

Raw

This solution is less intrusive but may have some impact on custom or 3rd party services.

Boot the system with the RHEL DVD into Rescue environment

Once booted, enter the chroot:

Raw
```
# chroot /mnt/sysimage
```

Tell systemd to always start services with /usr/lib64 taking precedence

Create a drop-in overriding DefaultEnvironment variable:

Raw

# mkdir -p /etc/systemd/system.conf.d
# cat > /etc/systemd/system.conf.d/kcs-3468321.conf << EOF
[Manager]
# Make sure /usr/lib64 takes precedence over paths specified in /etc/ld.so.conf.d
DefaultEnvironment="LD_LIBRARY_PATH=/usr/lib64"
EOF

Reboot
Update the 3rd party software scripts to set LD_LIBRARY_PATH if needed

If your 3rd party application doesn't start anymore, you may need to fix its startup script or startup service.
- If your 3rd party application uses a systemd service unit, create the following drop-in:
  
  Raw
```
# systemctl edit <your service>.service
... Editor opens, usually vim ...
[Unit]
Environment=/soft/oracle/product/client/12.1/lib
... Save and close the editor ...
```
- Otherwise, if you know which scripts are used to start the 3rd party application, edit the scripts to add the following statement so that the 3rd party libraries take precedence:
  
  Raw
```
export LD_LIBRARY_PATH="/soft/oracle/product/client/12.1/lib"
```
  If your 3rd party software vendor didn't explain how to use the LD_LIBRARY_PATH alternative to start their application, contact the vendor.

Root Cause

A path to 3rd party libraries has been specified in /etc/ld.so.conf.d, causing some incompatible library (usually libexpat.so.1) to take precedence over the corresponding RHEL library, which then prevents dbus from running properly, leading to issues with services requiring dbus

Diagnostic Steps

Run the following to confirm third party libraries are used instead of system ones

Raw
```
# ldd /usr/bin/dbus-daemon
```

This should output with something like the following(with a third party library replacing the libexpat.so.1):

Raw

# ldd /usr/bin/dbus-daemon 
    linux-vdso.so.1 =>  (0x00007ffd32564000)
    libdbus-1.so.3 => /lib64/libdbus-1.so.3 (0x00007f803a177000)
    libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007f8039f46000)
    libexpat.so.1 => /opt/oracle/database/11.2.0.3/lib/libexpat.so.1 (0x00007f3cc5f33000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f8039af5000)
    libaudit.so.1 => /lib64/libaudit.so.1 (0x00007f80398cc000)
    libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007f80396c6000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f80394aa000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f80390dd000)
    libcap.so.2 => /lib64/libcap.so.2 (0x00007f8038ed8000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f8038bd6000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f80389ce000)
    liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f80387a8000)
    liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f8038593000)
    libgcrypt.so.11 => /lib64/libgcrypt.so.11 (0x00007f8038312000)
    libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f803810d000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f8037ef4000)
    libdw.so.1 => /lib64/libdw.so.1 (0x00007f8037cad000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f8037aa9000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f8037893000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f803a5fc000)
    libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f8037631000)
    libattr.so.1 => /lib64/libattr.so.1 (0x00007f803742c000)
    libelf.so.1 => /lib64/libelf.so.1 (0x00007f8037214000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f8036ffe000)
    libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f8036dee000)