Ldd difference between RHEL 7 and 8 for mongo 3.6.17 above

Ops and Admin
1

Hi Team,

We are seeing few dynamic link libraries are missing in 3.6.17 for CentOS 8. if i install the same on CentOS7 its shows those additional libs.

3.6.17 mongod ldd output from CentOS7

ldd /usr/bin/mongo
linux-vdso.so.1 => (0x00007fff9778f000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f75ad456000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f75acff3000)
libssl.so.10 => /lib64/libssl.so.10 (0x00007f75acd81000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f75acb7d000)
librt.so.1 => /lib64/librt.so.1 (0x00007f75ac975000)
libm.so.6 => /lib64/libm.so.6 (0x00007f75ac673000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f75ac45d000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f75ac241000)
libc.so.6 => /lib64/libc.so.6 (0x00007f75abe73000)
/lib64/ld-linux-x86-64.so.2 (0x00007f75af230000)
libz.so.1 => /lib64/libz.so.1 (0x00007f75abc5d000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f75aba10000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f75ab727000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f75ab523000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f75ab2f0000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f75ab0e0000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f75aaedc000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f75aacb5000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f75aaa53000)

In CentOS8 the same mongo 3.6.17 shows only less libs

ldd /usr/bin/mongod
linux-vdso.so.1 (0x00007ffe834c4000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f2777452000)
libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f2776f6c000)
libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f2776cd8000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f2776ad4000)
librt.so.1 => /lib64/librt.so.1 (0x00007f27768cc000)
libm.so.6 => /lib64/libm.so.6 (0x00007f277654a000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f2776332000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f2776112000)
libc.so.6 => /lib64/libc.so.6 (0x00007f2775d4f000)
/lib64/ld-linux-x86-64.so.2 (0x00007f277a7a9000)
libz.so.1 => /lib64/libz.so.1 (0x00007f2775b38000)

Any idea why this difference is.? We have been debugging a mongo 3.6.17 version issue in newer CentOS8 and suspecting if this could cause an issue.

2

Hi @venkataraman_r,

You are comparing the ldd output of the mongo shell versus the mongod server binary, so the linked libraries are not expected to be identical. The linked library versions for mongod may also vary between O/S releases.

What problem are you trying to solve?

If you want to compare server build options, I suggest comparing the output of:

mongo --norc --eval “db.serverBuildInfo()”

Regards,
Stennie

3

Thank you @Stennie_X for pointing out. But when i check the same for mongod also the same difference i see.

db.serverBuildInfo() is helpful but i dont see any difference between 3.6.17 CentOS7 and 8.

We have been hitting mongo SECONDARY members getting into hung state and opened a JIRA (https://jira.mongodb.org/browse/SERVER-54805). But we are trying to understand why its happening only with CentOS8. CentOS7 with the same mongo version 3.6.17 is working great. So trying to analyze from all the angle seeing a ldd difference. So thinking if that could cause any issues.

4

Hi @venkataraman_r,

Based on the info in SERVER-54086, you are comparing MongoDB Community server on CentOS 7 with MongoDB Enterprise server on CentOS 7. MongoDB Enterprise uses additional libraries including GSSAPI and Kerberos.

This difference does not seem relevant to the problem you have described unless there are some other differences in configuration (for example, security & authentication) that might affect connections. However, you could install the same edition & release version of MongoDB server into the newer environment to remove that variation.

As noted in the Jira discussion, MongoDB 3.6 will reach End of Life (EOL) next month and we recommend you upgrade to MongoDB 4.0 to see if the issue is still reproducible. Once a server release series reaches end of life, no further bug fixes or security updates will be created.

Regards,
Stennie

5

No @Stennie_X, i dont have Enterprise version at all in my servers. Its the same community version.

6

Hi @venkataraman_r,

Can you provide the output of mongo --norc --eval “db.serverBuildInfo()” for both servers? Did you install official packages from MongoDB or build from source?

I’m curious to see if there is some difference in build options.

Thanks,
Stennie

8

CentOS 8 3.6.17

[root@abc ~]# mongo XYZ:27717 --norc --eval “db.serverBuildInfo()”
MongoDB shell version v3.6.17
connecting to: mongodb://XYZ:27717/test?gssapiServiceName=mongodb
Implicit session: session { “id” : UUID(“6431a77a-6845-4f3e-b70a-7b4481bcd41f”) }
MongoDB server version: 3.6.17
{
“version” : “3.6.17”,
“gitVersion” : “3d6953c361213c5bfab23e51ab274ce592edafe6”,
“modules” : ,
“allocator” : “tcmalloc”,
“javascriptEngine” : “mozjs”,
“sysInfo” : “deprecated”,
“versionArray” : [
3,
6,
17,
0
],
“openssl” : {
“running” : “OpenSSL 1.1.1g FIPS 21 Apr 2020”,
“compiled” : “OpenSSL 1.1.1 FIPS 11 Sep 2018”
},
“buildEnvironment” : {
“distmod” : “rhel80”,
“distarch” : “x86_64”,
“cc” : “/opt/mongodbtoolchain/v2/bin/gcc: gcc (GCC) 5.4.0”,
“ccflags” : “-fno-omit-frame-pointer -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -Werror -O2 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -Wno-unused-but-set-variable -Wno-missing-braces -fstack-protector-strong -fno-builtin-memcmp”,
“cxx” : “/opt/mongodbtoolchain/v2/bin/g++: g++ (GCC) 5.4.0”,
“cxxflags” : “-Woverloaded-virtual -Wno-maybe-uninitialized -std=c++14”,
“linkflags” : “-pthread -Wl,-z,now -rdynamic -Wl,–fatal-warnings -fstack-protector-strong -fuse-ld=gold -Wl,–build-id -Wl,–hash-style=gnu -Wl,-z,noexecstack -Wl,–warn-execstack -Wl,-z,relro”,
“target_arch” : “x86_64”,
“target_os” : “linux”
},
“bits” : 64,
“debug” : false,
“maxBsonObjectSize” : 16777216,
“storageEngines” : [
“devnull”,
“ephemeralForTest”,
“mmapv1”,
“wiredTiger”
],
“ok” : 1,
“operationTime” : Timestamp(1617065195, 2053),
“$clusterTime” : {
“clusterTime” : Timestamp(1617065195, 2053),
“signature” : {
“hash” : BinData(0,“AAAAAAAAAAAAAAAAAAAAAAAAAAA=”),
“keyId” : NumberLong(0)
}
}
}
[root@abc ~]# cat /etc/redhat-release
CentOS Linux release 8.1.1911 (Core)
[root@abc ~]# mongod --version
-bash: mongod: command not found
[root@abc ~]# ssh XYZ !!
ssh XYZ mongod --version
db version v3.6.17
git version: 3d6953c361213c5bfab23e51ab274ce592edafe6
OpenSSL version: OpenSSL 1.1.1g FIPS 21 Apr 2020
allocator: tcmalloc
modules: none
build environment:
distmod: rhel80
distarch: x86_64
target_arch: x86_64
[root@abc ~]#

CentOS 7 with 3.6.17

[root@abc ~]# mongo DEF:27717 --norc --eval “db.serverBuildInfo()”
MongoDB shell version v3.6.9
connecting to: mongodb://DEF:27717/test
Implicit session: session { “id” : UUID(“967d0f97-1419-44a6-8ca0-2d51ac367dc1”) }
MongoDB server version: 3.6.17
{
“version” : “3.6.17”,
“gitVersion” : “3d6953c361213c5bfab23e51ab274ce592edafe6”,
“modules” : ,
“allocator” : “tcmalloc”,
“javascriptEngine” : “mozjs”,
“sysInfo” : “deprecated”,
“versionArray” : [
3,
6,
17,
0
],
“openssl” : {
“running” : “OpenSSL 1.0.1e-fips 11 Feb 2013”,
“compiled” : “OpenSSL 1.0.1e-fips 11 Feb 2013”
},
“buildEnvironment” : {
“distmod” : “rhel70”,
“distarch” : “x86_64”,
“cc” : “/opt/mongodbtoolchain/v2/bin/gcc: gcc (GCC) 5.4.0”,
“ccflags” : “-fno-omit-frame-pointer -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -Werror -O2 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -Wno-unused-but-set-variable -Wno-missing-braces -fstack-protector-strong -fno-builtin-memcmp”,
“cxx” : “/opt/mongodbtoolchain/v2/bin/g++: g++ (GCC) 5.4.0”,
“cxxflags” : “-Woverloaded-virtual -Wno-maybe-uninitialized -std=c++14”,
“linkflags” : “-pthread -Wl,-z,now -rdynamic -Wl,–fatal-warnings -fstack-protector-strong -fuse-ld=gold -Wl,–build-id -Wl,–hash-style=gnu -Wl,-z,noexecstack -Wl,–warn-execstack -Wl,-z,relro”,
“target_arch” : “x86_64”,
“target_os” : “linux”
},
“bits” : 64,
“debug” : false,
“maxBsonObjectSize” : 16777216,
“storageEngines” : [
“devnull”,
“ephemeralForTest”,
“mmapv1”,
“wiredTiger”
],
“ok” : 1,
“operationTime” : Timestamp(1617065293, 297),
“$clusterTime” : {
“clusterTime” : Timestamp(1617065293, 297),
“signature” : {
“hash” : BinData(0,“AAAAAAAAAAAAAAAAAAAAAAAAAAA=”),
“keyId” : NumberLong(0)
}
}
}
[root@abc ~]# cat /etc/red
redhat-release redis-1.conf redis-2.conf redis-3.conf redis-4.conf redis-5.conf redis.conf
[root@abc ~]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
[root@abc ~]# ssh DEF mongod --version
db version v3.6.17
git version: 3d6953c361213c5bfab23e51ab274ce592edafe6
OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
allocator: tcmalloc
modules: none
build environment:
distmod: rhel70
distarch: x86_64
target_arch: x86_64
[root@abc ~]#

9

@Stennie_X, do you see any difference?

10

Hi @venkataraman_r,

I don’t think there is any meaningful difference in the mongod build options: both are built from the same gitVersion and appear to have relevant libraries for different CentOS release series.

ldd /usr/bin/mongod

While I don’t think this a useful path to continue pursuing, one of my colleagues shared a tip:

Try using readelf -a | grep NEEDED rather than ldd. Running ldd is going to show you not just the direct dependencies of mongod, but the transitive dependencies as well. Since system libraries may have changed their dependencies between RHEL 7 and RHEL 8, the output may well differ even if mongod is actually linked to the same system libraries. Using readelf to extract the DT_NEEDED entries will show only the direct dependencies that the mongod binary expresses.

I would follow the advice that Edwin gave in the server issue discussion:

We identified the behavior that occurs, but we don’t exactly know why it happens. The MMAPv1 storage engine has been deprecated in favor of WiredTiger since MongoDB 4.0, which correlates with when this issue went away. You can find more information on the storage engine, as well as migrating to WiredTiger from MMAPv1 as a replica set in our docs. We’d love to hear back from you if this issue persists after you’ve upgraded to MongoDB 4.0.

I know you aren’t prepared to upgrade to WiredTiger yet, but it has been the default storage engine since MongoDB 3.2 (December 2015) and there has been significant investment in improvements in performance and stability.

Is there a known issue with your use case that prevents using WiredTiger?

Regards,
Stennie

11

Thank you @Stennie_X for sharing the tip. Yes i dont see any difference with " readelf -a | grep NEEDED " command. . In past we had performance issue with WT. But we will consider WT in future. I’m fine to close this thread

1 Like
12

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.