How to make best use of the hex output and the backtrace when mongod crashes?

Every time mongod or mongos crashes, we’ll see some hex numbers and the backtraces. The backtraces looks messy and ugly and needs to be parsed into human readable things.

I want to know in mongodb team, how do you make best use of the hex number list, and the backtrace which may contains hex or symbols to find the potential cause of the crashes ? In extreme situation, do we have to use some kind of disassembly techniques to explain those hex numbers ?

As far as I know, there’re two tools , c++filt, and addr2line. Any other methods ?

Thanks.

And how to use this kind of json backtrace ? I cannot figure out the fields’ meaning.

image

Today I met:

Sun Oct 10 17:13:07.165 I ASIO     [TaskExecutorPool-5] Connecting to 100.117.151.187:7003
Sun Oct 10 17:13:07.165 I ASIO     [TaskExecutorPool-5] Connecting to 100.117.151.187:7003
Sun Oct 10 17:13:07.167 F -        [conn2017861] Got signal: 6 (Aborted).
 0x7fb610527301 0x7fb610526539 0x7fb610526a1d 0x7fb60de77630 0x7fb60dad0387 0x7fb60dad1a78 0x7fb60f8bf427 0x7fb60fdb67cb 0x7fb60fdaa509 0x7fb60fb6481b 0x7fb60fb67bfb 0x7fb60f9a0ae9 0x7fb60f9a3f40 0x7fb60ff0e0d9 0x7fb60fa11718 0x7fb60fa12d52 0x7fb60f910bfc 0x7fb60f9302aa 0x7fb60f92af37 0x7fb60f92e8b1 0x7fb60fdb3e52 0x7fb60f92890f 0x7fb60f92c455 0x7fb60f92a677 0x7fb60f92afca 0x7fb60f92e8b1 0x7fb60fdb43b5 0x7fb61049e304 0x7fb60de6fea5 0x7fb60db9896d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"7FB60F3CB000","o":"115C301","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7FB60F3CB000","o":"115B539"},{"b":"7FB60F3CB000","o":"115BA1D"},{"b":"7FB60DE68000","o":"F630"},{"b":"7FB60DA9A000","o":"36387","s":"gsignal"},{"b":"7FB60DA9A000","o":"37A78","s":"abort"},{"b":"7FB60F3CB000","o":"4F4427","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"7FB60F3CB000","o":"9EB7CB","s":"_ZN5mongo9transport18TransportLayerASIO9makeBatonEPNS_16OperationContextE"},{"b":"7FB60F3CB000","o":"9DF509","s":"_ZN5mongo9transport21TransportLayerManager9makeBatonEPNS_16OperationContextE"},{"b":"7FB60F3CB000","o":"79981B","s":"_ZN5mongo19AsyncRequestsSender13BatonDetacherC2EPNS_16OperationContextE"},{"b":"7FB60F3CB000","o":"79CBFB","s":"_ZN5mongo19AsyncRequestsSenderC2EPNS_16OperationContextEPNS_8executor12TaskExecutorENS_10StringDataERKSt6vectorINS0_7RequestESaIS8_EERKNS_21ReadPreferenceSettingENS_5Shard11RetryPolicyE"},{"b":"7FB60F3CB000","o":"5D5AE9"},{"b":"7FB60F3CB000","o":"5D8F40"},{"b":"7FB60F3CB000","o":"B430D9","s":"_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_19CommandReplyBuilderE"},{"b":"7FB60F3CB000","o":"646718"},{"b":"7FB60F3CB000","o":"647D52","s":"_ZN5mongo8Strategy13clientCommandEPNS_16OperationContextERKNS_7MessageE"},{"b":"7FB60F3CB000","o":"545BFC","s":"_ZN5mongo23ServiceEntryPointMongos13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"7FB60F3CB000","o":"5652AA","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"7FB60F3CB000","o":"55FF37","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"7FB60F3CB000","o":"5638B1"},{"b":"7FB60F3CB000","o":"9E8E52","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"7FB60F3CB000","o":"55D90F","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"7FB60F3CB000","o":"561455","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"7FB60F3CB000","o":"55F677","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"7FB60F3CB000","o":"55FFCA","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"7FB60F3CB000","o":"5638B1"},{"b":"7FB60F3CB000","o":"9E93B5"},{"b":"7FB60F3CB000","o":"10D3304"},{"b":"7FB60DE68000","o":"7EA5"},{"b":"7FB60DA9A000","o":"FE96D","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.0.3-cmongo-", "gitVersion" : "09a8bf0372a4d6d0a4018265ce91de7168fa8ac4-231", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.107-1-tlinux2_kvm_guest-0052", "version" : "#1 SMP Wed Jan 15 18:42:19 CST 2020", "machine" : "x86_64" }, "somap" : [ { "b" : "7FB60F3CB000", "elfType" : 3, "buildId" : "7466A6DD2E1308BE3F8BD82D12BA1827C3C8157E" }, { "b" : "7FFE57E93000", "elfType" : 3, "buildId" : "0A5EAA696352F4C4CA890C988945C05534E370A7" }, { "b" : "7FB60EE2C000", "path" : "/data/user_00/containers//77923053/bin/libstdc++.so", "elfType" : 3 }, { "b" : "7FB60F2BF000", "path" : "/lib64/libonion.so", "elfType" : 3 }, { "b" : "7FB60EBC2000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "56C5F10F267E857EB448E1E823B84552B0D16976" }, { "b" : "7FB60E9A8000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "B45C711D26DDD9F612D7814CE83B427927C8BC65" }, { "b" : "7FB60E7A0000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "3E44DF7055942478D052E40FDD1F5B7862B152B0" }, { "b" : "7FB60E59C000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "EAB4E94DD079F139912E15B9486E4893B22E9C77" }, { "b" : "7FB60E29A000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "1A1D39CE5355AF84CDB726059B6ADB887130A2CA" }, { "b" : "7FB60E084000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "AE41224E95BB4A4E10E96D6B6043ED889E9F9D99" }, { "b" : "7FB60DE68000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "0C8B4C7A74BEE674F7852032FD32BC45798981FD" }, { "b" : "7FB60DA9A000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "7296C032D4858C33DD6CAC6D59A34A4F0E857CD2" }, { "b" : "7FB60F1A7000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "FE2055668ABBA8E002C6E446430E0767D55ADDBD" }, { "b" : "7FB60D867000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "2B77BBEFFF65E94F3E0B71A4E89BEB68C4B476C5" }, { "b" : "7FB60D63A000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "6AD633E8DBCA215631BA3531F73E04D4D0FBE2AC" }, { "b" : "7FB60D3E1000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "F439579BA89E471324CF6AC261A33293C8BBA8A7" }, { "b" : "7FB60D1B9000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "DC0565F53320EFAC2CFF17230898E7D1DA1BA6CB" }, { "b" : "7FB60CE8A000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "7727F227C7A7707B53241EF20354B66EE042BA1B" }, { "b" : "7FB60CC5A000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "4D899A1162335B8D850016B72C224FF9793BAFC5" }, { "b" : "7FB60CA56000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "285DDB5653F966F7E4F627A8A3055B3DA53F5FFC" }, { "b" : "7FB60C851000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "1D5ECBE3C96448E09808B3914E26F406F5698FB6" }, { "b" : "7FB60C613000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "99A465293209F2D37459D20DD2778C858F2407E1" }, { "b" : "7FB60C3C6000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "032A36892E45853F0042C0030A49AC9FFD67B1EB" }, { "b" : "7FB60C0DD000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "8565EF264B9DB1E04A5F129DA260EF8585974F3B" }, { "b" : "7FB60BEAA000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "AD11DCA2EA33205263669E2B2B15AA5F5FB98AAE" }, { "b" : "7FB60BCA6000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "3A1166709F88740C49E060731832E3FAD2DFB66B" }, { "b" : "7FB60BA97000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "DFD1295E44DE6B24271B22F99453AA1B8EB76B7B" }, { "b" : "7FB60B842000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "4C1F14C63394A97BB4BE8ADEEE68B21715A93219" }, { "b" : "7FB60B62C000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "1982C8CDAE90F898D1AD26DC07E807333B4789D0" }, { "b" : "7FB60B3BA000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "FCF9ABA6A186EA1AF5E235B747F0BA7D83ABAEE3" }, { "b" : "7FB60AF57000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "79A77B78D6FB077849D085027A440DE7A8683C4B" }, { "b" : "7FB60AD47000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "64111D8724DC37ED3F8DE4428FBAA3D10AC90440" }, { "b" : "7FB60AB43000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7FB60A926000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "7F93B495437EEABFFC7FB12E87902C52D4930E69" }, { "b" : "7FB60A6FF000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "A1EBEED96ABA69B054613A61A2C366FF84E82B2B" }, { "b" : "7FB60A4C8000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "97BE6F9199FED4491B00AA91F7E6EACC4D5328F7" }, { "b" : "7FB60A266000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "1F9424FBDFE5E98281C2886F3644B187EC1BBB40" }, { "b" : "7FB60A063000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "403D7683FAD4AF7AFB808CDC54EF9AA28E4D0C81" } ] }} mongos(_ZN5mongo15printStackTraceERSo+0x41) [0x7fb610527301]
mongos(+0x115B539) [0x7fb610526539]
 mongos(+0x115BA1D) [0x7fb610526a1d]
 libpthread.so.0(+0xF630) [0x7fb60de77630]
 libc.so.6(gsignal+0x37) [0x7fb60dad0387]
 libc.so.6(abort+0x148) [0x7fb60dad1a78]
 mongos(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x7fb60f8bf427]
 mongos(_ZN5mongo9transport18TransportLayerASIO9makeBatonEPNS_16OperationContextE+0x31B) [0x7fb60fdb67cb]
 mongos(_ZN5mongo9transport21TransportLayerManager9makeBatonEPNS_16OperationContextE+0x69) [0x7fb60fdaa509]
 mongos(_ZN5mongo19AsyncRequestsSender13BatonDetacherC2EPNS_16OperationContextE+0x9B) [0x7fb60fb6481b]
 mongos(_ZN5mongo19AsyncRequestsSenderC2EPNS_16OperationContextEPNS_8executor12TaskExecutorENS_10StringDataERKSt6vectorINS0_7RequestESaIS8_EERKNS_21ReadPreferenceSettingENS_5Shard11RetryPolicyE+0x5B) [0x7fb60fb67bfb]
 mongos(+0x5D5AE9) [0x7fb60f9a0ae9]
 mongos(+0x5D8F40) [0x7fb60f9a3f40]
 mongos(_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_19CommandReplyBuilderE+0xD9) [0x7fb60ff0e0d9]
 mongos(+0x646718) [0x7fb60fa11718]
 mongos(_ZN5mongo8Strategy13clientCommandEPNS_16OperationContextERKNS_7MessageE+0x1D2) [0x7fb60fa12d52]
 mongos(_ZN5mongo23ServiceEntryPointMongos13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x68C) [0x7fb60f910bfc]
 mongos(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x7fb60f9302aa]
 mongos(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x7fb60f92af37]
 mongos(+0x5638B1) [0x7fb60f92e8b1]
 mongos(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x1A2) [0x7fb60fdb3e52]
 mongos(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x15F) [0x7fb60f92890f]
 mongos(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xAF5) [0x7fb60f92c455]
 mongos(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x357) [0x7fb60f92a677]
 mongos(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x12A) [0x7fb60f92afca]
 mongos(+0x5638B1) [0x7fb60f92e8b1]
 mongos(+0x9E93B5) [0x7fb60fdb43b5]
 mongos(+0x10D3304) [0x7fb61049e304]
 libpthread.so.0(+0x7EA5) [0x7fb60de6fea5]
 libc.so.6(clone+0x6D) [0x7fb60db9896d]
-----  END BACKTRACE  -----

It’s hard for me to figure out what is wrong. Can we get useful info from such output ?

Hi -

My recommendation would be to open tickets for these sort of events in the MongoDB JIRA under the SERVER project and provide the backtraces in the ticket. While there are some tools in the source tree that can symbolize these for you, getting them to work requires downloading and unpacking the matching debug symbols for the version of the product you are gunning, installing development tools like llvm-symbolizer, and cloning the server sources and running scripts to integrate that information. If you file a SERVER ticket, we end up doing that work instead of you.

You can file those tickets here: https://jira.mongodb.org/projects/SERVER

Thanks,
Andrew

2 Likes