SOLVED but curious - Error uploading loadMovieDetailsDataset.js

I must say I am delighted with the course and find Shannon’s teaching style outstanding and utterly clear! However I am puzzled by a minor problem I found:

When first attempting to upload the movieDetails in loadMovieDetailsDataset.js to my sandbox cluster, mongo shell violently dumped a stack trace and the process died leaving me out again in the bash shell. I could reproduce it and enclose the trace but hopefully we can find an explanation from the recovery steps I managed to take.

I edited loadMovieDetailsDataset.js leaving just some documents into insertMany and found out the small collection uploaded OK. So I suspected some stray character in some document later in the file.

Then I prepared a set of prefixes of said file containing 100, 200, 300 etc documents and fed each of them to the load() function in mongo shell. I was able to successfully feed the short files up to the one which had 1500 documents. The next one provoked the dump and death symptom. To be clear:

$ wc -l file*
 106 file-0105
 206 file-0205
 306 file-0305
 406 file-0405
 506 file-0505
 606 file-0605
 706 file-0705
 806 file-0805
 906 file-0905
1006 file-1005
1106 file-1105
1206 file-1205
1306 file-1305
1406 file-1405
1506 file-1505
1606 file-1605 <---
1706 file-1705
1806 file-1805
1906 file-1905
2006 file-2005
2106 file-2105
2206 file-2205

25432 total

The file ‘file-1605’ was the first to wreak havoc.

To my surprise, if I loaded() a shorter file such as ‘file-0105’ first and then ‘file-1605’, or any of the files larger than ‘file-1605’, up to and including the last and full one, the load was successfully completed. I was able to proceed with the course with no further problem. So my theory of a stray character was moot, and maybe it rather smells of some buffer size or response times to setup buffers involved on the cluster side?

Can an explanation be inferred from this behavior? Thank you very much in advance.

1 Like

Eduardo,

would you report:

  • the version of MongoDB you are using
  • the OS and version you are using
  • the output of running:
    sum loadMovieDetailsDataset.js

Regards,
Daniel.

Hi Daniel, thank you very much for your response

MongoDB Enterprise Cluster0-oso-shard-0:PRIMARY> version()
3.4.15

$ cat /etc/debian_version 
buster/sid
$ uname -a
Linux laptop-fai 4.17.0-1-amd64 #1 SMP Debian 4.17.8-1 (2018-07-20) x86_64 GNU/Linux

$ sum loadMovieDetailsDataset.js
22909  1350

I enclose the dreaded dump just in case it proves useful

MongoDB Enterprise Cluster0-oso-shard-0:PRIMARY> db.movieDetails.drop();
true
MongoDB Enterprise Cluster0-oso-shard-0:PRIMARY> load('loadMovieDetailsDataset.js')
2018-12-12T08:22:51.127-0300 F -        [thread1] Invalid access at address: 0
2018-12-12T08:22:51.834-0300 F -        [thread1] Got signal: 11 (Segmentation fault).

   0x557fa5c9560a 0x557fa5c948ce 0x557fa5c94f1c 0x557fa5dfba06 0x7f4eabc718e0 0x557fa62d0940 0x557fa62bbed6 0x557fa62d5feb 0x557fa62d66bd 0x557fa62d72f0 0x557fa60d1560 0x557fa60d1ab2 0x557fa60d1e5b 0x557fa60d209a 0x557fa62bd1e5 0x557fa62c4738 0x557fa6124ed8 0x557fa6299b1c 0x557fa62a8eec 0x557fa62a952c 0x557fa60848dd 0x557fa60849e5 0x557fa6084b0a 0x557fa5b827e6 0x557fa5b5e344 0x557fa5b916e3 0x557fa5b97203 0x557fa61e6001 0x557fa61d96b4 0x557fa61e5d20 0x557fa61ed380 0x557fa61ed4e1 0x557fa608387a 0x557fa6083920 0x557fa5b82886 0x557fa5bb11df 0x557fa5bb15df 0x557fa5b65eec 0x7f4eabee9adf 0x7f4eabc66f2a 0x7f4eabb99edf
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"557FA54F8000","o":"79D60A","s":"_ZN5mongo15printStackTraceERSo"},{"b":"557FA54F8000","o":"79C8CE"},{"b":"557FA54F8000","o":"79CF1C"},{"b":"557FA54F8000","o":"903A06"},{"b":"7F4EABC5F000","o":"128E0"},{"b":"557FA54F8000","o":"DD8940","s":"_ZN2js14TenuringTracer8traverseI8JSObjectEEvPPT_"},{"b":"557FA54F8000","o":"DC3ED6","s":"_ZN2js8frontend9ObjectBox5traceEP8JSTracer"},{"b":"557FA54F8000","o":"DDDFEB","s":"_ZN2JS12AutoGCRooter8traceAllEP8JSTracer"},{"b":"557FA54F8000","o":"DDE6BD","s":"_ZN2js2gc9GCRuntime11markRuntimeEP8JSTracerNS1_18TraceOrMarkRuntimeE"},{"b":"557FA54F8000","o":"DDF2F0","s":"_ZN2js7Nursery7collectEP9JSRuntimeN2JS8gcreason6ReasonEPN7mozilla6VectorIPNS_11ObjectGroupELm0ENS_17SystemAllocPolicyEEE"},{"b":"557FA54F8000","o":"BD9560","s":"_ZN2js2gc9GCRuntime7gcCycleEbRNS_11SliceBudgetEN2JS8gcreason6ReasonE"},{"b":"557FA54F8000","o":"BD9AB2","s":"_ZN2js2gc9GCRuntime7collectEbNS_11SliceBudgetEN2JS8gcreason6ReasonE"},{"b":"557FA54F8000","o":"BD9E5B","s":"_ZN2js2gc9GCRuntime7startGCE18JSGCInvocationKindN2JS8gcreason6ReasonEl"},{"b":"557FA54F8000","o":"BDA09A","s":"_ZN2js2gc9GCRuntime13gcIfRequestedEP9JSContext"},{"b":"557FA54F8000","o":"DC51E5","s":"_ZN2js2gc9GCRuntime23gcIfNeededPerAllocationEP9JSContext"},{"b":"557FA54F8000","o":"DCC738","s":"_ZN2js8AllocateI8JSScriptLNS_7AllowGCE1EEEPT_PNS_16ExclusiveContextE"},{"b":"557FA54F8000","o":"C2CED8","s":"_ZN8JSScript6CreateEPN2js16ExclusiveContextEN2JS6HandleIP8JSObjectEEbRKNS3_22ReadOnlyCompileOptionsES7_jj"},{"b":"557FA54F8000","o":"DA1B1C","s":"_ZN16BytecodeCompiler12createScriptEN2JS6HandleIP8JSObjectEEb"},{"b":"557FA54F8000","o":"DB0EEC","s":"_ZN16BytecodeCompiler13compileScriptEN2JS6HandleIP8JSObjectEENS1_IP8JSScriptEE"},{"b":"557FA54F8000","o":"DB152C","s":"_ZN2js8frontend13CompileScriptEPNS_16ExclusiveContextEPNS_9LifoAllocEN2JS6HandleIP8JSObjectEENS6_IPNS_11ScopeObjectEEENS6_IP8JSScriptEERKNS5_22ReadOnlyCompileOptionsERNS5_18SourceBufferHolderEP8JSStringPNS_21SourceCompressionTaskEPPNS_18ScriptSourceObjectE"},{"b":"557FA54F8000","o":"B8C8DD"},{"b":"557FA54F8000","o":"B8C9E5"},{"b":"557FA54F8000","o":"B8CB0A"},{"b":"557FA54F8000","o":"68A7E6","s":"_ZN5mongo5mozjs14MozJSImplScope4execENS_10StringDataERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbbbi"},{"b":"557FA54F8000","o":"666344","s":"_ZN5mongo5Scope8execFileERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbbi"},{"b":"557FA54F8000","o":"6996E3","s":"_ZN5mongo5mozjs17MongoExternalInfo9Functions4load4callEP9JSContextN2JS8CallArgsE"},{"b":"557FA54F8000","o":"69F203","s":"_ZN5mongo5mozjs7smUtils12wrapFunctionINS0_17MongoExternalInfo9Functions4loadEEEbP9JSContextjPN2JS5ValueE"},{"b":"557FA54F8000","o":"CEE001","s":"_ZN2js6InvokeEP9JSContextRKN2JS8CallArgsENS_14MaybeConstructE"},{"b":"557FA54F8000","o":"CE16B4"},{"b":"557FA54F8000","o":"CEDD20","s":"_ZN2js9RunScriptEP9JSContextRNS_8RunStateE"},{"b":"557FA54F8000","o":"CF5380","s":"_ZN2js13ExecuteKernelEP9JSContextN2JS6HandleIP8JSScriptEER8JSObjectRKNS2_5ValueENS_11ExecuteTypeENS_16AbstractFramePtrEPS9_"},{"b":"557FA54F8000","o":"CF54E1","s":"_ZN2js7ExecuteEP9JSContextN2JS6HandleIP8JSScriptEER8JSObjectPNS2_5ValueE"},{"b":"557FA54F8000","o":"B8B87A"},{"b":"557FA54F8000","o":"B8B920","s":"_Z16JS_ExecuteScriptP9JSContextN2JS6HandleIP8JSScriptEENS1_13MutableHandleINS1_5ValueEEE"},{"b":"557FA54F8000","o":"68A886","s":"_ZN5mongo5mozjs14MozJSImplScope4execENS_10StringDataERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbbbi"},{"b":"557FA54F8000","o":"6B91DF"},{"b":"557FA54F8000","o":"6B95DF","s":"_ZN5mongo5mozjs15MozJSProxyScope10implThreadEPv"},{"b":"557FA54F8000","o":"66DEEC","s":"_ZN4nspr6Thread13ThreadRoutineEPv"},{"b":"7F4EABE2E000","o":"BBADF"},{"b":"7F4EABC5F000","o":"7F2A"},{"b":"7F4EABAA2000","o":"F7EDF","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.15", "gitVersion" : "52e5b5fbaa3a2a5b1a217f5e647b5061817475f9", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.17.0-1-amd64", "version" : "#1 SMP Debian 4.17.8-1 (2018-07-20)", "machine" : "x86_64" }, "somap" : [ { "b" : "557FA54F8000", "elfType" : 3, "buildId" : "734884A07D43E2E44C21EE888058D8B2E9891C54" }, { "b" : "7FFFFAD78000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "79AABA4BFD44F4BE50C96048A7E3017519CA8983" }, { "b" : "7F4EAD228000", "path" : "/lib/x86_64-linux-gnu/libboost_program_options.so.1.62.0", "elfType" : 3, "buildId" : "B325670859E3A201351AB18B1F79B95C827D9C66" }, { "b" : "7F4EAD20D000", "path" : "/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0", "elfType" : 3, "buildId" : "3756F728B6BEE37C946FC5132650D7ACCFE462F6" }, { "b" : "7F4EAD1E3000", "path" : "/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0", "elfType" : 3, "buildId" : "30F7C3596336D6D5B9870E5CF1EC21F87116F268" }, { "b" : "7F4EAD1DE000", "path" : "/lib/x86_64-linux-gnu/libboost_system.so.1.62.0", "elfType" : 3, "buildId" : "B712A4F8FD2CD7228285EA30A5D7E7321F4AC1F3" }, { "b" : "7F4EAD1C5000", "path" : "/lib/x86_64-linux-gnu/libboost_iostreams.so.1.62.0", "elfType" : 3, "buildId" : "F712A6F97C9058F23BECC0B74BC6B1E13179376A" }, { "b" : "7F4EAD1BD000", "path" : "/lib/x86_64-linux-gnu/libboost_chrono.so.1.62.0", "elfType" : 3, "buildId" : "F5A272ACE6301113042BBDBA65A0D4E5195AC1C5" }, { "b" : "7F4EAD0A7000", "path" : "/lib/x86_64-linux-gnu/libboost_regex.so.1.62.0", "elfType" : 3, "buildId" : "7AF96B56873DAE8E6CF8CE123D297C674B5B0D9F" }, { "b" : "7F4EACCB1000", "path" : "/lib/x86_64-linux-gnu/libtcmalloc.so.4", "elfType" : 3, "buildId" : "5B2AC34ECEC8D63A2DF4EA65CB73F73EF4D8DBA6" }, { "b" : "7F4EACA93000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "3AF7C4BCEB19B6C83F76E2822B9A23041D85F6D1" }, { "b" : "7F4EAC842000", "path" : "/lib/x86_64-linux-gnu/libstemmer.so.0d", "elfType" : 3, "buildId" : "9CFDCA3EC241381801BA84204465310304FC7819" }, { "b" : "7F4EAC5C4000", "path" : "/lib/x86_64-linux-gnu/libyaml-cpp.so.0.5", "elfType" : 3, "buildId" : "B658AFC5A5081D3BC01514CD135E11A630ACADF9" }, { "b" : "7F4EAC3BC000", "path" : "/lib/x86_64-linux-gnu/libsnappy.so.1", "elfType" : 3, "buildId" : "5CACE6E4A1B7E4056635F7C863ACA22A16C8269E" }, { "b" : "7F4EAC346000", "path" : "/lib/x86_64-linux-gnu/libpcre.so.3", "elfType" : 3, "buildId" : "00EBA0162238FBA99AFD9A718829CC8EF4EB4C59" }, { "b" : "7F4EAC33A000", "path" : "/lib/x86_64-linux-gnu/libpcrecpp.so.0", "elfType" : 3, "buildId" : "FCDB6EF2DC1BB0BE08948B3718BEFADC8A64D6A2" }, { "b" : "7F4EAC2A9000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "0DCA1E57517AA2BAC1F21375AD48C7AB8C7BB88D" }, { "b" : "7F4EABFC2000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "B4827F5CE42E5EE05D3996A30F143706D9B29155" }, { "b" : "7F4EABFB8000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "1743408773C1C73E8871A5B6BED179F92229E607" }, { "b" : "7F4EABFB3000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "B7883B3FC771CFA5FCB452861BBB97A5B646259B" }, { "b" : "7F4EABE2E000", "path" : "/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "356267525E6FF4F6BF94C57FC1E570B2E3268DE4" }, { "b" : "7F4EABC9A000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FAB2857727406CACCD7AB22E1729B09CCF2C3EB7" }, { "b" : "7F4EABC80000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "7934C226EB35F973D2BF21C4F5546952313E26C9" }, { "b" : "7F4EABC5F000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "C1969B6AC0E7A64F9CD88FDCE8B584CCFC16623D" }, { "b" : "7F4EABAA2000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "DC87CD1E2B171A4C51139CB4E1F2EC630E711DE3" }, { "b" : "7F4EAB892000", "path" : "/lib/x86_64-linux-gnu/libbz2.so.1.0", "elfType" : 3, "buildId" : "06EA95CA3DF11E4A8E85791C1FF89A49FC3FE9C1" }, { "b" : "7F4EA9CE7000", "path" : "/lib/x86_64-linux-gnu/libicudata.so.60", "elfType" : 3, "buildId" : "215806AD2137691877D3841E39F581E11AF8A709" }, { "b" : "7F4EA9846000", "path" : "/lib/x86_64-linux-gnu/libicui18n.so.60", "elfType" : 3, "buildId" : "F1F21DC7F9BDA7CDAA5B532ECCF7E42A7845AE62" }, { "b" : "7F4EA948F000", "path" : "/lib/x86_64-linux-gnu/libicuuc.so.60", "elfType" : 3, "buildId" : "D80190F95BA80B08C2B2BDA83318789C3D9E1BF5" }, { "b" : "7F4EA9274000", "path" : "/lib/x86_64-linux-gnu/libunwind.so.8", "elfType" : 3, "buildId" : "37E14A4A8AD61BEEB7EB34B640381384EBE96D2D" }, { "b" : "7F4EAD2C7000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "DC5CB16F5E644116CAC64A4C3F5DA4D081B81A4F" }, { "b" : "7F4EA904C000", "path" : "/lib/x86_64-linux-gnu/liblzma.so.5", "elfType" : 3, "buildId" : "35CB083CB954DD492478DB6C7F9C998DD5053AE5" } ] }}
 mongo(_ZN5mongo15printStackTraceERSo+0x3A) [0x557fa5c9560a]
 mongo(+0x79C8CE) [0x557fa5c948ce]
 mongo(+0x79CF1C) [0x557fa5c94f1c]
 mongo(+0x903A06) [0x557fa5dfba06]
 libpthread.so.0(+0x128E0) [0x7f4eabc718e0]
 mongo(_ZN2js14TenuringTracer8traverseI8JSObjectEEvPPT_+0x0) [0x557fa62d0940]
 mongo(_ZN2js8frontend9ObjectBox5traceEP8JSTracer+0x36) [0x557fa62bbed6]
 mongo(_ZN2JS12AutoGCRooter8traceAllEP8JSTracer+0xDB) [0x557fa62d5feb]
 mongo(_ZN2js2gc9GCRuntime11markRuntimeEP8JSTracerNS1_18TraceOrMarkRuntimeE+0xAD) [0x557fa62d66bd]
 mongo(_ZN2js7Nursery7collectEP9JSRuntimeN2JS8gcreason6ReasonEPN7mozilla6VectorIPNS_11ObjectGroupELm0ENS_17SystemAllocPolicyEEE+0x290) [0x557fa62d72f0]
 mongo(_ZN2js2gc9GCRuntime7gcCycleEbRNS_11SliceBudgetEN2JS8gcreason6ReasonE+0x90) [0x557fa60d1560]
 mongo(_ZN2js2gc9GCRuntime7collectEbNS_11SliceBudgetEN2JS8gcreason6ReasonE+0x1D2) [0x557fa60d1ab2]
 mongo(_ZN2js2gc9GCRuntime7startGCE18JSGCInvocationKindN2JS8gcreason6ReasonEl+0x4B) [0x557fa60d1e5b]
 mongo(_ZN2js2gc9GCRuntime13gcIfRequestedEP9JSContext+0x8A) [0x557fa60d209a]
 mongo(_ZN2js2gc9GCRuntime23gcIfNeededPerAllocationEP9JSContext+0x145) [0x557fa62bd1e5]
 mongo(_ZN2js8AllocateI8JSScriptLNS_7AllowGCE1EEEPT_PNS_16ExclusiveContextE+0x28) [0x557fa62c4738]
 mongo(_ZN8JSScript6CreateEPN2js16ExclusiveContextEN2JS6HandleIP8JSObjectEEbRKNS3_22ReadOnlyCompileOptionsES7_jj+0x38) [0x557fa6124ed8]
 mongo(_ZN16BytecodeCompiler12createScriptEN2JS6HandleIP8JSObjectEEb+0x2C) [0x557fa6299b1c]
 mongo(_ZN16BytecodeCompiler13compileScriptEN2JS6HandleIP8JSObjectEENS1_IP8JSScriptEE+0x9C) [0x557fa62a8eec]
 mongo(_ZN2js8frontend13CompileScriptEPNS_16ExclusiveContextEPNS_9LifoAllocEN2JS6HandleIP8JSObjectEENS6_IPNS_11ScopeObjectEEENS6_IP8JSScriptEERKNS5_22ReadOnlyCompileOptionsERNS5_18SourceBufferHolderEP8JSStringPNS_21SourceCompressionTaskEPPNS_18ScriptSourceObjectE+0x6C) [0x557fa62a952c]
 mongo(+0xB8C8DD) [0x557fa60848dd]
 mongo(+0xB8C9E5) [0x557fa60849e5]
 mongo(+0xB8CB0A) [0x557fa6084b0a]
 mongo(_ZN5mongo5mozjs14MozJSImplScope4execENS_10StringDataERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbbbi+0x116) [0x557fa5b827e6]
 mongo(_ZN5mongo5Scope8execFileERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbbi+0x664) [0x557fa5b5e344]
 mongo(_ZN5mongo5mozjs17MongoExternalInfo9Functions4load4callEP9JSContextN2JS8CallArgsE+0xB3) [0x557fa5b916e3]
 mongo(_ZN5mongo5mozjs7smUtils12wrapFunctionINS0_17MongoExternalInfo9Functions4loadEEEbP9JSContextjPN2JS5ValueE+0x33) [0x557fa5b97203]
 mongo(_ZN2js6InvokeEP9JSContextRKN2JS8CallArgsENS_14MaybeConstructE+0xE1) [0x557fa61e6001]
 mongo(+0xCE16B4) [0x557fa61d96b4]
 mongo(_ZN2js9RunScriptEP9JSContextRNS_8RunStateE+0xA0) [0x557fa61e5d20]
 mongo(_ZN2js13ExecuteKernelEP9JSContextN2JS6HandleIP8JSScriptEER8JSObjectRKNS2_5ValueENS_11ExecuteTypeENS_16AbstractFramePtrEPS9_+0xF0) [0x557fa61ed380]
 mongo(_ZN2js7ExecuteEP9JSContextN2JS6HandleIP8JSScriptEER8JSObjectPNS2_5ValueE+0xB1) [0x557fa61ed4e1]
 mongo(+0xB8B87A) [0x557fa608387a]
 mongo(_Z16JS_ExecuteScriptP9JSContextN2JS6HandleIP8JSScriptEENS1_13MutableHandleINS1_5ValueEEE+0x60) [0x557fa6083920]
 mongo(_ZN5mongo5mozjs14MozJSImplScope4execENS_10StringDataERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEbbbi+0x1B6) [0x557fa5b82886]
 mongo(+0x6B91DF) [0x557fa5bb11df]
 mongo(_ZN5mongo5mozjs15MozJSProxyScope10implThreadEPv+0x13F) [0x557fa5bb15df]
 mongo(_ZN4nspr6Thread13ThreadRoutineEPv+0x1C) [0x557fa5b65eec]
 libstdc++.so.6(+0xBBADF) [0x7f4eabee9adf]
 libpthread.so.0(+0x7F2A) [0x7f4eabc66f2a]
 libc.so.6(clone+0x3F) [0x7f4eabb99edf]
-----  END BACKTRACE  -----
Segmentation fault
1 Like

Completely unrelated, but I’d like to applaud you for showing true hacker-thinking! Your thought process and the way you tackled this whole thing is outstanding.

The funny thing is that the process you’ve shown (cutting up data to find which content creates a segfault) is very close to the process one would use to investigate a buffer overflow attack.

1 Like

Eduardo,

it seems like your file is not corrupted.
I can’t reproduce the issue, using the same MongoDB version.
If you want to continue investigating, it may be worth trying to import the same file that fails from another machine/OS.

Regards,
Daniel

1 Like

Thank you very much, Daniel!

Hi everyone, i had that the some problem. I solved this using CentOs 7 in a VirtualBox, thats work to me.

There is some problem with 3.4.x mongo shell and this .js file. It segfaults for me too.
No problems though when I upgraded the shell client to 3.6.