Problems firing up mongod from systemd

I am trying to launch mongod with my own configuration from systemd, but it does not work. And I do not understand why. Any help by some more experienced person would be very welcome.

This is what I can say to start with:
1) When started outside of systemd, using the mongod command, the configuration works with no issue.
2) When I use systemd with the default configuration instead of mine, it also works with no issue.

This is the report provided by “systemctl status”, when using my own configuration:

root@ubuntu:/home/ubuntu# systemctl status mongod
● mongod.service - MongoDB Database Server
     Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Thu 2020-12-17 03:43:33 UTC; 15min ago
       Docs: https://docs.mongodb.org/manual
    Process: 1816 ExecStart=/usr/bin/mongod --config /home/ubuntu/mongo-1.cfg (code=exited, status=0/SUCCESS)
   Main PID: 1816 (code=exited, status=0/SUCCESS)

Dec 17 03:43:10 ubuntu systemd[1]: Started MongoDB Database Server.
Dec 17 03:43:18 ubuntu mongod[1816]: about to fork child process, waiting until server is ready for connections.
Dec 17 03:43:18 ubuntu mongod[1982]: forked process: 1982
Dec 17 03:43:24 ubuntu mongod[1816]: child process started successfully, parent exiting
Dec 17 03:43:33 ubuntu systemd[1]: mongod.service: Succeeded.
root@ubuntu:/home/ubuntu# 

Here is the content of the config file:

ubuntu@ubuntu:~$ cat mongo-1.cfg 
storage:
  dbPath: /mnt/mongoDB-One/DB_Data_1st
  journal:
    enabled: true
net:
  bindIp: localhost,192.168.1.2
  port: 22330
systemLog:
  destination: file
  path: /mnt/mongoDB-One/DB_Data_1st/mongod.log
  logAppend: true
processManagement:
  fork: true
replication:
  replSetName: mngoRepSet
ubuntu@ubuntu:~$ 

Here is the end of the server log file:

root@ubuntu:/home/ubuntu# tail -20 /mnt/mongoDB-One/DB_Data_1st/mongod.log
{"t":{"$date":"2020-12-17T03:43:25.146+00:00"},"s":"I",  "c":"FTDC",     "id":20626,   "ctx":"SignalHandler","msg":"Shutting down full-time diagnostic data capture"}
{"t":{"$date":"2020-12-17T03:43:25.160+00:00"},"s":"I",  "c":"STORAGE",  "id":4784927, "ctx":"SignalHandler","msg":"Shutting down the HealthLog"}
{"t":{"$date":"2020-12-17T03:43:25.160+00:00"},"s":"I",  "c":"STORAGE",  "id":4784929, "ctx":"SignalHandler","msg":"Acquiring the global lock for shutdown"}
{"t":{"$date":"2020-12-17T03:43:25.160+00:00"},"s":"I",  "c":"STORAGE",  "id":4784930, "ctx":"SignalHandler","msg":"Shutting down the storage engine"}
{"t":{"$date":"2020-12-17T03:43:25.160+00:00"},"s":"I",  "c":"STORAGE",  "id":20282,   "ctx":"SignalHandler","msg":"Deregistering all the collections"}
{"t":{"$date":"2020-12-17T03:43:25.161+00:00"},"s":"I",  "c":"STORAGE",  "id":22372,   "ctx":"OplogVisibilityThread","msg":"Oplog visibility thread shutting down."}
{"t":{"$date":"2020-12-17T03:43:25.161+00:00"},"s":"I",  "c":"STORAGE",  "id":22261,   "ctx":"SignalHandler","msg":"Timestamp monitor shutting down"}
{"t":{"$date":"2020-12-17T03:43:25.162+00:00"},"s":"I",  "c":"STORAGE",  "id":22317,   "ctx":"SignalHandler","msg":"WiredTigerKVEngine shutting down"}
{"t":{"$date":"2020-12-17T03:43:25.163+00:00"},"s":"I",  "c":"STORAGE",  "id":22318,   "ctx":"SignalHandler","msg":"Shutting down session sweeper thread"}
{"t":{"$date":"2020-12-17T03:43:25.164+00:00"},"s":"I",  "c":"STORAGE",  "id":22319,   "ctx":"SignalHandler","msg":"Finished shutting down session sweeper thread"}
{"t":{"$date":"2020-12-17T03:43:25.164+00:00"},"s":"I",  "c":"STORAGE",  "id":22320,   "ctx":"SignalHandler","msg":"Shutting down journal flusher thread"}
{"t":{"$date":"2020-12-17T03:43:25.164+00:00"},"s":"I",  "c":"STORAGE",  "id":22321,   "ctx":"SignalHandler","msg":"Finished shutting down journal flusher thread"}
{"t":{"$date":"2020-12-17T03:43:25.164+00:00"},"s":"I",  "c":"STORAGE",  "id":22322,   "ctx":"SignalHandler","msg":"Shutting down checkpoint thread"}
{"t":{"$date":"2020-12-17T03:43:25.164+00:00"},"s":"I",  "c":"STORAGE",  "id":22323,   "ctx":"SignalHandler","msg":"Finished shutting down checkpoint thread"}
{"t":{"$date":"2020-12-17T03:43:25.164+00:00"},"s":"I",  "c":"STORAGE",  "id":4795902, "ctx":"SignalHandler","msg":"Closing WiredTiger","attr":{"closeConfig":"leak_memory=true,"}}
{"t":{"$date":"2020-12-17T03:43:33.802+00:00"},"s":"I",  "c":"STORAGE",  "id":4795901, "ctx":"SignalHandler","msg":"WiredTiger closed","attr":{"durationMillis":8638}}
{"t":{"$date":"2020-12-17T03:43:33.802+00:00"},"s":"I",  "c":"STORAGE",  "id":22279,   "ctx":"SignalHandler","msg":"shutdown: removing fs lock..."}
{"t":{"$date":"2020-12-17T03:43:33.803+00:00"},"s":"I",  "c":"-",        "id":4784931, "ctx":"SignalHandler","msg":"Dropping the scope cache for shutdown"}
{"t":{"$date":"2020-12-17T03:43:33.803+00:00"},"s":"I",  "c":"CONTROL",  "id":20565,   "ctx":"SignalHandler","msg":"Now exiting"}
{"t":{"$date":"2020-12-17T03:43:33.803+00:00"},"s":"I",  "c":"CONTROL",  "id":23138,   "ctx":"SignalHandler","msg":"Shutting down","attr":{"exitCode":0}}
root@ubuntu:/home/ubuntu# 

This is the content of the mongod.service file:

root@ubuntu:/home/ubuntu# cat /lib/systemd/system/mongod.service
[Unit]
Description=MongoDB Database Server
Documentation=https://docs.mongodb.org/manual
After=network-online.target
Wants=network-online.target

[Service]
User=mongodb
Group=mongodb
EnvironmentFile=-/etc/default/mongod
#(Commented out by me)ExecStart=/usr/bin/mongod --config /etc/mongod.conf
ExecStart=/usr/bin/mongod --config /home/ubuntu/mongo-1.cfg
PIDFile=/var/run/mongodb/mongod.pid
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false

# Recommended limits for mongod as specified in
# https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings

[Install]
WantedBy=multi-user.target
root@ubuntu:/home/ubuntu#

Hello @Michel_Bouchet

  • You want to run your service as mongodb:mongodb . Are the access rights set accordingly?
    So does the mongodb:mongodb has access to the dbpath and log file? You can put 770 the parh and 600 for the files.
  • You mention that a manual start works fine. Did you tried this as user mongodb ? If not please do so. You can set the option fork in your mongo-1.cfg to false to get potential error messages on the console.
  • When you start via systemctl do you get any messages in /var/log/system?
  • The server.log shows a normal shutdown, is there something else indicating an error? Due to this I doubt actually problem with access rights
  • is there any other indication on an error before the shutdown in the mongod.log?

Regrades,
Michael

When I launched manually mongod, it was done as root.

Running it as mongodb is (also) one of the reasons I am trying to launch from systemd.

As far as the permissions are concerned I have run this command:

sudo chown -R mongodb:mongodb /mnt/mongoDB-One/DB_Data_1st

and the access rights are unchanged. For example:

ubuntu@ubuntu:~$ ls -l /mnt/mongoDB-One/DB_Data_1st/mongod.log
-rw------- 1 mongodb mongodb 402819 Dec 17 07:20 /mnt/mongoDB-One/DB_Data_1st/mongod.log
ubuntu@ubuntu:~$ 

As you can see, that is 600.

I presume there is no reason here to have permissions problems. Let me know if I am wrong.

I looked at /var/log/mongodb/mongod.log and didn’t see anything suspicious.

There is no /var/log/system.

Beside, I don’t know how to launch mongod as mongodb outside of systemd.
Because I can’t just log in as mongodb.

I changed the configuration file as an experiment:

I deleted these 2 lines:

processManagement:
  fork: true

And this is the result of systemctl status mongod

root@ubuntu:/home/ubuntu# systemctl status mongod
● mongod.service - MongoDB Database Server
     Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2020-12-17 07:48:24 UTC; 1min 12s ago
       Docs: https://docs.mongodb.org/manual
    Process: 1817 ExecStart=/usr/bin/mongod --config /home/ubuntu/mongo-1.cfg (code=exited, status=14)
   Main PID: 1817 (code=exited, status=14)

Dec 17 07:47:37 ubuntu systemd[1]: Started MongoDB Database Server.
Dec 17 07:48:24 ubuntu systemd[1]: mongod.service: Main process exited, code=exited, status=14/n/a
Dec 17 07:48:24 ubuntu systemd[1]: mongod.service: Failed with result 'exit-code'.
root@ubuntu:/home/ubuntu#

I am pretty sure that we will find the reason in the mongod.log
Can you please empty the log and try to start the service… If the log is not too big can you please post it here.
(Please make sure that there is nothing confidential posted)

I may already have an idea. By running this command:

ls -l /mnt/mongoDB-One/DB_Data_1st/* | grep -v mongodb

I found a few files with root onwnership. Maybe because I once ran mongod as root for test.
Anyway, I then reran:

sudo chown -R mongodb:mongodb /mnt/mongoDB-One/DB_Data_1st

and that seems to solve the problem. It means you were right to suspect some permission issue.

But there is something else I noticed. If I have these two lines in the config file:

processManagement:
  fork: true

then the server starts and then vanishes after a short while. On the other hand if I delete them, all seems to be fine. Do you know if this is normal?

Hello @Michel_Bouchet

What do you mean with this? The fork sends the process to the background so that would be ok, You can then connect to the mongodb shell via the mongo command.
Or do you mean that the process dies? In this case please clean the mongod log an, rerun and wait until the process dies and post the log.

Hi @Michel_Bouchet

I was looking at this the other day on another thread. Systemd expects a service to run in the foreground by default (Type=simple). As the forking process exits systemd will kill the whole cgroup for the service.

The Ubuntu .deb has a systemd service that uses simple and a default /etc/mongod.conf with no forking.

The RPM mongo version I looked at the other day(Centos 6 or 7) has forking enabled and the systemd has Type = Forking. The RedHat package also “fixes” any permissions errors in the PreExec stage.

2 Likes

Hi @chris

thanks for pointing that out. I never had a fork: true in my systemd configs and was not aware of this issue with the deb packages.

@Michel_Bouchet in your mongod.log should be something like:

"CONTROL",  "id":23377,   "ctx":"SignalHandler","msg":"Received signal","attr":"signal":15,"error":"Terminated"}}
"CONTROL",  "id":23378,   "ctx":"SignalHandler","msg":"Signal was sent by kill(2)","attr":{"pid":1,"uid":0}}

For something like this I was looking for, though I would not immediately have thought of the fork
Thanks Chris! You saved me some intense head scratching … I just tried it out to get the loglines…

Regards,
Michael

By this, I mean the fact that it does not work when I leave the 2 lines in the config;
and it works when I remove them.

The process going to the background when the 2 lines are in, is what happens when mongod is started by hand (And this is also what I expect).

When mongod is started by systemd, it does not behave that way. I don’t know why.
Maybe I need to follow your suggestions, erasing the logs first.

I see. Then this explains the strange behaviour I found then.
I just shouldn’t fork when using systemd then.

Thanks.

Yep, thanks to Chris we know now :slight_smile:

As I wrote you on the university forum, it’s very effective here. Problem solved even before I get a chance to participate.

1 Like

Hey … I know I’m late to the party, but I want to leave my 2cents here.

Systemd has a parameter called TimeoutSec (and if you want more control, there is TimeoutStartSec and TimeoutStopSec). In a few words, it controls for how long systemd will wait until service is fully started or stopped.

In some situations, if your mongod has been abruptly killed, when you restart it MongoDB has to do some housekeeping tasks before it fully starts the instance. If the time to finish these housekeeping tasks goes beyond the default TimeoutSec (which I believe is 90 secs, but don’t quote me on that), systemd will kill the daemon.

After you fixed your permissions, I believe your system was facing the issue I described above.

The solution is to add “TimeoutStartSec=3600” to the mongod.service file which tells systemd to kill mongod only after 1 hour if it didn’t fully start.

By adding that to the mongod.service, you don’t have to remove the fork: true option from the mongod.conf file.

All the best,

– Rodrigo

2 Likes

Yeah, thanks. That was a good advice. I keep the reference and use it sometime.

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.