LDAP server unavailable: SERVER

Discussion:

LDAP server unavailable: SERVER_DOWN

Marcel Bischoff

2018-01-08 14:00:35 UTC

Hi all,

I have recently set up a Kolab 16 installation, following this guide: https://docs.kolab.org/installation-guide/ubuntu-16.04.html

Initially everything worked alright until users started connecting continually. Now, from time to time, without immediate apparent reason, the LDAP service becomes unreachable (according to the log messages). This can only be solved by manually restarting the service with "systemctl restart ***@mx.service". Even restarting the server does nothing to recitify this, although the service is started at that time.

The only deviation from the installation instructions is the TLS setup as that appears not to be covered anywhere, which has caused me great frustration, especially with regard to GUAM.

Below is a relevant excerpt from /var/log/kolab/pykolab.log. Any ideas what the problem is here? I keep getting user complaints of not being able to log in, neither web nor email clients, which put me on the scent of the LDAP server being the central authentication instance.

2018-01-08 14:01:10,922 pykolab.wallace WARNING No contents configured for footer module
2018-01-08 14:04:05,940 pykolab.wallace WARNING No contents configured for footer module
2018-01-08 14:10:24,674 pykolab.wallace WARNING No contents configured for footer module
2018-01-08 14:15:49,913 pykolab.wallace WARNING No contents configured for footer module
2018-01-08 14:17:15,092 pykolab.wallace WARNING No contents configured for footer module
2018-01-08 14:17:18,222 pykolab.wallace WARNING No contents configured for footer module
2018-01-08 14:20:34,028 pykolab.wallace WARNING No contents configured for footer module
2018-01-08 14:39:08,989 pykolab.auth ERROR LDAP-Server nicht erreichbar: SERVER_DOWN({'desc': "Can't contact LDAP server"},)
2018-01-08 14:39:08,991 pykolab.auth ERROR Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line 3065, in _search
secondary_domains
File "<string>", line 10, in <module>
File "/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line 2963, in _regular_search
(_result_type, _result) = self.ldap.result(_search, False, 0)
File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 503, in result
resp_type, resp_data, resp_msgid = self.result2(msgid,all,timeout)
File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 507, in result2
resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all,timeout)
File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 514, in result3
resp_ctrl_classes=resp_ctrl_classes
File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 521, in result4
ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 106, in _ldap_call
result = func(*args,**kwargs)
SERVER_DOWN: {'desc': "Can't contact LDAP server"}

2018-01-08 14:39:08,991 pykolab.auth ERROR -- erneuter Verbindungsaufbau in 10 Sekunden.
2018-01-08 14:39:08,992 pykolab.auth ERROR LDAP server unavailable: SERVER_DOWN({'desc': "Can't contact LDAP server"},)
2018-01-08 14:39:08,993 pykolab.auth ERROR Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line 3065, in _search
secondary_domains
File "<string>", line 10, in <module>
File "/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line 2744, in _persistent_search
resp_ctrl_classes={ecnc.controlType:ecnc}
File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 521, in result4
ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 106, in _ldap_call
result = func(*args,**kwargs)
SERVER_DOWN: {'desc': "Can't contact LDAP server"}

2018-01-08 14:39:08,993 pykolab.auth ERROR -- reconnecting in 10 seconds.
2018-01-08 14:39:12,009 pykolab.wallace ERROR Module resources.heartbeat() failed with error: Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/wallace/__init__.py", line 89, in modules_heartbeat
modules.heartbeat(module, lastrun)
File "/usr/lib/python2.7/dist-packages/wallace/modules.py", line 128, in heartbeat
return modules[name]['heartbeat'](*args, **kw)
File "/usr/lib/python2.7/dist-packages/wallace/module_resources.py", line 433, in heartbeat
resource_dns = auth.find_resource('*')
File "/usr/lib/python2.7/dist-packages/pykolab/auth/__init__.py", line 220, in find_resource
result = self._auth.find_resource(address)
File "/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line 769, in find_resource
self._bind()
File "/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line 1441, in _bind
self.ldap.simple_bind_s(bind_dn, bind_pw)
File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 919, in simple_bind_s
res = self._apply_method_s(SimpleLDAPObject.simple_bind_s,*args,**kwargs)
File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 905, in _apply_method_s
self.reconnect(self._uri,retry_max=self._retry_max,retry_delay=self._retry_delay)
File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 871, in reconnect
self._apply_last_bind()
File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 844, in _apply_last_bind
SimpleLDAPObject.simple_bind_s(self,'','')
File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 223, in simple_bind_s
resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all=1,timeout=self.timeout)
File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 514, in result3
resp_ctrl_classes=resp_ctrl_classes
File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 521, in result4
ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 106, in _ldap_call
result = func(*args,**kwargs)
INAPPROPRIATE_AUTH: {'info': 'Anonymous access is not allowed', 'desc': 'Inappropriate authentication'}

2018-01-08 14:39:17,986 pykolab.auth ERROR Invalid DN, username and/or password.
2018-01-08 14:39:31,749 pykolab.wallace WARNING No contents configured for footer module
2018-01-08 14:39:32,067 pykolab.wallace WARNING No contents configured for footer module
2018-01-08 14:39:32,341 pykolab.wallace WARNING No contents configured for footer module
2018-01-08 14:39:32,916 pykolab.wallace WARNING No contents configured for footer module
2018-01-08 14:39:38,655 pykolab.wallace WARNING No contents configured for footer module

Thank you for any and all ideas how to make this more stable.

Best regards,
Marcel Bischoff

Marcel Bischoff

2018-01-08 15:03:42 UTC

Permalink

Post by Marcel Bischoff
I have recently set up a Kolab 16 installation, following this guide: https://docs.kolab.org/installation-guide/ubuntu-16.04.html
Initially everything worked alright until users started connecting
continually. Now, from time to time, without immediate apparent
reason, the LDAP service becomes unreachable (according to the log
messages). This can only be solved by manually restarting the service
does nothing to recitify this, although the service is started at that
time.

Can you have a look at the dirsrv logs (access&error)?

Good idea, I just did. Here is the tail of the "errors" file. The "access" file has way too many entries to be usefulin this context:

[07/Jan/2018:19:39:15 +0100] NSACLPlugin - acl_access_allowed: Resetting aclpb_pblock 7f146c7e7bd0 to pblock addr 7f14480088d0
[07/Jan/2018:20:42:50 +0100] NSACLPlugin - acl_access_allowed: Resetting aclpb_pblock 7f14747f7bd0 to pblock addr 7f13f400cf50
[07/Jan/2018:21:46:16 +0100] NSACLPlugin - acl_access_allowed: Resetting aclpb_pblock 7f1476ffcbd0 to pblock addr 7f14000097f0
[08/Jan/2018:09:35:30 +0100] NSACLPlugin - acl_access_allowed: Resetting aclpb_pblock 7f14727f3bd0 to pblock addr 7f1428008100
[08/Jan/2018:09:35:31 +0100] NSACLPlugin - acl_access_allowed: Resetting aclpb_pblock 7f14757f9bd0 to pblock addr 7f1420000d50
[08/Jan/2018:10:21:28 +0100] NSACLPlugin - acl_access_allowed: Resetting aclpb_pblock 7f14727f3bd0 to pblock addr 7f1428006770
[08/Jan/2018:11:16:07 +0100] - 389-Directory/1.3.4.9 B2016.109.158 starting up
[08/Jan/2018:11:16:07 +0100] - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[08/Jan/2018:11:16:08 +0100] - slapd started. Listening on All Interfaces port 389 for LDAP requests
[08/Jan/2018:11:24:53 +0100] - 389-Directory/1.3.4.9 B2016.109.158 starting up
[08/Jan/2018:11:24:53 +0100] - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[08/Jan/2018:11:24:53 +0100] - slapd started. Listening on All Interfaces port 389 for LDAP requests
[08/Jan/2018:11:26:41 +0100] NSACLPlugin - acl_access_allowed: Resetting aclpb_pblock 7f471dff2bd0 to pblock addr 7f46c4004900
[08/Jan/2018:14:39:09 +0100] - 389-Directory/1.3.4.9 B2016.109.158 starting up
[08/Jan/2018:14:39:09 +0100] - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
[08/Jan/2018:14:39:09 +0100] - slapd started. Listening on All Interfaces port 389 for LDAP requests
[08/Jan/2018:14:39:20 +0100] NSACLPlugin - acl_access_allowed: Resetting aclpb_pblock 7f0fdbffebd0 to pblock addr 7f0f5c00bc80
[08/Jan/2018:15:24:48 +0100] NSACLPlugin - acl_access_allowed: Resetting aclpb_pblock 7f0fdb7fdbd0 to pblock addr 7f0fac008480

Well... "Disorderly Shutdown" does not sound good at all.

Jochen Hein

2018-01-08 19:45:24 UTC

Permalink

Post by Marcel Bischoff

Can you have a look at the dirsrv logs (access&error)?

[08/Jan/2018:11:16:07 +0100] - Detected Disorderly Shutdown last time Directory Server was running, recovering database.

...

Post by Marcel Bischoff
Well... "Disorderly Shutdown" does not sound good at all.

Yes - and no hints either. Are there errors in dmesg? Have a look at
http://www.port389.org/docs/389ds/FAQ/faq.html#troubleshooting

Jochen

--
This space is intentionally left blank.

Skale, Franz

2018-01-09 09:36:15 UTC

Permalink

Hi,
this i an old bug you hit.
Since i have no problems with debian jessie ( equal to Ubuntu 14.04 LTS)
i think it's a library dependency problem with the ldap development
libs.
Bug report from 2015:
https://bugzilla.redhat.com/show_bug.cgi?id=1264224
This line tells it all:
NSACLPlugin - acl_access_allowed: Resetting aclpb_pblock 7f146c7e7bd0 to
pblock addr 7f14480088d0
I think jessie is stable but ubuntu 16.04 isn't 100% stable as it is for
stretch.
I tried to port the jessie packages to stretch and failed miserably.
Perhaps give it a try and open a Task on the kolab git:
https://git.kolab.org/

Regards.
Franz

Post by Jochen Hein

Post by Marcel Bischoff

Can you have a look at the dirsrv logs (access&error)?

Good idea, I just did. Here is the tail of the "errors" file. The
[08/Jan/2018:11:16:07 +0100] - Detected Disorderly Shutdown last time
Directory Server was running, recovering database.

...

Post by Marcel Bischoff
Well... "Disorderly Shutdown" does not sound good at all.

Yes - and no hints either. Are there errors in dmesg? Have a look at
http://www.port389.org/docs/389ds/FAQ/faq.html#troubleshooting
Jochen

hede

2018-01-09 10:34:17 UTC

Permalink

Post by Skale, Franz
[...]
I think jessie is stable but ubuntu 16.04 isn't 100% stable as it is
for stretch.
I tried to port the jessie packages to stretch and failed miserably.

I thought there are stretch packages already?

Jochen Hein

2018-01-09 11:08:11 UTC

Permalink

Post by hede

Post by Skale, Franz
[...]
I think jessie is stable but ubuntu 16.04 isn't 100% stable as it is
for stretch.
I tried to port the jessie packages to stretch and failed miserably.

I thought there are stretch packages already?

Only in Winterfell, not in Kolab 16. That's the reason my mail server
still runs Jessie.

Jochen

--
This space is intentionally left blank.

Skale, Franz

2018-01-09 12:54:16 UTC

Permalink

After investigating you can clearly see, that ubuntu stopped patching
of ds389-base with version 1.3.4.9 while debian stretch uses correct
patched version 1.3.5.15.
The ubuntu version is missing some CVE patches as well as non nss ldap
builds.
This could be your problem.
You can try packaging the debian stretch version on ubuntu 16.04.
I often port packages to unbuntu because of the inferior package
maintenance they provide.
Jessie uses a much older but stable version 1.3.3.5 though.
Cannot blame kolab for that.

Rgds.
Franz

Post by Marcel Bischoff

Post by Marcel Bischoff
https://docs.kolab.org/installation-guide/ubuntu-16.04.html
Initially everything worked alright until users started connecting
continually. Now, from time to time, without immediate apparent
reason, the LDAP service becomes unreachable (according to the log
messages). This can only be solved by manually restarting the service
does nothing to recitify this, although the service is started at that
time.

Can you have a look at the dirsrv logs (access&error)?

Good idea, I just did. Here is the tail of the "errors" file. The
Resetting aclpb_pblock 7f146c7e7bd0 to pblock addr 7f14480088d0
Resetting aclpb_pblock 7f14747f7bd0 to pblock addr 7f13f400cf50
Resetting aclpb_pblock 7f1476ffcbd0 to pblock addr 7f14000097f0
Resetting aclpb_pblock 7f14727f3bd0 to pblock addr 7f1428008100
Resetting aclpb_pblock 7f14757f9bd0 to pblock addr 7f1420000d50
Resetting aclpb_pblock 7f14727f3bd0 to pblock addr 7f1428006770
[08/Jan/2018:11:16:07 +0100] - 389-Directory/1.3.4.9 B2016.109.158 starting up
[08/Jan/2018:11:16:07 +0100] - Detected Disorderly Shutdown last time
Directory Server was running, recovering database.
[08/Jan/2018:11:16:08 +0100] - slapd started. Listening on All
Interfaces port 389 for LDAP requests
[08/Jan/2018:11:24:53 +0100] - 389-Directory/1.3.4.9 B2016.109.158 starting up
[08/Jan/2018:11:24:53 +0100] - Detected Disorderly Shutdown last time
Directory Server was running, recovering database.
[08/Jan/2018:11:24:53 +0100] - slapd started. Listening on All
Interfaces port 389 for LDAP requests
Resetting aclpb_pblock 7f471dff2bd0 to pblock addr 7f46c4004900
[08/Jan/2018:14:39:09 +0100] - 389-Directory/1.3.4.9 B2016.109.158 starting up
[08/Jan/2018:14:39:09 +0100] - Detected Disorderly Shutdown last time
Directory Server was running, recovering database.
[08/Jan/2018:14:39:09 +0100] - slapd started. Listening on All
Interfaces port 389 for LDAP requests
Resetting aclpb_pblock 7f0fdbffebd0 to pblock addr 7f0f5c00bc80
Resetting aclpb_pblock 7f0fdb7fdbd0 to pblock addr 7f0fac008480
Well... "Disorderly Shutdown" does not sound good at all.
_______________________________________________
users mailing list
https://lists.kolab.org/mailman/listinfo/users

Marcel Bischoff

2018-01-09 16:40:37 UTC

Permalink

Thanks. I will try to transplant the Debian Stretch package to the
Ubuntu machine tonight when there is no traffic on the server and report
on my success/failure.

Regarding blame, you are probably correct that Kolab is not directly to
blame. I would however expect a caveat notice or similar to be added to
the installation instruction overview pointing out issue like this, as
this one on particular is a breaking issue (not being able to log in
that is).

In case the transplantation is not successful, do you think a removal of
the respective packages followed by an installation from source be
possible without breaking things?

Best,
Marcel

Post by Skale, Franz
After investigating you can clearly see, that ubuntu stopped patching
of ds389-base with version 1.3.4.9 while debian stretch uses correct
patched version 1.3.5.15.
The ubuntu version is missing some CVE patches as well as non nss ldap
builds.
This could be your problem.
You can try packaging the debian stretch version on ubuntu 16.04.
I often port packages to unbuntu because of the inferior package
maintenance they provide.
Jessie uses a much older but stable version 1.3.3.5 though.
Cannot blame kolab for that.
Rgds.
Franz

Post by Marcel Bischoff

Can you have a look at the dirsrv logs (access&error)?

Skale, Franz

2018-01-09 18:12:56 UTC

Permalink

This post might be inappropriate. Click to display it.

Marcel Bischoff

2018-01-09 19:44:58 UTC

Permalink

Thank you very much for the testing and the pointers. I was successfully
able to port the package to the Ubuntu 16.04 machine. dpkg-buildpackage
complained about quite a lot of missing dependencies that were easily
install though, with a notable exception of a missing libsvrcore-dev (>=
1:4.1.2+dfsg1-3) dependency, so I had to build and install that as well,
similar to what you had to do on Jessie.

Restarting dirsrv was very quick compared to before and for now
everything appears to be running smooth. So I'm going to go ahead and
send a very big THANK YOU your way. If this works out, you will truly
have saved me from The Wrath of the Users(tm)... ;)

Have a good evening!

Best,
Marcel

Post by Skale, Franz
Hi,
https://packages.debian.org/stretch/389-ds-base
https://packages.debian.org/stretch/libsvrcore0
This will not be necessary for 16.04
Before building , check the debian/control file for dependencies.
The build with: dpkgbuildpackage -us -uc (after dpkg-source -x (dsc).
Since there are a lot of bugfixes, i gave it a try and it works
properly.
I did a backup of /var/lib/dirsrv and /etc/dirsrv before, because the
ldifs werde, of course, migrated too.
Nevertheless i would give it a try.
Rgds.
Franz

Post by Marcel Bischoff
Thanks. I will try to transplant the Debian Stretch package to the
Ubuntu machine tonight when there is no traffic on the server and report
on my success/failure.
Regarding blame, you are probably correct that Kolab is not directly to
blame. I would however expect a caveat notice or similar to be added to
the installation instruction overview pointing out issue like this, as
this one on particular is a breaking issue (not being able to log in
that is).
In case the transplantation is not successful, do you think a
removal of
the respective packages followed by an installation from source be
possible without breaking things?
Best,
Marcel

Post by Marcel Bischoff

Post by Marcel Bischoff
https://docs.kolab.org/installation-guide/ubuntu-16.04.html
Initially everything worked alright until users started connecting
continually. Now, from time to time, without immediate apparent
reason, the LDAP service becomes unreachable (according to the log
messages). This can only be solved by manually restarting
the service
does nothing to recitify this, although the service is started at that
time.

Can you have a look at the dirsrv logs (access&error)?

Skale, Franz

2018-01-10 06:17:17 UTC

Permalink

Hi Marcel,
cool, that it worked out for you.
I didn't have any problems with the new version 1.3.5.17 overnight.
Will keep it.
Ubuntu seems to neglect certain packages !

Best regards
Franz

Post by Marcel Bischoff
Thank you very much for the testing and the pointers. I was
successfully
able to port the package to the Ubuntu 16.04 machine. dpkg-buildpackage
complained about quite a lot of missing dependencies that were easily
install though, with a notable exception of a missing libsvrcore-dev (>=
1:4.1.2+dfsg1-3) dependency, so I had to build and install that as well,
similar to what you had to do on Jessie.
Restarting dirsrv was very quick compared to before and for now
everything appears to be running smooth. So I'm going to go ahead and
send a very big THANK YOU your way. If this works out, you will truly
have saved me from The Wrath of the Users(tm)... ;)
Have a good evening!
Best,
Marcel

Post by Marcel Bischoff
Thanks. I will try to transplant the Debian Stretch package to the
Ubuntu machine tonight when there is no traffic on the server and report
on my success/failure.
Regarding blame, you are probably correct that Kolab is not directly to
blame. I would however expect a caveat notice or similar to be added to
the installation instruction overview pointing out issue like this, as
this one on particular is a breaking issue (not being able to log in
that is).
In case the transplantation is not successful, do you think a removal of
the respective packages followed by an installation from source be
possible without breaking things?
Best,
Marcel

Post by Marcel Bischoff

Post by Marcel Bischoff
https://docs.kolab.org/installation-guide/ubuntu-16.04.html
Initially everything worked alright until users started
connecting
continually. Now, from time to time, without immediate apparent
reason, the LDAP service becomes unreachable (according to the log
messages). This can only be solved by manually restarting the service
does nothing to recitify this, although the service is started at that
time.

Can you have a look at the dirsrv logs (access&error)?

Marcel Bischoff

2018-01-10 13:40:38 UTC

Permalink

This post might be inappropriate. Click to display it.

Skale, Franz

2018-01-10 14:03:13 UTC

Permalink

Hi Marcel,
06:21 is the lograotation, so no problem. Same by me.
What strucks me is, that it seems that ns-slapd as to reallocate memory.
How much memory does your server have ?
send free -m
Do you have selinux enabled !
If so, disable it by adding selinux=0 to /etc/default/grub and rerun
update-grub.
Send the kernel version: uname -a

How much open file handles to your system allow per process ?
send: ulimit -a

send dmesg: (is there a segfault).
It really could be, that you have a failing memory module:
send dmidecode

Did you update your kernel days ago, if so, you sure ran into a buggy
kernel 4.9.65.
I built a 4.9.75 PTI enabled kernel which i send you to test.
Office 365 would be a bad and expensive choice.

Rgds.
Franz

Post by Marcel Bischoff
[10/Jan/2018:02:57:02.648374615 +0100] NSACLPlugin -
Resetting aclpb_pblock 0x7fc2707efa40 to pblock addr 0x7fc22c00b460
[10/Jan/2018:03:24:33.702338063 +0100] slapd shutting down - signaling
operation threads - op stack size 6 max work q size 2 max work q stack
size 2
[10/Jan/2018:03:24:33.877497688 +0100] slapd shutting down - closing
down internal subsystems and plugins
[10/Jan/2018:03:24:33.902644467 +0100] Waiting for 4 database threads to stop
[10/Jan/2018:03:24:34.821476000 +0100] All database threads now stopped
[10/Jan/2018:03:24:34.844761946 +0100] slapd shutting down - freed 2
work q stack objects - freed 8 op stack objects
[10/Jan/2018:03:24:35.702132753 +0100] slapd stopped.
[10/Jan/2018:03:25:09.710952820 +0100] 389-Directory/1.3.5.17
B2017.130.625 starting up
[10/Jan/2018:03:25:10.247208245 +0100] slapd started. Listening on
All Interfaces port 389 for LDAP requests
[10/Jan/2018:06:21:01.333124646 +0100] 389-Directory/1.3.5.17
B2017.130.625 starting up
[10/Jan/2018:06:21:02.463381179 +0100] Detected Disorderly Shutdown
last time Directory Server was running, recovering database.
[10/Jan/2018:06:21:10.733276501 +0100] slapd started. Listening on
All Interfaces port 389 for LDAP requests
[10/Jan/2018:10:43:21.361428667 +0100] slapd shutting down - signaling
operation threads - op stack size 7 max work q size 2 max work q stack
size 2
[10/Jan/2018:10:43:21.455623635 +0100] slapd shutting down - waiting
for 26 threads to terminate
[10/Jan/2018:10:43:21.467944267 +0100] slapd shutting down - closing
down internal subsystems and plugins
[10/Jan/2018:10:43:21.481536424 +0100] Waiting for 4 database threads to stop
[10/Jan/2018:10:43:22.157516146 +0100] All database threads now stopped
[10/Jan/2018:10:43:22.170111815 +0100] slapd shutting down - freed 2
work q stack objects - freed 7 op stack objects
[10/Jan/2018:10:43:23.158707276 +0100] slapd stopped.
[10/Jan/2018:10:43:23.301567366 +0100] 389-Directory/1.3.5.17
B2017.130.625 starting up
[10/Jan/2018:10:43:23.440775573 +0100] slapd started. Listening on
All Interfaces port 389 for LDAP requests
2018-01-10 10:42:54,024 pykolab.wallace WARNING No contents configured for footer module
SERVER_DOWN({'desc': "Can't contact LDAP server"},)
File
"/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line
3065, in _search
secondary_domains
File "<string>", line 10, in <module>
File
"/usr/lib/python2.7/dist-packages/pykolab/auth/ldap/__init__.py", line
2744, in _persistent_search
resp_ctrl_classes={ecnc.controlType:ecnc}
File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 521, in result4
ldap_result =
self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 106, in _ldap_call
result = func(*args,**kwargs)
SERVER_DOWN: {'desc': "Can't contact LDAP server"}
2018-01-10 10:43:22,177 pykolab.auth ERROR -- reconnecting in 10 seconds.
2018-01-10 10:48:48,333 pykolab.wallace WARNING No contents configured
for footer module
What I find notable is the fact that the service had a disorderly
shutdown at 06:21 but only when I restarted it around 10:43 did it
become unavailable. The latter is to be expected.
[10/Jan/2018:02:57:02.648374615 +0100] NSACLPlugin -
Resetting aclpb_pblock 0x7fc2707efa40 to pblock addr 0x7fc22c00b460
It may very well be that auth continued to work for everyone and my
restart was unnecessary. Being paged at 10:43, I just reflexively
restarted the service as that has been the recurring issue for the past
week, driving me nuts. Still, I don't like to see services crash
themselves for no clear and fixable reason. Also, experiencing all
thing, I just don't trust the installation. Well designed systems
usually work.
It looks like Ubuntu was a very bad choice as a platform for Kolab.
However, there's no way I can change that now. If this doesn't work out,
I fear management is going to put its fist down and orders Microsoft
Office 365. The way it's going with Kolab, I sadly cannot blame them.
Truly, I don't care whoever dropped the ball on quality control but this
is unacceptable. I have worked with several other mail server
implementations before and Kolab is the only one making a mess of
things. If Ubuntu is backwards, don't endorse it on the Website. Instead
of 5 different installation guides, do one that works, verified, on one
platform. I basically don't care what distribution I use, Ubuntu just
appeared to be the least dated one to choose from.
Best,
Marcel

Post by Skale, Franz
Hi Marcel,
cool, that it worked out for you.
I didn't have any problems with the new version 1.3.5.17 overnight.
Will keep it.
Ubuntu seems to neglect certain packages !
Best regards
Franz

Post by Marcel Bischoff
Thank you very much for the testing and the pointers. I was
successfully
able to port the package to the Ubuntu 16.04 machine.
dpkg-buildpackage
complained about quite a lot of missing dependencies that were easily
install though, with a notable exception of a missing libsvrcore-dev (>=
1:4.1.2+dfsg1-3) dependency, so I had to build and install that as well,
similar to what you had to do on Jessie.
Restarting dirsrv was very quick compared to before and for now
everything appears to be running smooth. So I'm going to go ahead and
send a very big THANK YOU your way. If this works out, you will truly
have saved me from The Wrath of the Users(tm)... ;)
Have a good evening!
Best,
Marcel

Post by Marcel Bischoff
Thanks. I will try to transplant the Debian Stretch package to the
Ubuntu machine tonight when there is no traffic on the server and report
on my success/failure.
Regarding blame, you are probably correct that Kolab is not directly to
blame. I would however expect a caveat notice or similar to be added to
the installation instruction overview pointing out issue like this, as
this one on particular is a breaking issue (not being able to log in
that is).
In case the transplantation is not successful, do you think a removal of
the respective packages followed by an installation from source be
possible without breaking things?
Best,
Marcel

Post by Marcel Bischoff

Can you have a look at the dirsrv logs (access&error)?

Good idea, I just did. Here is the tail of the "errors" file. The
Resetting aclpb_pblock 7f146c7e7bd0 to pblock addr 7f14480088d0
Resetting aclpb_pblock 7f14747f7bd0 to pblock addr 7f13f400cf50
Resetting aclpb_pblock 7f1476ffcbd0 to pblock addr 7f14000097f0
Resetting aclpb_pblock 7f14727f3bd0 to pblock addr 7f1428008100
Resetting aclpb_pblock 7f14757f9bd0 to pblock addr 7f1420000d50
Resetting aclpb_pblock 7f14727f3bd0 to pblock addr 7f1428006770
[08/Jan/2018:11:16:07 +0100] - 389-Directory/1.3.4.9
B2016.109.158
starting up
[08/Jan/2018:11:16:07 +0100] - Detected Disorderly Shutdown last time
Directory Server was running, recovering database.
[08/Jan/2018:11:16:08 +0100] - slapd started. Listening on All
Interfaces port 389 for LDAP requests
[08/Jan/2018:11:24:53 +0100] - 389-Directory/1.3.4.9
B2016.109.158
starting up
[08/Jan/2018:11:24:53 +0100] - Detected Disorderly Shutdown last time
Directory Server was running, recovering database.
[08/Jan/2018:11:24:53 +0100] - slapd started. Listening on All
Interfaces port 389 for LDAP requests
Resetting aclpb_pblock 7f471dff2bd0 to pblock addr 7f46c4004900
[08/Jan/2018:14:39:09 +0100] - 389-Directory/1.3.4.9
B2016.109.158
starting up
[08/Jan/2018:14:39:09 +0100] - Detected Disorderly Shutdown last time
Directory Server was running, recovering database.
[08/Jan/2018:14:39:09 +0100] - slapd started. Listening on All
Interfaces port 389 for LDAP requests
Resetting aclpb_pblock 7f0fdbffebd0 to pblock addr 7f0f5c00bc80
Resetting aclpb_pblock 7f0fdb7fdbd0 to pblock addr 7f0fac008480
Well... "Disorderly Shutdown" does not sound good at all.
_______________________________________________
users mailing list
https://lists.kolab.org/mailman/listinfo/users

Marcel Bischoff

2018-01-10 14:24:37 UTC

Permalink

Post by Skale, Franz
06:21 is the lograotation, so no problem. Same by me.

A bit reassuring but still: shouldn't the service cleanly restart
instead of barfing "Disorderly Shutdown"?

Post by Skale, Franz
What strucks me is, that it seems that ns-slapd as to reallocate memory.
How much memory does your server have ?
send free -m

$ free -m
total used free shared buff/cache available
Mem: 16045 1304 13096 56 1644 14371
Swap: 0 0 0

No problem I can see there.

Post by Skale, Franz
Do you have selinux enabled !
If so, disable it by adding selinux=0 to /etc/default/grub and rerun
update-grub.

No, not enabled.

Post by Skale, Franz
Send the kernel version: uname -a

Linux mx.example.com 4.10.0-42-generic #46~16.04.1-Ubuntu SMP Mon Dec 4
15:57:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Post by Skale, Franz
How much open file handles to your system allow per process ?
send: ulimit -a

$ ulimit -a
Maximum size of core files created (kB, -c) 0
Maximum size of a process’s data segment (kB, -d) unlimited
Maximum size of files created by the shell (kB, -f) unlimited
Maximum size that may be locked into memory (kB, -l) 64
Maximum resident set size (kB, -m) unlimited
Maximum number of open file descriptors (-n) 1024
Maximum stack size (kB, -s) 8192
Maximum amount of cpu time in seconds (seconds, -t) unlimited
Maximum number of processes available to a single user (-u) 64015
Maximum amount of virtual memory available to the shell (kB, -v) unlimited

Post by Skale, Franz
send dmesg: (is there a segfault).

The whole dmesg output is spammed by ufw and contains no useful
information whatsoever.

Post by Skale, Franz
send dmidecode

I don't think this is likely as this is a virtual server.

$ dmidecode
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.
10 structures occupying 408 bytes.
Table at 0x000F68A0.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: SeaBIOS
Version: 1.10.2
Release Date: 04/01/2014
Address: 0xE8000
Runtime Size: 96 kB
ROM Size: 64 kB
Characteristics:
BIOS characteristics not supported
Targeted content distribution is supported
BIOS Revision: 0.0

Handle 0x0100, DMI type 1, 27 bytes
System Information
Manufacturer: Hetzner
Product Name: vServer
Version: 2
Serial Number: Not Specified
UUID: A8236400-D36B-0135-FE8F-10BF48D7F2C6
Wake-up Type: Power Switch
SKU Number: a8236400-d36b-0135-fe8f-10bf48d7f2c6
Family: Not Specified

Handle 0x0300, DMI type 3, 21 bytes
Chassis Information
Manufacturer: QEMU
Type: Other
Lock: Not Present
Version: pc-i440fx-2.10
Serial Number: Not Specified
Asset Tag: Not Specified
Boot-up State: Safe
Power Supply State: Safe
Thermal State: Safe
Security Status: Unknown
OEM Information: 0x00000000
Height: Unspecified
Number Of Power Cords: Unspecified
Contained Elements: 0

Handle 0x0400, DMI type 4, 42 bytes
Processor Information
Socket Designation: CPU 0
Type: Central Processor
Family: Other
Manufacturer: QEMU
ID: A1 06 02 00 FF FB 8B 07
Version: pc-i440fx-2.10
Voltage: Unknown
External Clock: Unknown
Max Speed: 2000 MHz
Current Speed: 2000 MHz
Status: Populated, Enabled
Upgrade: Other
L1 Cache Handle: Not Provided
L2 Cache Handle: Not Provided
L3 Cache Handle: Not Provided
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Core Count: 4
Core Enabled: 4
Thread Count: 1
Characteristics: None

Handle 0x1000, DMI type 16, 23 bytes
Physical Memory Array
Location: Other
Use: System Memory
Error Correction Type: Multi-bit ECC
Maximum Capacity: 16 GB
Error Information Handle: Not Provided
Number Of Devices: 1

Handle 0x1100, DMI type 17, 40 bytes
Memory Device
Array Handle: 0x1000
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: 16384 MB
Form Factor: DIMM
Set: None
Locator: DIMM 0
Bank Locator: Not Specified
Type: RAM
Type Detail: Other
Speed: Unknown
Manufacturer: QEMU
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Rank: Unknown
Configured Clock Speed: Unknown
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown

Handle 0x1300, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00000000000
Ending Address: 0x000BFFFFFFF
Range Size: 3 GB
Physical Array Handle: 0x1000
Partition Width: 1

Handle 0x1301, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00100000000
Ending Address: 0x0043FFFFFFF
Range Size: 13 GB
Physical Array Handle: 0x1000
Partition Width: 1

Handle 0x2000, DMI type 32, 11 bytes
System Boot Information
Status: No errors detected

Handle 0x7F00, DMI type 127, 4 bytes
End Of Table

Post by Skale, Franz
Did you update your kernel days ago, if so, you sure ran into a buggy
kernel 4.9.65.
I built a 4.9.75 PTI enabled kernel which i send you to test.

4.10.0-42-generic

Post by Skale, Franz
Office 365 would be a bad and expensive choice.

I agree. I'd very much like to avoid it but when Kolab turn out to have
issues with the current setup, I doubt I'll get another shot.

Best,
Marcel

Marcel Bischoff

2018-01-10 15:50:10 UTC

Permalink

I have been able to pull the kernel messages from the logwatch output,
didn't think of this before. Maybe it helps in homing in on the cause. I
couldn't spot something obvious though. Maybe AppArmor does something
undesirable? I remember running into issues with it years ago in another
context.

I will uninstall it tonight, reboot the server and report on my progress
(or lack thereof).

Thanks again for bearing with me!

--------------------- Kernel Begin ------------------------

1 Time(s): #2
1 Time(s): #3
1 Time(s): 1 disabled
1 Time(s): 2 disabled
1 Time(s): 3 disabled
1 Time(s): 4 disabled
1 Time(s): 5 disabled
1 Time(s): 6 disabled
1 Time(s): 7 disabled
1 Time(s): ACPI: 1 ACPI AML tables successfully acquired and loaded
1 Time(s): ACPI: Added _OSI(Module Device)
1 Time(s): ACPI: Added _OSI(Processor Aggregator Device)
1 Time(s): ACPI: Added _OSI(Processor Device)
1 Time(s): ACPI: Early table checksum verification disabled
1 Time(s): ACPI: IRQ11 used by override.
1 Time(s): ACPI: IRQ5 used by override.
1 Time(s): ACPI: IRQ9 used by override.
1 Time(s): ACPI: Interpreter enabled
1 Time(s): ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
1 Time(s): ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
1 Time(s): ACPI: Power Button [PWRF]
1 Time(s): ACPI: Using IOAPIC for interrupt routing
1 Time(s): ACPI: bus type PCI registered
1 Time(s): ACPI: bus type USB registered
1 Time(s): AES CTR mode by8 optimization enabled
1 Time(s): AMD AuthenticAMD
1 Time(s): AVX version of gcm_enc/dec engaged.
1 Time(s): AppArmor: AppArmor Filesystem Enabled
1 Time(s): AppArmor: AppArmor initialized
1 Time(s): AppArmor: AppArmor sha1 policy hashing enabled
1 Time(s): Booting paravirtualized kernel on KVM
1 Time(s): Btrfs loaded, crc32c=crc32c-intel
1 Time(s): Build-time adjustment of leaf fanout to 64.
1 Time(s): Built 1 zonelists in Node order, mobility grouping on. Total pages: 4128613
1 Time(s): Calgary: Unable to locate Rio Grande table in EBDA - bailing!
1 Time(s): Calgary: detecting Calgary via BIOS EBDA area
1 Time(s): Calibrating delay loop (skipped) preset value.. 4199.99 BogoMIPS (lpj=8399992)
1 Time(s): Centaur CentaurHauls
1 Time(s): DMA zone: 21 pages reserved
1 Time(s): DMA zone: 64 pages used for memmap
1 Time(s): DMA32 zone: 12224 pages used for memmap
1 Time(s): Device empty
1 Time(s): EDD information not available.
1 Time(s): EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
1 Time(s): EXT4-fs (sda1): re-mounted. Opts: discard
1 Time(s): Early memory node ranges
1 Time(s): Freeing SMP alternatives memory: 32K
1 Time(s): Freeing unused kernel memory: 1156K
1 Time(s): Freeing unused kernel memory: 2228K
1 Time(s): Freeing unused kernel memory: 268K
1 Time(s): GHES: HEST is not enabled!
1 Time(s): Hierarchical RCU implementation.
1 Time(s): Hypervisor detected: KVM
1 Time(s): Initialise system trusted keyrings
1 Time(s): Intel GenuineIntel
1 Time(s): KERNEL supported cpus:
1 Time(s): KVM setup async PF for cpu 1
1 Time(s): KVM setup async PF for cpu 2
1 Time(s): KVM setup async PF for cpu 3
1 Time(s): Key type asymmetric registered
1 Time(s): Key type big_key registered
1 Time(s): Key type dns_resolver registered
1 Time(s): Key type encrypted registered
1 Time(s): Key type trusted registered
1 Time(s): MTRR default type: write-back
1 Time(s): MTRR fixed ranges enabled:
1 Time(s): MTRR variable ranges enabled:
1 Time(s): Magic number: 2:724:141
1 Time(s): Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
1 Time(s): Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes)
1 Time(s): Movable zone start for each node
1 Time(s): NET: Registered protocol family 1
1 Time(s): NET: Registered protocol family 16
1 Time(s): NET: Registered protocol family 17
1 Time(s): NET: Registered protocol family 2
1 Time(s): NR_IRQS:524544 nr_irqs:456 16
1 Time(s): NX (Execute Disable) protection: active
1 Time(s): NetLabel: domain hash size = 128
1 Time(s): NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
1 Time(s): NetLabel: unlabeled traffic allowed by default
1 Time(s): NetLabel: Initializing
1 Time(s): No NUMA configuration found
1 Time(s): Normal zone: 53248 pages used for memmap
1 Time(s): PCCT header not found.
1 Time(s): PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
1 Time(s): PCI: Using ACPI for IRQ routing
1 Time(s): PCI: Using configuration type 1 for base access
1 Time(s): PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
1 Time(s): PCI: pci_cache_line_size set to 64 bytes
1 Time(s): PM: Hibernation image not present or could not be loaded.
1 Time(s): PPP generic driver version 2.4.2
1 Time(s): Performance Events: unsupported p6 CPU model 42 no PMU driver, software events only.
1 Time(s): Policy zone: Normal
2 Time(s): Process accounting resumed
1 Time(s): RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4.
1 Time(s): RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=4
1 Time(s): SCSI subsystem initialized
1 Time(s): SMBIOS 2.8 present.
1 Time(s): Scanning 1 areas for low memory corruption
1 Time(s): Security Framework initialized
1 Time(s): Segment Routing with IPv6
1 Time(s): Switched APIC routing to physical x2apic.
1 Time(s): TSC deadline timer enabled
1 Time(s): UDP hash table entries: 8192 (order: 6, 262144 bytes)
1 Time(s): UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
1 Time(s): Unpacking initramfs...
1 Time(s): Using ACPI (MADT) for SMP configuration information
1 Time(s): Write protecting the kernel read-only data: 14336k
1 Time(s): Yama: becoming mindful.
1 Time(s): Zone ranges:
1 Time(s): acpi device:12: hash matches
1 Time(s): acpiphp: Slot [11] registered
1 Time(s): acpiphp: Slot [12] registered
1 Time(s): acpiphp: Slot [13] registered
1 Time(s): acpiphp: Slot [14] registered
1 Time(s): acpiphp: Slot [15] registered
1 Time(s): acpiphp: Slot [16] registered
1 Time(s): acpiphp: Slot [17] registered
1 Time(s): acpiphp: Slot [18] registered
1 Time(s): acpiphp: Slot [19] registered
1 Time(s): acpiphp: Slot [21] registered
1 Time(s): acpiphp: Slot [22] registered
1 Time(s): acpiphp: Slot [23] registered
1 Time(s): acpiphp: Slot [24] registered
1 Time(s): acpiphp: Slot [25] registered
1 Time(s): acpiphp: Slot [26] registered
1 Time(s): acpiphp: Slot [27] registered
1 Time(s): acpiphp: Slot [28] registered
1 Time(s): acpiphp: Slot [29] registered
1 Time(s): acpiphp: Slot [31] registered
1 Time(s): acpiphp: Slot [3] registered
1 Time(s): acpiphp: Slot [4] registered
1 Time(s): acpiphp: Slot [5] registered
1 Time(s): acpiphp: Slot [6] registered
1 Time(s): acpiphp: Slot [7] registered
1 Time(s): acpiphp: Slot [8] registered
1 Time(s): acpiphp: Slot [9] registered
1 Time(s): async_tx: api initialized (async)
1 Time(s): audit: initializing netlink subsys (disabled)
1 Time(s): clocksource: Switched to clocksource kvm-clock
1 Time(s): cpuidle: using governor ladder
1 Time(s): cpuidle: using governor menu
1 Time(s): devtmpfs: initialized
1 Time(s): ehci-pci: EHCI PCI platform driver
1 Time(s): ehci-platform: EHCI generic platform driver
1 Time(s): evm: security.SMACK64
1 Time(s): evm: security.SMACK64EXEC
1 Time(s): evm: security.SMACK64MMAP
1 Time(s): evm: security.SMACK64TRANSMUTE
1 Time(s): evm: security.capability
1 Time(s): evm: security.ima
1 Time(s): evm: security.selinux
1 Time(s): ftrace: allocating 34227 entries in 134 pages
1 Time(s): fuse init (API version 7.26)
1 Time(s): hidraw: raw HID events driver (C) Jiri Kosina
1 Time(s): hpet clockevent registered
1 Time(s): i2c /dev entries driver
1 Time(s): ima: No TPM chip found, activating TPM-bypass! (rc=-19)
1 Time(s): intel_idle: does not run on family 6 model 42
1 Time(s): io scheduler cfq registered
1 Time(s): io scheduler deadline registered
1 Time(s): io scheduler noop registered (default)
1 Time(s): ledtrig-cpu: registered to indicate activity on CPUs
1 Time(s): libphy: Fixed MDIO Bus: probed
1 Time(s): loop: module loaded
1 Time(s): mousedev: PS/2 mouse device common for all mice
1 Time(s): ohci-pci: OHCI PCI platform driver
1 Time(s): ohci-platform: OHCI generic platform driver
1 Time(s): ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
1 Time(s): pinctrl core: initialized pinctrl subsystem
1 Time(s): pnp: PnP ACPI init
1 Time(s): pnp: PnP ACPI: found 5 devices
1 Time(s): ppdev: user-space parallel port driver
1 Time(s): raid6: .... xor() 6424 MB/s, rmw enabled
1 Time(s): raid6: sse2x1 gen() 6723 MB/s
1 Time(s): raid6: sse2x1 xor() 5185 MB/s
1 Time(s): raid6: sse2x2 gen() 8399 MB/s
1 Time(s): raid6: sse2x2 xor() 5585 MB/s
1 Time(s): raid6: sse2x4 gen() 9942 MB/s
1 Time(s): raid6: sse2x4 xor() 6424 MB/s
1 Time(s): raid6: using algorithm sse2x4 gen() 9942 MB/s
1 Time(s): raid6: using ssse3x2 recovery algorithm
1 Time(s): random: crng init done
1 Time(s): random: fast init done
8 Time(s): random: systemd-udevd: uninitialized urandom read (16 bytes read)
2 Time(s): random: udevadm: uninitialized urandom read (16 bytes read)
1 Time(s): registered taskstats version 1
1 Time(s): scsi host1: ata_piix
1 Time(s): scsi host2: Virtio SCSI HBA
1 Time(s): sda: sda1
1 Time(s): setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1
1 Time(s): smp: Bringing up secondary CPUs ...
1 Time(s): smp: Brought up 1 node, 4 CPUs
1 Time(s): smpboot: Max logical packages: 1
1 Time(s): smpboot: Total of 4 processors activated (16799.98 BogoMIPS)
1 Time(s): tun: Universal TUN/TAP device driver, 1.6
1 Time(s): uhci_hcd: USB Universal Host Controller Interface driver
1 Time(s): usb 1-1: Manufacturer: QEMU
1 Time(s): usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=5
1 Time(s): usb 1-1: Product: QEMU USB Tablet
1 Time(s): usb 1-1: SerialNumber: 42
1 Time(s): usb 1-1: new full-speed USB device number 2 using uhci_hcd
1 Time(s): usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
1 Time(s): usb usb1: Product: UHCI Host Controller
1 Time(s): usbcore: registered new device driver usb
1 Time(s): usbcore: registered new interface driver hub
1 Time(s): usbcore: registered new interface driver usbfs
1 Time(s): usbcore: registered new interface driver usbhid
1 Time(s): usbhid: USB HID core driver
1 Time(s): vgaarb: loaded
1 Time(s): x2apic enabled
1 Time(s): x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
1 Time(s): x86/mm: Checked W+X mappings: passed, no W+X pages found.
1 Time(s): x86/mm: Memory block size: 128MB
1 Time(s): x86: Booting SMP configuration:
1 Time(s): xor: automatically using best checksumming function avx
1 Time(s): zbud: loaded
1 Time(s): zswap: loaded using pool lzo/zbud

---------------------- Kernel End -------------------------

Post by Marcel Bischoff

Post by Skale, Franz
06:21 is the lograotation, so no problem. Same by me.

A bit reassuring but still: shouldn't the service cleanly restart
instead of barfing "Disorderly Shutdown"?

Post by Skale, Franz
What strucks me is, that it seems that ns-slapd as to reallocate memory.
How much memory does your server have ?
send free -m

$ free -m
total used free shared buff/cache available
Mem: 16045 1304 13096 56 1644 14371
Swap: 0 0 0
No problem I can see there.

Post by Skale, Franz
Do you have selinux enabled !
If so, disable it by adding selinux=0 to /etc/default/grub and rerun
update-grub.

No, not enabled.

Post by Skale, Franz
Send the kernel version: uname -a

Linux mx.example.com 4.10.0-42-generic #46~16.04.1-Ubuntu SMP Mon Dec 4
15:57:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Post by Skale, Franz
How much open file handles to your system allow per process ?
send: ulimit -a

Post by Skale, Franz
send dmesg: (is there a segfault).

The whole dmesg output is spammed by ufw and contains no useful
information whatsoever.

Post by Skale, Franz
send dmidecode

I don't think this is likely as this is a virtual server.
$ dmidecode
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.
10 structures occupying 408 bytes.
Table at 0x000F68A0.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: SeaBIOS
Version: 1.10.2
Release Date: 04/01/2014
Address: 0xE8000
Runtime Size: 96 kB
ROM Size: 64 kB
BIOS characteristics not supported
Targeted content distribution is supported
BIOS Revision: 0.0
Handle 0x0100, DMI type 1, 27 bytes
System Information
Manufacturer: Hetzner
Product Name: vServer
Version: 2
Serial Number: Not Specified
UUID: A8236400-D36B-0135-FE8F-10BF48D7F2C6
Wake-up Type: Power Switch
SKU Number: a8236400-d36b-0135-fe8f-10bf48d7f2c6
Family: Not Specified
Handle 0x0300, DMI type 3, 21 bytes
Chassis Information
Manufacturer: QEMU
Type: Other
Lock: Not Present
Version: pc-i440fx-2.10
Serial Number: Not Specified
Asset Tag: Not Specified
Boot-up State: Safe
Power Supply State: Safe
Thermal State: Safe
Security Status: Unknown
OEM Information: 0x00000000
Height: Unspecified
Number Of Power Cords: Unspecified
Contained Elements: 0
Handle 0x0400, DMI type 4, 42 bytes
Processor Information
Socket Designation: CPU 0
Type: Central Processor
Family: Other
Manufacturer: QEMU
ID: A1 06 02 00 FF FB 8B 07
Version: pc-i440fx-2.10
Voltage: Unknown
External Clock: Unknown
Max Speed: 2000 MHz
Current Speed: 2000 MHz
Status: Populated, Enabled
Upgrade: Other
L1 Cache Handle: Not Provided
L2 Cache Handle: Not Provided
L3 Cache Handle: Not Provided
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Core Count: 4
Core Enabled: 4
Thread Count: 1
Characteristics: None
Handle 0x1000, DMI type 16, 23 bytes
Physical Memory Array
Location: Other
Use: System Memory
Error Correction Type: Multi-bit ECC
Maximum Capacity: 16 GB
Error Information Handle: Not Provided
Number Of Devices: 1
Handle 0x1100, DMI type 17, 40 bytes
Memory Device
Array Handle: 0x1000
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: 16384 MB
Form Factor: DIMM
Set: None
Locator: DIMM 0
Bank Locator: Not Specified
Type: RAM
Type Detail: Other
Speed: Unknown
Manufacturer: QEMU
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Rank: Unknown
Configured Clock Speed: Unknown
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown
Handle 0x1300, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00000000000
Ending Address: 0x000BFFFFFFF
Range Size: 3 GB
Physical Array Handle: 0x1000
Partition Width: 1
Handle 0x1301, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00100000000
Ending Address: 0x0043FFFFFFF
Range Size: 13 GB
Physical Array Handle: 0x1000
Partition Width: 1
Handle 0x2000, DMI type 32, 11 bytes
System Boot Information
Status: No errors detected
Handle 0x7F00, DMI type 127, 4 bytes
End Of Table

Post by Skale, Franz
Did you update your kernel days ago, if so, you sure ran into a buggy
kernel 4.9.65.
I built a 4.9.75 PTI enabled kernel which i send you to test.

4.10.0-42-generic

Post by Skale, Franz
Office 365 would be a bad and expensive choice.

I agree. I'd very much like to avoid it but when Kolab turn out to have
issues with the current setup, I doubt I'll get another shot.
Best,
Marcel

Marcel Bischoff

2018-01-10 21:05:03 UTC

Permalink

Hi Franz,

so much for trying to remove AppArmor:

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
389-admin 389-admin-console 389-console 389-ds 389-ds-base 389-ds-base-libs 389-ds-console 389-ds-console-doc 389-dsgw amavisd-new aspell aspell-en augeas-lenses chwala
clamav clamav-base clamav-daemon clamav-freshclam clamdscan cyrus-imapd dictionaries-common emacsen-common erlang-base erlang-crypto erlang-eimap erlang-goldrush erlang-lager
erlang-lager-syslog erlang-syntax-tools fontconfig guam irony kolab-cli kolab-conf kolab-freebusy kolab-imap kolab-ldap kolab-mta kolab-saslauthd kolab-schema kolab-server
kolab-syncroton kolab-webadmin kolab-xml ldap-utils libadminutil-data libadminutil0 libapache2-mod-nss libapache2-mod-php libapache2-mod-php7.0 libapparmor-perl libaspell15
libaudio2 libaugeas0 libauthen-sasl-perl libavahi-client3 libavahi-common-data libavahi-common3 libberkeleydb-perl libcalendaring libcgi-fast-perl libcgi-pm-perl libclamav7
libconvert-binhex-perl libconvert-tnef-perl libconvert-uulib-perl libcrypt-openssl-bignum-perl libcrypt-openssl-rsa-perl libcups2 libdigest-hmac-perl libds-admin-serv0
libencode-locale-perl libevent-core-2.0-5 libfcgi-perl libgd3 libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl libhttp-date-perl libhttp-message-perl libical1a
libidm-console-framework-java libio-html-perl libio-multiplex-perl libio-socket-inet6-perl libio-socket-ssl-perl libio-stringy-perl libjansson4 libjbig0 libjpeg-turbo8
libjpeg8 libjss-java libkolab2 libkolabxml1v5 liblcms2-2 libldap-java libllvm3.6v5 liblwp-mediatypes-perl libmail-dkim-perl libmail-spf-perl libmailtools-perl libmcrypt4
libmime-tools-perl libmng2 libmozilla-ldap-perl libmozldap-0d libnet-cidr-perl libnet-dns-perl libnet-ip-perl libnet-libidn-perl libnet-server-perl libnet-smtp-ssl-perl
libnet-ssleay-perl libnetaddr-ip-perl libnss3-tools libperl4-corelibs-perl libqt4-dbus libqt4-declarative libqt4-network libqt4-script libqt4-sql libqt4-sql-mysql libqt4-xml
libqt4-xmlpatterns libqtcore4 libqtdbus4 libqtgui4 libsctp1 libsocket-getaddrinfo-perl libsocket6-perl libtiff5 libunix-syslog-perl liburi-perl libvpx3 libxerces-c3.1
libxslt1.1 libzend-framework-php libzephyr4 mozldap-tools mysql-client mysql-client-5.7 mysql-client-core-5.7 mysql-server-core-5.7 pax php php-auth-sasl php-cli php-common
php-curl php-gd php-http-request2 php-intl php-kolab php-kolabformat php-ldap php-mail php-mail-mime php-mail-mimedecode php-mbstring php-mcrypt php-mdb2
php-mdb2-driver-mysql php-monolog php-mysql php-net-idna2 php-net-ldap2 php-net-ldap3 php-net-sieve php-net-smtp php-net-socket php-net-url2 php-pear php-pspell php-psr-log
php-sabre-dav-2.1 php-sabre-event php-sabre-http-3 php-sabre-vobject-3 php-xml php7.0 php7.0-cli php7.0-common php7.0-curl php7.0-fpm php7.0-gd php7.0-intl php7.0-json
php7.0-ldap php7.0-mbstring php7.0-mcrypt php7.0-mysql php7.0-opcache php7.0-pspell php7.0-readline php7.0-xml pykolab python-augeas python-cheetah python-dateutil
python-gnupg python-icalendar python-kolab python-kolabformat python-ldap python-pkg-resources python-pyasn1 python-pyasn1-modules python-pymysql python-six python-sqlalchemy
python-sqlalchemy-ext python-tz python-tzlocal qdbus qt-at-spi qtchooser qtcore4-l10n re2c roundcubemail roundcubemail-core roundcubemail-plugin-acl
roundcubemail-plugin-archive roundcubemail-plugin-calendar roundcubemail-plugin-contextmenu roundcubemail-plugin-filesystem-attachments roundcubemail-plugin-jqueryui
roundcubemail-plugin-kolab-activesync roundcubemail-plugin-kolab-addressbook roundcubemail-plugin-kolab-auth roundcubemail-plugin-kolab-config
roundcubemail-plugin-kolab-delegation roundcubemail-plugin-kolab-files roundcubemail-plugin-kolab-folders roundcubemail-plugin-kolab-notes roundcubemail-plugin-kolab-tags
roundcubemail-plugin-libcalendaring roundcubemail-plugin-libkolab roundcubemail-plugin-managesieve roundcubemail-plugin-newmail-notifier roundcubemail-plugin-odfviewer
roundcubemail-plugin-password roundcubemail-plugin-pdfviewer roundcubemail-plugin-redundant-attachments roundcubemail-plugin-tasklist roundcubemail-plugin-zipdownload
roundcubemail-plugins-kolab roundcubemail-skin-chameleon sa-compile smarty3 spamassassin spamc wallace zend-framework zend-framework-bin
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
apparmor kolab kolab-webclient mysql-server mysql-server-5.7
0 upgraded, 0 newly installed, 5 to remove and 0 not upgraded.
After this operation, 50.2 MB disk space will be freed.
Do you want to continue? [Y/n]

Which amounts to basically... everything.

At least apparmor_status lets me know:

apparmor module is loaded.
0 profiles are loaded.
0 profiles are in enforce mode.
0 profiles are in complain mode.
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

So that should be alright I guess.

I have raised the limits and will monitor the outcome. Hopefully things
will quiet down now.

Best,
Marcel

Post by Skale, Franz
Hi Marcel,
I would deactivate apparmor.
Also i find that your ulimit -a output is wrong.
You should raise the open files to 65k depending how much users you
have.
1024 is too small i guess.
http://directory.fedoraproject.org/docs/389ds/FAQ/performance-tuning.html#linux
Since you've a KVM guest, do you use PV or HVM ?
Did you install a PTI kernel on the host ?
If so, you should use a PTI kernel in the guest too.
Jan 7 18:10:46 localhost kernel: [ 0.000000] Kernel/User page
tables isolation: enabled e.g (using 4.9.75).
Just a thought.
Rgds.
Franz

Post by Marcel Bischoff
I have been able to pull the kernel messages from the logwatch output,
didn't think of this before. Maybe it helps in homing in on the cause. I
couldn't spot something obvious though. Maybe AppArmor does something
undesirable? I remember running into issues with it years ago in another
context.
I will uninstall it tonight, reboot the server and report on my progress
(or lack thereof).
Thanks again for bearing with me!
--------------------- Kernel Begin ------------------------
1 Time(s): #2
1 Time(s): #3
1 Time(s): 1 disabled
1 Time(s): 2 disabled
1 Time(s): 3 disabled
1 Time(s): 4 disabled
1 Time(s): 5 disabled
1 Time(s): 6 disabled
1 Time(s): 7 disabled
1 Time(s): ACPI: 1 ACPI AML tables successfully acquired and loaded
1 Time(s): ACPI: Added _OSI(Module Device)
1 Time(s): ACPI: Added _OSI(Processor Aggregator Device)
1 Time(s): ACPI: Added _OSI(Processor Device)
1 Time(s): ACPI: Early table checksum verification disabled
1 Time(s): ACPI: IRQ11 used by override.
1 Time(s): ACPI: IRQ5 used by override.
1 Time(s): ACPI: IRQ9 used by override.
1 Time(s): ACPI: Interpreter enabled
1 Time(s): ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
1 Time(s): ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
1 Time(s): ACPI: Power Button [PWRF]
1 Time(s): ACPI: Using IOAPIC for interrupt routing
1 Time(s): ACPI: bus type PCI registered
1 Time(s): ACPI: bus type USB registered
1 Time(s): AES CTR mode by8 optimization enabled
1 Time(s): AMD AuthenticAMD
1 Time(s): AVX version of gcm_enc/dec engaged.
1 Time(s): AppArmor: AppArmor Filesystem Enabled
1 Time(s): AppArmor: AppArmor initialized
1 Time(s): AppArmor: AppArmor sha1 policy hashing enabled
1 Time(s): Booting paravirtualized kernel on KVM
1 Time(s): Btrfs loaded, crc32c=crc32c-intel
1 Time(s): Build-time adjustment of leaf fanout to 64.
1 Time(s): Built 1 zonelists in Node order, mobility grouping on.
Total pages: 4128613
1 Time(s): Calgary: Unable to locate Rio Grande table in EBDA - bailing!
1 Time(s): Calgary: detecting Calgary via BIOS EBDA area
1 Time(s): Calibrating delay loop (skipped) preset value.. 4199.99 BogoMIPS (lpj=8399992)
1 Time(s): Centaur CentaurHauls
1 Time(s): DMA zone: 21 pages reserved
1 Time(s): DMA zone: 64 pages used for memmap
1 Time(s): DMA32 zone: 12224 pages used for memmap
1 Time(s): Device empty
1 Time(s): EDD information not available.
1 Time(s): EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
1 Time(s): EXT4-fs (sda1): re-mounted. Opts: discard
1 Time(s): Early memory node ranges
1 Time(s): Freeing SMP alternatives memory: 32K
1 Time(s): Freeing unused kernel memory: 1156K
1 Time(s): Freeing unused kernel memory: 2228K
1 Time(s): Freeing unused kernel memory: 268K
1 Time(s): GHES: HEST is not enabled!
1 Time(s): Hierarchical RCU implementation.
1 Time(s): Hypervisor detected: KVM
1 Time(s): Initialise system trusted keyrings
1 Time(s): Intel GenuineIntel
1 Time(s): KVM setup async PF for cpu 1
1 Time(s): KVM setup async PF for cpu 2
1 Time(s): KVM setup async PF for cpu 3
1 Time(s): Key type asymmetric registered
1 Time(s): Key type big_key registered
1 Time(s): Key type dns_resolver registered
1 Time(s): Key type encrypted registered
1 Time(s): Key type trusted registered
1 Time(s): MTRR default type: write-back
1 Time(s): Magic number: 2:724:141
1 Time(s): Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
1 Time(s): Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes)
1 Time(s): Movable zone start for each node
1 Time(s): NET: Registered protocol family 1
1 Time(s): NET: Registered protocol family 16
1 Time(s): NET: Registered protocol family 17
1 Time(s): NET: Registered protocol family 2
1 Time(s): NR_IRQS:524544 nr_irqs:456 16
1 Time(s): NX (Execute Disable) protection: active
1 Time(s): NetLabel: domain hash size = 128
1 Time(s): NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
1 Time(s): NetLabel: unlabeled traffic allowed by default
1 Time(s): NetLabel: Initializing
1 Time(s): No NUMA configuration found
1 Time(s): Normal zone: 53248 pages used for memmap
1 Time(s): PCCT header not found.
1 Time(s): PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
1 Time(s): PCI: Using ACPI for IRQ routing
1 Time(s): PCI: Using configuration type 1 for base access
1 Time(s): PCI: Using host bridge windows from ACPI; if necessary, use
"pci=nocrs" and report a bug
1 Time(s): PCI: pci_cache_line_size set to 64 bytes
1 Time(s): PM: Hibernation image not present or could not be loaded.
1 Time(s): PPP generic driver version 2.4.2
1 Time(s): Performance Events: unsupported p6 CPU model 42 no PMU
driver, software events only.
1 Time(s): Policy zone: Normal
2 Time(s): Process accounting resumed
1 Time(s): RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4.
1 Time(s): RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=4
1 Time(s): SCSI subsystem initialized
1 Time(s): SMBIOS 2.8 present.
1 Time(s): Scanning 1 areas for low memory corruption
1 Time(s): Security Framework initialized
1 Time(s): Segment Routing with IPv6
1 Time(s): Switched APIC routing to physical x2apic.
1 Time(s): TSC deadline timer enabled
1 Time(s): UDP hash table entries: 8192 (order: 6, 262144 bytes)
1 Time(s): UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
1 Time(s): Unpacking initramfs...
1 Time(s): Using ACPI (MADT) for SMP configuration information
1 Time(s): Write protecting the kernel read-only data: 14336k
1 Time(s): Yama: becoming mindful.
1 Time(s): acpi device:12: hash matches
1 Time(s): acpiphp: Slot [11] registered
1 Time(s): acpiphp: Slot [12] registered
1 Time(s): acpiphp: Slot [13] registered
1 Time(s): acpiphp: Slot [14] registered
1 Time(s): acpiphp: Slot [15] registered
1 Time(s): acpiphp: Slot [16] registered
1 Time(s): acpiphp: Slot [17] registered
1 Time(s): acpiphp: Slot [18] registered
1 Time(s): acpiphp: Slot [19] registered
1 Time(s): acpiphp: Slot [21] registered
1 Time(s): acpiphp: Slot [22] registered
1 Time(s): acpiphp: Slot [23] registered
1 Time(s): acpiphp: Slot [24] registered
1 Time(s): acpiphp: Slot [25] registered
1 Time(s): acpiphp: Slot [26] registered
1 Time(s): acpiphp: Slot [27] registered
1 Time(s): acpiphp: Slot [28] registered
1 Time(s): acpiphp: Slot [29] registered
1 Time(s): acpiphp: Slot [31] registered
1 Time(s): acpiphp: Slot [3] registered
1 Time(s): acpiphp: Slot [4] registered
1 Time(s): acpiphp: Slot [5] registered
1 Time(s): acpiphp: Slot [6] registered
1 Time(s): acpiphp: Slot [7] registered
1 Time(s): acpiphp: Slot [8] registered
1 Time(s): acpiphp: Slot [9] registered
1 Time(s): async_tx: api initialized (async)
1 Time(s): audit: initializing netlink subsys (disabled)
1 Time(s): clocksource: Switched to clocksource kvm-clock
1 Time(s): cpuidle: using governor ladder
1 Time(s): cpuidle: using governor menu
1 Time(s): devtmpfs: initialized
1 Time(s): ehci-pci: EHCI PCI platform driver
1 Time(s): ehci-platform: EHCI generic platform driver
1 Time(s): evm: security.SMACK64
1 Time(s): evm: security.SMACK64EXEC
1 Time(s): evm: security.SMACK64MMAP
1 Time(s): evm: security.SMACK64TRANSMUTE
1 Time(s): evm: security.capability
1 Time(s): evm: security.ima
1 Time(s): evm: security.selinux
1 Time(s): ftrace: allocating 34227 entries in 134 pages
1 Time(s): fuse init (API version 7.26)
1 Time(s): hidraw: raw HID events driver (C) Jiri Kosina
1 Time(s): hpet clockevent registered
1 Time(s): i2c /dev entries driver
1 Time(s): ima: No TPM chip found, activating TPM-bypass! (rc=-19)
1 Time(s): intel_idle: does not run on family 6 model 42
1 Time(s): io scheduler cfq registered
1 Time(s): io scheduler deadline registered
1 Time(s): io scheduler noop registered (default)
1 Time(s): ledtrig-cpu: registered to indicate activity on CPUs
1 Time(s): libphy: Fixed MDIO Bus: probed
1 Time(s): loop: module loaded
1 Time(s): mousedev: PS/2 mouse device common for all mice
1 Time(s): ohci-pci: OHCI PCI platform driver
1 Time(s): ohci-platform: OHCI generic platform driver
1 Time(s): ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
1 Time(s): pinctrl core: initialized pinctrl subsystem
1 Time(s): pnp: PnP ACPI init
1 Time(s): pnp: PnP ACPI: found 5 devices
1 Time(s): ppdev: user-space parallel port driver
1 Time(s): raid6: .... xor() 6424 MB/s, rmw enabled
1 Time(s): raid6: sse2x1 gen() 6723 MB/s
1 Time(s): raid6: sse2x1 xor() 5185 MB/s
1 Time(s): raid6: sse2x2 gen() 8399 MB/s
1 Time(s): raid6: sse2x2 xor() 5585 MB/s
1 Time(s): raid6: sse2x4 gen() 9942 MB/s
1 Time(s): raid6: sse2x4 xor() 6424 MB/s
1 Time(s): raid6: using algorithm sse2x4 gen() 9942 MB/s
1 Time(s): raid6: using ssse3x2 recovery algorithm
1 Time(s): random: crng init done
1 Time(s): random: fast init done
8 Time(s): random: systemd-udevd: uninitialized urandom read (16 bytes read)
2 Time(s): random: udevadm: uninitialized urandom read (16 bytes read)
1 Time(s): registered taskstats version 1
1 Time(s): scsi host1: ata_piix
1 Time(s): scsi host2: Virtio SCSI HBA
1 Time(s): sda: sda1
1 Time(s): setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1
1 Time(s): smp: Bringing up secondary CPUs ...
1 Time(s): smp: Brought up 1 node, 4 CPUs
1 Time(s): smpboot: Max logical packages: 1
1 Time(s): smpboot: Total of 4 processors activated (16799.98 BogoMIPS)
1 Time(s): tun: Universal TUN/TAP device driver, 1.6
1 Time(s): uhci_hcd: USB Universal Host Controller Interface driver
1 Time(s): usb 1-1: Manufacturer: QEMU
1 Time(s): usb 1-1: New USB device strings: Mfr=1, Product=3,
SerialNumber=5
1 Time(s): usb 1-1: Product: QEMU USB Tablet
1 Time(s): usb 1-1: SerialNumber: 42
1 Time(s): usb 1-1: new full-speed USB device number 2 using uhci_hcd
1 Time(s): usb usb1: New USB device strings: Mfr=3, Product=2,
SerialNumber=1
1 Time(s): usb usb1: Product: UHCI Host Controller
1 Time(s): usbcore: registered new device driver usb
1 Time(s): usbcore: registered new interface driver hub
1 Time(s): usbcore: registered new interface driver usbfs
1 Time(s): usbcore: registered new interface driver usbhid
1 Time(s): usbhid: USB HID core driver
1 Time(s): vgaarb: loaded
1 Time(s): x2apic enabled
1 Time(s): x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
1 Time(s): x86/mm: Checked W+X mappings: passed, no W+X pages found.
1 Time(s): x86/mm: Memory block size: 128MB
1 Time(s): xor: automatically using best checksumming function avx
1 Time(s): zbud: loaded
1 Time(s): zswap: loaded using pool lzo/zbud
---------------------- Kernel End -------------------------

Post by Marcel Bischoff

Post by Skale, Franz
06:21 is the lograotation, so no problem. Same by me.

A bit reassuring but still: shouldn't the service cleanly restart
instead of barfing "Disorderly Shutdown"?

Post by Skale, Franz
What strucks me is, that it seems that ns-slapd as to reallocate memory.
How much memory does your server have ?
send free -m

$ free -m
total used free shared buff/cache
available
Mem: 16045 1304 13096 56
1644 14371
Swap: 0 0 0
No problem I can see there.

Post by Skale, Franz
Do you have selinux enabled !
If so, disable it by adding selinux=0 to /etc/default/grub and rerun
update-grub.

No, not enabled.

Post by Skale, Franz
Send the kernel version: uname -a

Linux mx.example.com 4.10.0-42-generic #46~16.04.1-Ubuntu SMP Mon Dec 4
15:57:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Post by Skale, Franz
How much open file handles to your system allow per process ?
send: ulimit -a

$ ulimit -a
Maximum size of core files created (kB, -c) 0
Maximum size of a process’s data segment (kB, -d) unlimited
Maximum size of files created by the shell (kB, -f) unlimited
Maximum size that may be locked into memory (kB, -l) 64
Maximum resident set size (kB, -m) unlimited
Maximum number of open file descriptors
(-n) 1024
Maximum stack size (kB, -s) 8192
Maximum amount of cpu time in seconds (seconds, -t) unlimited
Maximum number of processes available to a single user
(-u) 64015
Maximum amount of virtual memory available to the shell (kB, -v) unlimited

Post by Skale, Franz
send dmesg: (is there a segfault).

The whole dmesg output is spammed by ufw and contains no useful
information whatsoever.

Post by Skale, Franz
send dmidecode

I don't think this is likely as this is a virtual server.
$ dmidecode
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.
10 structures occupying 408 bytes.
Table at 0x000F68A0.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: SeaBIOS
Version: 1.10.2
Release Date: 04/01/2014
Address: 0xE8000
Runtime Size: 96 kB
ROM Size: 64 kB
BIOS characteristics not supported
Targeted content distribution is supported
BIOS Revision: 0.0
Handle 0x0100, DMI type 1, 27 bytes
System Information
Manufacturer: Hetzner
Product Name: vServer
Version: 2
Serial Number: Not Specified
UUID: A8236400-D36B-0135-FE8F-10BF48D7F2C6
Wake-up Type: Power Switch
SKU Number: a8236400-d36b-0135-fe8f-10bf48d7f2c6
Family: Not Specified
Handle 0x0300, DMI type 3, 21 bytes
Chassis Information
Manufacturer: QEMU
Type: Other
Lock: Not Present
Version: pc-i440fx-2.10
Serial Number: Not Specified
Asset Tag: Not Specified
Boot-up State: Safe
Power Supply State: Safe
Thermal State: Safe
Security Status: Unknown
OEM Information: 0x00000000
Height: Unspecified
Number Of Power Cords: Unspecified
Contained Elements: 0
Handle 0x0400, DMI type 4, 42 bytes
Processor Information
Socket Designation: CPU 0
Type: Central Processor
Family: Other
Manufacturer: QEMU
ID: A1 06 02 00 FF FB 8B 07
Version: pc-i440fx-2.10
Voltage: Unknown
External Clock: Unknown
Max Speed: 2000 MHz
Current Speed: 2000 MHz
Status: Populated, Enabled
Upgrade: Other
L1 Cache Handle: Not Provided
L2 Cache Handle: Not Provided
L3 Cache Handle: Not Provided
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Core Count: 4
Core Enabled: 4
Thread Count: 1
Characteristics: None
Handle 0x1000, DMI type 16, 23 bytes
Physical Memory Array
Location: Other
Use: System Memory
Error Correction Type: Multi-bit ECC
Maximum Capacity: 16 GB
Error Information Handle: Not Provided
Number Of Devices: 1
Handle 0x1100, DMI type 17, 40 bytes
Memory Device
Array Handle: 0x1000
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: 16384 MB
Form Factor: DIMM
Set: None
Locator: DIMM 0
Bank Locator: Not Specified
Type: RAM
Type Detail: Other
Speed: Unknown
Manufacturer: QEMU
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Rank: Unknown
Configured Clock Speed: Unknown
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown
Handle 0x1300, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00000000000
Ending Address: 0x000BFFFFFFF
Range Size: 3 GB
Physical Array Handle: 0x1000
Partition Width: 1
Handle 0x1301, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00100000000
Ending Address: 0x0043FFFFFFF
Range Size: 13 GB
Physical Array Handle: 0x1000
Partition Width: 1
Handle 0x2000, DMI type 32, 11 bytes
System Boot Information
Status: No errors detected
Handle 0x7F00, DMI type 127, 4 bytes
End Of Table

Post by Skale, Franz
Did you update your kernel days ago, if so, you sure ran into a buggy
kernel 4.9.65.
I built a 4.9.75 PTI enabled kernel which i send you to test.

4.10.0-42-generic

Post by Skale, Franz
Office 365 would be a bad and expensive choice.

I agree. I'd very much like to avoid it but when Kolab turn out to have
issues with the current setup, I doubt I'll get another shot.
Best,
Marcel

Skale, Franz

2018-01-11 07:13:35 UTC

Permalink

Hi Marcel,
i now checked my ldap server and also found some messages regarding the
ACL Plugin:
Jan 10 22:32:54 localhost ns-slapd[98615]:
[10/Jan/2018:22:32:54.579885565 +0100] connection - conn=2940 fd=284
Attempt to release connection that is not acquire
Jan 10 22:32:54 localhost ns-slapd[98615]:
[10/Jan/2018:22:32:54.585434274 +0100] connection - conn=4470 fd=188
Attempt to release connection that is not acquire
Jan 10 23:50:07 localhost ns-slapd[98615]:
[10/Jan/2018:23:50:07.863504579 +0100] NSACLPlugin - acl_access_allowed:
Resetting aclpb_pblock 0x7fa3a3fe6a60 to pblo
Jan 11 00:07:34 localhost ns-slapd[98615]:
[11/Jan/2018:00:07:34.557315481 +0100] NSACLPlugin - acl_access_allowed:
Resetting aclpb_pblock 0x7fa3a9ff2a60 to pblo
Jan 11 03:12:32 localhost ns-slapd[98615]:
[11/Jan/2018:03:12:32.383353274 +0100] NSACLPlugin - acl_access_allowed:
Resetting aclpb_pblock 0x7fa3aeffca60 to pblo
Jan 11 06:47:56 localhost ns-slapd[98615]:
[11/Jan/2018:06:47:56.284946442 +0100] NSACLPlugin - acl_access_allowed:
Resetting aclpb_pblock 0x7fa3a7feea60 to pblo
Jan 11 06:48:00 localhost ns-slapd[98615]:
[11/Jan/2018:06:48:00.364639406 +0100] NSACLPlugin - acl_access_allowed:
Resetting aclpb_pblock 0x7fa3ae7fba60 to pblo
Jan 11 07:53:34 localhost ns-slapd[98615]:
[11/Jan/2018:07:53:34.001520691 +0100] NSACLPlugin - acl_access_allowed:
Resetting aclpb_pblock 0x7fa3a17e1a60 to pblo
Jan 11 07:58:34 localhost ns-slapd[98615]:
[11/Jan/2018:07:58:34.641383768 +0100] connection - conn=3240 fd=278
Attempt to release connection that is not acquire
Jan 11 07:58:34 localhost ns-slapd[98615]:
[11/Jan/2018:07:58:34.646688065 +0100] connection - conn=0 fd=0 Attempt
to release connection that is not acquired
But i have no unordered shutdown mentioned anywhere in the logs.
I also found out, that rasing the filelimit level doesn't work for the
dirsrv service.
Instead i raised it in the defaults config files:
/etc/default/dirsrv:
ulimit -n 65535
/etc/default/***@mailserver:
ulimit -n 65535
/etc/default/dirsrv.systemd:
# uncomment this line to raise the file descriptor limit
LimitNOFILE=65535
If your dirsrv shutdown unordered i personall think you have a problem
with either the host kernel or the guest kernel.
dirsrv is multithreaded and creates a thread for every connection
leaving alone filepointers it opens on demand.
So even on my testserver with no user created it consumes 200
filepointers after starting.

Rgds.
Franz

Post by Marcel Bischoff
Hi Franz,
Reading package lists... Done
Building dependency tree
Reading state information... Done
389-admin 389-admin-console 389-console 389-ds 389-ds-base
389-ds-base-libs 389-ds-console 389-ds-console-doc 389-dsgw
amavisd-new aspell aspell-en augeas-lenses chwala
clamav clamav-base clamav-daemon clamav-freshclam clamdscan
cyrus-imapd dictionaries-common emacsen-common erlang-base
erlang-crypto erlang-eimap erlang-goldrush erlang-lager
erlang-lager-syslog erlang-syntax-tools fontconfig guam irony
kolab-cli kolab-conf kolab-freebusy kolab-imap kolab-ldap kolab-mta
kolab-saslauthd kolab-schema kolab-server
kolab-syncroton kolab-webadmin kolab-xml ldap-utils libadminutil-data
libadminutil0 libapache2-mod-nss libapache2-mod-php
libapache2-mod-php7.0 libapparmor-perl libaspell15
libaudio2 libaugeas0 libauthen-sasl-perl libavahi-client3
libavahi-common-data libavahi-common3 libberkeleydb-perl
libcalendaring libcgi-fast-perl libcgi-pm-perl libclamav7
libconvert-binhex-perl libconvert-tnef-perl libconvert-uulib-perl
libcrypt-openssl-bignum-perl libcrypt-openssl-rsa-perl libcups2
libdigest-hmac-perl libds-admin-serv0
libencode-locale-perl libevent-core-2.0-5 libfcgi-perl libgd3
libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl
libhttp-date-perl libhttp-message-perl libical1a
libidm-console-framework-java libio-html-perl libio-multiplex-perl
libio-socket-inet6-perl libio-socket-ssl-perl libio-stringy-perl
libjansson4 libjbig0 libjpeg-turbo8
libjpeg8 libjss-java libkolab2 libkolabxml1v5 liblcms2-2 libldap-java
libllvm3.6v5 liblwp-mediatypes-perl libmail-dkim-perl libmail-spf-perl
libmailtools-perl libmcrypt4
libmime-tools-perl libmng2 libmozilla-ldap-perl libmozldap-0d
libnet-cidr-perl libnet-dns-perl libnet-ip-perl libnet-libidn-perl
libnet-server-perl libnet-smtp-ssl-perl
libnet-ssleay-perl libnetaddr-ip-perl libnss3-tools
libperl4-corelibs-perl libqt4-dbus libqt4-declarative libqt4-network
libqt4-script libqt4-sql libqt4-sql-mysql libqt4-xml
libqt4-xmlpatterns libqtcore4 libqtdbus4 libqtgui4 libsctp1
libsocket-getaddrinfo-perl libsocket6-perl libtiff5
libunix-syslog-perl liburi-perl libvpx3 libxerces-c3.1
libxslt1.1 libzend-framework-php libzephyr4 mozldap-tools
mysql-client mysql-client-5.7 mysql-client-core-5.7
mysql-server-core-5.7 pax php php-auth-sasl php-cli php-common
php-curl php-gd php-http-request2 php-intl php-kolab php-kolabformat
php-ldap php-mail php-mail-mime php-mail-mimedecode php-mbstring
php-mcrypt php-mdb2
php-mdb2-driver-mysql php-monolog php-mysql php-net-idna2
php-net-ldap2 php-net-ldap3 php-net-sieve php-net-smtp php-net-socket
php-net-url2 php-pear php-pspell php-psr-log
php-sabre-dav-2.1 php-sabre-event php-sabre-http-3
php-sabre-vobject-3 php-xml php7.0 php7.0-cli php7.0-common
php7.0-curl php7.0-fpm php7.0-gd php7.0-intl php7.0-json
php7.0-ldap php7.0-mbstring php7.0-mcrypt php7.0-mysql php7.0-opcache
php7.0-pspell php7.0-readline php7.0-xml pykolab python-augeas
python-cheetah python-dateutil
python-gnupg python-icalendar python-kolab python-kolabformat
python-ldap python-pkg-resources python-pyasn1 python-pyasn1-modules
python-pymysql python-six python-sqlalchemy
python-sqlalchemy-ext python-tz python-tzlocal qdbus qt-at-spi
qtchooser qtcore4-l10n re2c roundcubemail roundcubemail-core
roundcubemail-plugin-acl
roundcubemail-plugin-archive roundcubemail-plugin-calendar
roundcubemail-plugin-contextmenu
roundcubemail-plugin-filesystem-attachments
roundcubemail-plugin-jqueryui
roundcubemail-plugin-kolab-activesync
roundcubemail-plugin-kolab-addressbook roundcubemail-plugin-kolab-auth
roundcubemail-plugin-kolab-config
roundcubemail-plugin-kolab-delegation
roundcubemail-plugin-kolab-files roundcubemail-plugin-kolab-folders
roundcubemail-plugin-kolab-notes roundcubemail-plugin-kolab-tags
roundcubemail-plugin-libcalendaring roundcubemail-plugin-libkolab
roundcubemail-plugin-managesieve roundcubemail-plugin-newmail-notifier
roundcubemail-plugin-odfviewer
roundcubemail-plugin-password roundcubemail-plugin-pdfviewer
roundcubemail-plugin-redundant-attachments
roundcubemail-plugin-tasklist roundcubemail-plugin-zipdownload
roundcubemail-plugins-kolab roundcubemail-skin-chameleon sa-compile
smarty3 spamassassin spamc wallace zend-framework zend-framework-bin
Use 'apt autoremove' to remove them.
apparmor kolab kolab-webclient mysql-server mysql-server-5.7
0 upgraded, 0 newly installed, 5 to remove and 0 not upgraded.
After this operation, 50.2 MB disk space will be freed.
Do you want to continue? [Y/n]
Which amounts to basically... everything.
apparmor module is loaded.
0 profiles are loaded.
0 profiles are in enforce mode.
0 profiles are in complain mode.
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
So that should be alright I guess.
I have raised the limits and will monitor the outcome. Hopefully things
will quiet down now.
Best,
Marcel

Post by Marcel Bischoff
I have been able to pull the kernel messages from the logwatch output,
didn't think of this before. Maybe it helps in homing in on the cause. I
couldn't spot something obvious though. Maybe AppArmor does something
undesirable? I remember running into issues with it years ago in another
context.
I will uninstall it tonight, reboot the server and report on my progress
(or lack thereof).
Thanks again for bearing with me!
--------------------- Kernel Begin ------------------------
1 Time(s): #2
1 Time(s): #3
1 Time(s): 1 disabled
1 Time(s): 2 disabled
1 Time(s): 3 disabled
1 Time(s): 4 disabled
1 Time(s): 5 disabled
1 Time(s): 6 disabled
1 Time(s): 7 disabled
1 Time(s): ACPI: 1 ACPI AML tables successfully acquired and loaded
1 Time(s): ACPI: Added _OSI(Module Device)
1 Time(s): ACPI: Added _OSI(Processor Aggregator Device)
1 Time(s): ACPI: Added _OSI(Processor Device)
1 Time(s): ACPI: Early table checksum verification disabled
1 Time(s): ACPI: IRQ11 used by override.
1 Time(s): ACPI: IRQ5 used by override.
1 Time(s): ACPI: IRQ9 used by override.
1 Time(s): ACPI: Interpreter enabled
1 Time(s): ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
1 Time(s): ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
1 Time(s): ACPI: Power Button [PWRF]
1 Time(s): ACPI: Using IOAPIC for interrupt routing
1 Time(s): ACPI: bus type PCI registered
1 Time(s): ACPI: bus type USB registered
1 Time(s): AES CTR mode by8 optimization enabled
1 Time(s): AMD AuthenticAMD
1 Time(s): AVX version of gcm_enc/dec engaged.
1 Time(s): AppArmor: AppArmor Filesystem Enabled
1 Time(s): AppArmor: AppArmor initialized
1 Time(s): AppArmor: AppArmor sha1 policy hashing enabled
1 Time(s): Booting paravirtualized kernel on KVM
1 Time(s): Btrfs loaded, crc32c=crc32c-intel
1 Time(s): Build-time adjustment of leaf fanout to 64.
1 Time(s): Built 1 zonelists in Node order, mobility grouping on.
Total pages: 4128613
1 Time(s): Calgary: Unable to locate Rio Grande table in EBDA - bailing!
1 Time(s): Calgary: detecting Calgary via BIOS EBDA area
1 Time(s): Calibrating delay loop (skipped) preset value.. 4199.99
BogoMIPS (lpj=8399992)
1 Time(s): Centaur CentaurHauls
1 Time(s): DMA zone: 21 pages reserved
1 Time(s): DMA zone: 64 pages used for memmap
1 Time(s): DMA32 zone: 12224 pages used for memmap
1 Time(s): Device empty
1 Time(s): EDD information not available.
1 Time(s): EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
1 Time(s): EXT4-fs (sda1): re-mounted. Opts: discard
1 Time(s): Early memory node ranges
1 Time(s): Freeing SMP alternatives memory: 32K
1 Time(s): Freeing unused kernel memory: 1156K
1 Time(s): Freeing unused kernel memory: 2228K
1 Time(s): Freeing unused kernel memory: 268K
1 Time(s): GHES: HEST is not enabled!
1 Time(s): Hierarchical RCU implementation.
1 Time(s): Hypervisor detected: KVM
1 Time(s): Initialise system trusted keyrings
1 Time(s): Intel GenuineIntel
1 Time(s): KVM setup async PF for cpu 1
1 Time(s): KVM setup async PF for cpu 2
1 Time(s): KVM setup async PF for cpu 3
1 Time(s): Key type asymmetric registered
1 Time(s): Key type big_key registered
1 Time(s): Key type dns_resolver registered
1 Time(s): Key type encrypted registered
1 Time(s): Key type trusted registered
1 Time(s): MTRR default type: write-back
1 Time(s): Magic number: 2:724:141
1 Time(s): Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
1 Time(s): Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes)
1 Time(s): Movable zone start for each node
1 Time(s): NET: Registered protocol family 1
1 Time(s): NET: Registered protocol family 16
1 Time(s): NET: Registered protocol family 17
1 Time(s): NET: Registered protocol family 2
1 Time(s): NR_IRQS:524544 nr_irqs:456 16
1 Time(s): NX (Execute Disable) protection: active
1 Time(s): NetLabel: domain hash size = 128
1 Time(s): NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
1 Time(s): NetLabel: unlabeled traffic allowed by default
1 Time(s): NetLabel: Initializing
1 Time(s): No NUMA configuration found
1 Time(s): Normal zone: 53248 pages used for memmap
1 Time(s): PCCT header not found.
1 Time(s): PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
1 Time(s): PCI: Using ACPI for IRQ routing
1 Time(s): PCI: Using configuration type 1 for base access
1 Time(s): PCI: Using host bridge windows from ACPI; if necessary, use
"pci=nocrs" and report a bug
1 Time(s): PCI: pci_cache_line_size set to 64 bytes
1 Time(s): PM: Hibernation image not present or could not be loaded.
1 Time(s): PPP generic driver version 2.4.2
1 Time(s): Performance Events: unsupported p6 CPU model 42 no PMU
driver, software events only.
1 Time(s): Policy zone: Normal
2 Time(s): Process accounting resumed
1 Time(s): RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4.
1 Time(s): RCU: Adjusting geometry for rcu_fanout_leaf=64,
nr_cpu_ids=4
1 Time(s): SCSI subsystem initialized
1 Time(s): SMBIOS 2.8 present.
1 Time(s): Scanning 1 areas for low memory corruption
1 Time(s): Security Framework initialized
1 Time(s): Segment Routing with IPv6
1 Time(s): Switched APIC routing to physical x2apic.
1 Time(s): TSC deadline timer enabled
1 Time(s): UDP hash table entries: 8192 (order: 6, 262144 bytes)
1 Time(s): UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
1 Time(s): Unpacking initramfs...
1 Time(s): Using ACPI (MADT) for SMP configuration information
1 Time(s): Write protecting the kernel read-only data: 14336k
1 Time(s): Yama: becoming mindful.
1 Time(s): acpi device:12: hash matches
1 Time(s): acpiphp: Slot [11] registered
1 Time(s): acpiphp: Slot [12] registered
1 Time(s): acpiphp: Slot [13] registered
1 Time(s): acpiphp: Slot [14] registered
1 Time(s): acpiphp: Slot [15] registered
1 Time(s): acpiphp: Slot [16] registered
1 Time(s): acpiphp: Slot [17] registered
1 Time(s): acpiphp: Slot [18] registered
1 Time(s): acpiphp: Slot [19] registered
1 Time(s): acpiphp: Slot [21] registered
1 Time(s): acpiphp: Slot [22] registered
1 Time(s): acpiphp: Slot [23] registered
1 Time(s): acpiphp: Slot [24] registered
1 Time(s): acpiphp: Slot [25] registered
1 Time(s): acpiphp: Slot [26] registered
1 Time(s): acpiphp: Slot [27] registered
1 Time(s): acpiphp: Slot [28] registered
1 Time(s): acpiphp: Slot [29] registered
1 Time(s): acpiphp: Slot [31] registered
1 Time(s): acpiphp: Slot [3] registered
1 Time(s): acpiphp: Slot [4] registered
1 Time(s): acpiphp: Slot [5] registered
1 Time(s): acpiphp: Slot [6] registered
1 Time(s): acpiphp: Slot [7] registered
1 Time(s): acpiphp: Slot [8] registered
1 Time(s): acpiphp: Slot [9] registered
1 Time(s): async_tx: api initialized (async)
1 Time(s): audit: initializing netlink subsys (disabled)
1 Time(s): clocksource: Switched to clocksource kvm-clock
1 Time(s): cpuidle: using governor ladder
1 Time(s): cpuidle: using governor menu
1 Time(s): devtmpfs: initialized
1 Time(s): ehci-pci: EHCI PCI platform driver
1 Time(s): ehci-platform: EHCI generic platform driver
1 Time(s): evm: security.SMACK64
1 Time(s): evm: security.SMACK64EXEC
1 Time(s): evm: security.SMACK64MMAP
1 Time(s): evm: security.SMACK64TRANSMUTE
1 Time(s): evm: security.capability
1 Time(s): evm: security.ima
1 Time(s): evm: security.selinux
1 Time(s): ftrace: allocating 34227 entries in 134 pages
1 Time(s): fuse init (API version 7.26)
1 Time(s): hidraw: raw HID events driver (C) Jiri Kosina
1 Time(s): hpet clockevent registered
1 Time(s): i2c /dev entries driver
1 Time(s): ima: No TPM chip found, activating TPM-bypass! (rc=-19)
1 Time(s): intel_idle: does not run on family 6 model 42
1 Time(s): io scheduler cfq registered
1 Time(s): io scheduler deadline registered
1 Time(s): io scheduler noop registered (default)
1 Time(s): ledtrig-cpu: registered to indicate activity on CPUs
1 Time(s): libphy: Fixed MDIO Bus: probed
1 Time(s): loop: module loaded
1 Time(s): mousedev: PS/2 mouse device common for all mice
1 Time(s): ohci-pci: OHCI PCI platform driver
1 Time(s): ohci-platform: OHCI generic platform driver
1 Time(s): ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
1 Time(s): pinctrl core: initialized pinctrl subsystem
1 Time(s): pnp: PnP ACPI init
1 Time(s): pnp: PnP ACPI: found 5 devices
1 Time(s): ppdev: user-space parallel port driver
1 Time(s): raid6: .... xor() 6424 MB/s, rmw enabled
1 Time(s): raid6: sse2x1 gen() 6723 MB/s
1 Time(s): raid6: sse2x1 xor() 5185 MB/s
1 Time(s): raid6: sse2x2 gen() 8399 MB/s
1 Time(s): raid6: sse2x2 xor() 5585 MB/s
1 Time(s): raid6: sse2x4 gen() 9942 MB/s
1 Time(s): raid6: sse2x4 xor() 6424 MB/s
1 Time(s): raid6: using algorithm sse2x4 gen() 9942 MB/s
1 Time(s): raid6: using ssse3x2 recovery algorithm
1 Time(s): random: crng init done
1 Time(s): random: fast init done
8 Time(s): random: systemd-udevd: uninitialized urandom read (16 bytes read)
2 Time(s): random: udevadm: uninitialized urandom read (16 bytes read)
1 Time(s): registered taskstats version 1
1 Time(s): scsi host1: ata_piix
1 Time(s): scsi host2: Virtio SCSI HBA
1 Time(s): sda: sda1
1 Time(s): setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1
1 Time(s): smp: Bringing up secondary CPUs ...
1 Time(s): smp: Brought up 1 node, 4 CPUs
1 Time(s): smpboot: Max logical packages: 1
1 Time(s): smpboot: Total of 4 processors activated (16799.98 BogoMIPS)
1 Time(s): tun: Universal TUN/TAP device driver, 1.6
1 Time(s): uhci_hcd: USB Universal Host Controller Interface driver
1 Time(s): usb 1-1: Manufacturer: QEMU
1 Time(s): usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=5
1 Time(s): usb 1-1: Product: QEMU USB Tablet
1 Time(s): usb 1-1: SerialNumber: 42
1 Time(s): usb 1-1: new full-speed USB device number 2 using uhci_hcd
1 Time(s): usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
1 Time(s): usb usb1: Product: UHCI Host Controller
1 Time(s): usbcore: registered new device driver usb
1 Time(s): usbcore: registered new interface driver hub
1 Time(s): usbcore: registered new interface driver usbfs
1 Time(s): usbcore: registered new interface driver usbhid
1 Time(s): usbhid: USB HID core driver
1 Time(s): vgaarb: loaded
1 Time(s): x2apic enabled
1 Time(s): x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
1 Time(s): x86/mm: Checked W+X mappings: passed, no W+X pages found.
1 Time(s): x86/mm: Memory block size: 128MB
1 Time(s): xor: automatically using best checksumming function avx
1 Time(s): zbud: loaded
1 Time(s): zswap: loaded using pool lzo/zbud
---------------------- Kernel End -------------------------

Post by Marcel Bischoff

Post by Skale, Franz
06:21 is the lograotation, so no problem. Same by me.

A bit reassuring but still: shouldn't the service cleanly restart
instead of barfing "Disorderly Shutdown"?

Post by Skale, Franz
What strucks me is, that it seems that ns-slapd as to reallocate memory.
How much memory does your server have ?
send free -m

$ free -m
total used free shared buff/cache
available
Mem: 16045 1304 13096 56
1644 14371
Swap: 0 0 0
No problem I can see there.

Post by Skale, Franz
Do you have selinux enabled !
If so, disable it by adding selinux=0 to /etc/default/grub and rerun
update-grub.

No, not enabled.

Post by Skale, Franz
Send the kernel version: uname -a

Linux mx.example.com 4.10.0-42-generic #46~16.04.1-Ubuntu SMP Mon Dec 4
15:57:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Post by Skale, Franz
How much open file handles to your system allow per process ?
send: ulimit -a

$ ulimit -a
Maximum size of core files created (kB, -c) 0
Maximum size of a process’s data segment (kB, -d) unlimited
Maximum size of files created by the shell (kB, -f) unlimited
Maximum size that may be locked into memory (kB, -l) 64
Maximum resident set size (kB, -m) unlimited
Maximum number of open file descriptors
(-n) 1024
Maximum stack size (kB, -s) 8192
Maximum amount of cpu time in seconds (seconds, -t) unlimited
Maximum number of processes available to a single user
(-u) 64015
Maximum amount of virtual memory available to the shell (kB, -v) unlimited

Post by Skale, Franz
send dmesg: (is there a segfault).

The whole dmesg output is spammed by ufw and contains no useful
information whatsoever.

Post by Skale, Franz
send dmidecode

I don't think this is likely as this is a virtual server.
$ dmidecode
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.
10 structures occupying 408 bytes.
Table at 0x000F68A0.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: SeaBIOS
Version: 1.10.2
Release Date: 04/01/2014
Address: 0xE8000
Runtime Size: 96 kB
ROM Size: 64 kB
BIOS characteristics not supported
Targeted content distribution is supported
BIOS Revision: 0.0
Handle 0x0100, DMI type 1, 27 bytes
System Information
Manufacturer: Hetzner
Product Name: vServer
Version: 2
Serial Number: Not Specified
UUID: A8236400-D36B-0135-FE8F-10BF48D7F2C6
Wake-up Type: Power Switch
SKU Number: a8236400-d36b-0135-fe8f-10bf48d7f2c6
Family: Not Specified
Handle 0x0300, DMI type 3, 21 bytes
Chassis Information
Manufacturer: QEMU
Type: Other
Lock: Not Present
Version: pc-i440fx-2.10
Serial Number: Not Specified
Asset Tag: Not Specified
Boot-up State: Safe
Power Supply State: Safe
Thermal State: Safe
Security Status: Unknown
OEM Information: 0x00000000
Height: Unspecified
Number Of Power Cords: Unspecified
Contained Elements: 0
Handle 0x0400, DMI type 4, 42 bytes
Processor Information
Socket Designation: CPU 0
Type: Central Processor
Family: Other
Manufacturer: QEMU
ID: A1 06 02 00 FF FB 8B 07
Version: pc-i440fx-2.10
Voltage: Unknown
External Clock: Unknown
Max Speed: 2000 MHz
Current Speed: 2000 MHz
Status: Populated, Enabled
Upgrade: Other
L1 Cache Handle: Not Provided
L2 Cache Handle: Not Provided
L3 Cache Handle: Not Provided
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Core Count: 4
Core Enabled: 4
Thread Count: 1
Characteristics: None
Handle 0x1000, DMI type 16, 23 bytes
Physical Memory Array
Location: Other
Use: System Memory
Error Correction Type: Multi-bit ECC
Maximum Capacity: 16 GB
Error Information Handle: Not Provided
Number Of Devices: 1
Handle 0x1100, DMI type 17, 40 bytes
Memory Device
Array Handle: 0x1000
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: 16384 MB
Form Factor: DIMM
Set: None
Locator: DIMM 0
Bank Locator: Not Specified
Type: RAM
Type Detail: Other
Speed: Unknown
Manufacturer: QEMU
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Rank: Unknown
Configured Clock Speed: Unknown
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown
Handle 0x1300, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00000000000
Ending Address: 0x000BFFFFFFF
Range Size: 3 GB
Physical Array Handle: 0x1000
Partition Width: 1
Handle 0x1301, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00100000000
Ending Address: 0x0043FFFFFFF
Range Size: 13 GB
Physical Array Handle: 0x1000
Partition Width: 1
Handle 0x2000, DMI type 32, 11 bytes
System Boot Information
Status: No errors detected
Handle 0x7F00, DMI type 127, 4 bytes
End Of Table

Post by Skale, Franz
Did you update your kernel days ago, if so, you sure ran into a buggy
kernel 4.9.65.
I built a 4.9.75 PTI enabled kernel which i send you to test.

4.10.0-42-generic

Post by Skale, Franz
Office 365 would be a bad and expensive choice.

I agree. I'd very much like to avoid it but when Kolab turn out to have
issues with the current setup, I doubt I'll get another shot.
Best,
Marcel

Marcel Bischoff

2018-01-11 13:19:08 UTC

Permalink

Hey Franz,

what do you know — tonight there wasn't a disorderly shutdown. The main
ingredient changed was a kernel update from 4.10.0-42-generic to
4.13.0-26-generic (a rather unusual jump), which was distributed through
the main package sources. So this supports your theory about something
amiss with the guest kernel.

I will monitor the situation carefully. Slowly I re-build some
confidence in this setup. Additionally, I will not touch Ubuntu again
for new deployments. Not with a stick. The AppArmor issue (everything
depends on it) broke the camels' back.

Best,
Marcel

Post by Skale, Franz
Hi Marcel,
i now checked my ldap server and also found some messages regarding
[10/Jan/2018:22:32:54.579885565 +0100] connection - conn=2940 fd=284
Attempt to release connection that is not acquire
[10/Jan/2018:22:32:54.585434274 +0100] connection - conn=4470 fd=188
Attempt to release connection that is not acquire
[10/Jan/2018:23:50:07.863504579 +0100] NSACLPlugin -
acl_access_allowed: Resetting aclpb_pblock 0x7fa3a3fe6a60 to pblo
[11/Jan/2018:00:07:34.557315481 +0100] NSACLPlugin -
acl_access_allowed: Resetting aclpb_pblock 0x7fa3a9ff2a60 to pblo
[11/Jan/2018:03:12:32.383353274 +0100] NSACLPlugin -
acl_access_allowed: Resetting aclpb_pblock 0x7fa3aeffca60 to pblo
[11/Jan/2018:06:47:56.284946442 +0100] NSACLPlugin -
acl_access_allowed: Resetting aclpb_pblock 0x7fa3a7feea60 to pblo
[11/Jan/2018:06:48:00.364639406 +0100] NSACLPlugin -
acl_access_allowed: Resetting aclpb_pblock 0x7fa3ae7fba60 to pblo
[11/Jan/2018:07:53:34.001520691 +0100] NSACLPlugin -
acl_access_allowed: Resetting aclpb_pblock 0x7fa3a17e1a60 to pblo
[11/Jan/2018:07:58:34.641383768 +0100] connection - conn=3240 fd=278
Attempt to release connection that is not acquire
[11/Jan/2018:07:58:34.646688065 +0100] connection - conn=0 fd=0
Attempt to release connection that is not acquired
But i have no unordered shutdown mentioned anywhere in the logs.
I also found out, that rasing the filelimit level doesn't work for the
dirsrv service.
ulimit -n 65535
ulimit -n 65535
# uncomment this line to raise the file descriptor limit
LimitNOFILE=65535
If your dirsrv shutdown unordered i personall think you have a problem
with either the host kernel or the guest kernel.
dirsrv is multithreaded and creates a thread for every connection
leaving alone filepointers it opens on demand.
So even on my testserver with no user created it consumes 200
filepointers after starting.
Rgds.
Franz

Post by Marcel Bischoff
Hi Franz,
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no
389-admin 389-admin-console 389-console 389-ds 389-ds-base
389-ds-base-libs 389-ds-console 389-ds-console-doc 389-dsgw
amavisd-new aspell aspell-en augeas-lenses chwala
clamav clamav-base clamav-daemon clamav-freshclam clamdscan
cyrus-imapd dictionaries-common emacsen-common erlang-base
erlang-crypto erlang-eimap erlang-goldrush erlang-lager
erlang-lager-syslog erlang-syntax-tools fontconfig guam irony
kolab-cli kolab-conf kolab-freebusy kolab-imap kolab-ldap kolab-mta
kolab-saslauthd kolab-schema kolab-server
kolab-syncroton kolab-webadmin kolab-xml ldap-utils libadminutil-data
libadminutil0 libapache2-mod-nss libapache2-mod-php
libapache2-mod-php7.0 libapparmor-perl libaspell15
libaudio2 libaugeas0 libauthen-sasl-perl libavahi-client3
libavahi-common-data libavahi-common3 libberkeleydb-perl
libcalendaring libcgi-fast-perl libcgi-pm-perl libclamav7
libconvert-binhex-perl libconvert-tnef-perl libconvert-uulib-perl
libcrypt-openssl-bignum-perl libcrypt-openssl-rsa-perl libcups2
libdigest-hmac-perl libds-admin-serv0
libencode-locale-perl libevent-core-2.0-5 libfcgi-perl libgd3
libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl
libhttp-date-perl libhttp-message-perl libical1a
libidm-console-framework-java libio-html-perl libio-multiplex-perl
libio-socket-inet6-perl libio-socket-ssl-perl libio-stringy-perl
libjansson4 libjbig0 libjpeg-turbo8
libjpeg8 libjss-java libkolab2 libkolabxml1v5 liblcms2-2 libldap-java
libllvm3.6v5 liblwp-mediatypes-perl libmail-dkim-perl libmail-spf-perl
libmailtools-perl libmcrypt4
libmime-tools-perl libmng2 libmozilla-ldap-perl libmozldap-0d
libnet-cidr-perl libnet-dns-perl libnet-ip-perl libnet-libidn-perl
libnet-server-perl libnet-smtp-ssl-perl
libnet-ssleay-perl libnetaddr-ip-perl libnss3-tools
libperl4-corelibs-perl libqt4-dbus libqt4-declarative libqt4-network
libqt4-script libqt4-sql libqt4-sql-mysql libqt4-xml
libqt4-xmlpatterns libqtcore4 libqtdbus4 libqtgui4 libsctp1
libsocket-getaddrinfo-perl libsocket6-perl libtiff5
libunix-syslog-perl liburi-perl libvpx3 libxerces-c3.1
libxslt1.1 libzend-framework-php libzephyr4 mozldap-tools
mysql-client mysql-client-5.7 mysql-client-core-5.7
mysql-server-core-5.7 pax php php-auth-sasl php-cli php-common
php-curl php-gd php-http-request2 php-intl php-kolab php-kolabformat
php-ldap php-mail php-mail-mime php-mail-mimedecode php-mbstring
php-mcrypt php-mdb2
php-mdb2-driver-mysql php-monolog php-mysql php-net-idna2
php-net-ldap2 php-net-ldap3 php-net-sieve php-net-smtp php-net-socket
php-net-url2 php-pear php-pspell php-psr-log
php-sabre-dav-2.1 php-sabre-event php-sabre-http-3
php-sabre-vobject-3 php-xml php7.0 php7.0-cli php7.0-common
php7.0-curl php7.0-fpm php7.0-gd php7.0-intl php7.0-json
php7.0-ldap php7.0-mbstring php7.0-mcrypt php7.0-mysql php7.0-opcache
php7.0-pspell php7.0-readline php7.0-xml pykolab python-augeas
python-cheetah python-dateutil
python-gnupg python-icalendar python-kolab python-kolabformat
python-ldap python-pkg-resources python-pyasn1 python-pyasn1-modules
python-pymysql python-six python-sqlalchemy
python-sqlalchemy-ext python-tz python-tzlocal qdbus qt-at-spi
qtchooser qtcore4-l10n re2c roundcubemail roundcubemail-core
roundcubemail-plugin-acl
roundcubemail-plugin-archive roundcubemail-plugin-calendar
roundcubemail-plugin-contextmenu
roundcubemail-plugin-filesystem-attachments
roundcubemail-plugin-jqueryui
roundcubemail-plugin-kolab-activesync
roundcubemail-plugin-kolab-addressbook roundcubemail-plugin-kolab-auth
roundcubemail-plugin-kolab-config
roundcubemail-plugin-kolab-delegation
roundcubemail-plugin-kolab-files roundcubemail-plugin-kolab-folders
roundcubemail-plugin-kolab-notes roundcubemail-plugin-kolab-tags
roundcubemail-plugin-libcalendaring roundcubemail-plugin-libkolab
roundcubemail-plugin-managesieve roundcubemail-plugin-newmail-notifier
roundcubemail-plugin-odfviewer
roundcubemail-plugin-password roundcubemail-plugin-pdfviewer
roundcubemail-plugin-redundant-attachments
roundcubemail-plugin-tasklist roundcubemail-plugin-zipdownload
roundcubemail-plugins-kolab roundcubemail-skin-chameleon sa-compile
smarty3 spamassassin spamc wallace zend-framework zend-framework-bin
Use 'apt autoremove' to remove them.
apparmor kolab kolab-webclient mysql-server mysql-server-5.7
0 upgraded, 0 newly installed, 5 to remove and 0 not upgraded.
After this operation, 50.2 MB disk space will be freed.
Do you want to continue? [Y/n]
Which amounts to basically... everything.
apparmor module is loaded.
0 profiles are loaded.
0 profiles are in enforce mode.
0 profiles are in complain mode.
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
So that should be alright I guess.
I have raised the limits and will monitor the outcome. Hopefully things
will quiet down now.
Best,
Marcel

Post by Marcel Bischoff
I have been able to pull the kernel messages from the logwatch output,
didn't think of this before. Maybe it helps in homing in on the cause. I
couldn't spot something obvious though. Maybe AppArmor does something
undesirable? I remember running into issues with it years ago in another
context.
I will uninstall it tonight, reboot the server and report on my progress
(or lack thereof).
Thanks again for bearing with me!
--------------------- Kernel Begin ------------------------
1 Time(s): #2
1 Time(s): #3
1 Time(s): 1 disabled
1 Time(s): 2 disabled
1 Time(s): 3 disabled
1 Time(s): 4 disabled
1 Time(s): 5 disabled
1 Time(s): 6 disabled
1 Time(s): 7 disabled
1 Time(s): ACPI: 1 ACPI AML tables successfully acquired and loaded
1 Time(s): ACPI: Added _OSI(Module Device)
1 Time(s): ACPI: Added _OSI(Processor Aggregator Device)
1 Time(s): ACPI: Added _OSI(Processor Device)
1 Time(s): ACPI: Early table checksum verification disabled
1 Time(s): ACPI: IRQ11 used by override.
1 Time(s): ACPI: IRQ5 used by override.
1 Time(s): ACPI: IRQ9 used by override.
1 Time(s): ACPI: Interpreter enabled
1 Time(s): ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
1 Time(s): ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
1 Time(s): ACPI: Power Button [PWRF]
1 Time(s): ACPI: Using IOAPIC for interrupt routing
1 Time(s): ACPI: bus type PCI registered
1 Time(s): ACPI: bus type USB registered
1 Time(s): AES CTR mode by8 optimization enabled
1 Time(s): AMD AuthenticAMD
1 Time(s): AVX version of gcm_enc/dec engaged.
1 Time(s): AppArmor: AppArmor Filesystem Enabled
1 Time(s): AppArmor: AppArmor initialized
1 Time(s): AppArmor: AppArmor sha1 policy hashing enabled
1 Time(s): Booting paravirtualized kernel on KVM
1 Time(s): Btrfs loaded, crc32c=crc32c-intel
1 Time(s): Build-time adjustment of leaf fanout to 64.
1 Time(s): Built 1 zonelists in Node order, mobility grouping on.
Total pages: 4128613
1 Time(s): Calgary: Unable to locate Rio Grande table in EBDA - bailing!
1 Time(s): Calgary: detecting Calgary via BIOS EBDA area
1 Time(s): Calibrating delay loop (skipped) preset value.. 4199.99
BogoMIPS (lpj=8399992)
1 Time(s): Centaur CentaurHauls
1 Time(s): DMA zone: 21 pages reserved
1 Time(s): DMA zone: 64 pages used for memmap
1 Time(s): DMA32 zone: 12224 pages used for memmap
1 Time(s): Device empty
1 Time(s): EDD information not available.
1 Time(s): EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
1 Time(s): EXT4-fs (sda1): re-mounted. Opts: discard
1 Time(s): Early memory node ranges
1 Time(s): Freeing SMP alternatives memory: 32K
1 Time(s): Freeing unused kernel memory: 1156K
1 Time(s): Freeing unused kernel memory: 2228K
1 Time(s): Freeing unused kernel memory: 268K
1 Time(s): GHES: HEST is not enabled!
1 Time(s): Hierarchical RCU implementation.
1 Time(s): Hypervisor detected: KVM
1 Time(s): Initialise system trusted keyrings
1 Time(s): Intel GenuineIntel
1 Time(s): KVM setup async PF for cpu 1
1 Time(s): KVM setup async PF for cpu 2
1 Time(s): KVM setup async PF for cpu 3
1 Time(s): Key type asymmetric registered
1 Time(s): Key type big_key registered
1 Time(s): Key type dns_resolver registered
1 Time(s): Key type encrypted registered
1 Time(s): Key type trusted registered
1 Time(s): MTRR default type: write-back
1 Time(s): Magic number: 2:724:141
1 Time(s): Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
1 Time(s): Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes)
1 Time(s): Movable zone start for each node
1 Time(s): NET: Registered protocol family 1
1 Time(s): NET: Registered protocol family 16
1 Time(s): NET: Registered protocol family 17
1 Time(s): NET: Registered protocol family 2
1 Time(s): NR_IRQS:524544 nr_irqs:456 16
1 Time(s): NX (Execute Disable) protection: active
1 Time(s): NetLabel: domain hash size = 128
1 Time(s): NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
1 Time(s): NetLabel: unlabeled traffic allowed by default
1 Time(s): NetLabel: Initializing
1 Time(s): No NUMA configuration found
1 Time(s): Normal zone: 53248 pages used for memmap
1 Time(s): PCCT header not found.
1 Time(s): PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
1 Time(s): PCI: Using ACPI for IRQ routing
1 Time(s): PCI: Using configuration type 1 for base access
1 Time(s): PCI: Using host bridge windows from ACPI; if
necessary, use
"pci=nocrs" and report a bug
1 Time(s): PCI: pci_cache_line_size set to 64 bytes
1 Time(s): PM: Hibernation image not present or could not be loaded.
1 Time(s): PPP generic driver version 2.4.2
1 Time(s): Performance Events: unsupported p6 CPU model 42 no PMU
driver, software events only.
1 Time(s): Policy zone: Normal
2 Time(s): Process accounting resumed
1 Time(s): RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4.
1 Time(s): RCU: Adjusting geometry for rcu_fanout_leaf=64,
nr_cpu_ids=4
1 Time(s): SCSI subsystem initialized
1 Time(s): SMBIOS 2.8 present.
1 Time(s): Scanning 1 areas for low memory corruption
1 Time(s): Security Framework initialized
1 Time(s): Segment Routing with IPv6
1 Time(s): Switched APIC routing to physical x2apic.
1 Time(s): TSC deadline timer enabled
1 Time(s): UDP hash table entries: 8192 (order: 6, 262144 bytes)
1 Time(s): UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
1 Time(s): Unpacking initramfs...
1 Time(s): Using ACPI (MADT) for SMP configuration information
1 Time(s): Write protecting the kernel read-only data: 14336k
1 Time(s): Yama: becoming mindful.
1 Time(s): acpi device:12: hash matches
1 Time(s): acpiphp: Slot [11] registered
1 Time(s): acpiphp: Slot [12] registered
1 Time(s): acpiphp: Slot [13] registered
1 Time(s): acpiphp: Slot [14] registered
1 Time(s): acpiphp: Slot [15] registered
1 Time(s): acpiphp: Slot [16] registered
1 Time(s): acpiphp: Slot [17] registered
1 Time(s): acpiphp: Slot [18] registered
1 Time(s): acpiphp: Slot [19] registered
1 Time(s): acpiphp: Slot [21] registered
1 Time(s): acpiphp: Slot [22] registered
1 Time(s): acpiphp: Slot [23] registered
1 Time(s): acpiphp: Slot [24] registered
1 Time(s): acpiphp: Slot [25] registered
1 Time(s): acpiphp: Slot [26] registered
1 Time(s): acpiphp: Slot [27] registered
1 Time(s): acpiphp: Slot [28] registered
1 Time(s): acpiphp: Slot [29] registered
1 Time(s): acpiphp: Slot [31] registered
1 Time(s): acpiphp: Slot [3] registered
1 Time(s): acpiphp: Slot [4] registered
1 Time(s): acpiphp: Slot [5] registered
1 Time(s): acpiphp: Slot [6] registered
1 Time(s): acpiphp: Slot [7] registered
1 Time(s): acpiphp: Slot [8] registered
1 Time(s): acpiphp: Slot [9] registered
1 Time(s): async_tx: api initialized (async)
1 Time(s): audit: initializing netlink subsys (disabled)
1 Time(s): clocksource: Switched to clocksource kvm-clock
1 Time(s): cpuidle: using governor ladder
1 Time(s): cpuidle: using governor menu
1 Time(s): devtmpfs: initialized
1 Time(s): ehci-pci: EHCI PCI platform driver
1 Time(s): ehci-platform: EHCI generic platform driver
1 Time(s): evm: security.SMACK64
1 Time(s): evm: security.SMACK64EXEC
1 Time(s): evm: security.SMACK64MMAP
1 Time(s): evm: security.SMACK64TRANSMUTE
1 Time(s): evm: security.capability
1 Time(s): evm: security.ima
1 Time(s): evm: security.selinux
1 Time(s): ftrace: allocating 34227 entries in 134 pages
1 Time(s): fuse init (API version 7.26)
1 Time(s): hidraw: raw HID events driver (C) Jiri Kosina
1 Time(s): hpet clockevent registered
1 Time(s): i2c /dev entries driver
1 Time(s): ima: No TPM chip found, activating TPM-bypass! (rc=-19)
1 Time(s): intel_idle: does not run on family 6 model 42
1 Time(s): io scheduler cfq registered
1 Time(s): io scheduler deadline registered
1 Time(s): io scheduler noop registered (default)
1 Time(s): ledtrig-cpu: registered to indicate activity on CPUs
1 Time(s): libphy: Fixed MDIO Bus: probed
1 Time(s): loop: module loaded
1 Time(s): mousedev: PS/2 mouse device common for all mice
1 Time(s): ohci-pci: OHCI PCI platform driver
1 Time(s): ohci-platform: OHCI generic platform driver
1 Time(s): ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
1 Time(s): pinctrl core: initialized pinctrl subsystem
1 Time(s): pnp: PnP ACPI init
1 Time(s): pnp: PnP ACPI: found 5 devices
1 Time(s): ppdev: user-space parallel port driver
1 Time(s): raid6: .... xor() 6424 MB/s, rmw enabled
1 Time(s): raid6: sse2x1 gen() 6723 MB/s
1 Time(s): raid6: sse2x1 xor() 5185 MB/s
1 Time(s): raid6: sse2x2 gen() 8399 MB/s
1 Time(s): raid6: sse2x2 xor() 5585 MB/s
1 Time(s): raid6: sse2x4 gen() 9942 MB/s
1 Time(s): raid6: sse2x4 xor() 6424 MB/s
1 Time(s): raid6: using algorithm sse2x4 gen() 9942 MB/s
1 Time(s): raid6: using ssse3x2 recovery algorithm
1 Time(s): random: crng init done
1 Time(s): random: fast init done
8 Time(s): random: systemd-udevd: uninitialized urandom read (16 bytes read)
2 Time(s): random: udevadm: uninitialized urandom read (16 bytes read)
1 Time(s): registered taskstats version 1
1 Time(s): scsi host1: ata_piix
1 Time(s): scsi host2: Virtio SCSI HBA
1 Time(s): sda: sda1
1 Time(s): setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1
1 Time(s): smp: Bringing up secondary CPUs ...
1 Time(s): smp: Brought up 1 node, 4 CPUs
1 Time(s): smpboot: Max logical packages: 1
1 Time(s): smpboot: Total of 4 processors activated (16799.98 BogoMIPS)
1 Time(s): tun: Universal TUN/TAP device driver, 1.6
1 Time(s): uhci_hcd: USB Universal Host Controller Interface driver
1 Time(s): usb 1-1: Manufacturer: QEMU
1 Time(s): usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=5
1 Time(s): usb 1-1: Product: QEMU USB Tablet
1 Time(s): usb 1-1: SerialNumber: 42
1 Time(s): usb 1-1: new full-speed USB device number 2 using uhci_hcd
1 Time(s): usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
1 Time(s): usb usb1: Product: UHCI Host Controller
1 Time(s): usbcore: registered new device driver usb
1 Time(s): usbcore: registered new interface driver hub
1 Time(s): usbcore: registered new interface driver usbfs
1 Time(s): usbcore: registered new interface driver usbhid
1 Time(s): usbhid: USB HID core driver
1 Time(s): vgaarb: loaded
1 Time(s): x2apic enabled
1 Time(s): x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
1 Time(s): x86/mm: Checked W+X mappings: passed, no W+X pages found.
1 Time(s): x86/mm: Memory block size: 128MB
1 Time(s): xor: automatically using best checksumming function avx
1 Time(s): zbud: loaded
1 Time(s): zswap: loaded using pool lzo/zbud
---------------------- Kernel End -------------------------

Post by Marcel Bischoff

Post by Skale, Franz
06:21 is the lograotation, so no problem. Same by me.

A bit reassuring but still: shouldn't the service cleanly restart
instead of barfing "Disorderly Shutdown"?

Post by Skale, Franz
What strucks me is, that it seems that ns-slapd as to reallocate memory.
How much memory does your server have ?
send free -m

$ free -m
total used free shared buff/cache
available
Mem: 16045 1304 13096 56
1644 14371
Swap: 0 0 0
No problem I can see there.

Post by Skale, Franz
Do you have selinux enabled !
If so, disable it by adding selinux=0 to /etc/default/grub
and rerun
update-grub.

No, not enabled.

Post by Skale, Franz
Send the kernel version: uname -a

Linux mx.example.com 4.10.0-42-generic #46~16.04.1-Ubuntu SMP Mon Dec 4
15:57:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Post by Skale, Franz
How much open file handles to your system allow per process ?
send: ulimit -a

$ ulimit -a
Maximum size of core files created (kB, -c) 0
Maximum size of a process’s data segment (kB, -d) unlimited
Maximum size of files created by the shell (kB, -f) unlimited
Maximum size that may be locked into memory (kB, -l) 64
Maximum resident set size (kB, -m) unlimited
Maximum number of open file descriptors
(-n) 1024
Maximum stack size (kB, -s) 8192
Maximum amount of cpu time in seconds (seconds, -t) unlimited
Maximum number of processes available to a single user
(-u) 64015
Maximum amount of virtual memory available to the shell (kB, -v) unlimited

Post by Skale, Franz
send dmesg: (is there a segfault).

The whole dmesg output is spammed by ufw and contains no useful
information whatsoever.

Post by Skale, Franz
send dmidecode

I don't think this is likely as this is a virtual server.
$ dmidecode
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.
10 structures occupying 408 bytes.
Table at 0x000F68A0.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: SeaBIOS
Version: 1.10.2
Release Date: 04/01/2014
Address: 0xE8000
Runtime Size: 96 kB
ROM Size: 64 kB
BIOS characteristics not supported
Targeted content distribution is supported
BIOS Revision: 0.0
Handle 0x0100, DMI type 1, 27 bytes
System Information
Manufacturer: Hetzner
Product Name: vServer
Version: 2
Serial Number: Not Specified
UUID: A8236400-D36B-0135-FE8F-10BF48D7F2C6
Wake-up Type: Power Switch
SKU Number: a8236400-d36b-0135-fe8f-10bf48d7f2c6
Family: Not Specified
Handle 0x0300, DMI type 3, 21 bytes
Chassis Information
Manufacturer: QEMU
Type: Other
Lock: Not Present
Version: pc-i440fx-2.10
Serial Number: Not Specified
Asset Tag: Not Specified
Boot-up State: Safe
Power Supply State: Safe
Thermal State: Safe
Security Status: Unknown
OEM Information: 0x00000000
Height: Unspecified
Number Of Power Cords: Unspecified
Contained Elements: 0
Handle 0x0400, DMI type 4, 42 bytes
Processor Information
Socket Designation: CPU 0
Type: Central Processor
Family: Other
Manufacturer: QEMU
ID: A1 06 02 00 FF FB 8B 07
Version: pc-i440fx-2.10
Voltage: Unknown
External Clock: Unknown
Max Speed: 2000 MHz
Current Speed: 2000 MHz
Status: Populated, Enabled
Upgrade: Other
L1 Cache Handle: Not Provided
L2 Cache Handle: Not Provided
L3 Cache Handle: Not Provided
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Core Count: 4
Core Enabled: 4
Thread Count: 1
Characteristics: None
Handle 0x1000, DMI type 16, 23 bytes
Physical Memory Array
Location: Other
Use: System Memory
Error Correction Type: Multi-bit ECC
Maximum Capacity: 16 GB
Error Information Handle: Not Provided
Number Of Devices: 1
Handle 0x1100, DMI type 17, 40 bytes
Memory Device
Array Handle: 0x1000
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: 16384 MB
Form Factor: DIMM
Set: None
Locator: DIMM 0
Bank Locator: Not Specified
Type: RAM
Type Detail: Other
Speed: Unknown
Manufacturer: QEMU
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Rank: Unknown
Configured Clock Speed: Unknown
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown
Handle 0x1300, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00000000000
Ending Address: 0x000BFFFFFFF
Range Size: 3 GB
Physical Array Handle: 0x1000
Partition Width: 1
Handle 0x1301, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00100000000
Ending Address: 0x0043FFFFFFF
Range Size: 13 GB
Physical Array Handle: 0x1000
Partition Width: 1
Handle 0x2000, DMI type 32, 11 bytes
System Boot Information
Status: No errors detected
Handle 0x7F00, DMI type 127, 4 bytes
End Of Table

Post by Skale, Franz
Did you update your kernel days ago, if so, you sure ran
into a buggy
kernel 4.9.65.
I built a 4.9.75 PTI enabled kernel which i send you to test.

4.10.0-42-generic

Post by Skale, Franz
Office 365 would be a bad and expensive choice.

I agree. I'd very much like to avoid it but when Kolab turn out to have
issues with the current setup, I doubt I'll get another shot.
Best,
Marcel

Skale, Franz

2018-01-11 13:33:40 UTC

Permalink

Hi Marcel,
as a matter of fact, there's a major change in the so called kaiser
patch (KPTI) for users who use KVM and linux guests. There i build a new
kernel 4.9.75 as of 06.01.2018. You have to patch the Host and the guest
though.
Good to hear that the unordered shutdown disappeared. I also don't have
unordered shutdowns using the new 389 base package.
Tuning the limits will sure help !

Rgds.
Franz

Post by Marcel Bischoff
Hey Franz,
what do you know — tonight there wasn't a disorderly shutdown. The main
ingredient changed was a kernel update from 4.10.0-42-generic to
4.13.0-26-generic (a rather unusual jump), which was distributed through
the main package sources. So this supports your theory about something
amiss with the guest kernel.
I will monitor the situation carefully. Slowly I re-build some
confidence in this setup. Additionally, I will not touch Ubuntu again
for new deployments. Not with a stick. The AppArmor issue (everything
depends on it) broke the camels' back.
Best,
Marcel

Post by Marcel Bischoff
Hi Franz,
Reading package lists... Done
Building dependency tree
Reading state information... Done
389-admin 389-admin-console 389-console 389-ds 389-ds-base
389-ds-base-libs 389-ds-console 389-ds-console-doc 389-dsgw
amavisd-new aspell aspell-en augeas-lenses chwala
clamav clamav-base clamav-daemon clamav-freshclam clamdscan
cyrus-imapd dictionaries-common emacsen-common erlang-base
erlang-crypto erlang-eimap erlang-goldrush erlang-lager
erlang-lager-syslog erlang-syntax-tools fontconfig guam irony
kolab-cli kolab-conf kolab-freebusy kolab-imap kolab-ldap kolab-mta
kolab-saslauthd kolab-schema kolab-server
kolab-syncroton kolab-webadmin kolab-xml ldap-utils libadminutil-data
libadminutil0 libapache2-mod-nss libapache2-mod-php
libapache2-mod-php7.0 libapparmor-perl libaspell15
libaudio2 libaugeas0 libauthen-sasl-perl libavahi-client3
libavahi-common-data libavahi-common3 libberkeleydb-perl
libcalendaring libcgi-fast-perl libcgi-pm-perl libclamav7
libconvert-binhex-perl libconvert-tnef-perl libconvert-uulib-perl
libcrypt-openssl-bignum-perl libcrypt-openssl-rsa-perl libcups2
libdigest-hmac-perl libds-admin-serv0
libencode-locale-perl libevent-core-2.0-5 libfcgi-perl libgd3
libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl
libhttp-date-perl libhttp-message-perl libical1a
libidm-console-framework-java libio-html-perl libio-multiplex-perl
libio-socket-inet6-perl libio-socket-ssl-perl libio-stringy-perl
libjansson4 libjbig0 libjpeg-turbo8
libjpeg8 libjss-java libkolab2 libkolabxml1v5 liblcms2-2 libldap-java
libllvm3.6v5 liblwp-mediatypes-perl libmail-dkim-perl
libmail-spf-perl
libmailtools-perl libmcrypt4
libmime-tools-perl libmng2 libmozilla-ldap-perl libmozldap-0d
libnet-cidr-perl libnet-dns-perl libnet-ip-perl libnet-libidn-perl
libnet-server-perl libnet-smtp-ssl-perl
libnet-ssleay-perl libnetaddr-ip-perl libnss3-tools
libperl4-corelibs-perl libqt4-dbus libqt4-declarative libqt4-network
libqt4-script libqt4-sql libqt4-sql-mysql libqt4-xml
libqt4-xmlpatterns libqtcore4 libqtdbus4 libqtgui4 libsctp1
libsocket-getaddrinfo-perl libsocket6-perl libtiff5
libunix-syslog-perl liburi-perl libvpx3 libxerces-c3.1
libxslt1.1 libzend-framework-php libzephyr4 mozldap-tools
mysql-client mysql-client-5.7 mysql-client-core-5.7
mysql-server-core-5.7 pax php php-auth-sasl php-cli php-common
php-curl php-gd php-http-request2 php-intl php-kolab php-kolabformat
php-ldap php-mail php-mail-mime php-mail-mimedecode php-mbstring
php-mcrypt php-mdb2
php-mdb2-driver-mysql php-monolog php-mysql php-net-idna2
php-net-ldap2 php-net-ldap3 php-net-sieve php-net-smtp php-net-socket
php-net-url2 php-pear php-pspell php-psr-log
php-sabre-dav-2.1 php-sabre-event php-sabre-http-3
php-sabre-vobject-3 php-xml php7.0 php7.0-cli php7.0-common
php7.0-curl php7.0-fpm php7.0-gd php7.0-intl php7.0-json
php7.0-ldap php7.0-mbstring php7.0-mcrypt php7.0-mysql php7.0-opcache
php7.0-pspell php7.0-readline php7.0-xml pykolab python-augeas
python-cheetah python-dateutil
python-gnupg python-icalendar python-kolab python-kolabformat
python-ldap python-pkg-resources python-pyasn1 python-pyasn1-modules
python-pymysql python-six python-sqlalchemy
python-sqlalchemy-ext python-tz python-tzlocal qdbus qt-at-spi
qtchooser qtcore4-l10n re2c roundcubemail roundcubemail-core
roundcubemail-plugin-acl
roundcubemail-plugin-archive roundcubemail-plugin-calendar
roundcubemail-plugin-contextmenu
roundcubemail-plugin-filesystem-attachments
roundcubemail-plugin-jqueryui
roundcubemail-plugin-kolab-activesync
roundcubemail-plugin-kolab-addressbook
roundcubemail-plugin-kolab-auth
roundcubemail-plugin-kolab-config
roundcubemail-plugin-kolab-delegation
roundcubemail-plugin-kolab-files roundcubemail-plugin-kolab-folders
roundcubemail-plugin-kolab-notes roundcubemail-plugin-kolab-tags
roundcubemail-plugin-libcalendaring roundcubemail-plugin-libkolab
roundcubemail-plugin-managesieve
roundcubemail-plugin-newmail-notifier
roundcubemail-plugin-odfviewer
roundcubemail-plugin-password roundcubemail-plugin-pdfviewer
roundcubemail-plugin-redundant-attachments
roundcubemail-plugin-tasklist roundcubemail-plugin-zipdownload
roundcubemail-plugins-kolab roundcubemail-skin-chameleon sa-compile
smarty3 spamassassin spamc wallace zend-framework zend-framework-bin
Use 'apt autoremove' to remove them.
apparmor kolab kolab-webclient mysql-server mysql-server-5.7
0 upgraded, 0 newly installed, 5 to remove and 0 not upgraded.
After this operation, 50.2 MB disk space will be freed.
Do you want to continue? [Y/n]
Which amounts to basically... everything.
apparmor module is loaded.
0 profiles are loaded.
0 profiles are in enforce mode.
0 profiles are in complain mode.
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
So that should be alright I guess.
I have raised the limits and will monitor the outcome. Hopefully things
will quiet down now.
Best,
Marcel

Post by Marcel Bischoff
I have been able to pull the kernel messages from the logwatch output,
didn't think of this before. Maybe it helps in homing in on the cause. I
couldn't spot something obvious though. Maybe AppArmor does something
undesirable? I remember running into issues with it years ago in another
context.
I will uninstall it tonight, reboot the server and report on my progress
(or lack thereof).
Thanks again for bearing with me!
--------------------- Kernel Begin ------------------------
1 Time(s): #2
1 Time(s): #3
1 Time(s): 1 disabled
1 Time(s): 2 disabled
1 Time(s): 3 disabled
1 Time(s): 4 disabled
1 Time(s): 5 disabled
1 Time(s): 6 disabled
1 Time(s): 7 disabled
1 Time(s): ACPI: 1 ACPI AML tables successfully acquired and loaded
1 Time(s): ACPI: Added _OSI(Module Device)
1 Time(s): ACPI: Added _OSI(Processor Aggregator Device)
1 Time(s): ACPI: Added _OSI(Processor Device)
1 Time(s): ACPI: Early table checksum verification disabled
1 Time(s): ACPI: IRQ11 used by override.
1 Time(s): ACPI: IRQ5 used by override.
1 Time(s): ACPI: IRQ9 used by override.
1 Time(s): ACPI: Interpreter enabled
1 Time(s): ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
1 Time(s): ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
1 Time(s): ACPI: Power Button [PWRF]
1 Time(s): ACPI: Using IOAPIC for interrupt routing
1 Time(s): ACPI: bus type PCI registered
1 Time(s): ACPI: bus type USB registered
1 Time(s): AES CTR mode by8 optimization enabled
1 Time(s): AMD AuthenticAMD
1 Time(s): AVX version of gcm_enc/dec engaged.
1 Time(s): AppArmor: AppArmor Filesystem Enabled
1 Time(s): AppArmor: AppArmor initialized
1 Time(s): AppArmor: AppArmor sha1 policy hashing enabled
1 Time(s): Booting paravirtualized kernel on KVM
1 Time(s): Btrfs loaded, crc32c=crc32c-intel
1 Time(s): Build-time adjustment of leaf fanout to 64.
1 Time(s): Built 1 zonelists in Node order, mobility grouping on.
Total pages: 4128613
1 Time(s): Calgary: Unable to locate Rio Grande table in EBDA - bailing!
1 Time(s): Calgary: detecting Calgary via BIOS EBDA area
1 Time(s): Calibrating delay loop (skipped) preset value.. 4199.99
BogoMIPS (lpj=8399992)
1 Time(s): Centaur CentaurHauls
1 Time(s): DMA zone: 21 pages reserved
1 Time(s): DMA zone: 64 pages used for memmap
1 Time(s): DMA32 zone: 12224 pages used for memmap
1 Time(s): Device empty
1 Time(s): EDD information not available.
1 Time(s): EXT4-fs (sda1): mounted filesystem with ordered data
mode.
Opts: (null)
1 Time(s): EXT4-fs (sda1): re-mounted. Opts: discard
1 Time(s): Early memory node ranges
1 Time(s): Freeing SMP alternatives memory: 32K
1 Time(s): Freeing unused kernel memory: 1156K
1 Time(s): Freeing unused kernel memory: 2228K
1 Time(s): Freeing unused kernel memory: 268K
1 Time(s): GHES: HEST is not enabled!
1 Time(s): Hierarchical RCU implementation.
1 Time(s): Hypervisor detected: KVM
1 Time(s): Initialise system trusted keyrings
1 Time(s): Intel GenuineIntel
1 Time(s): KVM setup async PF for cpu 1
1 Time(s): KVM setup async PF for cpu 2
1 Time(s): KVM setup async PF for cpu 3
1 Time(s): Key type asymmetric registered
1 Time(s): Key type big_key registered
1 Time(s): Key type dns_resolver registered
1 Time(s): Key type encrypted registered
1 Time(s): Key type trusted registered
1 Time(s): MTRR default type: write-back
1 Time(s): Magic number: 2:724:141
1 Time(s): Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
1 Time(s): Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes)
1 Time(s): Movable zone start for each node
1 Time(s): NET: Registered protocol family 1
1 Time(s): NET: Registered protocol family 16
1 Time(s): NET: Registered protocol family 17
1 Time(s): NET: Registered protocol family 2
1 Time(s): NR_IRQS:524544 nr_irqs:456 16
1 Time(s): NX (Execute Disable) protection: active
1 Time(s): NetLabel: domain hash size = 128
1 Time(s): NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
1 Time(s): NetLabel: unlabeled traffic allowed by default
1 Time(s): NetLabel: Initializing
1 Time(s): No NUMA configuration found
1 Time(s): Normal zone: 53248 pages used for memmap
1 Time(s): PCCT header not found.
1 Time(s): PCI-DMA: Using software bounce buffering for IO
(SWIOTLB)
1 Time(s): PCI: Using ACPI for IRQ routing
1 Time(s): PCI: Using configuration type 1 for base access
1 Time(s): PCI: Using host bridge windows from ACPI; if
necessary, use
"pci=nocrs" and report a bug
1 Time(s): PCI: pci_cache_line_size set to 64 bytes
1 Time(s): PM: Hibernation image not present or could not be loaded.
1 Time(s): PPP generic driver version 2.4.2
1 Time(s): Performance Events: unsupported p6 CPU model 42 no PMU
driver, software events only.
1 Time(s): Policy zone: Normal
2 Time(s): Process accounting resumed
1 Time(s): RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4.
1 Time(s): RCU: Adjusting geometry for rcu_fanout_leaf=64,
nr_cpu_ids=4
1 Time(s): SCSI subsystem initialized
1 Time(s): SMBIOS 2.8 present.
1 Time(s): Scanning 1 areas for low memory corruption
1 Time(s): Security Framework initialized
1 Time(s): Segment Routing with IPv6
1 Time(s): Switched APIC routing to physical x2apic.
1 Time(s): TSC deadline timer enabled
1 Time(s): UDP hash table entries: 8192 (order: 6, 262144 bytes)
1 Time(s): UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
1 Time(s): Unpacking initramfs...
1 Time(s): Using ACPI (MADT) for SMP configuration information
1 Time(s): Write protecting the kernel read-only data: 14336k
1 Time(s): Yama: becoming mindful.
1 Time(s): acpi device:12: hash matches
1 Time(s): acpiphp: Slot [11] registered
1 Time(s): acpiphp: Slot [12] registered
1 Time(s): acpiphp: Slot [13] registered
1 Time(s): acpiphp: Slot [14] registered
1 Time(s): acpiphp: Slot [15] registered
1 Time(s): acpiphp: Slot [16] registered
1 Time(s): acpiphp: Slot [17] registered
1 Time(s): acpiphp: Slot [18] registered
1 Time(s): acpiphp: Slot [19] registered
1 Time(s): acpiphp: Slot [21] registered
1 Time(s): acpiphp: Slot [22] registered
1 Time(s): acpiphp: Slot [23] registered
1 Time(s): acpiphp: Slot [24] registered
1 Time(s): acpiphp: Slot [25] registered
1 Time(s): acpiphp: Slot [26] registered
1 Time(s): acpiphp: Slot [27] registered
1 Time(s): acpiphp: Slot [28] registered
1 Time(s): acpiphp: Slot [29] registered
1 Time(s): acpiphp: Slot [31] registered
1 Time(s): acpiphp: Slot [3] registered
1 Time(s): acpiphp: Slot [4] registered
1 Time(s): acpiphp: Slot [5] registered
1 Time(s): acpiphp: Slot [6] registered
1 Time(s): acpiphp: Slot [7] registered
1 Time(s): acpiphp: Slot [8] registered
1 Time(s): acpiphp: Slot [9] registered
1 Time(s): async_tx: api initialized (async)
1 Time(s): audit: initializing netlink subsys (disabled)
1 Time(s): clocksource: Switched to clocksource kvm-clock
1 Time(s): cpuidle: using governor ladder
1 Time(s): cpuidle: using governor menu
1 Time(s): devtmpfs: initialized
1 Time(s): ehci-pci: EHCI PCI platform driver
1 Time(s): ehci-platform: EHCI generic platform driver
1 Time(s): evm: security.SMACK64
1 Time(s): evm: security.SMACK64EXEC
1 Time(s): evm: security.SMACK64MMAP
1 Time(s): evm: security.SMACK64TRANSMUTE
1 Time(s): evm: security.capability
1 Time(s): evm: security.ima
1 Time(s): evm: security.selinux
1 Time(s): ftrace: allocating 34227 entries in 134 pages
1 Time(s): fuse init (API version 7.26)
1 Time(s): hidraw: raw HID events driver (C) Jiri Kosina
1 Time(s): hpet clockevent registered
1 Time(s): i2c /dev entries driver
1 Time(s): ima: No TPM chip found, activating TPM-bypass! (rc=-19)
1 Time(s): intel_idle: does not run on family 6 model 42
1 Time(s): io scheduler cfq registered
1 Time(s): io scheduler deadline registered
1 Time(s): io scheduler noop registered (default)
1 Time(s): ledtrig-cpu: registered to indicate activity on CPUs
1 Time(s): libphy: Fixed MDIO Bus: probed
1 Time(s): loop: module loaded
1 Time(s): mousedev: PS/2 mouse device common for all mice
1 Time(s): ohci-pci: OHCI PCI platform driver
1 Time(s): ohci-platform: OHCI generic platform driver
1 Time(s): ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
1 Time(s): pinctrl core: initialized pinctrl subsystem
1 Time(s): pnp: PnP ACPI init
1 Time(s): pnp: PnP ACPI: found 5 devices
1 Time(s): ppdev: user-space parallel port driver
1 Time(s): raid6: .... xor() 6424 MB/s, rmw enabled
1 Time(s): raid6: sse2x1 gen() 6723 MB/s
1 Time(s): raid6: sse2x1 xor() 5185 MB/s
1 Time(s): raid6: sse2x2 gen() 8399 MB/s
1 Time(s): raid6: sse2x2 xor() 5585 MB/s
1 Time(s): raid6: sse2x4 gen() 9942 MB/s
1 Time(s): raid6: sse2x4 xor() 6424 MB/s
1 Time(s): raid6: using algorithm sse2x4 gen() 9942 MB/s
1 Time(s): raid6: using ssse3x2 recovery algorithm
1 Time(s): random: crng init done
1 Time(s): random: fast init done
8 Time(s): random: systemd-udevd: uninitialized urandom read (16 bytes read)
2 Time(s): random: udevadm: uninitialized urandom read (16 bytes read)
1 Time(s): registered taskstats version 1
1 Time(s): scsi host1: ata_piix
1 Time(s): scsi host2: Virtio SCSI HBA
1 Time(s): sda: sda1
1 Time(s): setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4
nr_cpu_ids:4
nr_node_ids:1
1 Time(s): smp: Bringing up secondary CPUs ...
1 Time(s): smp: Brought up 1 node, 4 CPUs
1 Time(s): smpboot: Max logical packages: 1
1 Time(s): smpboot: Total of 4 processors activated (16799.98 BogoMIPS)
1 Time(s): tun: Universal TUN/TAP device driver, 1.6
1 Time(s): uhci_hcd: USB Universal Host Controller Interface driver
1 Time(s): usb 1-1: Manufacturer: QEMU
1 Time(s): usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=5
1 Time(s): usb 1-1: Product: QEMU USB Tablet
1 Time(s): usb 1-1: SerialNumber: 42
1 Time(s): usb 1-1: new full-speed USB device number 2 using uhci_hcd
1 Time(s): usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
1 Time(s): usb usb1: Product: UHCI Host Controller
1 Time(s): usbcore: registered new device driver usb
1 Time(s): usbcore: registered new interface driver hub
1 Time(s): usbcore: registered new interface driver usbfs
1 Time(s): usbcore: registered new interface driver usbhid
1 Time(s): usbhid: USB HID core driver
1 Time(s): vgaarb: loaded
1 Time(s): x2apic enabled
1 Time(s): x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
1 Time(s): x86/mm: Checked W+X mappings: passed, no W+X pages found.
1 Time(s): x86/mm: Memory block size: 128MB
1 Time(s): xor: automatically using best checksumming function
avx
1 Time(s): zbud: loaded
1 Time(s): zswap: loaded using pool lzo/zbud
---------------------- Kernel End -------------------------

Post by Marcel Bischoff

Post by Skale, Franz
06:21 is the lograotation, so no problem. Same by me.

A bit reassuring but still: shouldn't the service cleanly restart
instead of barfing "Disorderly Shutdown"?

Post by Skale, Franz
What strucks me is, that it seems that ns-slapd as to reallocate memory.
How much memory does your server have ?
send free -m

$ free -m
total used free shared buff/cache
available
Mem: 16045 1304 13096 56
1644 14371
Swap: 0 0 0
No problem I can see there.

Post by Skale, Franz
Do you have selinux enabled !
If so, disable it by adding selinux=0 to /etc/default/grub and rerun
update-grub.

No, not enabled.

Post by Skale, Franz
Send the kernel version: uname -a

Linux mx.example.com 4.10.0-42-generic #46~16.04.1-Ubuntu SMP Mon Dec 4
15:57:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Post by Skale, Franz
How much open file handles to your system allow per process ?
send: ulimit -a

$ ulimit -a
Maximum size of core files created (kB, -c) 0
Maximum size of a process’s data segment (kB, -d) unlimited
Maximum size of files created by the shell (kB, -f) unlimited
Maximum size that may be locked into memory (kB, -l) 64
Maximum resident set size (kB, -m) unlimited
Maximum number of open file descriptors
(-n) 1024
Maximum stack size (kB, -s) 8192
Maximum amount of cpu time in seconds (seconds, -t) unlimited
Maximum number of processes available to a single user
(-u) 64015
Maximum amount of virtual memory available to the shell (kB, -v) unlimited

Post by Skale, Franz
send dmesg: (is there a segfault).

The whole dmesg output is spammed by ufw and contains no useful
information whatsoever.

Post by Skale, Franz
send dmidecode

I don't think this is likely as this is a virtual server.
$ dmidecode
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.
10 structures occupying 408 bytes.
Table at 0x000F68A0.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: SeaBIOS
Version: 1.10.2
Release Date: 04/01/2014
Address: 0xE8000
Runtime Size: 96 kB
ROM Size: 64 kB
BIOS characteristics not supported
Targeted content distribution is supported
BIOS Revision: 0.0
Handle 0x0100, DMI type 1, 27 bytes
System Information
Manufacturer: Hetzner
Product Name: vServer
Version: 2
Serial Number: Not Specified
UUID: A8236400-D36B-0135-FE8F-10BF48D7F2C6
Wake-up Type: Power Switch
SKU Number: a8236400-d36b-0135-fe8f-10bf48d7f2c6
Family: Not Specified
Handle 0x0300, DMI type 3, 21 bytes
Chassis Information
Manufacturer: QEMU
Type: Other
Lock: Not Present
Version: pc-i440fx-2.10
Serial Number: Not Specified
Asset Tag: Not Specified
Boot-up State: Safe
Power Supply State: Safe
Thermal State: Safe
Security Status: Unknown
OEM Information: 0x00000000
Height: Unspecified
Number Of Power Cords: Unspecified
Contained Elements: 0
Handle 0x0400, DMI type 4, 42 bytes
Processor Information
Socket Designation: CPU 0
Type: Central Processor
Family: Other
Manufacturer: QEMU
ID: A1 06 02 00 FF FB 8B 07
Version: pc-i440fx-2.10
Voltage: Unknown
External Clock: Unknown
Max Speed: 2000 MHz
Current Speed: 2000 MHz
Status: Populated, Enabled
Upgrade: Other
L1 Cache Handle: Not Provided
L2 Cache Handle: Not Provided
L3 Cache Handle: Not Provided
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Core Count: 4
Core Enabled: 4
Thread Count: 1
Characteristics: None
Handle 0x1000, DMI type 16, 23 bytes
Physical Memory Array
Location: Other
Use: System Memory
Error Correction Type: Multi-bit ECC
Maximum Capacity: 16 GB
Error Information Handle: Not Provided
Number Of Devices: 1
Handle 0x1100, DMI type 17, 40 bytes
Memory Device
Array Handle: 0x1000
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: 16384 MB
Form Factor: DIMM
Set: None
Locator: DIMM 0
Bank Locator: Not Specified
Type: RAM
Type Detail: Other
Speed: Unknown
Manufacturer: QEMU
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Rank: Unknown
Configured Clock Speed: Unknown
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown
Handle 0x1300, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00000000000
Ending Address: 0x000BFFFFFFF
Range Size: 3 GB
Physical Array Handle: 0x1000
Partition Width: 1
Handle 0x1301, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00100000000
Ending Address: 0x0043FFFFFFF
Range Size: 13 GB
Physical Array Handle: 0x1000
Partition Width: 1
Handle 0x2000, DMI type 32, 11 bytes
System Boot Information
Status: No errors detected
Handle 0x7F00, DMI type 127, 4 bytes
End Of Table

Post by Skale, Franz
Did you update your kernel days ago, if so, you sure ran
into a buggy
kernel 4.9.65.
I built a 4.9.75 PTI enabled kernel which i send you to test.

4.10.0-42-generic

Post by Skale, Franz
Office 365 would be a bad and expensive choice.

I agree. I'd very much like to avoid it but when Kolab turn out to have
issues with the current setup, I doubt I'll get another shot.
Best,
Marcel

Marcel Bischoff

2018-01-11 15:06:58 UTC

Permalink

Hey Franz,

I adjusted the limits and hope for the best now. Thanks again for taking
the time to troubleshoot this with me!

Best,
Marcel

Post by Skale, Franz
Hi Marcel,
as a matter of fact, there's a major change in the so called kaiser
patch (KPTI) for users who use KVM and linux guests. There i build a
new kernel 4.9.75 as of 06.01.2018. You have to patch the Host and the
guest though.
Good to hear that the unordered shutdown disappeared. I also don't
have unordered shutdowns using the new 389 base package.
Tuning the limits will sure help !
Rgds.
Franz

Post by Marcel Bischoff
Hi Franz,
Reading package lists... Done
Building dependency tree
Reading state information... Done
389-admin 389-admin-console 389-console 389-ds 389-ds-base
389-ds-base-libs 389-ds-console 389-ds-console-doc 389-dsgw
amavisd-new aspell aspell-en augeas-lenses chwala
clamav clamav-base clamav-daemon clamav-freshclam clamdscan
cyrus-imapd dictionaries-common emacsen-common erlang-base
erlang-crypto erlang-eimap erlang-goldrush erlang-lager
erlang-lager-syslog erlang-syntax-tools fontconfig guam irony
kolab-cli kolab-conf kolab-freebusy kolab-imap kolab-ldap kolab-mta
kolab-saslauthd kolab-schema kolab-server
kolab-syncroton kolab-webadmin kolab-xml ldap-utils libadminutil-data
libadminutil0 libapache2-mod-nss libapache2-mod-php
libapache2-mod-php7.0 libapparmor-perl libaspell15
libaudio2 libaugeas0 libauthen-sasl-perl libavahi-client3
libavahi-common-data libavahi-common3 libberkeleydb-perl
libcalendaring libcgi-fast-perl libcgi-pm-perl libclamav7
libconvert-binhex-perl libconvert-tnef-perl libconvert-uulib-perl
libcrypt-openssl-bignum-perl libcrypt-openssl-rsa-perl libcups2
libdigest-hmac-perl libds-admin-serv0
libencode-locale-perl libevent-core-2.0-5 libfcgi-perl libgd3
libhtml-parser-perl libhtml-tagset-perl libhtml-template-perl
libhttp-date-perl libhttp-message-perl libical1a
libidm-console-framework-java libio-html-perl libio-multiplex-perl
libio-socket-inet6-perl libio-socket-ssl-perl libio-stringy-perl
libjansson4 libjbig0 libjpeg-turbo8
libjpeg8 libjss-java libkolab2 libkolabxml1v5 liblcms2-2 libldap-java
libllvm3.6v5 liblwp-mediatypes-perl libmail-dkim-perl
libmail-spf-perl
libmailtools-perl libmcrypt4
libmime-tools-perl libmng2 libmozilla-ldap-perl libmozldap-0d
libnet-cidr-perl libnet-dns-perl libnet-ip-perl libnet-libidn-perl
libnet-server-perl libnet-smtp-ssl-perl
libnet-ssleay-perl libnetaddr-ip-perl libnss3-tools
libperl4-corelibs-perl libqt4-dbus libqt4-declarative libqt4-network
libqt4-script libqt4-sql libqt4-sql-mysql libqt4-xml
libqt4-xmlpatterns libqtcore4 libqtdbus4 libqtgui4 libsctp1
libsocket-getaddrinfo-perl libsocket6-perl libtiff5
libunix-syslog-perl liburi-perl libvpx3 libxerces-c3.1
libxslt1.1 libzend-framework-php libzephyr4 mozldap-tools
mysql-client mysql-client-5.7 mysql-client-core-5.7
mysql-server-core-5.7 pax php php-auth-sasl php-cli php-common
php-curl php-gd php-http-request2 php-intl php-kolab php-kolabformat
php-ldap php-mail php-mail-mime php-mail-mimedecode php-mbstring
php-mcrypt php-mdb2
php-mdb2-driver-mysql php-monolog php-mysql php-net-idna2
php-net-ldap2 php-net-ldap3 php-net-sieve php-net-smtp php-net-socket
php-net-url2 php-pear php-pspell php-psr-log
php-sabre-dav-2.1 php-sabre-event php-sabre-http-3
php-sabre-vobject-3 php-xml php7.0 php7.0-cli php7.0-common
php7.0-curl php7.0-fpm php7.0-gd php7.0-intl php7.0-json
php7.0-ldap php7.0-mbstring php7.0-mcrypt php7.0-mysql php7.0-opcache
php7.0-pspell php7.0-readline php7.0-xml pykolab python-augeas
python-cheetah python-dateutil
python-gnupg python-icalendar python-kolab python-kolabformat
python-ldap python-pkg-resources python-pyasn1 python-pyasn1-modules
python-pymysql python-six python-sqlalchemy
python-sqlalchemy-ext python-tz python-tzlocal qdbus qt-at-spi
qtchooser qtcore4-l10n re2c roundcubemail roundcubemail-core
roundcubemail-plugin-acl
roundcubemail-plugin-archive roundcubemail-plugin-calendar
roundcubemail-plugin-contextmenu
roundcubemail-plugin-filesystem-attachments
roundcubemail-plugin-jqueryui
roundcubemail-plugin-kolab-activesync
roundcubemail-plugin-kolab-addressbook
roundcubemail-plugin-kolab-auth
roundcubemail-plugin-kolab-config
roundcubemail-plugin-kolab-delegation
roundcubemail-plugin-kolab-files roundcubemail-plugin-kolab-folders
roundcubemail-plugin-kolab-notes roundcubemail-plugin-kolab-tags
roundcubemail-plugin-libcalendaring roundcubemail-plugin-libkolab
roundcubemail-plugin-managesieve
roundcubemail-plugin-newmail-notifier
roundcubemail-plugin-odfviewer
roundcubemail-plugin-password roundcubemail-plugin-pdfviewer
roundcubemail-plugin-redundant-attachments
roundcubemail-plugin-tasklist roundcubemail-plugin-zipdownload
roundcubemail-plugins-kolab roundcubemail-skin-chameleon sa-compile
smarty3 spamassassin spamc wallace zend-framework zend-framework-bin
Use 'apt autoremove' to remove them.
apparmor kolab kolab-webclient mysql-server mysql-server-5.7
0 upgraded, 0 newly installed, 5 to remove and 0 not upgraded.
After this operation, 50.2 MB disk space will be freed.
Do you want to continue? [Y/n]
Which amounts to basically... everything.
apparmor module is loaded.
0 profiles are loaded.
0 profiles are in enforce mode.
0 profiles are in complain mode.
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
So that should be alright I guess.
I have raised the limits and will monitor the outcome. Hopefully things
will quiet down now.
Best,
Marcel

Post by Marcel Bischoff
I have been able to pull the kernel messages from the logwatch output,
didn't think of this before. Maybe it helps in homing in on the cause. I
couldn't spot something obvious though. Maybe AppArmor does something
undesirable? I remember running into issues with it years ago in another
context.
I will uninstall it tonight, reboot the server and report on my progress
(or lack thereof).
Thanks again for bearing with me!
--------------------- Kernel Begin ------------------------
1 Time(s): #2
1 Time(s): #3
1 Time(s): 1 disabled
1 Time(s): 2 disabled
1 Time(s): 3 disabled
1 Time(s): 4 disabled
1 Time(s): 5 disabled
1 Time(s): 6 disabled
1 Time(s): 7 disabled
1 Time(s): ACPI: 1 ACPI AML tables successfully acquired and loaded
1 Time(s): ACPI: Added _OSI(Module Device)
1 Time(s): ACPI: Added _OSI(Processor Aggregator Device)
1 Time(s): ACPI: Added _OSI(Processor Device)
1 Time(s): ACPI: Early table checksum verification disabled
1 Time(s): ACPI: IRQ11 used by override.
1 Time(s): ACPI: IRQ5 used by override.
1 Time(s): ACPI: IRQ9 used by override.
1 Time(s): ACPI: Interpreter enabled
1 Time(s): ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
1 Time(s): ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
1 Time(s): ACPI: Power Button [PWRF]
1 Time(s): ACPI: Using IOAPIC for interrupt routing
1 Time(s): ACPI: bus type PCI registered
1 Time(s): ACPI: bus type USB registered
1 Time(s): AES CTR mode by8 optimization enabled
1 Time(s): AMD AuthenticAMD
1 Time(s): AVX version of gcm_enc/dec engaged.
1 Time(s): AppArmor: AppArmor Filesystem Enabled
1 Time(s): AppArmor: AppArmor initialized
1 Time(s): AppArmor: AppArmor sha1 policy hashing enabled
1 Time(s): Booting paravirtualized kernel on KVM
1 Time(s): Btrfs loaded, crc32c=crc32c-intel
1 Time(s): Build-time adjustment of leaf fanout to 64.
1 Time(s): Built 1 zonelists in Node order, mobility grouping on.
Total pages: 4128613
1 Time(s): Calgary: Unable to locate Rio Grande table in EBDA - bailing!
1 Time(s): Calgary: detecting Calgary via BIOS EBDA area
1 Time(s): Calibrating delay loop (skipped) preset value.. 4199.99
BogoMIPS (lpj=8399992)
1 Time(s): Centaur CentaurHauls
1 Time(s): DMA zone: 21 pages reserved
1 Time(s): DMA zone: 64 pages used for memmap
1 Time(s): DMA32 zone: 12224 pages used for memmap
1 Time(s): Device empty
1 Time(s): EDD information not available.
1 Time(s): EXT4-fs (sda1): mounted filesystem with ordered
data mode.
Opts: (null)
1 Time(s): EXT4-fs (sda1): re-mounted. Opts: discard
1 Time(s): Early memory node ranges
1 Time(s): Freeing SMP alternatives memory: 32K
1 Time(s): Freeing unused kernel memory: 1156K
1 Time(s): Freeing unused kernel memory: 2228K
1 Time(s): Freeing unused kernel memory: 268K
1 Time(s): GHES: HEST is not enabled!
1 Time(s): Hierarchical RCU implementation.
1 Time(s): Hypervisor detected: KVM
1 Time(s): Initialise system trusted keyrings
1 Time(s): Intel GenuineIntel
1 Time(s): KVM setup async PF for cpu 1
1 Time(s): KVM setup async PF for cpu 2
1 Time(s): KVM setup async PF for cpu 3
1 Time(s): Key type asymmetric registered
1 Time(s): Key type big_key registered
1 Time(s): Key type dns_resolver registered
1 Time(s): Key type encrypted registered
1 Time(s): Key type trusted registered
1 Time(s): MTRR default type: write-back
1 Time(s): Magic number: 2:724:141
1 Time(s): Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
1 Time(s): Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes)
1 Time(s): Movable zone start for each node
1 Time(s): NET: Registered protocol family 1
1 Time(s): NET: Registered protocol family 16
1 Time(s): NET: Registered protocol family 17
1 Time(s): NET: Registered protocol family 2
1 Time(s): NR_IRQS:524544 nr_irqs:456 16
1 Time(s): NX (Execute Disable) protection: active
1 Time(s): NetLabel: domain hash size = 128
1 Time(s): NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
1 Time(s): NetLabel: unlabeled traffic allowed by default
1 Time(s): NetLabel: Initializing
1 Time(s): No NUMA configuration found
1 Time(s): Normal zone: 53248 pages used for memmap
1 Time(s): PCCT header not found.
1 Time(s): PCI-DMA: Using software bounce buffering for IO
(SWIOTLB)
1 Time(s): PCI: Using ACPI for IRQ routing
1 Time(s): PCI: Using configuration type 1 for base access
1 Time(s): PCI: Using host bridge windows from ACPI; if
necessary, use
"pci=nocrs" and report a bug
1 Time(s): PCI: pci_cache_line_size set to 64 bytes
1 Time(s): PM: Hibernation image not present or could not be loaded.
1 Time(s): PPP generic driver version 2.4.2
1 Time(s): Performance Events: unsupported p6 CPU model 42 no PMU
driver, software events only.
1 Time(s): Policy zone: Normal
2 Time(s): Process accounting resumed
1 Time(s): RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4.
1 Time(s): RCU: Adjusting geometry for rcu_fanout_leaf=64,
nr_cpu_ids=4
1 Time(s): SCSI subsystem initialized
1 Time(s): SMBIOS 2.8 present.
1 Time(s): Scanning 1 areas for low memory corruption
1 Time(s): Security Framework initialized
1 Time(s): Segment Routing with IPv6
1 Time(s): Switched APIC routing to physical x2apic.
1 Time(s): TSC deadline timer enabled
1 Time(s): UDP hash table entries: 8192 (order: 6, 262144 bytes)
1 Time(s): UDP-Lite hash table entries: 8192 (order: 6,
262144 bytes)
1 Time(s): Unpacking initramfs...
1 Time(s): Using ACPI (MADT) for SMP configuration information
1 Time(s): Write protecting the kernel read-only data: 14336k
1 Time(s): Yama: becoming mindful.
1 Time(s): acpi device:12: hash matches
1 Time(s): acpiphp: Slot [11] registered
1 Time(s): acpiphp: Slot [12] registered
1 Time(s): acpiphp: Slot [13] registered
1 Time(s): acpiphp: Slot [14] registered
1 Time(s): acpiphp: Slot [15] registered
1 Time(s): acpiphp: Slot [16] registered
1 Time(s): acpiphp: Slot [17] registered
1 Time(s): acpiphp: Slot [18] registered
1 Time(s): acpiphp: Slot [19] registered
1 Time(s): acpiphp: Slot [21] registered
1 Time(s): acpiphp: Slot [22] registered
1 Time(s): acpiphp: Slot [23] registered
1 Time(s): acpiphp: Slot [24] registered
1 Time(s): acpiphp: Slot [25] registered
1 Time(s): acpiphp: Slot [26] registered
1 Time(s): acpiphp: Slot [27] registered
1 Time(s): acpiphp: Slot [28] registered
1 Time(s): acpiphp: Slot [29] registered
1 Time(s): acpiphp: Slot [31] registered
1 Time(s): acpiphp: Slot [3] registered
1 Time(s): acpiphp: Slot [4] registered
1 Time(s): acpiphp: Slot [5] registered
1 Time(s): acpiphp: Slot [6] registered
1 Time(s): acpiphp: Slot [7] registered
1 Time(s): acpiphp: Slot [8] registered
1 Time(s): acpiphp: Slot [9] registered
1 Time(s): async_tx: api initialized (async)
1 Time(s): audit: initializing netlink subsys (disabled)
1 Time(s): clocksource: Switched to clocksource kvm-clock
1 Time(s): cpuidle: using governor ladder
1 Time(s): cpuidle: using governor menu
1 Time(s): devtmpfs: initialized
1 Time(s): ehci-pci: EHCI PCI platform driver
1 Time(s): ehci-platform: EHCI generic platform driver
1 Time(s): evm: security.SMACK64
1 Time(s): evm: security.SMACK64EXEC
1 Time(s): evm: security.SMACK64MMAP
1 Time(s): evm: security.SMACK64TRANSMUTE
1 Time(s): evm: security.capability
1 Time(s): evm: security.ima
1 Time(s): evm: security.selinux
1 Time(s): ftrace: allocating 34227 entries in 134 pages
1 Time(s): fuse init (API version 7.26)
1 Time(s): hidraw: raw HID events driver (C) Jiri Kosina
1 Time(s): hpet clockevent registered
1 Time(s): i2c /dev entries driver
1 Time(s): ima: No TPM chip found, activating TPM-bypass! (rc=-19)
1 Time(s): intel_idle: does not run on family 6 model 42
1 Time(s): io scheduler cfq registered
1 Time(s): io scheduler deadline registered
1 Time(s): io scheduler noop registered (default)
1 Time(s): ledtrig-cpu: registered to indicate activity on CPUs
1 Time(s): libphy: Fixed MDIO Bus: probed
1 Time(s): loop: module loaded
1 Time(s): mousedev: PS/2 mouse device common for all mice
1 Time(s): ohci-pci: OHCI PCI platform driver
1 Time(s): ohci-platform: OHCI generic platform driver
1 Time(s): ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
1 Time(s): pinctrl core: initialized pinctrl subsystem
1 Time(s): pnp: PnP ACPI init
1 Time(s): pnp: PnP ACPI: found 5 devices
1 Time(s): ppdev: user-space parallel port driver
1 Time(s): raid6: .... xor() 6424 MB/s, rmw enabled
1 Time(s): raid6: sse2x1 gen() 6723 MB/s
1 Time(s): raid6: sse2x1 xor() 5185 MB/s
1 Time(s): raid6: sse2x2 gen() 8399 MB/s
1 Time(s): raid6: sse2x2 xor() 5585 MB/s
1 Time(s): raid6: sse2x4 gen() 9942 MB/s
1 Time(s): raid6: sse2x4 xor() 6424 MB/s
1 Time(s): raid6: using algorithm sse2x4 gen() 9942 MB/s
1 Time(s): raid6: using ssse3x2 recovery algorithm
1 Time(s): random: crng init done
1 Time(s): random: fast init done
8 Time(s): random: systemd-udevd: uninitialized urandom read (16 bytes read)
2 Time(s): random: udevadm: uninitialized urandom read (16 bytes read)
1 Time(s): registered taskstats version 1
1 Time(s): scsi host1: ata_piix
1 Time(s): scsi host2: Virtio SCSI HBA
1 Time(s): sda: sda1
1 Time(s): setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4
nr_cpu_ids:4
nr_node_ids:1
1 Time(s): smp: Bringing up secondary CPUs ...
1 Time(s): smp: Brought up 1 node, 4 CPUs
1 Time(s): smpboot: Max logical packages: 1
1 Time(s): smpboot: Total of 4 processors activated (16799.98 BogoMIPS)
1 Time(s): tun: Universal TUN/TAP device driver, 1.6
1 Time(s): uhci_hcd: USB Universal Host Controller Interface driver
1 Time(s): usb 1-1: Manufacturer: QEMU
1 Time(s): usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=5
1 Time(s): usb 1-1: Product: QEMU USB Tablet
1 Time(s): usb 1-1: SerialNumber: 42
1 Time(s): usb 1-1: new full-speed USB device number 2 using uhci_hcd
1 Time(s): usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
1 Time(s): usb usb1: Product: UHCI Host Controller
1 Time(s): usbcore: registered new device driver usb
1 Time(s): usbcore: registered new interface driver hub
1 Time(s): usbcore: registered new interface driver usbfs
1 Time(s): usbcore: registered new interface driver usbhid
1 Time(s): usbhid: USB HID core driver
1 Time(s): vgaarb: loaded
1 Time(s): x2apic enabled
1 Time(s): x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
1 Time(s): x86/mm: Checked W+X mappings: passed, no W+X
pages found.
1 Time(s): x86/mm: Memory block size: 128MB
1 Time(s): xor: automatically using best checksumming
function avx
1 Time(s): zbud: loaded
1 Time(s): zswap: loaded using pool lzo/zbud
---------------------- Kernel End -------------------------

Post by Marcel Bischoff

Post by Skale, Franz
06:21 is the lograotation, so no problem. Same by me.

A bit reassuring but still: shouldn't the service cleanly restart
instead of barfing "Disorderly Shutdown"?

Post by Skale, Franz
What strucks me is, that it seems that ns-slapd as to reallocate memory.
How much memory does your server have ?
send free -m

$ free -m
total used free shared buff/cache
available
Mem: 16045 1304 13096 56
1644 14371
Swap: 0 0 0
No problem I can see there.

Post by Skale, Franz
Do you have selinux enabled !
If so, disable it by adding selinux=0 to /etc/default/grub and rerun
update-grub.

No, not enabled.

Post by Skale, Franz
Send the kernel version: uname -a

Linux mx.example.com 4.10.0-42-generic #46~16.04.1-Ubuntu SMP Mon Dec 4
15:57:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Post by Skale, Franz
How much open file handles to your system allow per process ?
send: ulimit -a

$ ulimit -a
Maximum size of core files created (kB, -c) 0
Maximum size of a process’s data segment (kB, -d) unlimited
Maximum size of files created by the shell (kB, -f) unlimited
Maximum size that may be locked into memory (kB, -l) 64
Maximum resident set size (kB, -m) unlimited
Maximum number of open file descriptors
(-n) 1024
Maximum stack size (kB, -s) 8192
Maximum amount of cpu time in seconds (seconds, -t) unlimited
Maximum number of processes available to a single user
(-u) 64015
Maximum amount of virtual memory available to the shell (kB, -v) unlimited

Post by Skale, Franz
send dmesg: (is there a segfault).

The whole dmesg output is spammed by ufw and contains no useful
information whatsoever.

Post by Skale, Franz
send dmidecode

I don't think this is likely as this is a virtual server.
$ dmidecode
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.
10 structures occupying 408 bytes.
Table at 0x000F68A0.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: SeaBIOS
Version: 1.10.2
Release Date: 04/01/2014
Address: 0xE8000
Runtime Size: 96 kB
ROM Size: 64 kB
BIOS characteristics not supported
Targeted content distribution is supported
BIOS Revision: 0.0
Handle 0x0100, DMI type 1, 27 bytes
System Information
Manufacturer: Hetzner
Product Name: vServer
Version: 2
Serial Number: Not Specified
UUID: A8236400-D36B-0135-FE8F-10BF48D7F2C6
Wake-up Type: Power Switch
SKU Number: a8236400-d36b-0135-fe8f-10bf48d7f2c6
Family: Not Specified
Handle 0x0300, DMI type 3, 21 bytes
Chassis Information
Manufacturer: QEMU
Type: Other
Lock: Not Present
Version: pc-i440fx-2.10
Serial Number: Not Specified
Asset Tag: Not Specified
Boot-up State: Safe
Power Supply State: Safe
Thermal State: Safe
Security Status: Unknown
OEM Information: 0x00000000
Height: Unspecified
Number Of Power Cords: Unspecified
Contained Elements: 0
Handle 0x0400, DMI type 4, 42 bytes
Processor Information
Socket Designation: CPU 0
Type: Central Processor
Family: Other
Manufacturer: QEMU
ID: A1 06 02 00 FF FB 8B 07
Version: pc-i440fx-2.10
Voltage: Unknown
External Clock: Unknown
Max Speed: 2000 MHz
Current Speed: 2000 MHz
Status: Populated, Enabled
Upgrade: Other
L1 Cache Handle: Not Provided
L2 Cache Handle: Not Provided
L3 Cache Handle: Not Provided
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Core Count: 4
Core Enabled: 4
Thread Count: 1
Characteristics: None
Handle 0x1000, DMI type 16, 23 bytes
Physical Memory Array
Location: Other
Use: System Memory
Error Correction Type: Multi-bit ECC
Maximum Capacity: 16 GB
Error Information Handle: Not Provided
Number Of Devices: 1
Handle 0x1100, DMI type 17, 40 bytes
Memory Device
Array Handle: 0x1000
Error Information Handle: Not Provided
Total Width: Unknown
Data Width: Unknown
Size: 16384 MB
Form Factor: DIMM
Set: None
Locator: DIMM 0
Bank Locator: Not Specified
Type: RAM
Type Detail: Other
Speed: Unknown
Manufacturer: QEMU
Serial Number: Not Specified
Asset Tag: Not Specified
Part Number: Not Specified
Rank: Unknown
Configured Clock Speed: Unknown
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown
Handle 0x1300, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00000000000
Ending Address: 0x000BFFFFFFF
Range Size: 3 GB
Physical Array Handle: 0x1000
Partition Width: 1
Handle 0x1301, DMI type 19, 31 bytes
Memory Array Mapped Address
Starting Address: 0x00100000000
Ending Address: 0x0043FFFFFFF
Range Size: 13 GB
Physical Array Handle: 0x1000
Partition Width: 1
Handle 0x2000, DMI type 32, 11 bytes
System Boot Information
Status: No errors detected
Handle 0x7F00, DMI type 127, 4 bytes
End Of Table

Post by Skale, Franz
Did you update your kernel days ago, if so, you sure ran
into a buggy
kernel 4.9.65.
I built a 4.9.75 PTI enabled kernel which i send you to test.

4.10.0-42-generic

Post by Skale, Franz
Office 365 would be a bad and expensive choice.

I agree. I'd very much like to avoid it but when Kolab turn out to have
issues with the current setup, I doubt I'll get another shot.
Best,
Marcel

Continue reading on narkive:

Search results for 'LDAP server unavailable: SERVER_DOWN' (Questions and Answers)

replies

what is internet love ?

started 2006-02-11 18:44:19 UTC

family & relationships

replies

what is the internet?

started 2006-01-28 05:26:33 UTC

internet

replies

who win the match for jonh and randy ortan?

started 2007-08-19 06:00:21 UTC

rugby league