Stephen Switzer
2016-09-12 04:13:19 UTC
I recently was looking at 9folders (android client) and why I can't see
any folders on it other than inbox & sent. I decided to see what updates
were available on Kolab16 under Centos. After the update, I'm now
running 7.2.1511.
Once the update was done, I rebooted, and quickly noticed that I could
not send email.I sa errors like this in the maillog:
Sep 11 23:01:14 kolab16 postfix/trivial-rewrite[11366]: warning:
ldap:/etc/postfix/ldap/mydestination.cf: table lookup problem
Sep 11 23:01:16 kolab16 imaps[3445]: badlogin: localhost [127.0.0.1]
login [SASL(-13): authentication failure: checkpass failed]
Sep 11 23:01:24 kolab16 postfix/trivial-rewrite[11366]: warning:
dict_ldap_connect: Unable to bind to server ldap://localhost:389 with dn
uid=kolab-service,ou=Special Users,dc=sbsroc,dc=com: -5 (Timed out)
Somehow I didn't notice the ldap error at first glance and looked at
other things, noticing the following in /var/log/kolab/pykolab.log:
2016-09-11 23:04:02,420 pykolab.wallace ERROR Module
resources.heartbeat() failed with error: Traceback (most recent call
last):
File "/usr/lib/python2.7/site-packages/wallace/__init__.py", line 89,
in modules_heartbeat
modules.heartbeat(module, lastrun)
File "/usr/lib/python2.7/site-packages/wallace/modules.py", line 128,
in heartbeat
return modules[name]['heartbeat'](*args, **kw)
File "/usr/lib/python2.7/site-packages/wallace/module_resources.py",
line 417, in heartbeat
resource_dns = auth.find_resource('*')
File "/usr/lib/python2.7/site-packages/pykolab/auth/__init__.py", line
220, in find_resource
result = self._auth.find_resource(address)
File "/usr/lib/python2.7/site-packages/pykolab/auth/ldap/__init__.py",
line 765, in find_resource
self._bind()
File "/usr/lib/python2.7/site-packages/pykolab/auth/ldap/__init__.py",
line 1435, in _bind
self.ldap.simple_bind_s(bind_dn, bind_pw)
File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
837, in simple_bind_s
res =
self._apply_method_s(SimpleLDAPObject.simple_bind_s,*args,**kwargs)
File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
818, in _apply_method_s
return func(self,*args,**kwargs)
File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
208, in simple_bind_s
resp_type, resp_data, resp_msgid, resp_ctrls =
self.result3(msgid,all=1,timeout=self.timeout)
File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
469, in result3
resp_ctrl_classes=resp_ctrl_classes
File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
476, in result4
ldap_result =
self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 99,
in _ldap_call
result = func(*args,**kwargs)
TIMEOUT
When I finally honed in on the ldap server, I tried to start it:
[***@kolab16 ~]# systemctl restart ***@service.service
Job for ***@service.service failed because a configured resource
limit was exceeded. See "systemctl status ***@service.service" and
"journalctl -xe" for details.
So I ran the suggestion:
[***@kolab16 ~]# journalctl -xe
Sep 11 23:17:44 kolab16.sbsllc.local imaps[12089]: starttls: TLSv1.2
with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits reused) no
authentication
Sep 11 23:17:46 kolab16.sbsllc.local imaps[11890]: badlogin: localhost
[127.0.0.1] plaintext ***@mydomain.com SASL(-13): authentication
failure: checkpass fai
Sep 11 23:18:14 kolab16.sbsllc.local imaps[12089]: timeout_select:
reading from ptloader: Connection timed out
Sep 11 23:18:14 kolab16.sbsllc.local imaps[12089]: ptload failed: but
canonified ***@mydomain.com -> ***@mydomain.com
Sep 11 23:18:16 kolab16.sbsllc.local polkitd[926]: Registered
Authentication Agent for unix-process:12166:120602 (system bus name
:1.36 [/usr/bin/pkttyagent
Sep 11 23:18:16 kolab16.sbsllc.local systemd[1]: Failed to load
environment files: No such file or directory
Sep 11 23:18:16 kolab16.sbsllc.local systemd[1]: ***@service.service
failed to run 'start' task: No such file or directory
Sep 11 23:18:16 kolab16.sbsllc.local systemd[1]: Failed to start 389
Directory Server service..
-- Subject: Unit ***@service.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit ***@service.service has failed.
--
-- The result is failed.
The 389 server is not running... I couldn't log in to roundcube, could
send email with thunderbird, etc. the web admin panel also didn't let me
log in. Since the auth backend is down, that's expected.
I rolled back to a snapshot of this virtual machine, and booted it up...
same issue!! Since the old versio behaved the same way, I went back to
the newly updated snapshot. I figure if it's failing, I might as well
work on the latest version.
After letting it sit for a while during my google research, I received a
"new mail" desktop notification from my browser. Uhmmmmm. It started
working on its own!
I checked the service status, and it still says it failed:
[***@kolab16 ~]# systemctl status ***@service.service
â ***@service.service - 389 Directory Server service.
Loaded: loaded (/usr/lib/systemd/system/***@.service; enabled;
vendor preset: disabled)
Active: failed (Result: resources)
Sep 11 23:10:56 kolab16.sbsllc.local systemd[1]: Failed to load
environment files: No such file or directory
Sep 11 23:10:56 kolab16.sbsllc.local systemd[1]: ***@service.service
failed to run 'start' task: No such file or directory
Sep 11 23:10:56 kolab16.sbsllc.local systemd[1]: Failed to start 389
Directory Server service..
Sep 11 23:10:56 kolab16.sbsllc.local systemd[1]: ***@service.service
failed.
Sep 11 23:10:56 kolab16.sbsllc.local systemd[1]: Starting 389 Directory
Server service....
Sep 11 23:18:16 kolab16.sbsllc.local systemd[1]: Failed to load
environment files: No such file or directory
Sep 11 23:18:16 kolab16.sbsllc.local systemd[1]: ***@service.service
failed to run 'start' task: No such file or directory
Sep 11 23:18:16 kolab16.sbsllc.local systemd[1]: Failed to start 389
Directory Server service..
Sep 11 23:18:16 kolab16.sbsllc.local systemd[1]: ***@service.service
failed.
Sep 11 23:18:16 kolab16.sbsllc.local systemd[1]: Starting 389 Directory
Server service....
But... the LDAP port is open and listening:
[***@kolab16 ~]# netstat -nap | grep 389 | grep "LISTEN "
tcp6 0 0 :::389 :::*
LISTEN 705/ns-slapd
[***@kolab16 ~]#
...and I'm still seeing errors all over the kolab logs.
2016-09-11 23:46:54,908 pykolab.auth ERROR Traceback (most recent call
last):
File "/usr/lib/python2.7/site-packages/pykolab/auth/ldap/__init__.py",
line 3059, in _search
secondary_domains
File "<string>", line 10, in <module>
File "/usr/lib/python2.7/site-packages/pykolab/auth/ldap/__init__.py",
line 2738, in _persistent_search
resp_ctrl_classes={ecnc.controlType:ecnc}
File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
476, in result4
ldap_result =
self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 99,
in _ldap_call
result = func(*args,**kwargs)
TIMEOUT
There must be something wrong underlying, but the system is working...
(I just sent this email!) I'd like to make sure it's working the the
right reason and that it'll stay up. Thank you to anyone that can lend
me some input.
--
Best regards,
STEPHEN H. SWITZER
_Technical Consultant & System Engineer_
- VMware VSP
- Microsoft MCP, Desktop/Server
SWITZER BUSINESS SOLUTIONS, LLC
- VMware VIP Partner
- HP Authorized Business Development Partner
- Xorcom Certified Dealer
any folders on it other than inbox & sent. I decided to see what updates
were available on Kolab16 under Centos. After the update, I'm now
running 7.2.1511.
Once the update was done, I rebooted, and quickly noticed that I could
not send email.I sa errors like this in the maillog:
Sep 11 23:01:14 kolab16 postfix/trivial-rewrite[11366]: warning:
ldap:/etc/postfix/ldap/mydestination.cf: table lookup problem
Sep 11 23:01:16 kolab16 imaps[3445]: badlogin: localhost [127.0.0.1]
login [SASL(-13): authentication failure: checkpass failed]
Sep 11 23:01:24 kolab16 postfix/trivial-rewrite[11366]: warning:
dict_ldap_connect: Unable to bind to server ldap://localhost:389 with dn
uid=kolab-service,ou=Special Users,dc=sbsroc,dc=com: -5 (Timed out)
Somehow I didn't notice the ldap error at first glance and looked at
other things, noticing the following in /var/log/kolab/pykolab.log:
2016-09-11 23:04:02,420 pykolab.wallace ERROR Module
resources.heartbeat() failed with error: Traceback (most recent call
last):
File "/usr/lib/python2.7/site-packages/wallace/__init__.py", line 89,
in modules_heartbeat
modules.heartbeat(module, lastrun)
File "/usr/lib/python2.7/site-packages/wallace/modules.py", line 128,
in heartbeat
return modules[name]['heartbeat'](*args, **kw)
File "/usr/lib/python2.7/site-packages/wallace/module_resources.py",
line 417, in heartbeat
resource_dns = auth.find_resource('*')
File "/usr/lib/python2.7/site-packages/pykolab/auth/__init__.py", line
220, in find_resource
result = self._auth.find_resource(address)
File "/usr/lib/python2.7/site-packages/pykolab/auth/ldap/__init__.py",
line 765, in find_resource
self._bind()
File "/usr/lib/python2.7/site-packages/pykolab/auth/ldap/__init__.py",
line 1435, in _bind
self.ldap.simple_bind_s(bind_dn, bind_pw)
File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
837, in simple_bind_s
res =
self._apply_method_s(SimpleLDAPObject.simple_bind_s,*args,**kwargs)
File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
818, in _apply_method_s
return func(self,*args,**kwargs)
File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
208, in simple_bind_s
resp_type, resp_data, resp_msgid, resp_ctrls =
self.result3(msgid,all=1,timeout=self.timeout)
File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
469, in result3
resp_ctrl_classes=resp_ctrl_classes
File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
476, in result4
ldap_result =
self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 99,
in _ldap_call
result = func(*args,**kwargs)
TIMEOUT
When I finally honed in on the ldap server, I tried to start it:
[***@kolab16 ~]# systemctl restart ***@service.service
Job for ***@service.service failed because a configured resource
limit was exceeded. See "systemctl status ***@service.service" and
"journalctl -xe" for details.
So I ran the suggestion:
[***@kolab16 ~]# journalctl -xe
Sep 11 23:17:44 kolab16.sbsllc.local imaps[12089]: starttls: TLSv1.2
with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits reused) no
authentication
Sep 11 23:17:46 kolab16.sbsllc.local imaps[11890]: badlogin: localhost
[127.0.0.1] plaintext ***@mydomain.com SASL(-13): authentication
failure: checkpass fai
Sep 11 23:18:14 kolab16.sbsllc.local imaps[12089]: timeout_select:
reading from ptloader: Connection timed out
Sep 11 23:18:14 kolab16.sbsllc.local imaps[12089]: ptload failed: but
canonified ***@mydomain.com -> ***@mydomain.com
Sep 11 23:18:16 kolab16.sbsllc.local polkitd[926]: Registered
Authentication Agent for unix-process:12166:120602 (system bus name
:1.36 [/usr/bin/pkttyagent
Sep 11 23:18:16 kolab16.sbsllc.local systemd[1]: Failed to load
environment files: No such file or directory
Sep 11 23:18:16 kolab16.sbsllc.local systemd[1]: ***@service.service
failed to run 'start' task: No such file or directory
Sep 11 23:18:16 kolab16.sbsllc.local systemd[1]: Failed to start 389
Directory Server service..
-- Subject: Unit ***@service.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit ***@service.service has failed.
--
-- The result is failed.
The 389 server is not running... I couldn't log in to roundcube, could
send email with thunderbird, etc. the web admin panel also didn't let me
log in. Since the auth backend is down, that's expected.
I rolled back to a snapshot of this virtual machine, and booted it up...
same issue!! Since the old versio behaved the same way, I went back to
the newly updated snapshot. I figure if it's failing, I might as well
work on the latest version.
After letting it sit for a while during my google research, I received a
"new mail" desktop notification from my browser. Uhmmmmm. It started
working on its own!
I checked the service status, and it still says it failed:
[***@kolab16 ~]# systemctl status ***@service.service
â ***@service.service - 389 Directory Server service.
Loaded: loaded (/usr/lib/systemd/system/***@.service; enabled;
vendor preset: disabled)
Active: failed (Result: resources)
Sep 11 23:10:56 kolab16.sbsllc.local systemd[1]: Failed to load
environment files: No such file or directory
Sep 11 23:10:56 kolab16.sbsllc.local systemd[1]: ***@service.service
failed to run 'start' task: No such file or directory
Sep 11 23:10:56 kolab16.sbsllc.local systemd[1]: Failed to start 389
Directory Server service..
Sep 11 23:10:56 kolab16.sbsllc.local systemd[1]: ***@service.service
failed.
Sep 11 23:10:56 kolab16.sbsllc.local systemd[1]: Starting 389 Directory
Server service....
Sep 11 23:18:16 kolab16.sbsllc.local systemd[1]: Failed to load
environment files: No such file or directory
Sep 11 23:18:16 kolab16.sbsllc.local systemd[1]: ***@service.service
failed to run 'start' task: No such file or directory
Sep 11 23:18:16 kolab16.sbsllc.local systemd[1]: Failed to start 389
Directory Server service..
Sep 11 23:18:16 kolab16.sbsllc.local systemd[1]: ***@service.service
failed.
Sep 11 23:18:16 kolab16.sbsllc.local systemd[1]: Starting 389 Directory
Server service....
But... the LDAP port is open and listening:
[***@kolab16 ~]# netstat -nap | grep 389 | grep "LISTEN "
tcp6 0 0 :::389 :::*
LISTEN 705/ns-slapd
[***@kolab16 ~]#
...and I'm still seeing errors all over the kolab logs.
2016-09-11 23:46:54,908 pykolab.auth ERROR Traceback (most recent call
last):
File "/usr/lib/python2.7/site-packages/pykolab/auth/ldap/__init__.py",
line 3059, in _search
secondary_domains
File "<string>", line 10, in <module>
File "/usr/lib/python2.7/site-packages/pykolab/auth/ldap/__init__.py",
line 2738, in _persistent_search
resp_ctrl_classes={ecnc.controlType:ecnc}
File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
476, in result4
ldap_result =
self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
File "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 99,
in _ldap_call
result = func(*args,**kwargs)
TIMEOUT
There must be something wrong underlying, but the system is working...
(I just sent this email!) I'd like to make sure it's working the the
right reason and that it'll stay up. Thank you to anyone that can lend
me some input.
--
Best regards,
STEPHEN H. SWITZER
_Technical Consultant & System Engineer_
- VMware VSP
- Microsoft MCP, Desktop/Server
SWITZER BUSINESS SOLUTIONS, LLC
- VMware VIP Partner
- HP Authorized Business Development Partner
- Xorcom Certified Dealer
