Migrating Ubuntu -> CentOS IMAP Troubles

[Tobias Brunner]

Hi everyone,

I'm trying to migrate a Kolab server from an Ubuntu installation to a
CentOS installation and I struggle with IMAP data migration. All data
(/var/lib/imap and /var/spool/imap) is rsynced to the new server with
Cyrus stopped. When I start Cyrus all *.db files in /var/lib/imap get
deleted an recreated. The old files are preserved as *.db.berkeley

On the old Ubuntu server, Cyrus is installed and running as version
"2.5~dev2015021301-0~kolab2", the new server uses
"cyrus-imapd-2.5.8-13.4.el7.kolab_16.x86_64".

Does anyone have an idea what I can do? F.e. I already tried to first
convert the dbs on the old server to flat and then on the new server
back to twoskip, but that didn't help. And I'm pretty sure that the old
server doesn't use BDB as backend.

Cheers,
Tobias

[Andy Kopciuch]

Post by Tobias Brunner
Does anyone have an idea what I can do? F.e. I already tried to first
convert the dbs on the old server to flat and then on the new server
back to twoskip, but that didn't help. And I'm pretty sure that the old
server doesn't use BDB as backend.

If you convert the db files to a format other than twoskip (which is the
default now in cyrus > 2.5) ... then you'll need to change some options in
imap.conf to match the format you converted them into.

https://cyrusimap.org/docs/cyrus-imapd/2.2.13p1/man/imapd.conf.5.php

Check out the options named *_db. (annotations_db, mailboxes_db, etc). They
default to twoskip. So if you want to convert them to berkely, or skiplist,
or flat then you need to specify that format in the config as well.

annotations_db = flat

I believe all of the databases will be recreated if they are not available
_except_ for the mailboxes database. So if it can not read the database files
you have in place, it will re-create them.

I would make sure whatever format you decide to use for each database is set
in imap.conf accordingly.



Andy

[Andy Kopciuch]

Post by Andy Kopciuch
https://cyrusimap.org/docs/cyrus-imapd/2.2.13p1/man/imapd.conf.5.php

Sorry, that was a very old version. Grabbed the URL from the wrong browser
window. The man for version 2.5.8 is here :

https://cyrusimap.org/docs/cyrus-imapd/2.5.8/man/imapd.conf.5.php


Andy

[Tobias Brunner]

Post by Andy Kopciuch

Post by Tobias Brunner
Does anyone have an idea what I can do? F.e. I already tried to first
convert the dbs on the old server to flat and then on the new server
back to twoskip, but that didn't help. And I'm pretty sure that the old
server doesn't use BDB as backend.

If you convert the db files to a format other than twoskip (which is the
default now in cyrus > 2.5) ... then you'll need to change some options in
imap.conf to match the format you converted them into.
https://cyrusimap.org/docs/cyrus-imapd/2.2.13p1/man/imapd.conf.5.php
Check out the options named *_db. (annotations_db, mailboxes_db, etc). They
default to twoskip. So if you want to convert them to berkely, or skiplist,
or flat then you need to specify that format in the config as well.
annotations_db = flat
I believe all of the databases will be recreated if they are not available
_except_ for the mailboxes database. So if it can not read the database files
you have in place, it will re-create them.
I would make sure whatever format you decide to use for each database is set
in imap.conf accordingly.

I only converted the files to flat on the old server to copy it to the
new server. On the new server I converted the DBs back to twoskip with
cvt_cyrusdb which worked as expected per the tool output. *_db is not
configured, therefore Cyrus should use twoskip.

As this doesn't seem to solve the issue, I tried again to just rsync
/var/lib/imap/ from the old to the new server. Before starting Cyrus the
content of the directory looks like this:

# ls -Alh /var/lib/imap/*.db*
-rw-------. 1 cyrus mail 176K Sep 28 14:09 /var/lib/imap/annotations.db
-rw-------. 1 cyrus mail 1.2M Sep 28 22:01 /var/lib/imap/deliver.db
-rw-------. 1 cyrus mail 181K Sep 28 21:28 /var/lib/imap/mailboxes.db
-rw-------. 1 cyrus mail 336 Mar 10 2015 /var/lib/imap/statuscache.db
-rw-------. 1 cyrus mail 200K Sep 28 22:11 /var/lib/imap/tls_sessions.db

After starting cyrus-imapd, it looks like this:

# ls -Alh /var/lib/imap/*.db*
-rw-------. 1 cyrus mail 3.8K Sep 28 22:12 /var/lib/imap/annotations.db
-rw-------. 1 cyrus mail 176K Sep 28 14:09
/var/lib/imap/annotations.db.berkeley
-rw-------. 1 cyrus mail 336 Sep 28 22:12 /var/lib/imap/deliver.db
-rw-------. 1 cyrus mail 1.2M Sep 28 22:01 /var/lib/imap/deliver.db.berkeley
-rw-------. 1 cyrus mail 3.7K Sep 28 22:12 /var/lib/imap/mailboxes.db
-rw-------. 1 cyrus mail 181K Sep 28 21:28
/var/lib/imap/mailboxes.db.berkeley
-rw-------. 1 cyrus mail 336 Sep 28 22:12 /var/lib/imap/statuscache.db
-rw-------. 1 cyrus mail 336 Mar 10 2015
/var/lib/imap/statuscache.db.berkeley
-rw-------. 1 cyrus mail 768 Sep 28 22:12 /var/lib/imap/tls_sessions.db
-rw-------. 1 cyrus mail 200K Sep 28 22:11
/var/lib/imap/tls_sessions.db.berkeley

So somehow Cyrus thinks that the files are in BDB format and recreates
them from scratch. Opening a file of the old server in an editor, it
contains "twoskip file" at the beginning.

Unfortunately the logfile /var/log/maillog doesn't log that much on startup:

Sep 28 22:17:06 kolab2 ctl_cyrusdb[3309]: skiplist: clean shutdown file
missing, updating recovery stamp
Sep 28 22:17:06 kolab2 ctl_cyrusdb[3309]: recovering cyrus databases
Sep 28 22:17:06 kolab2 ctl_cyrusdb[3309]: done recovering cyrus databases
Sep 28 22:17:07 kolab2 master[3307]: unable to setsocketopt(IP_TOS)
service ptloader/unix: Operation not supported
Sep 28 22:17:07 kolab2 master[3307]: unable to setsocketopt(IP_TOS)
service lmtpunix/unix: Operation not supported
Sep 28 22:17:07 kolab2 master[3307]: unable to setsocketopt(IP_TOS)
service notify/unix: Operation not supported
Sep 28 22:17:07 kolab2 ctl_cyrusdb[3312]: checkpointing cyrus databases
Sep 28 22:17:07 kolab2 ctl_cyrusdb[3312]: done checkpointing cyrus databases

I even tried to set CYRUS_VERBOSE=1 in /etc/sysconfig/cyrus-imapd which
didn't change anything regarding logging output. Also configuring
local6.debug in rsyslog didn't produce more verbose logging.

Any other hints what I can do to debug this sort of issue?

Cheers,
Tobias

[Andy Kopciuch]

Post by Tobias Brunner
Any other hints what I can do to debug this sort of issue?

1. I would be curious to see what exists in the annotations.db (or another
database) on the new server that is created when it ignores your files. Just
to confirm that it is in fact configured to use twoskip format. If it is not
(you don't have "twoskip file" in the top) ... that might give you some clues.

2. You could forcefully set those configuration options in imap.conf to set
the database formats to twoskip. Just to be positive it is expecting that
format, and the files you have are that format.

3. I really don't think it matters much if they are re-created ... except for
mailboxes.db That's the one that has the list of mailboxes in the user's
accounts. I would focus on getting mailboxes.db to work ... and ignore the
rest of them.

--

When I recently had some database problems, this page was some good insight
(although written quite some time ago) :
https://www.banquise.org/software/how-to-recover-from-cyrus-when-you-have-some-db-errors/

I just found this in a link :
https://gist.github.com/tpokorra/5643410

Did you switch from a 32 to 64 bit server?



Andy

[Tobias Brunner]

Post by Andy Kopciuch

Post by Tobias Brunner
Any other hints what I can do to debug this sort of issue?

1. I would be curious to see what exists in the annotations.db (or another
database) on the new server that is created when it ignores your files. Just
to confirm that it is in fact configured to use twoskip format. If it is not
(you don't have "twoskip file" in the top) ... that might give you some clues.
2. You could forcefully set those configuration options in imap.conf to set
the database formats to twoskip. Just to be positive it is expecting that
format, and the files you have are that format.
3. I really don't think it matters much if they are re-created ... except for
mailboxes.db That's the one that has the list of mailboxes in the user's
accounts. I would focus on getting mailboxes.db to work ... and ignore the
rest of them.
--
When I recently had some database problems, this page was some good insight
https://www.banquise.org/software/how-to-recover-from-cyrus-when-you-have-some-db-errors/
https://gist.github.com/tpokorra/5643410
Did you switch from a 32 to 64 bit server?

Before I answer the questions, I've discovered an odd behaviour which I
don't understand: After a completely fresh Kolab 16 install on CentOS 7
everything looks fine. I can create mailboxes, stop/start Cyrus, all
works as it should do. The contents of /var/lib/imap looks like this:

-rw------- 1 cyrus mail 3.8K Sep 29 14:48 annotations.db
drwxr-x--- 2 cyrus mail 6 Sep 22 17:25 backup
drwxr-x--- 2 cyrus mail 22 Sep 29 14:48 db
drwx------ 2 cyrus mail 46 Sep 29 14:48 db.backup1
-rw------- 1 cyrus mail 336 Sep 29 14:48 deliver.db
drwx------ 3 cyrus mail 14 Sep 29 14:48 domain
drwx------ 5 cyrus mail 35 Sep 29 14:48 lock
drwxr-x--- 2 cyrus mail 6 Sep 22 17:25 log
-rw------- 1 cyrus mail 3.7K Sep 29 14:48 mailboxes.db
drwxr-x--- 2 cyrus mail 6 Sep 22 17:25 md5
drwxr-x--- 2 cyrus mail 6 Sep 22 17:25 meta
drwxr-x--- 2 cyrus mail 6 Sep 22 17:25 msg
drwxr-x--- 2 cyrus mail 18 Sep 29 14:56 proc
drwxr-x--- 2 cyrus mail 37 Sep 29 14:48 ptclient
drwxr-x--- 2 cyrus mail 6 Sep 22 17:25 quota
drwxr-x--- 2 cyrus mail 51 Sep 29 14:48 rpm
drwxr-x--- 2 cyrus mail 6 Sep 22 17:25 sieve
drwxr-x--- 2 cyrus mail 4.0K Sep 29 14:56 socket
-rw------- 1 cyrus mail 336 Sep 29 14:48 statuscache.db
drwxr-x--- 2 cyrus mail 6 Sep 22 17:25 sync
-rw------- 1 cyrus mail 768 Sep 29 14:48 tls_sessions.db
drwxr-x--- 2 cyrus mail 6 Sep 22 17:25 user

Then I do:

1. Stop Cyrus: systemctl stop cyrus-imapd. Log says:

Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: attempting clean
shutdown on signal
Sep 29 14:57:59 kolabv.vagrant.dev pop3[16965]: graceful shutdown
initiated by unexpected process 1 (/usr/lib/systemd/systemd
--switched-root --system --deserialize 21)
Sep 29 14:57:59 kolabv.vagrant.dev pop3[16971]: graceful shutdown
initiated by unexpected process 1 (/usr/lib/systemd/systemd
--switched-root --system --deserialize 21)
Sep 29 14:57:59 kolabv.vagrant.dev lmtpunix[16967]: graceful shutdown
initiated by unexpected process 1 (/usr/lib/systemd/systemd
--switched-root --system --deserialize 21)
Sep 29 14:57:59 kolabv.vagrant.dev systemd[1]: Stopping Cyrus-imapd
IMAP/POP3 email server...
Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
name:imap path:/usr/lib/cyrus-imapd/imapd age:561.385s pid:16959 exited,
status 75
Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
name:imap path:/usr/lib/cyrus-imapd/imapd age:561.385s pid:16961 exited,
status 75
Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
name:pop3 path:/usr/lib/cyrus-imapd/pop3d age:561.385s pid:16963 exited,
status 75
Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
name:pop3 path:/usr/lib/cyrus-imapd/pop3d age:561.385s pid:16964 exited,
status 75
Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
name:pop3s path:/usr/lib/cyrus-imapd/pop3d age:561.385s pid:16966
exited, status 75
Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
name:imap path:/usr/lib/cyrus-imapd/imapd age:561.387s pid:16958 exited,
status 75
Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
name:imap path:/usr/lib/cyrus-imapd/imapd age:561.389s pid:16960 exited,
status 75
Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
name:pop3 path:/usr/lib/cyrus-imapd/pop3d age:561.389s pid:16965 exited,
status 75
Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
name:lmtpunix path:/usr/lib/cyrus-imapd/lmtpd age:561.388s pid:16967
exited, status 75
Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
name:notify path:/usr/lib/cyrus-imapd/notifyd age:561.388s pid:16968
exited, status 75
Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
name:pop3 path:/usr/lib/cyrus-imapd/pop3d age:561.388s pid:16969 exited,
status 75
Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
name:pop3 path:/usr/lib/cyrus-imapd/pop3d age:561.388s pid:16970
signaled to death by signal 15 (Terminated)
Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
name:pop3 path:/usr/lib/cyrus-imapd/pop3d age:561.388s pid:16971 exited,
status 75
Sep 29 14:57:59 kolabv.vagrant.dev master[16944]: process type:SERVICE
name:pop3s path:/usr/lib/cyrus-imapd/pop3d age:561.388s pid:16972
exited, status 75
Sep 29 14:57:59 kolabv.vagrant.dev systemd[1]: Stopped Cyrus-imapd
IMAP/POP3 email server.

2. Remove all content of /var/lib/imap: rm -rf /var/lib/imap/*
3. Start Cyrus, which automatically recreates the folder structure under
/var/lib/imap. The folder content:

-rw------- 1 cyrus mail 3.8K Sep 29 15:00 annotations.db
drwxr-xr-x 2 cyrus mail 22 Sep 29 15:00 db
drwx------ 2 cyrus mail 46 Sep 29 15:00 db.backup1
-rw------- 1 cyrus mail 336 Sep 29 15:00 deliver.db
drwx------ 3 cyrus mail 14 Sep 29 15:00 domain
drwx------ 5 cyrus mail 35 Sep 29 15:00 lock
drwxr-xr-x 2 cyrus mail 6 Sep 29 15:00 log
-rw------- 1 cyrus mail 3.6K Sep 29 15:00 mailboxes.db
drwxr-xr-x 2 cyrus mail 6 Sep 29 15:00 msg
drwxr-xr-x 2 cyrus mail 18 Sep 29 15:00 proc
drwxr-xr-x 2 cyrus mail 37 Sep 29 15:00 ptclient
drwxr-xr-x 2 cyrus mail 26 Sep 29 15:00 rpm
drwxr-xr-x 2 cyrus mail 4.0K Sep 29 15:00 socket
-rw------- 1 cyrus mail 336 Sep 29 15:00 statuscache.db
drwxr-xr-x 2 cyrus mail 6 Sep 29 15:00 sync
-rw------- 1 cyrus mail 1.3K Sep 29 15:00 tls_sessions.db

4. Stop Cyrus which now leads to broken DBs, renamed to *.db.berkeley:

-rw------- 1 cyrus mail 3.8K Sep 29 15:00 annotations.db.berkeley
drwxr-xr-x 2 cyrus mail 22 Sep 29 15:00 db
drwx------ 2 cyrus mail 46 Sep 29 15:00 db.backup1
-rw------- 1 cyrus mail 336 Sep 29 15:00 deliver.db.berkeley
drwx------ 3 cyrus mail 14 Sep 29 15:00 domain
drwx------ 5 cyrus mail 35 Sep 29 15:00 lock
drwxr-xr-x 2 cyrus mail 6 Sep 29 15:00 log
-rw------- 1 cyrus mail 3.6K Sep 29 15:00 mailboxes.db.berkeley
drwxr-xr-x 2 cyrus mail 6 Sep 29 15:00 msg
drwxr-xr-x 2 cyrus mail 6 Sep 29 15:01 proc
drwxr-xr-x 2 cyrus mail 46 Sep 29 15:01 ptclient
drwxr-xr-x 2 cyrus mail 46 Sep 29 15:01 rpm
drwxr-xr-x 2 cyrus mail 4.0K Sep 29 15:01 socket
-rw------- 1 cyrus mail 336 Sep 29 15:00 statuscache.db.berkeley
drwxr-xr-x 2 cyrus mail 6 Sep 29 15:00 sync
-rw------- 1 cyrus mail 1.5K Sep 29 15:00 tls_sessions.db.berkeley

5. Start Cyrus, files are recreated:

-rw------- 1 cyrus mail 336 Sep 29 15:01 annotations.db
-rw------- 1 cyrus mail 3.8K Sep 29 15:00 annotations.db.berkeley
drwxr-xr-x 2 cyrus mail 22 Sep 29 15:00 db
drwx------ 2 cyrus mail 46 Sep 29 15:01 db.backup1
drwx------ 2 cyrus mail 46 Sep 29 15:00 db.backup2
-rw------- 1 cyrus mail 336 Sep 29 15:01 deliver.db
-rw------- 1 cyrus mail 336 Sep 29 15:00 deliver.db.berkeley
drwx------ 3 cyrus mail 14 Sep 29 15:00 domain
drwx------ 5 cyrus mail 35 Sep 29 15:00 lock
drwxr-xr-x 2 cyrus mail 6 Sep 29 15:00 log
-rw------- 1 cyrus mail 336 Sep 29 15:01 mailboxes.db
-rw------- 1 cyrus mail 3.6K Sep 29 15:00 mailboxes.db.berkeley
drwxr-xr-x 2 cyrus mail 6 Sep 29 15:00 msg
drwxr-xr-x 2 cyrus mail 6 Sep 29 15:01 proc
drwxr-xr-x 2 cyrus mail 46 Sep 29 15:01 ptclient
drwxr-xr-x 2 cyrus mail 46 Sep 29 15:01 rpm
drwxr-xr-x 2 cyrus mail 4.0K Sep 29 15:01 socket
-rw------- 1 cyrus mail 336 Sep 29 15:01 statuscache.db
-rw------- 1 cyrus mail 336 Sep 29 15:00 statuscache.db.berkeley
drwxr-xr-x 2 cyrus mail 6 Sep 29 15:00 sync
-rw------- 1 cyrus mail 1.5K Sep 29 15:00 tls_sessions.db.berkeley

I even tried mkimap before step 3 which basically creates the same
folder structure as Cyrus does on startup.

It seems to me that Cyrus is doing some bogus stuff. Has anyone an
explanation for this odd behaviour? What am I missing here? How is the
initial folder structure created which seems to be fine?

Cheers,
Tobias