Hi Franz,
Post by Skale, FranzHi,
i see no mangling of the message body other than setting the default
locale to UTF-8 then encode it quoted printable. The header will be
parsed and changed (invitation etc.).
yes, that's it. It's changing the Content-Transfer-Encoding to
quoted-printable. If this is done after dkim signing it breaks signature.
Post by Skale, FranzIt the message contains html, it will be parsed too.
Since you didn't supply a debug example i can only urge you to enable
wallace debugging.
I can't see anything in debug log - except if smtplib is called the data
is already qouted printable:
2018-06-02 01:25:54,004 pykolab.wallace INFO Accepted connection
2018-06-02 01:25:54,018 pykolab.wallace DEBUG [8771]: Resource
Management called for ('/var/spool/pykolab/wallace/tmpk9DNom',), {}
2018-06-02 01:25:54,019 pykolab.wallace DEBUG [8771]: Renaming
'/var/spool/pykolab/wallace/tmpk9DNom' to
'/var/spool/pykolab/wallace/resources/incoming/tmpk9DNom'
2018-06-02 01:25:54,020 pykolab.wallace DEBUG [8771]: Nachricht ist
keine iTip Nachricht (keine Multipart Nachricht)
2018-06-02 01:25:54,020 pykolab.wallace INFO Message is not an iTip
message or does not contain any (valid) iTip.
2018-06-02 01:25:54,020 pykolab.wallace DEBUG [8771]: No itips, no
resources, pass along
'/var/spool/pykolab/wallace/resources/incoming/tmpk9DNom'
2018-06-02 01:25:54,020 pykolab.wallace DEBUG [8771]: Invitation policy
called for ('/var/spool/pykolab/wallace/resources/incoming/tmpk9DNom',), {}
2018-06-02 01:25:54,021 pykolab.wallace DEBUG [8771]: Invitation policy
executing for '/var/spool/pykolab/wallace/resources/incoming/tmpk9DNom',
False
2018-06-02 01:25:54,021 pykolab.wallace DEBUG [8771]: Renaming
'/var/spool/pykolab/wallace/resources/incoming/tmpk9DNom' to
'/var/spool/pykolab/wallace/invitationpolicy/incoming/tmpk9DNom'
2018-06-02 01:25:54,021 pykolab.wallace DEBUG [8771]: Nachricht ist
keine iTip Nachricht (keine Multipart Nachricht)
2018-06-02 01:25:54,022 pykolab.wallace INFO Message is not an iTip
message or does not contain any (valid) iTip objects.
2018-06-02 01:25:54,022 pykolab.wallace DEBUG [8771]: No itips, no
users, pass along
'/var/spool/pykolab/wallace/invitationpolicy/incoming/tmpk9DNom'
2018-06-02 01:25:54,022 pykolab.wallace INFO Akzeptiere Nachricht in
/var/spool/pykolab/wallace/invitationpolicy/incoming/tmpk9DNom (durch
Modul wallace)
2018-06-02 01:25:54,022 pykolab.wallace DEBUG [8771]: Akzeptiere
Nachricht in:
'/var/spool/pykolab/wallace/invitationpolicy/incoming/tmpk9DNom'
2018-06-02 01:25:54,023 pykolab.wallace DEBUG [8771]: recipients:
['***@example.net']
send: 'ehlo mx0.example.net\r\n'
reply: '250-mx0.example.net\r\n'
reply: '250-PIPELINING\r\n'
reply: '250-SIZE 20480000\r\n'
reply: '250-VRFY\r\n'
reply: '250-ETRN\r\n'
reply: '250-STARTTLS\r\n'
reply: '250-XFORWARD NAME ADDR PROTO HELO SOURCE PORT IDENT\r\n'
reply: '250-ENHANCEDSTATUSCODES\r\n'
reply: '250-8BITMIME\r\n'
reply: '250 DSN\r\n'
reply: retcode (250); Msg: mx0.example.net
PIPELINING
SIZE 20480000
VRFY
ETRN
STARTTLS
XFORWARD NAME ADDR PROTO HELO SOURCE PORT IDENT
ENHANCEDSTATUSCODES
8BITMIME
DSN
send: 'mail FROM:<***@example.net> size=1123\r\n'
reply: '250 2.1.0 Ok\r\n'
reply: retcode (250); Msg: 2.1.0 Ok
send: 'rcpt TO:<***@example.net>\r\n'
reply: '250 2.1.5 Ok\r\n'
reply: retcode (250); Msg: 2.1.5 Ok
send: 'data\r\n'
reply: '354 End data with <CR><LF>.<CR><LF>\r\n'
reply: retcode (354); Msg: End data with <CR><LF>.<CR><LF>
data: (354, 'End data with <CR><LF>.<CR><LF>')
send: 'Sender: ***@example.net\r\nDKIM-Signature: v=1;
a=rsa-sha256; c=relaxed/relaxed; d=example.net;\r\n s=dkim201805;
t=1527895386;\r\n
h=from:from:sender:sender:reply-to:subject:subject:date:date:\r\n
message-id:message-id:to:to:cc:mime-version:mime-version:\r\n
content-type:content-type:\r\n
content-transfer-encoding:content-transfer-encoding:in-reply-to:\r\n
references; bh=IYC/RDnaNtcM6arkuRe/LIW86LUe+V8zvrkFPp/dOoY=;\r\n
b=XjBEXPP/CfSc9RqxX6G+zVW0gorAevrouaSNdQXIx2GhJVvUvheJszeils1SKtRYV7h3oK\r\n
ricUH0upeecCDgQJPyGc90aY/JwsoLs2ZpANomt53fQxOJiSyIiuqGbRAyZgsddK0BoW77\r\n
+TpL2Xatf9c5u017mxvzAWJngXzD52hV7txlM/gKcGy3SZR48F74JNyGdIJX3qmMBe0dSo\r\n
oGj6g0YHN4nTdtvJ995J7eYYgofUJlUglOezF58rQV7n4Vh44pncZZ+vMDKNaQ2h9eKPw6\r\n
AcypUALlZvnssOWyuHRBzHqy7Aet2F9dH8sBvOAZCxcLnuOlC8kruqNAF34/WA==\r\nTo:
Test User <***@example.net>\r\nFrom: cu-test
<***@example.net>\r\nSubject: test encoding 2\r\nMessage-ID:
<7b2f68db-4e2a-1762-a223-***@example.net>\r\nDate: Sat, 2 Jun
2018 01:25:50 +0200\r\nMIME-Version: 1.0\r\nContent-Type: text/plain;
charset=utf-8\r\nContent-Language: de-LU\r\nContent-Transfer-Encoding:
quoted-printable\r\n\r\ntest encoding2\r\n=C3=A4=C3=B6=C3=BC\r\n.\r\n'
reply: '250 2.0.0 Ok: queued as 09238B14\r\n'
reply: retcode (250); Msg: 2.0.0 Ok: queued as 09238B14
data: (250, '2.0.0 Ok: queued as 09238B14')
send: 'quit\r\n'
reply: '221 2.0.0 Bye\r\n'
reply: retcode (221); Msg: 2.0.0 Bye
2018-06-02 01:25:54,153 pykolab.wallace DEBUG [9175]: Worker process
PoolWorker-7 initializing
Post by Skale, FranzBe sure that your default locale on the server is utf-8 !
yes. it is.
What I actually don't understand: Where exactly wallace is changing the
encoding? I send email with 8bit encoding and utf-8.
This is my original mail:
Date: Sat, 2 Jun 2018 01:25:50 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
Thunderbird/52.8.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Language: de-LU
Content-Transfer-Encoding: 8bit
test encoding2
äöü
(encoded in utf-8 - not quoted printable)
I found the function for converting in the footer and invitation
modules. But even if I disable this modules the encoding still is
changed. It's not if I disable wallace in postfix master.cf. I wondering
if this is done by python smtplib.
My submission part of master.cf:
submission inet n - n - -
smtpd
-o cleanup_service_name=cleanup_submission
-o syslog_name=postfix/submission
-o smtpd_tls_security_level=encrypt
-o smtpd_sasl_auth_enable=yes
-o smtpd_sasl_authenticated_header=yes
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
-o smtpd_helo_restrictions=$mua_helo_restrictions
-o smtpd_data_restrictions=$submission_data_restrictions
-o smtpd_recipient_restrictions=$submission_recipient_restrictions
-o smtpd_sender_restrictions=$submission_sender_restrictions
-o content_filter=smtp-wallace:[127.0.0.1]:10026
Because I don't have amavis I call wallace directly in submission.
Milter is called in main.cf:
smtpd_milters = inet:mailpd.example.net:11332
non_smtpd_milters = inet:mailpd.example.net:11332
milter_protocol = 6
milter_mail_macros = i {mail_addr} {client_addr} {client_name}
{auth_authen}
milter_default_action = accept
So the mailflow now is: going to prequeue-miltering and signs dkim.
After that there is the content filter set to wallaces - which alteres
the message and breaks signature.
Post by Skale, FranzAlso, reordering of the content_filter directive might help.
It's not easy, because wallace always is content filter - and milter is
prequeue (opendkim or rspamd doesn't matter).
What I tried: using rspamd as a content_filter which is possible. So I
could pass mails from submission to wallace:
submission inet n - n - -
smtpd
-o cleanup_service_name=cleanup_submission
-o syslog_name=postfix/submission
-o smtpd_tls_security_level=encrypt
-o smtpd_sasl_auth_enable=yes
-o smtpd_sasl_authenticated_header=yes
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
-o smtpd_helo_restrictions=$mua_helo_restrictions
-o smtpd_data_restrictions=$submission_data_restrictions
-o smtpd_recipient_restrictions=$submission_recipient_restrictions
-o smtpd_sender_restrictions=$submission_sender_restrictions
# overwrite the default miter - we can't do that on submission,
because we have first go to wallace
-o smtpd_milters=
-o content_filter=smtp-wallace:[127.0.0.1]:10026
And then in wallace:
# Listener to re-inject email from Wallace into Postfix
127.0.0.1:10027 inet n - n - 100
smtpd
-o cleanup_service_name=cleanup_internal
-o content_filter=
-o local_recipient_maps=
-o relay_recipient_maps=
-o smtpd_milters=
-o smtpd_restriction_classes=
-o smtpd_client_restrictions=
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o mynetworks=127.0.0.0/8
-o smtpd_authorized_xforward_hosts=127.0.0.0/8
-o content_filter=smtp:[127.0.0.1]:2525
in the last line I call the rspamd content filter which I define hiere:
# rspamd as content filter
127.0.0.1:2525 inet n - n - - smtpd
-o syslog_name=postfix/content-filter
-o mynetworks=127.0.0.0/8
-o content_filter=
-o smtpd_milters=${rspamd}
-o smtpd_tls_security_level=none
-o smtpd_client_restrictions=
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o smtpd_relay_restrictions=permit_mynetworks,reject
-o smtpd_authorized_xforward_hosts=${mynetworks}
Now everything works - but on smtp we have the mailflow:
-> rspamd as milter -> wallace -> rspamd as content_filter
I haven't got any Idea how to either to call rspamd content filter in
case of submission but not of smtpd. Or have one directive where both
content filters are called. As far as I understand there is no possility
in postfix to add more then one content_filter without reinjection. But
even in this case - still the problem to distinguish between smtpd and
submission.
Regard
Jan
Post by Skale, FranzPost by Jan KowalskyHi Franz,
thanks for answer,
Post by Skale, FranzHi,
DKIM uses non standard quoted printable encoding !
http://dkim.org/specs/rfc4871-dkimbase.html#dkim-qp
But as far as I understand, dkim itselfs doesn't change the mail body at
all. The qp encoding in my understanding is only for calculating hash.
Post by Skale, FranzWallace, of course, uses standard QP encoding and doesn't take care for
special cases, though it's a feature request.
But wallaces alters the mailbody itself. It changes all to quoted
printable and, if configured, adds footer/header.
So dkim signing in my experiences is valid if the email already is
quoted printable from the mua. Roundcube is fine, thunderbird in default
not. But we do not have control about how users are configure there
email programs and 8bit email transfer isn't very uncommon any more.
The problem is, that wallace alteres mails after dkim milter is applied.
Even if the encoding is no problem we run in trubles as far as wallace
adds e.g. footers.
Regards
Jan