SA, normalize_charset, проблемы с body тестами

[loopback]

Привет.
Может кто сталкивался с проблемой - в sa не работают правила body с кирилицей и нормалайзом.
SpamAssassin-3.3.0_3, perl 5.8.8 (пробовал и с последней версией - то же самое). В конфиге включен normalize_charse,
применен патч (который помогал с более старыми версиями SA):
======================
use bytes;
use utf8;
use POSIX qw (locale_h);
POSIX::setlocale(&POSIX::LC_CTYPE,'ru_RU.UTF-8');
=============================
В случае SA 3.2.X правила работают.
В 3.3 - нет.
Все правила написаны в utf-8

---------------------

Решил проблему.

[Хостинг HostFood.ru]

Тарифы на хостинг в России, от 12 рублей: https://www.host-food.ru/tariffs/hosting/
Тарифы на виртуальные сервера (VPS/VDS/KVM) в РФ, от 189 руб.: https://www.host-food.ru/tariffs/virtualny-server-vps/
Выделенные сервера, Россия, Москва, от 2000 рублей (HP Proliant G5, Intel Xeon E5430 (2.66GHz, Quad-Core, 12Mb), 8Gb RAM, 2x300Gb SAS HDD, P400i, 512Mb, BBU):
https://www.host-food.ru/tariffs/vydelennyi-server-ds/
Недорогие домены в популярных зонах: https://www.host-food.ru/domains/

[Alex Keda]

ф топку ассасин

[taurus.rpd]

Как решить аналогичную проблему для версии 3.3.1-1? Неужто проблема до сих пор не решена?
Конфигурационный файл:

Код: Выделить всё

rewrite_header Subject *****SPAM*****
report_safe 0
trusted_networks x.x.x.x/y
required_score 6.0
use_bayes 1
bayes_auto_learn 1
bayes_learn_to_journal 1
bayes_auto_learn_threshold_spam         15
bayes_auto_learn_threshold_nonspam      0.1
bayes_journal_max_size 0
bayes_expiry_max_db_size 200000

ok_locales              ru uk en
ok_languages            ru uk en
normalize_charset	1
skip_rbl_checks         0
use_razor2              1
use_dcc                 1
use_pyzor               1

bayes_ignore_header X-Bogosity
bayes_ignore_header X-Spam-Flag
bayes_ignore_header X-Spam-Status
bayes_ignore_header X-Spam-Report
bayes_ignore_header X-FILTER-SPAM
bayes_ignore_header X-SENDER-INFO
bayes_ignore_header X-SENDER-INFO
bayes_ignore_header X-RCPT-TO
bayes_ignore_header X-REPORT-RETURN
bayes_ignore_header X-HEADER-TO

[taurus.rpd]

Немного детализирую:
при таких настройках

Код: Выделить всё

ok_locales              en
ok_languages            uk ru en
normalize_charset      0
inactive_languages      af am ar be bg bs ca cs cy da de el eo es et eu fa fi fr fy ga gd he hi hr hu hy id is it ja ka ko la lt lv mr ms ne nl no pl pt qu rm ro sa sco sk sl sq sr sv sw ta th tl tr vi yi zh zh.big5 zh.gb2312
textcat_max_languages   5

При тестировании имеем:

Код: Выделить всё

store tmp # spamassassin -D < test_utf8.msg
...
Фев 18 12:18:58.164 [19562] dbg: plugin: Mail::SpamAssassin::Plugin::TextCat=HASH(0xa12b6a8) implements 'extract_metadata', priority 0
Фев 18 12:18:58.164 [19562] dbg: message: ---- MIME PARSER START ----
Фев 18 12:18:58.164 [19562] dbg: message: parsing multipart, got boundary: _----------=_1298017287235347
Фев 18 12:18:58.165 [19562] dbg: message: found part of type text/plain, boundary: _----------=_1298017287235347
Фев 18 12:18:58.165 [19562] dbg: message: added part, type: text/plain
Фев 18 12:18:58.166 [19562] dbg: message: found part of type text/html, boundary: _----------=_1298017287235347
Фев 18 12:18:58.166 [19562] dbg: message: added part, type: text/html
Фев 18 12:18:58.167 [19562] dbg: message: parsing normal part
Фев 18 12:18:58.167 [19562] dbg: message: parsing normal part
Фев 18 12:18:58.167 [19562] dbg: message: ---- MIME PARSER END ----
Фев 18 12:18:58.167 [19562] dbg: message: decoding other encoding type (binary), ignoring
Фев 18 12:18:58.168 [19562] dbg: message: decoding other encoding type (binary), ignoring
Фев 18 12:18:58.181 [19562] dbg: textcat: classifying, skipping: pt ne hi tr sco es da zh.gb2312 sw no lv fr ro vi sa ta id th sr et tl cy ko fi lt hr de be cs yi af bs is qu sl la ga hy ms am ja eu ka mr bg sv zh.big5 rm it he zh hu sq pl eo ca fa fy nl ar gd sk el
Фев 18 12:18:58.257 [19562] dbg: textcat: language possibly: ru.iso-8859-5,uk.koi8-r,ru.koi8-r,ru.windows-1251,en
Фев 18 12:18:58.258 [19562] dbg: textcat: X-Languages: "ru.iso-8859-5 uk.koi8-r ru.koi8-r ru.windows-1251 en", X-Languages-Length: 10000
Фев 18 12:18:58.258 [19562] dbg: plugin: Mail::SpamAssassin::Plugin::URIDNSBL=HASH(0x9cb6210) implements 'parsed_metadata', priority 0
...
Фев 18 12:19:02.275 [19562] dbg: rules: ran body rule LOCAL_TEST ======> got hit: "test"
Фев 18 12:19:07.827 [19562] dbg: rules: ran body rule LOCAL_UTF ======> got hit: "порно"
Фев 18 12:19:08.062 [19562] dbg: rules: ran body rule MY_PORNO2 ======> got hit: "порно"
Фев 18 12:19:08.936 [19562] dbg: rules: ran body rule LOCAL_ALL ======> got hit: "порно"
...

Код: Выделить всё

store tmp # spamassassin -D < test_cp1251.msg
...
Фев 18 12:23:43.262 [19613] dbg: plugin: Mail::SpamAssassin::Plugin::TextCat=HASH(0xa589ca8) implements 'extract_metadata', priority 0
Фев 18 12:23:43.262 [19613] dbg: message: ---- MIME PARSER START ----
Фев 18 12:23:43.262 [19613] dbg: message: parsing multipart, got boundary: _----------=_1298017287235347
Фев 18 12:23:43.263 [19613] dbg: message: found part of type text/plain, boundary: _----------=_1298017287235347
Фев 18 12:23:43.264 [19613] dbg: message: added part, type: text/plain
Фев 18 12:23:43.265 [19613] dbg: message: found part of type text/html, boundary: _----------=_1298017287235347
Фев 18 12:23:43.265 [19613] dbg: message: added part, type: text/html
Фев 18 12:23:43.265 [19613] dbg: message: parsing normal part
Фев 18 12:23:43.265 [19613] dbg: message: parsing normal part
Фев 18 12:23:43.265 [19613] dbg: message: ---- MIME PARSER END ----
Фев 18 12:23:43.265 [19613] dbg: message: decoding other encoding type (binary), ignoring
Фев 18 12:23:43.266 [19613] dbg: message: decoding other encoding type (binary), ignoring
Фев 18 12:23:43.278 [19613] dbg: textcat: classifying, skipping: pt ne hi tr sco es da zh.gb2312 sw no lv fr ro vi sa ta id th sr et tl cy ko fi lt hr de be cs yi af bs is qu sl la ga hy ms am ja eu ka mr bg sv zh.big5 rm it he zh hu sq pl eo ca fa fy nl ar gd sk el
Фев 18 12:23:43.346 [19613] dbg: textcat: language possibly: ru.windows-1251
Фев 18 12:23:43.346 [19613] dbg: textcat: X-Languages: "ru.windows-1251", X-Languages-Length: 8331
Фев 18 12:23:43.346 [19613] dbg: plugin: Mail::SpamAssassin::Plugin::URIDNSBL=HASH(0xa114818) implements 'parsed_metadata', priority 0
...
Фев 18 12:23:46.037 [19613] dbg: rules: ran body rule LOCAL_TEST ======> got hit: "test"
Фев 18 12:23:49.214 [19613] dbg: rules: ran body rule LOCAL_WIN ======> got hit: "?????"
...

при

Код: Выделить всё

normalize_charset      1

Код: Выделить всё

store tmp # spamassassin -D < test_utf8.msg
...
Фев 18 12:26:53.804 [19722] dbg: plugin: Mail::SpamAssassin::Plugin::TextCat=HASH(0x9a466a8) implements 'extract_metadata', priority 0
Фев 18 12:26:53.804 [19722] dbg: message: ---- MIME PARSER START ----
Фев 18 12:26:53.804 [19722] dbg: message: parsing multipart, got boundary: _----------=_1298017287235347
Фев 18 12:26:53.805 [19722] dbg: message: found part of type text/plain, boundary: _----------=_1298017287235347
Фев 18 12:26:53.805 [19722] dbg: message: added part, type: text/plain
Фев 18 12:26:53.806 [19722] dbg: message: found part of type text/html, boundary: _----------=_1298017287235347
Фев 18 12:26:53.806 [19722] dbg: message: added part, type: text/html
Фев 18 12:26:53.807 [19722] dbg: message: parsing normal part
Фев 18 12:26:53.807 [19722] dbg: message: parsing normal part
Фев 18 12:26:53.807 [19722] dbg: message: ---- MIME PARSER END ----
Фев 18 12:26:53.807 [19722] dbg: message: decoding other encoding type (binary), ignoring
Фев 18 12:26:53.808 [19722] dbg: message: Converting...
Фев 18 12:26:53.809 [19722] dbg: message: decoding other encoding type (binary), ignoring
Фев 18 12:26:53.809 [19722] dbg: message: Converting...
Фев 18 12:26:53.874 [19722] dbg: textcat: classifying, skipping: pt ne hi tr sco es da zh.gb2312 sw no lv fr ro vi sa ta id th sr et tl cy ko fi lt hr de be cs yi af bs is qu sl la ga hy ms am ja eu ka mr bg sv zh.big5 rm it he zh hu sq pl eo ca fa fy nl ar gd sk el
Фев 18 12:26:53.923 [19722] dbg: textcat: language possibly: ru.iso-8859-5,uk.koi8-r,ru.koi8-r,ru.windows-1251,en
Фев 18 12:26:53.923 [19722] dbg: textcat: X-Languages: "ru.iso-8859-5 uk.koi8-r ru.koi8-r ru.windows-1251 en", X-Languages-Length: 10000
Фев 18 12:26:53.923 [19722] dbg: plugin: Mail::SpamAssassin::Plugin::URIDNSBL=HASH(0x95d1210) implements 'parsed_metadata', priority 0
...
Фев 18 12:26:57.830 [19722] dbg: rules: ran body rule LOCAL_TEST ======> got hit: "test"
...

Код: Выделить всё

store tmp # spamassassin -D < test_cp1251.msg
...
Фев 18 12:28:38.054 [19791] dbg: plugin: Mail::SpamAssassin::Plugin::TextCat=HASH(0x9769cc0) implements 'extract_metadata', priority 0
Фев 18 12:28:38.054 [19791] dbg: message: ---- MIME PARSER START ----
Фев 18 12:28:38.054 [19791] dbg: message: parsing multipart, got boundary: _----------=_1298017287235347
Фев 18 12:28:38.056 [19791] dbg: message: found part of type text/plain, boundary: _----------=_1298017287235347
Фев 18 12:28:38.056 [19791] dbg: message: added part, type: text/plain
Фев 18 12:28:38.057 [19791] dbg: message: found part of type text/html, boundary: _----------=_1298017287235347
Фев 18 12:28:38.057 [19791] dbg: message: added part, type: text/html
Фев 18 12:28:38.057 [19791] dbg: message: parsing normal part
Фев 18 12:28:38.057 [19791] dbg: message: parsing normal part
Фев 18 12:28:38.057 [19791] dbg: message: ---- MIME PARSER END ----
Фев 18 12:28:38.058 [19791] dbg: message: decoding other encoding type (binary), ignoring
Фев 18 12:28:38.058 [19791] dbg: message: Using labeled charset windows-1251
Фев 18 12:28:38.058 [19791] dbg: message: Converting...
Фев 18 12:28:38.059 [19791] dbg: message: decoding other encoding type (binary), ignoring
Фев 18 12:28:38.059 [19791] dbg: message: Using labeled charset windows-1251
Фев 18 12:28:38.059 [19791] dbg: message: Converting...
Фев 18 12:28:38.123 [19791] dbg: textcat: classifying, skipping: pt ne hi tr sco es da zh.gb2312 sw no lv fr ro vi sa ta id th sr et tl cy ko fi lt hr de be cs yi af bs is qu sl la ga hy ms am ja eu ka mr bg sv zh.big5 rm it he zh hu sq pl eo ca fa fy nl ar gd sk el
Фев 18 12:28:38.172 [19791] dbg: textcat: language possibly: ru.iso-8859-5,uk.koi8-r,ru.koi8-r,ru.windows-1251,en
Фев 18 12:28:38.172 [19791] dbg: textcat: X-Languages: "ru.iso-8859-5 uk.koi8-r ru.koi8-r ru.windows-1251 en", X-Languages-Length: 10000
Фев 18 12:28:38.172 [19791] dbg: plugin: Mail::SpamAssassin::Plugin::URIDNSBL=HASH(0x92f4838) implements 'parsed_metadata', priority 0
...
Фев 18 11:31:22.483 [18199] dbg: rules: ran body rule LOCAL_TEST ======> got hit: "test"
...

Пример правила которое тестируется:

Код: Выделить всё

body MY_PORNO2 /порно/i
score MY_PORNO2 1.0

body    LOCAL_TEST      /test/
score   LOCAL_TEST      1.0

body    LOCAL_WIN /\xEF\xEE\xF0\xED\xEE/
score   LOCAL_WIN 1.0

body    LOCAL_UTF /\xd0\xbf\xd0\xbe\xd1\x80\xd0\xbd\xd0\xbe/
score   LOCAL_UTF 1.0

body    LOCAL_ALL /порно/
score   LOCAL_ALL 1.0

Если я правильно понимаю, то при включении "normalize_charset" кодировки перестают определяться. Ето поправить можно?
Локаль

Код: Выделить всё

store spamassassin # locale
LANG=ru_UA.UTF-8
LC_CTYPE="ru_UA.UTF-8"
LC_NUMERIC="ru_UA.UTF-8"
LC_TIME="ru_UA.UTF-8"
LC_COLLATE="ru_UA.UTF-8"
LC_MONETARY="ru_UA.UTF-8"
LC_MESSAGES="ru_UA.UTF-8"
LC_PAPER="ru_UA.UTF-8"
LC_NAME="ru_UA.UTF-8"
LC_ADDRESS="ru_UA.UTF-8"
LC_TELEPHONE="ru_UA.UTF-8"
LC_MEASUREMENT="ru_UA.UTF-8"
LC_IDENTIFICATION="ru_UA.UTF-8"
LC_ALL=

Как то так...