pf-nat & muiltipath

Вопросы настройки и работы с этой ОС.
Правила форума
Убедительная просьба юзать теги [cоde] при оформлении листингов.
Сообщения не оформленные должным образом имеют все шансы быть незамеченными.
johny87
ефрейтор
Сообщения: 65
Зарегистрирован: 2010-01-21 11:56:12

pf-nat & muiltipath

Непрочитанное сообщение johny87 » 2017-06-19 9:19:01

Все привет!

Имеем - openBSD 6.1. Настроен multipath на два интерфейса :

#ifconfig pppoe0

Код: Выделить всё

pppoe0: flags=208851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST,AUTOCONF6> mtu 1492
        index 36 priority 0 llprio 3
        dev: em1 state: session
        sid: 0x1 PADI retries: 0 PADR retries: 0 time: 3d 11:43:40
        sppp: phase network authproto chap authname "XXXXXXXXX"
        groups: pppoe egress
        status: active
        inet6 fe80::80de:c984:7711:4488%pppoe0 ->  prefixlen 64 scopeid 0x24
        inet 46.146.232.XXX --> XXX.XXX.XXX.XXX netmask 0xffffffff
#ifconfig pppoe1

Код: Выделить всё

pppoe1: flags=208851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST,AUTOCONF6> mtu 1492
        index 38 priority 0 llprio 3
        dev: em2 state: session
        sid: 0x13a PADI retries: 0 PADR retries: 0 time: 3d 11:42:18
        sppp: phase network authproto pap authname "XXXXXXXXX"
        groups: pppoe egress
        status: active
        inet6 fe80::80de:c984:7711:4488%pppoe1 ->  prefixlen 64 scopeid 0x26
        inet 178.47.140.XXX --> XXX.XXX.XXX.XXX netmask 0xffffffff

#route show :

Код: Выделить всё

Internet:
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
default            XXX.XXX.XXX.XXX      UGSP     368 65825941     -     8 pppoe0
default            XXX.XXX.XXX.XXX      UGSP     129 86899766     -     8 pppoe1
#sysctl -a :

Код: Выделить всё

net.inet.ip.forwarding=1
net.inet.ip.mforwarding=1
net.inet6.ip6.forwarding=1
net.inet6.ip6.mforwarding=1
net.inet.ip.multipath=1
net.inet6.ip6.multipath=1


#cat /etc/pf.conf

Код: Выделить всё

intif = "em0"
extif = "pppoe0"
extif2 = "pppoe1"
table <lan> { 192.168.10.0/24 !192.168.10.1 }
set skip on { lo }
match in all scrub (no-df)
match on pppoe0 all scrub (max-mss 1440)
match on pppoe1 all scrub (max-mss 1440)
pass all flags S/SA
match out on em2 from <lan> to any nat-to (em2) round-robin

match out on pppoe0 inet from 178.47.140.XXX to any nat-to (pppoe0) round-robin
match out on pppoe0 inet from <lan> to any nat-to (pppoe0) round-robin
match out on pppoe1 inet from 46.146.232.XXX to any nat-to (pppoe1) round-robin
match out on pppoe1 inet from <lan> to any nat-to (pppoe1) round-robin
match out on pppoe0 inet from 192.168.10.254 to any nat-to (pppoe0) round-robin static-port
match out on pppoe1 inet from 192.168.10.254 to any nat-to (pppoe1) round-robin static-port

match in on pppoe0 inet proto tcp from any to 46.146.232.XXX port = 443 rdr-to (pppoe0) port 22 round-robin
match in on pppoe1 inet proto tcp from any to 178.47.140.XXX port = 443 rdr-to (pppoe1) port 22 round-robin
match in on pppoe0 inet proto udp from any to 46.146.232.XXX port = 5060 rdr-to 192.168.10.254
match in on pppoe0 inet proto udp from any to 46.146.232.XXX port 10000:20000 rdr-to 192.168.10.254
match in on pppoe1 inet proto udp from any to 178.47.140.XXX port = 5060 rdr-to 192.168.10.254
match in on pppoe1 inet proto udp from any to 178.47.140.XXX port 10000:20000 rdr-to 192.168.10.254
Из локальной сети NAT работает отлично, пакеты ходят сразу на два интерфейса и ip заменяется на внешний.
Проблема в том, что как только я использую например rtorrent или transmission на сервере, в пакетах на интерейфесе pppoe1 ip не заменяется, а идет с pppoe0 :

#tcpdump -ni pppoe1

Код: Выделить всё

11:04:03.589417 46.146.232.XXX.6890 > 88.123.170.114.46475: . ack 312256 win 535 <nop,nop,timestamp 631086195 1768427598,nop,nop,sack 1 {315112:316540} >
Т.е. транзитный трафик натится, а с локалхоста получается что нет. Что я делаю не так ? Помогите написать правило.


#uname -rsvp

Код: Выделить всё

OpenBSD 6.1 GENERIC.MP#20 amd64

Хостинговая компания Host-Food.ru
Хостинг HostFood.ru
 

Услуги хостинговой компании Host-Food.ru

Хостинг HostFood.ru

Тарифы на хостинг в России, от 12 рублей: https://www.host-food.ru/tariffs/hosting/
Тарифы на виртуальные сервера (VPS/VDS/KVM) в РФ, от 189 руб.: https://www.host-food.ru/tariffs/virtualny-server-vps/
Выделенные сервера, Россия, Москва, от 2000 рублей (HP Proliant G5, Intel Xeon E5430 (2.66GHz, Quad-Core, 12Mb), 8Gb RAM, 2x300Gb SAS HDD, P400i, 512Mb, BBU):
https://www.host-food.ru/tariffs/vydelennyi-server-ds/
Недорогие домены в популярных зонах: https://www.host-food.ru/domains/

johny87
ефрейтор
Сообщения: 65
Зарегистрирован: 2010-01-21 11:56:12

pf-nat & muiltipath

Непрочитанное сообщение johny87 » 2017-06-21 9:59:13

Дополнение : запускаю rtorrent с дефолтными настройками, и смотрю tcpdump.

rtorrent

Код: Выделить всё

(11:42:46) Could not read resource file: ~/.rtorrent.rc
[Throttle off/off KB] [Rate   0.0/  0.0 KB] [Port: 6886] [U 0/0] [D 0/0] [H 0/3
rtorrent слушает по всем интерфейсам порт 6886
netstat -f inet -nat | grep 6886

Код: Выделить всё

tcp          0      0  *.6886                 *.*                    LISTEN
запускаю загрузку торрента (например ubuntu).
tcpdump -ni pppoe1

Код: Выделить всё

11:45:28.950726 46.146.232.XXX.6886 > 82.149.101.196.52980: . ack 120696 win 250 <nop,nop,timestamp 2345641191 1791786> [tos 0x8]
11:45:28.968719 46.146.232.XXX.6886 > 2.230.145.7.9480: . ack 3440 win 245 <nop,nop,timestamp 1641821338 358803> [tos 0x8]
11:45:28.982560 46.146.232.XXX.6886 > 2.230.145.7.9480: . ack 4868 win 267 <nop,nop,timestamp 1641821338 358803> [tos 0x8]
11:45:28.987651 46.146.232.XXX.6886 > 78.43.41.12.44453: . ack 465796 win 509 <nop,nop,timestamp 2985472704 3509342895> [tos 0x8]
11:45:28.987733 46.146.232.XXX.6886 > 78.43.41.12.44453: . ack 465796 win 552 <nop,nop,timestamp 2985472704 3509342895> [tos 0x8]
11:45:28.998118 46.146.232.XXX.6886 > 78.43.41.12.44453: . ack 468556 win 530 <nop,nop,timestamp 2985472704 3509342901> [tos 0x8]
11:45:28.998179 46.146.232.XXX.6886 > 78.43.41.12.44453: . ack 469936 win 552 <nop,nop,timestamp 2985472704 3509342901> [tos 0x8]
11:45:28.999104 46.146.232.XXX.6886 > 78.43.41.12.44453: . ack 472696 win 530 <nop,nop,timestamp 2985472704 3509342901> [tos 0x8]
11:45:28.999160 46.146.232.XXX.6886 > 78.43.41.12.44453: . ack 474076 win 552 <nop,nop,timestamp 2985472704 3509342906> [tos 0x8]
11:45:28.999621 46.146.232.XXX.6886 > 78.43.41.12.44453: . ack 476836 win 509 <nop,nop,timestamp 2985472704 3509342906> [tos 0x8]
11:45:28.999670 46.146.232.XXX.6886 > 78.43.41.12.44453: . ack 476836 win 552 <nop,nop,timestamp 2985472704 3509342906> [tos 0x8]
11:45:28.999779 46.146.232.XXX.6886 > 78.43.41.12.44453: P 1516:1610(94) ack 476836 win 552 <nop,nop,timestamp 2985472704 3509342906> [tos 0x8]
11:45:29.000115 46.146.232.XXX.6886 > 78.43.41.12.44453: . ack 479596 win 509 <nop,nop,timestamp 2985472704 3509342906> [tos 0x8]
11:45:29.000181 46.146.232.XXX.6886 > 78.43.41.12.44453: . ack 480976 win 552 <nop,nop,timestamp 2985472704 3509342906> [tos 0x8]
11:45:29.002636 46.146.232.XXX.6886 > 78.43.41.12.44453: . ack 483736 win 509 <nop,nop,timestamp 2985472704 3509342906> [tos 0x8]
11:45:29.002701 46.146.232.XXX.6886 > 78.43.41.12.44453: . ack 483736 win 552 <nop,nop,timestamp 2985472704 3509342906> [tos 0x8]
11:45:29.008825 46.146.232.XXX.6886 > 82.149.101.196.52980: P 491:535(44) ack 122124 win 267 <nop,nop,timestamp 2345641191 1791811> [tos 0x8]
11:45:29.011629 46.146.232.XXX.6886 > 2.230.145.7.9480: . ack 7724 win 245 <nop,nop,timestamp 1641821338 358803> [tos 0x8]
11:45:29.023068 46.146.232.XXX.6886 > 2.230.145.7.9480: . ack 9152 win 267 <nop,nop,timestamp 1641821338 358803> [tos 0x8]
11:45:29.028493 46.146.232.XXX.6886 > 5.94.192.45.46608: P 188:572(384) ack 69 win 267 <nop,nop,timestamp 3665927213 600817> [tos 0x8]
11:45:29.032316 46.146.232.XXX.6886 > 78.43.41.12.44453: . ack 486496 win 530 <nop,nop,timestamp 2985472704 3509342906> [tos 0x8]
11:45:29.037634 46.146.232.XXX.6886 > 78.43.41.12.44453: . ack 488712 win 517 <nop,nop,timestamp 2985472704 3509342945> [tos 0x8]
11:45:29.048669 46.146.232.XXX.6886 > 78.43.41.12.44453: . ack 490092 win 552 <nop,nop,timestamp 2985472704 3509342956> [tos 0x8]
11:45:29.049094 46.146.232.XXX.6886 > 78.43.41.12.44453: . ack 492852 win 509 <nop,nop,timestamp 2985472704 3509342956> [tos 0x8]
11:45:29.049144 46.146.232.XXX.6886 > 78.43.41.12.44453: . ack 492852 win 552 <nop,nop,timestamp 2985472704 3509342956> [tos 0x8]

В tcpdump на интерфейсе pppoe1 исходящий ip-адрес с интерфейса pppoe0..pf как будто не работает.
Правила pf (обратите внимание на правила @6 и @7)

pfctl -vvvvvvvvvsr

Код: Выделить всё

@0 match in all scrub (no-df)
  [ Evaluations: 3         Packets: 6         Bytes: 406         States: 3     ]
  [ Inserted: uid 0 pid 94697 State Creations: 0     ]
@1 match on pppoe0 all scrub (max-mss 1440)
  [ Evaluations: 3         Packets: 6         Bytes: 406         States: 3     ]
  [ Inserted: uid 0 pid 94697 State Creations: 0     ]
@2 match on pppoe1 all scrub (max-mss 1440)
  [ Evaluations: 3         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 94697 State Creations: 0     ]
@3 match out on em2 from <lan:2> to any nat-to (em2:1) round-robin
  [ Evaluations: 3         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 94697 State Creations: 0     ]
@4 match out on pppoe0 inet from <lan:2> to any nat-to (pppoe0:1) round-robin
  [ Evaluations: 3         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 94697 State Creations: 0     ]
@5 match out on pppoe1 inet from <lan:2> to any nat-to (pppoe1:1) round-robin
  [ Evaluations: 0         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 94697 State Creations: 0     ]
@6 match out on pppoe0 inet from 178.47.140.XXX to any nat-to (pppoe0:1) round-robin
  [ Evaluations: 0         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 94697 State Creations: 0     ]
@7 match out on pppoe1 inet from 46.146.232.XXX to any nat-to (pppoe1:1) round-robin
  [ Evaluations: 0         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 94697 State Creations: 0     ]
@8 match out on pppoe0 inet from 192.168.10.254 to any nat-to (pppoe0:1) round-robin static-port
  [ Evaluations: 0         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 94697 State Creations: 0     ]
@9 match out on pppoe1 inet from 192.168.10.254 to any nat-to (pppoe1:1) round-robin static-port
  [ Evaluations: 0         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 94697 State Creations: 0     ]
@10 match in on pppoe0 inet proto tcp from any to 46.146.232.XXX port = 443 rdr-to (pppoe0:1) port 22 round-robin
  [ Evaluations: 3         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 94697 State Creations: 0     ]
@11 match in on pppoe1 inet proto tcp from any to 178.47.140.XXX port = 443 rdr-to (pppoe1:1) port 22 round-robin
  [ Evaluations: 0         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 94697 State Creations: 0     ]
@12 match in on pppoe0 inet proto udp from any to 46.146.232.XXX port = 5060 rdr-to 192.168.10.254
  [ Evaluations: 3         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 94697 State Creations: 0     ]
@13 match in on pppoe0 inet proto udp from any to 46.146.232.XXX port 10000:20000 rdr-to 192.168.10.254
  [ Evaluations: 3         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 94697 State Creations: 0     ]
@14 match in on pppoe1 inet proto udp from any to 178.47.140.XXX port = 5060 rdr-to 192.168.10.254
  [ Evaluations: 3         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 94697 State Creations: 0     ]
@15 match in on pppoe1 inet proto udp from any to 178.47.140.XXX port 10000:20000 rdr-to 192.168.10.254
  [ Evaluations: 0         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 94697 State Creations: 0     ]
@16 pass all flags S/SA
  [ Evaluations: 3         Packets: 6         Bytes: 406         States: 3     ]
  [ Inserted: uid 0 pid 94697 State Creations: 3     ]