Paper, PDF, 2.07MB - BSDCon Europe 2002

MTAPerformanceComparison:
sendmailvs.postfixon*BSD
BradKnowles
SeniorConsultantforSnow,BV
brad.knowles@snow.nl
http://www.shub-internet.org/brad/papers/mtacomparison/
Entirecontentscopyright 2002byBradKnowles,allrightsreserved
Overview
• Goal
• Meta Information
– Hardware Used
– Software Tested
– Tools Used
– Methodology
• Test Results
• Conclusions
2002-11-16
Copyright 2002byBradKnowles
2
Goal
• Show you what it looks like to do MTA
performance tuning
– On as many *BSD platforms as I could
• Focus on the process, not the numbers
or expected/desired outcome
– You get to see (most) everything, warts
and all
2002-11-16
Copyright 2002byBradKnowles
3
Goal
• Basically, chapter 3
from the book
sendmail
Performance Tuning by
Nick Christenson
– See
http://www.jetcafe.org/~n
pc/book/sendmail/
2002-11-16
Copyright 2002byBradKnowles
4
MetaInformation
•
•
•
•
Hardware Used
Software Tested
Benchmark Tools
Methodology
2002-11-16
Copyright 2002byBradKnowles
5
HardwareUsed
• NetBSD
– Twinhead “Twinstation 5G” (Sun SPARCstation 5
clone)
– Not UltraSPARC, but the ancient SPARC 5
•
•
•
•
OS:
CPU:
RAM:
NIC:
NetBSD 1.6-RELEASE
microsSPARC-II @ 110MHz
32MB real, 384MB virtual
On-board “Lance” 10Base-T Ethernet
& SBus QuadFastEthernet
• Unfortunately, disk drives died before testing could be
performed (bug in NetBSD regarding SCSI tagged
command queueing for old drives?)
2002-11-16
Copyright 2002byBradKnowles
6
HardwareUsed
• FreeBSD
– Compaq Armada 4131T
•
•
•
•
OS:
CPU:
RAM:
NIC:
• HD:
FreeBSD 4.6.2-RELEASE
Pentium 133
48MB real, 384MB virtual
Asanté FriendlyNET AL1011
“Prism2” 802.11b WiFi PCMCIA
10GB IBM Travelstar 20GN
– 4200 RPM
– 12ms avg. seek
2002-11-16
Copyright 2002byBradKnowles
7
HardwareUsed:
FreeBSD
Image copyright © 2001 Sunset Computer Services, Inc. All Rights Reserved.
2002-11-16
Copyright 2002byBradKnowles
8
HardwareUsed
• MacOS X (Part 1)
– PowerBook G3 “Pismo”
•
•
•
•
•
OS:
CPU:
RAM:
NIC:
HD:
MacOS X 10.2.1
PowerPC G3 @ 400MHz
1GB real, 2GB virtual
Apple AirPort 802.11b WiFi
48GB IBM Travelstar 40GH
– 5400 RPM
– 12ms avg. seek
2002-11-16
Copyright 2002byBradKnowles
9
HardwareUsed:
MacOSX(Part1)
Image copyright © 2001 Sunset Computer Services, Inc. All Rights Reserved.
2002-11-16
Copyright 2002byBradKnowles
10
HardwareUsed
• MacOS X (Part 2)
– PowerBook G4 (DVI)
•
•
•
•
•
OS:
CPU:
RAM:
NIC:
HD:
MacOS X 10.2.2
PowerPC G4 @ 800MHz
1GB real, 2GB virtual
Apple AirPort 802.11b WiFi
40GB Toshiba MK4018GAS
– 4200 RPM
– 12ms avg. seek
2002-11-16
Copyright 2002byBradKnowles
11
HardwareUsed:
MacOSX(Part2)
Image copyright © 2001 Sunset Computer Services, Inc. All Rights Reserved.
2002-11-16
Copyright 2002byBradKnowles
12
SoftwareTested
• Sendmail 8.12.6
– ftp://ftp.sendmail/org/pub/sendmail/sendmail.8.12.6.tar.gz
• Postfix 1.1.11
– http://www.postfix.org/ftp-sites.html
• Exim 4.10
– Tried every way I could think of, but could
not get it configured to relay back to an IP
address or port on the same machine
2002-11-16
Copyright 2002byBradKnowles
13
ToolsUsed
• Disk Benchmarking
– PostMark from Network Appliance
• http://www.netapp.com/tech_library/3022.html
– Bonnie++ from Russell Coker
• http://www.coker.com.au/bonnie++/
• SMTP Benchmarking
– smtp-source/smtp-sink from Wietse Venema
• Included with postfix distribution
– Postal from Russell Coker (with smtp-sink)
• http://www.coker.com.au/postal/
2002-11-16
Copyright 2002byBradKnowles
14
Methodology
• Get disk subsystem baseline by
performing benchmarking
– For each platform & filesystem
configuration
• PostMark
– See http://www.shub-internet.org/brad/FreeBSD/postmark.html
• Bonnie++
– Configured to test similar parameters
» File sizes, numbers, & subdirectories according
to the four PostMark tests
2002-11-16
Copyright 2002byBradKnowles
15
Methodology
• For FreeBSD, we only test turning on or off “soft
updates”
– We do not test (turned on by default)
• vfs.vmiodirenable
• dirprefs
• dirhash
– For data on all the possible combinations of the above, see
“Recent Filesystem Optimisations in FreeBSD” at
http://www.usenix.org/events/usenix02/tech/freenix/dowse.html
– For comparison data of Journaling vs. Soft Updates, see
“Journaling Versus Soft Updates: Asynchronous Meta-data
Protection in File Systems” at
http://www.usenix.org/publications/library/proceedings/usenix2000/
general/seltzer.html
2002-11-16
Copyright 2002byBradKnowles
16
Methodology
• For MacOS X
– We test only
• HFS+
– Apple’s Extended Hierarchical File System
• MFS
– Memory-based filesystem created using
RamBunctious 2.0
2002-11-16
Copyright 2002byBradKnowles
17
Methodology
• For MacOS X
– We do NOT test
• UFS
– IMO, Apple broke it while porting HFS+ to Mach
– Many applications do not recognize it, in addition to
MacOS 9 itself
– “Classic” requires that boot device be HFS+ and
MacOS 9 must be installed before MacOS X
• HFS+ with Journaling
– Too new, introduced with MacOS X Server 10.2.2
– Initial reports indicate ~20% performance loss
2002-11-16
Copyright 2002byBradKnowles
18
Methodology
• For MacOS X
– Does not have a native “poll.h” or “libpoll”
(required by postal)
• Install manually from source
http://www.clapper.org/software/poll/
• Install using “fink”
– MacOS X package/port manager based on Debian
“dselect”
– Unable to test Bonnie++
• Need further assistance from the author to find out why it
is crashing
2002-11-16
Copyright 2002byBradKnowles
19
Methodology
• Perform SMTP benchmarking
– For each hardware & software configuration, test
• Test direct source->sink connection
– Demonstrate maximum possible performance
• Test source->relay->sink
– Show relay performance
– Note
• Testing only relaying, not local delivery
• Test generated on, relayed through, and terminated on
“loopback” network
– Eliminate all question of network or NIC performance
2002-11-16
Copyright 2002byBradKnowles
20
TestResults
• Chart data only
– PostMark table data available from
http://www.shub-internet.org/brad/FreeBSD/postmark.html
– Bonnie++ table data not yet available
– smtp-source/smtp-sink data not yet
available
• `hostname` must be legal (I.e., does not end
with “.”)
– Postal data not yet available
2002-11-16
Copyright 2002byBradKnowles
21
TestResults
• Dumb “Driver” Scripts
– My “runpostmark” script available at
http://www.shub-internet.org/brad/runpostmark
– My “runbon” script available at
http://www.shub-internet.org/brad/runbon
– My “runsmtp” script available at
http://www.shub-internet.org/brad/runsmtp
2002-11-16
Copyright 2002byBradKnowles
22
TestResults:
PostMarkSampleRun
% cd /var/tmp
% postmark
PostMark v1.5 : 3/27/01
pm>set number 1000
pm>set transactions 50000
pm>show
Current configuration is:
The base number of files is 1000
Transactions: 50000
Files range between 500 bytes and 9.77 kilobytes in size
Working directory: /var/tmp
Block sizes are: read=512 bytes, write=512 bytes
Biases are: read/append=5, create/delete=5
Using Unix buffered file I/O
Random number generator seed is 42
Report format is verbose.
pm>run
2002-11-16
Copyright 2002byBradKnowles
23
TestResults:
PostMarkSampleRun
Creating files...Done
Performing transactions........Done
Deleting files...Done
Time:
497 seconds total
492 seconds of transactions (101 per second)
Files:
26014 created (52 per second)
Creation alone: 1000 files (500 per second)
Mixed with transactions: 25014 files (50 per
second)
24868 read (50 per second)
24880 appended (50 per second)
26014 deleted (52 per second)
Deletion alone: 1028 files (342 per second)
Mixed with transactions: 24986 files (50 per
second)
Data:
161.15 megabytes read (332.02 kilobytes per second)
168.38 megabytes written (346.92 kilobytes per second)
2002-11-16
Copyright 2002byBradKnowles
24
PostMarkTestResults
PostmarkTest#1:1000Files&50kTransactions
90
8,000.00
2,500
80
7,000.00
70
2,000
6,000.00
60
1,500
50
Trans/Sec
KB/Sec
5,000.00
4,000.00
40
1,000
3,000.00
30
East
West
North
DataRead
DataWritten
Trans/Sec
2,000.00
20
500
1,000.00
10
PT
P3
+U
FS
2x
P3
+S
OF
T
2x
P3
+M
FS
U5
+U
FS
U5
+L
og
U5
+T
M
PF
S
PB
G3
/H
FS
+
PB
G3
/M
FS
PB
G4
/H
FS
+
PB
G4
/M
Co
FS
mp
aq
+U
Co
FS
mp
aq
+S
of
t
+D
P3
inu
x
T
OF
P3
/L
2nd Qtr
2x
+S
PP
ro
F6
30
30
S/
F3
NF
1st Qtr
NF
R5
S/
S/
R0
OD
S/
PF
2002-11-16
OD
TM
UF
0
S
0
S
0.00
Platform
3rd Qtr
4th Qtr
Copyright 2002byBradKnowles
25
PostMarkTestResults
PostmarkTest#2:20kFiles&50kTransactions
90
3,000.00
700
80
70
600
60
500
50
400
2,500.00
1,500.00
40
300
30
200
1,000.00
20
500.00
Trans/Sec
KB/Sec
2,000.00
East
West
North
DataRead
DataWritten
Trans/Sec
100
10
Platform
/H
PB
G3
PB
U5
3rd Qtr
FS
+
G3
/M
FS
PB
G4
/H
FS
+
PB
G4
/M
FS
Co
mp
aq
+U
FS
Co
mp
aq
+S
of
t
S
PF
g
+T
M
U5
+L
o
FT
2x
P3
+M
FS
U5
+U
FS
+S
O
P3
2nd Qtr
2x
3+
U
FS
PT
+D
P3
2x
P
x
T
inu
/L
P3
+S
OF
ro
F6
30
PP
30
S/
F3
NF
1st Qtr
NF
S/
S/
R5
R0
OD
S/
PF
2002-11-16
OD
TM
UF
0
S
0
S
0.00
4th Qtr
Copyright 2002byBradKnowles
26
PostMarkTestResults
PostmarkTest#3:20kFiles&100kTransactions
90
2,500.00
80
700
70
600
60
500
KB/Sec
1,500.00
50
400
40
300
30
200
1,000.00
Trans/Sec
2,000.00
DataRead
East
DataWritten
Trans/Sec
West
North
500.00
Platform
ft
+S
o
FS
+U
mp
aq
aq
mp
+
FS
PB
G4
/M
FS
Copyright 2002byBradKnowles
Co
4th Qtr
Co
FS
/H
G4
+
FS
G3
/M
PB
PB
3rd Qtr
PB
FS
MP
G3
/H
og
+L
U5
+T
2nd Qtr
U5
63
0
ro
+S
OF
T
P3
/L
inu
x
P3
+D
PT
2x
P3
+U
FS
2x
P3
+S
OF
T
2x
P3
+M
FS
U5
+U
FS
1st Qtr
PP
/F
33
S/
/F
NF
S
NF
S
/R
PF
2002-11-16
OD
OD
S
TM
0
0
0
R5
10.000
0
100
S
20
27
PostMarkTestResults
PostmarkTest#4:20kFiles,100kTransactions,&100subdirectories
90
6,000.00
1,400
80
70
1,200
60
1,000
50
800
5,000.00
3,000.00
40
600
30
400
2,000.00
20
1,000.00
Trans/Sec
KB/Sec
4,000.00
East
West
North
DataRead
DataWritten
Trans/Sec
200
10
ft
mp
aq
+S
o
FS
FS
4th Qtr
Copyright 2002byBradKnowles
Co
/M
Co
mp
aq
+U
PB
3rd Qtr
PB
G4
G4
/H
FS
+
FS
G3
/M
FS
+
Platform
PB
M
+T
U5
2nd Qtr
PB
G3
/H
S
PF
og
+L
U5
FS
U5
+U
FS
3+
M
2x
P
P3
+S
2x
P
2002-11-16
1st Qtr
2x
x
inu
P3
/L
0
OF
T
0
3+
UF
S
0.00
28
PostMarkTestResults
PostmarkTest#1:1000Files&50kTransactions
90
3,500.00
1,200
80
3,317.76
3,174.40
70
3,000.00
1,000
980
60
2,500.00
800
KB/Sec
40
600
1,638.40
1,566.72
1,500.00
485
30
492
Trans/Sec
50
2,000.00
East
West
North
DataRead
DataWritten
Trans/Sec
400
1,000.00
20
10
500.00
0
268.32 280.35
82
0.00
1st Qtr
PBG3/HFS+
2002-11-16
489.66 511.62
149
332.02 346.92
PBG3/MFS
2nd Qtr
200
235.40 245.96
72
PBG4/HFS+
3rd Qtr
PBG4/MFS
Platform
Compaq+UFS
4th Qtr
Copyright 2002byBradKnowles
0
Compaq+Soft
29
PostMarkTestResults
PostmarkTest#2:20kFiles&50kTransactions
90
3,000.00
700
80
2,816.00
666
70
600
2,500.00
60
500
KB/Sec
50
400
40
1,500.00
1,495.04
349
1,392.64
300
30
Trans/Sec
2,000.00
East
West
North
DataRead
DataWritten
Trans/Sec
1,000.00
20
200
737.57
500.00
10
0
100
82.00
0.00
154.75
29
1st Qtr
PBG3/HFS+
2002-11-16
99.14
PBG3/MFS
187.09
36
2nd Qtr
38.95
PBG4/HFS+
3rd Qtr
PBG4/MFS
Platform
73.51
15
Compaq+UFS
66.57
125.63
23
4th Qtr
Copyright 2002byBradKnowles
0
Compaq+Soft
30
PostMarkTestResults
PostmarkTest#3:20kFiles&100kTransactions
90
2,500.00
700
80
2,334.72
613
70
600
2,000.00
60
500
50
KB/Sec
1,500.00
400
40
1,177.60
310
300
1,000.00
30
Trans/Sec
1,628.16
East
West
North
DataRead
DataWritten
Trans/Sec
819.89
200
20
500.00
10
0
100
90.20
0.00
129.65
31
1st Qtr
PBG3/HFS+
2002-11-16
94.42
PBG3/MFS
135.72
33
2nd Qtr
42.68
PBG4/HFS+
3rd Qtr
PBG4/MFS
Platform
69.72 100.21
22
61.35
15
Compaq+UFS
4th Qtr
Copyright 2002byBradKnowles
0
Compaq+Soft
31
PostMarkTestResults
PostmarkTest#4:20kFiles,100kTransactions,&100subdirectories
90
2,500.00
700
80
2,129.92
70
600
574
2,000.00
60
500
50
KB/Sec
400
40
300
1,015.93
277
1,000.00
30
708.74
20
Trans/Sec
1,484.80
1,500.00
East
West
North
DataRead
DataWritten
Trans/Sec
200
500.00
10
0
100
65.43
0.00
93.79
24
1st Qtr
PBG3/HFS+
2002-11-16
77.21
PBG3/MFS
110.68
28
2nd Qtr
38.81
PBG4/HFS+
3rd Qtr
PBG4/MFS
Platform
55.63
14
Compaq+UFS
40.19
4th Qtr
Copyright 2002byBradKnowles
57.61
13
0
Compaq+Soft
32
TestResults:
Bonnie++SampleRun
% bonnie++ -d /var/tmp/bonnie -n 1:10k:0 -m compaq-soft -r 48m
Writing with putc()...done
Writing intelligently...done
Rewriting…done
Reading with getc()...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.02d
------Sequential Output------ --Sequential Input- --Random-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-Machine
Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
compaq-soft
300M 2407 67 4584 39 2052 21 1948 53 3644 18 57.8
3
------Sequential Create------ --------Random Create--------Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-files:max
/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
1:10:0
402 31 +++++ +++
832 49
564 46 +++++ +++ 1746 99
compaq-soft,300M,2407,67,4584,39,2052,21,1948,53,3644,18,57.8,3,1:10:0,402,31,
+++++,+++,832,49,564,46,+++++,+++,1746,99
2002-11-16
Copyright 2002byBradKnowles
33
Bonnie++Results
Bonnie++BlockI/O(SequentialCreate)
450
60%
402
400
50%
50%
47%
350
340
331
40%
250
31%
200
32%
31%
30%
181
23%
150
150
100
%CPUBusy
FilesCreated/Sec
300
FilesCreated/Sec
%CPUBusy
20%
76
10%
50
0
0%
compaq-ufs/1:10:0
compaqufs/20:10:0
compaqufs/20:10:0:100
compaqsoft/1:10:0
compaqsoft/20:10:0
compaqsoft/20:10:0:100
Platform-Filesystem/Test
2002-11-16
Copyright 2002byBradKnowles
34
Bonnie++Results
Bonnie++BlockI/O(SequentialRead)
800
50%
704
43%
700
67043%
69544%
45%
40%
600
500
30%
400
25%
20%
300
%CPUBusy
FilesRead/Sec
35%
FilesRead/Sec
%CPUBusy
15%
200
165
10%
10%
100
5%
0
0%
compaq-ufs/20:10:0
compaq-ufs/20:10:0:100
compaq-soft/20:10:0
compaq-soft/20:10:0:100
Platform-Filesystem/Test
2002-11-16
Copyright 2002byBradKnowles
35
Bonnie++Results
Bonnie++BlockI/O(SequentialDelete)
2,500
70%
2,06260%
2,000
1,857
53%
50%
49%
48%
1,500
40%
30%
1,000
25%
%CPUBusy
52%
FilesDeleted/Sec
60%
FilesDeleted/Sec
%CPUBusy
832
20%
500
215
10%
188
103
0
0%
compaq-ufs/1:10:0
compaqufs/20:10:0
compaqufs/20:10:0:100
compaqsoft/1:10:0
compaqsoft/20:10:0
compaqsoft/20:10:0:100
Platform-Filesystem/Test
2002-11-16
Copyright 2002byBradKnowles
36
Bonnie++Results
Bonnie++BlockI/O(RandomCreate)
600
57%
500
60%
564
50%
48%
46%
40%
330
33%
350
35%
300
200
30%
24%
201
%CPUBusy
FilesCreated/Sec
400
FilesCreated/Sec
%CPUBusy
20%
154
100
10%
78
0
0%
compaq-ufs/1:10:0
compaqufs/20:10:0
compaqufs/20:10:0:100
compaqsoft/1:10:0
compaqsoft/20:10:0
compaqsoft/20:10:0:100
Platform-Filesystem/Test
2002-11-16
Copyright 2002byBradKnowles
37
Bonnie++Results
Bonnie++BlockI/O(RandomRead)
90
6%
81
80
79
80
5%
5%
5%
5%
70
4%
50
46
3%
40
30
2%
%CPUBusy
FilesRead/Sec
60
FilesRead/Sec
%CPUBusy
2%
20
1%
10
0
0%
compaq-ufs/20:10:0
compaq-ufs/20:10:0:100
compaq-soft/20:10:0
compaq-soft/20:10:0:100
Platform-Filesystem/Test
2002-11-16
Copyright 2002byBradKnowles
38
Bonnie++Results
Bonnie++BlockI/O(RandomDelete)
2,000
120%
1,890
1,800
1,811
1,746
100%
99%
1,600
80%
1,200
1,000
60%
53%
53%
800
42%
%CPUBusy
FilesDeleted/Sec
1,400
FilesDeleted/Sec
%CPUBusy
40%
600
400
20%
170
200
40 7%
397%
0
0%
compaq-ufs/1:10:0
compaqufs/20:10:0
compaqufs/20:10:0:100
compaqsoft/1:10:0
compaqsoft/20:10:0
compaqsoft/20:10:0:100
Platform-Filesystem/Test
2002-11-16
Copyright 2002byBradKnowles
39
TestResults:
SmtpStoneSampleRun
% sudo smtp-sink 127.0.0.1:25 1024 &
% foreach I (1 2 3 4 5 6 7 8 9 10)
foreach? /usr/bin/time smtp-source -l 10240 -f fred@example.net \
-t george@example.net -m 1000 -s 512 127.0.0.1:25
foreach? end
8.06 real
0.22 user
1.65 sys
9.03 real
0.26 user
1.45 sys
smtp-source: fatal: connect: Operation timed out
32.00 real
0.17 user
1.68 sys
13.08 real
0.19 user
1.43 sys
10.98 real
0.31 user
1.28 sys
9.02 real
0.24 user
1.52 sys
8.04 real
0.25 user
1.57 sys
smtp-source: fatal: lost connection while reading server greeting
16.51 real
0.19 user
1.61 sys
10.99 real
0.19 user
1.53 sys
16.52 real
0.25 user
1.50 sys
2002-11-16
Copyright 2002byBradKnowles
40
SmtpStoneResults
PowerBookG4SmtpStonePerformance(1000messages)
250.00
9.00
8.05
8.00
208.29
7.58
7.00
154.16
6.00
150.78
150.00
5.00
4.00
100.00
84.20
79.10
2.00
39.98
0.98
0.42
256
128
64
32
9.980.46
0.15
0.00
512
1.00
19.98
0.64
16
Msgs/Sec.
Std.Dev.
3.00
68.19
50.00
StandardDeviation
MessagesperSecond
200.00
0.10
8
0.07
4
2
5.00
0.00 0.00
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
41
SmtpStoneResults
PowerBookG4SmtpStonePerformance(20000messages)
250.00
45.00
237.28
40.00
38.23
200.00
35.78
35.00
32.42
29.34
30.00
155.99
150.00
25.00
123.55
100.00
97.38
93.87
100.96
101.69
21.14
93.57 96.08
20.00
19.76
79.20
9.34
50.00
39.83
2048
1024
512
256
128
64
32
2.56
1.24
0.00
4096
10.00
5.00
2.58
8192
Msgs/Sec.
Std.Dev.
15.00
5.10
16384
StandardDeviation
MessagesperSecond
32.92
188.79
16
0.00
8
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
42
SmtpStoneResults
CompaqSmtpStonePerformance(1000Messages)
30.00
4.50
28.39
4.23
4.00
25.00
20.00
3.00
16.62
2.50
15.00
2.00
10.00
9.00
StandardDeviation
MessagesperSecond
3.50
Msgs/Sec
Std.Dev.
1.50
1.00
5.00
0.50
0.17
0.02
0.00
4
2
0.00
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
43
TestResults:
PostalSampleRun
% sudo smtp-sink 127.0.0.1:25 1024 &
% postal -m 10k -p 64 -c 1 127.0.0.1 postal.user time,messages,data(K),errors,connections,SSL connections
23:05,7651,41262,0,7715,0
23:06,7795,41656,0,7795,0
23:07,7847,42305,0,7847,0
23:08,7616,41528,0,7616,0
23:09,7670,41659,0,7670,0
23:10,7413,39922,0,7413,0
23:11,7765,41886,0,7765,0
23:12,7752,41960,0,7752,0
23:13,7423,39835,0,7423,0
23:14,7475,39697,0,7475,0
23:15,7458,39981,0,7458,0
23:16,7462,40332,0,7462,0
23:17,7281,39294,0,7280,0
23:18,7408,40022,0,7409,0
23:19,7595,40882,0,7586,0
23:20,7737,41991,0,7746,0
23:21,7646,41122,0,7646,0
^C
2002-11-16
Copyright 2002byBradKnowles
44
TestResults:
PostalSampleRun
% sudo smtp-sink 127.0.0.1:25 1024 &
% postal -m 10k -p 128 -c 1 127.0.0.1 postal.user time,messages,data(K),errors,connections,SSL connections
23:27,8283,43988,0,8411,0
23:28,8965,48165,0,8965,0
23:29,8952,48100,0,8952,0
23:30,8785,46803,0,8785,0
23:31,7948,42811,0,7933,0
Server timed out on read.
Server timed out on read.
[ .. .. Deletia .. .. ]
Server timed out on read.
Server timed out on read.
23:32,8016,43367,28,8059,0
23:33,8960,47748,0,8960,0
23:34,9054,48507,0,9022,0
23:35,8737,47175,0,8769,0
23:36,9057,48586,0,9057,0
23:37,8800,46722,0,8800,0
23:38,8933,48092,0,8933,0
23:39,8969,48155,0,8955,0
23:40,8984,48576,0,8998,0
^C
2002-11-16
Copyright 2002byBradKnowles
45
TestResults:
PostalSampleRun
% sudo smtp-sink 127.0.0.1:25 1024 &
% postal -m 10k -p 256 -c 1 127.0.0.1 postal.user Usage: postal [-m maximum-message-size] [-p processes] [-l local-address]
[-c messages-per-connection] [-r messages-per-minute] [-a]
[-b [no]netscape] [-[z|Z] debug-file]
[-s ssl-percentage]
smtp-server user-list-filename conversion-filename
Postal Version: 0.61
% postal -m 10k -p 32 -c 1 127.0.0.1 postal.user time,messages,data(K),errors,connections,SSL connections
Can't connect to 127.0.0.1 port 25.
Can't connect to 127.0.0.1 port 25.
Can't connect to 127.0.0.1 port 25.
Can't connect to 127.0.0.1 port 25.
Can't connect to 127.0.0.1 port 25.
2002-11-16
Copyright 2002byBradKnowles
46
PostalResults
PowerBookG4PostalPerformance
160.00
400.00
378.02
145.77
140.00
350.00
120.00
300.00
100.00
250.00
79.69
80.00
200.00
164.94
60.00
150.00
39.92
40.00
9.98
27.51
10.06
0.00
128
64
32
16
Msgs/Sec
Std.Dev.
100.00
19.97
20.00
StandardDeviation
MessagesperSecond
126.46
3.67
8
2.74
50.00
0.00
4
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
47
NotesforTesting
• When testing real MTAs, you need to watch more
than just the output from the testing program
– Not unusual for MTAs to accept mail faster than they can
process it, especially under heavy load
• Also watch
– Mail queue size
• mailq | grep '[0-9][0-9]:[0-9][0-9]' | wc -l
– CPU load
• iostat 5
– Memory utilization
• vmstat 5
2002-11-16
Copyright 2002byBradKnowles
48
NotesforPostfix
• Make sure that you modify
/etc/postfix/main.cf to be suitable
– Need to boost
• local_destination_concurrency_limit
• default_destination_concurrency_limit
• Check /etc/postfix/master.cf to make
sure there are no artificial limits on processes
like smtp, smtpd, bounce, cleanup,
rewrite etc…
2002-11-16
Copyright 2002byBradKnowles
49
NotesforMacOSX
• Unfortunately, Apple broke iostat
– Still mostly okay for what we want to watch
(CPU user %, system %, and idle %)
• Apple doesn’t even provide vmstat
– vm_stat is not the same program,
although it is similar
2002-11-16
Copyright 2002byBradKnowles
50
PostfixSmtpStoneResults
PowerBookG4PostfixSmtpStonePerformance(1000messages)
70.00
0.90
0.85
0.84
60.00
0.80
57.22
0.70
52.26
0.64
0.60
39.50
40.00
0.50
0.40
30.00
19.98
20.00
0.29
StandardDeviation
MessagesperSecond
50.00
Msgs/Sec
Std.Dev.
0.30
0.20
10.00
9.92
0.13
4.99
0.08
0.00
0.10
0.00
32
16
8
4
2
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
51
PostfixSmtpStoneResults
Compaq+UFSSmtpStonePerformance(1000messages)
12.00
1.40
10.95
10.36
1.20
10.00
1.14
6.41
0.80
6.00
0.60
0.48
4.00
StandardDeviation
MessagesperSecond
1.00
8.00
Msgs/Sec
Std.Dev.
0.40
0.33
2.00
0.20
0.00
0.00
4
2
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
52
PostfixSmtpStoneResults
Compaq+SoftSmtpStonePerformance(1000messages)
12.00
1.80
10.65
9.40
1.40
8.00
1.20
0.95
6.00
6.14
0.94
1.00
0.80
4.00
StandardDeviation
MessagesperSecond
1.60
1.55
10.00
Msgs/Sec
Std.Dev.
0.60
0.40
2.00
0.20
0.00
0.00
4
2
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
53
PostfixSmtpStoneResults
CompaqPostfixSmtpStoneResults(20000messages/4Threads)
12.00
350.00
11.08
10.00
303.20
300.00
7.53
200.00
6.00
150.00
StandardDeviation
MessagesperSecond
250.00
8.00
Msgs/Sec
Std.Dev.
4.00
100.00
2.00
50.00
17.72
0.00
0.00
compaq-ufs
compaq+soft
Platform-Filesystem
2002-11-16
Copyright 2002byBradKnowles
54
SendmailSmtpStoneResults
PowerBookG4SendmailSmtpStoneResults(1000messages)
20.00
20.00
18.59
18.19
MessagesperSecond
16.00
18.00
14.89
16.00
14.36
14.00
14.00
12.00
12.00
9.69
10.00
10.00
8.00
6.00
8.00
5.37
4.97
4.75
4.00
1.92
16
8
6.00
2.00
1.47
0.68
0.37
32
Msgs/Sec
Std.Dev.
4.00
2.00
0.00
StandardDeviation
18.00
4
2
0.00
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
55
SendmailSmtpStoneResults
PowerBookG4Sendmail+AsyncSmtpStoneResults(1000
messages)
20.00
1.40
18.01
18.00
1.20
1.16
16.00
MessagesperSecond
12.00
0.80
9.92
10.00
0.60
8.00
6.00
5.36
0.42
4.97
0.38
StandardDeviation
1.00
14.00
Msgs/Sec
Std.Dev.
0.40
4.00
2.48
0.20
2.00
0.12
0.20
1.25
0.10
0.00
0.00
32
16
8
4
2
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
56
SendmailSmtpStoneResults
CompaqSendmail-UFSSmtpStoneResults(1000messages)
6.00
35.00
5.0231.04
5.00
30.00
4.00
25.00
3.92
3.65
20.00
3.00
2.49
14.08
15.00
StandardDeviation
MessagesperSecond
4.56
Msgs/Sec.
Std.Dev.
2.00
10.00
9.67
1.00
4.17
5.00
4.04
0.00
0.00
16
8
4
2
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
57
SendmailSmtpStoneResults
CompaqSendmail+SoftSmtpStoneResults(1000messages)
6.00
90.00
6.61
82.20
5.91
80.00
72.11
70.00
MessagesperSecond
5.00
62.62
60.00
4.10
4.00
50.00
3.00
40.00
2.81
2.35
StandardDeviation
7.00
Msgs/Sec.
Std.Dev.
30.00
2.00
20.37
1.00
10.93
20.00
10.00
0.00
0.00
16
8
4
2
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
58
PerformanceTuning:Syslog
• Problem
– Local syslog can cause excessive load on busy mail
system
• Solutions
– Configure syslog to write to logs asynchronously
• Perhaps via a “–” (dash) before the filename
– Run syslog on a central log server, across the network
• Also allows you to perform more in-depth log processing,
without interfering with mail services
• Generally logs are much easier to manage centrally
• Can also combine with other central syslog processing
applications
– E.g., Addamark Log Management System
2002-11-16
Copyright 2002byBradKnowles
59
PerformanceTuning:Syslog
• Problem
– Classic syslog uses UDP, can lose 75%
or more of packets sent across the network
• Solution
– Replace classic syslog with nsyslog,
ssyslog, or syslog-ng and use TCP
instead of UDP
• Depending on your particular requirements for
hardware, OS, environment, etc….
2002-11-16
Copyright 2002byBradKnowles
60
PostfixSmtpStoneResults
PowerBookG4(nosyslog)PostfixSmtpStone(1000messages)
70.00
3.00
2.80
60.89
60.00
2.50
52.59
MessagesperSecond
50.00
2.06
40.00
1.67
2.00
36.32
1.50
1.41
30.00
26.78
1.23
Msgs/Sec
Std.Dev.
1.00
19.30
20.00
StandardDeviation
52.04
0.80
9.96
10.00
0.35
0.47
0.50
4.98
0.00
0.00
128
64
32
16
8
4
2
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
61
SendmailSmtpStoneResults
PowerBookG4Sendmail+Async+NoSyslogSmtpStoneResults
(1000messages)
20.00
8.00
18.00
7.10
17.20
7.00
16.00
6.00
5.50
5.00
12.00
9.92
10.00
4.00
8.00
6.00
3.00
5.24
4.97
2.50
2.00
1.24
0.36
0.10
0.00
32
16
8
0.12
4
Msgs/Sec
Std.Dev.
2.00
1.76
4.00
StandardDeviation
MessagesperSecond
14.00
2
1.00
0.00
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
62
SendmailSmtpStoneResults
CompaqSendmail-UFS-NoSyslogSmtpStoneResults
(1000messages)
6.00
30.00
5.64
5.00
25.00
24.92
4.00
20.00
3.63
3.00
15.00
2.30
11.43
2.00
10.00
StandardDeviation
MessagesperSecond
4.46
Msgs/Sec.
Std.Dev.
8.44
1.00
5.00
2.31
0.00
0.00
8
4
2
1
NumberofThreads
2002-11-16
Copyright 2002byBradKnowles
63
Conclusions
• This process can be difficult and timeconsuming to get right, but it pays off
• Graphs & charts can help a great deal
– But they don’t tell the whole story
• Software
– Postfix is a lot faster than even I had thought
– Wasn’t able to get sendmail tuned for the
performance I’m confident it can deliver
– Benchmarking tools (esp. Bonnie++ and postal)
need more work to be useful
2002-11-16
Copyright 2002byBradKnowles
64
Conclusions
• Use the Scientific Method
– Try to set aside personal bias
– Trust only what you can prove
– Process
• Generate Hypothesis
• Generate test for Hypothesis
• Test Hypothesis
– Prove or Disprove
• If disproven, modify or generate new
Hypothesis
2002-11-16
Copyright 2002byBradKnowles
65
RelatedWork
• Nick Christenson
– sendmail Performance Tuning book
http://www.jetcafe.org/~npc/book/sendmail/
– “Performance Tuning Sendmail Systems”
slides
http://www.jetcafe.org/~npc/doc/performance_tuning.pdf
• Brad Knowles
– “Sendmail Performance Tuning for Large
Systems” slides
http://www.shub-internet.org/brad/papers/sendmail-tuning/
2002-11-16
Copyright 2002byBradKnowles
66
RelatedWork
• Matthias Andree
http://www-dt.e-technik.uni-dortmund.de/~ma/postfix/vsqmail.html
http://www-dt.e-technik.uni-dortmund.de/~ma/postfix/bench2.html
• Nakamura Motonori
http://www.kyoto.wide.ad.jp/mta/eval1/eindex.html
• Ciprian Ascante
http://www.benchmarks.dmz.ro/index.php?topic=LinMSB
2002-11-16
Copyright 2002byBradKnowles
67