Blackhole Documentation
Release 1.6.0
Kura
April 30, 2013
CONTENTS
1
Using the blackhole.io service
1.1 Testing via telnet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Testing SSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3
3
3
2
Getting the source code
5
3
Running your own server
3.1 Python versions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Third party libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Getting started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7
7
7
7
4
Tests & Coverage
4.1 Running tests manually . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Third party CI/Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
13
13
5
Blackhole + PyPy
15
6
FAQ
17
7
Reference
7.1 blackhole.connection
7.2 blackhole.data . . . . .
7.3 blackhole.log . . . . . .
7.4 blackhole.opts . . . . .
7.5 blackhole.ssl_utils .
7.6 blackhole.state . . . .
7.7 blackhole.utils . . . .
8
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
19
19
19
20
20
20
21
21
Indices and tables
23
Python Module Index
25
i
ii
Blackhole Documentation, Release 1.6.0
.o8
oooo
"888
‘888
888oooo.
888
.oooo.
d88’ ‘88b 888 ‘P )88b
888
888 888
.oP"888
888
888 888 d8( 888
‘Y8bod8P’ o888o ‘Y888""8o
oooo
oooo
‘888
‘888
.ooooo.
888 oooo
888 .oo.
d88’ ‘"Y8 888 .8P’
888P"Y88b
888
888888.
888
888
888
.o8 888 ‘88b.
888
888
‘Y8bod8P’ o888o o888o o888o o888o
.ooooo.
d88’ ‘88b
888
888
888
888
‘Y8bod8P’
oooo
‘888
888
888
888
888
o888o
o8o
‘"’
.ooooo.
oooo
d88’ ‘88b
‘888
888ooo888
888
888
.o .o. 888
‘Y8bod8P’ Y8P o888o
Blackhole is a Tornado powered MTA (mail transport agent) that is designed for handling large volumes of email
without handling any of the messages and doing no disk bound I/O.
Blackhole is designed mostly for testing purposes and can be used to test numerous things suchs as;
• Email send rates, if you need to test how much mail you can send per minute, hour etc
• Email integration testing and finally
• if you work in the real world, chances are you’ll need work on a copy of production data from time to time. You
can try to anonymous all the data but there is always a chance you’ll miss something. Configuring blackhole
as your applications default SMTP gateway will remove any chance of a real person receiving an email they
shouldn’t have received.
CONTENTS
1
.oo
d88’
888
888
‘Y8b
Blackhole Documentation, Release 1.6.0
2
CONTENTS
CHAPTER
ONE
USING THE BLACKHOLE.IO SERVICE
All data sent to blackhole.io will be forgotten instantly, we store nothing you send.
1. Point your application’s outgoing SMTP server to ‘blackhole.io’,
2. Sit back and watch mail never get delivered to a real user.
or, send an email to blackhole.io using an @blackhole.io address, any address is fine e.g.:
[email protected]
1.1 Testing via telnet
$ telnet blackhole 25
Trying 198.199.126.159...
Connected to blackhole.io.
Escape character is ’^]’.
220 2.2.0 OK, ready
HELO fake.mail.server
250 2.5.0 OK, done
MAIL FROM:<[email protected]>
250 2.5.0 OK, done
RCPT TO:<[email protected]>
250 2.5.0 OK, done
DATA
354 3.5.4 Start mail input; end with <CRLF>.<CRLF>
To: Someone <[email protected]>
From: User <[email protected]>
Subject: Bye
Bye bye email
.
251 2.5.1 OK, user not local, will forward
QUIT
221 2.2.1 Thank you for speaking to me
Connection closed by foreign host.
1.2 Testing SSL
3
Blackhole Documentation, Release 1.6.0
$ openssl s_client -connect blackhole.io:465
CONNECTED(00000003)
depth=0 C = GB, ST = London, L = London, O = blackhole.io, OU = blackhole.io, CN = blackhole.io, emai
... snip ...
--220 2.2.0 OK, ready
HELO fake.mail.server
250 2.5.0 OK, done
MAIL FROM:<[email protected]>
250 2.5.0 OK, done
RCPT TO:<[email protected]>
250 2.5.0 OK, done
DATA
354 3.5.4 Start mail input; end with <CRLF>.<CRLF>
To: Someone <[email protected]>
From: User <[email protected]>
Subject: Bye
Bye bye email
.
251 2.5.1 OK, user not local, will forward
QUIT
221 2.2.1 Thank you for speaking to me
DONE
4
Chapter 1. Using the blackhole.io service
CHAPTER
TWO
GETTING THE SOURCE CODE
The source code is available under the MIT license from GitHub.
5
Blackhole Documentation, Release 1.6.0
6
Chapter 2. Getting the source code
CHAPTER
THREE
RUNNING YOUR OWN SERVER
3.1 Python versions
Python 2.6
Python 2.7
Python 3.2
Python 3.3
PyPy 1.9
PyPy 2.0
# see notes below
# see notes below
Blackhole works on Python 2.6 and 2.7, it also works with PyPy (see Blackhole + PyPy section below).
3.2 Third party libraries
tornado>=2.2.1,<=3.1
setproctitle>=1.1.6
deiman>=0.1.4
# setproctitle 1.1.7 and above are required for all PyPy versions
# older version of Deiman will not work because of API changes
3.3 Getting started
3.3.1 Installation
Packaged
From PyPI
pip install blackhole
Or
easy_install blackhole
From GitHub
pip install -e git+git://github.com/kura/blackhole.git#egg=blackhole
7
Blackhole Documentation, Release 1.6.0
From source
Download the latest tarball from PyPI or GitHub. Unpack and run:
python setup.py install
3.3.2 Configuration options
Configuration options can be passed via the command line as below:
--workers=NUM
Number of child workers to spawn (default: # of CPUs/cores + 1
master)
--host=IP
IP address to bind go (default: 0.0.0.0)
--port=PORT
Port to listen for connections on (default: 25)
--pid=FILE
File to write process information to (default: /tmp/blackhole.pid)
--log=FILE
File to write logs to (not very verbose) (default: /tmp/blackhole.log)
--user=USER
User to drop privs to during run time. (default: blackhole)
--group=GROUP
Group to drop privs to during run time. (default: blackhole)
--mode=MODE
Mode to run blackhole in (accept, bounce, random, unavailable, offline) (default: accept) - see Modes section
--debug=BOOL
Will set the debug flag and causes all received and sent command/email data to be logged. Will degrade performance with high
throughput (default: False)
--ssl=BOOL
Enabled or disable SSL, requires SSL compiled in to Python and
OpenSSL. True or False (default: True)
--ssl_port=PORT
Port to listen for SSL enabled connections (default: 465)
--ssl_key=PATH
X509 SSL keyfile
--ssl_cert=PATH
X509 SSL certificate file
SSL options
You can also specify the –conf option to load configuration from a file:
--conf=FILE
Config file to parse and use. Overrides command line args
For more information on the configuration file format read Configuration file example.
Deprecated options
The following options are deprecated and should not be used.
--ssl_ca_certs_dir=PATH Path to your operating system’s repository of certificates authorities (default: None)
3.3.3 Configuration file example
8
Chapter 3. Running your own server
Blackhole Documentation, Release 1.6.0
host="0.0.0.0"
port=25
pid="/tmp/blackhole.io"
mode="offline"
ssl_key=/etc/ssl/private/blackhole.io.key
ssl_cert=/etc/ssl/certs/blackhole.io.crt
3.3.4 Debug Flag
After much debate and several emails from developers a debug flag has been added to blackhole.
You can enable debug using command line options Configuration options or via the configuration file Configuration
file example.
Enabling the debug flag will log all incoming and outgoing commands to the blackhole log file as well as all data sent
after a DATA command.
Each connection/email gets it’s own unique identifier, this indentifier uses only hexidecimal characters and is 10
character in length, which means at some point the identifier will be reused.
Example debug output
2013-04-14
2013-04-14
2013-04-14
2013-04-14
2013-04-14
2013-04-14
2013-04-14
2013-04-14
2013-04-14
2013-04-14
2013-04-14
2013-04-14
2013-04-14
2013-04-14
2013-04-14
2013-04-14
2013-04-14
2013-04-14
2013-04-14
07:30:30,788
07:30:31,813
07:30:31,814
07:30:35,074
07:30:35,075
07:30:38,402
07:30:38,402
07:30:40,573
07:30:40,573
07:30:47,143
07:30:52,181
07:30:54,647
07:30:56,550
07:30:57,372
07:30:58,286
07:30:58,287
07:31:04,023
07:31:04,024
07:31:04,024
-
Connection from ’127.0.0.1’
[9AF260AB7F] RECV: HELO
[9AF260AB7F] SEND: 250 2.5.0 OK, done
[9AF260AB7F] RECV: MAIL FROM:<[email protected]>
[9AF260AB7F] SEND: 250 2.5.0 OK, done
[9AF260AB7F] RECV: RCPT TO:<[email protected]>
[9AF260AB7F] SEND: 250 2.5.0 OK, done
[9AF260AB7F] RECV: DATA
[9AF260AB7F] SEND: 354 3.5.4 Start mail input; end with <CRLF>.<CRLF>
[9AF260AB7F] RECV: From: <[email protected]>
[9AF260AB7F] RECV: To: <[email protected]>
[9AF260AB7F] RECV: Subject: Testing debug
[9AF260AB7F] RECV: Hi [email protected]
[9AF260AB7F] RECV: This is a debug message
[9AF260AB7F] RECV: .
[9AF260AB7F] SEND: 250 2.5.0 OK, done
[9AF260AB7F] RECV: QUIT
[9AF260AB7F] SEND: 221 2.2.1 Thank you for speaking to me
Closing
3.3.5 Controlling the server
The following commands are for use when using blackhole without an init script.
blackhole start
blackhole stop
blackhole restart
blackhole status
Starts the server
Stops the server
Restarts the server
Shows the status of the server, running, not running etc
For more information on controlling the server using init.d/rc.d read Controlling the server with init.d/rc.d.
3.3. Getting started
9
Blackhole Documentation, Release 1.6.0
3.3.6 Controlling the server with init.d/rc.d
/etc/init.d/blackhole start
/etc/init.d/blackhole stop
/etc/init.d/blackhole restart
/etc/init.d/blackhole status
Starts the server
Stops the server
Restarts the server
Shows the status of the server, running, not running etc
Installating the init.d/rc.d scripts
The init script depends on /etc/blackhole.conf being in place and configured.
Blackhole comes with a script that works with init.d/rc.d, to install it copy it from the init.d/YOUR_DISTRO folder in
the root directory of this project to /etc/init.d/.
The init scripts can be found here.
i.e. for Debian/Ubuntu users, mv the file from init.d/debian-ubuntu/ to /etc/init.d/.
Then make sure it’s executable:
chmod +x /etc/init.d/blackhole
To make blackhole start on a reboot use the following:
update-rc.d blackhole defaults
3.3.7 Modes
The server can run in several different modes, these are outlined below.
See the Response codes section for more information on responses and which mode responds with which codes.
3.3.8 accept
Accept all email with code 250, 251, 252 or 253.
3.3.9 bounce
Bounce all email with a random code, excluding 250, 251, 252, 253.
3.3.10 random
Randomly accept or bounce all email with a random code.
3.3.11 unavailable
Server always respondes with code 421 - service is unavailable.
3.3.12 offline
Server always responds with code 521 - server does not accept mail.
10
Chapter 3. Running your own server
Blackhole Documentation, Release 1.6.0
3.3.13 Response codes
All
Code
220
221
250
251
252
253
354
355
421
450
451
452
454
458
459
500
501
502
503
504
521
530
534
538
550
551
552
553
554
571
message
OK, ready
Thank you for speaking to me
OK, done
OK, user not local, will forward
OK, cannot VRFY user but will attempt delivery
OK, messages pending
Start mail input; end with <CRLF>.<CRLF>
Octet-offset is the transaction offset
Service not available, closing transmission channel
Requested mail action not taken: mailbox unavailable
Requested action aborted: local error in processing
Requested action not taken: insufficient system storage
TLS not available due to temporary reason
Unable to queue message
Not allowed: unknown reason
Command not recognized
Syntax error, no parameters allowed
Command not implemented
Bad sequence of commands
Command parameter not implemented
Machine does not accept mail
Must issue a STARTTLS command first
Authentication mechanism is too weak
Encryption required for requested authentication mechanism
Requested action not taken: mailbox unavailable
User not local
Requested mail action aborted: exceeded storage allocation
Requested action not taken: mailbox name not allowed
Transaction failed
Blocked
Accept
This mode will respond with the following codes:
Codes
250 251
252
253
Bounce
This mode will respond with the following codes:
Codes
421 431
521 534
450
550
451
551
3.3. Getting started
452
552
454
553
458
554
459
571
11
Blackhole Documentation, Release 1.6.0
Offline
This mode will respond with the following codes:
Codes
521
Unavailable
This mode will respond with the following codes:
Codes
421
12
Chapter 3. Running your own server
CHAPTER
FOUR
TESTS & COVERAGE
4.1 Running tests manually
Running tests manually is pretty simple, there is a Make target dedicated to it.
The test suite relies on unittest2 and nose, both if which get installed by the Make target during test running.
make tests
There is also a Make target for generating coverage:
make coverage
4.2 Third party CI/Coverage
You can find the latest build status on travis.
And the test coverage report on coveralls.
13
Blackhole Documentation, Release 1.6.0
14
Chapter 4. Tests & Coverage
CHAPTER
FIVE
BLACKHOLE + PYPY
PyPy is a Python interpreter and just-in-time compiler. PyPy focuses on speed, efficiency and compatibility with the
original CPython interpreter.
Blackhole works well under PyPy 1.9, 2.0 beta1 and 2.0 beta2, you can see performance improvements of up to 30%
in certain situations.
However, blackhole does have issues with both PyPy 1.9 and 2.0 beta1 and 2.0 beta2 when using the pre-compiled
binaries, this is due to a conflict in the version of OpenSSL compiled in to PyPy and the version compiled in to your
CPython installation. If you wish to use blackhole with SSL support on PyPy I suggest you either compile PyPy
yourself or try to make sure your PyPy and CPython have the same versions.
15
Blackhole Documentation, Release 1.6.0
16
Chapter 5. Blackhole + PyPy
CHAPTER
SIX
FAQ
A few people have emailed me questions about why blackhole exists, how I use it, why Tornado and things like that
so I have outlined some questions and responses in an faq.
17
Blackhole Documentation, Release 1.6.0
18
Chapter 6. FAQ
CHAPTER
SEVEN
REFERENCE
7.1 blackhole.connection
Provide mechanisms for processing socket data.
This module provides methods for Blackhole to use internally for binding and listening on sockets as well as process
all incoming socket data and responding appropriately.
7.1.1 blackhole.connection
blackhole.connection.sockets()
Spawn a looper which loops over socket data and creates the sockets.
It should only ever loop over a maximum of two - standard (std) and SSL (ssl).
This way we’re able to detect incoming connection vectors and handle them accordingly.
A dictionary of sockets is then returned to later be added to the IOLoop.
blackhole.connection.connection_stream(connection)
Detect which socket the connection is being made on, create and iostream for the connection, wrapping it in
SSL if connected over the SSL socket.
The parameter ‘connection’ is an instance of ‘socket’ from stdlib.
blackhole.connection.handle_command(line, mail_state)
Handle each SMTP command as it’s sent to the server
The paramater ‘line’ is the currently stream of data ending in ‘n’. ‘mail_state’ is an instance of ‘blackhole.state.MailState’.
blackhole.connection.connection_ready(sock, fd, events)
Accepts the socket connections and passes them off to be handled.
‘sock’ is an instance of ‘socket’. ‘fd’ is an open file descriptor for the current connection. ‘events’ is an integer
of the number of events on the socket.
blackhole.connection.write_response(stream, mail_state, resp)
Write the response back to the stream
7.2 blackhole.data
Provides SMTP response codes and methods for returning the correct response code.
19
Blackhole Documentation, Release 1.6.0
This module contains all usable SMTP response codes for returning through the socket. It also provides mechanisms
for picking response codes that mean a mail message has been accepted, rejected or that the server is offline.
7.2.1 blackhole.data
blackhole.data.response(response)
Return an SMTP response code and message. ‘response’ an string refering to the code you wish to return.
blackhole.data.get_response()
Get our response from available responses based on configuration settings.
blackhole.data.random_choice(response_list)
Pick a random choice for the configured choices dictionary.
‘response_list’ is a list of available response types from ‘blackhole.data’, this can be: - ACCEPT_RESPONSES - BOUCNE_RESPONSES - OFFLINE_RESPONSES - UNAVAILABLE_RESPONSES
- RANDOM_RESPONSES
blackhole.data.response_message(response)
Format our response in ESMTP format.
‘response’ is a string containing the reponse code you wish to return.
7.3 blackhole.log
Logging configuration for the blackhole server.
7.4 blackhole.opts
Command line options for the blackhole server.
Also creates a list of available ports for the server to be run on, based on configuration and responds with the help
menu when requested or invalid options are given.
7.4.1 blackhole.opts
blackhole.opts.ports()
Create and return a list of sockets we need to create.
A maximum of two will be returned, default is standard (std) and SSL (ssl) it is enabled.
blackhole.opts.print_help(file)
Prints all the command line options to stdout.
7.5 blackhole.ssl_utils
Utility functions for SSL wrapped sockets.
This module provides a simple default SSL configuration for the blackhole server and also exposes a custom ‘BlackholeSSLException’ for SSL-based exceptions.
20
Chapter 7. Reference
Blackhole Documentation, Release 1.6.0
7.5.1 blackhole.ssl_utils
exception blackhole.ssl_utils.BlackholeSSLException
A simple Exception class
blackhole.ssl_utils.verify_ssl_opts()
Verify our SSL configuration variables are correctly set-up.
7.6 blackhole.state
State object for the current connection.
7.6.1 blackhole.state
class blackhole.state.MailState
A state object used for remembering the current connections place in our runtime.
This is mostly used for figuring out if we’re receiving SMTP commands or have trigger the DATA command.
email_id
Email ID is used to assign commands sent and received against an email/connection ID.
Only utilized when debug flag is set.
reading
MailState’s ‘reading’ property, used for figuring out where we are in the state chain.
7.7 blackhole.utils
A utility module used for methods and features that do not belong in their own module.
7.7.1 blackhole.utils
blackhole.utils.setgid()
Change our existing group.
Used to drop from root privileges down to a less privileged group.
MUST be called BEFORE setuid, not after.
blackhole.utils.setuid()
Change our existing user.
Used to drop from root privileges down to a less privileged user
MUST be called AFTER setgid, not before.
blackhole.utils.terminate(signum, frame)
Terminate the parent process and send signals to shut down it’s children
Iterates over the child pids in the frame and sends the SIGTERM signal to shut them down.
blackhole.utils.set_process_title()
Set the title of the process.
If the process is the master, set a master title, otherwise set worker.
7.6. blackhole.state
21
Blackhole Documentation, Release 1.6.0
blackhole.utils.email_id()
Generate an HEX ID to assign to each connection.
Will be reused later down the line due to the limited number of characters.
22
Chapter 7. Reference
CHAPTER
EIGHT
INDICES AND TABLES
• genindex
• modindex
• search
23
Blackhole Documentation, Release 1.6.0
24
Chapter 8. Indices and tables
PYTHON MODULE INDEX
b
blackhole.connection (Unix), 19
blackhole.data (Unix), 19
blackhole.log (Unix), 20
blackhole.opts (Unix), 20
blackhole.ssl_utils (Unix), 20
blackhole.state (Unix), 21
blackhole.utils (Unix), 21
25