Email filtering

What is it?

"Email filtering" -- sometimes called "email blocking" -- is the use of software to limit transmission of electronic mail messages.

If you're interested in learning how email filtering works, and the general considerations for using it, read on.  If you just want the instructions for managing the filtering process at the School of Medicine, click here.

Filtering in both directions

Filtering is now commonly applied to inbound email, to block spam, phishes and other annoying/dangerous messages, as well as to intercept email with potentially dangerous attachments (containing malware or spyware).

Filtering may also be applied to outbound email messages, to control transmission of sensitive information, or to intercept messages that may risk legal liability for other reasons.

Filtering by clients and servers

Filtering capabilities may be built into an email client -- the software that is installed on your personal computer or portable computing device.  Filters may also be installed at an email server level to protect all users within an organization's network. 

The School of Medicine has server-level filters protecting the campus networks and all workplace email systems.  Most Internet Service Providers (ISPs) now offer some sort of network-level filtering for their customers too, though the quality varies.

How does it work?

Email filters can pass a incoming message through unchanged, alter its contents before allowing transmission (such as by removing a potentially dangerous attachment), re-route the message to another destination (such as a "holding area" for human inspection), or simply discard it outright. 

All these actions are also options for outgoing email.  In addition, some systems have the capability of triggering an encryption service to encode the message; that encoding may be optional or mandatory depending on the setup.  (Outbound filtering is less common than inbound filtering, but its use is growing.)

Filtering rules

How does the filtering software "decide" what to do?   Decisions are based on a combination of rules applied to each message's content.  Filtering programs may:

  • check the address or other header information for the message;
  • look for particular keywords in the message body, its subject field, or the message contents; or
  • monitor for particular types of attachments (such as .exe executables).

Sophisticated statistical techniques may be used to compute probabilities -- and take action -- based on particular combinations, frequencies and spellings of words.  The strange spelling and syntax common to spam messages is not a reflection of spammers' poor grammatical skills, at least not entirely; it's an attempt to outsmart the email filters.

Probabilities and decision rules for filtering may be based on recent local experience, such as the spam coming into an organization's network that day or week, or on statistical analyses of recent spam traffic world-wide.  It may also be based on an email user's explicit declaration that something is or is not to be filtered -- more on that process below.

The dynamic adaptability of today's filtering software is the key to its effectiveness in a world that is increasingly filled with junk email.  Unfortunately, that also means that you cannot entirely predict what will get into your inbox (or out of your outbox) based on past experience.

What about privacy?

All filtered email is being "read" -- but only by a computer program.  However, keep in mind that one of the actions a filtering program may take is to forward suspicious email to a holding area for inspection by a human.

Be aware that you have no privacy rights with respect to emails sent via workplace systems.  The law considers these to be corporate documents, subject to inspection not only by the organization itself but also by legal/regulatory authorities.  That's one of many reasons why you should confine your personal correspondence to a personal email account.

What can go wrong?

While email filters can be very accurate, they cannot be perfect.  Intercepting spam is a particularly daunting task, since the people who send spam are constantly adjusting their messages to beat the filtering software. 

Email filter rules must always trade off two kinds of error -- "false negatives" and "false positives" -- and you need to be aware of the consequences of each.

False negatives

Sometimes filters fail to catch inbound messages that should be intercepted.   If the message is spam, simply delete it.  We recommend that you do not respond -- even to try to opt out of future mailings -- and that you never, ever click on any links in a suspicious message or open any file attachments accompanying it.  (For more anti-spam tips, click here.)

At the School of Medicine, you can also forward the spam to spam@med.miami.edu to help us adjust our filtering rules.  We don't need you to do this every time, but it is appreciated for spam that you receive repeatedly or which appears to contain particularly dangerous content.

Some email software allows you to construct your own filtering rules to intercept more messages:

  • Add the particular sender -- that is, the particular email address -- to a "blocked sender" list to prevent all future emails.  (This option is not always effective at preventing spam, however, since many spammers continually change the particular sending address.)
     
  • Add the particular domain -- blocking everyone with an @domain email address -- to a blocked sender list.  
      
  • Construct your own rules -- specifying combinations of addresses, subjects and other elements that identify the kinds of message you always want rejected. 

Such blocking list features should be used with caution, particularly ones that block whole domains, since you may cut off correspondence that you actually want.

False positives

Sometimes filters will block messages that should not be intercepted.  This is potentially a more serious problem, since you generally don't know about messages that you did not receive, and sometimes that means you'll miss an important communication.

At the School of Medicine, most Exchange users receive a list of filtered emails (sent every twenty-four hours).  This spam reporting can be discontinued if you find that all the intercepted messages are indeed ones you want intercepted, or you just get tired of reading the reports and are willing to take your chances.  (Send your opt-out request to the Help Desk at help@med.miami.edu.)

As with blocking, some email software allows you to construct your own filtering rules to allow more messages:

  • Add the particular sender -- again, the particular email address -- to a "safe sender" list to allow all future emails. 
     
  • Designate the entire domain -- everyone with an @domain address -- as safe. 
     
  • Construct your own rules -- combinations of addresses, subjects and other elements that identify a safe message.

Particularly sophisticated email software may be capable of bring "trained" by your history of accept and reject decisions, without you having to specify any particular rules.  But no matter how sophisticated, there will always be some errors.

If receipt of a particular email is critical, we urge you to ask for an explicit email response from the recipient, or a confirmatory follow-up by telephone.  Although the vast majority of emails get to their intended destination without incident, it does not happen 100 percent of the time.  Filtering is only one of the reasons why, though it is an increasingly common reason.

Learn more

Bayesian spam filtering (Wikipedia)
Using probability theory to filter the "spam" from the "ham"

Category: spam filtering (Wikipedia)
Links to more than 30 entries on the topic

Spamhaus (The Spamhaus Project)
Lots of information about spam and other malicious email, with real-time data

TrustedSource.org (Ciphertrust) **on-campus only**
More real-time data on malicious email, including an animated "ZombieLocator" map