Microsoft® Exchange Server 2007

Microsoft® Exchange Server 2007
Microsoft® Exchange
Server 2007:
Tony Redmond’s Guide to
Successful Implementation
This page intentionally left blank
Microsoft® Exchange
Server 2007:
Tony Redmond’s Guide to
Successful Implementation
Tony Redmond
Amsterdam • Boston • Heidelberg • London • New York • Oxford
Paris • San Diego• San Francisco • Singapore • Sydney • Tokyo
Digital Press is an imprint of Elsevier
Digital Press is an imprint of Elsevier
30 Corporate Drive, Suite 400, Burlington, MA 01803, USA
Linacre House, Jordan Hill, Oxford OX2 8DP, UK
Copyright © 2007, Hewlett-Packard Development Company, L.P. Published by
Elsevier. All rights reserved.
No part of this publication may be reproduced, stored in a retrieval system, or
transmitted in any form or by any means, electronic, mechanical, photocopying,
recording, or otherwise, without the prior written permission of the publisher.
Permissions may be sought directly from Elsevier’s Science & Technology Rights
Department in Oxford, UK: phone: (+44) 1865 843830, fax: (+44) 1865 853333,
E-mail: [email protected] You may also complete your request online
via the Elsevier homepage (, by selecting “Support & Contact”
then “Copyright and Permission” and then “Obtaining Permissions.”
Recognizing the importance of preserving what has been written, Elsevier prints its
books on acid-free paper whenever possible.
Library of Congress Cataloging-in-Publication Data
Application Submitted.
British Library Cataloguing-in-Publication Data
A catalogue record for this book is available from the British Library.
ISBN: 978-1-55558-347-7
For information on all Elsevier Digital Press publications visit our Web site at
Printed in the United States of America
07 08 09 10 11 12 10 9 8 7 6 5 4 3 2 1
A decade and counting of Exchange deployments
The way we were
The protocol wars
Ever increasing mobility
Third-party products and management
Some interesting projects
The not so good points
Exchange’s connection with the Active Directory
Reviewing predictions made in 1996
Microsoft’s themes for Exchange 2007
The happy prospect of a migration
Preparing for Exchange 2007
Installing Exchange 2007
Modifying and removing servers
Validating the installation
Third-party software
Server roles
Version numbers
32-bit Exchange 2007?
Challenges for Exchange 2007
Into the future
Exchange, Windows, and the Active Directory
Active Directory and Exchange
Domain Designs
Active Directory replication
Replication basics
When Active Directory replication happens
Active Directory naming contexts
Transforming Domain controllers into
Global Catalogs 58
USNs and replication
Urgent replication
Intrasite and Intersite replication
High-watermark vector and up-to-date vector tables
Changes in Active Directory replication in Windows 2003
Exchange’s Active Directory Topology service
DSAccess (or ADAccess)
How many Global Catalog servers do I need?
Where are my Global Catalogs?
Recovering deleted Active Directory accounts
Exchange and the Active Directory schema
Updating the schema with an installation
Changing the schema
Active Directory custom attributes for Exchange
Updating the schema to allow Ambiguous
Name Resolution
Exchange-specific permissions
Exchange property sets
Longhorn and Exchange 2007
The very important LegacyExchangeDN attribute
Brain surgery for the Active Directory: ADSIEDIT
Active Directory for Exchange
The Basics of Managing Exchange 2007
Exchange Management Console
The importance of filters
Managing mixed organizations
Running EMC remotely or on a workstation
No more AD Users and Computers
Changing columns
Visual effects
Why some options have disappeared from EMC
Coping with change
3.3 Changes in the Exchange delegation model
3.4 Customized Recipient Management
Adieu RUS
Recipient types
3.5 Moving users
Moving mailboxes
Logging mailbox moves
3.6 Using distribution groups
Forming groups
Group changes in Exchange 2007
Expanding distribution lists
How many objects can I have in a group?
Managing group membership
Protected groups (and users)
3.7 Using groups for permissions
Managing distribution groups from Outlook
3.8 Dynamic distribution groups
Changing filters and conditions for dynamic
distribution groups
A note on OPATH
A new UI for dynamic groups
Creating New dynamic groups
Using dynamic Distribution groups
3.9 Mailbox quotas
Setting mailbox quotas
3.10 Email address policies
3.10.1 Mailbox moves and email address policies
3.10.2 Queries that drive email address policies
3.11 Address lists
3.11.1 Upgrading Address Lists to Exchange 2007 format
3.12 User naming conventions
3.13 Server naming conventions
3.14 Moving from the basics
The Exchange Management Shell
EMS: Exchange’s management shell
Working with PowerShell commands
Exchange shell commands
Command editing
Getting at more information about something
Using common and user-defined variables
Working in a multi-domain forest
PowerShell in batch
4.1.10 Execution policies
4.1.11 Sending email from the shell
4.2 Learning from EMC
4.3 Using EMS to work with mailboxes
Creating a new mailbox with a template
Setting and retrieving mailbox properties
Other ways of interacting with mailboxes
Moving mailboxes
Accessing another user’s mailbox
Different commands and different properties
4.4 Working with distribution groups
Working with dynamic distribution groups
Advanced group properties
4.5 Delegation through the shell
4.6 Creating efficient filters
4.7 Bulk updates
Creating sets of mailboxes
4.8 Reporting mailbox data
Special properties
4.9 Using the shell for other management tasks
4.10 Command validation
4.11 Working with remote servers
4.12 Working with non-Exchange 2007 servers
4.13 Testing Exchange 2007
4.13.1 Client connections
4.13.2 Mail Flow
4.13.3 Miscellaneous test commands
4.14 PowerShell for Exchange administrators
The Store
Introducing the Store
Differences in the Exchange 2007 Store
Are 64 bits that important?
Trading memory for I/O
The decrease in storage costs
5.3 No more streaming database
5.4 Tables and items
5.5 Storage groups
Creating a new storage group and database
Working with storage groups and databases
5.6 Transaction logs
Circular logging
Creating new transaction logs
Reserved logs
Transactions, buffers, and commitment
Transaction log I/O
Protecting transaction logs
Transaction log checksum
Maximum database size
5.7 Database portability
Zero database pages
5.8 MAPI connections and logons
5.9 The Deleted Items cache
Cleaning the Deleted Items cache
Recovering items and mailboxes
5.10 Background maintenance
5.10.1 Background tasks
5.10.2 Tracking background maintenance
5.11 Fixing failed databases
5.12 Exchange 2007 content indexing
5.12.1 Using content indexing
5.13 Public folders
5.13.1 Public folders and Exchange 2007
5.13.2 Changes in public folders administration since
Exchange 2003
5.13.3 Calming replication storms
5.13.4 Managing public folders with Exchange 2007
5.13.5 Permissions on top-level folders
5.13.6 Referrals
5.13.7 Migrating public folder content
5.14 Removing database size limits
5.15 Backups
5.15.1 NTBackup
5.15.2 Other commercial backup products
5.15.3 Creating a backup strategy
5.15.4 Backups and storage groups
5.15.5 Checkpoint file
5.15.6 The future of streaming backups
5.16 Moving from the Store
Exchange Transport and Routing
The evolution of routing
Change through experience
Hidden administrative and routing groups
Exchange 2007 transport architecture
The critical role of hub transport servers
Receive connectors
Send connectors
Linking Exchange 2003 and Exchange 2007
Multiple routes into Exchange 2003
Decommissioning Exchange 2003 routing groups
Handling Exchange 2003 link state updates
during migration
Foreign connectors
6.3.10 Accepted domains
6.3.11 Transport storage
Routing ABC
Resolving multiple paths
Most specific connector
Connector cost
Closest proximity
The role of hub routing sites
Site link costs versus routing costs
Instructing mailbox servers
Bypassing some connections
Protocol logging
6.4.10 X.400 support
6.4.11 Bifurcation
6.4.12 Header firewalls
Transport configuration
Transport configuration file
Routing logs
The Queue Viewer
The Unreachable queue
Poison messages
6.7 Back Pressure
6.8 Delivery Status Notifications
Customizing DSNs
Postmaster addresses
6.9 Transport agents
6.10 Transport summary
6.11 Edge servers
6.11.1 Edge or hub?
6.11.2 Basic Edge
6.11.3 Edge Synchronization
6.11.4 Basic Edge security
6.11.5 Fighting spam and email viruses
6.11.6 Defense in depth
6.11.7 Microsoft’s approach to mail hygiene
6.11.8 Forefront for Exchange
6.11.9 Mail Hygiene Agents
6.11.10 Agent logs
6.11.11 Connection filtering
6.11.12 Sender filtering
6.11.13 Address Rewrite agent
6.11.14 Sender ID agent
6.11.15 Content filtering
6.11.16 Content Filter updates
6.11.17 Per-user SCL processing
6.11.18 Safelist Aggregation
6.11.19 Sender reputation
6.11.20 Recipient filtering
6.11.21 Blocking file attachments
6.11.22 Attachment filtering
6.11.23 Edge transport rules
6.11.24 Available Edge
6.12 Client-side spam suppression
6.12.1 Outlook’s Junk Mail Filter
6.12.2 Postmarks
6.12.3 Restricting OOF and other notifications
6.13 Routing onwards
Outlook web services
Understanding Outlook’s relationship with Exchange
Deploying cached Exchange mode
Address caching
MAPI compression and buffers
Conflict resolution
Preventing MAPI clients from connecting
Outlook 2007 and Exchange 5.5
Offline and personal Stores
Personal folders
Mail delivery to personal folders
Configuring PSTs
PST archiving
Offline folder files
OST synchronization
When things go wrong with your OST
Out of Office changes
The big question: Is Outlook 2007 worth the upgrade?
The Offline Address Book (OAB)
Downloading the OAB
OAB files on the PC
The evolving OAB format
OAB and cached Exchange mode
OAB generation and distribution
Creating a customized OAB
Allocating OABs to users
Outlook Anywhere
Outlook Web Access
New features in Outlook Web Access 2007
Outlook Web Access Light
International versions
Accessing legacy data
Managing Outlook Web Access
Controlling attachments
7.7.10 Themes
7.7.11 Client settings
Internet client access protocols
The Exchange 2007 IMAP server
7.9 Mobile clients
Selecting mobile devices
Server-based ActiveSync
7.10 Windows Mobile 6.0 and Exchange 2007
7.10.1 ActiveSync policies
7.10.2 Managing mobile devices through EMC
7.10.3 Moving mailboxes to Exchange 2007 and ActiveSync
7.10.4 Estimating network traffic for mobile devices
7.10.5 Analyzing ActiveSync logs
7.10.6 Wiping mobile devices
7.10.7 Debugging synchronization
7.11 Comparing Windows Mobile and BlackBerry
7.11.1 Processing the mail
7.11.2 Other messaging options for Windows Mobile
7.11.3 Power management
7.11.4 Input flexibility
7.12 Unified Communications
7.13 Unified Messaging
7.13.1 Client Access to voicemail
7.13.2 Dealing with voicemail
7.13.3 Voice synthesis
7.13.4 Pure voicemail
7.13.5 The magic of SIP
7.13.6 Speech Grammars
7.13.7 Phonetic names
7.13.8 Cross-forest UM
7.14 Special mailboxes
7.15 Clients and users
Managing Users
Room and equipment mailboxes
Managing properties of room and equipment mailboxes
Converting old mailboxes to rooms
8.2 Helping users to use email better
Eliminating bad habits
Out-of-Office Notifications
The last few bad email habits
Customizing display templates
Exchange 2007 and compliance
The growing need for compliance
Transport rules
Using a rule to add disclaimer text to outgoing messages
Capturing selected messages
Becoming more complicated
Creating an ethical firewall
Transport rule storage
Rules and the shell
Journal rules
Messaging Record Management
Managing default folders
Managing custom folders
Allocating managed folders with policies
Applying policies to users
The Managed Folder Assistant
Logging Managed Folder activity
Using Managed Folders
Harvesting information from managed folders
Message classifications
Adding intelligence to classification through rules
Copying user mailboxes
Free and busy
Looking at free and busy data
Free and busy in Exchange 2007
Changes in Outlook 2007
Cross-forest free and busy
Hardware and Performance
Moving toward 64-bit Exchange
Buying servers for Exchange 2007
The storage question
RPC pop-ups
Clusters and Exchange
Continuous replication and Exchange 2007
9.7 Deploying Local Continuous Replication (LCR)
How LCR works
LCR operations
LCR restrictions
LCR database transition
9.8 Deploying Cluster Continuous Replication (CCR)
Comparing CCR and traditional clusters
CCR in practice
CCR failovers
Lost Log Resilience
The transport dumpster
Standby Continuous Replication
9.9 Continuous Log Replication: Good or bad?
9.10 Virtual Exchange
10 More useful things to Know about Exchange
10.1 Automated analysis
10.1.1 SSCP
10.1.2 Microsoft’s Release to Web (RTW) strategy
10.2 The Exchange Toolbox
10.2.1 Updates
10.2.2 Database Recovery Management
10.2.3 Database Troubleshooter
10.2.4 Mail Flow Troubleshooter
10.3 Messaging tracking logs
10.3.1 Generating message tracking logs
10.3.2 Log sizes and ages
10.3.3 Keeping track of message subjects
10.3.4 Accessing message tracking logs
10.3.5 Using the Troubleshooting Assistant to track messages
10.3.6 Tracking messages with EMS
10.3.7 Message delivery latency
10.4 Management frameworks
10.5 Utilities
10.5.1 Performance testing
10.5.2 The MFCMAPI utility
10.5.3 MDBVU32
10.5.4 ExMon—Exchange User Monitor
10.5.5 PFDavAdmin
10.5.6 LogParser
10.5.7 Outlook Spy
10.6 Bits and pieces
10.6.1 Where the Exchange team hangs out
10.6.2 Online Forums
10.7 Conferences
10.7.1 Magazines
10.7.2 How Exchange uses registry keys
10.8 Good reference books
Message Tracking Log Format
Events noted in Message Tracking Logs
Important Exchange PowerShell commands
Recipient management commands
Exchange server administrative Commands
Databases and Storage Groups
Address Lists and Email Policies
Queues and Messages
Edge Synchronization
Public folders
Transport and journal rules
Active Directory commands
Testing Exchange 2007
Basic PowerShell
PowerShell control commands
By their very nature, every book that seeks to describe how technology
works face challenges during its creation. Dealing with beta software and
attempting to resolve the difference between how the software works and
how the developers say it will work in the final version is a problem faced by
any author, which is one reason why it is often best to wait to finalize text
after you have a chance to work with released software. Looking back at this
project, in some ways, this has been the hardest book of the seven that I
have written about Exchange. I think that there are four reasons why this
might be so.
First, Exchange 2007 marks the boundary for substantial architectural
change within the product, so it is similar to the degree of change that we
experienced when we moved from Exchange 5.5 to Exchange 2000. Second,
the nature of software is that it becomes more complex over time as the
developers add new features and this is certainly true of Exchange 2007. The
new features have to be considered, probed, and documented, all of which
takes time. Third, the Exchange development team has done an excellent job
since 2004 to document all aspects of Exchange in a more comprehensive
manner than ever before. The Exchange 2007 help file, TechNet, MSDN,
and the excellent Exchange team blog at
default.aspx are interesting and productive hoards of information for authors
to mine. Unfortunately, there is often too much material (a good complaint
to have) and the material needs to be interpreted and analyzed in the light of
your own experience with Exchange. Engineers write great blogs, but the
scourge of cognitive dissonance often means that they omit some detail that
makes all the difference to a newcomer in understanding why a component
works the way that it does.
Last but not least, you should not underestimate the degree of cultural
change that Microsoft has incorporated into Exchange 2007 in the transition
from a predominantly GUI-centric approach to server management to the
use of the PowerShell scripting language as the basis of many management
operations. The need to understand and appreciate the change has to occur
before you can adequately document and describe the benefits and this
increases the effort required to write the book. I must admit that it took me
time to realize the full benefit of interacting with Exchange through the shell,
but now I am at the point where I wonder why Microsoft never provided
such a powerful interface in the past!
The degree of change that exists in Exchange 2007 means that it is difficult to cover everything in one book. I have therefore elected to cover the
parts of Exchange that I think are of most interest to the majority of administrators and have left other components for you to discover through the
material that Microsoft publishes or perhaps another book, written by me or
someone else. Please accept my apology if I have not covered something that
you think is important and treat this as a challenge and opportunity for you
to write about the topic yourself. There are many magazines, blogs, and other
ways of spreading information about Exchange.
From time to time, I wander back down the path to consider some
aspect of Exchange 2003. While this book is firmly focused on Exchange
2007, the vast majority of companies that will deploy Exchange 2007 will do
so by migrating from Exchange 2003 and will therefore run both products
alongside each other for some period. For large organizations, the period
might extend to a year or more as it is unlikely that few will complete their
migration to a pure Exchange 2007 environment quickly. With this in mind,
it is fair and reasonable to document how things work with Exchange 2003,
especially when these servers operate with Exchange 2007.
So what is in the book? To set the context, Chapter 1 starts with an overview of the development of Exchange from 4.0 to 2007 and then describes the
themes that Microsoft employed to focus the development priorities for
Exchange 2007 and some of the changes that occur in this release. All successful deployments of Exchange since Exchange 2000 operate on a solid Active
Directory foundation, so Chapter 2 reviews some of the critical intersection
points between Exchange and the Active Directory including replication, the
schema, and Global Catalogs. Chapter 3 goes into the basics of managing
Exchange 2007 through the Exchange Management Console. Chapter 4 takes
the management topic further by exploring the ins and outs of the new
Exchange Management Shell, perhaps the most fundamental change to the
product that Microsoft has made in Exchange 2007. Chapter 5 goes to the
heart of Exchange and reviews how the Store works including topics such as
databases, storage groups, and transaction logs to content indexing and backups. Chapter 6 looks at how the new transport system routes messages and
includes topics such as the Edge server and anti-spam protection. Chapter 7
explains how clients from Outlook to Outlook Web Access to mobile devices
allow users to work with their mailboxes. Chapter 8 then moves on to consider some elements of user management, including the important topic of
compliance and records management. Chapter 9 addresses one of the more
difficult topics in hardware and performance. It is difficult because hardware
capabilities change so rapidly that it is hard to give any advice about performance in anything other than outline detail. Finally, Chapter 10 wraps things
up with some miscellaneous items that are important to Exchange, or at least
that I think are important for Exchange administrators to know. I hope that
the book hangs together as a coherent whole.
It is inevitable that I have omitted some topics that you might like me to
have covered. There is so much technology in and around Exchange 2007
that it would take a 2,000 page book to cover it in any detail.
My experience is mostly in the enterprise space, so it should not be a
surprise that many of the opinions expressed in the book reflect that bias.
One of my reviewers noticed this point, and complained that I did not think
that POP3 was an important protocol. Using Exchange 2007 as a hosting
platform is a pretty specialized business and I apologize in advance if I offend
anyone by my concentration on how to deploy Exchange 2007 most effectively for medium to large enterprises.
All errors and omissions are mine, especially in the code samples selected
to illustrate the power of the Exchange Management Shell. PowerShell samples are indicated in the courier typeface like so:
Get-Mailbox –id Redmond | Select DisplayName
Any output from the commands is shown as follows:
Tony Redmond
While all the code worked on one or more test systems, experience tells
me that errors can creep in the process required to take code from a system
through editing and publishing to the final content in a book. This is especially so when the underlying code changes from build to build as the engineers push to finish the product and generate a knock-on effect of changes to
commands and individual parameters. This book does not pretend to be a
comprehensive guide to PowerShell programming or to the Exchange Management Shell and the examples are there to give you a taste of what you can
now do to automate management operations, so any errors that do creep in
should be pretty obvious and easily solved—I hope!
Books do not happen overnight and they represent a lot of work. I have
gained enormously from being able to work alongside some tremendous
experts in enterprise messaging, both inside and outside HP. I acknowledge
the contribution of groups such as my own team, who humored me when I
was writing. The Exchange 2007 academy tutors allowed me to ask many
questions as I probed the content that they generated to train HP consultants
and customers. I must also acknowledge the huge contribution made by the
enterprise messaging team at HP including Kathy Pollert, Mike Ireland, and
Stan Foster (an honorary member), who let me into the details of how
Exchange 2007 into the huge Windows infrastructure that HP operates.
There are many people at Microsoft who patiently answered questions even
if they didn’t realize that this was happening; the amount of information that
Microsoft now generates in help files, blogs, MSDN, TechNet, and Knowledge Base articles is truly staggering and has become a big challenge for people to understand and assimilate. It is great that the information is there, but
just sometimes…. I should also acknowledge and thank the mass of enthusiasts who attend conferences such as Windows and Exchange Connections
who asked about an Exchange 2007 book and eventually prompted me to
start writing.
On my first day with the Exchange team in 2001, I was handed a copy of
Tony Redmond’s Exchange 2000 book, “Here, read this!” It did take me a
while to make my way through that tome, but I still recall thinking that it
was well worth the time, as it laid the foundation for everything that was to
come for me in Exchange.
They were obviously there before me, but I can personally attest that
since that day, Tony’s team at HP have been outstanding partners with us in
designing Exchange 2003 and 2007, helping us test the software throughout
the development, and ultimately working with many customers on their
deployments, migrations, and operations.
We designed Exchange 2007 with three audiences in mind:
The IT executive looking for cost reduction, security, and compliance.
The IT professional looking for operational efficiency.
The end user looking for anywhere access to their email.
I hope you will find with your deployment of Exchange 2007 that
we’ve delighted all three. Since 2005, we’ve been testing Exchange 2007
with more organizations and more end users than any previous release of
Exchange. The end result is a product that we are very proud of here in
Redmond, Washington. We look forward to receiving your feedback about
Exchange 2007 over the coming years.
On behalf of the entire Exchange team, thank you for choosing
Microsoft Exchange!
Terry Myerson ([email protected])
General Manager, Exchange Server
Microsoft Corporation
This page intentionally left blank
A decade and counting of Exchange deployments
Microsoft shipped Exchange 4.0 in March 1996 after a gestation period of
some four years. The new messaging server went through many different
design phases. Microsoft grappled with the challenge of enterprises and small
companies, figured out what they had to do to be competitive, understood
how best to migrate users from other platforms (including their own), and
achieved the necessary performance and scalability levels—albeit limited by
the capabilities of Windows NT 3.51 and the available hardware.
Exchange replaced Microsoft Mail and went into immediate competition with other messaging systems such as those favored by large corporations (IBM PROFS, Digital Equipment Corporation’s ALL-IN-1 and
MailWorks, and HP OpenMail) and the PC LAN-based systems such as
Lotus cc:Mail, Banyan Vines, Novell GroupWise, and Lotus Notes.
Exchange 4.0 was the first version that implemented the initial Exchange
architecture and this generation subsequently spanned Exchange 5.0 and 5.5,
released in March and November 1997 respectively. The second generation
arrived with Exchange 2000 in 2000 and Microsoft developed this version of
the architecture further with Exchange 2003. Exchange 2007 advances the
state of the art by implementing the third distinct architecture for Exchange.
It is hard to realize just how much progress messaging technology has
made since 1996. Exchange has improved its capabilities dramatically in
terms of functionality, robustness, security, and connectivity since 1996. We
have also seen other important advances in the standards that dictate how
systems connect together, the networks that we use, Windows and associated
technology such as IIS, the power and usefulness of the devices that we connect to our mailboxes, and the other technology that has established the type
of world we work in.The web is the best and most pervasive example of a
technology that has influenced Exchange. The volume and depth of change
over the decade has posed a challenge for administrators to keep up to date
1.1 A decade and counting of Exchange deployments
with new developments, and hopefully the articles published about Exchange
and associated technologies in that time have helped to bridge the gap.
The way we were
The messaging market was more fragmented in 1996 than it is in 2007. The
administrator who set out to deploy Exchange 4.0 had to cope with a plethora of competing standards, connections, and clients. Companies such as
SoftSwitch (later bought by Lotus), WorldTalk, and LinkAge (later bought
by Microsoft as part of their push to migrate companies from Notes) built
healthy businesses by producing software to connect different email systems
so that companies could communicate together. The war between the proponents of the international messaging standards (X.400 and X.500) and the
Internet standards hadn’t reached a satisfactory conclusion in 1996, so we
struggled to communicate in a world where you needed a great deal of magic
incantations to send even a plain text message addressed to a single recipient
to a foreign email system.
Government and telecommunications bodies led the charge toward a
common standard for directories that eventually resulted in the X.500 standard. While X.500 offered the potential that it could eventually result in a
global directory standard that everyone used to connect directories to, directory synchronization was another black art in 1996. It was common to have
weekly or monthly synchronization runs to merge directory data to provide a
common view of users across multiple systems. Email addresses were more
convoluted (mine was then [email protected]) than today
as most organizations now use the standard SMTP convention of [email protected] Of course, X.500 has long since faded into the
background and LDAP is now the most widely used standard for directory
access and interoperability. We can still see the influence of X.500 in some
enterprise directories and in the design principles that Microsoft followed to
build the original Exchange Directory Store and then the Active Directory,
but few Exchange administrators bother about X.500 now.
The ease of connectivity established by SMTP, its extensions (ESMTP),
and the easy access that we now enjoy to the Internet has revolutionized
email. This is true for corporate users and personal users. Ten years ago it
would have been difficult to predict the success and ease of access that people around the world enjoy to email systems such as Hotmail, Gmail, and
Yahoo mail.
The protocol wars
MAPI is the great survivor of the protocol wars. MAPI is actually an API, but
many people refer to MAPI as a protocol, in the same way as they refer to
1.1 A decade and counting of Exchange deployments
IMAP4 or POP3; MAPI is also a message format as used in Exchange, so the
email community uses the term in different ways. Microsoft introduced the
first version of MAPI in Microsoft Mail, but this was a very simple version of
the API that Outlook clients use today as it only supported twelve functions.
Capone, the original Exchange client shipped with Exchange 4.0, was the
first client to exploit the full range of MAPI capabilities as made available in
the MAPI 1.0 release. Microsoft developed the Exchange RPC protocol to
wrap around MAPI and Exchange 2007 continues to use Exchange RPCs
(often called MAPI RPCs or often just MAPI) to connect Outlook clients to
servers. There’s also server-side MAPI, which is what Exchange servers use for
server applications that need to access the Store, such as the System Attendant and the Exchange management console. The server variation of MAPI
in Exchange 2003 is tuned to support the kind of multi-threaded applications that you find on servers better, but the difference between the MAPI
library distributed with Exchange and the version that came along with Outlook confused administrators in the past. For example, you could not install
Outlook on an Exchange server because the two versions of the MAPI library
would cause a conflict when they were loaded into the same process space.
Exchange 2007 introduces MAPI.Net—a thoroughly modern version of
server-side MAPI that Exchange uses for communication between servers.
For instance, all of the traffic between mailbox and hub transport servers is
via MAPI.Net RPCs. A side effect of the new version of MAPI is that you
can now install Outlook quite happily on an Exchange 2007 server because
the two versions do not clash anymore. While it’s still possible (but difficult
and time consuming) to write highly efficient and effective MAPI code to
run on the server, Microsoft’s strategy is to move programmers away from
MAPI to use Exchange Web services. The promise is that code will be easier
to write and debug, will deliver better performance, and should be more supportable over the long term. Microsoft tuned the server variation of MAPI in
Exchange 2003 to better support the kind of multi-threaded applications
that servers run, but the difference between the MAPI library distributed
with Exchange 2003 and the version that came along with any version of
Outlook confused administrators in the past.
Exchange 2007 introduces MAPI.Net—a thoroughly modern version of
server-side MAPI. Back on the client side, Microsoft referred to the Capone
client as a “viewer.” This seemed to be an odd name to give to a client, but it
reflected a software engineering perspective that the client was an application
that allowed users to view Exchange data. Capone was elegant, simple, and
precise, but the first release of Outlook in 1997 rapidly passed out the original Exchange client in terms of functionality. Today Outlook boasts a range
of features that most users (except Sue Mosher, the guru of Outlook) find all
the features difficult to comprehend, let alone use. Despite rumblings over
the years (many from within Microsoft), that Exchange should drop MAPI
and use Internet protocols for its clients instead, no Internet client protocol
Chapter 1
1.1 A decade and counting of Exchange deployments
has emerged that could deliver the same functionality as MAPI, so it continues to be the foundation for Exchange 2007 and Outlook 2007. MAPI
remains a mystery to many, so if you’re interested in finding out more, head
over to, the Web site dedicated to Inside MAPI, the
definitive book on the API (out of print for many years).
Of course, Outlook is not the only client that you can connect to
Exchange. Ever since Microsoft realized that they had to support the Internet after the famous memo written by Bill Gates galvanized Microsoft’s
engineering groups in 1996, Exchange has been able to support other client protocols. Exchange 5.0 (released in early 1997) was the first version to
support Internet protocols. Today, Exchange 2007 supports a broad range
of Internet protocols from POP3 and IMAP4 on the client side to SMTP
as the basis for messaging connectivity and transport, to HTTP for Web
access, plus extensions that provide better security and functionality, like
The Outlook Web Access (OWA) client is a real success story for
Microsoft. Like many other projects that come out of Redmond, the initial
version (shipped with Exchange 5.0 and then improved significantly in 5.5)
was slow. This was due to some aspects of its architecture, its interaction with
the Store, and various implementation details, all of which combined to limit
its scalability to be less than the number of MAPI clients that a server could
support. The version of Outlook Web Access that shipped with Exchange
2000 marked a dramatic step forward in the UI and performance and Outlook Web Access became a client that you could actually use as a replacement
for Outlook. Microsoft made further improvements to Outlook Web Access
in Exchange 2003, not least to respond to the needs of the service providers
who wanted to deliver segmented functionality to their users, and further
improvements, not least in an upgraded and highly functional user interface,
are delivered by Exchange 2007. Some suggest that it is difficult to tell the
difference between Outlook Web Access 2007 and Outlook 2007. The test
works at a distance (of at least five feet), if not when you actually start to use
the two clients where Outlook is still the superior client. Nevertheless, the
bottom line with Outlook Web Access is that many users who work in offices
with reliable network connections find that they do not need to use Outlook
as all the functionality that they need is in Outlook Web Access.
Ever increasing mobility
We were just getting used to having cell phones in 1996 (but cell phone
bills were dramatically more expensive than today), so Exchange 4.0 and
the versions that followed really did not have to do much to accommodate
mobility. Alphanumeric pagers were the most common mobile device that
people carried if they needed to keep in touch with the office. RIM
( was founded in 1984 and developed its BlackBerry device
1.1 A decade and counting of Exchange deployments
as a solution that was initially targeted at executives. Today, BlackBerry has
become a term that people understand to mean constant connection to the
office and many of those connections are to Exchange. Of course, BlackBerry is not the only mobile device that Exchange supports. The GoodLink
server (—now owned by Motorola) connects BlackBerry
devices to Exchange along with its own devices and those running Palm OS
and Microsoft-powered PDAs. You can choose from a wide range of SmartPhones as well.
Microsoft continues to focus on mobile access to information as one of
its key development strategies for Exchange. They have poured in a huge
amount of effort to improve connectivity for mobile devices from Exchange
2003 onwards, especially for devices that run Windows Mobile 5.0 or later
releases. The good news is that the combination of new devices and
Exchange 2007 deliver even better functionality and performance, if not
quite yet to the standard that BlackBerry delivers.
In Chapter 7, we explore how Exchange 2007 delivers even more functionality for mobile users, as long as you are prepared to buy devices that run
the latest version of Windows Mobile. Exchange 2007 also includes
Microsoft’s first venture into the unified messaging market to deliver an integrated inbox that accommodates voicemail as well as email, plus the ability
for users to access their mailbox and calendar data through Outlook Voice
Access. Microsoft’s favorite demo for unified messaging is to show how you
can ring Exchange while you are en route to the office and cancel all your
appointments for the day, perhaps because you are feeling ill. Having such a
wide range of connectivity options is very convenient for users, but the sheer
number of connections that an Exchange server now supports has put a huge
load on kernel mode resources that Windows finds hard to satisfy. Increasing
the server’s ability to support client connections is one of the primary reasons
why Microsoft made the decision to make Exchange 2007 available only on
the x86-641 Windows platform. It is also fair to say that the increase in the
number of options available to users to connect to Exchange made server and
network administration more complex because of the increased number of
places where things can go wrong and disrupt the messaging flow. Security of
data, especially data carried around on mobile devices, is also a concern,
largely because of the number of mobile devices that are lost annually. It is a
personal disaster to lose your contacts because you mislaid a phone; it is a
professional and business disaster if your SmartPhone contains the company’s business plan and you have not protected the device.
Exchange 2007 runs only on the 64-bit Intel and AMD platforms. It does not run on the IA64 “Itanium” platform.
Chapter 1
1.1 A decade and counting of Exchange deployments
Third-party products and management
Exchange began with a sparse ecosystem surrounding the product. In 1996,
the threat horizon was not what it is today, so there was not the same need
for anti-virus and spam-suppression products. Management software was
limited to the basic Exchange administration program. Developers had not
even begun to think about the range of reporting and analysis software that
we enjoy today. In short, the only add-on software that was available for
Exchange was some messaging connectors and migration products to help
Microsoft migrate customers from other email systems.
The situation today is very different and we now enjoy a huge range of
add-on software that help administrators to deploy, operate, manage, report,
and debug their installations. Many software companies have come and gone
and a wave of recent mergers and acquisitions has reduced the number of
companies who create add-on products, amongst them HP (with its OpenView suite) and Quest Software (, which sells many useful
products for an Exchange environment. Microsoft has been active in this area
too and despite some false starts when it comes to APIs, Microsoft has created a lot of helpful software that it makes available to customers through
Web downloads from (see Chapter 10). Taken
with the improvements in the base software, the upshot of all the third-party
activity is that it is easier to manage an Exchange server than ever before.
Some interesting projects
Like any technology, Exchange is useless until you deploy it in projects to
solve customer business problems. The early projects were all about migration and usually contained some interesting technical problems. For
instance, one European post office wanted to replace PROFS with Exchange,
but they only had 2Kbps connections to each post office. The link was OK
for “green screen email,” but could not handle the RPCs that Exchange and
MAPI clients depend on. Later on, we faced the challenge of bringing
Exchange and Windows together as Exchange 2000 absolutely depended on
a solid implementation of Active Directory before it could function. Active
Directory posed a huge learning curve for administrators, system designers,
and consultants alike, but we have past it now and the combination of Windows 2003 and Exchange 2003 is a much more stable platform. The new
combination of 64-bit Windows and 64-bit Exchange 2007 will take some
time for administrators to become accustomed to, but it should be at least as
stable as Windows 2003/Exchange 2003 in terms of its performance in production environments.
Introducing Exchange to the world of Internet service providers (ISPs)
broke a lot of new ground around the turn of the century. Microsoft had not
1.1 A decade and counting of Exchange deployments
designed the first versions of Exchange to deal with the demands of ISPs, yet
they expected Exchange to replace MCIS, their previous email solution for
ISPs. The world of ISPs is significantly different to enterprise deployments as
the focus is all about short connections for huge numbers of POP3 and
IMAP4 clients instead of the leisurely-extended connections enjoyed by corporate users. Maybe the most interesting project was the system deployed to
provide email to political parties. Even within the same party, users did not
trust each other and the politicians were not happy to have their email stored
on the same computer as data owned by other politicians. This was not a
technology challenge, except in convincing users that Exchange and Windows could provide the necessary security to isolate everyone’s data and keep
it secure, but there were many interesting debates along the way.
The not so good points
Not everything has gone well for Exchange since 1996. Public folders are
probably the biggest piece of functionality that has underperformed and disappointed across a large number of deployments. When Microsoft was stoking the market before they shipped Exchange 4.0, they made enormous play
about the capabilities of public folders, especially when you linked them to
the power of the 16-bit Visual Basic–like Electronic Forms Designer (EFD).
With EFD, you could quickly put together a form such as a travel request or
expense claim, link it to a public folder, and allow users to create and use the
form much more efficiently than paper equivalents. With replication, you
could move that information around your organization and collate it centrally. It all looked promising, but in practice EFD was a disaster as it generated forms that performed well with a couple of users or with a small
number of items in a folder, but rapidly ran out of steam after that. EFD
sank quickly while public folders have lingered on. Microsoft has made a
couple of runs at improving public folders, most notably when they introduced multiple folder hierarchies in Exchange 2000, but no one seemed to
be interested because public folders are difficult to manage and maintain
and it did not seem like a good idea to introduce more complexity with the
extra folder hierarchies. The net result is that many companies have large
numbers of public folders, but no good way to audit, clean up, report on, or
effectively manage their contents. We will not mourn the passing of public
folders when Microsoft eventually puts a bullet through them, as long as
migration utilities exist to allow companies to move their data to a new platform. Exchange 2007 marks the start of the phase-out process for public
folders, albeit one that may take several more versions before Microsoft can
finally pull the plug on this functionality. Microsoft has promised to support public folders until at least 2016, so you can take that as an indication
of the work that Microsoft and customers have to do to transition the contents of public folders and whatever applications still depend on public foldChapter 1
1.1 A decade and counting of Exchange deployments
ers to new platforms. Other technologies, such as SharePoint Portal Server,
did not exist in 1996 and do a much better job of categorizing, searching,
and managing data, so it will be a relief to move.
Clustering is a disappointment on the hardware side. I had great hopes
for Microsoft clustering when the original “Wolfpack” release appeared
alongside Exchange 5.5 in late 1997. Part of my optimism arose from my
history at Digital where OpenVMS clustering set a bar in the mid-1980s that
Microsoft clustering has still approached today. My optimism went alongside
a realization that Exchange was vulnerable to hardware failure, especially in
the disk subsystem where the disks that were available in 1997 were not as
reliable or intelligent as they are today and few companies had started to use
Storage Area Networks (SANs) as the backbone of their Exchange deployment. Vulnerability increased as we increased the user load on servers, which
had reached a point where even the basic 32-bit Pentium II–based servers
could cheerfully accept the load of several thousand concurrent users.
Microsoft’s original implementation of clustering was expensive because you
could only run two servers in a cluster and one of those was passive, waiting
for its twin to fail. The passive server had to be licensed and have a similar
configuration to its partner, so only deployments that absolutely needed the
highest protection against failure stumped up the necessary investment to
implement clustering.
Microsoft revamped clustering in Windows 2000 and 2003 and
upgraded Exchange 2000 and 2003 to take advantage of active-active clusters. Active-active means that every node in a cluster can support work and
despite being limited to four Exchange servers in a cluster, it seemed like an
advance. However, problems with virtual memory fragmentation led to the
inability to transfer storage groups from a failed node and Microsoft revisited
its support of Exchange on clusters to impose an active-passive model where
you had to keep at least one passive server in the cluster to accept the workload should a failure occur. Obviously, this was a retrograde step because it
increased the cost of clustering again as you could not use all of the hardware
in a cluster as productively as before. Another problem was that not all thirdparty software was cluster aware, which caused problems for companies who
wanted to deploy clusters but also wanted to use a common set of software
for purposes such as monitoring, anti-virus, or backup. Finally, some of the
Exchange components did not run on clusters (like messaging gateways), so
introducing a cluster became an expensive business when you counted the
extra servers that were required to support the complete environment.
To their credit, Microsoft made a commitment to use clusters for their
internal deployment of Exchange and demonstrated that they could support
16,000 mailboxes on a seven-node cluster (four active nodes running
Exchange, one passive node, two servers performing backup and other
administrative work). Of course, the cynics pointed out that it would be easy
1.1 A decade and counting of Exchange deployments
to deploy and manage such a cluster if you had the Windows and Exchange
development groups on site all the time. The net is that clustering began with
great hopes and has receded to a point where it is useful to those who can
afford to deploy the necessary hardware and understand the somewhat special administrative environment that clusters represent. It would have been
great if clustering had become the de facto standard for Exchange deployments, but the obvious deficiencies in the implementation and the cost premium meant that this could never happen.
Exchange 2007 now offers a choice between “traditional” clusters where
the databases are located on shared storage, and cluster continuous replication (CCR), a feature that allows you to deploy a cluster built from two physical nodes and keep the database used by a virtual Exchange server up to date
on both nodes through asynchronous log shipping. CCR lays the foundation
for stretched clusters and allows for a new level of protection against physical
datacenter outages. Exchange 2007 also includes local continuous replication
(LCR) to provide an additional level of protection against a disk failure that
affects the Exchange Store to address the most obvious single point of failure
in all previous versions of Exchange. Chapter 9 covers these technologies in
some detail.
Because of the very nature of the beast, disaster recovery is always difficult, but it is the role of software to automate recovery operations to
guide administrators and assist them in getting servers back online as
quickly as possible while also avoiding mistakes like overwriting transaction logs. Until the introduction of the Recovery Storage Group in
Exchange 2003, you had to maintain extra servers to use if a disaster
occurred, and in the early days of virtualization this required physical hardware. Better hardware and fewer software bugs steadily reduced the number
and impact of database corruptions, but it is surprising that we have had to
wait until Exchange 2007 for features such as log shipping (as employed in
both CCR and LCR) and the database troubleshooting assistant. Even
though we’ve had to wait, now that we have the ability to deploy CCR and
LCR, it will be interesting to see how these technologies are used to build a
new level of resistance to database outages.
APIs are the other disaster area for Exchange. Microsoft needed
Exchange to have great programming capabilities to help wean companies off
Lotus Notes. Notes is not a great messaging engine, but it has extremely
strong collaborative and programming capabilities that companies exploit to
put together mail-enabled applications that take advantage of the Notes replication engine (also better than the replication implemented in Exchange
public folders). We have endured multiple attempts by Microsoft to deliver
an equivalent development platform for Exchange. To give Microsoft credit,
they are persistent, and they have been very persistent, but also have a very
awful track record with the APIs that have shipped with Exchange. We have
Chapter 1
1.1 A decade and counting of Exchange deployments
seen CDO2, CDOEXM, Exchange Routing Objects, the infamous EFD,
WMI, client-side MAPI and server-side MAPI, WebDAV, and so on. The
highest figure I ever heard was that there have been 32 different ways a programmer can write code to access Exchange data over the years. I cannot verify the count, but it does not surprise me.
Microsoft knows that they have a mess on their hands and the advent
of PowerShell (see Chapter 10) support in Exchange 2007 means that we
have a solid and robust interface that we can use to build a new set of management scripts and other tools. Exchange 2007 also delivers a new set of
Web services that may mark the start of the process of breaking Exchange
up into a series of Web services that other applications can consume.
Decomposing a mammoth application will take time, but Microsoft has
made a good start in Exchange 2007. Outside of the limited set of Web services that can only access a tiny portion of the overall functionality of
Exchange, there is still no good way to develop mission-critical client side
applications that exploit the storage and messaging power of Exchange and
we await developments in this area.
Exchange’s connection with the Active Directory
When Microsoft moved Exchange away from its own directory store to support the Active Directory in Exchange 2000, some predicted that the transition would make Exchange harder to manage and deploy. To some extent,
this assertion is true as the need to deploy Active Directory first slowed down
the migration from Exchange 5.5 to Exchange 2000. Indeed, some companies have not yet migrated away from Exchange 5.5!
Over time, I think that Active Directory has been good for Exchange.
After the initial set of hiccups that slowed adoption, the body of knowledge
around the Active Directory grew and some solid deployments ensued.
This is not to say that the deployments were perfect and certainly some
have corroded over time in terms of their effectiveness. The transition to
Exchange 2007 is a perfect opportunity to revisit Active Directory deployments to ask the question whether they are as effective as they could be if
they reflect current best practice, and to consider whether you can consolidate sites, domains, and servers to reduce complexity and cost from the
infrastructure. You realize the worth of Active Directory to Exchange 2007
in the new dependency that exists on the site topology and site links as the
basis for message routing, replacing the routing group structure used by
Exchange 2000/2003.
The only big problem that I now have with the Active Directory is the
inflexible way that Exchange uses it. Exchange uses a container in the Active
CDO 1.2.1 remains supported for use with Exchange 2007, but only on 32-bit platforms.
1.1 A decade and counting of Exchange deployments
Directory configuration naming context to store its configuration data.
Exchange gains great value from this implementation, not least because the
Active Directory replicates all of Exchange’s configuration data automatically
to domain controllers around the forest. However, no one has ever been able
to explain to me why the Active Directory can host only a single Exchange
organization, as it does not seem to be complex to store several containers,
one for each organization, in the directory. It would be nice to see this restriction lifted in the future, if only to remove the requirement to deploy multiple
forests (and the hardware to support multiple forests) if you need to support
multiple Exchange organizations. Sometimes it is good to have the separation
between organizations that different Active Directory forests afford, but it
would be better to have the option to store everything in one place.
Reviewing predictions made in 1996
Scary as it seems, I have been writing about Exchange since 1996. The vast
bulk of my scribbling has appeared in Windows IT Pro magazine (and its
predecessors—see and the Exchange Administrator
newsletter, and I hope that the articles have helped you understand and
exploit Exchange to the maximum over the years. However, my first article
appeared in a publication called Exchange Manager that did not last very
long. I wrote an article called “Scaling Exchange” where I looked at the
practical issues involved in scaling Exchange 4.0 to deal with hundreds of
users (my advice was not to support more than 300 users on a server). I
wrote: “Lots of people get hung up about the 16GB limit for the Information Store…. I don’t, because it’s a limit that most of us will never encounter.” I was right in one respect because Microsoft only upped the 16GB
limit to 75GB for the standard edition of Exchange in Exchange 2003 SP2
(and then removed the limit completely in Exchange 2007), but the sheer
number of messages that we send and the average size of those messages has
exploded since 1996. Then, most messages in corporate email systems were
between 5KB and 10KB. Now, they are bloated through a mixture of user
indiscipline (horrible autosignature files, too many replies to replies, etc.)
and huge attachments.
I went on to ask: “What was the last time you saw a Windows NT system
that had more than 100GB of disk attached? Or more than 4 CPUs? Or even
more than 256MB of memory or 512MB on a RISC system?” How times
have changed. In my defense, our first Exchange servers boasted 66MHz 486
CPUs, had 64MB of memory, and 4GB of disk. Today, the best advice is to
buy powerful 64-bit servers to run Exchange 2007, preferably with multicore processors, gigabytes of memory, and lots of disk. In their justification
(see the commentary in
12/29/416613.aspx) for the move to an exclusive 64-bit platform for
Exchange 2007, Microsoft cites the fact that they believe that 500GB disks
Chapter 1
1.2 Microsoft’s themes for Exchange 2007
will be standard when Exchange 2007 ships and that 1TB disks will be
available. Note that 64-bit means the x64 platform as Exchange 2007 does
not support the Itanium (IA64) platform. Support for a version of
Exchange running on IA64 may come in the future as SQL has already
demonstrated the huge scalability potential of the IA64 platform.
I looked into the future by predicting that: “In the long term, the evolution of Windows NT to support 64-bit computing will raise the performance
bar even further and allow people to consider even larger systems … systems
that can support thousands of users on a daily basis.” I went on to ask
Microsoft to consider raising the limit on the Information Store from 16GB
to 16TB (Microsoft did this for the Enterprise version in Exchange 2000);
support clustering (Microsoft shipped Wolfpack clustering in 1997 with
Exchange 5.5, but the Exchange clustering story has been an uneven success
since); support a single mailbox restore (possible with third-party products),
since 1997; PSS generated the ExMerge utility in 1998 (a utility that
Microsoft no longer supports in Exchange 2007); the Mailbox Recovery
Center arrived in Exchange 2003; provide better support for multiple processors (done in Exchange 5.5 and much improved since); and optimize the
code for non-Intel processors. Alas, the multi-platform play for NT and
Exchange terminated after Windows NT 4.0/Exchange 5.5 when Microsoft
halted their support for Windows NT on the Alpha CPU, but the 64-bit
AMD and Intel platforms are now a great success for Exchange 2007. Looking back, it was not a bad list to ask for.
Microsoft’s themes for Exchange 2007
Over the last ten years, the environment surrounding Exchange has evolved
in many dimensions. Here are just a few of the most important technology
influences that have affected the evolution of Exchange.
The base operating system has moved from Windows NT 3.51 on a
32-bit platform to Windows 2003 R2 on a 64-bit platform.
Storage has moved from small and expensive direct attached storage
to a huge range of solutions spanning anything from JBOD (just a
bunch of inexpensive disks) to very large SANs.
Systems that might have been lucky to operate with 128MB of memory have moved to a point where many email servers are equipped
with 8GB or more, and Microsoft’s recommendations for memory
for some Exchange 2007 servers will make 32GB or 64GB a common configuration in the near future.
1.2 Microsoft’s themes for Exchange 2007
Microsoft has poured enormous engineering effort to bring Exchange
through three distinct generations of software from the original focus
on PC LAN-centric deployments and the need to migrate from competing products such as Lotus cc:Mail to the ability to deal with massive corporate deployments.
Exchange’s administrative model has moved from a purely graphical
interface that dealt well with the needs of a few hundred users but struggled with large organizations to a point where Exchange 2007 complements an administrative GUI with a sophisticated shell that
administrators can program to automate many management operations.
The protocols that are important to Exchange have moved from a
mishmash of messaging and directory protocols, both proprietary
and international, to a point where we deal with a consistent set of
Internet protocols such as SMTP, LDAP, HTTP, and IMAP.
The computing model for Windows has evolved from an approach
that usually focused on a one-server, one-application model to something that more closely resembles the kind of deployments seen on
other corporate computing platforms.
The range of clients that Exchange supports has expanded dramatically from a single Microsoft client that could only run on Windows
to a variety of clients that accommodate the spectrum of user needs
from traditional PC workstations to many variations of handheld
When Microsoft began to design the third generation of Exchange, they
decided to use three broad themes as the general thrust for the development
effort. These are:
Built-in Protection: While Exchange has long had good protection against
viruses and spam through third-party products and Exchange 2003 offers
good filtering capabilities to block incoming messages from undesirable parties, Microsoft knew that they had to offer out-of-the-box protection for
Exchange to remain competitive and to protect customers. Some of this work
started with the release of the Internet Message Filter (IMF) for Exchange
2003 and the SmartScreen junk mail filtering technology that Outlook 2003
and 2007 incorporate into their code.
After they had released Exchange 2003, Microsoft’s original plan was to
deliver a server designed for deployment within the DMZ (the original Edge
project). Microsoft based the original Edge project on Exchange 2003 technology, but they cancelled the project in late 2005. Microsoft bought market-leading anti-virus and anti-spam technology in-house through the
acquisition of Sybari Software Inc. in 2006. They have since refined its capaChapter 1
1.2 Microsoft’s themes for Exchange 2007
bilities since to produce the ForeFront Security for Exchange product, which
is bundled with the Enterprise edition of Exchange 2007. In addition,
Microsoft has dedicated considerable engineering effort to research how best
to defend against email threats and contributed to projects such as the Sender
ID initiative. While the result of this work is spread throughout Exchange
2007, much of it is focused in the Edge and hub transport server roles. The
Edge server essentially completes the project that Microsoft started out to
build some years ago with the added advantage that it is built on the
Exchange 2007 code base. The bottom line is that Microsoft intends
Exchange 2007 to be the most secure email server available anywhere. This is
a laudable goal, but given the evolving nature of network threat, we will not
really know whether Microsoft has succeeded until companies have moved to
Exchange 2007 and moved away from the previous generation of servers.
Anywhere Access: This theme reflects the mobile nature of the world that
we live in today rather than the tethered nature of traditional email access.
Exchange has offered Web-based access since 1996 and support for handheld
devices since 1999, first with RIM BlackBerry devices and then later Windows Mobile handhelds, but only through add-on products. Exchange 2003
introduced server-based ActiveSync and Outlook Mobile Access (support for
WAP browsers). Neither offering was on par with the leaders in the market.
Outlook Web Access was a reasonable Web-based interface, but it needed to
move away from protocols such as WebDAV that had seemed the way forward when Microsoft introduced WebDAV support in Exchange 2000 but
had now been bypassed, and create a new user interface based on the latest
Web technologies such as ASP.NET. However, while increasing the functionality and power of Outlook Web Access and ActiveSync reflected some
immediate technical imperatives of this theme, the more interesting aspect
was the desire to incorporate voice technology into the Exchange platform
for the first time. This was not a new technical challenge because third parties
such as Nortel and Avaya had created good voicemail integrations with
Exchange and Outlook as far back as 1998. The interesting challenge was to
create a different type of integration than merely playing back received voicemail through Outlook messages and PC speakers. As they set out to create
the Exchange 2007 Unified Messaging server, Microsoft wanted to deliver an
integrated server that used the Exchange Store as the repository for voice and
data and the Exchange messaging infrastructure to move voice messages
around. On the client side, Outlook Voice Access lets users access their messaging, calendar, and directory data through a range of telephones from the
latest SmartPhone to a plain old touch-tone phone.
Operational Efficiency: Even its best friends would not hold Exchange
2003 up as the best example of an operationally efficient messaging system.
Exchange 2003 offers great functionality to end users at the expense of a lot
of hard work by administrators at the back end. Issues included:
1.2 Microsoft’s themes for Exchange 2007
Common administrative tasks such as moving mailboxes were hard to
The user interface presented by the administrative tools were sometimes confusing; there were too many black boxes in Exchange (think
of how the Recipient Update Service stamps email addresses on newly
created mailboxes).
There are far too many places where Exchange suddenly enabled features if an administrator would only update the system registry with a
magic key.
Performance of some of the tools was acceptable for small to medium
businesses but not for large organizations.
Sometimes you felt that the developers had simply lost interest when the
time came to write the system management components of Exchange
because they all wanted to create cool new features that were appreciated by
users. This is quite a litany of complaints. I have had practice in composing
this list because I have been quite vocal on the point both when speaking at
conferences and when talking with the Exchange developers. Exchange 2003
is not bad software to have to manage, as long as you know its quirks and
took the time to learn all about how it works. The trouble was that many
people did not make the necessary effort to learn Exchange and the inevitable
result was dissatisfaction, product issues, and system outages.
Perhaps the biggest change in attitude and focus that Microsoft has
made since the release of Exchange 2003 is the effort that the development
group has poured into making Exchange more automated, more manageable,
and easier to deploy. Microsoft has pumped out a huge increase in wizards,
automated management and analysis tools, and documentation since 2005.
A new attitude seems to have infused the development group with the desire
to improve the administrative characteristics of Exchange and you can see the
result in Exchange 2007. While the most obvious change is in the Exchange
Management Console, the real magic is in the introduction of the Exchange
Management Shell because this is the basis for a new era of automation for
common and not so common administrative tasks.
Figure 1.1 illustrates an example of how Microsoft has made Exchange
2007 more manageable than any previous version. Users receive nondelivery
messages all the time, but the content of messages generated by older versions
is often not too helpful to the average user. Exchange 2007 generates messages that are easy for users to understand what problem has occurred and
include some additional information for administrators to figure out why the
problem has occurred. For example, the bottom portion of the message
shown in the left-hand screen of Figure 1.1 includes the error text, and then a
trace of all of the servers (not illustrated) that the message passed through
Chapter 1
1.2 Microsoft’s themes for Exchange 2007
Figure 1.1
Some improved
system messages
Exchange 2007
before Exchange detected the error. You see the same attention to detail in
other places too, like the diagnostics information available from Outlook
Web Access (Figure 1.2) to help administrators figure out why things may
not be working as normal, the mailbox quota exceeded messages sent to users
(the right-hand screen in Figure 1.1), and so on.
Figure 1.2
Outlook Web Access
reveals all to help
fix problems
Microsoft has removed two major distinguishing features of the
Exchange 2000/2003 architecture in Exchange 2007. When Microsoft introduced Exchange 2000, they had to provide backwards compatibility with
1.2 Microsoft’s themes for Exchange 2007
Exchange 5.5 in terms of management and routing, so they introduced the
concept of administrative groups (comparable to Exchange 5.5 sites) and
routing groups (comparable to Exchange 5.5 sites in terms of the routing
topology). Unfortunately, the original plans to make administrative groups
more flexible in terms of server management never quite worked out. As
implemented in Exchange 2000 and 2003, administrative groups are an
unintelligent container for servers and not much else. Experience with
administrative groups has demonstrated that they were far too rigid in operation (you could not move a server between administrative groups) and not
granular enough when it came to server management (delegation applied to
all of the servers in the group rather than down to the individual server). The
result is that many Exchange administrators ignored administrative groups
and kept all the servers in the default administrative group.
According to surveys conducted by Microsoft at events like TechEd, the
only administrators who paid much attention to administrative groups
worked in large enterprises where the sheer number of servers and their distribution across multiple physical sites meant that administrative groups
offered some benefits. Administrators of small to medium Exchange organizations often did not know about administrative groups because they never
needed to do anything else but install all their servers into the default administrative group. The experience with routing groups was more positive
because they offered flexibility and a way to control the routing topology but
they required quite a lot of manual intervention to set up and manage. In
addition, the move to a new SMTP routing engine that leveraged the Active
Directory and used point-to-point TCP connections to create a full-mesh
routing network based on deterministic routing meant that routing groups
were no longer required. As we will see later on, a default administrative and
a default routing group still linger on within Exchange 2007, but only as a
method to allow management and routing to continue seamlessly in a mixedmode organization.
Better security was another important focus for Exchange 2007.
Microsoft has steadily increased the default level of security in its products
to cope with the increased level of threat that exists within the network
today. You will find that Exchange 2007 components are secure by default.
For example, Outlook Web Access connects via HTTPS instead of HTTP
as previously used; POP3 and IMAP4 clients connect over secure ports
rather than the insecure default ports for these protocols; Exchange servers
authenticate and connect securely before they transfer messages; and connectors are secure out of the box. Overall, Exchange 2007 is a much more
secure product than its predecessors are. This is not to say that administrators are unable to punch holes in Exchange’s security by removing restrictions, but even if you leave the default security settings, you will find that
your installation is more secure than before. Tools provided by Microsoft to
review and tighten security (such as the Windows 2003 Security ConfiguraChapter 1
1.2 Microsoft’s themes for Exchange 2007
tion Wizard) are helpful to ensure that you do not leave holes open on
Exchange 2007 servers.
It is important to point out that Microsoft management gave these three
major focus areas to Microsoft engineers to create cohesion in very large software engineering projects. However, you should not attribute the same
degree of importance to these areas because they may not be critical to your
organization. In fact, there may be specific areas inside Exchange 2007 that
Microsoft thinks are hypercritical and you think are unimportant—such is
the nature of very large and complex software products. You can often measure the real success of software in its longevity. Microsoft shipped Exchange
5.5 in 1997 and perhaps Microsoft’s relative lack of success in persuading
companies to move off Exchange 5.5 is due to its usability and relative ease of
management. It will be interesting to see how Exchange 2007 has fared ten
years after customers first deploy it and compare whether administrators feel
the same positive way about it as many clearly do about Exchange 5.5.
The happy prospect of a migration
I doubt that there is a single administrator of a messaging system in the
world who can honestly say that they enjoy the process of moving an email
system from one software version to another. Over the three generations of
Exchange, we have experienced relatively easy migrations, such as upgrading
servers from Exchange 5.0 to 5.5 or from Exchange 2000 to 2003. These
migrations required careful planning to ensure no disruption for users, but
the actual mechanics of the upgrade were simple and easy to perform because
there was no change to the fundamentals of Exchange—the operating system, hardware, or internal architecture. On the other hand, some migrations
have caused massive upheaval because of the degree of change in those same
fundamentals. Upgrading an Exchange organization from 5.5 to 2000 was
not easy because of the requirement to install the Active Directory, which
required a change in the base platform from Windows NT to Windows
2000. Administrators had enough of a steep learning curve to understand
how Active Directory worked, especially around security boundaries and replication, and when you heaped the massive change in the Exchange architecture that Microsoft introduced in Exchange 2000, you had a recipe for many
long hours of work to plan, deploy, and support the migration. From
Microsoft’s perspective, their decision to base Exchange 2000 around Active
Directory was a great move because it forced many companies to introduce a
corporate directory service long before they might have wanted to do so. Any
company that wanted to use Exchange was forced to deploy the Active Directory first. The long-term effect is that the Active Directory is an embedded
part of the core IT for any company who uses Exchange.
In the most part, we have mastered Active Directory now. While
Exchange 2007 does not require disruption of the kind seen when you intro-
1.2 Microsoft’s themes for Exchange 2007
duce a new corporate directory that simply has to work before applications
can function, it does include two huge changes. First, Exchange 2007 only
runs on 64-bit Windows on x64 servers. Second, Microsoft has torn up the
administrative model used in Exchange 2000 and 2003 in favor of a simplified GUI for the management console and a whole new focus on a highly
programmable scripting language implemented through a UNIX-like command shell. Administrators therefore face the need to refresh their server
hardware inventory completely for both Exchange and Windows while at the
same time they need to learn a whole bag of new tricks to work with and
manage Exchange 2007.
The server refresh is not straightforward because you should not simply
replace 32-bit servers for 64-bit models on a one-for-one basis. Such an
approach could be an incredible waste of money and negate many of the
advantages of the new platform. Exchange 2007 is an opportunity to completely revamp the original Active Directory and other base Windows infrastructure components laid down to support Exchange 2000 and 2003 with
the intention of simplifying the infrastructure through consolidation of servers into a much smaller set of domain controllers, global catalog servers,
DHCP servers, and the like. Consolidation is possible because a 64-bit Windows server is able to handle much more work than its 32-bit counterparts
are and additional cheap network capacity usually makes it possible to bring
the work to a smaller set of servers than to distribute it around the network.
For the same reason, the same kind of consolidation may be possible with
Exchange 2007. In one way, the introduction of server roles prompts you to
think about the number and function of servers to deploy for Exchange
2007. How many mailbox servers will you deploy? How many servers of the
other roles will you need and should you run multi-role servers or single-role
servers? For example, if you want to deploy CCR on MNS clusters, these
servers will be single-role mailbox servers whereas you can use LCR on multirole servers. How will you take advantage of the larger memory model to
support more mailboxes? How can you exploit SANs better to support the
data requirements for consolidated servers in a datacenter? Exchange 2007
requires more memory than ever before to compensate for a large reduction
in I/O operations. This leads on to a dramatically different I/O profile in
terms of the demands that Exchange 2007 makes on storage subsystems.
Large SANs will be able to support more servers and you can build servers
around newer storage technologies that offer better performance and lower
cost. There is quite a bit of work to do to figure out the most appropriate
configuration for Exchange 2007 servers in any organization.
With so much upheaval caused by the transition to 64-bit servers, you
should take the opportunity to have a long hard look at the server and storage infrastructure you use for Exchange 2003 and think about how you can
consolidate to remove cost, complexity, and administrative overhead. You
cannot upgrade an Exchange 2003 server to Exchange 2007 server, so you
Chapter 1
1.3 Preparing for Exchange 2007
have to use the move mailbox function to get user data across to the new
servers. With this in mind, there is an obvious opportunity to plan for a
smaller number of mailbox servers that support larger user communities.
Another point that you should consider is whether the time is right for
you to virtualize some of your Windows infrastructure, including domain
controllers and Global Catalog servers. There is no doubt that virtual servers
will gradually replace physical servers over time in order to maximize the use
that we can get from increasingly powerful hardware. The capabilities of the
virtual server software that is available today are incredible, even if it is not all
from Microsoft. Virtual servers should definitely be on your agenda to support test servers at the very least, but it is now time to take a hard look at
your Windows servers to figure out what you can deploy on virtual servers
and what has to stay on physical servers.
Of course, server consolidation is not easy and you need to do a lot of
planning to make sure that you deploy and populate new servers with data
without causing disruption to users. The end goal is worthwhile because
fewer servers are easier and cheaper to manage, so the lower operational costs
may pay for some or all of the overall migration effort.
The impact of the change in the administrative model is harder to predict. Some administrators will take to the Exchange Management Shell with
gusto and will become very comfortable with the 360+ new commands that
Microsoft has made available to manage Exchange 2007 through PowerShell.
Other administrators will be like lost lambs in a storm, wondering where
their familiar graphical management console has gone and struggling with
the syntax of the shell. To a large degree, the change is in terms of culture and
philosophy as well as technology, and it will take people time to adjust.
To return to the original point—migrations are not easy and we face
some big changes in Exchange 2007 that make the migration to this version
more difficult to plan for. However, the good thing is that careful planning,
attention to detail, and dedication to mastering the new technology will
bring success, albeit at the expense of a lot of time and effort.
Preparing for Exchange 2007
Because of its dependence on the Active Directory, just like Exchange 2000
and 2003, you have to prepare the forest and every domain that will host
mail-enabled objects before you can install Exchange 2007. Consult the
Exchange 2007 documentation and the release notes to get the most up-todate information on what you need to do to install Exchange 2007. In general, the major considerations that you need to incorporate into a deployment plan are:
1.3 Preparing for Exchange 2007
Ensure that the schema master, domain controllers, and Global Catalogs in any domain that will support an Exchange 2007 server runs
Windows 2003 SP1, R2, or later.
Ensure that the Active Directory functional level for the forest is
Windows 2003. This is not a strict prerequisite. However, I recommend that you move the forest to Windows 2003 functional level
before you deploy Exchange 2007.
Ensure that any legacy Exchange servers are running Exchange 2003
SP2 or Exchange 2000 SP3 with the post-SP3 update rollup (available from Microsoft’s Web site).3 No Exchange 5.5 servers can be in
the forest. Unless Microsoft supports you through a beta program,
you will have to remove and reinstall any servers that run beta versions of Exchange 2007.
Ensure that you have a Global Catalog server in every Active Directory site that you intend deploying an Exchange 2007 server. Note
that Exchange 2007 setup will fail if the Active Directory site contains a Windows 2000 domain controller. You should upgrade these
servers to Windows 2003 SP1 before you deploy Exchange 2007.
Run Setup /PrepareAD in the forest root to prepare the forest creating forest-wide objects used by Exchange 2007.
Run Setup /PrepareSchema to extend the Active Directory schema to
support Exchange 2007. You need to take this step even if you have
extended the schema to support Exchange 2000 or 2003.
Run Setup /PrepareDomain for every domain in the forest that will
host mail-enabled objects.
If you are installing Exchange 2007 into an existing Exchange
2000/2003 organization, you have to run Setup with the /PrepareLegacyExchangePermissions switch.
Install the first Exchange 2007 server into an Active Directory site
that is close (in network terms) to the site that supports the schema
Exchange 2007 has its own SMTP stack, so you should remove the
standard Windows SMTP server or the SMTP server installed by IIS
before you install Exchange hub transport or Edge servers.
You can run ExBPA, the Exchange Best Practice Analyzer tool, to verify
that your organization is ready to deploy Exchange 2007. ExBPA performs
It is important that legacy Exchange servers run the latest service packs because this software includes the updates to
allow the Exchange System Manager console to deal with Exchange 2007 objects correctly.
Chapter 1
1.4 Installing Exchange 2007
checks against the Active Directory, validates that you have the correct version of software deployed on legacy servers, and generally checks that you
will not meet any surprises when you come to install your first Exchange
2007 server. See Chapter 10 for more information on ExBPA.
It is important that servers run at least Windows 2003 SP1 because
Exchange 2007 depends on its notification mechanism to advise components
that an update has occurred in the Active Directory. In addition, Outlook
Web Access 2007 depends on improvements made in Windows 2003 SP1 to
exploit a feature called VLV (Virtual List View) to perform more efficient
lookups against the GAL.
While you must install Exchange 2007 on new 64-bit servers, you do
not necessarily have to deploy 64-bit domain controllers and Global Catalog
servers, as Exchange is happy to use the Active Directory on 32-bit servers.
There is no doubt that 64-bit Windows servers are much more scalable than
their 32-bit equivalent, largely because they can cache even very large Active
Directory databases completely in memory and so improve performance for
applications that depend on the directory, like Exchange. It is a good idea to
plan for the introduction of 64-bit Windows servers to provide the essential
infrastructure to support Exchange 2007 and to deploy this infrastructure
well before you plan to install Exchange 2007, just to make sure that the supporting infrastructure is fully bedded down.
Installing Exchange 2007
In some respects, the task of the engineers who created the Exchange 2007
setup program was simple. After all, because it is not possible to upgrade an
old 32-bit Exchange 2000 or 2003 server to Exchange 2007, they only had
to build a program to install Exchange on brand-new 64-bit servers, and conceptually, there is nothing easier than taking software off an installation kit
and arranging it across the available space on a disk. Of course, it is easy
enough to say, but difficult to do, especially when you want to create an
installation procedure that is intelligent, robust, and difficult for administrators to make mistakes when they install Exchange. The latter point is important for Microsoft because a good installation program reduces the number
of support calls that they receive, especially after the launch of a new version
of a product. Support costs are very expensive because they often require
human intervention to solve the reported problem.
The Exchange developers have done a good job with the Exchange 2007
installation program. A lot of intelligence is incorporated into the program so
that even before you begin to get near to installing Exchange, the program
does a comprehensive job of scanning the target server to ensure that you
have already installed all the necessary pre-requisite software (such as PowerShell version 1.0, version 2.0 of the .NET framework or later, and MMC
1.4 Installing Exchange 2007
3.0). The program also checks the Active Directory to ensure that the schema
is at the necessary version. Even better, unlike other installations that tell you
that something is wrong and then promptly exit to leave the poor administrator to figure out how to fix the problem, a strong point of Exchange
2007’s installation program is its integration with Microsoft’s support network. When it detects a problem, the installation program points you to the
right Web page to download the components that are necessary to fix the
problem. Of course, this will not do you much good if you have to install
Exchange 2007 in a secure location that’s isolated from the Internet, but it’s
great for the vast majority of installations.
Figure 1.3
Exchange 2007
The user interface that allows you to select the options for the installation are well thought out and easy to follow. Figure 1.3 shows how you can
select a typical installation (install the hub transport, client access, and mailbox roles, and the Exchange Management Console, Help files, and the
Exchange Management Shell extensions for PowerShell) or a customized
installation. If you select a customized installation, you can decide what roles
to install on a server and, if you want to create a mailbox server, whether you
want to use a cluster configuration. Custom installation is also the place to go
when you want to deploy an Edge server or Unified Messaging server.
Before beginning the actual installation, the program checks to see
whether any new best practice advice is available from Microsoft and downloads the advice if Microsoft has updated it since they built the installation
kit. This is a good example of how Microsoft is leveraging the investment
that they have made since 2003 in building intelligent analysis and reporting
tools for Exchange because the installation program can access the same
knowledge base as the Exchange Best Practice Analyzer to find out about
nonsoftware items that have to be resolved before a successful installation can
proceed. For example, Figure 1.4 shows that an installation of an Edge server
detected two problems before proceeding. In both cases, the installation program can connect you to a Web page that contains the steps necessary to fix
Chapter 1
1.4 Installing Exchange 2007
the problem. In this case, because Microsoft designed the Edge server to
operate within the DMZ and because Exchange 2007 uses its own SMTP
stack, it is necessary to turn off the standard SMTP service that is part of IIS.
If you install a server that supports multiple roles, the installation program
will determine the order that it installs the roles. However, if you install a single-role server and then decide to install other roles on that server, the suggested order to do this is Client Access Server, Hub Transport, and Mailbox
followed by Unified Messaging if required.
Figure 1.4
An error
detected by the
Exchange 2007
setup program
Note that you must have at least one hub transport server in an Active
Directory site if the site supports a mailbox server. A mailbox server cannot
communicate on its own and needs the hub transport server to act as its
point of contact with the rest of the organization. A hub transport server is
also required if you deploy a Unified Messaging server. Exchange 2007 uses
server-side RPCs to connect servers together within the same site. Microsoft
has never optimized RPCs to work over extended WAN connections that
you might expect between Active Directory sites. For this reason, if you have
an Active Directory site that supports a small number of users, you will
deploy either a single multi-role server (mailbox, hub, and client access) or
route client connections to servers in another site.
Exchange 2007 supports command line installations if you do not like
the GUI. Command line installations are very useful if you want to use unattended installations. You have to log onto the target server and then execute
Setup, passing the options that you want to use for the installation. For
example: /Mode:install /Roles:C, H, M /On:XYZ / /EnableLegacyOutlook /
LegacyRoutingServer:Exch2003BHS /DisableErrorReporting
1.4 Installing Exchange 2007
This command line tells the installation procedure to:
Install the Client Access, Hub Transport, and Mailbox roles in the
XYZ organization.
Use as the domain controller.
Enable support for legacy Outlook clients (Outlook 2003 and previous). Essentially, this means that Exchange 2007 will be installed
with public folders as legacy clients are heavily dependent on public
Create a routing group connector to link back to legacy Exchange
servers; the Exch2003BHS server is the bridgehead on the legacy side
of the connector.
Disable the option to report errors back to Microsoft for inclusion in
their quality improvement program (some companies have internal
rules that stop them from reporting error data in this way).
Exchange 2007 also has the ability for an administrator to assign the
responsibility for completing the installation of a server to another user with
the /NewProvisionedServer command switch. In this scenario, an enterprise
administrator executes the heavily permissioned parts of the installation procedure to create the server object and add the delegated account to the local
administrators group on the target server. The delegated user account is also
added to the Exchange View Only administrators group. Effectively, the
enterprise administrator delegates enough authority to allow the designated
user to run Setup to complete the actual creation of the physical server (such
as moving files to the right location on the disk, etc.). The other user only has
the necessary permission to be able to complete the installation of the designated server and can take no other action. See the section on “Changes in the
Exchange Delegation Model” in Chapter 3 for more information.
Behind the scenes, the Exchange 2007 installation program writes a vast
amount of information into a set of log files that it creates in the \ExchangeSetupLogs directory on the drive that you use to install Exchange. Amongst the
files that you’ll find there are some PowerShell scripts (the exact number
depends on the kind of installation that you perform on the server) that
Exchange generates to install itself. Figure 1.5 shows an extract of a script generated during the installation of the mailbox role on a server. You can see commands to stop services that must not be running when Exchange is being
installed followed by commands to precompile the software components used
by Exchange.
Chapter 1
1.4 Installing Exchange 2007
Overall, Microsoft has done an excellent job to create a nice tool for
administrators in the Exchange 2007 installation program. Running the
installation program should not cause many concerns for your deployment
and may be the easiest part of the process. Following the installation, you
normally have some work to do that is more complicated. For example, if
you do not want to configure a routing group connector to link your first
Exchange 2007 server with a legacy organization during the installation, you
could do it after you have checked out the newly installed server and made
sure that everything is working as expected. Another task to support legacy
clients is to configure the Offline Address Book for legacy Outlook clients.
Other tasks include making sure that the Availability service is running
smoothly, installing some anti-virus software, setting up an Edge server, or
even plunging into the new world of Unified Messaging to allow Exchange
users to access their mailbox through the phone. We will get to all of these
tasks later on in the book.
Figure 1.5
Extract of a
PowerShell script
used to install
Exchange 2007
After you complete a server installation, you should consider running
the Microsoft Security Configuration Wizard (SCW) to harden the server.
Exchange 2007 includes support for SCW, so you can follow the directions
in the installation guide to run SCW and configure security for the different
server roles. Before you begin using SCW, you need to register the SCW
extensions for the server roles. You can do this with these commands. The
first command registers the extensions for any role except the Edge server;
the second registers the extension for the Edge server.
1.4 Installing Exchange 2007
Scwcmd register /kbname:"Ex2007KB" /kbfile:"%programfiles%\
Microsoft\Exchange Server\scripts\Exchange2007.xml"
Scwcmd register /kbname:"Ex2007EdgeKB" /
kbfile:"%programfiles%\Microsoft\Exchange Server\scripts\
Modifying and removing servers
To modify or remove an existing Exchange 2007 server, you can use either
the Windows Control Panel (to add/remove software) or the ExSetup program, which you’ll find in the \Program Files\Microsoft\Exchange Server\Bin
directory. This program has several modes:
/Install: Add a new role to an existing Exchange 2007 server.
/Uninstall: Remove a role or a complete Exchange 2007 server. For
example, to remove the client access role from a server, use ExSetup /
Mode:Uninstall /Roles:CA.
/Upgrade: Upgrade to a new release.
/Recover Server: Recover a server using the information about it
stored in the Active Directory.
/Clustered: Designate an active role in a cluster.
Validating the installation
After the installation process checks, you may want to validate the installation by testing that the installation procedure has generated a functioning
Exchange server. The installation procedure logs all of the operations that it
performs in the Exchange setup log file at C:\ExchangeSetupLogs\ExchangeSetup.log. It is certainly possible to review the setup log, but it is verbose in
the level of detail that it captures. You can filter the application event log for
events written by MSExchangeSetup and check for events 1000 (starting to
install a server role) and 1001 (successfully installed a server role). Alternately, you can use a PowerShell script called Get-SetupLog.ps14 to parse the
contents of the setup log and report the entries found there. Figure 1.6 shows
what you can expect to see from both the Get-SetupLog script and the application event log.
Of course, another way of looking at things is to say that you have not
installed a server until it works and that the only good test validates that
Located in \Program Files\Microsoft\Exchange Server\Scripts.You can find details of how to use the script in TechNet or in
the header of the script.
Chapter 1
1.5 Server roles
Figure 1.6
Checking the
installation log
the server works in all its various roles. You can use some simple tests for
this purpose. First, use the Get-ExchangeServer PowerShell command to
see whether the output of the command includes the new server in the list
of Exchange servers in the organization. Second, log onto the server and
use the special test commands supported by the Exchange Management
Shell work as expected. You can find details of how to use these commands
in Chapter 4.
Third-party software
Migrations can only proceed at the pace of the slowest component and often
that slowest component is some essential piece of third-party software that’s
used with Exchange in your production environment. Table 1.1 lists a representative sample of third parties that engineer software that supports
Exchange 2007 in some way. This list is likely to grow over time, so you
should consult the online list that Microsoft maintains at http:// for the latest information. In addition, you need to check with the vendor to ensure that they do
not have any special prerequisites for deployment alongside Exchange 2007,
such as an inability to support clusters, a requirement to deploy on a single
role server, or anything else.
Server roles
The notion of giving a server a particular role to fulfil within a messaging
infrastructure is not new and messaging architects have assigned roles such as
1.5 Server roles
Table 1.1
Representative sample of third-party product support for Exchange 2007
Software Category
Microsoft ForeFront Security for Exchange,
TrendMicro, McAfee, Symantec, GFI, Kasperski,
Sophos, F-Secure
Archiving (compliance)
HP, Commvault, Symantec, Hummingbird,
Mobius, Overtone, Atempo, AXS-One, Zantaz
Content Filtering
IXOS, Meridio
FAX Gateway
Fenestrae, GFI, Captaris, BNS Group
IP/PBX Gateway
Audiocodes, Dialogics
Line of Business Applications
Newsgator (RSS), nCipher, K2 (workflow)
Quest Software, NetIQ, Computer Associates,
HP, Zenprise
Migration from Legacy Exchange
to Exchange 2007
Quest Software
Mobile Clients and Management
RIM (BlackBerry), Good (GoodLink)
Public Folder Migration
Quest Software, Tzunami
Secure Messaging
nCipher, RSA, CheckPoint
Storage (Backup, Disaster Recovery, High Availability)
HP, NetApp, Hitachi, CommVault, Symantec,
Doubletake, Computer Associates, UltraBack,
F5, Teneros, Azaleos, Cemaphore, Xiotech,
front-end servers, mailbox servers, and mail hygiene servers within their
designs for years. The difference in Exchange 2007 is that Microsoft has
defined server roles within the product so that, for example, you now manage
and view servers through ESM according to roles. To make the definition of
roles effective, Microsoft has divided the Exchange code base so that you
only install and operate the code on a server that is absolutely necessary.
There is no doubt that the more code you run on a server, the higher potential exists for you to uncover bugs, some of which are urgent problems such
as security holes that need to be plugged with hot fixes. In addition, if you
have a service pack or hot fix to apply, it is a lot easier when you only have to
apply it to a server that hosts the role that requires the updated code.
The previous generation of Exchange became more and more like a
monolithic messaging behemoth as Exchange 2000 went through four service packs, evolved into Exchange 2003 and added a new set of service packs,
Chapter 1
1.5 Server roles
not to mention the inevitable collection of bug fixes. Microsoft’s decision to
divide Exchange’s functionality into different roles allowed them to reduce
the code that runs on a server to just the set that is required to implement the
desired functionality and no more. Over the long term, deploying Exchange
in this way should increase server stability and remove complexity (because
you will not have unexpected or unwanted options appearing on servers), so
the introduction of server roles is rather more strategic than a simple exercise
in packaging.
In practice, because an Exchange 2007 server can take on multiple roles,
there is nothing to stop you deploying from Exchange 2007 servers in exactly
the way that you have deployed Exchange 2000 or 2003 servers in the past.
Many small to medium companies will deploy multi-role servers because
they don’t want to operate multiple physical computers just for Exchange.
The advantage of dividing Exchange functionality into different roles is really
for the enterprise market where companies commonly operate tens or hundreds of servers within a distributed network. Apart from the Edge role, you
can install and remove roles from a server to meet changing circumstances,
assuming that the servers have enough hardware capacity to take on the new
The set of server roles in Exchange 2007 are:
Client Access servers: These servers are responsible for proxying client
connections to the servers that host user mailboxes, much like
Exchange 2003 front-end servers handle incoming connections from
POP3, IMAP4, Web, and mobile clients before relaying them to
back-end servers that host user mailboxes. Client Access servers do
the work to render the Outlook Web Access user interface for Web
clients and they also provide the Exchange 2007 Web services such as
the Availability service to clients. Users whose mailboxes are on
Exchange 2003 servers can access their mailboxes through Exchange
2007 Client Access Servers. Microsoft does not support access to
Exchange 2007 mailboxes through Exchange 2003 front-end servers.
Mailbox servers: These servers are responsible for managing user mailboxes and handling MAPI connections to mailboxes and public folders. Mailbox servers are the only role that you can run on a cluster.
Hub transport servers: These servers form the basis of the messaging
topology and are responsible for routing messages between mailbox
servers inside the organization and the transmission of messages via
connectors to external messaging systems.
Edge transport servers: These servers are designed to operate in standalone mode and are not part of the Exchange organization or the
Active Directory forest, so they can be placed outside the firewall
1.5 Server roles
inside the DMZ to act as the first point of contact in the messaging
infrastructure. Edge servers run anti-spam and anti-virus agents to
cleanse the incoming message stream before handing it over to hub
transport servers for routing within the internal Exchange organization. No other server role can exist on an Edge server.
Unified messaging servers: This is a new server role for Microsoft, but
UM is not a new technology and other vendors such as Nortel,
Adomo, and Cisco have offered integration between PABX and
Exchange for years. The unified messaging servers are responsible for
the routing of messages between a PABX and Exchange so that you
can use the phone to communicate with Exchange through a new
product offering called Outlook Voice Access. You need to deploy a
gateway (IP-PBX or VoIP) to connect the UM server with your PBX
before you can use Outlook Voice Access.
The suggested deployment order for the most common roles is Client
Access server, mailbox server, and hub transport server. If you deploy server
roles on different physical servers, you need to ensure fast network links
between them. For example, you need at least 100Mbit links between Client
Access and mailbox servers—1Gbit is better.
You can only deploy mailbox servers on clusters (both traditional clusters—SCC—and the new continuous log replication—CCR—clusters),
largely because Microsoft has not done the testing to support the other roles
on when clustered. In addition, you can argue that server roles that typically
deal only with transient data (such as hub transport servers) do not need the
resilience to storage failure that clusters can deliver for mailbox servers. You
can achieve the necessary degree of resilience for other server roles without
clustering. For example, you can deploy several hub transport servers in a site
to load balance the messaging workload—if one of the hub transport servers
fails, the remaining hub transport servers take up the workload. The same is
true for Client Access and Edge servers.
Apart from the Edge server, you can combine server roles on a single
physical computer (similar in concept to the way that you deploy Exchange
2003 servers that host mailboxes and connectors) except clustered servers.
The design and code base of the Edge server is very different to the other
roles. There is some debate as to what is the best practice for installing an
Edge server. Some administrators advocate installing Edge servers into a
Windows workgroup to keep them well isolated from the production environment. Others prefer to install a separate version of the Active Directory
for the DMZ and install the Edge server into that domain. In either case,
every Edge server has its own and separate instance of ADAM (Active Directory Application Mode) to hold a read-only copy of the Exchange organization configuration (so that it can participate in routing) that is synchronized
Chapter 1
1.5 Server roles
from a hub transport server through a push process. The ADAM instance
also holds configuration data specific to the Edge server. To complete the picture, Table 1.2 summarizes how each of the server roles interacts with Active
Following a successful installation, you will find that a number of separate
services related to Exchange 2007 are active on your server. A Windows service is an executable file designed to run independently as a background process that Windows can launch during system start-up or when an application
starts. The exact set of Exchange services that you find on a server depends
on the roles that the server supports. All previous versions of Exchange have
Table 1.2
How different Exchange 2007 server roles interact with Active Directory
Active Directory Use
Authentication of MAPI client connections. Retrieval of information about mail-enabled objects from the directory plus the
necessary data required to enforce email address policies.
Client Access
Authentication of non-MAPI client connections (IMAP, POP,
ActiveSync, Outlook Web Access, Outlook Voice Access).
Lookup of server that hosts target mailbox.
Hub Transport
Lookup against the directory during message categorization.
Expansion of distribution groups. Execution of queries against
the directory to populate dynamic distribution groups. Expansion
of groups used in transport rules plus storage of transport and
journal rules in Exchange configuration data. Use of Active Directory site topology as the fundamental underpinning for routing.
Unified Messaging
Storage for global configuration such as dial plans, IP gateways,
and hunt groups. Active Directory is also used to build speech
grammars that are used to find users and to match telephone
numbers to accounts.
Uses an ADAM instance rather than Active Directory to hold its
configuration and other data, but synchronizes recipient and
configuration information from the production Active Directory so that the Edge server understands how to route messages
effectively. Edge synchronization also includes safe list information generated by users to improve the effectiveness of antispam processing.
1.5 Server roles
used a similar set of services; the difference is that Exchange 2007 uses rather
more services as a by-product of server role segmentation. Figure 1.7 illustrates a set of services active on a server that supports the mailbox, Client
Access, and hub transport server roles. Some services such as IMAP4 and
POP3 access are not used extensively with Exchange in most enterprises outside companies that provide hosting services. It is quite safe to leave these to
be started manually until you find that they are needed. Others, such as the
Store and System Attendant, represent the fundamental core of Exchange
2007 and run on any server. The Transport service is the other critical component, but this only runs on hub and Edge servers.
Figure 1.7
The set of
Exchange services
Table 1.3 lists the full set of Exchange services with a brief description of
what the service does, the server roles that it is valid for, and a note whether
the service is required or optional.
Table 1.3
Exchange 2007 services
Service name
Server Roles
Active Directory Topology (MSExchangeADTopology)
Runs under LocalSystem. Interrogates the Active Directory and
returns configuration and other data
to Exchange.
ADAM (ADAM_MSExchange)
Network service. Manages data about
Exchange configuration and recipient
data that is synchronized from the
Active Directory to an Edge Transport server.
(only by ET
Chapter 1
1.5 Server roles
Table 1.3
Exchange 2007 services (continued)
Anti-spam Update service (MSExchangeAntiSpamUpdate)
Runs under LocalSystem to automatically download anti-spam updates
from Microsoft.
(you have to
license the
Credential service (EdgeCredentialSync)
Runs under LocalSystem to monitor
any credential changes that occur in
ADAM and then implements the
changes on the Edge Transport Server
(only by ET
EdgeSync (MSExchangeEdgeSync)
Runs under LocalSystem on a HT
server to connect via LDAP to an
ADAM instance on an ET server to
synchronize data from the Active
Directory to ADAM. Depends on
AD Topology service.
(only if ET
servers are
File Distribution Service (MSExchangeFDS)
Runs under LocalSystem to copy the
files that make up Offline Address
Books to Web distribution points.
Also distributes any custom prompts
used by Unified Messaging. Depends
on AD Topology and workstation
IMAP4 (MSExchangeIMAP4)
Runs under LocalSystem to allow
IMAP4 clients to connect to mailboxes. Dependent on AD Topology
Information Store (MSExchangeIS)
Runs under LocalSystem to manage
mailboxes and public folders. Dependent on several Windows services
including NT LM Security, RPC,
Server, and workstation.
Mail submission service (MsExchangeMailSubmission)
Runs under LocalSystem to transfer
messages from a mailbox server to a
HT server. Dependent on AD Topology service.
Mailbox Assistants (MsExchangeMailboxAssistants)
Runs under LocalSystem to manage
the calendar, resource booking, OOF,
and managed folder assistants.
Dependent on AD Topology service.
1.5 Server roles
Table 1.3
Exchange 2007 services (continued)
Monitoring (MSExchangeMonitoring)
Runs under LocalSystem to provide
an RPC server that can be used to
execute diagnostic commands on a
POP3 (MSExchangePOP3)
Runs under LocalSystem to allow
POP3 clients to connect to mailboxes. Dependent on AD Topology
Replication (MSExchangeRepl)
Runs under LocalSystem to manage
the process of transaction log shipping at the heart of LCR (Local Continuous Replication) and CCR
(Cluster Continuous Replication).
Dependent on AD Topology service.
Search (MSFTESQL-Exchange)
Runs as LocalSystem to insert new
items provided by the Search Indexer
into the full-text index maintained
for mailbox databases. This is a customized version of the general-purpose Microsoft Search engine and is
dependent on the RPC service.
Optional (if
you want to
use content
Search Indexer (MSExchangeSearch)
Runs under LocalSystem to provide
new items to the Search service.
Dependent on AD Topology and
Search services.
Optional (if
you want to
use content
Service Host (MSExchangeServiceHost)
Runs under LocalSystem to manage
the RPC virtual directory in IIS and
the registry data required for Outlook
Anywhere. Dependent on the AD
Topology Service.
Speech Engine (MSS)
Runs as network service to provide
UM speech processing services.
Dependent on Windows WMI service.
(for UM)
System Attendant (MSExchangeSA)
Runs under LocalSystem to provide
monitoring, maintenance, and directory lookup services for Exchange.
Dependent on the same set of Windows services as the Store.
Chapter 1
1.6 Licensing
Table 1.3
Exchange 2007 services (continued)
Transport (MSExchangeTransport)
Runs as network service to manage
the Exchange SMTP service and the
transport stack. Dependent on the
AD Topology service.
Transport Log Search (MSExchangeTransportLogSearch)
Runs under LocalSystem to allow
administrators to trace the path of
messages through the message tracking log.
(must be
before message tracking can
Unified Messaging (MSExchangeUM)
Runs as LocalSystem to manage the
Exchange UM engine and provide the
services that make up Outlook Voice
Access. Dependent on the AD Topology and Speech Engine services.
(for UM)
Just for the sake of comparison, Exchange 4.0 had just four core services
(System Attendant, MTA, Information Store, and Directory Store). The
Internet Mail Connector (for SMTP) was an optional extra that you had to
pay for and we did not have to worry so much about spam and viruses.
Things have certainly changed since 1996.
There are two different versions of Exchange 2007: the standard edition and
the enterprise edition. Microsoft targets the standard edition at small to
medium companies and is the version that is included in Microsoft Small
Business Server. The enterprise edition is more appropriate to the messaging
needs of large corporations. As you can see from Table 1.4, the major differences between the two editions occur within the Store. Because the enterprise
edition focuses on large corporations, it is logical that it should support a
much larger amount of storage groups and databases.
Apart from deciding on what server edition to run, you also have to
make sure that every user (or device) has a client access license (CAL). There
are two types of CAL: standard and enterprise. The standard CAL provides
access to the basic functionality that makes Exchange an email server. The
enterprise CAL is additive, meaning that you have to purchase a standard
CAL first and then add the enterprise CAL to gain access to the enterpriselevel features. Table 1.5 lists the functionality enabled by each CAL type.
1.6 Licensing
Table 1.4
Table 1.5
Exchange database features in standard and enterprise editions
Maximum number of Storage
Up to five
Up to fifty
Maximum number of databases
Up to five
Up to fifty
Maximum size of an individual
No software limit
No software limit
Not supported
Support for both MNS
(CCR) and traditional
(SCC) clusters
Continuous replication
LCR (local) only
LCR (local) and CCR
Functionality gained through standard and enterprise CALs
Standard CAL
Enterprise CAL
Outlook Web Access
Unified Messaging
Per-User/Per-Group Journaling
Managed Folders
Exchange Hosted Filtering
Forefront Security for Exchange Server
Exchange hosted filtering and Forefront Security for Exchange Server
are not covered by licenses sold in the retail channel. You have to purchase
licenses from Microsoft through their Volume Licensing Program if you want
to use these features. Hosting filtering means that Microsoft takes responsibility for filtering email traffic en route to an organization by passing it
through a set of servers that Microsoft operates before redirecting the mesChapter 1
1.6 Licensing
Figure 1.8
EMC detects an
unlicensed server
sage stream to its original destination. This is a similar offering to that of
other companies such as Postini, Symantec, and MessageLabs.
Exchange 2007 is more demanding in terms of license enforcement. It
prompts administrators that they need to take corrective action if their servers remain unlicensed. Every time the Exchange Management Console
starts, it checks all of the current servers in the organization and reports any
that are unlicensed, as shown in Figure 1.8. All servers are unlicensed immediately after you install them. You move them to licensed status when you
input a valid product key (a standard 25 character alphanumeric string in
five groups separated by hyphens) that you purchase from Microsoft. You
can select a server and enter the product key or use the Set-ExchangeServer
PowerShell command to enter the product key. For example, this command
enters a product key for the ExchMbxSvr1 server:
Set-ExchangeServer –id ExchMbxSvr1 –ProductKey ‘10554-2127070916-06623’
Unlicensed editions expire 120 days after you install them, but you can
continue running these versions in environments such as test laboratories as
long as you can tolerate the constant nagging to enter a product key. Alternatively, you can remove and reinstall the test servers to start an extra 120 days
of nag-free work.
You can perform the same check as EMC with a single line of PowerShell code:
1.6 Licensing
Get-ExchangeServer | Where {$_.AdminDisplayVersion –iLike ‘*8.0*’ –and
$_.ProductId –eq $Null} | Select Name, Edition, RemainingTrialPeriod
We’ll get to do a lot more to work with Exchange data through PowerShell in Chapter 4, but for now it’s enough to know that this command queries the Exchange configuration data in the Active Directory for a list of
Exchange servers in the organization. A filter then reduces the set to just the
servers that run Exchange 2007 and have a blank ProductId property, which
indicates an unlicensed state. Edge servers always return a blank ProductId
property until they have taken out a subscription to a hub transport server
and joined the organization. The synchronization process between the Edge
and hub transport servers inserts the licensing information from the Edge
server into the Exchange configuration so that Exchange acknowledges the
license and EMC ceases to worry about the status of the server. The valid edition types5 are “Unknown,” “Standard,” “Standard Evaluation,” “Enterprise,” and “Enterprise Evaluation.”
Another interesting difference in the Exchange 2007 license that may
affect more companies is that it no longer includes a client license for Outlook. All previous versions of Exchange have included a PC client from the
original Capone client distributed with Exchange 4.0 to Outlook 2003 with
Exchange 2003. Microsoft believes that Outlook Web Access has progressed
to a state where it delivers so much functionality that it serves as an adequate
replacement, but as covered in Chapter 7, the inability to use the client
offline and lack of support for public folders until SP1 will make Outlook
Web Access a bad swap for many companies. The choice that you have is to
continue to use Outlook 2003 with Exchange 2007 or to negotiate a separate
upgrade with Microsoft to use Office 2007. As always with licensing, the
conditions and licenses that companies have vary greatly, so your company
should have a discussion with your local Microsoft representative to understand exactly what your Software Assurance or Enterprise Agreement covers
in terms of upgrades to new software versions.
Use the [Enum]::GetValues([Microsoft.Exchange.Data.ServerEditionType]) shell command
to retrieve the valid edition types.
Chapter 1
1.6 Licensing
Version numbers
Exchange 2007 is Exchange 2007, or is it? Well, it all depends what version
number you look at. The code name for the product was Exchange 12, which
is the version that Microsoft used until their marketing team made a decision
about the real product name. Internally, if you ask PowerShell to report the
two properties that contain version information for an Exchange 2007 server,
you get different answers in different formats:
Get-ExchangeServer –id ExchMbxSvr1 | Select ExchangeVersion,
--------------0.1 (8.0.535.0)
------------------------Version 8.0 (Build 685.24)
Figure 1.9
What version is
Exchange 2007?
The data reported in the ExchangeVersion property is not a product version at all. Instead, it is the minimum version of Exchange that can read the
object. In this case, any version of Exchange from 8.0.535 can read the
object, which is an Exchange 2007 server. Microsoft refers to Exchange 2007
as Exchange version 8; Exchange 2003 is version 6.5, Exchange 2000 is version 6.0, and so on through 5.5, 5.0, and 4.0. The AdminDisplayVersion
property reports the build number as well, so we know that this server is running the RTM6 release because that is build 685.24—or maybe 685.25 (as
always reported by an Edge server). Then you look at the Help About screen
and Exchange reports itself to be running build 685.018 (Figure 1.9). All in
RTM means Release to Manufacturing. This is the version that the engineering team builds to deliver to customers at the
end of their development cycle. The fact that Exchange 2007 reports 685.24 in the Admin-DisplayVersion property is due
to a bug.
1.6 Licensing
all, the different version numbers are slightly confusing and a good source for
a Trivial Pursuit question.
32-bit Exchange 2007?
All through its development, Microsoft insisted that the only version of
Exchange 2007 that they would support would be on the 64-bit x64 platform. They would provide a 32-bit version during the development period to
allow partners to understand the new release without deploying a 64-bit
environment, to support training facilities where it would be too expensive to
deploy 64-bit servers, to allow for installation on laptops (often used for test
environments), or to run on a virtual server. At the end of development,
Microsoft generated two versions of the RTM code: a 32-bit version and the
expected 64-bit version. What happened?
When the time came to release Exchange 2007, Microsoft discovered
that they could not keep to a hard line on 64-bit everywhere because it just
was not practical in real life. Administrators had 32-bit Windows XP workstations that they wanted to run the EMC console on and they wanted to
run the Exchange setup program on 32-bit Windows 2003 servers that supported the Active Directory to prepare the Active Directory for Exchange
2007 (extend the schema, etc.). These are obvious examples of why
Microsoft needed to provide 32-bit versions of Exchange 2007. A far less
obvious example is the limited support that Microsoft permits for using the
32-bit version of Exchange 2007 to run test servers to try out Exchange 2007
clusters (SCC and CCR), even though the 32-bit version is only provided for
the Standard Edition of Exchange 2007 that doesn’t support clustering. You
can also use the 32-bit version to try out Unified Messaging. In effect, this
decision was a pragmatic step by Microsoft to help customers get to grips
with new technology in the hope that customers would then accelerate their
adoption rate for Exchange 2007. However, you still cannot install a 32-bit
Exchange 2007 server into production and there is absolutely no guarantee
or commitment that Microsoft will ever update the 32-bit code base to
incorporate service packs or new functionality. Other missing functionality
includes no support for automatic anti-spam updates (you do not need automatic updates because you are not going to run this software in production)
and the same restriction on the number of storage groups and databases as
you have with the standard edition of Exchange 2007.
ExchangeQA/ for more information about the conditions under which
Microsoft supports the use of 32-bit Exchange 2007.
Chapter 1
1.8 Challenges for Exchange 2007
Exchange 2007 incorporates a new support mechanism. Instead of issuing
separate hot fixes for individual issues as they are reported, Microsoft plans to
create cumulative fixes that they will issue regularly, perhaps every six to eight
weeks at the start (when new software always has most bugs that need to be
fixed) and potentially less often thereafter. Cumulative means that the kits
contain every fix that Microsoft has made to Exchange 2007 since they
released the original version. Kits are cumulative from one to another, so Kit
5 contains all the fixes in Kits 1 through 4. You will be able to download the
fixes from Microsoft Update.
This approach simplifies life for administrators because you will know
that a patch kit will be available at regular intervals that you can incorporate
into regular server maintenance windows rather than having to constantly
check for new Microsoft Knowledge Base articles that document a new bug
and often mean that you have to install a patch.
Challenges for Exchange 2007
No one ever deploys a software product without encountering some
upheaval. The situation is more interesting because Exchange has a large
installed base of customers as well as the companies that service those customers with products of their own. In short, because Microsoft has elected to
change the architecture of Exchange, including the APIs that allow other
products to connect to and use the services offered by Exchange, the complete ecosystem undergoes a transformation around Exchange 2007. Challenges exist for Microsoft, their customers, service partners, and independent
software vendors (ISVs) to prepare for and then deploy Exchange 2007.
In some respects, the challenge before Microsoft is the simplest and
most straightforward. They have to deliver a high-quality, reliable, scalable,
and robust product that interfaces well with previous versions. They have to
perform the necessary testing and quality control to ensure that all of the features work as designed, that they achieve the necessary interoperability with
previous versions and other messaging systems so that clients can connect as
expected, that the bug count is within acceptable boundaries, and that the
software appears on time. This is an enormous effort, which is why Microsoft
has hundreds of engineers working on the Exchange 2007 project, but it is a
development effort that Microsoft has gone through many times before with
products such as Exchange, SQL, SharePoint, and Windows, so the task is
within expected boundaries. The biggest challenge facing Microsoft is the
smooth transfer from older versions of Exchange as customers migrate to
Exchange 2007. Migrations have always caused problems, but this migration
is more difficult because of the move to the 64-bit platform. Exchange has
1.8 Challenges for Exchange 2007
moved from one architecture to another before, as when we migrated from
Exchange 5.5 to Exchange 2000, but that operation could proceed on the
same Windows platform (if you upgraded to Windows 2000 first). With
Exchange 2007, we have to install new 64-bit servers, deploy the necessary
surrounding infrastructure (anything from Global Catalog servers to antivirus software), and then move mailboxes. Microsoft cannot build utilities to
transform a 32-bit Windows 2003/Exchange 2003 server to Exchange 2007,
so they have to test and document to be sure that everyone understands what
needs to be done and when it has to be done to achieve a graceful migration.
Customers will take the information provided by Microsoft and put it
into context with their own infrastructures to build a tailored migration plan.
Complexity increases because there are so many different ways to deploy and
use Exchange, including international and multi-lingual scenarios. Customers have to train their system administrators and support staff so that they
understand the new architecture. They have to make a decision whether they
need outside help from external consultants to plan and deploy Exchange
2007. They need to build and agree on hardware budgets so that 64-bit hardware is available when the deployment commences.
Possibly the most fundamental decision for any deployment of
Exchange 2007 is how long the migration project will take and when to start.
These decisions drive many other activities, such as training and hardware
procurement. You also have to consider user training if you plan to deploy
new clients alongside Exchange, especially if you want to perform a desktop
refresh and change operating systems at the same time perhaps to deploy a
combination of Office 2007 and Vista. Other issues come into play if you
plan a desktop refresh, such as cost, the sheer work involved in deploying
new desktop hardware, how to distribute and install the client software, and
so on.
Service partners (such as HP) track the development of new software
products carefully because they have to be ready to assist customers before
the formal ship date in projects such as Microsoft’s Technology Adoption
Program (TAP), which is the formal way that Microsoft engages customers to
test and comment on new versions of products like Exchange. As the product
approaches its official ship date, service partners have to be ready to scale up
to support customer deployments. Some service partners depend on
Microsoft to train their consultants while others build their own training
programs, as they want to include their own consulting methodologies and
best practice for Exchange. In either case, the trick is to have people ready at
the right time. If training happens too early, the risk exists that Microsoft has
not fully completed the software, so training cannot be complete. It is also
true that best practice evolves over time and with experience, so early training
often cannot cover best practice because it does not exist (or is pure guesswork). On the other hand, if you train people too late, you miss the chance
Chapter 1
1.8 Challenges for Exchange 2007
to take “first mover” advantage and the market may perceive you to be a follower rather than an influence.
The change in Exchange architecture automatically throws any existing
infrastructure into some doubt because you have to validate that the plan
that you used to deploy Exchange 2000 or 2003 will be equally successful
with Exchange 2007. Given the fundamental nature of the change in the
underlying operating system, hardware platform, Exchange software, and
influences such as increased automation and virtualization, it’s a stretch to
imagine that a design laid down a few years ago that focused on Exchange
2000 or 2003 will be right for Exchange 2007. Best practice changes over
time as our knowledge evolves about how we use software in production, so
this is another thing to take into account. For example, early designs for
Active Directory deployments assumed that domains represented security
boundaries, but we soon discovered that the forest is the only real security
boundary that you can depend on once Active Directory was subjected to the
stress and strain of real-life deployments. This discovery had a huge impact
on designs. Another truth about IT architectures is that time has a corrosive
effect on their effectiveness. The architectures work well best on the first day
that you deploy them, but they tend to degrade afterwards as users load servers with data, administrators make mistakes, software upgrades and patches
are applied, and new products are introduced to the mix. The best architectures survive the longest by being most resistant to influence of change, but
all architectures inevitably come around to the point when it is time to
rebuild and refresh. The combination of the move to the Windows 64-bit
platform, the new design for Exchange 2007, and the demands for more
mobility, more security, and more functionality is a good indicator that now
is a good time to take a fresh look at the architecture that underpins any
existing Exchange deployment and ask whether the migration to Exchange
2007 can be used as a catalyst for improvement.
In seeking the right type of change to make, it is always attractive to see
if you can drive cost out of the infrastructure by eliminating components
that you no longer need or that you can consolidate into a smaller set of servers. For example, Exchange 2007 offers the potential to consolidate many
32-bit servers to a smaller set of more scalable 64-bit servers. The same is true
of the basic Windows infrastructure, where you can look at consolidating 32bit domain controllers and Global Catalog servers onto a smaller set of 64-bit
equivalents. Seeking opportunities for server consolidation is enough reason
to conduct a review of any current infrastructure.
The move from routing groups to Windows sites as the basis for message
routing requires attention because it is unlikely that your Windows sites map
the routing infrastructure as implemented in routing groups. You probably
need to review your management procedures and validate that they work as
well with Exchange 2007 as they did with Exchange 2003.
1.9 Into the future
ISVs face another balancing act. They want to be ready to support a new
version when Microsoft ships the product, but they do not want to invest too
early because they will have to work against incomplete code. Some ISVs are
ready with an upgraded version of their product when Microsoft ships the
final code, others have a policy where they delay until the first service pack
on the basis that many customers opt to wait for this version before they
want to upgrade.
Into the future
Overall, working with Exchange since Microsoft introduced Exchange 4.0 in
1996 has been a real blast. Even though Microsoft has gone down some blind
alleyways (with application programming interfaces in particular) and we
still don’t fully understand how and when public folders will shuffle out of
the product, there is no doubt that the software has improved consistently
and usually has been easy to work with. In addition, I have met great people
across the Exchange community and enjoyed the projects that I have worked
on with customers around the world. Microsoft has taken a big step forward
with Exchange 2007, but before we go into the details of what they have
done, we need to set the foundation for success—and that is in a solid
deployment of Windows and the Active Directory.
Chapter 1
This page intentionally left blank
Exchange, Windows, and the Active Directory
No server software operates in a vacuum. Exchange depends on a solid
deployment of Windows to underpin the success of its own ecosystem. In
this chapter, we review the essential interoperation between Exchange and
Windows and the critical points where Exchange depends on Windows. In
particular, we will look at how Exchange uses the Active Directory, how
Active Directory replicates data, the role of domain controllers and Global
Catalog servers, and how Exchange updates and extends the Active Directory
for its own purposes.
Active Directory and Exchange
A good design and efficient operation of the Active Directory has been core
to any deployment of Exchange ever since Exchange 2000. Exchange makes
extensive use of the Active Directory to:
Store the configuration data for the Exchange organization. The
Active Directory stores this information in the “Microsoft Exchange”
container (Figure 2.1) in the configuration naming context and replicates it to every domain controller in the forest. Among the configuration data, Exchange holds information about valid routing paths
and connectors that the transport system uses to determine how best
to route messages.
Store permissions used to access Exchange data such as server objects.
Store information about mail-enabled objects—user accounts (mailboxes), public folders, contacts, distribution groups, and dynamic
distribution groups.
The most fundamental difference that exists between Exchange 2007
and previous versions when it comes to the Active Directory is the depen47
2.1 Active Directory and Exchange
Figure 2.1
data in the
Active Directory
dency that Exchange 2007 now has on the site definitions that you apply
within a forest. Previously, sites are interesting because they tell Exchange
servers where the closest Global Catalog and domain controller servers are,
but now Active Directory sites provide the information Exchange needs to
route messages when direct connections between hub transport servers are
unavailable using the IP subnet information that you define as the foundation of each site (Figure 2.2). In effect, the Active Directory site structure
that you deploy lays down the routing topology for your Exchange organization and every Active Directory site that contains a hub Exchange 2007
server can participate in message routing.
It’s fair to say that any Active Directory design made before the advent
of Exchange 2007 will not have taken message routing into consideration, so
it is a good idea to review the existing site layout and decide what sites will
host hub transport servers and what sites will remain outside the routing
Domain Designs
The classic Active Directory design deploys a root domain to hold the
FMSO servers and business or geographic-centric domains to hold the user
accounts. This design is based on the logic that you isolate enterprise components into the root domain, where you can protect the components against
inadvertent administrative mistakes by restricting access to the domain to a
small set of enterprise administrators. All of the application servers and the
user accounts are then deployed in one or more domains under the root
domain. Another reason that proved erroneous was the idea that domains
represented a security boundary, something that Microsoft eventually
2.1 Active Directory and Exchange
Figure 2.2
IP subnets define
an Active
Directory site
acknowledged to be untrue in 2001—but it is amazing that in some people’s
minds, the myth that the domain is a security boundary still persists. The
forest represents the only true security boundary for the Active Directory, so
if you need to operate multiple security domains, you will be forced to
deploy multiple forests, with all the attendant overhead of cross-directory
synchronization, authentication, and so on.
The multi-domain model is certainly valid and it has proven its worth in
deployments in some very large enterprises since Active Directory first
appeared in 1999 and the design continues to work in companies such as HP
where three regional-based domains (for the Americas, Europe, and AsiaPacific) organize user accounts on a geographic basis. Indeed, the multidomain design is the only way to implement specific features, such as applying different password policies to each domain. If you do decide to deploy
multiple user domains, make sure that you restrict the number of Enterprise
Admin accounts to as few a number as possible and use delegation to allocate
the rights to allow administrators to take control of the elements that they
need to manage, such as an organizational unit or server. You should also
establish clear rules for situations where decisions are needed that influence
multiple domains, such as the appointment of a new enterprise administrator
or the allocation of rights over the root domain, as you do not want situations to arise where an inadvertent action on the part of an administrator
impacts multiple domains.
Chapter 2
2.2 Active Directory replication
In the light of experience and in the spirit of removing complexity
whenever possible, the advice now is to have as few domains as possible and
place the FSMO servers in the same domain structure as user accounts.
Instead of depending on domain access to protect sensitive components, you
place the components into Organizational Units that you protect with
appropriate access controls. The ideal situation in most situations is to deploy
a single domain for everything.
Microsoft designed Exchange 2000 to support a single organization per
Active Directory forest and the one-directory, one-organization restriction
remains in place with Exchange 2007. Many companies find it limiting to be
only able to deploy a single Exchange organization, as they want to segment
users across multiple messaging systems and connect the systems with SMTP.
While it might appear that dividing users across multiple systems generates
increased cost and complexity in operations, email address synchronization,
and message transport, in some cases it is the right solution because a real
need exists to maintain separate messaging communities within the one company. For example, a financial services company might have divisions that
handle trading and investment banking and need to control the communications between the two divisions to satisfy legislative or regulatory requirements. Setting up two messaging systems is one approach, as is the
deployment of specialized software to monitor communications between the
two sides. However, if you elect to go with two messaging systems and want
to use Exchange, you have to deploy two Active Directory forests.
Active Directory replication
The Active Directory replicates configuration data about itself and applications such as Exchange together with information about objects such as
users, contacts, and distribution lists (groups) between domain controllers in
a domain and to Global Catalog servers within the forest. Replication is the
term that describes the propagation of data from an originating server to
other connected servers, in this case within a Windows domain or forest of
domains. We could spend a lot of time diving into the details of how Active
Directory replication works, but this topic deserves its own book. Instead, we
will look at the basics of Active Directory replication between domain controllers and Global Catalog servers.
Understanding how the Active Directory replicates information is
important for Windows and Exchange administrators alike. If replication
does not flow smoothly then Exchange or any other application that depends
on the Active Directory will experience problems. Successful Exchange
projects invariably have a solid and well-managed Active Directory deployment in place before the installation of the first Exchange server. The move to
use Active Directory sites as the basis for the routing topology in Exchange
2.2 Active Directory replication
2007 further underlines the need to make successful replication a high-priority item for any Exchange 2007 deployment.
Replication basics
The goal of any replication mechanism is to propagate data as quickly as possible so that all participants in replication receive updates and can build their
own synchronized copy of the data. After replication has finished, all copies
should be consistent. In a fast-moving networked environment where users
and computers update data all the time, it is difficult to achieve a state of perfect replication, so you can consider the Active Directory partition hosted on
any individual domain controller to be in a state of loose consistency. In
other words, while the vast majority of the data in the Active Directory is
consistent and synchronized with the server’s replication partners, it is likely
that outstanding replication operations are en route somewhere in the network. In a perfect world, we would have only one copy of the directory
(unlikely a worldwide deployment) or be able to freeze changes for a time to
allow all servers to perform any outstanding replication operations to achieve
perfect synchronization throughout the network. Unfortunately, we do not
live in a perfect world, so we have to be satisfied that the Active Directory is
loosely consistent at any point in time.
Windows domain controllers are responsible for initiating and participating in replication operations. Each domain controller holds a complete
read-write copy of the objects for its domain and acts as a replication partner
for the other domain controllers within the domain. Replication only occurs
between domain controllers as member servers do not hold a copy of the
Active Directory.
Global Catalog servers hold a partial read-only copy of the objects for
the other domains in the forest as well as the fully writable set of objects for
its own domain. We say partial copy because Windows replicates a subset of
the full attribute set of objects from domains to every Global Catalog along
with a complete copy of all objects from its own domain. You can change the
Active Directory schema to force replication of additional attributes. See
page 80 for details.
The Active Directory performs attribute-level replication to reduce the
amount of data that a replication partner sends or receives after a change has
occurred. This means that the Active Directory only replicates the content of
changed attributes, plus the GUID to identify the object and data to track
replication rather than replicating the complete object including all the
attributes that haven’t changed. For example, if you change the display name
for a user account, then the Active Directory only sends that attribute to replication partners. The net effect is that the Active Directory throttles back
network traffic to only the size of the changed data and a buffer (perhaps 600
Chapter 2
2.2 Active Directory replication
bytes in total, depending on the attribute that you update). In addition, the
Active Directory uses a multi-threaded replication agent to be able to handle
a high volume of replication requests. Given the speed of the network that
connects many Windows servers today, you might not think that it’s important to save a couple of thousand bytes on the wire here and there, but you
have to remember that the Active Directory continues to hold more data
about objects as applications exploit its ability to act as a repository for information. Exchange 2007 allows users to publish their safe lists as attributes in
the Active Directory so that Exchange can then synchronize this information
with Edge servers, which then use the data to check the sender information
on incoming messages to detect spam. Exchange 2007 also introduces the
Unified Messaging server and this adds a slew of new attributes to the Active
Directory. The average size of an Active Directory object is getting larger as
time goes by and there are many more attributes that applications can
update, so attribute-level replication is a very important feature.
To help monitor and control the flow of replication, the Active Directory maintains a high watermark vector table on each domain controller. The
table contains a row for every replication partner of the domain controller,
including the partner’s highest known USN (unique serial number). A USN
is a 64-bit number maintained by each controller to indicate how many
changes have occurred on the domain controller. The Active Directory also
maintains USNs for every object in the directory and for every attribute in
an object, so multiple USNs are involved in the replication process. Maintaining USNs at an attribute level allows the Active Directory to control replication at that level rather than replicating complete objects.
As you can see in Figure 2.3, the Active Directory tracks the changes
made to an object and you can see both the USN value when the Active
Directory first created the object (73,476) and the current USN
(1,019,026,971). These USN values are specific to a domain controller and
you will see other values for the same object if you connect to a different
domain controller in the same domain. However, if we take the figures at
face value, then this domain controller (part of HP’s forest) has
processed nearly a billion separate changes that have been replicated to this
domain controller in 76 months. Some of these changes originated on the
domain controller, but given the size of HP’s Active Directory and the number of domain controllers in the four domains that make up the domain, the
majority of changes are likely to have originated on other domain controller.
Another way of looking at this is that the domain controller processed over
465,000 changes in each of the 2,316 days since the object was originally created. This kind of churn in the directory is indicative of major change within
an organization that is typical when mergers and acquisitions occur.
It is common to see such a heavy volume of changes in a large organization and this illustrates the replication traffic that the Active Directory can
2.2 Active Directory replication
Figure 2.3
handle, plus the wisdom of using a 64-bit number (a GUID) to hold USNs.
Technically, when the number overflows because of the number of changes
applied to a controller, the USN reverts to one and starts over again. This will
provoke a huge amount of replication to occur within a network as the controller will ask its replication partners for “any change since USN 1,” but you
shouldn’t worry too much about this situation as it is extremely unlikely to
occur in our lifetime (just like Y2K!).
When Active Directory replication happens
The Active Directory does not dispatch replication updates immediately after
changes occur. If this was to happen, replication updates might swamp the
network if a program applied many changes over a short period. Consider
what happens when you synchronize data from an external directory with the
Active Directory. The typical synchronization routine process uses a load file
or another mechanism (such as a direct LDAP connection) as input to a program that synchronizes the content of another directory with matching
entries in the Active Directory. For example, HP’s LDSU (LDAP Synchronization Utility) is able to synchronize the Active Directory with any LDAPcompliant directory or any directory that can generate an LDIF-format load
Chapter 2
2.2 Active Directory replication
file. During the synchronization process, the Active Directory process
instructions are to add, change, or delete objects very quickly (programs have
been written to add objects at well over 500 objects per second), and if replication were triggered after each individual update, the network would grind
to a halt due to all of the replication updates. Instead, the replication mechanism on a domain controller gathers changes together and sends out packages of replication data that its replication partners can apply. The exact
method depends on whether the Active Directory needs to replicate data
within a single Windows site or between Windows sites, but in all cases, the
Active Directory packages groups of updates together instead of sending
individual changes.
Any change to the Active Directory provokes some replication activity.
The basic operations are:
Object creation: For example, when you create a new user object,
contact, or group.
Object modification: For example, when you add a new SMTP
address to a user, or add a user or contact to a mail-enabled group, or
if you move a user from one organizational unit to another.
Object deletion: The Active Directory does not remove an object
immediately. Instead, it creates a tombstone for the object and replicates the tombstone to other domain controllers to inform them that
the object has been marked as deleted. Tombstones have a default
expiry time of 60 days, and after this time, the Active Directory permanently removes all trace of the object and completes the deletion
by replicating the final state change.
The Active Directory makes a distinction between two types of write
operations. When you initiate an operation on a domain controller, the
update that the Active Directory applies to its local copy is called an originating write. A replicated write is an operation that occurs as the result of
incoming data that arrives from a replication partner for application to the
local Active Directory database. For example, if we increase the mailbox
quota for a user on the HPDC1 domain controller and then replicate the
information to the HPDC2 domain controller, we refer to the update
applied on HPDC2 as a replicated write.
The Active Directory maintains two internal GUIDs for each controller,
one for the server itself and one for the database. The server GUID remains
constant, even if the server name changes. The Active Directory allocates a
unique Server GUID to each domain controller and uses it to reference and
identify specific controllers during replication operations. The Active Direc-
2.2 Active Directory replication
tory also creates a GUID to identify the copy of a database on a domain controller and sets it to be the same value as the Server GUID. If you ever need
to restore a database from a backup, the Active Directory alters the database
GUID to inform other domain controllers that you have restored the database, which may therefore be in an inconsistent state and need to generate
some backfill requests to recover some updates that occurred after the backup
was taken.
Active Directory naming contexts
In Windows 2000, the Active Directory organizes data into three separate
naming contexts (NC). Windows 2003 introduces the application NC. A
naming context is a tree of objects. In some definitions of directory terminology, a naming context represented the level of replication or referral within
the directory.
The configuration NC contains the objects that represent the structure of the directory (controllers, site topology, etc.). Because it uses
the Active Directory as its repository and stores information about its
organization (servers, roles, rules, and so on) in the configuration
NC, this part of the Active Directory is very important to Exchange
The schema NC contains all the classes and attributes that define
individual objects and their attributes.
The domain NC defines all the other objects such as users, groups,
contacts, and computers. A separate domain NC exists for each
domain in the forest.
Domains act as partitions within the Active Directory. To some degree,
domains also act as a replication boundary, but not for the schema, which the
Active Directory shares throughout the forest, or for the partial set of domain
data that the Active Directory replicates to Global Catalogs throughout the
forest. Of course, if your deployment manages to use just one domain across
a high-speed network then the replication issue becomes very simple indeed
and you will probably not have to worry about where you replicate data to
and how long it takes to get there. On the other hand, even a single domain
can run into replication issues if low-bandwidth links connect the sites and
the topology is overly complex.
When Microsoft first shipped Windows 2000, best practice for corporate deployments recommended the use of multiple domains arranged into a
single forest. This approach was liked by administrators because in some
respects, it mimicked the way that Windows NT domains divided up the
Chapter 2
2.2 Active Directory replication
administrative workload and responsibilities. As time has gone by, best practice focused more on removing complexity within the Active Directory as
much as possible by reducing the number of domains, so a design exercise
started today is less likely to create multiple domains within a forest than previously. Even if you have just two domains within a forest, the scope of replication becomes a very important issue, especially when you deploy an
application that makes extensive use of the Active Directory (like Exchange)
on top of the basic Windows infrastructure. For example, if Active Directory
replication does not work predictably across the forest, then the contents of
the GAL are likely to be inaccurate.
Naming contexts set boundaries for data replication. The configuration
and schema are unique within a forest, which means that the Active Directory must replicate these contexts to every domain controller in the forest,
so they have a forest-wide scope. In Exchange terms, this means that an
Exchange server can query any domain controller to discover a complete
picture of the Exchange organization data. The Active Directory only replicates domain objects to the controllers within a domain, so this naming
context has a domain-wide scope. Besides storing the domain NC of its own
domain, the Active Directory includes a partial replica of all other domain
NCs in the database held by a Global Catalog to build a complete picture
across the forest. The partial replica of a domain NC contains roughly 30%
of the data stored in the full replica of the NC, so each Global Catalog that
you create incurs a replication penalty that grows with the number of
domains in the forest.
Exchange exploits the fact that the Global Catalog provides a full view
of the user objects in a forest to build address lists like the GAL through
LDAP filters that it applies to the copy of the Active Directory hosted on a
Global Catalog. The LDAP filter essentially “finds” every mail-enabled object
known to the Global Catalog whose showInAddressBook flag is set to true.
Because its content forms the basis of the GAL, the Global Catalog is
extremely important to Exchange. For example, when Outlook clients want
to look up an address in the GAL, behind the scenes Exchange fetches the
information from the Global Catalog and responds to the client. The
Exchange transport engine makes heavy use of Global Catalog data because it
needs to check email addresses in message headers to decide how best to
route messages.
A typical forest that supports Exchange has one GAL, but you can create
more address lists by specifying filters that focus in on a specific collection of
mail-enabled objects. This feature is often used by service providers that supply an email service to companies so that each company is able to see their
own users but no one else. In a Windows 2003 forest, the technical limit for
the number of GALs or address lists in the forest is 1,300. However, this is a
theoretical limit and you would quickly lose yourself in a flurry of address
2.2 Active Directory replication
Figure 2.4
Exchange routing
information in the
configuration NC
lists if you attempt to create that many. The vast bulk of Exchange implementations are happy with just one GAL.
While Exchange makes heavy use of the Global Catalog for address validation and message routing, it also needs to access configuration data for
management and other operational purposes. For example, you cannot navigate through a complete view of the organization if the EMS console cannot
read it from the data held in the Microsoft Exchange container in the configuration NC. Figure 2.4 illustrates how Exchange stores information about
the domains that Exchange 2007 will process email for in the configuration
NC in the Transport Settings section under “Microsoft Exchange.” When
you use the EMC to manage Exchange, EMC reads the configuration data
from the nearest domain controller. Generally, the larger the Exchange organization, the slower EMC performs because of the amount of data it fetches
from the Active Directory.
Generally, you should only work with Exchange configuration data
through EMC or through shell commands as the console and commands will
stop you doing something silly, like deleting essential information from
Exchange’s configuration data. Sometimes, the need might arise to eradicate
completely all traces of Exchange from a forest, perhaps because you have
made a mistake (like using the wrong organization name) when you first
installed Exchange and you want to start over. You can use the ADSIEDIT
utility (see page 88) to remove Exchange from an organization very quickly.
Simply select the “Microsoft Exchange” container, right-click, and select
“Delete” from the menu. Afterwards, you need to wait for replication to
Chapter 2
2.2 Active Directory replication
occur to complete the removal of Exchange from the forest. This action will
not roll back the schema changes that Exchange makes to the Active Directory, but it does mean that you can restart the deployment of a brand new
organization. Note that removing Exchange in this way is a very drastic
option, so be sure that you have good reason to remove the organization in
this way before you proceed. Knowledge Base article 312878 documents the
approved Microsoft method to remove an Exchange organization from the
Active Directory, while article 260378 also provides some useful background
Transforming Domain controllers into
Global Catalogs
You turn domain controllers into Global Catalogs by changing the server’s
NTDS Settings property through the Active Directory Sites and Services console. Navigate to the site that holds the server, select the server, and view its
NTDS Settings properties, as shown in Figure 2.5. Check the “Global Catalog” box to make the controller start to pull information from other domains.
If you clear the checkbox, the server will revert to become a domain controller
and the Active Directory will flush data belonging to other domains from the
local database. This process may not happen quickly because it depends on
the number of objects from other domains that exist in the local database. The
Knowledge Consistency Checker (KCC) component removes these objects
every time it runs and spreads the load by deleting about 8,000 objects an
hour. It is therefore obvious that it can take some days before a DC finally
purges data for other domains from its database and during this time, you
cannot reverse the process and make the domain controller into a Global Catalog again. In large forests and especially where a relatively low-bandwidth
connection exists between the server that you’re working with and the nearest
domain controller, it may be faster to remove the domain controller completely, reinstall the server from scratch, and then reload the Active Directory
from media!
Transforming a domain controller into a Global Catalog generates replication activity. If you add a second Global Catalog to a site, the new Global Catalog copies the partial replicas of other domain NCs from the
existing Global Catalog rather than requesting data from outside the site.
This step avoids the need to broadcast replication requests outside the site
boundary and thus potentially generate a flood of replication data within
the forest. Otherwise, the exact impact on a network and Windows infrastructure depends on the complexity of the site topology, the number of
domains in the forest, and the number of controllers involved in the replication process.
2.2 Active Directory replication
There is no good answer to how quickly the Active Directory can replicate data from one part of a forest to another. Microsoft recommends using
tools like the Performance Monitor to review the performance of the NTDS
Objects (such as DRA Outward Bytes Total and DRA Inward Bytes Total, or
the total bytes consumed in outward and inward replication activity). However, this is a rudimentary approach to the problem and it is better to use the
more sophisticated tools from companies like Quest Software or NetPro. In
addition, to help gain an insight into potential problem areas for replication
between sites and controllers, you can use tools like HP’s OpenView Active
Directory replication monitor to measure end-to-end replication latency and
the overall effectiveness of replication inside a forest.
Figure 2.5
Setting the Global
Catalog property
for a domain
Exchange’s dependency on consistent availability of Global Catalogs
and the underlying reliable replication of Active Directory data illustrates
the point that you cannot approach a design exercise for the Active Directory in isolation from the applications that depend on the infrastructure.
This is a major and far-reaching change for any company that traditionally
keeps infrastructure design teams separated from application design
teams—and we see the same issue arising with Unified Messaging where
successful deployments depend on a close working relationship between the
telephony and messaging teams. The intimate dependency of Exchange
2007 on the Active Directory means that successful design teams treat the
infrastructure as a whole rather than as separate parts. It is worth noting that
while Exchange has a tremendous dependency on proper Active Directory
Chapter 2
2.2 Active Directory replication
deployment and operation, you also have to manage other Windows and
network components to bring everything together. For example, if you do
not deploy DNS properly, Exchange may not be able to locate domain controllers and Global Catalogs to read information about the organization’s
configuration and mail-enabled objects.
USNs and replication
As mentioned earlier, the Active Directory uses USNs to track changes made
to its objects. Two different USNs are maintained for each object. The
domain controller sets the value of the USNCreated attribute for a new object
to the value of the server USN when it creates the object and then updates
the USNChanged attribute each time it processes an update for the object.
When an object is replicated to another domain controller, the Active Directory on the receiving controller sets the value of USNChanged to the value of
the server USN of the server where the originating write occurs. You can see
the value of a server USN by examining server properties, as shown in Figure
2.6. Now that we know the basics, we can track what happens during a simple set of replication operations.
Figure 2.6
Server USN
This scenario uses two controllers in a domain called HPDC1 and
HPDC2. The servers have been in operation for some time, so the values of the
2.2 Active Directory replication
server USNs reflect the number of AD updates. For this example, we assume
that HPDC1 begins with USN 10925 and HPDC2 starts with 11933.
We begin replication by creating a new user object on HPDC1. The
Active Directory automatically allocates the next USN in sequence based on
the server USN, or 10926. Note that the allocation of a USN is a one-time
operation and the USN can never be reused even if the create operation is
interrupted and fails to complete. The next update on HPDC1 increases the
USN by one. Table 2.1 lists a subset of the values maintained in the user
object table in the database on HPDC1.
Table 2.1
User Table in Active Directory
First name
[email protected]
The extract illustrates a number of important points:
The originating domain controller automatically adds a number of
values to each attribute to control replication.
Each attribute populated as the Active Directory creates the new user
object receives the USN allocated to the object. The “Originating”
USN is set to the same value because the operation is initiated on the
local server. This value will change as the Active Directory updates
attributes over the lifetime of the object.
The Active Directory populates the “Originating DC GUID” with
the GUID from the server where it creates the object. Again, this
value can change if another domain controller updates an attribute.
The version number is set to “1” because the originating domain controller has just created the object. This number is incremented as
updates are applied. The Active Directory uses the version number to
resolve replication conflicts that might occur if two or more controllers attempt to update the same attribute at roughly the same time.
Chapter 2
2.2 Active Directory replication
Conflicts are resolved by accepting the update with the highest version number. This is especially important when resolving conflicts for
multivalued attributes (such as groups). For example, if you make
two changes to a group on one domain controller that overwrites a
single change made to the same group on another domain controller
later on.
A timestamp extracted from the current date and time on the server is
saved. The Active Directory uses timestamps as a last resort to resolve
replication conflicts that might occur if two or more controllers
attempt to update the same attribute at roughly the same time and
end up with a version number that is the same. Conflicts are resolved
by accepting the last update according to the timestamps, using the
timestamp from the originating write to arbitrate the conflict.
Note also that the Active Directory assigns the new object the current
value of the server USN (10926) in its USNCreated and USNChanged
attributes. We now replicate the information to HPDC2, which inserts the
new information into its copy of the directory. Table 2.2 shows the values
from HPDC2.
Table 2.2
User Table in Active Directory (2)
First name
[email protected]
The Active Directory changes the USN to reflect the server USN on
HPDC2 and also assigns this value (11934) to the USNChanged attribute
because this is the initial creation of the object on this controller. The timestamp now contains the date and time when HPDC2 applied the update. The
originating DC GUID and the originating USN are still the values from
HPDC1 because this is the controller where the update originated. We now
see the difference between an originating write and a replicated write.
2.2 Active Directory replication
Table 2.3
User Table in Active Directory
[email protected]
Table 2.3 demonstrates what happens after you update the user’s title on
HPDC2. We populate a previously blank attribute, which forces the USN on
HPDC2 to move to 11935. The Active Directory also updates the USNChanged value to 11935 and then sets the originating DC GUID for this
attribute to HPDC2 because the update originates from this controller, so
the Active Directory also updates the originating USN to the USN from
HPDC2. Eventually, replication will occur back to HPDC1, which results in
the values shown in Table 2.4. The timestamp is set to the value of the originating write.
Table 2.4
User Table in Active Directory
[email protected]
First name
Chapter 2
2.2 Active Directory replication
The important things here are that the USN has been incremented
according to the server USN for HPDC1, but because the write originated
on HPDC2, the Active Directory takes its server GUID and USN and uses
them to update the database on HPDC1. The Active Directory also updates
the timestamp. However, the Active Directory leaves the USNCreated value
for the object unaltered at 10926, but updates the USNChanged value to
10927. Note that the Active Directory leaves all the other attributes alone, as
it only needs to update the value of the Title attribute. While it may seem
obvious to keep track of the controller that last updated an attribute by holding the server GUID, the role of the originating USN might be more
obscure. The way that the Active Directory uses the originating USN
becomes more important as you increase the number of domain controllers
that participate in replication. In fact, the Active Directory uses the originating USN for propagation dampening, or the ability to stop replication operations progressing if the Active Directory has already updated the local
database after an interchange of replication data with another controller.
Propagation dampening is very important in large networks as it eliminates
unnecessary traffic and reduces the amount of system resources that are
required to keep the directory in a consistent state.
Urgent replication
The normal replication process is sufficient to ensure that the Active Directory replicates updates to attributes such as telephone numbers, titles, or even
email addresses between DCs reasonably quickly. Two situations exist when
fast replication is required. These are when a user account is locked out and
when a new set of identifiers is issued by the RID master.
The Active Directory supports the concept of “forced” replication to get
updates to domain controllers as quickly as possible after you lockout or disable accounts. This feature exists to prevent users being able to move between
domain controllers and log in after an administrator has disabled or locked
their account. If the Active Directory depended on normal replication in
these instances, the danger exists that the newly locked out user could still
authenticate against a controller that has not yet received the update and so
continue to access resources. You should realize that this mechanism does not
prevent users who previously logged onto the network from accessing the
resources until their Kerberos ticket lifetime expires. In case of NTLM
authentication, the users can continue to access resources until they logoff.
Note that the Active Directory does not consider password updates as
urgent. When you update your password, the controller that you make the
change to replicates the new password to the PDC emulator master, which
then replicates the updated password to other domain controllers in the next
replication cycle. Note that the site topology might prevent a controller com-
2.2 Active Directory replication
municating with the PDC emulator, but normal inter-site replication will
eventually get the change through.
Finally, if the RID master issues a new set of identifiers to a domain controller to allow it to generate unique identifiers for new accounts, the Active
Directory must send that information quickly to its destination. In all these
cases, fast replication or rather a faster form of replication is performed by
immediately sending out change notifications to replication partners, which
then respond by pulling the update from the originating controller.
Intrasite and Intersite replication
Exchange 4.0 introduced the notion of a site to reflect locality in terms of
network connectivity. The Active Directory also uses the concept of a site,
but with some subtle differences. An Exchange 4.0 site was defined as a collection of servers that share good network connectivity, often because all the
servers in the site are in a common location (office, city, country, or even
continent). The definition of a Windows site is somewhat stricter. An Active
Directory site is composed of a collection of one or more IP subnets, but like
an Exchange site, an Active Directory site shares LAN-quality bandwidth
between the servers.
In logical terms, a Windows infrastructure builds sites on top of the
physical network and sites map the physical network to create the Active
Directory site topology. Another way of thinking about this is to remember
that sites are collections of servers in a location whereas domains are collections of objects that may exist across multiple sites. In actuality, when you
deploy your first Windows domain, it goes into the default site (called
“default-first-site”). New domain controllers are also installed into the same
site unless you decide to create other sites to accommodate the requirements
of the network. You introduce a domain into a site by creating a replica
(domain controller) for the domain in the site. A domain may span many
sites, and a site can host multiple domains as long as a domain controller for
each domain is present in the site. Exchange 2007 imposes another requirement by mandating that one of the domain controllers in the site must also
be a Global Catalog before the site can participate in message routing.
Windows limits domain NC replication to synchronous RPCs. This
means that you can only include servers in a domain if they can establish
RPC connectivity with the other servers in the domain. The connection is
not limited to a LAN, but within a WAN, you find that RPCs are sensitive to
latency and network availability and may time out or otherwise fail to complete. The requirement to support RPC connectivity means that some
designs use domains to restrict replication and end up with more domains
than strictly necessary. The Active Directory can replicate the configuration
and schema NCs through asynchronous SMTP messages, so replication can
Chapter 2
2.2 Active Directory replication
truly span the world. In this case, the Active Directory uses the Windows
SMTP service to send the messages containing replication data.
When you first create the forest, Windows creates a default site link. You
cannot create a site without associating it with at least one site link, so you
can either use the default link or create new ones. Site links connect together
to allow replication to proceed. The existence of a site link indicates that network connectivity is available between the two sites. Unlike the automatic
connection objects created by Windows to replicate data between two partners in a specific NC, you must create site links before intersite replication
can proceed.
The Knowledge Consistency Checker (KCC) manages the creation of
the intrasite topology and works with the ISTG (Inter-Site Topology Generator, a subprocess of the KCC) to ensure that Windows optimizes Active
Directory replication. The KCC is a service that runs on every domain controller to generate and optimize the Active Directory replication topology by
creating connection agreements between domain controllers. Costs range
from 1 to 32767, with 100 being the default. In general, the lower the cost,
the better the network connectivity that exists between sites. Because they
link sites, which are IP subnets, you can think of site links as WAN connections. Each site link has a cost, and the ISTG uses the cost and site link
schedule to determine which connection objects it must create to enable replication. The connection objects created by ISTG to link the different sites
form a spanning tree, designed to avoid message looping as updates flow
between bridgehead servers in the sites. This is important with asynchronous
replication (see below), where messages that contain replication data may
take some time to reach their final destination. Administrators can also create
connection objects, but the usual approach is to let the ISTG create a default
set and only interfere if necessary afterwards.
Because good network connectivity is assumed, directory replication
occurs automatically and frequently inside and between sites. Replication
partners notify each other when they have updates, and the partners then
pull the data from the originating server to update their directory. Even if
no updates exist for replication, the domain controllers in a site exchange
details of their latest USNs to update their vector tables and ensure that they
miss no data. The Active Directory uses the same pull mechanism for intesite replication. In this case, bridgehead servers hold data until their replication partners (bridgehead servers in other sites) request updates and then
pull the data.
Because the Active Directory assumes good connectivity exists between
servers in a site, it never compresses replication data and uses RPCs to communicate between servers. By default, if there are two domain controllers in a
site, they replicate every 5 minutes. If more than three domain controllers are
present in the same site, the KCC will set up the replication connections to
2.2 Active Directory replication
ensure that changes from any domain controller within the site reach all
other domain controllers within three hops. That is, any changes replicate to
all domain controllers within 15 minutes. On the other hand, the Active
Directory always performs intersite replication according to a schedule,
which allows administrators to distribute changes when they feel appropriate.
If replication data is over 50KB, the Active Directory compresses it to
between 10–15% of its original size before transmission, trading network
consumption against the processing to compress and decompress the data.
The Active Directory can replicate synchronously or asynchronously
between sites. Synchronous replication occurs between two bridgehead servers within the same NC. The bridgehead server acknowledges receiving the
data and then distributes it to other domain controllers in the site. Synchronous replication can only happen over a reliable and relatively high-speed
connection. Asynchronous replication allows replication to occur over slow
or unreliable links by sending SMTP messages using a component called
Inter-Site Messaging (ISM-SMTP). ISM generates messages containing replication data and sends them through the basic SMTP service included in
every Windows. However, you cannot replicate the domain NC over SMTP
and the experience gained in enterprise deployments demonstrates that few if
any large companies use SMTP replication for the Active Directory. Because
Active Directory replication is so important, most large companies deploy
domain controllers within the parts of their networks that have best connectivity to ensure that replication works predictably. The reduced cost of network experienced in most countries since Microsoft introduced Windows
2000 has also reduced the attractiveness of asynchronous replication, so to
some degree you can consider this a feature that may help in very specific circumstances (such as Active Directory deployments over extended low-speed
networks), but one that has hardly been proved in practice. Table 2.5 summarizes the differences between intra- and intersite replication for the AD.
Table 2.5
Characteristics of Active Directory replication
Spanning tree
Replication timing
Automatic when necessary
(every 5 minutes by
According to schedule as
defined by administrator
Replication model
Notify and Pull
Store and Forward
Full (if over 50KB)
Chapter 2
2.2 Active Directory replication
Within a site, Windows sets the default schedule for domain controllers
to send notifications of updates to their replication partners to 300 seconds
(5 minutes). You can change this interval for all members of a site, but there
is little reason to do this normally unless you want to tune back the amount
of replication traffic that the Active Directory generates. One exception to
the rule is when you want to send changes to the bridgehead server in the site
so that it begins to replicate with its partner sites as soon as possible. This
technique can propagate changes faster within a distributed Active Directory,
but you need to test and measure results before committing it to deployment.
Each site link and connection object has a schedule (Figure 2.7), which
defines when the domain controllers associated with the connection object
will replicate. Each time a replication slot occurs in the schedule, the domain
controllers inside the site exchange information with each other to establish
whether they need to replicate. The site link schedule takes precedence over
the connection object schedule for intrasite replication.
Figure 2.7
Connection Object
High-watermark vector and up-to-date
vector tables
The Active Directory incorporates a number of propagation dampening
mechanisms to control the amount of replication within the network. Propagation dampening means that a domain controller can suppress or ignore
unnecessary replication under specific circumstances. In other words, if a
2.2 Active Directory replication
domain controller receives information that it already knows about, such a
request to create a new user object in their copy of the Active Directory, the
domain controller can discard the update. Elimination of unnecessary replication activities becomes more important as the number of controllers
increase. Duplicating some work between two controllers is probably not
important, but involving a hundred or two hundred controllers in a replication mechanism that generates “n” unnecessary activities per replication partner is a recipe for disaster.
Windows uses two tables to control propagation—the high-watermark
vector table and the up-to-date vector table (also sometimes called the state
vector table), and maintains the two tables on every domain controller. The
contents of the tables represent the current state of replication as known to
an individual domain controller. The Active Directory increments the controller USN as it actions each change, no matter whether it is to add, update,
or delete an object. It then stores the USN with the object and any updated
attributes. Increments also happen for unsuccessful operations, such as the
failure to create a user account.
The high-watermark vector table tracks the highest known USN from
each replication partner. This information allows a domain controller to
know whether it needs to request additional information from a replication
partner to backfill data that may be missing, perhaps because of a failure to
replicate properly in the past. The Active Directory uses the high-watermark
vector table to detect recent changes on a replication partner. If a domain
controller has not received a recent update from a replication partner, it
broadcasts its highest known USN to the partner. The receiving DC can then
verify this data against its own high-watermark vector table to discover
whether any outstanding replication exists. If this is true, the domain controller can then request the information from a replication partner.
Knowing the USN from a replication partner also allows a controller to
request precisely the data required to update its own directory without having
to request “all changes.” For example, if the highest known USN on controller
DC20 is 17754, and a replication notification arrives from DC20 saying that
its current USN is 17794, then the receiving domain controller knows that it
still has to apply 40 updates to its copy of the directory and is able to issue a
request to DC20 to provide the missing information.
The up-to-date vector table maintains a list of all replication partners
and the highest originating write on each. When Windows has fully synchronized all of the controllers in a domain, the up-to-date table is the same
everywhere. Each domain controller sends its up-to-date vector table to its
replication partners as part of the replication cycle. The replication partner
matches the USN in the up-to-date vector table with its high-watermark vector table to identify any missing data. If the replication partner finds that
some replication operations are outstanding, it will request updates. OtherChapter 2
2.2 Active Directory replication
wise, if the replication partner has already received the data—through replication with another domain controller—then it makes no further attempt to
replicate because of propagation dampening.
Within a small domain, these tables are not very important. However,
their importance grows in line with the number of domain controllers. Each
domain controller is likely to have a set of different replication partners, so
replication data can flow along many different paths. If no mechanisms were
in place to stop unnecessary replication, then a domain controller might process updates multiple times after it contacts different replication partners, all
of whom want to provide the domain controller with the same information.
Changes in Active Directory replication in
Windows 2003
Most Exchange servers run on Windows 2003 today and you have to
upgrade to Windows 2003 to deploy Exchange 2007. While it is possible to
run a very effective Active Directory infrastructure on Windows 2000, Windows 2003 introduces many important Active Directory upgrades, enhancements, and fixes. The following are the most critical changes in terms of
operation and deployment:
You can now promote a Windows 2003 server to become a domain
controller using a copy of the Active Directory on removable media.
This makes it much easier to deploy controllers in a large network
because it removes the replication load necessary for the Active Directory to populate a database on a new DC. For example, within HP’s
forest, network based replication using Windows 2000 used to take
between three and five days to complete to promote a new controller.
Using the load from media feature, a new controller is online within
an hour.
The Windows 2003 Active Directory is better at cleaning up “lingering” objects (ones that do not disappear completely in all controllers
after the replication process completes) that remain in the database
after deletion.
Intersite replication is more efficient and consumes less bandwidth.
The overall size of the Active Directory database is usually smaller
because the database engine is more efficient and uses mechanisms
like single-instance storage of security descriptors. For example, when
HP moved its Windows forest into native Windows 2003 mode, the
DIT file shrank from 12GB to 7.5GB.
2.3 Exchange’s Active Directory Topology service
Global Catalog servers do not commence a full synchronization when
you add new attributes to the partial attribute set defined in the
Active Directory schema, such as those added by the schema upgrade
required to support Exchange 2007. Large forests feel the effect of
this change most because of the number of controllers that participate in replication. Where changes to the partial attribute set in Windows 2000 might create a replication storm over five days before all
of the Global Catalogs have a fully populated database, now you can
assume that replication completes in under a day, assuming no network outages.
Last logon timestamps are replicated.
ITSG makes better decisions about the site connections that it
The Active Directory supports per-value replication for multi-value
attributes. This is very important for groups (and Exchange makes
extensive use of distribution groups), which hold group membership
in a single multi-value attribute. Therefore, any change to group
membership used to force the Active Directory to replicate the complete membership now just replicates the changed value.
While the Active Directory is a critical component of successful Windows infrastructures, it is not the only component to manage. File Replication Services (FRS), which replicates SYSVOL and policies, is also a
dependency for many applications. For example, if FRS replication is not
working properly, then the changes to server security policies applied by the
Exchange installation program will not replicate from the server that you perform the installation on to other servers in the domain. One side effect of
this failure is that you may be able to start the set of Exchange services on
other servers, but you will not be able to mount the Store databases because
the services will not possess the necessary privileges!
Exchange’s Active Directory Topology service
Exchange 2007 uses an Active Directory API to access information that is
stored in Active Directory. The Active Directory Topology service runs on all
Exchange 2007 servers. This service reads information from all
Active Directory partitions. The data that is retrieved is cached and is used by
Exchange 2007 servers to discover the Active Directory site location of all
Exchange services in the organization. The Active Directory Topology service
checks the site membership attribute on the Exchange server object when the
server starts. If the site attribute has to be updated, the Active Directory
Topology service stamps the attribute with the new value. The Active DirecChapter 2
2.3 Exchange’s Active Directory Topology service
tory Topology service verifies the site attribute value every 15 minutes and
updates the value if site membership has changed. The Active Directory
Topology service uses the Net Logon service to obtain current site membership. The Net Logon service updates site membership every 5 minutes. This
means that up to a 20-minute latency period may pass between the time that
site membership changes and the new value is stamped on the site attribute.
DSAccess (or ADAccess)
DSAccess was a pretty important component for Exchange 2000 and 2003
because it manages the interaction between Exchange and the Active Directory. It also managed a cache of recipients that were read from the Active
Directory as the transport engine resolved message headers to decide how
best to route messages. Over time, Exchange built up a cache of recipient
information that the transport engine could access from memory without
having to make an expensive call to a Global Catalog server. Renamed as
ADAccess, DSAccess is still used in Exchange 2007, but its recipient cache
is no more. Instead, four services load ADAccess to gain a consistent view of
the current topology data from the Active Directory and to make sure that a
single consistent view of the topology is shared by all services running on a
Active Directory Topology
Information Store
System Attendant
Client Access server (through the WWW Publishing service)
The role of ADAccess is to:
Perform suitability testing to determine that a selected domain controller and Global Catalog server can function correctly before
ADAccess uses them (see below).
Manage load balancing of LDAP requests within the local Active
Directory site if more than ten domain controllers and Global Catalog servers are present.
Perform graceful failovers to a new domain controller or Global Catalog server to minimize the impact on clients and other Exchange
components such as the transport service. ADAccess uses information
about Windows site links and connection costs to determine the
most appropriate controller to fail over to.
2.3 Exchange’s Active Directory Topology service
In failover situations, ADAccess monitors for the local domain controller and Global Catalog to detect when they are available again. If
this happens within 5 minutes, ADAccess automatically reconnects
to that controller.
Diagnostic logging to help troubleshoot problems.
Figure 2.8
ADAccess discovers
Active Directory
Every 15 minutes, ADAccess scans the Active Directory to build a view
of the current topology. ADAccess divides the servers that it discovers into
those that are in the same Active Directory site as the Exchange server and
those that are outside the site. Figure 2.8 illustrates the data returned by
The name of the server: A list of the FQDNs of the controllers that
are available inside and outside the Exchange server’s Active Directory
site. For example,
The roles that this server can fulfill: D indicates that the server is a
domain controller, G means Global Catalog, C means that the server
Chapter 2
2.3 Exchange’s Active Directory Topology service
is acting as the Configuration DC. In this case, CDG means that the
selected server is able to act as the Configuration domain controller, a
regular domain controller, and as a Global Catalog server.
Whether or not the server is reachable: This value is a bit mask where
“1” means that the server is reachable via LDAP as a Global Catalog
through port 3268, “2” means that it is reachable as a domain controller through port 389, and “4” means that it can act as a configuration domain controller (through port 389 also). A value of “7”
indicates that the server is reachable through all ports and can act in
any role. A value of “0” indicates that the server is completely
unreachable and therefore ADAccess cannot use it.
Whether or not the server is synchronized: The same bit mask values
are used, so “1” means that the Global Catalog is synchronized, “2”
that the domain controller is synchronized, and “4” that the configuration domain controller is synchronized. A value of “7” means that
the server is completely synchronized.
Whether the server is capable of acting as a Global Catalog: “1” indicates that it is.
Whether or not the server is the PDC emulator for a domain: “0”
indicates that it is not.
Whether the server passes the SACL (System Access Control List)
test: This test determines whether the server resides inside a domain
that you have prepared to host Exchange servers. A value of “1” indicates that the SACL test passed.
Whether the server hosts critical data: A value of “1” indicates that
the Microsoft Exchange container exists in the configuration naming
context on this domain controller. The Exchange container stores
critical data such as server names and roles, routing information, and
so on that must be available for the transport service to work. ADAccess only selects a domain controller if it hosts this container.
The set of services that rely on ADAccess are used throughout Exchange
2007, so ADAccess is a very important component, but a curious feature of
its introduction is the elimination of Exchange’s ability to use cached recipient data. Instead of using a centralized cache for data fetched from the Active
Directory, Microsoft allows each service running inside Exchange 2007 to
decide how long they want to keep data for as long as they need it and no
longer. Microsoft believes that this approach results in more predictable and
better performance, keeps code simpler, and that it ensures that the data used
is always up to date (in other words, there is no danger that it has aged in the
cache). In the case of the Exchange 2007 transport engine, it only retains
2.3 Exchange’s Active Directory Topology service
recipient information for as long as it takes to route a message, which is very
different to the behavior of Exchange 2003. If services need to keep Active
Directory data for extended periods, they can subscribe for notifications
from the Active Directory so that they know when to refresh. In addition,
they can perform periodic checks to ensure that the data that they use is consistent with the data in the Active Directory.
Perhaps Microsoft believes that the migration of Active Directory to
64-bit Windows servers will remove the general need for Exchange to cache
information because, assuming that enough memory is available to the
server, it is far more possible to cache the complete Active Directory database in memory on a 64-bit Global Catalog than it is on a 32-bit Global
Catalog. Thus, even if Exchange needs to make a massive amount of directory requests to resolve recipient addresses, the requests will flow across a
short network hop to a Global Catalog that can respond immediately
because the necessary data is in its cache. It’s entirely possible that this is
true, but it does underline the need to deploy sufficient high-performance
Global Catalog servers in any Active Directory site that hosts several hub
transport servers. How many is sufficient? Only time, experience, planning,
and some realistic data gained through testing will answer that question for
your company, so be prepared to do some work to find out.
How many Global Catalog servers do I need?
The best advice that I ever heard was to avoid skimping on Global Catalog
servers for any medium- to large-scale Exchange deployment. This is an attitude that perhaps runs contrary to the approach taken by Windows administrators, who may not understand the vital role that Global Catalogs play in
Exchange, especially in GAL interaction (for example, to update a distribution group or lookup the properties of a recipient) performed by clients and
address validation performed by Exchange to route messages efficiently. There
are many instances when deployments start with a relatively small number of
Global Catalogs and everything proceeds smoothly until Exchange servers
begin to operate. Up to this point, the demand on the Global Catalogs is relatively small and is generated largely by client logins and replication. Immediately you introduce an Exchange server and start to use it to send and receive
mail, the load on the Global Catalogs increases dramatically.
The general rule of thumb to determine the number of Global Catalogs
was originally one Global Catalog per four Exchange servers. As experience
with Exchange deployments grew, the rule changed to one Global Catalog
CPU per four Exchange CPUs. In other words, if you have two dual-CPU
Exchange servers, you need a Global Catalog. If you have six dual-CPU
Exchange servers, you need three Global Catalog servers, and so on. The
Global Catalog servers do not have to be heavy-duty systems and it is easy to
build relatively inexpensive servers to carry the load—nd even deploy some
Chapter 2
2.3 Exchange’s Active Directory Topology service
as virtualized servers running on much bigger systems. The important thing
is to be resilient to failure as nothing stops Exchange working as quickly as a
Global Catalog outage, and to make sure that there is enough capacity to
handle the peak load generated by Windows, Exchange, and any other application that uses a Global Catalog.
Another issue to consider is the placement of Global Catalogs alongside
Exchange servers. The rule of thumb here is simple too. If you have an
Exchange server, you need a Global Catalog on the same WAN segment. In
other words, you must ensure that any network outage has a minimal effect
on Exchange. Satisfying the rule may not always be possible, especially if you
feel that you have to place an Exchange server in a branch office (to speed up
email for local users) and cannot afford to add a Global Catalog. In this case,
you may want to channel Active Directory lookups to the Global Catalog
across a WAN link. However, while it seems an attractive option to use the
network in this way, it will inevitably lead to outages and user dissatisfaction
and the better options are to either deploy a local Global Catalog or centralize Exchange and Global Catalogs together in one location and use clients
such as Outlook in cached Exchange mode to connect over the WAN.
Where are my Global Catalogs?
Because you need to have at least one Global Catalog server in each site that
supports an Exchange 2007 server, Global Catalog servers are a critical component of your Exchange 2007 deployment. As you roll out servers to support Exchange 2007, you may want to investigate what Global Catalogs are
deployed within your infrastructure. Some of these computers may not be
required as you can normally consolidate the workload done by several 32-bit
Global Catalog servers into a smaller number of 64-bit servers. This is especially valuable when your Active Directory includes more objects than can be
cached in the memory available to 32-bit Windows servers. It’s difficult to be
precise as to exactly how many objects this represents because the average size
of an Active Directory object depends on the object type (user objects are
bigger than contacts or groups, for instance) and how many of their properties are populated. A good rule of thumb is deploy 64-bit Global Catalogs
when the forest contains more than 100,000 objects. Of course, over time,
the natural server replacement cycle will replace the 32-bit version of Windows with 64-bit servers and the process will complete naturally, but it’s certainly a good idea to move faster when you have large directories to manage.
As an example, tests inside HP’s production Windows environment
showed that a single 64-bit Global Catalog server (dual CPU, either Opteron
or IA64, equipped with 14GB of memory) can handle the load previously
processed by eleven 32-bit servers for 20,000 users in a single large campus
site. In HP’s case, the size of the DIT (AD database file) varies between 10
and 11GB to hold approximately 350,000 objects, depending on the
2.3 Exchange’s Active Directory Topology service
amount of white space in the database, so it is possible to cache the entire
database if the server is equipped with enough physical memory. Normal
Exchange messaging activity in a reasonably busy site will populate the cache
on a Global Catalog server quickly and once the cache is populated, further
access is very fast indeed. Depending on the size of your Exchange and Active
Directory infrastructure, your mileage will vary, so do not take these figures
as anything more than a guideline to what might be possible. It’s also obviously not a good idea to depend on a single large Global Catalog to handle
the workload generated by a set of Exchange servers as a failure on the Global
Catalog will affect thousands of users, so in a production environment, you’d
have at least two or three Global Catalogs for a site that supports 20,000
users, even if one can handle the workload.
Before you can decide what Global Catalogs to refresh or consolidate,
you need to know where they are. There are at least three ways of discovering
what Global Catalog servers exist inside the forest. First, you can use the
NSLOOKUP utility to check what servers have registered themselves as
offering a Global Catalog service:
The difficulty here is that servers might have registered themselves as
Global Catalogs at some point in the past and subsequently ceased this role
but never removed their information from Active Directory. You can go to
every Domain Controller and query it to see if its isGlobalCatalogReady
attribute is true, but this can be quite a lot of work.
Microsoft also provides the REPLMON utility as part of the Windows
support tools. You can add a server to its list and then right click on the
server and select the “Show Global Catalog Servers in the Enterprise” option
to view the data. Other products such as the HP OpenView Smart plug-in
for Active Directory are able to graphically chart the replication topology and
show you what’s going on inside your network. These approaches work
equally well with both before and after you deploy Exchange 2007. Once
you start to deploy Exchange 2007, you’ll be able to use PowerShell, the new
Windows scripting language. There is much more coverage of PowerShell
and how you can use it to manage Exchange 2007 from Chapter 4 onwards.
For now, we should recognize that PowerShell has a huge influence over how
you will manage Windows, Exchange, and other components going forward,
so it’s not surprising to discover that you can use PowerShell to interrogate
the Active Directory and find out what Global Catalog servers exist.
After you start up PowerShell, we start off by populating a variable with
details of the forest:
Chapter 2
2.4 Recovering deleted Active Directory accounts
$Forest =
You can view details of the forest by typing the name of the variable and
this reveals some interesting information that you can capture by piping the
output to a text file:
$Forest > c:\temp\Forest.txt
You’ll see something like this:
: {San Francisco, Dublin, Atlanta,
Amsterdam, Bellevue, Warrington, Adelaide, Paris, London,
Chicago, Zurich}
: {}
: {,,
ApplicationPartitions : {DC=DomainDnsZones,DC=xyz,DC=com,
: Windows2003Forest
: CN=Schema,CN=Configuration,DC=xyz,
This list is easy to understand in a small organization, but it can be difficult to interpret in more complicated environments, so we can break out the
Global Catalogs and list them as follows:
$GClist = $Forest.FindAllGlobalCatalogs()
ForEach-Object ($GC in $GClist) { $GC.Name }
Recovering deleted Active Directory accounts
Even the best administrator can make a mistake and if you delete an AD
account in error, you will find that it is difficult to restore the account
quickly and simply. Sometimes, the quickest recovery method is to recreate
the account from scratch, but this means that the account loses the access it
2.4 Recovering deleted Active Directory accounts
has to different resources because the SID for the recreated account is different from the original. Another problem is that the new account does not
have membership of the different groups that the original group belonged to,
something that is easier to cope with albeit only if you know all the groups to
add the account to.
The normal method used to recover a deleted account from a backup
tape is to perform an authoritative restore on a domain controller and then
use the NTDSUTIL program to recover the account. You can only do this if
you know the full DN of the account, as you need this information to
retrieve the data from the backup. However, there are other difficulties to
overcome even after you recover the deleted account as its group membership
may not be intact and therefore the account may lose access to some
In large organizations, where the potential for accidental deletions is
higher, you may want to consider setting up a special Windows site that
contains a domain controller. The delayed site replicates with the rest of the
domain once or twice a week so that its copy of the AD database is complete
after replication finishes. If you catch the error before replication to the
delayed site occurs, you know that a copy of the deleted account still exists
in the copy of the AD in the delayed site, so you do not have to restore any
backup tapes. Instead, you can log onto the domain controller in the
delayed site to find the deleted object (and establish its DN) and note its
group membership, then boot the domain controller into Directory Services
Restore Mode and use the NTDSUTIL program to run an authoritative
restore for the object. The effect of this operation is to increase the USN for
the restored object by 100,000, which means that the update to the object
in the delayed site will win any replication conflict caused by updates from
other sites. After the restore, you reboot the domain controller as normal
and then force replication to replicate the data back into the rest of the
domain. After replication is complete, you can reestablish the account’s
membership of the groups that it lost because of the accidental deletion.
A variation of the delayed site technique is to deploy domain controllers
in two different sites, each with different replication schedules. For example,
the first site might replicate every Tuesday, while the second replicates every
Friday. With this setup, even if you do not catch the accidental deletion
quickly and replication occurs to one site, you have a backstop to rescue you.
However, there is obvious additional cost to maintain the second site that
rules this variation out for all but the largest Windows deployments.
Even in its simplest form, the delayed site technique is not for every
organization. If you have a small company, it is unlikely that you will want to
maintain the overhead to create a delayed site as you probably will not realize
much of a benefit. Virtualized servers can help because it is possible to host
multiple domain controllers for different sites on one large physical server.
Chapter 2
2.5 Exchange and the Active Directory schema
The bottom line is that in return for some extra cost, the technique may save
you hours of work especially in large multi-site organizations.
Exchange and the Active Directory schema
Schemas define the structure of a database, including the object classes and
the attributes held for each object. The AD schema is extensible by applications or enterprises to allow it to hold additional information. For example,
you could extend the directory to add a new “Birthday” attribute for user
objects, and then write an AD-enabled application that ran when a user
logged on to check whether it was their birthday, generating an appropriate
message on the day in question. First, generation Exchange includes 15 “custom” attributes for mailboxes that you can use for similar purposes, but you
cannot extend the schema of the Exchange Directory.
Exchange was the first application to extend the AD schema by adding
new attributes that can then be associated with recipients (users, contacts,
and groups) as well as configuration information about Exchange such as
administrative and routing groups. For example, Exchange extends the user
object with storage attributes to allow users to be associated with the Store
where their mailbox is located as well as any quotas placed on the mailbox.
Exchange also extends the schema to accommodate the requirements of its
own subsystems, such as the changes to hold information about Exchange
ActiveSync and the mobility policies that Exchange can impose on certain
mobile devices. Schema extensions for Exchange must be in place (and replicated throughout the forest) before you can install Exchange servers into an
AD forest, so this is a step that you have to include in your deployment plan.
If you have Exchange 2003 servers already in the forest, there are just a few
extensions to install and replicate before you can install Exchange 2007, but
there are well over 2,000 changes necessary to prepare a brand new AD forest
for Exchange 2007.
Updating the schema with an installation
The easiest way to update the schema is to just go ahead with the first
Exchange server installation in a forest, but you can update the schema
beforehand by utilizing two options that are included in the Exchange installation procedure. You execute SETUP with these options from an administrator account that has full permission to modify the Active Directory. Once
the Active Directory is prepared, you can perform subsequent installations of
Exchange using accounts that have local administrative access, but are not
privileged to change forest-wide settings such as updating the schema or adding the Exchange container to the configuration-naming context.
The options are:
2.5 Exchange and the Active Directory schema
Setup /PrepareAD: This option runs in the root domain of the forest
(or the domain that hosts the schema master, which is normally the
root domain). /PrepareAD performs the set of changes to the schema,
instantiates the Exchange organization, adds the Exchange container
to the configuration naming context, and creates the security groups
used by Exchange. You cannot execute this command unless you are
able to log on with Enterprise Admin and Schema Admin privileges.
Setup /DomainPrep: You run this option in every domain where an
Exchange 2007 server is located. The option performs tasks such as
creating the global groups used for Exchange administration. You
must be a domain administrator to be able to run this option.
If you are installing Exchange 2007 into an existing Exchange 2000/
2003 organization, you have to run Setup /PrepareLegacyExchangePermissions to ensure that the correct security context is created to
support a mixture of server versions. In particular, this process allows
the Exchange Enterprise Server and Exchange Domain Servers security groups access to the new property sets that Exchange creates in
the Active Directory and ensures that the Recipient Update Service
can continue to stamp attributes on new objects after Exchange 2007
joins the organization. Note that if you add a new domain to the forest and install Exchange 2003 servers into the domain, you must run
Setup /PrepareLegacyExchangePermissions in this domain to allow
the Recipient Update Service to work.
If you install Exchange 2007 in a domain in another forest to the forest that holds your user accounts, you have to run Setup /ForeignForestFQDN in the forest that holds the user accounts to allow the
accounts to access their mailboxes in the other forest.
The /PrepareAD option is a useful thing to execute if you want to replicate schema updates throughout the forest before you begin server installations. The sheer number of changes applied to the schema (Exchange adds
57 attributes to the Windows Partial Attribute Set that is replicated to all
Global Catalogs and creates over 500 new object identifiers, including 13
indexed attributes) is a good reason to perform the installation (or schema
update) of the first Exchange server close (at least in network terms) to the
schema master as this will speed up processing of the schema changes. Windows makes schema changes to the configuration container of the Active
Directory on the target controller and then replicates them to the other controllers throughout the forest.
Figure 2.9 shows the ADSIEDIT utility examining the properties of the
container used to hold configuration details for Exchange. Every object in
Chapter 2
2.5 Exchange and the Active Directory schema
the Active Directory has a distinguished name, so the container used to hold
Exchange data is named <domain-name>/Configuration/Services/Microsoft
Exchange and it holds a number of other containers to store details of entities
such as policies, address lists, connectors, and so on.
Figure 2.9
The Exchange
Changing the schema
The provision of an extendible schema is a major feature of the Active Directory, but whereas it is great to be able to customize the Active Directory, the
feature introduces some new issues for consideration. Updating the schema
does not impose a performance penalty. The new attributes occupy no space
in the database unless you populate the attributes with data. Changing the
schema is something that you must treat very seriously and only perform
when a change is justified and you fully understand the ramifications. For
this reason, you should agree any change up front with all of the domain
administrators. Ideally, someone within the organization should take responsibility for arbitration of schema changes and anyone who wishes to make a
change should first consult that person. Schema anarchy is not a pretty sight!
For this reason, some companies keep the membership of the Schema
Admins group empty until they know they need to make a change, whereupon they add the necessary user to the group until after the change when
they revoke the membership.
It is also a good idea to apply as many schema changes as possible at the
start of an implementation as this means that every new domain controller
will inherit the fully updated schema as part of the DCPROMO procedure.
The alternative is to make schema changes as the need arises, but this means
that you have to let the Active Directory replicate each set of changes
2.5 Exchange and the Active Directory schema
throughout the forest before you can proceed to deploy applications that
depend on the schema update.
Attributes can be single-valued (like your home telephone number) or
multi-valued (like the membership of a group). Before you change the
schema to add a new attribute, you need the following information:
The name of the new attribute and its purpose: In directory terminology, this is the common name. You can provide a separate description, although in many cases for the standard set of attributes used by
Windows or Exchange, the description is very similar to the name.
Because the roots of Active Directory are in X.500, each attribute and
object has a unique X.500 object identifier, or OID. A national registration authority, such as the American National Standards Institute
(ANSI)1, issues OIDs. You can make up the value for an OID, and as
long as the value does not clash with the OID of another attribute,
then you will not have a problem. However, if an application comes
along in the future and attempts to add an attribute with an existing
OID, then you will have a problem and the application will not be
able to add the new attribute to the schema. One method that is
sometimes used is to take the base OID for the DSA provider and
append your company’s international telephone number plus some
sequence number to it to create the new OID. This method usually
guarantees uniqueness.
The type or syntax of the attribute and its maximum and minimum
range: For example, an attribute held as string values will be stored as
Unicode strings and can range in size from 1 to whatever number of
bytes is required to hold the maximum string length.
Whether or not the Active Directory should replicate the attribute to
Global Catalog servers: Clearly, the more attributes that are replicated, the more data is transmitted across the network. Some
attributes are not required enterprise-wide and can be restricted to an
object’s home domain. The Active Directory has to replicate others,
like the attribute that holds details of an Exchange user’s home mailbox server, throughout the forest to enable specific functionality, like
message routing.
Whether the Active Directory indexes the attribute and includes it in
the Ambiguous Name Resolution (ANR) process. Exchange has supported ANR since Exchange 4.0.
Chapter 2
2.5 Exchange and the Active Directory schema
Before we look at the detail of how to change the schema, we need to
update the system registry on the server where we want to apply the change.
Ideally, you should apply all updates to the schema at the schema master. If
not, the server that you use to make the change needs a fast connection to the
schema master to make the change. Applications like Exchange often include
commands to update the schema in their installation procedures. Exchange
2007 updates the Active Directory and extends the schema when you run the
Setup program with the /PrepareAD switch. After the schema is updated,
you can browse through it, as shown in Figure 2.10, to view details of the
additional attributes used for Exchange. Figure 2.10 shows the schema snapin loaded in an MMC console, which is how we are able to view details of the
schema. You may find that MMC doesn’t list the schema snap-in when you
go to load it. This is probably because the schema snap-in is not registered for
use with MMC. To solve the problem, issue this command at the command
Figure 2.10
Viewing Exchange
attributes in the
AD schema
The schema is loaded into memory on every domain controller in an
area called the schema cache. To ensure that clients access changes promptly,
the Active Directory reloads the schema cache from disk every five minutes.
If you change the schema, you can force the Active Directory to reload the
cache immediately from an option in the Schema snap-in. This ensures that
all the changes are active before you attempt to use them.
2.5 Exchange and the Active Directory schema
Active Directory custom attributes
for Exchange
Exchange has always provided a set of custom attributes that you can use to
store information about users in the directory. It is highly unlikely that the
designer of any directory will ever incorporate a set of attributes that satisfies
every possible customer, so the Active Directory includes 15 custom
attributes that you can populate according to your needs. Prior to Exchange
2007, you fill in the attributes by first selecting the advanced view for AD
Users and Computers and then viewing the Exchange Advanced property
page. In Exchange 2007, you select the object using EMC, edit the properties, and click on the “Custom Properties” button (Figure 2.11).
Before anyone begins to add values to the custom property set, you
should clearly define what these values will hold and how to populate them.
Otherwise, one administrator may think that custom attribute 1 holds an
employee badge number whereas everyone else uses custom attribute 2 for
this purpose. Other common uses for these attributes include cost center
identifiers, whether a user is a permanent or temporary employee, social
security numbers (some countries bar you by law from holding personal
information like this in a general-purpose directory), job codes, and so on.
Figure 2.11
Setting values for
custom properties
Chapter 2
2.5 Exchange and the Active Directory schema
Updating the schema to allow Ambiguous
Name Resolution
Ambiguous Name Resolution (sometimes called Automatic Name Resolution) is the process used by Exchange to search the Active Directory to find
addresses that match information provided by users. In other words,
Exchange searches the Active Directory against a number of different fields
to find any or all matching addresses. If Exchange finds a number of matching addresses we have an ambiguous name, and Exchange prompts the client
to present the addresses to the user in a dialog for them to make the final
Exchange checks attributes such as the first name, surname, display
name, and mailbox alias during the ANR process. ANR is invoked whenever
a user presses the CTRL/K sequence, or when an unresolved address is
detected in the TO:, CC:, or BCC: fields of a message header. Outlook and
Outlook Web Access clients support ANR. In Figure 2.12, you can see how
Outlook Web Access supports ANR in Exchange 2007 (left) and Exchange
2003 (right). Outlook Express also performs ANR if you configure the client
to check names against the Active Directory before it sends messages.
Figure 2.12
Ambiguous names
detected by
Outlook Web Access
It may be advantageous to alter the schema to include a customized field
in the ANR process. You could populate one of the extension properties with
details such as a department code and add this attribute to the ANR process,
if this makes sense within the company. Remember that changes to the
schema are forest-wide and you cannot reverse a change once made, so be
sure to consider all aspects of making the change before implementation.
Three different properties govern whether Exchange can use an attribute
for ANR.
Figure 2.13 illustrates these properties as set for one of the custom
extension properties. You might wonder why I say that three properties govern ANR when only one is marked as such. The reason is simple. The
Ambiguous Name Resolution property determines whether the ANR process
uses the attribute, but it is not very intelligent to mark an attribute for ANR
2.5 Exchange and the Active Directory schema
Figure 2.13
Where to enable
ANR resolution for
a selected attribute
if it is not indexed, and only slightly more intelligent to keep an attribute
used by ANR out of the Global Catalog. After all, if the Active Directory
does not index the attribute, any attempt to check a name against the
attribute will be very slow and frustrate users, and if the attribute is restricted
to one domain, its value is useless anywhere else in the forest. For best effect,
check all the properties.
Exchange-specific permissions
Along with an upgraded schema, Exchange also adds a set of permissions to
the Active Directory to allow administrators to perform operations such as
managing databases. The permissions, which Microsoft refers to as “extended
rights,” exist in the configuration container under “Extended-Rights,” as
shown in Figure 2.14. All of the rights that Exchange uses have an “ms-Exch”
prefix, so they are easy to find.
Property sets are a special type of extended rights. A property set is a
group of object properties that you can use to define access control on an
object by manipulating multiple permissions in a single operation. For example, if you enable the Exchange Information (a property set) permission on a
user object, you actually enable users to see the values in the object’s email
addresses, manager, and common name attributes. Exchange uses property
sets extensively to simplify administrative operations. If it did not, we would
Chapter 2
2.5 Exchange and the Active Directory schema
Figure 2.14
Extended AD
permissions used by
be constantly updating permissions on Active Directory objects to try to
assign the necessary rights to get work done.
Exchange property sets
Chapter 3 reviews how Exchange and the Windows administrators perform
different tasks in an Exchange 2000/2003 environment. The split administrative model is not natural in that Microsoft didn’t build a comprehensive
design for administrative delegation that spans both Exchange and Windows.
Instead of being able to assign some out-of-the-box roles such as “Administrator of a Branch Office” that incorporates the necessary permissions to perform all the tasks that you’d expect someone in this role to have to be able to
do, it’s a case of creating a roll-your-own solution by assigning different permissions over different objects to different people on a very individual basis.
You can argue that this is the right approach because every company is different and all have their own way of managing servers. While this assertion is
correct, it would still be nice to have a flexible administrative framework to
work within.
The problem in solving the problem through selective allocation of
permissions to individuals is that you run the danger of bloat in access control lists. In other words, to assign administrators a fine level of control
over the attributes of mailboxes and other recipient types, you have to
assign permissions over a large number of individual attributes. The result
is that the access control lists increase in size and become difficult to man-
2.5 Exchange and the Active Directory schema
age. The Active Directory database file, NTDS.DIT, also grows in size to
accommodate the expanded Access Control Lists. Exchange 2007 solves
this problem by defining property sets that cover the vast majority of
attributes associated with mail recipients. A property set is a grouping of
attributes that can be controlled by setting a single Access Control Entry
(ACE) on the set rather than setting individual ACEs on each attribute. An
attribute can only be a member of a single property set. For example, the
Personal Information property set includes information such as the given
name, company name, telephone number, and department name, all of
which are attributes of user objects. Exchange recipient administrators
need to be able to work with these attributes, so by grouping all of the
attributes that a recipient administrator needs to work with into a property
set, we can set the necessary ACE to allow recipient administrators to work
with those attributes in a single operation.
Exchange 2003 did not use specific property sets. Instead, the Exchange
2003 installation added some Exchange-specific attributes to the standard
Active Directory Personal Information and Public Information property sets.
On the surface, this seems like a good idea, but the problem is that Exchange
has no control over these property sets, which may be amended by a future
version of Active Directory that generates a subsequent knock-on impact for
Exchange. Exchange 2007 resolves the problem by creating two new property sets, Exchange Information and Exchange Personal Information, that are
installed into the forest when you run Setup /PrepareAD. These property sets
contain attributes that are defined by the Exchange schema extensions to
Active Directory and are under the control of Exchange. These property sets
are created when you extend the Active Directory schema prior to installing
the first Exchange 2007 server in an organization. Attributes that were added
by Exchange 2003 to the standard Active Directory property sets are
removed from these property sets and incorporated into the new Exchange
property sets. Examples of the attributes that are in the Exchange Information property set are:
While examples of the attributes in the Exchange Personal Information property set include:
Chapter 2
2.6 Longhorn and Exchange 2007
The last two of these attributes are interesting because they allow
Exchange to capture and store details of users’ safe recipients and safe sender
data in their accounts in the Active Directory. Subsequently, Exchange can
synchronize this data to the ADAM instance used by Edge servers so that
they can integrate user data into the tests that the Edge servers use to detect
and suppress spam.
Creating a new Recipient Administrator role provides a logical solution
to the problem, and that is what you get in Exchange 2007, where you can
grant a user the Exchange Recipient Administrator role through EMC or by
using the Add-ExchangeAdministrator command. Having Microsoft determine what permissions are required to perform the recipient management
task is a much more secure solution than each organization figuring out what
ACLs have to be set on Active Directory objects or permissions held by user
accounts. It is inevitable that mistakes will occur along the way when multiple attempts are made to solve the problem and mistakes lead to loopholes in
security, which can lead to further issues with data integrity and so on.
Longhorn and Exchange 2007
Longhorn is the code name for the major release of the Windows server operating system that Microsoft anticipates shipping in late 2007 or early 2008.
Dates are always difficult to predict when it comes to the availability of a new
version of an operating system as much depends on the quality of the software and the kind of bugs that the developers encounter as they drive to meet
the ship date.
Because Longhorn was still in the beta development phase when
Microsoft shipped Exchange 2007, it is logical that you cannot deploy
Exchange 2007 on servers that run Longhorn, nor can you deploy Exchange
2007 servers inside an Active Directory site that supports Longhorn domain
controllers or Global Catalogs. After all, the Exchange developers can test
Exchange 2007 on the latest version of Longhorn that is available to them
when they put the final changes on Exchange 2007, but that is no guarantee
that code will run on the final version. There is no doubt that a future version of Exchange 2007 (probably a full-service pack) will support Longhorn,
so you can therefore factor Longhorn into your long-term plans for your
Windows infrastructure.
Even though you cannot use Longhorn as a platform to run the first
release of Exchange 2007 and have to wait until Exchange 2007 SP1 for
2.7 The very important LegacyExchangeDN attribute
Microsoft to support the two software releases working together, you can use
Longhorn to solve specific issues. For example, Longhorn supports a readonly version of a domain controller that is well suited for deployment within
a DMZ when you want to allow applications access to the Active Directory
to retrieve account information, but you don’t want to allow applications to
write into or update the Active Directory in case the DMZ is compromised
by an attacker. Exchange 2007 could use this functionality to support its
Edge server role but as Longhorn is not available, Exchange 2007 uses
ADAM instead.
The very important LegacyExchangeDN attribute
The original Exchange architecture allocated a fixed distinguished name to
every object in the directory and used this value as the primary key for the
object in the directory. Exchange composed distinguished names from the
names of the containers along the path to an object in the directory. Primary
keys are difficult to change and this implementation led to some problems in
terms of flexibility. You could not change the name of an Exchange 5.5 site or
move users between containers because these operations required the object to
be deleted and then recreated with a new distinguished name that reflected the
new location of the object within the directory.
During Exchange’s transition to use the Active Directory, the developers
were careful to avoid the same issue. While the Active Directory still uses distinguished names to reference objects, the distinguished name is no longer
the primary key. Instead, when it creates a new object, the Active Directory
allocates a GUID (Global Unique Identifier) to it. You can move objects
around the directory between containers as much as you want, which results
in multiple changes to their distinguished names, but their GUIDs remain
constant. Unlike previous versions of Exchange, no correlation exists
between a user’s distinguished name and a site or other administrative, security, or routing boundary.
You can view the distinguished name for an object using the ADSIEDIT utility. Select the object and then look at the value of the “distinguished name” attribute. You will see values like this:
CN=Tony Redmond,OU=Exchange Users,DC=xyz,DC=com
There is no correlation between distinguished names and email
addresses. The change in distinguished name format occurs automatically as
accounts are migrated and Exchange hides the change in distinguished name
format from users. However, many different pieces of the messaging puzzle
Chapter 2
2.7 The very important LegacyExchangeDN attribute
store the old Exchange-format distinguished name (such as the headers of old
messages and MAPI profiles) to ensure backwards compatibility.
Because two different formats are in use, some method is needed to
ensure that the older addresses stored within the infrastructure are still valid.
The Active Directory accomplishes this goal by maintaining a separate
attribute for every Exchange object called the LegacyExchangeDN. You can
think of LegacyExchangeDN as the “old” name for an Exchange object;
something to ensure backwards compatibility by always having something
that an older version of Exchange, older clients, or a utility built for an older
version of Exchange can recognize and use. Another way of considering the
attribute is to think of it as the equivalent of a SID for MAPI because all versions of MAPI clients and servers built to date can use the LegacyExchangeDN to locate objects. As an example, here are two LegacyExchangeDN
values for a mailbox created in an Exchange 2007 organization and the special administrative group that Exchange 2007 uses for backwards compatibility with Exchange 2000/2003. The values that you see in legacy
organizations will be slightly different.
/o=xyz/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/
/o=xyz/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)
The Active Directory indexes the LegacyExchangeDN attribute to
enable fast lookup and automatically searches the index if a client attempts to
search the Active Directory using a distinguished name in the old format.
Thus, MAPI profiles continue to work and clients can continue to reply to
messages using old addresses because the Active Directory responds to the
search executed by the transport engine when it attempts to resolve the
It is possible that Exchange will move away from using the LegacyExchangeDN attribute in future versions, but the need to retain backwards
compatibility means that we’ll probably run a new convention (like an
SMTP address) in tandem with the old LegacyExchangeDN addresses for at
least two versions. To do this, Microsoft will have to create parallel code to be
able to handle both formats and they will have to move to a position where a
new version of Exchange cannot support previous versions of Outlook. Even
though LegacyExchangeDN is used throughout Exchange, creating parallel
code is a well-understood engineering technique that Microsoft already uses
in the availability service so that the previous method of free and busy lookups work for older Outlook clients and the new method used by Outlook
2007. However, the big issue is that there are many older Outlook clients
used with Exchange and customer response to a forced upgrades a new ver-
2.8 Brain surgery for the Active Directory: ADSIEDIT
sion of Outlook is unlikely to be positive. The net result is that this a difficult
problem to solve and that the LegacyExchangeDN attribute is likely to be
with us for a while yet.
Brain surgery for the Active Directory: ADSIEDIT
Normally, you access the Active Directory through consoles such as the AD
Users and Computers snap-in. The user interface of MMC and the snap-ins
enforce rules and restrictions that stop administrators from making mistakes
when they work with important data. In most conditions, it is perfectly
acceptable to work with the regular consoles, but sometimes you want or
need to be able to go behind the scenes to interrogate the Active Directory in
order to find out how things really work and that is where ADSIEDIT proves
its value.
ADSIEDIT allows administrators to browse and manipulate raw Active
Directory data, such as the configuration data for an Exchange organization
or a single attribute setting for a storage group. ADSIEDIT is a double-edged
sword: While it allows you to interrogate and view details of any Active
Directory data, it is also very possible to change something best left
untouched. You could do much the same thing with the original Exchange
ADMIN program by running it in raw mode as this exposed the contents of
the Exchange directory that the user interface normally blocked. However,
when you can work with raw data, you can make some raw mistakes, such as
deleting something important that affected the way that Exchange works.
Occasionally, mistakes required a complete restore of the directory to fix the
problem. The same power and potential for problems exists with ADSIEDIT, and an inadvertent slip of the finger can stop Exchange working if
you make a mistake in the wrong place. The best idea when working with
ADSIEDIT is to practice safe directory access by never changing an object
unless you know the exact consequences of the action.
Before you can use ADSIEDIT, you have to install the Windows Support Tools. You can get the Windows Support Tools from the Windows 2003
server CD or by downloading the kit from Microsoft’s Web site. Once
installed, you load the ADSIEDIT snap-in by starting MMC, clicking on the
Add/Remove Snap-in option on the File menu, then clicking on the “Add”
button to view the snap-ins that are available on the server, and then select
the ADSIEDIT snap-in from the list (Figure 2.15). Click Add, Close, and
then OK to make the snap-in active in the console.
After you have loaded ADSIEDIT into MMC, you need to connect to a
naming context before you can work with the Active Directory. Right-click
on the ADSIEDIT root and select the “Connect to” option to force ADSIEDIT to display the Connection Settings dialog shown in Figure 2.16.
Chapter 2
2.8 Brain surgery for the Active Directory: ADSIEDIT
Figure 2.15
Adding the
ADSIEDIT snapin to a console
Figure 2.16
naming context
If you want to work with account and contact information, you connect
to the Domain naming context; if you want to work with configuration
information, such as Exchange organizational data, then you connect to the
configuration naming context. By default, ADSIEDIT attempts to connect
to a Global Catalog in the local Windows site. You can click on the “ADSI
Edit” root and connect to another controller elsewhere in the forest by typing
the full name of the server. Note that the default connection is to the domain
2.8 Brain surgery for the Active Directory: ADSIEDIT
naming context for the selected server and that ADSIEDIT maintains any
existing connections, including to the schema and configuration naming
contexts unless you use the “Remove” option to release the connection. Once
connected, you can browse the contents of each of the naming contexts in
the same way as any other MMC console. Figure 2.17 shows ADSIEDIT
loaded into an MMC console being used to browse user account information
through the domain naming context. Once you have connected to the Active
Directory, you can use ADSIEDIT to browse the contents of each of the
naming contexts in the same way as any other MMC console. Figure 2.17
shows the contents of the domain naming context, so objects like user
accounts are listed. Note the dynamic distribution group object shown here.
Figure 2.17
Accessing Active
Directory data
Administrators most commonly use ADSIEDIT to examine the properties of objects and verify that the right values exist in the various attributes.
There are many and varied reasons why you use ADSIEDIT for this purpose.
Even the Advanced View of AD Users and Computers only reveals a limited
selection of the attributes of an object so you have to use a different tool if
you want to examine the hidden attributes, and ADSIEDIT is very convenient for this purpose. Many of these attributes are very important in directory synchronization projects. For example, if you are using your own
procedures to synchronize the Active Directory with another metadirectory,
you may want to check that the synchronization process is writing values
from the metadirectory into the correct attributes in the Active Directory.
As an example, let us assume that the metadirectory holds information
about business units and we have implemented code to transfer this data to
the extensionAttribute15 attribute in the Active Directory. To check that the
synchronization code works correctly, we use ADSIEDIT to select any user
Chapter 2
2.8 Brain surgery for the Active Directory: ADSIEDIT
object and then view its properties, selecting the appropriate attribute as
shown in Figure 2.18. As you can see, the string “Thinker” exists in
extensionAttribute15. If this is the value in the metadirectory, we know that
the synchronization code works. Of course, if the wrong value is present we
have entirely another problem to solve!
Figure 2.18
to view properties
of a user object
Windows administrators also use ADSIEDIT for many purposes. For
example, assume that a domain controller suffers a catastrophic hardware
failure and you do not want to reinstall the server. Because an administrator
has not run the DCPROMO process to demote the server, it still exists in the
Active Directory as a domain controller with all of the associated roles and
responsibilities such as acting as a replication partner. In this case, you could
use ADSIEDIT to remove all references of the server from the AD. Later on,
if you wanted to reintroduce the controller, you can rebuild the server and
then run DCPROMO to make it a domain controller.
Note that you shouldn’t run DCPROMO on a server that has Exchange
2007 installed because Microsoft doesn’t support this operation. In fact,
Microsoft is pretty emphatic that you shouldn’t run Exchange on a server
that supports the Active Directory. See TechNet for more details.
If you want to get even closer to raw data, the LDP utility is available. While
ADSIEDIT is quite capable of performing brain surgery on the AD, LDP is
able to examine individual nerves. LDP is a good example of a utility that
works brilliantly in the hands of a software engineer or someone who knows
2.8 Brain surgery for the Active Directory: ADSIEDIT
their way around LDAP and needs to retrieve diagnostic information. If you
just want to poke around the innards of the AD for a casual browse, use
ADSIEDIT. LDP is just too dangerous because the GUI does not protect
you, so a simple slip or mistype of one character can have enormous consequences. The big difference between the two utilities is that you can examine
many attributes together for an object with ADSIEDIT whereas you can
look at many together with LDP. In addition, even though the Windows
2003 version of LDP is improved, its user interface is very basic and it is easy
to make a mistake. For example, you have to type in the name of an attribute
when you modify it, whereas ADSIEDIT allows you to select the attribute
through from lists. Knowledge Base article 224543 is a good introduction to
LDP and explains many of its functions.
Figure 2.19
LDP utility
Figure 2.19 shows the LDP utility connected to a DC. In this case, I
began searching in the OU that holds some user accounts, selected one, and
then began to modify an attribute. The Browse menu allows you to add,
delete, or modify attributes for selected objects, but in most cases, you simply
want to interrogate the AD to discover information about objects. You can
see the contents of the OU in the tree view in the left-hand pane and some of
the attributes and their values in the right-hand pane.
When you examine a user object, you can see a large amount of data,
including the links to the distribution groups that the user belongs to and
any organizational data that exists (their manager and direct reports). You
can also look at security descriptors, while the replication option allows you
to see replication metadata. This information is very valuable if you need to
provide it to Microsoft PSS to help solve a complex problem, but it is not
usually very valuable in day-to-day administration. After you have investigated the AD with LDP, you may want to change the values of some
attributes. You can certainly do this with ADSIEDIT, but again you have to
operate on one object at a time.
Chapter 2
2.8 Brain surgery for the Active Directory: ADSIEDIT
LDIFDE is a utility that exports or imports LDAP-compliant data into
the AD. Knowledge Base article 237677 gives you a good overview of the
process, but you will also need to become conversant with LDAP syntax and
the set of attributes supported by the AD to process anything other than simple exports and imports. RFC2254 is the authoritative source for information about LDAP search strings, while you can discover the names of the
attributes that you want to work with by examining the properties of a
selected object with ADSIEDIT or LDP.
LDIFDE works on very basic principles. You create a file that contains
the operations you want to perform against the AD and then feed it to
LDIFDE for execution. In most cases, the easiest way to accomplish the goal
is to export data about the objects you want to process first, use a text editor
to make whatever changes you want, and then run LDIFDE again to import
the file and make the changes.
Active Directory for Exchange
The Active Directory continues to provide the fundamental underpinning
for Exchange 2007, just as it has done since Exchange 2000. Seven years or
so later, we don’t have to experience the same trauma that occurred when
everyone had to upgrade to Windows 2000 and the Active Directory before
they could deploy Exchange 2000, and most Active Directory deployments
are in good shape now. Exchange also depends on the Active Directory for its
permissions model and to provide the link to Edge servers so that anti-spam
protection works smoothly, if you decide to deploy Edge servers for this purpose. For all these reasons and more, the Active Directory still deserves and
requires attention from an Exchange administrator.
The big change for Exchange 2007 is the dependency on Active Directory sites as the basis for the message routing topology. If Active Directory
replication works efficiently today, you should be in good shape to move
forward with Exchange 2007. Moving to the 64-bit version of Windows
may be a more interesting challenge because of the potential that exists to
consolidate domain controllers and Global Catalogs into a smaller set of
physical servers and it will be interesting to see how administrators approach
this work. Leaving Active Directory behind (but always lurking in the background), we move on to discuss the basics of managing Exchange 2007.
The Basics of Managing Exchange 2007
Managing Exchange has often been a challenge for system administrators,
especially in large corporate environments. Because of the huge variety of
companies that use Exchange, Microsoft had to perform a balancing act
when they built the management tools that they provide with Exchange.
They had to create tools that administrators could use in any situation from
the small deployments based on the version of Exchange integrated with
Microsoft Small Business Server to deployments that spanned the world and
hundreds of thousands of users. This is a very difficult problem to crack
because it is not easy to create software that can adapt to a huge variety of different operating circumstances, yet this is what Microsoft attempted to do in
Exchange 2000 and 2003 with the Exchange System Manager (ESM) console. As a system management tool, ESM was okay, but no more. ESM
reflected the view of the world in 1998–99 and was not flexible enough to
deal with the demands of the largest Exchange deployments.
To Microsoft’s credit, they recognized that the development approach
that resulted in ESM had run out of steam and needed to be changed. ESM
represents an inflexible object-centric view of management that limits
administrators to working with objects such as servers, connectors, databases, and mailboxes within the boundaries set by the GUI built by the
Exchange developers. If you want to go outside the GUI, you face the hurdle of incomplete and undocumented programming interfaces that have
changed across different versions of Exchange and that require a good deal
of knowledge before you can attempt even the most fundamental of tasks,
such as retrieving a list of mailboxes from a server. Microsoft’s response in
Exchange 2007 can be broken down into two major pieces—a radically
redesigned management console that looks a lot better and behaves more
intuitively than previous versions and an automation shell. For Exchange
2007, Microsoft created EMS, the Exchange Management Shell, on top of
the standard Windows PowerShell scripting language. The idea is that you
can use EMS to program common management tasks quickly and easily.
The interesting thing is that Microsoft has built the Exchange Management
Console (EMC) from the task-based commands that they expose through
3.1 Exchange Management Console
PowerShell. For the first time, because Microsoft built EMC on the same
code as they use for the PowerShell commands, Exchange administrators are
able to create scripts that automate tasks to meet the demands of their own
operating environment without having to depend on an Exchange developer doing the job for them.
The complete overhaul of the management console takes some time
for even experienced Exchange administrators to become used to. It is not
just the change in the appearance of EMC or the introduction of newly
formalized concepts such as server roles or the appearance of multiple wizards to perform tasks. In addition, you have to become accustomed to the
new terminology that Exchange uses throughout the product allied to the
new focus on programmability. Finally, it is the fact that you can only perform some tasks through PowerShell. Apart from the challenge of understanding the new interface and terms, administrators have the option of
becoming a GUI “whimp” or an EMS wizard. With this in mind, this
chapter explores how to use the Exchange Management Console to perform common management tasks and how to use Exchange PowerShell to
automate the same tasks.
Exchange Management Console
The Exchange Management Console is the latest version of the administrative tool for Exchange servers. It replaces the Exchange System Manager
(ESM) console that features in Exchange 2000 and 2003 that replaced the
original ADMIN.EXE administration program used by Exchange 4.0 to 5.5.
EMC and ESM are both MMC snap-ins. EMC is based on Version 3.0 of
the Microsoft Management Console (MMC) framework and shares some of
the design philosophies seen in other consoles, such as those used for ISA
2004 or 2006, MOM 2005 or 2007, and Windows Storage Server. MMC
3.0 is included in Windows 2003 R2 and also supports Windows 2003 SP1,
where it requires you to install the .NET framework (version 2.0.50727 or
Experience with the original version of ESM demonstrated that it
worked well for small to medium organizations but had some performance
problems in larger organizations (more than a hundred servers). ESM was
also handicapped in that you cannot create new mail-enabled Active Directory objects from the console, so you ended up in a split management
model where some tasks (such as creating a new user account) were performed with the AD Users and Computers console and others (like creating
a new messaging connector) were performed with ESM. While you could
argue that the split in responsibilities was logical because you wanted those
who administered the Active Directory to have full control over the objects
that were created, in practice the split was not terribly logical and ended up
3.1 Exchange Management Console
confusing more people than it helped. Finally, it was not always easy to find
information within ESM as getting to a specific option might require seven
or eight different navigation operations to expose the desired data.
Microsoft’s goal was to make EMC better performing and to simplify navigation within the console.
Figure 3.1
Exchange 2003
ESM and
Exchange 2007
Figure 3.1 demonstrates the difference in design philosophies in console
layout between Exchange 2003 and Exchange 2007. The Exchange 2003
ESM (left) has a traditional tree that you navigate down through to reach the
administrative group, then the server, then the storage group, and then the
database before you can do any real work. As it happens, ESM reveals more
information about mailboxes in a database when you eventually get there,
because it displays the mailboxes and data such as mailbox size and total
items alongside. However, you have had to do quite a lot of clicking to get
there. The Exchange 2007 EMC (right) has a cleaner design that organizes
information into four distinct nodes or sections:
Organization Configuration: This area holds all the global or
system-wide configuration data such as email address policies 1,
address lists, and accepted domains. The sub-tree in this area
shows servers categorized by role so that you can see all the configuration data for a server role in one place. The top level of the
tree holds all the general global settings that do not map to a
server role, much as it does when you work with organizational
settings in the Exchange 2003 version of ESM.
Recipient Policies are the rough equivalent in Exchange 2000 and 2003.
Chapter 3
3.1 Exchange Management Console
Server Configuration: As the name implies, this area holds configuration settings for individual servers such as the databases that
are present on the servers. It also allows you to work with servers
to enable options such as local continuous replication for a storage group (see Chapter 9 for more details on how local continuous replication works). Like the organization area, the server area
presents servers grouped in roles, but you can also click on the
Server Configuration node to see an aggregate view of all servers
in the organization. Think of how you work with individual servers in ESM; the difference is that you can get to an individual
server more quickly because you don’t have to navigate through
administrative groups.
Recipient Configuration: This is where you manage all types of
mail-enabled recipients such as mailboxes, distribution groups,
and contacts. EMC groups the various recipient types under the
Recipient Configuration node and if you click on the node, you
see a list of all recipients, or at least, up to the first 1,000 recipients in the organization. This operation could take a long time in
a large organization, but Microsoft is confident that they have
tweaked the performance of the predefined filters included in
Exchange 2007 (for example, show only mailboxes) to deliver the
necessary responsiveness. As we will discuss later on in this chapter, you can create tailored filters to match the exact requirements
of your environment and speed things up even further. One small
concern is the naming convention that Microsoft has chosen for
the “Disable” and “Remove” options (Figure 3.2) because the
“Disable” option really means, “Leave the Active Directory
account there, but remove the Exchange-specific attributes”
whereas “Remove” means “Delete the Active Directory account
and the mailbox completely.” Apart from the obvious removal of
access to a mailbox, removing a mailbox can have other consequences. For instance, if you use the mailbox as a journal recipient, its removal has an obvious effect on the ability of Exchange
to journal messages to that destination.
Figure 3.2
Disable and
Remove a mailbox
EMC lists any disconnected mailboxes in this node. Disconnected mailboxes are mailboxes that exist in a mailbox database
but do not have a connection to an Active Directory account. You
3.1 Exchange Management Console
can use this node to connect these mailboxes with an account
after the mailbox was previously disabled or following a disasterrecovery exercise. Because entries for disconnected mailboxes do
not exist within the Active Directory, Exchange queries a specific
mailbox store to discover any disconnected mailboxes that may
exist in that database’s mailbox table. If you suspect a database
includes some disconnected mailboxes that do not show up in
EMC, you can use the Clean-MailboxDatabase shell command
to scan the database to check for disconnected mailboxes. This is
the equivalent to the Run Cleanup Agent option in the Exchange
2003 ESM. You can then delete the disconnected mailboxes with
the Remove-Mailbox command.
Exchange Toolbox: Some refer to this node as a “launching pad”
for additional tools from Microsoft (and potentially third parties)
that you can use to manage Exchange. For example, Microsoft
includes troubleshooting, diagnostics, and analysis tools such as
the Message Tracking Center, the Exchange Best Practice Analyzer tool, and the Queue Viewer. See Chapter 10 for an explanation of these tools. ESM supports the Message Tracking Center,
but Microsoft has only created many of the tools that you now
find in the Toolbox since they shipped Exchange 2003. Previously, you would have to download each tool separately from
Microsoft’s web site and install it onto selected servers.
The four task areas do not include anything to do with the Edge server
role. This is because Edge servers operate in a standalone mode quite separate from the rest of the Exchange organization. Edge servers share in the
configuration of the organization through the EdgeSync process that pushes
the data from the Active Directory to individual Edge servers, but they play
no part in the organization as a whole and are therefore omitted from EMC.
Microsoft implemented the new layout for EMC because they believe
that this is a logical division of labor across common tasks for Exchange
administrators. They refer to each area as a “work center,” which Microsoft
defines as a top-level tree node with sub (or child) nodes. A major difference
in EMC is that any dynamic data that results as you navigate through different object types is shown in the result pane rather than appearing as part of
the tree. For example, if you enable a mailbox for ActiveSync, an option to
allow you to work with properties of the mobile devices associated with the
mailbox appears in the action pane. EMC divides its screen estate into the
navigation tree, the result pane, and the action pane (which you can turn off
by customizing the console view if you prefer to use the right-click contextsensitive menu to reveal the options that are available to work with objects).
The server node also reveals a separate work pane to allow you to view details
Chapter 3
3.1 Exchange Management Console
Figure 3.3
EMC Error
provider in action
of objects related to individual servers, such as the databases on a mailbox
server. EMC now offers a tabbed view to group similar features and data
together in some cases, such as when you view the configuration details of a
server role. The result of all the changes is that EMC navigation now goes
just three levels deep rather than eight or more as was the case in ESM. The
change takes a little to get used to, but it is reasonably intuitive.
One nice touch in EMC that did not exist in ESM is the addition of an
error indicator when a value that you enter into a field is incorrect. As you can
see in Figure 3.3, I attempted to enter a huge value for the “Issue warning at”
storage limit. EMC flags the error with an exclamation mark in a red icon and
if you move the mouse pointer over the icon, you see some explanatory text. In
this case, the limit for a quota ranges from 0KB to 2,147,483,647KB, or just
under 2,048GB, which should be enough for anyone.
The importance of filters
For performance reasons, EMC only attempts to fetch the first 1,000 items
that match the filter in use. The same limit of 1,000 items exists everywhere
that Exchange fetches information from the Active Directory, including PowerShell commands. The default filter for EMC is to “get everything,” so
EMC limits itself to working with the first 1,000 mailboxes, contacts, or
groups when you access these nodes under Recipient Configuration, or a
combination of all mail-enabled objects when you work at the Recipient
Configuration level.
For performance reasons, EMC uses a feature called “list view virtual
page” to avoid the need to render the items until it needs to display them, so
even if you ask for more items, you will not take on much more of a performance penalty until EMC has to display the new items. The default
approach of showing the first 1,000 items is perfectly reasonable in a small
Exchange organization, such as those operating in small businesses or in test
environments. In any medium to large organization, it is likely that you will
3.1 Exchange Management Console
want to apply a filter to restrict the items that EMC displays in order to not
slow down the responsiveness of EMC. Alternatively, if you really need to
view thousands of items, you will have to specify that EMC should process a
larger number of items from the Active Directory. Once you find a filter that
works well in your environment, you can save it as the default view (use the
Save Current Filter as Default option on the View menu to do this). For
example, opting to view all the mailboxes on the server that you manage is a
common option to take.
Figure 3.4 shows two ways to define what data you want to work within
EMC. First, you can set the scope for the organizational unit to fetch users
from the default (access everything from the top-level organizational unit
down, so all mail-enabled objects in the directory are fetched) to another
point in the OU structure. This approach works well if you have administrative responsibility over a domain or organizational unit and do not need
EMC to scan the entire directory. It is also an important point for administrators who work with Exchange inside a multi-domain forest. The default
starting point for the recipient scope is the domain that the Exchange server
is located in, so if you want to work with objects from another domain, you
have to click on the “Modify Recipient Scope” option at the Recipient Configuration level and select the domain that you need.
The other option to control the set of objects that you work with allows
you to increase the number of items that EMC fetches from the Active
Figure 3.4
Increasing the
numbers of items
and applying a new
scope to EMC
Chapter 3
3.1 Exchange Management Console
Directory. The default number of items returned is 1,000 and if there are
more objects available (as defined by the current recipient scope), you will see
the error displayed in Figure 3.5. However, you can increase the size of the
set that EMC fetches by setting the “Modify the Maximum Number of
Recipients to Display” option, so that even if you select a very large organizational unit to work with, you can fetch all the items. As we will see in Chapter 4, you can do the same thing when you work with shell commands by
specifying a number for the set of objects that you want to work with
through the –ResultSize parameter.
Figure 3.5
Too many objects
in the current scope
Do not rush to fetch a larger number of items from the Active Directory. Depending on network conditions and the load on the Global Catalog server that Exchange connects to, it could take EMC a long time to
fetch more than the default set of 1,000 items. For example, I changed the
maximum number of recipients setting to force EMC to fetch and display
60,000 recipients in a very large organization (Figure 3.6). EMC took
approximately two minutes to fetch all the recipients, but it displayed them
without a problem. If you do set too high a figure to load, you can always
use the “Stop Loading” button to stop and then click on F5 or the
“Refresh” option to get going again.
Figure 3.6
Allowing EMC to
display sixty
thousand recipients
Once you have the complete set cached, EMC does not go back to the
Active Directory unless you explicitly ask for the set to be refreshed. In
another test, I increased the number of recipients to 30,000 and EMC took
about a minute to load these objects. I then increased the number to 50,000
and EMC took approximately 30 seconds to load the first 30,000 and then
slowed down to load the last 20,000 in just over another minute. Even
though EMC still verified the objects, the first 30,000 objects loaded faster
3.1 Exchange Management Console
because they were already in memory whereas the last 20,000 had to come
off disk.
You do not want to wait around for EMC to load, so when your organization spans more than a few thousand users, it is more efficient to create and
apply a filter to work with the objects that you need. Figure 3.7 shows how to
focus in on a set of objects to work with. In this case, we have defined a filter
to show only the users who work in a department starting with “L.” Examples of other common filters are “all mailboxes on server ExchMbxSvr1” or
“All users in London” to focus in on a specific set of users. Because it is difficult to know exactly what filters you need, EMC allows you to use a wide
range of properties to build filters.
Figure 3.7
Applying a filter
to EMC
For example, if you work at the Recipient Configuration level, you can
filter based on the following properties:
ActiveSync Mailbox Policy
CustomAttribute1 to CustomAttribute15
Display Name
Email Addresses
External Email Address
First Name
Last Name
Managed Folder Mailbox Policy
Chapter 3
3.1 Exchange Management Console
Recipient Type Details
State or Province
UM Enabled
Unified Messaging Mailbox Policy
User logon name (pre-Windows 2000)
User logon name (User Principal Name)
Some of these properties are available for certain recipient types. For
example, the 15 custom attributes are only available for mailboxes, so if you
create a filter based on one of these attributes, the filter will only return mailboxes and contacts and groups will be ignored. The same is true for a server,
because only mailboxes are associated with a server. When you build filters,
you can use the following operators to compare values:
Does Not Contain
Does Not Equal
Ends With
Starts With
Not all of the operators are available for every property. For example,
you can create a filter of “City Equals Dublin” or “Office Starts with L,” but
you can only use the “Equals” and “Does not Equal” operators with the
Server property. However, to make things easier, EMC does at least provide a
browse button for you to select the server that you want to include in the filter. After you add a filter, you apply it by clicking on the “Apply Filter” button. You can add additional expressions for the filter by clicking on the “Add
Expression” button. The left-hand screen in Figure 3.8 shows a two-step filter
that searches for mailboxes that have an office starting with “Bel” and a city
containing “Dub.” In fact, this is not a very efficient filter because EMC has
to do a lot of work to scan records any time you use the “Contains” operator.
This filter works, but it would be more efficient (and precise) if we change
the second expression to “City starts with Dub.” You will not notice the per-
3.1 Exchange Management Console
Figure 3.8
Adding expressions
to the filter
formance hit on a test system, but you will on a production system if you
support more than a few thousand users.
Note that you can add multiple expressions that operate against the same
property, in which case Exchange treats the filter as an implicit OR operation.
If you create multiple expressions against different properties, Exchange treats
the filter as an AND expression. Look at the right-hand screen of Figure 3.8
and see that because we have changed the expression that compares the value of
the City property from “Contains” to “Starts with,” the number of found items
falls from five to three because the filter now drops the two items that contain
“Dub” somewhere in the City property.
You can add up to ten additional expressions for your filter. Each expression adds to the complexity of the filter and may slow its performance, so
you should ensure that you only add expressions that are absolutely required
to zero in on the objects that you want to see. EMC is optimized to use
server-side filtering whenever possible, meaning that Exchange executes the
filter on the server rather than bringing all possible items to the client and
then applying the filter. This approach minimizes network traffic and maximizes performance, but even so, an inefficient filter is never going to execute
quickly. Removing the filter to revert back to the default filter is simply a
matter of clicking the “Remove Filter” button.
Managing mixed organizations
As Figure 3.9 illustrates, if you operate a mixed mode organization to migrate
legacy Exchange servers, Exchange 2007 servers are visible through the
Exchange 2000/2003 management console. You will also notice that the
Exchange 2007 servers occupy their own special administrative and routing
group that allows them to participate in routing within a mixed mode organization. However, while you can view organizational information, the version of ESM provided in Exchange 2003 automatically blocks any attempt to
edit an object that is under the control of Exchange 2007. You should always
manage Exchange 2007 objects through EMC or EMS. Here are some recommendations on how to approach management when you have a mixture
of servers in your Exchange organization:
Chapter 3
3.1 Exchange Management Console
Figure 3.9
Exchange 2007
servers listed
in ESM
Do not attempt to use ESM to manage Exchange 2007 storage
groups and databases. These objects have been modified to function
correctly on an Exchange 2007 server. If you attempt to use ESM to
view the properties of these objects, Exchange returns an error similar
to the one shown in Figure 3.10.
Always manage Exchange 2007 mailboxes with EMC or EMS. Why?
Because if you attempt to manage them with AD Users and Computers on a Windows 2003 server that has the Exchange 2003 add-ins
loaded, you may corrupt some important attributes. Microsoft does
not impose a block on editing Exchange 2007 mailboxes from AD
Users and Computers, but you do this at your own peril.
You can edit, remove, or move Exchange 2003 mailboxes with EMC
or EMS, but do not attempt to create a mailbox on an Exchange
2003 server with EMC or EMS. Why? Because the RUS has to process new mailboxes on an Exchange 2003 server and if you create it
through EMC or EMS, the mailbox may not be fully functional.
You can manage mail-enabled contacts from either Exchange 2003 or
Exchange 2007.
Figure 3.10
ESM cannot deal
with an Exchange
2007 server
3.1 Exchange Management Console
You can manage mail-enabled groups from either Exchange 2003 or
Exchange 2007. However, be careful managing dynamic distribution
groups because the recipient filters differ between the two versions.
Exchange 2003 uses LDAP filters and Exchange 2007 use OPATH
filters. See page 156 for more information on this topic.
Do not use the Exchange 2003 move mailbox feature to move mailboxes to an Exchange 2007 server. Always use the Exchange 2007
move mailbox feature (or the Move-Mailbox shell command), because
it is the only method that can move data between both server versions.
Apart from the Move-Mailbox command, a number of other shell
commands work against legacy Exchange servers. We will get to more
detail on this topic in Chapter 4, but in general, you should not
expect shell commands to work, if only because the majority of the
commands focus on Exchange 2007 objects and data structures.
Do not edit objects such as the Default Offline Address List or Email
Address policies with ESM after you update them with Exchange
Do not attempt to use the ESM functions to monitor servers or view
queues on an Exchange 2007 server, as these functions will not work.
You can look at the details of the routing group connector that links
your Exchange 2003 servers with the Exchange 2007 servers, but you
cannot modify it. ESM flags the problem with the dialog shown in
Figure 3.11, which indicates that you need to use a server running
Exchange version 8 (2007) to work with the object.
Figure 3.11
Unable to edit an
object with ESM
The Exchange 2003 version of ESM understands the difference in
object version numbers, so it is able to stop you editing Exchange 2007
objects. However, the Exchange 2000 version of ESM has no knowledge
about object version numbers or Exchange 2007 unless you have updated
your servers to run Exchange 2000 SP3 and the August 2004 hot-fix roll-up
(described in Microsoft Knowledge Base article 870540). In addition to the
servers, you obviously need to update any workstations that run the
Exchange 2000 version of ESM.
While you can use EMC to manage objects on an Exchange 2003 server,
if you want predictable results, it is best to follow the old advice of using
management tools designed for a particular environment to manage that
Chapter 3
3.1 Exchange Management Console
environment. In other words, use the mixture of AD Users and Computers
and ESM to manage Exchange 2003 objects and use EMC and EMS to
manage Exchange 2007 objects. The bottom line is that you should keep
ESM to manage the Exchange 2003 and 2000 servers until they are decommissioned and to administer components that have been deprecated in
Exchange 2007, such as public folders (see Chapter 4 for a discussion on how
to manage public folders through the Exchange Management Shell).
Running EMC remotely or on a workstation
You can install the Exchange 2007 management tools on a server that does
not support Exchange 2007 or onto a workstation to allow you to perform
Exchange management remotely. To do this, you run the Exchange 2007
installation program and select the option to install only the Administration
components. You can download the x32 version of the Administration components (EMC, EMS, Exchange Best Practice Analyzer, Exchange Troubleshooter, and the Exchange help files) and install them on the following
Windows Server 2003 with SP1 or SP2 (x86 or x64 versions)
Windows XP SP2 (x86 and x64)
In all cases, you must already have installed MMC 3.0, PowerShell 1.0,
and at least version 2.0 of the .NET framework, and whatever hot fixes are
required when you are installing the Exchange management tools. If you
install on an XP workstation, you also have to install IIS common services
(use the Add/Remove Windows components from the Add/Remove software
option in Control Panel and make sure that you do not install other IIS components such as the SMTP server). If you want to manage some third-party
components that link to Exchange, you may have to install some of their
management code to integrate with EMC and that code may not be supported on a workstation.
Figure 3.12 shows a mixture of Outlook 2007, EMC, and EMS running
on a Windows XP workstation. Everything progresses smoothly and the only
issue that I ever see is a slight delay caused by the extended WAN connection
between my PC (in Dublin) and the server (in Houston). After careful net2.
Microsoft will not support Vista as a platform for Exchange administration until they have had time to test Vista workstations. Expect formal support as part of an early service pack, or expect to pick up tips on how to install and configure the
necessary 32-bit components to get EMC and EMS working on Vista as the result of testing and persistence within the
Exchange community.
3.1 Exchange Management Console
Figure 3.12
Running EMC,
EMS, and
Outlook 2007 on
Windows XP
work planning, HP consolidated all its Exchange 2007 servers into three
datacenters in the US, so all client traffic to an Exchange server flows over
extended connections..
You can only install the Exchange management tools on a workstation
that is part of an Active Directory forest. The management tools depend on
the schema extensions for Exchange 2007, so you must deploy them first.
When you start up EMC or EMS on the workstation, it automatically connects to the Exchange organization if one is present in the forest. There is no
way (that I have found) to tell EMC and EMS to connect to another
Exchange organization, such as one that you might run in a test forest.
Another equally effective method is to use the inbuilt terminal services
capability of Windows 2003 servers to allow Windows XP Professional and
Vista workstations to log onto an Exchange server and be able to perform
administrative activities remotely. This approach works very well and is a
great way of being able to manage Exchange without the need to install anything on a workstation.
No more AD Users and Computers
We touched on the split in responsibilities between Active Directory management and Exchange management that occurs in ESM. Another important
aspect of the introduction of EMC is that you do not need to use the AD
Users and Computers console to work with the Active Directory objects that
are important to Exchange. In the past, you had to create user and other
objects through AD Users and Computers and ESM was only able to list
objects and report on their properties. If you needed to update an object, you
Chapter 3
3.1 Exchange Management Console
had to go back to AD Users and Computers, so administrators needed to
move continually between the two consoles to get work done. Obviously, if
you wanted to delegate access to another administrator and allow them to
manage mailboxes, you had to grant access to both consoles. The split in
operations between AD Users and Computers and ESM was confusing for
administrators and difficult to manage in some production environments if
you wanted to allocate different responsibilities to different administrators.
Exchange 2007 allows you to work with mailboxes, contacts, and
groups through EMC without ever having to go near AD Users and Computers. In fact, the Exchange 2007 installation program does not install the
add-ins that Exchange 2000 and 2003 use to allow AD Users and Computers to work with Exchange properties, so you can only use AD Users and
Computers to work with properties that are not used by Exchange. For anything specific to Exchange 2007, such as enabling features like ActiveSync
(Figure 3.13), you have to access mailbox properties through EMC, or
indeed use the PowerShell commands that Exchange provides to work with
the objects. You still need to use AD Users and Computers, but only for
tasks that are fundamental to the Active Directory, such as creating a new
organizational unit.
Figure 3.13
Working with
Exchange mailbox
If you create users through AD Users and Computers, you can use the
New Mailbox option in EMC to mail-enable them. As shown in Figure 3.14,
3.1 Exchange Management Console
Figure 3.14
Selecting a
Windows user to
be mail-enabled
you can select “Existing User” rather than “New Mailbox” to be able to
choose a user who is not mail-enabled and assign them a mailbox.
One small note about creating new mailboxes through EMC – the wizard that creates the mailbox only supports the default UPN suffix even
though a forest can support multiple UPNs. In addition, the UI (Figure
3.15) uses a drop down list for the UPN suffix that obviously indicates that
Microsoft planned to include this functionality. However, it is just a small UI
bug that has crept into Exchange 2007. If you want to create new mailboxes
that use a non-default UPN suffix, you will have to do this by using the shell
New-Mailbox command that we will explore in Chapter 4. Alternatively, you
could create a non-mail enabled user object through the AD Users and Computers console and set the UPN there before using EMC (or the shell) to
mail-enable the new user. Microsoft addresses the issue in Exchange 2007
SP1 and you will be able to select a UPN from a drop-down list that an
administrator can create.
Changing columns
Like any MMC console, you can use the Add/Remove option on the View
menu to change the columns that you see when you access different types of
Chapter 3
3.1 Exchange Management Console
Figure 3.15
one UPN
suffix allowed!
objects. In Figure 3.16, we are positioned at the mailbox node under Recipient Configuration. The default columns are Display Name, Alias, Recipient
Type Details, Primary SMTP Address, Server, and Organizational Unit. You
can actually only see the first three of these columns in the picture because of
limited screen real estate, but EMC would display them if enough space was
As you can see, you can remove some of the default columns and replace
them with others that might be of more use to you. For example, some
administrators replace the alias column with department because they manage mailboxes on the basis of department and so want to be able to sort what
EMC displays by department order (you can sort on any available column by
clicking on its heading).
Because EMC deals with different types of objects depending on where
you are positioned, it is logical that it offers different columns. Figure 3.17
shows the columns available when EMC is positioned on the hub transport
node under Organizational Configuration. You can see that the columns that
are available here are very different to those available when EMC displays
Visual effects
EMC offers a visual effects option on the View menu that allows you to
chose whether these effects are always on (default), never used, or automatic
3.1 Exchange Management Console
Figure 3.16
Changing the
columns used
by EMC
Figure 3.17
Columns available
for send connectors
(meaning that EMC selects). Visual effects control whether the Exchange
wizards use the “Exchange 2007 skin” or classic Windows look. Figure 3.18
illustrates the difference. On the left, you see the classic Windows look as a
wizard creates a new mail-enabled contact. On the right, you see the
Exchange 2007 skin.
Chapter 3
3.2 Why some options have disappeared from EMC
Figure 3.18
Classic Windows
and Exchange
2007 visual effects
You see the big difference in the two effects when you work over a lowspeed connection such as when you manage an Exchange server using Windows Remote Desktop Connection across the public Internet via a VPN.
While the Exchange developers believe that their skin is cool, it is also slow
and gets in the way because it takes too long for the wizard to paint screens.
Why some options have disappeared from EMC
You can manage Exchange 2007 objects through EMC or EMS, but you
have to understand that EMC is a subset of the capabilities offered by EMS.
The result is that administrators have to be comfortable with EMS if they
want to exert maximum control over Exchange 2007, because the more you
look at Exchange 2007, the more you discover that there are many tasks that
can be only performed through EMS. The introduction of EMS is therefore
a huge influence on Exchange administration.
Even accepting that EMS is the most powerful tool at the disposal of
Exchange administrators, it is still a tremendously interesting aspect of the
transition from ESM to EMC that Microsoft has removed a number of
options that used to feature in ESM. You cannot assume that everything that
you used to be able to do through the GUI is still possible in Exchange 2007.
For example, you cannot alter server properties to enable message tracking
for a server. In Exchange 2007, you enable this feature through EMS (see the
discussion on message tracking in Chapter 10). As you can see from Figure
3.19, there is no diagnostics tab available when you view server properties, so
you cannot turn up diagnostics on a server to force Exchange to log more
information into the application event log. Like every other time when you
discover a missing feature that used to be in the GUI, you now have to resort
to EMS. In this case, you would use the Get-EventLogLevel –Server command to return the current diagnostic level on a server and the Set-EventLo-
3.2 Why some options have disappeared from EMC
Figure 3.19
Server properties—
no diagnostics!
command to set a new diagnostic level (lowest, low, medium, high,
or expert) for a selected component. For example:
Set-EventLogLevel –id ‘MSExchangeTransport\Categorizer’
–Level High
Of course, you may not know what components you can tweak to
increase the diagnostic level, but you can check this with the GetEventLogLevel command, passing the desired server name as a parameter:
Get-EventLogLevel –Server ExchMbxSvr1
Other examples of options that are available in the ESM GUI in
Exchange 2003 that are not supported by EMC in Exchange 2007 include
managing the properties of Exchange’s POP3 or IMAP4 servers as well as
almost anything to do with routing management such as organizing servers
into routing groups (which have disappeared from Exchange 2007 anyway).
Another example is the inability to view or set permissions on Exchange
objects through the GUI. In Exchange 2000 and 2003, you can set a regisChapter 3
3.2 Why some options have disappeared from EMC
try value to force ESM to expose a “Permissions” property page when you
view objects, but the Exchange developers didn’t have time to complete this
work for Exchange 2007. Instead, you have to use the Add-AdPermission,
Get-AdPermission, and Remove-AdPermission commands to work with
In addition to permissions, there are a number of other management
tasks that you can only perform through PowerShell, such as managing Cluster Continuous Replication or managing public folders. In fact, if you want
to manage public folders through a GUI, you have to use the Exchange 2003
ESM or tools such as PFDavAdmin running on an Exchange 2003 server.
That is, until Microsoft updates Exchange 2007 to incorporate some GUI to
manage public folders through EMC, an update that is expected in a future
service pack.
The developers did not make the changes lightly. They wanted to create
a streamlined, powerful, and uncluttered console that included all of the
operations that Exchange administrators perform most often. This was not
always the case. For example, administrators can perform Outlook Web
Access management in Exchange 2003 with the OWAAdmin tool, a separate utility that you have to download and install before you can use it.
There are many other examples where administrators had to change
Exchange’s behavior by making changes to the registry or to Exchange’s configuration data in the Active Directory. Often, Microsoft had documented
these settings poorly or had not documented them at all, which made
Exchange 2003 more difficult to support, especially if administrators made
mistakes when they edited configuration data. Finally, there are parts of
ESM that are difficult to extend to add new functionality because business
logic is buried alongside the user interface.
As they designed EMC and decided what options to present through the
GUI, the developers put features into four buckets:
Options that would remain and EMC might be able to enhance.
For example, you can now control what features Outlook Web
Access presents to users. In addition, EMC displays some information that is not covered. For example, if you view mailbox
details, EMC displays the last login time for the user, the total
items in the mailbox, and the consumed quota (Figure 3.20).
This is a nice feature, but it can slow you down if the mailbox
data is not cached as EMC is then forced to retrieve the data from
the Store.
An option that most administrators never needed to use. For
example, Microsoft discovered that the Recovery Storage Group
was not used by many administrators, so it is now available as
3.2 Why some options have disappeared from EMC
part of the Toolbox. However, you never see the Recovery Storage
Group as all of the interaction with it is via a wizard. Another
example is configuration of the IMAP and POP protocols. Few
enterprise installations use IMAP to access Exchange and fewer
use POP, so it makes sense to move these options into EMS.
These options helped ESM to become over complicated in parts,
so moving them out helps to create the streamlined UI that we
see today.
An option that was no longer required by Exchange 2007. The
best example is the elimination of the Recipient Update Service.
Although they will be around for some time to come, you might
also consider the removal of public folder management to be in
this category.
An option that has been replaced by a better solution in Exchange
2007. The inclusion of recipient management in EMC is a good
example because it removes the split of Exchange 2003 management operations that exists between AD Users and Computers
and ESM and it allows Exchange to use a common code base to
work with recipients in EMC and EMS. System policies are no
longer required because you can use shell commands to perform
bulk management of servers, mailboxes, contacts, and groups.
If they just removed features from the console, the developers would
have created an easier console to use but a less powerful management environment. The advent of EMS is a mitigating factor because it offers the
developers a way to include all the seldom used features in Exchange and a
way to replace making changes to Exchange’s configuration with tools like
ADSIEdit. Do not read these comments as being critical of the developers
who built ESM in Exchange 2000 and 2003 as they did their best with the
technology available at that time. Exchange leverages the MMC framework
as much as it can through components like the snap-ins for AD Users and
Computers to add the Exchange property pages, but there is a limit to what
you can do with a graphical user interface. The review done for Exchange
2007 identified opportunities to improve matters using new technology, new
approaches, and feedback gleaned from the experience of many thousands of
customers who have managed Exchange since 2000.
The Exchange developers believe that EMC is a much better working
environment for administrators and that the combination of EMC and EMS
delivers the power and flexibility to manage even the largest and most complex Exchange organization. Microsoft also believes that they will be far better able to extend Exchange in the future in a way that they never have to
resort to the kind of add-on utilities and registry hacks that we’ve experienced in the past. If they are right, then we’ll all be happy.
Chapter 3
3.2 Why some options have disappeared from EMC
Figure 3.20
Coping with change
Like every other human, administrators accumulate habits over time and we
all have our own methods to manage Exchange that we have honed over
time. The changes that the developers have made to the console and the fact
that some options are missing force you to go and learn where the equivalent
functionality exists in Exchange 2007.
Microsoft provides a huge array of commands to handle all the work to
deploy and manage Exchange. There is huge power and flexibility now available for those who want to roll their sleeves up and write scripts to build
management the way that they see best fit, but others, especially part-time
administrators, will feel that it would have been nice had Microsoft provided
a more comprehensive and feature-rich GUI for EMC. Such an attitude
rather misses the essential point of what has occurred for Exchange management. The transition from GUI options to EMS commands is a fundamental
cultural transformation. Griping about the new GUI and the missing
options that used to be available in Exchange 2003 ESM is only a waste of
time. The real value of EMS is not in one-off tasks: The value of EMS comes
about through the ability to automate common tasks, especially when you
3.2 Why some options have disappeared from EMC
need to process more than one object at a time. Once you discover how to
perform a task through EMS, you can capture the necessary commands in a
script and reuse that script time after time, something that becomes a lot
more interesting than the alternative of clicking your way through the GUI
multiple times.
For example, while ESM can list information about mailboxes in a
selected mailbox store, it cannot give you a complete list of all mailboxes
across the organization, so you need to select every mailbox store to retrieve
this data and then combine the data that you extract from each mailbox store
to form the complete picture. If you have 20 mailbox servers that host 3
mailbox stores each within your organization, you have to repeat the select,
open, export operations 60 times before you can even begin to combine the
data. By comparison, you can combine the Get-ExchangeServer and GetMailboxStatistics commands to extract and report data from all servers in
one operation. It might take the script some time to complete the processing
necessary to fetch data from all your servers, but it is a lot easier when the
computer does the work.
Another example of how the Get-MailboxStatistics command helps
to locate specific data quickly is how to discover how many disconnected
mailboxes exist within a mailbox database. You would not expect to have a
GUI option for this operation because it is not something that administrators need to perform often, but you can do it with a single command line:
Get-MailboxStatistics –Database ‘VIP Mailboxes’ | WhereObject {$_.DisconnectDate –ne $Null}
Do not worry too much about the syntax of EMS commands for now,
as we will get to explore the finer points of working with the shell in Chapter
4. The point is that you can execute commands like this to do things that
were not possible in previous versions of Exchange without writing a lot of
code. Here is another example to illustrate the point. With Exchange 2003,
unless you have just one server with just one mailbox database, it is very hard
to get a quick snapshot of the distribution of mailboxes across the different
mailbox stores in an organization. To complete the task, you have to go to
every server, open up each mailbox store, record the data, and then collate it.
With Exchange 2007, you can use the Get-Mailbox command to do the job
like this:
Get-Mailbox | Group-Object database | Sort Count -descending
| Format-List Count, Name
Chapter 3
3.3 Changes in the Exchange delegation model
Again, this is a command that will take some time to execute in a large
organization, but it is not something that you will do every day.
All of this goes to prove that a fundamental change in system management philosophy occurs in Exchange 2007. The changeover from expecting
to perform management through a GUI that is limited by the imagination of
the developers to working through a powerful shell poses a cultural and
learning challenge for administrators. There is no doubt that you can do
many things with just a few lines of PowerShell, but Microsoft’s strong and
frequent assertions that administrators will find that the change from GUI to
PowerShell makes Exchange easier to manage understates the dimension of
the problem that faces administrators as they move to Exchange 2007. I suspect that a little bit of cognitive dissonance may have crept in here because
the Microsoft developers have worked with PowerShell for so long. The
changeover in thinking from GUI to shell is a challenge for us all; some will
struggle to get to grips with PowerShell and it will take everyone some time
to discover how to get things done effectively and efficiently through the
Changes in the Exchange delegation model
Exchange 2007 includes many changes to the delegation model that enables
administrators to work with servers and other Exchange data. The major
changes are:
Creation of a wider and more useful set of out-of-the-box roles
(Table 3.1) for common administrative tasks such as taking backups
and working with recipients. The logic here is that many large organizations delegate tasks like these to help desk personnel, who do
not need complete administrative control over a server to do these
jobs. The permissions therefore reflect role-specific tasks that you
can assign to individuals through the Permission Delegation Wizard
(Figure 3.21).
Implementation of a clearer separation between Exchange and Windows properties in recipient management: While Exchange depends
on a large set of properties for mail-enabled Active Directory objects,
the separation between the set of attributes that Exchange administrators need to work with and those required by Windows was not
clear in previous versions. Exchange 2007 is more definitive in this
respect and you cannot access a Windows property unless you are a
Windows administrator. Apart from anything else, the separation of
Exchange and Windows properties for mail-enabled objects means
that there is less chance of inadvertent errors.
3.3 Changes in the Exchange delegation model
Figure 3.21
Adding a new
Exchange 2007 creates an Exchange Servers universal group to hold
all Exchange servers and to replace the previous Exchange Domain
servers and Exchange Enterprise servers groups. This change reflects
experience of Exchange in production. There must have been a good
reason to have several groups previously, but that reason is lost in the
mists of time.
Implementation of simpler recipient provisioning: Creating new
mail-enabled objects in Exchange 2003 is a complex multi-step process that requires an administrator to create the object followed by
processing by the Recipient Update Service (RUS), which takes care
of creating email addresses according to policy. Exchange 2007 uses
simplified email address policies to define the format of email
addresses to apply immediately after an administrator creates a new
mail-enabled object. If you modify a template, Exchange applies the
change to all recipients covered by the policy. In effect, the RUS is
neutered—and not before time! See page 130 for more information
about how Exchange 2007 now performs the functions previously
taken care of by the RUS.
The changes in the delegation model will be most useful in enterprise
deployments where there are multiple administrators and multiple roles. In
smaller deployments, where one or two people administer Windows and
Exchange and do everything, you can continue as before in the “all-powerful” Exchange Full Administrator role.
Exchange automatically assigns the Exchange Organization Administrator role to the account that installs the first Exchange server in an organizaChapter 3
3.3 Changes in the Exchange delegation model
Table 3.1
Exchange administrative roles
Exchange Organization
This is the highest level of permissions within an
Exchange organization and is designed to manage data
with a global scope, so it has full control over the
Exchange configuration container in Active Directory.
This role is able to modify any setting for Exchange at global or local level, for example, create new managed folders, set policies, establish a new connector, or set a new
global restriction for message size.
Exchange Server
Designed to perform administrative tasks tied to a specific
server. Is able to modify any setting on that server.
Exchange Recipient
Is able to perform administrative operations on mailenabled objects (for example, set a new mailbox quota) or
execute recipient management operations for the complete organization. This role is facilitated through access
to the Exchange Personal Information and Exchange
Information property sets (see later in this chapter).
Exchange View-Only
Is able to view information about the complete organization. One “viewlike” option that these administrators cannot take is Message Tracking, as this requires authorized
access to the Exchange Transport Log Search service on
the server where the search is being performed.
tion. Subsequently, you can use this account to delegate administrative roles
to other accounts. Behind the scenes, the /PrepareAD phase of the Exchange
installation program creates Windows universal security groups to support
the roles in the Microsoft Exchange Security Groups OU in the domain
where /PrepareAD runs. The groups shown in Figure 3.22 are:
Exchange Organization Administrators: Accounts that hold the
Exchange Organization Administrator role are members of this group.
Exchange Recipient Administrators: The delegation wizard adds
accounts that hold the Exchange Recipient Administrator role to this
Exchange View-Only Administrators: The delegation wizard adds
accounts that you grant the Exchange View-Only Administrator
role to this group. In addition, the wizard adds accounts that hold
the Exchange Server Administrator role for a specific server to the
3.3 Changes in the Exchange delegation model
group to allow a server administrator to be able to view the complete organization.
ExchangeLegacyInterop: Used to secure operations between
Exchange 2007 and legacy (Exchange 2003 and 2000) servers.
Exchange Servers: This group contains all Exchange 2007 servers in
the organization.
Figure 3.22
Exchange Security
The addition of the ability to delegate administrative permission for a
specific server to an individual solves a major issue that often occurred in
Exchange 2000/2003 deployments where a single Exchange server was
deployed in branch offices to support a small community. In these cases, it
was common practice to create a separate administrative group to host the
server and then allocate administrative control over the administrative group
to a user in the branch office. This approach was logical because the administrative group represents the boundary for server administration in Exchange
2000/2003. However, even though it was logical to create small administrative groups, the result was a proliferation of administrative groups and complicated organizations. Exchange 2007 does not use administrative groups, so
something had to change, which is why servers are now the basic administrative entity for Exchange 2007.
Following on the ability in Exchange 2007 to restrict administration
rights to a single server, the Exchange developers added an interesting option
to define who can install an Exchange 2007 server. This functionality is
extremely useful when you need to delegate the ability to install a server to
someone else in the organization or you simply want to prepopulate the
entire organization with servers that you will eventually get around to installing.3 To perform the delegation, you run the ExSetup program and specify
the FQDN of the server that you want to install and the account name
(which has to be a member of the Local Administrators group for the target
server) that you want to be able to install the server. For example:
There is an excellent entry in the Exchange team’s blog that describes scripts to prepopulate an Exchange organization
using the provisioned server feature.
Chapter 3
3.4 Customized Recipient Management
ExSetup / /
ServerAdmin xyz\Redmond
The ExSetup program then creates the necessary entries in the Exchange
configuration in the Active Directory and assigns the appropriate permissions
to allow the nominated account to run the installation on the server. The delegated account is also granted the Exchange Server Administrator role for the
target server and the Exchange View-Only Administrators role for the organization. To reverse the process, you can run the ExSetup program with the /
RemoveProvisionedServer switch:
ExSetup /
You can also delegate administrative access to the organization, recipients, and specific servers through the Exchange Management Shell. See
Chapter 4 for details.
In addition to the groups created in the Microsoft Exchange Security
Groups container, you will find another group in the Microsoft Exchange
System Objects container called Exchange Install Domain Servers. When
you install an Exchange 2007 server, the installation procedure adds the
server’s computer account to the Exchange Servers universal security group.
The Exchange Servers group is hosted in the root domain of the forest, but if
you install a server into another domain, the Exchange services may fail to
start because the LocalSystem account is not authorized to run these services.
The normal root cause is that the Active Directory has not been able to replicate the updated membership of the Exchange Servers group to the local
domain. As you can see from Figure 3.23, the Exchange Install Domain Servers group is not designed for general viewing. Its purpose is to ensure that
Exchange can start its services immediately and avoid the dependency on
Active Directory replication. Logically, in a multi-domain forest, you can
expect to find an Exchange Install Domain Servers group in each domain.
Customized Recipient Management
As discussed earlier, a split permissions model exists in Exchange 2000/2003
where administrators perform some operations with ESM and some through
the AD Users and Computers console. Recipient management is therefore
sometimes a challenge in Exchange 2000/2003 if you were not a Windows
administrator who had the necessary access to the Active Directory to be able
to update the Exchange-specific properties on mail-enabled objects. This
wasn’t a problem for small to medium deployments because it is usual to
have the same administrators taking care of Windows and Exchange, but
3.4 Customized Recipient Management
Figure 3.23
The Exchange
Install Domain
Servers group
larger deployments tend to separate out responsibilities to different teams
and want to do things such as allocate the power to update user properties to
help desk personnel. Microsoft supports different administrative roles for
Exchange 2000/2003, so you can assign users to hold the roles of Exchange
Server Administrator, Exchange View-Only Administrator, and Exchange
Organization Administrator. These roles are implemented as canned sets of
Windows permissions that Exchange gathers together so that you can assign
the roles to users to enable them to perform different types of management
activities. However, the roles defined in Exchange 2000/2003 are focused on
server rather than recipient management, so while you could be allocated a
role to perform Exchange server management, you had to receive another set
of Windows permissions to be able to create users, contacts, and groups.
Recipient Management is better organized in Exchange 2007. First, you can
assign the Exchange Recipient Admin role to users who need to perform this
role. Second, you can create a customized console that brings together all of
the functionality required to work with Exchange mailboxes, groups, and
contacts. Follow these steps to create the console:
Open MMC.
Add the Exchange Server 2007 snap-in to the console.
Select the Recipient Configuration node and right click. Select
“New Window from here” to force MMC to open a new window
and display just the Recipient Configuration node.
Select File. Options to give the new console a suitable name and
to lock it down to User Mode, single window, with limited access.
Chapter 3
3.4 Customized Recipient Management
Save the console as an .MSC file. You can distribute the .MSC file
to any user who needs to act as a Recipient Administrator.
You can see the effect of the customized console in Figure 3.24.
Figure 3.24
Creating a
Adieu RUS
Exchange 2000 was the first version to use the Recipient Update Service
(RUS) to discover and provision mail-enabled objects with email addresses
according to recipient policies and to ensure that the new objects joined
whatever address lists they should belong to. Mail-enabled objects created
through AD Users and Computers are in a state of “email limbo” until the
RUS stamps them with an email address and other key properties such as
HomeMDB (pointer to the database that holds a user’s mailbox) and
LegacyExchangeDN (legacy Exchange distinguished name) that are required
to make them fully functional for Exchange.4 In Exchange 2000/2003, the
RUS is implemented as an API that calculates the value for the properties
used by Exchange and a process that runs as part of the System Attendant
that scans for new and updated objects and then updates their properties to
comply with recipient policies. The two-step process was necessary in
Exchange 2000/2003 because of the split between Active Directory and
Exchange management of objects. Exchange 2007 uses a different approach
that removes the need for the RUS process because it takes full responsibility
for the management of any mail-enabled object through EMC or EMS.
When you create a new mail-enabled object on an Exchange 2007 server
through EMC or EMS, the new object is immediately fully provisioned and
See Microsoft Knowledge Base article 253770 for more details about the processing that the RUS performs for new mailenabled objects created on Exchange 2000 or 2003 servers.
3.4 Customized Recipient Management
able to function. There is no need for the RUS process to run asynchronously
to validate that the email addresses on the new objects comply with email
address policies, that the new objects possess a full array of Exchange properties, or that they have joined address lists such as the GAL. Behind the
scenes, the commands that create new mail-enabled objects call the RUS API
to calculate values for the properties before the Exchange stamps them onto
the objects. The advantage of incorporating a synchronous process to update
new mail-enabled objects is obvious: Once an administrator creates a new
mail-enabled object, the user can access their mailbox immediately. This is
much simpler for all concerned. With the benefit of hindsight, even the most
ardent defender of the RUS would probably admit that it is the way that
things should have been done from the start.
Recipient policies provided a major input to the RUS in Exchange 2000/
2003. In their place, Exchange 2007 uses Email Address Policies to define
what form of email proxy addresses you want to create within an organization.
The wizard provided in Exchange 2007 to work with Email Address Policies
create a much more structured way of defining and setting email proxy
addresses and you do not run the risk of wreaking havoc within an organization after changing a recipient policy. Changing a recipient policy to alter an
email proxy address forces the RUS into action to apply the new policy, but
unfortunately the RUS often seemed to ignore some email addresses. If an
administrator wanted to be sure that all email addresses compiled with policy,
they would have to apply it explicitly through ESM. In Exchange 2007, if you
update an Email Address Policy, you have a choice of updating objects immediately or at a future time that you schedule, so you know when updates will
occur. In addition, you can have confidence that the underlying commands
will enforce the new Email Address Policy every time an administrator adds or
updates a mail-enabled object through EMC or EMS. See page 173 for more
information on Email Address Policies.
The change to remove the RUS from the mailbox creation and update
process is not before time. When it worked everything flowed smoothly and
new mail-enabled objects could be used reasonably soon after they were created, but when the RUS stopped functioning or merely hiccupped, it was
enormously difficult to determine what was going on. The root of the problem was that Microsoft never documented the RUS in any depth, so it
became one of the black boxes that ran on an Exchange server. The lack of
documentation or feedback from the RUS meant that you could never be
quite sure when the RUS would wake up to process new or amended mailenabled objects, which lead to administrator confusion and frustration.
Depending on your environment, the RUS might take anything from a few
minutes to a few hours before it had stamped a new object with the email
addresses that the object needed to become email-enabled.
Chapter 3
3.4 Customized Recipient Management
It is possible that you run another process to create new mail-enabled
objects, perhaps as part of a system that provisions new users across multiple
systems. For example, a new employee joins and is entered into a HR system
that automatically spawns jobs to create a new user in the Active Directory.
In the past, the RUS would find the new mailbox and stamp the object with
the necessary properties to mail-enable the object. Because the RUS no
longer runs in Exchange 2007, you will have to periodically run the UpdateEmailAddressPolicy and Update-AddressList commands through EMS to
detect new recipient objects that have to be updated. These commands apply
the appropriate email proxy address and address list policies to any object
that they determine should be processed. For example, the command:
Update-EmailAddressPolicy ‘Default Policy’ –Domain XYZ-DC1
Looks for objects and applies the default email address policy to create new
email proxy addresses when required. Note that the domain controller to
provide information about recipients and where to apply the updates is specified with the –Domain parameter. This command can take some time to
execute for any moderately large organization. Perhaps the best advice is to
test how best to integrate these commands into whatever procedures you
operate today in order to ensure that provisioning continues to work after
you introduce Exchange 2007.
One final point—you can see Exchange 2007 servers through ESM so
you may be tempted to select an Exchange 2007 server to host the RUS. Do
not do this as the Exchange 2007 server has no knowledge of how the
Exchange 2003 RUS functions and you will only create a brain-dead nonfunctioning RUS.
Recipient types
Exchange 2003 really offers just one mailbox type: a mailbox used by a
human being to store messages. However, in the real world, it is common to
find different types of mailboxes in use, and Exchange 2007 recognizes this
by supporting several implicit mailbox types that you can use as a filter when
viewing mailboxes through ESM:
User mailbox: A standard mailbox hosted on an Exchange 2007
Room mailbox: A mailbox associated with a disabled Active Directory account used to schedule a conference room.
Equipment mailbox: A mailbox associated with a disabled Active
Directory account used to schedule an item of equipment.
3.5 Moving users
Linked mailbox: A mailbox accessed through an account in a different Active Directory forest that is accessible through a two-way trust.
Shared mailbox: These are mailboxes that multiple user accounts have
permission to log on to. The actual account that is associated with the
shared mailbox is usually disabled because no one uses this account to
log on to the mailbox.
Legacy mailbox: These are mailboxes that are homed on Exchange
2000 or 2003 servers. Even though they are fully functional user
mailboxes, they are called “legacy” because the Exchange 2007 ESM
cannot apply all features to these mailboxes. Once migrated to an
Exchange 2007 server, these mailboxes become standard mailboxes. If
you ever find that you have a mailbox shown as “Legacy” that you
have definitely migrated to Exchange 2007, you can upgrade it with
the shell command:
Set-Mailbox –id Old-Mailbox –ApplyMandatoryProperties
This command only works with mailboxes that are hosted on an
Exchange 2007 server.
A similar model of implicit typing applies when it comes to distribution
groups, breaking them out into:
Mail-enabled universal security group.
Mail-enabled universal distribution group.
Mail-enabled nonuniversal group (global or domain local groups):
These groups are not recommended for use by Exchange because the
expansion of group membership is unpredictable unless you can
guarantee that you always connect to a DC that holds a complete
copy of the group membership.
Dynamic (or query-based) distribution group.
Contacts are much simpler because they are objects with an associated
email address.
Moving users
You’re going to have to move mailboxes if you migrate from a legacy version
of Exchange to Exchange 2007. Once you get to Exchange 2007, normal
Chapter 3
3.5 Moving users
messaging operations will dictate the need to move mailboxes for business
reasons, to balance the load on servers, or to redistribute mailboxes across
Moving mailboxes
Because you can’t upgrade an Exchange 2003 mailbox server to be an
Exchange 2007 mailbox server, the only way that you can transfer existing
mailboxes to Exchange 2007 is to use the Move Mailbox feature. You can
move mailboxes from Exchange 2000 (SP3 or later), Exchange 2003 (SP1 or
later), or Exchange 2007 servers to Exchange 2003 (SP3 or later), Exchange
2003 (SP1 or later), or Exchange 2007 servers. You can also move mailboxes
between different Exchange organizations, providing that the necessary trusts
exist between the two forests.
In most cases, you’ll be moving to Exchange 2007, but it’s nice to know
that you can use the feature to move mailboxes between older servers if necessary. To move a mailbox to an Exchange 2007 server, your account needs to
hold administrative permissions for the target server and you’ll obviously
need administrative permission to access the source mailbox. Apart from
migrations, common reasons to move mailboxes include rebalancing across
databases on a server or across servers within the organization, or to move
mailboxes off a server that is being decommissioned for some reason such as a
datacenter consolidation.
You can use two approaches to move mailboxes in Exchange 2007: the
traditional wizard-based method available through EMC or by using the
Move-Mailbox shell command (see Chapter 4 for details). Because they are
based on the same code base, both methods get the job done, but you can
specify additional parameters to exert more control using the Move-Mailbox
command. In most cases, you won’t need the additional degree of control to
move a few mailboxes around on an ad-hoc basis, but the extra parameters
become more important as you dive into the possibilities that scripting offers
to automate moves for hundreds of mailboxes during the migration to
Exchange 2007.
Microsoft made many improvements such as support for multiple
threads to the Move Mailbox wizard in Exchange 2003. Exchange 2007 adds
prevalidation checking to ensure that any obvious errors that could stop a
successful mailbox move are detected before the move commences. The
checks include:
Verify that the user account exists and that they have a valid mailbox.
Check that the user performing the move has sufficient permissions
to perform the move. This check is done by connecting to both
source and target server.
3.5 Moving users
Check the mailbox quota limit on the target database to ensure that
the mailbox does not exceed the quota.
Stop attempts to move system mailboxes.
Check that the target database is mounted and available to accept the
incoming mailbox.
The Move Mailbox wizard always performs these checks before it
attempts to move a mailbox. If you use the Move-Mailbox shell command,
you can specify the –ValidateOnly parameter to force Exchange to make the
Figure 3.25
The Exchange
2007 Move
Mailbox Wizard
Figure 3.25 illustrates the first two screens of the Exchange 2007 Move
Mailbox wizard.
The first choice to make is what are the target server and database. As
long as a network link is available, you can move a mailbox. My own
mailbox was transferred from a server in Dublin, Ireland, to a server
in Houston, Texas, during HP’s migration to Exchange 2007 but
clearly, the closer the two servers are in terms of network connectivity
and bandwidth, the faster that the move can progress.
The wizard can continue processing and move the mailbox even if it
encounters some corrupt messages in the mailbox. You can decide
how many corrupt messages the wizard should skip. In Exchange
2003, the wizard had a hard-coded limit of 100, but Exchange 2007
allows you to specify much higher figures (the highest that I have
gone is 1,000). Clearly, if you meet a mailbox that has many corrupt
messages, it is a sign that the underlying mailbox store is probably
suffering from some sort of corruption that deserves your attention.
Chapter 3
3.5 Moving users
Note that if the wizard skips any corrupt messages, it generates a
report to show you what those messages are.
You can schedule moves to occur at a particular time. The System
Attendant tracks and performs scheduled jobs at the requesting time,
allowing you to ensure that mailboxes move when users are asleep!
You can move multiple mailboxes concurrently by selecting multiple
mailboxes and then taking the “Move Mailbox” option. Of course,
you can achieve further acceleration by using multiple workstations
to move mailboxes. The Move Mailbox wizard uses up to four
threads per session to move mailbox content (you can increase the
number of threads if you move mailboxes with the Move-Mailbox
command). Figure 3.26 shows how to set a schedule through the wizard and how EMC reports details of a multi-mailbox move as it
Figure 3.26
Setting a schedule
and multiple
mailbox move
Depending on system load, moving large mailboxes (> 100 MB) will
take between five and ten minutes to perform if the two servers share the
same LAN, up to approximately 500 MB an hour. Moving a mailbox is a
binary operation—it either works or not. If a failure occurs somewhere
along the process, you have to start again. Note that you need Exchange
administrator permission for the administrative group for both the source
and target server to be able to perform the move. Always make sure that you
ask the mailbox owner to log off before commencing the move, and ensure
that enough disk space is available to accommodate all the transaction logs
that the move operation generates. Each transferred message is a transaction
for both the source and target databases, so the disk that holds the transaction logs will handle a substantial I/O load during the transfer. One small
thing that occasionally trips people up is the fact that when you move mailboxes, the contents of the deleted items cache remain behind in the source
3.5 Moving users
database. Therefore, a user is not able to retrieve a deleted item after their
mailbox is moved.
If users run Outlook 2003/2007 and have a cached Exchange mode
mailbox, it is possible for them to continue working in a disconnected state
while Exchange moves their mailbox. Exchange will not deliver new messages
to the mailbox and queues them on the source server while the move proceeds. When it has moved the mailbox, Exchange releases the messages and
delivers them from the queue. Exchange notifies Outlook when the moved
mailbox is ready to use, and Outlook then switches across (transparently) to
access the new server and then transmits any messages that the user sent during the mailbox move. While it is possible to move Outlook 2003/2007 users
who do not work in cached Exchange mode while they are online, it is definitely something that tempts fate and invites problems. Therefore, it is
always a good idea to have users disconnect before the move starts and have
them set their connection status to “Work Offline.” In addition, if a user
attempts to log on during a mailbox move (as opposed to continue working
during the move), the Store detects the connection attempt and logs event
9660, identifying the user name and distinguished name. The Store also flags
an error to tell the user that a mailbox move is under way. Finally, you should
ask users to wait for 20 minutes after their mailbox is moved to a new server
before attempting to connect. This step allows any cached credentials to
expire and forces Outlook to refresh its information about the user’s mailbox
and so discover that the mailbox is now on a new server.5
After Exchange has moved the mailboxes, it is best to leave a period of
at least 15 minutes before you attempt to access them on the new server.
This is because the Store process caches information about mailboxes so that
it does not have to go back to Active Directory each time a user requests to
access their mailbox. If a mailbox is not accessed in 15 minutes, the Store
removes the data from its cache. When you move mailboxes, Active Directory replication may create a slight time-lag before it reflects the true location of the mailbox across all domain controllers, which may then lead to a
situation where a user attempts to log on to their mailbox but cannot
because Active Directory is still pointing to the old location. The Store
caches the incorrect login data and will refer to this data if the user makes
another attempt to access their mailbox. The net result is confusion and
frustration, so it is best to wait for replication to occur before you attempt to
access a moved mailbox.
After a mailbox move, Exchange redirects Outlook Web Access connections to the new server the next time that a user attempts to logon to their
mailbox, but you have to update settings manually for IMAP4 and POP3 cli5.
Some installations have reported that Outlook 2003/2007 clients that work with RPC over HTTP have difficulty connecting to Exchange 2007 after a mailbox move. It may be that Microsoft fixes this problem in a service pack, but if you
encounter the issue, the easiest solution is to delete the user’s profile and recreate it to point at the new server.
Chapter 3
3.5 Moving users
ents. In some cases, administrators report that newly moved mailboxes use
larger quotas (in other words, the same number of messages occupy more
space) when hosted by Exchange 2007 than on a legacy server. This is anecdotal evidence that is not reported consistently from implementation to
implementation and is probably highly dependent on the type and format of
items stored in a mailbox. In any case, it is a good idea to check mailbox quotas before and after the move to ensure that users can keep working.
Logging mailbox moves
In addition to the prevalidation checks, Microsoft has improved the logging
of mailbox moves. All movements are logged in the application event log
(Figure 3.27), including the deletion of any Active Directory objects such as
source mailboxes involved in cross-organization moves. Exchange 2003 generates XML reports (Figure 3.28) for mailbox moves and the detail is
improved in Exchange 2007 because the report now includes items such as
the domain controllers and Global Catalog servers referenced in the move,
the mailbox size, the number of mailboxes moved, time taken, and so on. By
default, Exchange puts the move mailbox logs in the \Logging directory
under the installation directory, but if you use the Move-Mailbox command,
you can specify the –ReportFile parameter to select a different location.
Figure 3.27
A mailbox move
is logged
3.5 Moving users
Figure 3.28
Move Mailbox
XML report
In addition to the XML report, Exchange 2007 generates a detailed text
format log of all operations performed in the move that is helpful when troubleshooting failures. This log is stored in the same directory with a file name
of Move-MailboxHHMMSS.log. An example extract of what you can expect
to find in the troubleshooting log is shown below:
[12/18/2006 1:03:26 PM] [0] [Bond] Updating attributes.
[12/18/2006 1:03:26 PM] [0] [Bond] Messages moved. Closing connections.
[12/18/2006 1:03:26 PM] [0] [Bond] Trying to unlock mailbox:
pguidMdb: {903F6615-A656-4F08-84FD-26069DEFFBC3}
pguidMailbox: {E33F05B7-59DC-4961-B9B1-A6E1EA4867C9}
[12/18/2006 1:03:26 PM] [0] [Bond] Mailbox was unlocked successfully.
[12/18/2006 1:03:26 PM] [0] [Bond] Deleting mailbox from mailbox database:
pguidMdb: {DC280469-1A69-426E-BB84-DF3406FE47FC}
pguidMailbox: {E33F05B7-59DC-4961-B9B1-A6E1EA4867C9}
[12/18/2006 1:03:26 PM] [0] [Bond] Mailbox '{E33F05B7-59DC-4961-B9B1A6E1EA4867C9}' was deleted successfully.
Chapter 3
3.6 Using distribution groups
[12/18/2006 1:03:26 PM] [0] [Bond] The operation has finished.
[12/18/2006 1:03:26 PM] [0] Searching objects "
Users/ATG/James Bond" of type "ADUser" under the root "$null" using "".
[12/18/2006 1:03:27 PM] [0] Searching objects ""
of type "Server" under the root "$null" using "".
[12/18/2006 1:03:27 PM] [0] [Bond] Before applying RUS, the proxy addresses are:
smtp:[email protected]
SMTP:[email protected]
[12/18/2006 1:03:27 PM] [0] Applying RUS policy to the given recipient
" Users/ATG/James Bond" with the home Domain
Controller "".
[12/18/2006 1:03:27 PM] [0] The RUS server to apply policies on the given
recipient is "".
[12/18/2006 1:03:27 PM] [0] [Bond] After applying RUS, the proxy addresses are:
smtp:[email protected]
SMTP:[email protected]
Using distribution groups
As shown in Table 3.2, Windows supports four types of groups. For email
purposes, you need to mail-enable groups before users can send messages to
the members of the group. Groups also hold security principals, so you can
use them to control access to resources. However, we will focus on the use of
groups for email purposes in this discussion.
Universal groups are available anywhere within a forest. In other words,
you can use a universal group to control access to any resource in the forest.
Windows publishes details of universal groups in the Global Catalog, and clients resolve membership of universal groups when they log-on by reference
to a Global Catalog. This step ensures that a client receives a full set of credentials and is able to access resources through membership of any universal
group they may hold.
Somewhat confusingly given the name, Windows does not publish
details of global groups in the Global Catalog. Similar to the definition of
global groups as used in Windows NT, global groups can be present in the
ACL of any object in the forest, but they can only contain members from
the domain that “owns” the group. Domain local groups are the most
restricted as they only apply within the domain in which they are created.
Again, Windows does not publish details of domain local groups in the
Global Catalog. Domain local groups can contain universal or global
groups along with individual users. There is no real value in attempting to
use domain local or global groups with Exchange. Unless you operate
inside a single-domain forest, universal groups are the only logical type of
3.6 Using distribution groups
Table 3.2
Group type
Exchange Group Type
Universal group
MailEnabledUniversalSecurityGroup or
The preferred group type
for Exchange 2007
Global group
Not useful for Exchange
unless in a single domain
Domain local group
Not useful for Exchange
unless in a single domain
Query based group
Group whose membership
depends on the execution of
an OPATH query against
the Active Directory
group to use for email-enabled recipients because you can guarantee that
Exchange will always be able to expand the membership of the group accurately by reference to a Global Catalog. Exchange 2007 is the first version
of Exchange to enforce this requirement.
Here is why the problem exists for Exchange 2000 and 2003. In these
releases, you can use AD Users and Computers to create mail-enabled global
groups or mail-enabled domain local groups. Everything works if you operate in a single domain, but if Exchange operates in multi-domain forests and
mail-enabled objects exist in multiple domains, you can experience problems
when Exchange attempts to expand the contents of groups to build the recipient list for message delivery. Everything will go well and Exchange will be
able to expand the group to determine its full membership if the Global Catalog server that Exchange queries holds a complete copy of all the group
members. However, if Exchange connects to a Global Catalog that has an
incomplete copy (for example, a universal group that contains a link to a
nested domain local group from a different domain), then it is impossible to
resolve the group to its complete membership. Even worse, users will see different results from one Exchange server to another. Enforcing a common
approach to groups is the reason why Microsoft changed Exchange 2007 to
require you to use universal groups. It is also the reason why Microsoft
requires you to operate the Active Directory in native mode. If you have nonuniversal groups in use, you can use the Set-Group command to convert
these groups. For example:
Set-Group –id ‘Exchange Gurus’ –Universal:$True
Chapter 3
3.6 Using distribution groups
This command converts the “Exchange Gurus” group to be a universal
group if it is not already. If it is, you will get an error.
If you want, you can scan for mail-enabled groups that are not universal
with this command:
Get-DistributionGroup –Filter {RecipientTypeDetails –eq
If you add the command to set the group to universal, you can scan and
convert all at one time:
Get-DistributionGroup –Filter {RecipientTypeDetails –eq
‘MailNonUniversalGroup’} | Set-Group –Universal:$True
As you can see from Figure 3.29, Windows also defines groups by type.
If a group is a “security” type, it means that the group holds a security principal and you can use it to control access to resources. A group that is a “distribution” type does not have a security principal, but you can use it with
Exchange as a way to distribute messages, which is very close in style to the
original definition of an Exchange distribution list. Figure 3.29 also shows
that a security group can be mail-enabled (the group has a published email
address that is obscured in the picture). A mail-enabled universal group that
holds a security principal is the most powerful group. It is available anywhere
in the forest, can be used to secure resources, and you can use the group as a
traditional e-mail distribution list. Of course, you do not create groups using
AD Users and Computers when you work with Exchange 2007. Instead, you
can use either EMC or EMS to create and manage both security and distribution groups. All the groups that Exchange 2007 creates are universal.
Because they provide a convenient way to address large numbers of users
in a single operation, groups or distribution lists have always been tremendously important to email systems. Exchange is no different in this respect.
For example, Microsoft IT report that they have some 175,000 groups in use
in their Exchange environment and HP has over 150,000 (90,000 of which
have been created since I last wrote on this topic some four years ago). While
these are two very large deployments of Exchange, they prove the point of
the importance of groups and provide us with a good reason to look into
groups in some detail.
Forming groups
Windows groups are composed of a set of backwards pointers to individual
directory objects (other groups, users, and contacts) that form the group. The
3.6 Using distribution groups
Figure 3.29
properties of a
Windows Group
left-hand screen in Figure 3.30 shows group membership as viewed through
EMC. The right-hand screen shows the membership of the same group, this
time viewed through the ADSIEDIT utility. Here we see that the Active
Directory holds the membership of a group in a single multi-valued attribute
called “Member” and that the attribute is formed by the distinguished name
of each of the members. The AD uses the distinguished name as the pointer
to the records for the individual members when it has to resolve or expand
group membership. Because Windows builds a group through a set of pointers to Active Directory objects you have to be careful if you need to perform
an authoritative restore of the Active Directory to recover from a database
failure, as you may affect group membership if the restore removes some of
the pointers. For this reason, you should take great care to check important
group membership after completing any restore of the Active Directory.
The Active Directory uses attribute-level replication, but any change
made to the membership of a group results in essentially the complete object
being replicated because the membership is held in a single attribute. This is
a true statement for Windows 2000, but the situation is a little different in
Windows 2003, which includes a new mechanism called linked value replication (LVR). LVR addresses some of the problems associated with large distribution groups that span more than a few hundred members. People tend
to leave and join groups on a regular basis, so the “churn” in membership
generates an obvious issue in excessive network traffic to handle replication
of the membership attribute. In addition, because the Active Directory is a
Chapter 3
3.6 Using distribution groups
Figure 3.30
Viewing group
multi-master directory where any replica (on a domain controller) can originate an update, the danger exists that the Active Directory ignores updates
when users make changes to group membership in multiple places at approximately the same time.
In this situation, the Active Directory must reconcile changes coming in
from multiple places without the benefit of human intelligence, and it does
not always make the right decision. In Windows 2000, the Active Directory
is handicapped because it only updates one attribute—the membership—for
each of the changes it must reconcile. Thus, the change originating from the
New York domain controller looks the same as the change from London.
Which change should win? The Active Directory uses its normal conflictresolution techniques to decide the “winner” (the update that is finally
made). Timestamps eventually help to resolve the conflict, and the Active
Directory uses the last timed update to the group membership and applies
that update to the group. Administrators may only know about a reconciliation snafu when a user complains that they are not able to access a public
folder or that they do not receive email sent to a list. LVR helps to solve the
conflict issue because the Active Directory is now able to resolve changes at
the level of individual members rather than handling the complete group as
one entity.
At this point, it is also worth noting that restoring the Active Directory in
a multi-domain environment may lose some updates applied to any data that
the Active Directory holds in a multi-value attribute. An authoritative restore
will recover any links associated with the domain that you perform the restore
for, so the problem only arises for links that tie objects together across multiple domains. Group membership is the obvious example, but Exchange uses
3.6 Using distribution groups
multi-value attributes for data such as organizational structures (reporting
relationships between manager and employee, for instance). When you restore
data, you essentially roll back to the point in time when the backup occurred.
Many changes may take place between that time and the current time and
there is no guarantee that the Active Directory replication mechanism will be
able to reconcile everything accurately because (unlike public folders) there is
no concept of a backfill request to allow one replica of the directory to request
a complete refresh from another replica. There is also no way that you can
know whether the Active Directory is in an inconsistent state. For these reasons, it is a good idea to practice restores of the Active Directory and to have a
plan to take frequent backups of the system state of a domain controller from
every domain in the forest, as you will rely on this data during restores. You
may also want to check with companies that build Active Directory utilities to
see if they have a program to help with this problem. With all of this in mind,
it makes sense from a pure Active Directory perspective to limit the amount of
updates applied to groups whenever possible, especially for universal groups as
the Active Directory replicates them to every Global Catalog in the forest.
However, universal groups are very important to Exchange, as they are the
only way to implement secure access to public folders and as email distribution lists within multi-domain implementations.
You can reduce the replication load by building the membership of universal groups from a set of global groups. This means that the effect of
changes to group membership are restricted to the host domain of the users
or contacts in the group, and the only replication to the Global Catalog
occurs if a global group is added or removed from the universal group. Seeking to reduce replication in this way is a great example of where the needs of
the base operating system do not match the need of applications. If you
include nested global groups in a universal distribution group, only Exchange
servers in the same domain will be fully able to resolve group membership.
This leads to a situation where you can send email to the same group from
accounts in different domains and have messages delivered to different sets of
people! Even worse, neither users nor administrators will be aware of the
problem because no problem exists in purely technical terms and Exchange
signals no error messages. For this reason, it is best practice only to use universal groups with Exchange.
Group changes in Exchange 2007
Up to Exchange 2007, you worked with mail-enabled groups through the
AD Users and Computers console, just like any other Active Directory
object. With Exchange 2007, you should work with groups through EMC to
ensure that the correct set of attributes is stamped on groups when you create
or work with them. Part of the reason for the change is that the Exchange
Management Console (EMC) now performs the work that the Recipient
Chapter 3
3.6 Using distribution groups
Update Service (RUS) does in Exchange 2000 and 2003. Part is because
Exchange exercises much closer control over Active Directory objects when
they are managed completely through the EMC interface than when administrators go behind the scenes and update objects with AD Users and Computers, ADSIEDIT, or any other utility.
Figure 3.31
Comparing the
properties of
two groups
Even in an Exchange 2007 environment, AD Users and Computers
allows you to continue creating distribution groups and you can even assign
them a valid email address. However, any group that you create through AD
Users and Computers is not functionally equivalent to a group created
through EMC. On the surface, everything looks similar, but below—in the
set of properties stamped on an object when you create or update a group
using EMC—things are very different.
For example, compare the properties of the two groups as viewed
through ADSIEDIT shown in Figure 3.31. The “Operations Committee”
group (left) was created with AD Users and Computers. The “Senior Executives” group (right) was created with EMC. Two immediate and important
differences are apparent. The Senior Executives group has values for the
MailNickName and LegacyExchangeDN attributes. If we scroll down
through the list of properties, you would find other differences, such as the
presence of a value for MsExchPoliciesIncluded, MsExchRecipientDisplayType, MsExchRequireAuthToSendTo, and so on. These properties complete
the process of making a standard Windows group into a fully-fledged
Exchange object, and while it is entirely possible for you to set the necessary
properties manually, there is a huge risk of error that then leads to further
difficulties. If you send a message to an object that is not fully mail-enabled,
you will receive an NDR that says something like this:
3.6 Using distribution groups
Diagnostic information for administrators:
Generating server:
[email protected]
#550 5.1.1 RESOLVER.ADR.RecipNotFound; not found ##
The rule is therefore to use AD Users and Computers to work with
objects that are unique to Windows and to use EMC to maintain those
required by Exchange. See page 100 for more information about how to
work with EMC.
Expanding distribution lists
Exchange expands the membership of any distribution group that a user
includes in a message header by checking a Global Catalog to find the preferred email address of each object in the lists. As Figure 3.32 shows, you can
assign the task of group expansion to a particular server. When a server is
defined for group expansion, Exchange routes any message addressed to that
group to the specified expansion server, which then expands the group into
the individual addressees. If the expansion server is not available, Exchange
queues the message until the server becomes available again.
Most groups do not have an expansion server specified, so Exchange can
expand the group on the first hub transport server (or Exchange 2003 server)
that the message is routed through en route to its destination. To expand a
group, the transport engine decomposes it into the set of individual email
addresses for each member in the group and then uses the email addresses to
help determine the optimum routing path for the message to reach each
recipient. All of this work occurs in memory.
Exchange does not update message headers sent to groups with the
details of the individual group members for two good reasons:
Adding email addresses for the group members would increase
the size of the message header. It is much more efficient to transport a single address than a complete set.
Group membership changes over time, so keeping the group
address in the header ensures that Exchange always uses up-todate membership if a user replies to a message.
If you use a personal distribution list to address a message, Outlook (or
the client you use) must expand the contents of the distribution list and add
addresses for all of the recipients to the message header before it can submit
the message to Exchange. You can see the impact of list expansion on the size
Chapter 3
3.6 Using distribution groups
Figure 3.32
Defining a
preferred expansion
for DL expansion
of a message by comparing similar messages sent to a group and a list, each
containing 20 members. The message sent to the group is always far smaller.
In the past, the expansion of large distribution groups (that contain
more than a few hundred members), or those that contain nested groups,
could impose a considerable demand on system resources. Given the speed of
current computers you shouldn’t see any problem unless you use groups that
expand to literally thousands of users. Even so, users can notice a delay when
they send a message to a large list (if they are a member of the list), as
Exchange can take up to a minute or so to deliver a copy of the message back
to the originator.
How many objects can I have in a group?
The AD Users and Computers UI imposes a limit of approximately 5,000
objects for membership of a group, although you can manipulate the member
attribute of a group programmatically to add more entries. However, because
of some technical issues involved in replicating group membership, Microsoft
recommends that you avoid building groups with 5,000 entries. My own
opinion is a little stronger on this topic—the limit is technical, not realistic,
and you create the potential for problems if you insist on using very large
3.6 Using distribution groups
groups. Best practice is to keep group membership under 1,000 to make
everything (membership, replication, update collisions, traffic) a lot easier to
control. This is particularly true in large multi-domain implementations.
Note the use of the word “objects” when referring to group membership.
An object can be a user account or contact, but it can also be another group,
so you can build groups from a collection of nested groups and build very
large logical groups in this manner. Nested groups greatly expand the total
possible user population that you can address through a single directory
entry. However, while it is nice to know that it is technically possible to create a mega-distribution list to address thousands of users at a single keystroke, once a list contains more than 200 users or so, it is also a weapon
waiting for people to use as a form of internal spam.
Windows does not impose a hard-coded limit to the number of groups
that can exist in a forest, but you can run into a situation called “token
bloat” that can prevent users logging on. Token bloat occurs when the number of groups that a user belongs to is so large that Windows cannot fit all
the groups into the security token. Because they are so useful for email, an
Exchange environment is likely to use far more groups than a simple Windows deployment, and you may run into token bloat when you migrate
from an earlier version of Exchange and Windows upgrades the distribution
lists to groups. This is good reason to review group membership and remove
users from groups that they do not need to belong to. Note that you are less
likely to run into a problem with tokens on the Windows 64 platform
because Windows has more memory available to deal with AD structures.
Managing group membership
It is easy to set up new distribution groups but some forget the hard
work to maintain membership. In theory, updates should be automatic and
the Active Directory adjusts group membership as it deletes accounts and
contacts. The normal problems include:
Adding new members: Whom should someone contact if they want
to join the list? An easy way to inform people is to include instructions in the “Notes” section of the list’s properties as this text is visible to Outlook when you look at its properties through the GAL
(Figure 3.33).
Note that a user is only able to manage groups that are part of the
user’s own domain and that you can control the ability for a manager
to update the membership list by an attribute of the list. You cannot
manage groups from other domains because you are using an incomplete copy of the group replicated to a local Global Catalog rather
Chapter 3
3.6 Using distribution groups
Figure 3.33
Details of a list
than the full copy maintained by domain controllers in the owning
domain. In addition, the group maintainer must have the correct
Windows permissions (Allow Read Members and Allow Write Members) to be able to change the membership. You can grant these permissions at either the organizational unit (if the same maintainer
takes care of multiple groups) or individual group level.
The informational text can state whether the potential member
should email a request to the maintainer or take another action to
join the list. Some companies have sophisticated web-based applications to allow users to maintain their own group membership, using
LDAP-based utilities or PowerShell scripts to communicate with the
Active Directory behind the scenes.
You know you have a problem when the list begins to receive “please
let me in” messages from people who want to join as this is a good
indication that your procedures are not well known or do not work.
You need to know who the members of the list are. This is easy for
small lists, but it becomes increasingly difficult as the numbers grow
because the standard tools for list maintenance (through Outlook or
EMC) have no way to generate membership reports. You can use the
Get-DistributionGroupMembership command to list the members
of a distribution group and then export by piping the output to the
Export-CSV command to create a CSV file that you can manipulate
with another program for formatting purposes, such as Excel or
Access. Figure 3.34 shows how Get-DistributionGroupMembership
lists the members of a group. If you want to see all the properties for
3.6 Using distribution groups
each member, pipe the output to the Format-List command. For
Get-DistributionGroupMembership ‘Senior Executives’ | FormatList
Removing obsolete members: If your practice is to disable accounts
and keep mailboxes for a period after their owners leave the company,
you should remove them from all groups to prevent the mailboxes
receiving unwanted copies of email. It is best practice for group owners to review the list regularly to remove recipients who no longer
work for the company. Apart from filling Store databases with messages that no one will ever read, this step prevents the other group
members from becoming annoyed when they send messages to the
group only to receive multiple nondelivery notifications back. The
most common reason for NDRs is that Exchange cannot deliver messages to the mailboxes of the disabled accounts (perhaps because the
mailboxes have exceeded their quotas with all the messages that have
arrived since the owner left the company).
Figure 3.34
Listing group
You can block most nondelivery notifications by setting the “Advanced”
properties (Figure 3.35) of the distribution group to:
Suppress notifications completely;
Send notifications back to the message originator; or
Send notifications to the owner of the group as defined in the “Group
Information” property page.
The easiest option is to suppress nondelivery notifications for distribution lists, but this means that you never see any indication that someone has
a problem—their mailbox quota is exceeded or Exchange makes an attempt
Chapter 3
3.6 Using distribution groups
Figure 3.35
Properties of a
distribution group
to deliver to a mailbox that no longer exists, which could mean that directory
replication is not working. Sending notifications back to the owner is acceptable if that person can take appropriate action. For example, if you allocate
ownership to an administrative assistant and they do not have the correct
Windows permissions to modify group membership, then you will end up
doing the work to maintain the group. Note that these properties are only
valid for a single forest and single Exchange organization. If you operate in
an environment where you use multiple Exchange organizations, the settings
you make in one organization do not replicate to the others and you will still
see notifications unless you turn them off everywhere.
Protected groups (and users)
Sometimes, the SMTP address of a distribution group gets outside a company. Apart from the obvious irritation if spam (otherwise called unsolicited
commercial email) starts arriving for everyone on the list, there is a danger
that unauthorized users could exploit the list to identify personnel within the
company, for example, by a recruiter looking for specific talent. The first
attempt to block messages from unauthorized users was implemented in
Exchange 2000, which allows administrators to configure delivery restric-
3.6 Using distribution groups
tions for groups by checking that it could resolve the sender’s address against
the directory, but spammers can work around this check by spoofing their
identity by inserting a perfectly valid email address in a message’s “From:”
field. For example, you might restrict a group so that only senior executives
can send messages to it, only to be embarrassed when a spammer uses your
chief executive’s email address (which is often published or known externally)
to address the troops. In Exchange 2007, you can put a deny permission
called ms-Exch-SMTP-Accept-Authoritative-Domain-Sender on send connectors that process incoming traffic from the Internet if you want to block
this kind of message. Send connectors are discussed in Chapter 6.
Figure 3.36
Limiting a group to
users only
With Exchange 2003 and Exchange 2007, you can block such access by
only allowing authenticated users (those who possess recognized NT credentials) to send messages to the list. For Exchange 2003, use AD Users and
Computers to select the group you want to protect and set the “From
authenticated users only” checkbox on the Exchange General property page.
For Exchange 2007, use EMC to select the group and access the Message
Delivery Restrictions options from the Mail Flow property page (Figure
3.36). This feature depends on a new attribute that Exchange 2003 introduced for user and distribution group objects (ms-Exch-AuthenticatedOnly), so you can only set authenticated restrictions on a group when working on a server that has the Exchange 2003 or Exchange 2007 management
components installed.
Chapter 3
3.7 Using groups for permissions
You can implement the same restrictions on an individual user’s mailbox. For example, it is reasonably common to create two mailboxes for senior
executives. One is their “working” mailbox, which they and their support
staff use for day-to-day email. Access to this mailbox is restricted to specific
users and groups. The second mailbox is intended for public access so that
anyone can send a message to it. Messages that arrive in the public mailbox
are screened by a member of the executive’s support staff before they are forwarded to the working mailbox for a response.
Using groups for permissions
Because Exchange treats them like any other recipient, mail-enabled security
groups are a useful way of maintaining permissions on public folders, much
along the same lines as delivery restrictions. When you give permission to a
group, all members of the list inherit the permission. The advantage lies in
the fact that it is very much easier to grant permission to a single entity, the
group, than it is to manage separate permissions for each individual in the
group. Better still, when people join a group, they automatically inherit the
permissions held by the group and so have access to all of the public folders
that are available to the group. Later on, when you remove some members
from the group, Windows revokes the permissions they inherit from the
group and there is no danger that someone will gain access to confidential
information after they no longer need this privilege.
Managing distribution groups from Outlook
I use Outlook to manage distribution groups all the time because it is a quick
and effective way to manage group membership. Outlook does not differentiate between distribution and security groups; all it cares about is whether
the group is mail-enabled, and if it is, then Exchange includes the group into
the GAL and so Outlook accesses and manages it.
You need to have the appropriate permissions before you can manage a
group with Outlook. This means that you either are an administrator of the
OU that the group belongs to or an administrator explicitly allocates you the
permission to manage and update the group by updating the group’s properties through EMC as shown in Figure 3.37.
Note that it is also possible to update the group’s maintainer through PowerShell. For example, we could assign the group to Julie Smith and tell people
who to go to if they want to be included in the group with this command:
Set-Group –Id ‘Legal Executives’ –ManagedBy ‘Julie Smith’ –Notes
‘This group is managed by Julie Smith. Please contact her to request
group membership’
3.7 Using groups for permissions
Figure 3.37
Permissions to
manage a group
Once the permissions are set, you should be able to update group membership from Outlook unless your account belongs to a different domain to
the domain that owns the group. The inability to manage groups across
domains, even in the same forest, is a major flaw that has existed in all versions since Exchange 2000. The problem exists because of the way Exchange
proxies connections from some versions of Outlook to a Global Catalog
server through the DSProxy mechanism. DSProxy exists to support older
Outlook clients that assume a direct connection to the directory, as existed
with Exchange 5.5. The Global Catalog that Exchange connects an Outlook
client to contains details of all of the mail-enabled objects in the forest, but
only a partial read-only attribute set of objects from domains outside the
Global Catalog server’s own domain. When a user updates group membership from Outlook, DSProxy relays an NSPI request to perform the update
to the Active Directory, but the NSPI request fails if it goes to a Global Catalog that cannot update the target group (because the Global Catalog only has
a read-only copy of the group). The failure is silent, meaning that the user
will not be aware that the update failed unless they check the group membership afterwards.
Updates to user objects, such as adding delegates (to allow other users to
process email on your behalf ) to your mailbox, can exhibit the same problem
Chapter 3
3.8 Dynamic distribution groups
if Outlook connects to a Global Catalog that does not belong to the same
domain as the target user object. Once again, you attempt to make the
change with Outlook and everything seems to go okay only for the interaction between Exchange and the Global Catalog to fail. The delegate information remains unaltered and the user is unaware of the problem unless they
check afterwards that Exchange updated the Active Directory.
It is best to manage groups with accounts that are in the same domain
and never to create an Active Directory design that places users from one
domain into a situation where they are likely to connect to Global Catalogs
that belong to another domain. This is one reason why some experts recommend that you place all user accounts in one domain.
Dynamic distribution groups
Large companies like HP use distribution groups (or lists) extensively to create email-addressable objects that reach many people who share a common
need or interest. However, it is typical to find a proliferation of distribution
groups in any large email deployment, leading to a situation where people
spend a lot of time updating the groups to make sure that the right people
receive the right data all the time. For example, the HP GAL includes over
150,000 distribution groups, all of which have different maintainers. Some
maintainers are very conscientious about how they maintain membership
and the group is always accurate; others are less so and you can never be sure
that the group contains everyone that it should.
Conscientious maintainers remove members who no longer want to
receive messages sent to the group or those who have left the company. Others simply leave the groups to degrade gently until they are no longer wanted.
At that point, system administrators have to clean up the directory by removing redundant groups. However, cleaning up the directory or maintaining
group membership are low-priority tasks for hard-pressed administrators, so
they are easy tasks to overlook. The result is a directory cluttered with obsolete groups and groups that do not reach the right membership. Exchange
2003 introduced support for query-based distribution groups to help automate list maintenance. Only Exchange 2003/2007 and Exchange 2000 (SP3
onwards) can use dynamic distribution groups as the Active Directory must
have the necessary schema extension to support these lists (object type msExch-Dynamic-Distribution-List).
The only difference a user sees between a dynamic group and a normal
group is that you cannot view the membership of a dynamic group from its
GAL properties. This is quite logical because the GAL has no way of resolving
the query to determine group membership. From an administrative perspective, you cannot use dynamic groups to control access to objects like connectors and you cannot nest these groups inside other distribution groups.
3.8 Dynamic distribution groups
Changing filters and conditions for dynamic
distribution groups
When users submit a message addressed to a dynamic distribution group, the
message travels from the originating mailbox to a hub transport server where
the transport system identifies that the group requires a query to determine
the addresses to receive the message. The categorizer component then
expands the group membership into individual addresses by using the query
to interrogate the Global Catalog server. The categorizer can then use the
group membership to create the list of recipients and decide how to best
route the message. The need to execute the query to build a list before message submission increases load on both the hub transport server and the Global Catalog server and delays the message slightly, but this is usually a
justifiable trade between usefulness and overhead.
In Exchange 2007, Microsoft renames the query-based distribution
groups that they introduced in Exchange 2003 to be “dynamic” distribution
groups. The name may have changed, but the resolution of the group into
individual routable addresses still depends on queries that Exchange executes
against the Active Directory. The big difference is that Exchange 2003 querybased distribution groups use queries that are expressed in LDAP format
while Exchange 2007 dynamic distribution groups use queries in OPATH
You may ask the question why Microsoft has changed query types from
LDAP to OPATH in Exchange 2007. After all, this is a move away from a
standard for queries that is well understood and supported by many products, including utilities that you can use to build queries against LDAP-compliant directories. It seems strange that Microsoft would want to move away
from LDAP to use OPATH and require administrators to master new skills.
OPATH isn’t difficult, but it is something more to learn.
The complete answer is probably pretty complicated, but I think that
the history of how Exchange queries the Active Directory reflects the components that the developers had to work with when they introduced querybased distribution groups in Exchange 2003. They didn’t have the time to
create a wizard or other UI to help administrators create new groups, so they
used the Active Directory LDAP picker to create the criteria for the queries.
The groups were managed through AD Users and Computers, just like all of
the other mail-enabled objects used by Exchange, and they were forced to use
LDAP because it was the only directory query language available. Everything
worked, but the implementation was flawed. Four years later, the Exchange
developers have a completely new environment to work in. First, PowerShell
has arrived to provide the scripting and automation language for Windows
and application such as Exchange. As discussed in Chapter 4, Exchange 2007
does an impressive job of extending the basic PowerShell framework to
Chapter 3
3.8 Dynamic distribution groups
include support for mail-enabled objects, servers, and other messaging components. OPATH is the filtering syntax used by PowerShell, so it is a natural
progression for Exchange 2007 to embrace OPATH as its preferred syntax,
especially as it also removes the need for administrators to learn and master
the somewhat obscure and opaque queries that LDAP often generates. The
change to OPATH allows administrators to learn one query syntax and use it
everywhere, whether they want to write one-line PowerShell commands that
filter objects or complex filtering expressions used in scripts or to drive
objects such as dynamic distribution groups, email address policies, address
lists, and global address lists.
Query-based distribution groups never made much of an impact in
many organizations that deployed Exchange 2003, so now is as good a time
as any to change the filter syntax. You can use the LDIFDE utility to gain an
insight into the information that the Active Directory holds about dynamic
distribution groups. Figure 3.38 shows an edited version of LDIFDE output
generated for a group by running the command:
C:> LDIFDE -v -d "CN=Exchange Power Users,
OU=Groups,OU=Exchange Users,DC=hpqbox,DC=com" -f x.tmp
Figure 3.38
LDIFDE output
for a dynamic
distribution group
dn: CN=Exchange Power Users,OU=Groups,OU=Exchange
Users,DC=hpqbox, DC=com
objectClass: msExchDynamicDistributionList
cn: Exchange Power Users
instanceType: 4
displayName: Exchange Power Users
mailNickname: Exchange_Power_Users
name: Exchange Power Users
mail: [email protected]
((RecipientType -eq 'UserMailbox' -and Department -eq
'Exchange') -and -not(Name -like 'SystemMailbox{*'))
msExchRecipientDisplayType: 3
As you can see, many of the attributes that you expect for any other distribution group including email addresses also exist for dynamic groups. The
big difference is the value of the msExchQueryFilter attribute, which holds
3.8 Dynamic distribution groups
the OPATH filter that the Active Directory resolves to determine the membership of the list. The msExchDynamicDLFilter holds the equivalent LDAP
query to allow an Exchange 2003 server to expand the group membership.
While the effects of the two filters are the same, the difference in syntax can
be a learning hurdle for administrators who want to convert older querybased distribution groups to new dynamic distribution groups. The LDAP
filter is a read-only property that also allows administrators who are experienced with LDAP to compare and understand the syntax of the OPATH filter. Exchange generates the value of the LDAP filter when it creates the
OPATH filter and you should not attempt to set the LDAP filter with other
tools as you may generate incompatible or unusable results.
A note on OPATH
Dynamic distribution groups are the most obvious opportunity for Exchange
administrators to use OPATH. Exchange uses the same filtering behavior for
email address policies, address lists, and global address lists, so clearly
OPATH is a big thing for Exchange administrators to get to know (and
maybe love). All of these objects support precanned filters and all allow you
to build custom filters using the same syntax and filtering properties. If you
learn how to build OPATH filters for dynamic distribution groups, you can
use the same techniques with the other objects.
Some basic points about OPATH syntax for building custom recipient
filters are:
OPATH requires a hyphen before –and, –or, and –not operators.
Comparison operators include –eq (equal), –ne (not equal), –lt (less
than), –gt (greater than), –like (like), –ilike, and –notlike. –Like and
–notlike are wildcard string compares. –iLike and –inotLike are case
The RecipientFilter property specifies the OPATH query specified in
the syntax given above and including the properties and values that
we want to filter on. Specify the filter within braces. For example,
{Office –eq “Dublin”} means that we want to filter on objects whose
“Office” property is set to “Dublin.” When you create a dynamic distribution group through EMC, Exchange automatically generates the
necessary filter and stores it in the RecipientFilter property.
The RecipientContainer property specifies the scope of the query
within the Active Directory. For example, if you pass “Exchange
Mailboxes” in this property, the query covers the “Exchange Users”
organizational unit and all its child organization units below.
Chapter 3
3.8 Dynamic distribution groups
The IncludedRecipients property allows you to specify the type of
recipients to include in the query. For example, if you pass “MailboxUsers” in this property, the query includes only mailboxes.
Do not worry too much about how to construct OPATH queries for now,
as we will review many examples of filters in action when we enter the world of
PowerShell in Chapter 4. For now, all we have to realize is that you have to
update older LDAP filters created for dynamic distribution lists in Exchange
2003 to use OPATH filters if you want to support them in Exchange 2007.
You cannot edit a query-based distribution group created with Exchange 2003
with EMC or EMS unless you use the Set-DynamicDistributionGroup command to force an upgrade of the group using the –ForceUpgrade parameter. At
this point, you might find that you need to provide a new query because
Exchange can’t automatically convert the LDAP query to OPATH format.
Note that whenever you upgrade a filter to use OPATH format, Exchange
automatically translates the OPATH filter into LDAP syntax so that legacy
Exchange servers can continue to use the query.
As you migrate to Exchange 2007, the best plan is to remove the old
query-based distribution groups and replace them whenever possible with
new Exchange 2007 dynamic distribution groups. Because the Active Directory properties for both types of groups contain queries that Exchange 2003
and 2007 can execute, you can use both types of group during the migration
period. Over time you will want to phase out the Exchange 2003 servers and
will therefore lose the ability to manage the older query-based distribution
groups, so you will need to change them over. In some cases, the process to
create new equivalent dynamic distribution groups is a matter of point and
click through the Exchange 2007 Management Console; in other cases where
complex queries are required to build the membership for dynamic groups,
you will need to create and manage the new groups through PowerShell.
A new UI for dynamic groups
At the same time as they moved to OPATH, the Exchange team was designing a new administration GUI. This allowed them to get rid-of -components
that they can’t control, like AD Users and Computers. It also allowed them
to focus on the question of how to help administrators create well-structured
and high-performing queries and to avoid some of the overly complex queries that were generated in Exchange 2003.
When you create a dynamic group for Exchange 2003, you have a huge
amount of flexibility in creating the query used to expand group membership. The Active Directory query builder allows you to construct very sophisticated and precise searches against the Active Directory, but on the other
hand, an unsuspecting or careless administrator is able to build queries that
3.8 Dynamic distribution groups
take hours to execute. For example, using the Active Directory Finder, you
can create Exchange 2003 dynamic groups based on queries that examine:
Objects in a particular department or set of departments.
Users who have mailboxes on a specific Exchange server.
Users and contacts who have a specific title (for example, all marketing managers).
Identify users who are temporary employees or full-time employees,
using a customized attribute to locate the members of either set. You
might like to restrict the ability to send to a group to certain users or
groups (see Figure 3.36). For example, you could limit the ability to
send messages to full-time employees to authenticated users so that
someone outside the company cannot send a message to all employees.
Objects in a particular country.
Queries that use values in the set of customized attributes reserved for
your own deployment purposes. For example, you could store
employee job codes in one of the customized attributes and search
against these values.
You can also combine searches for values in different attributes. For
example, find everyone named Tom who has a mailbox on a specific server
and lives in Ireland. The Active Directory is able to resolve such a query but
it will take some time. Real-life experience showed that it was all too easy for
Exchange administrators to fall into the trap of creating queries that stressed
servers as the Active Directory attempted to build lists of thousands of
matching objects followed by Exchange sending each object a copy of a message. You could create a query that ended up addressing a message to every
user in your organization—something that is probably okay for anyone to do
inside a small company, but definitely a problem if you have tens of thousands of recipients. The problem was compounded by the fact that the processing that ensued after a user sent a message to a dynamic group was a
black hole and administrators could not get any insight into what was happening or their servers, nor could they halt the processing of a dynamic
group after a user submitted a message.
Flexibility often results in complexity, and in turn, can lead to some of
the problems discussed above. The new wizard-driven UI in the Exchange
2007 EMC leads administrators through the steps to create dynamic distribution groups based on three questions. The answers to these questions
determine the OPATH query that the wizard generates.
Chapter 3
3.8 Dynamic distribution groups
What is the organizational scope for the recipients that should be
included in the query? The scope is established by an organizational unit in a domain and all of its child organizational units.
Scope cannot extend across domain boundaries, so if you need
groups that cover multiple domains, you need to create separate
groups for each domain and then combine the groups together
into a “super group.”
What sort of recipients are included in the query? Are they just
mailboxes or can mail-enabled contacts be included as well?
What additional properties of the recipients should be used to
select the right people?
The aim of the GUI is to enable administrators to create the vast majority of all of the dynamic distribution groups that they will need by providing
enough flexibility to handle most scenarios. The filters that the wizard creates
are “precanned” rather than “custom” and are tuned for performance. You
can only edit precanned filters through the wizard, so if you need to make a
change to a filter that can’t be done with the options presented by the wizard,
you will have to delete the original group and create a new one through PowerShell and specify the necessary recipient filter then.
Creating New dynamic groups
Before creating a new dynamic group, you should know what purpose you
want the group to serve as well as the attributes that you will use to query the
Active Directory to build the group. In an Exchange 2003 organization, you
create dynamic distribution groups through the AD Users and Computers
console. After you install Exchange 2007 on a server, the option to create a
dynamic (query) distribution group still exists in AD Users and Computers,
but the necessary add-in code that allows AD Users and Computers to create
new mail-enabled objects that was previously installed by Exchange 2003 is
missing on an Exchange 2007 server. You can choose the option, but you will
not be able to create the group. Some administrators like to create one or
more organizational units (OU) to hold the query-based groups instead of
holding them with other objects in “regular” OUs. For example, you might
call this OU “Dynamic Groups” to make its purpose obvious. This decision
is a matter of taste and policy and is not a requirement to use these groups.
To create a new Exchange 2007 dynamic group, open EMC and click
on Distribution Groups under the Recipient Configuration node (Figure
3.39) to launch the new object wizard. The first step is to select the OU
where you want to create the new object along with its name and alias. It is a
good idea to create groups in a logical structure within the Active Directory
3.8 Dynamic distribution groups
Figure 3.39
Creating a new
distribution group
and you may well want to keep all the dynamic groups together in a specific
OU. You then decide what type of objects will be included in the query. In
most cases, you will probably want to address recipients with Exchange mailboxes although you can also select other object types such as mail-enabled
contacts or equipment mailboxes. The next stage is to create the LDAP query
that determines group membership. Exchange will execute against the Active
Directory to group membership. It is at this point that the biggest change
occurs between Exchange 2003 and Exchange 2007.
The use of a wizard-driven GUI may make you assume that Exchange
2007 dynamic groups are less flexible than their Exchange 2003 counterparts. This is true to some degree (but any gap can be addressed by creating a
dynamic group through PowerShell). In any case, it is possible to create
dynamic distribution groups in Exchange 2007 that are flexible and powerful. The major difference is that the UI restricts you to creating relatively
simple and well-bounded queries for dynamic distribution groups, but at
least you’ll know that the queries will work and you won’t have to grapple
with OPATH syntax as Exchange will generate the necessary code for you.
Microsoft’s decision to restrict the ability to build dynamic groups
based on simple queries through EMC is easier to understand when you
reflect on the type of dynamic groups that administrators actually create. In
most cases, the queries are reasonably straightforward. Microsoft analyzed
the queries and realized that the majority can be satisfied by the following
Chapter 3
3.8 Dynamic distribution groups
Figure 3.40
Setting query
A specific organizational scope (for example, a domain, or an
organizational unit and its children).
A specific collection of recipient types (for example, mailboxes,
groups, contacts, or any mail-enabled object).
Checks against a number of popular Active Directory fields that
are most commonly used to create dynamic groups:
a. Recipient is in a specific State or Province (check against
the StateOrProvince property in Active Directory). This
value is stored in the ConditionalStateOrProvince property in the filter.
b. Recipient is in a specific Department (check against the
Department property in Active Directory). This value is
stored in the ConditionalDepartment property in the filter.
c. Recipient is in a specific Company (check against the
Company property in Active Directory). This value is
stored in the ConditionalCompany property in the filter.
d. Checks against any of the 15 customized attributes
(CustomAttribute1 through CustomAttribute15) created for use with Exchange. These values are stored in the
ConditionalCustomAttribute1 through ConditionalCustomAttribute15 properties in the filter.
3.8 Dynamic distribution groups
Figure 3.41
Previewing the
results of the query
The wizard helps administrators to create dynamic groups based on the
conditions described above. Figure 3.40 illustrates the dynamic group wizard
in action. In this case, we have already decided the scope for the query and
the collection of objects that we want to select. In this step of the wizard we
have the opportunity to refine the collection further by comparing values
against the three Active Directory properties that we just discussed. In all
cases, you can specify multiple values, such as looking for recipients who are
in both the “Legal” and “Law” departments. You can also combine the different criteria. For example, the query specified in Figure 3.40 searches for any
recipient in the four countries that constitute the United Kingdom provided
that they also have HP or Hewlett Packard specified in the company field.
When you have defined the query parameters, you can test a query with the
“Preview” button. Exchange executes the OPATH query and outputs the
result as shown in the left-hand screen of Figure 3.41. It’s a good idea to test
the query to ensure that it generates the expected list of names, if only to
avoid the possibility that users send messages to unintended recipients. Of
course, the membership of dynamic groups change as administrators update
the underlying Active Directory objects and can only function as you expect
if data is correctly populated and maintained in the directory.
After it creates the new dynamic group, EMC displays the PowerShell
command that it uses. You can use CTRL/C to copy this command to review
it or to use the code as the basis for scripting the creation of future dynamic
distribution groups. For example, here’s the code that EMC generated to create the group that we just reviewed:
New-DynamicDistributionGroup -Name:'UK employees'
-DisplayName:'UK employees' -Alias:'UK_employees'
Chapter 3
3.8 Dynamic distribution groups
-IncludedRecipients:'MailboxUsers' -Company:'HP','Hewlett Packard'
–StateOrProvince:'England','Scotland','Northern Ireland',‘Wales’
The OPATH filter for the group is specified in the –IncludedRecipients
parameter. If you do not believe the query conditions that you see through
EMC, you can check behind the scenes to examine the queries that Exchange
generates by looking at the Active Directory group object with ADSIEDIT.
The LDAP query is held in the MsExchDynamicDLFilter attribute, as shown
in the right-hand screen of Figure 3.41 and we can see the OPATH filter by
looking at the msExchQueryFilter attribute. It is very easy to make a mistake
if you update attributes through ADSIEdit, so it is not a good idea to
attempt to change either the OPATH or LDAP filters through ADSIEDIT
(and with Exchange 2007, the OPATH filter is the one that is important),
even if you are the world’s best and most fluent expert in these syntaxes.
The thought might now cross your minds as to what Exchange 2007
does with the LDAP query that you established when you created dynamic
groups in Exchange 2003 if you subsequently edit the groups using the
Exchange 2007 EMC. The answer is that the Exchange 2007 EMC can only
generate simple queries and cannot display the kind of complex queries that
administrators often built using the Active Directory finder. You therefore
cannot update older query-based distribution groups using EMC and have to
update them with EMS. If you want to create a dynamic distribution group
that is functionally equivalent to a query-based distribution group, you will
have to translate the LDAP recipient filter contained in the msExchDynamicDLFilter attribute of the group and write the equivalent OPATH recipient filter into the msExchQueryFilter attribute with the SetDynamicDistributionList command. For this reason, you should review the
use of query-based distribution groups in Exchange 2003 and determine how
you can create equivalent Exchange 2007 dynamic distribution groups with
new OPATH queries as replacements during your migration to Exchange
2007. Remember that it is possible to make a mistake and create queries that
generate different results for Exchange 2003 and the new Exchange 2007
groups, so some careful testing is required.
Note that Exchange 2007 limits the recipients that can be included in a
dynamic distribution group to a single domain. In other words, if you operate in a multi-domain forest and want to create and use dynamic distribution
groups, you will have to create separate groups for each domain that contains
recipients that you want to include and then create a separate “all-in” normal
distribution group that contains all the dynamic distribution groups for the
different domains.
3.8 Dynamic distribution groups
Using dynamic Distribution groups
Outlook 2007 and Outlook Web Access 2007 clients support dynamic
groups completely. The support in the 2003 versions of these clients is less
complete. For example, while you can view the dynamic groups in the Outlook 2003 GAL, if you view their membership, the list appears to be blank.
By comparison, if you view the membership of a dynamic group through
Outlook Web Access 2007, you see all the members. It is possible that the
masking of group members will cause users some concern, but this is an easy
point to cover in training, as long as your users accept that some magic happens behind the scenes to get their messages to the right recipients. Figure
3.42 illustrates what you can expect to see for a dynamic group using Outlook Web Access 2007 (left) and Outlook 2003 (right).
Because a dynamic group only “exists” when an application like
Exchange resolves its membership against the AD, it follows that you cannot
maintain group membership in the same way as users can update membership of regular groups by selecting the list from the GAL and then modifying
its properties using Outlook. You have to make any change to a dynamic
group through EMC.
Outlook Express and other non-Microsoft POP3/IMAP4 clients do not
support dynamic groups, so these clients have some problems when you
attempt to address or send email to these groups. For example, you cannot
use the CTRL/K feature with Outlook Express to check the name of a
dynamic group against AD. In addition, the Outlook Express interface to
AD (a straightforward LDAP search) cannot find these entries, so it is best to
create personal contacts that include the dynamic group’s SMTP address if
you want to use dynamic groups often. You can then use the contact or type
Figure 3.42
Viewing a dynamic
group from
Outlook Web Access
2007 and
Outlook 2003
in the SMTP address into a message header and Exchange will safely deliver
the message by expanding the group membership after it resolves the address
on the incoming message against the AD.
Chapter 3
3.9 Mailbox quotas
Dynamic distribution groups cannot hold Windows security principals
so you cannot use them to control access to public folders or other resources
such as file shares. In a similar manner, you cannot use dynamic groups to
place restrictions on objects like connectors. In addition, you cannot nest
dynamic groups and cannot add them to global or universal distribution
groups: they stand on their own. Finally, the data captured by Exchange to
chart the progress of a message sent to a dynamic group in a server's message
tracking logs is the same as for regular groups.
Dynamic distribution groups are a useful feature that enables administrators to create self-maintaining groups. Like anything that depends on a
directory, the accuracy of directory information will dictate the effectiveness
of dynamic groups in practice. For example, if you want to create a dynamic
group to address all users in a particular country, the group can only be successful if you identify every user in that country by updating their user
objects with the necessary information in the directory.
Mailbox quotas
Mailbox quotas have grown from the 25–30MB typically assigned to corporate users on an Exchange 4.0 server to a point where many companies consider a 1GB mailbox to be a nice round figure to use for a mailbox quota and
5GB is not a very unusually large mailbox. One of Microsoft’s big reasons for
going to the 64-bit platform is to allow users to have bigger mailboxes. In
short, Microsoft recognizes that we are human packrats who cannot survive
within the confines of a 100MB mailbox and need far more space to hold all
the important information that we accumulate through work and personal
activities. The truth, of course, is different. Users need larger mailboxes
because they are not disciplined enough to clean out their mailboxes and
keep them somewhat under control, and anyway, storage is cheap, so what is
the problem with having a large inbox? It’s hard to get agreement on what
people consider to be a large mailbox or rather a large quota. A user thinks
that their mailbox is large enough when they don’t get messages to say that
their quota is exceeded; an administrator thinks that mailboxes are too large
at this point because they are responsible for backing up mailbox content to
protect the data; and managers often look at cost and don’t think about mailbox sizes at all!
The real reason why users and administrators struggle to agree on mailbox quotas may lie between lack of discipline on the user side and the time
that administrators spend figuring out how to manage mailbox data. In some
cases, the lack of user discipline may actually be due to lack of time for users
to be able to review and clean out old mailbox content. If this is the case,
then perhaps administrators should concentrate on growing mailbox quotas
gracefully to meet user demand so that the Exchange servers are not stressed,
3.9 Mailbox quotas
that backups can proceed quickly, and that data can be restored efficiently in
case of disasters. This is perhaps the core reasoning behind Microsoft’s argument that they need to support very large mailboxes in Exchange 2007.
If we take the opportunity presented by cheaper storage and the ability
of Exchange 2007 to manage more data to allocate larger storage quotas for
user mailboxes then perhaps we can eliminate the security problem that PSTs
pose. The limitations of PSTs are well known, yet users continue to create
these files and populate them with information because they cannot keep the
data on the server as their mailbox quotas are too small. From a philosophical
perspective, I would cheerfully trade server storage to eliminate PSTs because
I could then:
Assure users that they can access their mailbox information from a
variety of clients instead of being restricted to Outlook.
Assure the legal department that the Exchange deployment can satisfy any legislative requirements for the company to achieve compliance because all messages stay on the server and can be captured in a
journal on the server instead of some being delivered to PST inboxes.
Provide better resilience to failure than you can get from any file that
is hosted on a local hard drive of a workstation or a laptop.
Provide better protection against viruses because the anti-virus scanners can access all messages and attachments in server mailboxes. This
is much preferable to attempting to scan every PST in the organization (which can be done, if the anti-virus scanners can gain access to
all the PSTs).
Provide better all-round security to data because information in mailboxes is far harder for someone to break into than a PST, which can
be opened (even if it is password protected) very easily.
The list goes on (and we discuss this topic in detail in Chapter 7), but
the point is that Exchange 2007 presents administrators with an opportunity
to change user behavior for the better. However, to be effective, you will have
to use a group policy setting to disable PST use from Outlook. The answer
lies in the DisablePST registry value that Outlook 98 (and all versions from
then) supports. This value is under the HKEY_LOCAL_MACHINE\Software\ Microsoft\Office\11.0\Outlook subkey (this is for Outlook 2003, use
10.0 for Outlook XP, 9.0 for Outlook 2000, and 12.0 for Outlook 2007).
You create the value if it is not present and set it to 1 to disable PST access.
As you might imagine, the default is “0” to allow access. The net effect is that
DisablePST turns off the ability to archive to a PST, to export or import data
to and from a PST, and to create a new PST. Smart users can get around the
Chapter 3
3.9 Mailbox quotas
disabling block by opening an existing PST through the File.Open.Outlook
Data File command, so a complete solution to the issue requires you to use
group policy to disable the Outlook Data File command by blocking the ID
for the command (5576). This is a little complex, but it works and it stops
users messing with PSTs, which is what you want to achieve.
Setting mailbox quotas
Exchange has a structured flow to set and respect mailbox quotas. When you
create a new mailbox, it automatically inherits the default storage limits set
for the database that hosts the mailbox, and as you can see in Figure 3.43, the
default limits are quite high. Exchange 2007 is the first version that sets
default limits as these values are left blank for databases created by previous
versions of Exchange. Table 3.3 lists the default storage limits together with
their meaning. The table also lists the EMS shell parameter that you use to
retrieve information about a user’s current quota consumption with the GetMailbox command and the Active Directory property that stores the storage
limits for the user object.
Microsoft set the default storage limits by beginning with an assumption
that users will have large 2GB mailboxes, which is aligned with their general
idea that Exchange 2007 is better able to support such large mailboxes and
that the move toward unified messaging means that more items will end up
in mailboxes. This assertion is true, if you decide to deploy Exchange 2007
Unified Messaging. However, voicemail messages do not generally take up
much extra space in a mailbox, so it’s hard to accept that mailbox quotas
need to increase quite so much to accommodate voicemail.
Table 3.3
Default mailbox storage limits
Active Directory Property
EMS Shell Parameter
Meaning and Default Value
Issue a warning that you are close to the
limit when the mailbox reaches 1.9GB
Stop the user sending new messages when
the mailbox reaches 2GB
Stop the user sending and receiving messages when the mailbox reaches 2.3GB
Flag to control whether mailbox uses
default storage quotas set for the database;
default is “true”
After settling on a default mailbox size of 2GB, Microsoft then calculated the point to start issuing warnings by assuming that an average user
3.9 Mailbox quotas
receives a hundred 50KB messages a day, or about 5MB daily and 100MB
over 20 days. The gap between a warning starting at 1.9GB and a user being
unable to send new messages at 2GB is sufficient for even the tardiest user to
take note of the warnings and clean up their mailbox. They then decided to
leave a gap of 0.3GB before the next restriction kicks in and the user is
unable to receive new mail to accommodate the situation when users take
extended vacations and are unable to clean out their mailbox.
You may decide that Microsoft’s calculations are off and come up with
your own default values for what limitations should be applied to mailboxes
within the organization. In any event, once you have decided on quotas, you
should apply the values by policy before you start to move users to Exchange
2007 so that the new mailboxes inherit the desired quotas. For specific users,
you can set different quotas by editing mailbox properties and clearing the
“Use mailbox database defaults” checkbox and setting new values for warning, prohibit send, and prohibit send and receive. If you want to create a
mailbox that has a truly unlimited quota, then clear the “Use mailbox database defaults” checkbox and set no values for warning, prohibit send, and
prohibit send and receive.
Figure 3.43
How mailbox
quotas are applied
When a user exceeds their mailbox quota, Exchange logs event 8528 in
the application event log (Figure 3.44). Users also see a notification about
their quota problem the next time they connect to Exchange with Outlook
or the premium edition of Outlook Web Access 2007 (Figure 3.45). The
Chapter 3
3.9 Mailbox quotas
light edition of Outlook Web Access, IMAP and POP clients will not
detect a problem with mailbox quota until a user attempts to create and
send a new message.
Apart from scanning the event log regularly to detect users that exceed
their quotas, you can take a more proactive approach by checking with a
PowerShell command. We will get to the details of how to use PowerShell
with Exchange 2007 in some detail in Chapter 4, but for now, you can use a
command like this to check for problem mailboxes on a server:
Get-MailboxStatistics –Server ExchMbxSvr1 | Where-Object
{$_.StorageLimitStatus –eq ‘MailboxDisabled’} | Select
This one-line PowerShell command scans the ExchMbxSvr1 server to
look for disabled mailboxes because they have exceeded quota. We can do a
lot more with PowerShell to help us identify mailboxes that need some
administrative intervention, but we will leave the details of how to create
more powerful scripts until Chapter 4.
Figure 3.44
Event logging
mailbox quota
Most users who struggle to keep under quota will probably admit that
their Inbox and Sent Items folders are stuffed full of messages that they really
do not need to retain. On Exchange 2007 servers, you can use managed
3.10 Email address policies
Figure 3.45
Outlook Web Access
flags a quota
folder policies to help users keep some control of these folders by expiring
and deleting messages after a defined period (for instance, 60 days). Managed
folder policies are discussed in Chapter 8.
3.10 Email address policies
Email address policies dictate the primary email address of mail-enabled
objects in an Exchange organization. Objects can have many different SMTP
addresses that can be used to receive email, but one address is always primary
and is the address that is stamped on outgoing messages. When you install a
new organization, Exchange automatically creates a default email address
policy that creates addresses of the form:
Object alias + organization name
For example, if you are in an organization called and you create
a new mailbox with an alias of “Redmond,” then the default email address
policy stamps the new account with an SMTP primary address of [email protected]
An organization can have multiple address policies and each policy has a
priority so that Exchange knows the order in which to apply the policies. Figure 3.46 shows how EMC lists email address policies. As you can see, the
default policy has lowest priority, meaning that it is applied last if no other
policy covers an object being processed. Exchange creates the default email
address policy automatically for you when you install the organization to
ensure that all new mail-enabled objects receive valid email addresses. The
default policy serves as a backstop, meaning that Exchange applies this policy
to create an email address for new objects if no other policy applies.
Part of creating a policy is to establish a filter that determines what
objects Exchange will apply the policy to. You can create a policy that applies
to every mail-enabled object in the organization, a specific type such as mailboxes or contacts, or a filter based on an attribute such as department or
Chapter 3
3.10 Email address policies
Figure 3.46
Email address
Figure 3.47
Email address
policies in the
Active Directory
country. For example, you can create a policy that creates email addresses
specifically for mailboxes belonging to users in the Legal Department in
You define Email Address policies at organizational level, so they apply
to all Exchange servers within the organization. Exchange stores details of the
email address policies in its configuration data in the Active Directory (Figure 3.47), so the policies are replicated to all domain controllers in the forest.
The Active Directory Topology service is responsible for refreshing policy
information to local mailbox servers so that they can implement the policies
as mail-enabled objects are added or updated. The Recipient Update Service
is responsible for stamping objects with email addresses according to policy
on Exchange 2000 and 2003 servers and the reliance on an additional service
meant that an unpredictable interval (anything from a few minutes to an
hour or more) often occurred before a new object received its email address
and could be used. Exchange 2007 avoids this problem by incorporating pol-
3.10 Email address policies
icies directly into the process that updates the attributes of mail-enabled
objects, so that if you call Set-Mailbox, Set-Contact, or the other commands that work with these objects and update an attribute, Exchange
checks for policy and updates email addresses to comply with policy.
Figure 3.48
Creating a new
email address policy
Apart from deciding what objects the policy applies to, the most important part of an email address policy is the format of the email address that the
policy will generate for the objects. As you can see in Figure 3.48, Exchange
supports a number of different formats for SMTP addresses. The most common format used by large organizations is “first name.last name,” so that’s
the format selected here. Note that the email domain that you specify here
must be in the list of accepted domains defined under the Hub Transport
node under the Organization Configuration in EMC.
You can apply the new policy immediately as the last step in the wizard
or schedule the update to occur at some point in the future (Figure 3.49). If
you opt to update immediately, the wizard will cycle through every object
covered by the scope of the policy and update the primary SMTP address for
each object. The advantage is that the policy is enacted immediately; the disadvantage is that you could have to wait a long time for the wizard to finish.
Some simple tests revealed that Exchange processes between 25 and 35
objects per minute, so applying a policy to 5,000 mailboxes might take
between 142 and 200 minutes to complete! Your mileage will vary.
It is also possible to create, amend, and apply email address policies
through the shell. You can invoke the Update-AddressList shell command if
you want to update a set of users (identified by a known address list). To find
out what email address policies exist within the organization, the format of
email addresses they generate, and their priority, use the command:
Get-AddressList | Select Name, EnabledEmailAddressTemplates,
Chapter 3
3.10 Email address policies
Figure 3.49
The wizard applies
an email
address policy
If conflicts arise when the email address policy is applied to existing
objects or as new objects are created because the policy results in duplicate
email addresses, Exchange resolves the conflicts by adding a number starting at
2 to the address and continues to do this until it generates a unique email
address. For example, if you have three users named John Smith, the email
addresses generated by Exchange under a first name.last [email protected] policy
will be:
[email protected]
[email protected]
[email protected]
Mail-enabled objects can have several email addresses and several email
addresses of the same type. Figure 3.50 shows a mailbox with three SMTP
addresses and one X.400 address. The bolded SMTP address is the default
reply address that Exchange uses to identify the user in outgoing messages.
The checkbox “Automatically updated email addresses based on recipient policy” allows you to control whether or not Exchange applies its email address
policies to an object. If you clear this checkbox, you take full responsibility for
3.10 Email address policies
the format and content of the email addresses given to the object. If the
checkbox is set, Exchange automatically validates any email addresses that you
add or update to ensure that policies are respected. Exchange also validates
that email addresses are unique in the organization to ensure that a condition
never occurs when objects share email addresses. The ability to state that the
email addresses for an object are not under the control of a policy is a change
in behavior as performing a rebuild of email addresses with the RUS in previous versions of Exchange processed all objects, whether or not you wanted the
RUS to do anything with those objects.
Figure 3.50
Email addresses for
a mailbox
There are situations where companies will provision and update mailboxes as part of automated HR processes that control all of the steps necessary
to bring a new employee on board and may not want to use email address policies. In other cases, companies may want to use an email address format that
isn’t included in the wizard and will write their own code to stamp the mailboxes for new users with email addresses in the desired format. These procedures should also make sure that the EmailAddressPolicyEnabled property of
the mailbox is set to False. As we will see in Chapter 4, setting a property for a
mailbox requires a one-line PowerShell command:
Chapter 3
3.10 Email address policies
Set-Mailbox –id ‘James Bond’
It’s hard to satisfy everyone, but the basic email address policies provided
by Exchange do enough to satisfy the needs of 90% or more of deployments.
Mailbox moves and email address policies
One interesting side-effect of moving mailboxes from an Exchange 2003
server to Exchange 2007 is that the mailboxes will be processed by an email
address policy. Given the somewhat erratic way that Exchange 2003 applies
recipient policies, it is entirely possible that you move some mailboxes that
have not had policies applied to them. Exchange 2007 offers far less latitude
(or loopholes) about how email address policies are applied so when the mailboxes are moved they may be stamped with a new email address to comply
with policy and perhaps even a new primary SMTP address, which could
cause confusion for users.
To avoid the problem, you can use the AD Users and Computers console on the Exchange 2003 server to amend the mailbox properties so that
policies are not applied or you can run a shell command to do the same thing
as explained in the last section. Even better, if you know that you are moving
a group of mailboxes and you can identify them (all in a CSV file, all mailboxes from a server or database, everyone in a distribution group), then a
piped command such as the one shown below will work:
Get-Mailbox D
– atabase ‘VIP Mailboxes’ | Set-Mailbox
After the mailboxes move to Exchange 2007, you can decide whether or
not to reenable email address policies for them or leave them alone.
Queries that drive email address policies
Behind the scenes, each email address policy has a recipient filter that allows
Exchange to determine what objects are under the control of the policy. For
example, the default email address policy has a very broad recipient filter
because it has to apply to any object that Exchange can apply no other email
address policy to. The recipient filter for the default email address policy is
very simple as the filter catches everything. You can see details of the policy
with the Get-EmailAddressPolicy command:
3.10 Email address policies
Get-EmailAddressPolicy –id ‘Default Policy’ | Select Name,
*Filter*, *Exch*
Default Policy
0.0 (6.5.6500.0)
The interesting thing about this output, which you will see if you look
at the default policy in an Exchange organization that contains legacy servers,
is that:
The default email address policy does not include a recipient filter. As
we learned earlier on in this chapter, Exchange 2007 uses OPATH
syntax recipient filters instead of the LDAP filters used by Exchange
2003. Because this organization contains legacy servers, the LDAP
filter remains in place and will be translated into OPATH on the fly
by Exchange 2007 mailbox servers when they need to apply it to new
objects. We can also see that the RecipientFilterType is “Legacy” and
that the ExchangeVersion property specifies 6.5, which indicates that
the object can be worked with by legacy servers.
The LDAP recipient filter (mailnickname=*) is exactly what you’d
expect to find for every object in the organization, so it applies to
every object if Exchange has not been able to apply a previous policy.
If you attempt to edit this email address policy, EMC signals an error as
shown in Figure 3.51. This does not mean that you have to take immediate
action to upgrade or otherwise change the policy because it is quite valid to
retain the policy in its current form until you decide that you want to use the
Set-EmailAddressPolicy command to upgrade the policy, perhaps after you
have removed the last legacy Exchange server from the organization. Use this
command to perform the upgrade:
Set-EmailAddressPolicy –id ‘Default Policy’ –
IncludedRecipients AllRecipients
Exchange will prompt you to confirm that you want the upgrade to
occur. If you agree, Exchange will rewrite the email address policy and
Chapter 3
3.10 Email address policies
Figure 3.51
EMC detects a
legacy email
address policy
upgrade its version and the RecipientFilter. You can suppress the question by
including the –ForceUpgrade parameter, which is provided to allow you to
upgrade email address policies in a script, however, it does nothing to automatically write new recipient filters and you have to provide that information
yourself. Do not worry about upgrading the default email address policy even
if you still have some legacy Exchange servers. The fact that Exchange maintains an LDAP filter within the email address policy allows the Recipient
Update Service to continue to stamp email addresses for new objects created
on the legacy servers. However, because of the upgrade, you will not be able
to edit the default email address policy using ESM.
On the other hand, rewriting the default email address policy in this
way will not allow you to edit the recipient filter through EMC either
because you have programmatically set the filter conditions. If you need to
adjust the recipient filter in future, you will have to do it with the SetEmailAddressPolicy command. This should not be a problem because
most organizations never change their default email address policy after it
is first set.
You can check for email address policies that are still in legacy status
with the following command:
Get-EmailAddressPolicy | Where {$_.ExchangeVersion –Like
‘*6.5*’} | Select Name, *Filter*
If you look at an email address policy that EMC created, you see some
differences in the filters:
Get-EmailAddressPolicy –id ‘Legal Department Members’ |
Select Name, *Filter*, *Exch*
: Legal Department Members
: (RecipientType -eq 'MailboxUser'
-and (Department -eq 'Legal' -or Department -eq 'Law'))
: False
3.10 Email address policies
: Precanned
: 0.1 (8.0.535.0)
Remember that the OPATH syntax used by email address policies is the
same as that used by dynamic distribution groups and the other Exchange
objects that use recipient filters, so the form taken by the recipient filter
should be familiar. Looking at the details of the email address policy that we
created through EMC, we see that:
The email address policy contains a recipient filter in both OPATH
and LDAP format. The two filters generate the same result when
applied to find users. However, we are still functioning in an organization that contains legacy servers, so the Recipient Update Service
will continue to use the LDAP filter.
Exchange has upgraded the ExchangeVersion property to 8.0 to indicate that we can manage the policy with EMC.
The recipient filter is “Precanned,” meaning that it has been generated by setting conditions through the wizard that EMC calls to
allow you to define what objects are covered by a policy. Figure 3.52
shows the wizard editing the email address policy to set the conditions that eventually result in the recipient filter shown above.
If you need to apply a very complex filter for an email address policy
that the wizard does not support, you can create the policy with the NewEmailAddressPolicy shell command and specify the recipient filter that you
need. In this situation, the syntax rules that apply are the same as those used
to specify recipient filters for dynamic distribution groups. See the section on
using the shell to manipulate dynamic distribution groups in Chapter 4 for
more information. As an example, let’s assume that you want to create an
email address policy that only applies to mailbox users in the Dublin office.
You could use this command:
New-EmailAddressPolicy –Name ‘Users in Dublin’
–RecipientFilter {Office –eq ‘Dublin’ –and
RecipientTypeDetails –eq ‘UserMailbox’}
–EnabledPrimarySMTPAddressTemplate ‘’
This command creates an email address policy with a custom recipient
filter, so you will not be able to edit the policy through EMC afterwards.
Therefore, if you need to update the recipient filter subsequently, you will
have to do it by writing a new recipient filter with the Set-EmailAddressPolicy command. If we examine the details of the email address policy creChapter 3
3.10 Email address policies
Figure 3.52
Setting a query for
an email
address policy
ated with the above command, we see the output shown below. Note that
Exchange has generated the appropriate LDAP filter and that the RecipientFilterType is now “Custom.”
: Users in Dublin
: (Office -eq 'Dublin' -and RecipientTypeDetails -eq
: False
: Custom
: 0.1 (8.0.535.0)
3.11 Address lists
3.11 Address lists
Exchange uses address lists to segment the set of available addressees within
an organization into different types of useful lists such as All Users, All Contacts, Public Folders, and the Global Address List itself. All of these lists are
available in Exchange 2003. Exchange 2007 adds the All Rooms Address List
to allow users to access a list of available room mailboxes. You work with
Address Lists through the Mailbox section of the Organization Configuration node in EMC (Figure 3.53). Each Address List has a recipient filter that
establishes the objects included in the list. You can create new Address Lists
by providing Exchange with the necessary recipient filter to build the list.
Figure 3.53
Address Lists
Conceptually, an Address List is similar to a dynamic distribution group
in that Exchange uses a recipient filter to locate all of the objects to form the
Address List. Whereas a dynamic distribution group is transient and its
membership only exists when Exchange expands the group to add the recipients to a message addressed to the group, Address Lists are more persistent as
users can access them through a client. Figure 3.54 shows how Outlook presents the set of available Address Lists.
You can see the same information through the Exchange Management
Shell with the Get-AddressList command:
Chapter 3
3.11 Address lists
Figure 3.54
Address lists as
shown through
Get-AddressList | Select Name, RecipientFilter
---All Rooms
All Users
Everyone in Dublin
eq 'Dublin')
All Groups
All Contacts
Public Folders
UK Employees
or Company...
(RecipientType -eq 'MailboxUser' -and StateOrProvince -
(((Alias -ne $null -and (Company -eq 'Hewlett Packard' (RecipientType -eq 'MailboxUser' -and Department -eq
As you can see, some of these Address Lists have custom recipient filters.
The standard Address Lists provided by Exchange use precanned recipient
filters. We can see this by examining the properties of one of the lists:
3.11 Address lists
Get-AddressList –id ‘All Users’ | Select Name, *Filt*, *Exch*
: All Users
: (& (mailnickname=*) (|
))) ))
LastUpdatedRecipientFilter :
: Legacy
: 0.0 (6.5.6500.0)
Figure 3.55
Properties of the
Default Offline
Address List
Just like Email Address Policies, we can see that this Address List is managed by a legacy Exchange server and that the filter is provided in LDAP format. We will examine how to upgrade these Address Lists later on in this
To create a new Address List, go to the Mailbox section under the Configuration node in EMC and click on the New Address List option. Creating
a new Address List is very similar to creating a dynamic distribution group in
Chapter 3
3.11 Address lists
that you specify the types of recipients that you want to include in the list
and then the filter that Exchange will use to build the list. You also specify
the schedule that Exchange uses to build the list. Figure 3.56 shows the first
two steps in the wizard used to create a new Address List. Note the Preview
option on the right-hand screen that allows you to verify that the filter you
create selects the right objects to include in the list.
Companies that provide a hosted Exchange service to other companies
often use customized Address Lists to create customized Offline Address
Books. In this case, if we examine the Address List that we have just created,
we will see some differences between it and the standard Address Lists.
Get-AddressList –id ‘Everyone in Dublin’ | Select Name, *Fil*, *Exch*
: Everyone in Dublin
: (RecipientType -eq 'MailboxUser' -and
StateOrProvince -eq 'Dublin')
: Precanned
: 0.1 (8.0.535.0)
Figure 3.56
Creating a new
Address List
Note that the version number has increased to 8.0, meaning that you
cannot manage this Address List in Exchange 2003. The recipient filter is in
both OPATH and LDAP syntax and because it is a precanned filter, we know
that we can manage it through EMC.
3.11 Address lists
Upgrading Address Lists to Exchange 2007 format
As with Email Address Policies, the Address Lists created by a legacy
Exchange organization will function quite happily under Exchange 2007. If
you want to upgrade them to Exchange 2007 format, you take roughly the
same approach as to upgrade Email Address Policies by using the SetAddressList command to update the Address List with a new recipient filter.
For example:
Set-AddressList –id ‘All Users’ –IncludedRecipients
MailboxUsers -ForceUpgrade
If you don’t supply the –ForceUpgrade parameter, you’ll be prompted to
go ahead with the upgrade. After the upgrade, you can examine the properties of the Address List to see what happened. We’ll see that a recipient filter
in OPATH format exists, the older LDAP format is also available to support
Exchange 2003 servers, and the Exchange version has been upgraded. Once
again, because we have upgraded the Address List through the shell, we can
only change the recipient filter in the future with Set-AddressList.
Get-AddressList –id ‘All Users’ | Select Name, *Filt*, *Exch*
: All Users
: RecipientType -eq 'UserMailbox'
: Precanned
: 0.1 (8.0.535.0)
There are two Address Lists that you have to take special care with when
you upgrade them from their legacy versions: Public Folders and the Default
Global Address List. No precanned filter exists for mail-enabled public folders, so you have to specify the filter:
Set-AddressList –id ‘Public Folders’ –RecipientFilter
{RecipientType –eq ‘PublicFolder’}
The Default Global Address List is a little more complex, as we see when
we examine its properties with the Get-GlobalAddressList command, and
we see the following:
Chapter 3
3.12 User naming conventions
Get-GlobalAddressList | Select Name, *Filter*, *Exch*
: Default Global Address List
: (& (mailnickname=*) (|
ctCategory=publicFolder)(objectCategory=msExchDynamicDistributionList) ))
: Legacy
: 0.0 (6.5.6500.0)
The syntax for the LDAP recipient filter looks complicated because it
includes all the different recipient types that are included in the GAL. To
upgrade the Default Global Address List, we have to use the following command to specify the new recipient filter in OPATH format:
Set-GlobalAddressList i
– d ‘Default Global Address List’
-RecipientFilter {(Alias -ne $null -and (ObjectClass -eq
‘User’ -or ObjectClass -eq ‘contact’ -or ObjectClass -eq
‘msExchDynamicDistributionList’ -or ObjectClass -eq
‘Group’ -or ObjectClass -eq ‘PublicFolder’))}
You may have custom Address Lists that you created with Exchange
2003. The process that you take to upgrade custom Address Lists to
Exchange 2007 format is exactly that described above. You have to determine
what the recipient filter is in OPATH format and then use the Set-AddressList command to add the new recipient filter to the object. To help you figure out the correct syntax for OPATH recipient filters, Microsoft has
published a PowerShell script on the Exchange team’s blog. You can use this
script to convert an LDAP filter to OPATH format.
3.12 User naming conventions
Email Address policies allow you to define and apply different patterns for
SMTP addresses, however, Exchange 2007 does not allow you to define policies to control the generation of display names. The default pattern for display names is %g %s—in other words, first name <space> last name, or
“Tony Redmond.” This is an acceptable naming convention for small implementations where everyone knows each other and it is easy to find the correct
recipient by browsing the GAL, but problems arise as the number of directory entries increases. Even though they have a very large GAL, Microsoft
3.12 User naming conventions
actually uses this naming convention, but I suspect that this is more to do
with tradition than design.
On balance, more variation within any messaging community occurs in
surnames rather than given names. Users are accustomed to browse through
telephone directories by surname, so it makes sense to organize the GAL by
surname. Think of a common given name, like “John” or “Mary,” and imagine how many times they occur in a large GAL. If the GAL is sorted by given
name, you might have to look through several hundred entries before you
locate the right recipient. It is easier to search using a surname, even with
common surnames, like “Smith,” “Chan,” or “Ng.”
Exchange builds the GAL from Active Directory data. Because
Exchange 2000 and 2003 use the AD Users and Computers console to create
user objects, the DisplayName attribute dictates the way that display names
are generated when new objects are created and therefore dictates the sort
order for the GAL. In these versions, it was common practice to change the
pattern that Active Directory uses to create display names by altering the
properties of the User-Display attribute of the “Display Specifier” object in
the configuration naming context with the ADSIEDIT utility and change
the pattern to %<sn>, %<givenname> as this then forces Active Directory to
create new display names in Surname, First Name format, which then generates the GAL in the desired format. The same trick is possible for contacts
and if you make a change for user objects, you should make the same change
for contacts in order to maintain consistency in the GAL. Groups are normally left alone because you don’t normally assign surnames to groups.
The problem that we encounter with the naming of user objects when
we upgrade to Exchange 2007 is that EMC now has full control over the creation of mailbox objects. Therefore, the changes that you make to Active
Directory to effect how the Active Directory creates display names for new
objects don’t apply and are ignored by EMC.
A further complication for organizations that run Exchange in multiple
languages arises because the Exchange 2007 EMC includes logic to respect
locales so that name-related edit boxes are reordered to reflect language-specific
naming conventions. EMC supports the following languages:
Chinese (Simplified)
Chinese (Traditional)
Chapter 3
3.12 User naming conventions
The language that you run EMC in determines the fields that are
exposed when you create new objects and the order that the fields are used.
For example, EMC doesn’t display the initials field in the French version. If
you create Chinese users, the boxes are presented by EMC in surname, initials, and first name order, which is exactly what some organizations who run
everything in English would like to use. However, you can’t force this order
to occur for other locales as the ordering is hard-coded in EMC. Thus, if you
want to apply a different naming convention, you can either:
Allow EMC to create the mailboxes and contacts as normal and then
edit the Display Name.
Create mailboxes and contacts using shell scripts so that you have
complete control over the format used for display names.
Changing the display name pattern only effects the display name for
objects created after you make the change, so in either case, you may have
existing mailboxes and contacts that do not have display names that comply
with your desired naming convention. Here is a simple script that can do the
job of changing the display names for mailboxes and mail-enabled contacts
objects to the last name, first name standard:
$Users = Get-User -Filter {RecipientTypeDetails -eq ‘UserMailbox’ or
RecipientTypeDetails –eq ‘LegacyMailbox’}
ForEach-Object ($U in $Users)
$Mailbox = $U.Name
$DN = $U.LastName + ", " + $U.FirstName
Write-Host "Setting display name for mailbox " $DN
Set-User $Mailbox -DisplayName $DN
$DN = $Null
$Contacts = Get-Contact -Filter {RecipientTypeDetails -eq ‘MailContact’}
3.12 User naming conventions
ForEach-Object ($C in $Contacts)
$Contact = $C.Name
$DN = $C.LastName + ", " + $C.FirstName
Write-Host "Setting display name for contact " $DN
Set-Contact $Contact -DisplayName $DN
$DN = $Null
Figure 3.57
A well-ordered
Apart from groups, you may also want to leave resource and equipment
mailboxes unchanged, again because they don’t have surnames. There may be
other circumstances where you have mailboxes that you don’t want to use the
last name, first name convention, such as those used for journaling, but these
can be dealt with on an exception basis. Figure 3.57 shows an example of a
GAL with a mixture of users and groups where the mailboxes and contacts
use the last name, first name convention and the groups use the default.
Of course, at any time, you can change the display name for objects to
make them consistent or indeed to input any value you like to help users
identify the correct user. For example, you could include a department name,
location, job title, or other identifier to help resolve users that share the same
name. Thus:
Chapter 3
3.13 Server naming conventions
Smith, John (Glasgow)
Smith, John (Security Department)
Smith, John (VP, Technology)
3.13 Server naming conventions
Is it important to have a naming convention for all the servers in your infrastructure or for both servers and workstations? The answer depends on the
number of servers in the infrastructure. If you only have a small number of
servers, you probably know each server intimately and you will be able to recognize the location of the server and its relative importance to the infrastructure if a problem arises. In these cases, you can afford to deploy whimsical
names such as calling all the servers after characters in Star Trek, Lord of the
Rings, or whatever your favorite film happens to be.
However, things are more complicated in large, distributed infrastructures that span hundreds of servers. In these cases, you need some help to
locate a server if your beeper goes off to notify you that your monitoring service has identified a problem with a specific system. For example, if the
beeper says “problem with SERVER27,” you will have to rack your brains to
figure out where SERVER27 is physically located and what it does. If
SERVER27 is a backup server for a production system, maybe you do not
have to rush to fix the problem, but if it hosts Exchange for senior management, you probably have a different response. Being able to find servers
quickly when things go wrong is the best and most practical reason to use a
naming convention.
The next step is to decide what kind of naming convention to use. Here
are some:
Office or location codes: If you work in a company that has many
offices, you may be able to use office or location codes to identify
where servers are located. For example, one company that I worked in
used location codes such as “ZKO,” “MRO,” “REO,” “DBO,” and
so on. If several buildings exist in a location, you can add a number to
identify the exact building, giving values such as “REO1,” “DBO2,”
etc. The problem with office or location codes is if your company
closes down an office and you relocate servers elsewhere.
Geographic names: The best thing about geographic names like
“London”, “Paris”, and “New York” is that they are highly unlikely to
change before you need to replace a server. For this reason, many
companies use three or four-character abbreviations of geographic
names as the base of their naming convention, resulting in values
such as “LON”, “PAR”, and “NYC.” Again, you can combine these
3.13 Server naming conventions
values with an indicator of what building the server is located in to
make the name more effective.
Function indicator: Knowing where a problem server is valuable, but
knowing its function is even more valuable. Many companies use values such as those listed below to indicate how important the server is
to the infrastructure.
EXMBX—Exchange mailbox server
EXCAS—Exchange Client Access Server
EXBHS—Exchange bridgehead server (Exchange 2003)
EXHUB—Exchange Hub transport server
EXPFS—Exchange public folder server (Exchange 2003)
EXCH—Exchange multiple role server
EXEDGE—Exchange Edge server
GC—Global Catalog Server
DC—Domain Controller
DHCP—DHCP server
WS—individual workstation
Identifier: You may have more than one server of a specific type in a
location, so you need to allocate a value to create a unique identifier.
Some companies number their servers, while others assign values that
provide more information about the server. For example, they might
use “CL1” to indicate the first node in a cluster, or “PL380-4” to
indicate that the server is a 4-way ProLiant 380. Going to this degree
of specificity is somewhat excessive and the simple numbering
scheme is usually sufficient.
Putting everything together, you have a naming convention that names
servers like this:
Meaning that this server is located in building 1 in New York City and is
the first Exchange server in the office. Note that hyphens in server names are
not a problem for Windows infrastructures but they can pose an issue for legacy environments. Other administrators think that it is a good idea to
include an indication of the user community that Exchange servers support,
so they add suffixes such as “-VIP” (VIP mailboxes), “-FIN” (Financial
Department mailboxes), and so on, meaning that you might decide to have a
naming convention that named servers like:
Chapter 3
3.14 Moving from the basics
On a cautionary note, before you include the server’s role in your naming convention, make sure that your security team is happy for this to happen as some security teams believe that adding information like this makes
certain servers more of a target for attack. Security teams also dislike location indicators in server names, as they hate to expose any data to potential
attack. However, it is fair to say that security teams are paranoid due to the
very nature of their business and that they would prefer you to give servers
names like:
In fact, if Windows would allow you to include some special characters
in server names, the security team would want you to take advantage of that
feature too. As always, the important thing is to achieve a balance in your
naming convention between utility, ease of management, and security. It is
wise not to be too hung up on if servers do not follow the naming convention precisely as few organizations manage to achieve 100% compliance.
Some administrators approach the question of server naming with the
attitude that it is not important what name you start with because it is possible to rename servers. This is true, but in practice, most administrators do
not want to do the work that is involved to rename a server and leave servers
with their original names until the servers gently degrade over time and eventually leave the infrastructure when they become obsolete. For this reason, it
is best to select a naming convention that is most resilient to change while
meeting whatever restrictions others in the organization (such as your auditors or IT security) place upon you.
3.14 Moving from the basics
The Exchange 2007 management console is an easy tool to work with. The
changes that the Exchange developers have made may confuse at first sight,
but rapidly become familiar. With a solid grasp of the basics of how to
approach the management of Exchange 2007 servers, it is time to move onto
something more complex. We met some aspects of PowerShell when we discussed recipient filters and the wonders of OPATH when used with dynamic
distribution groups, Email Address Policies, and Address Lists. It is now time
to move into PowerShell in some detail because you cannot spend all your
time working through the management console if you expect to manage an
Exchange 2007 organization effectively.
The Exchange Management Shell
The original Exchange administration program, ADMIN.EXE, did a good
job of managing small groups of servers (a site), but Microsoft did not
design the ADMIN program for the enterprise, so it faltered in large
deployments because it struggled to deal with large numbers of servers or a
very distributed environment. ADMIN did not support a generally available
API, so what you got with ADMIN was what you used. Things improved
somewhat in Exchange 2000 with the introduction of the Exchange System
Manager (ESM), based on the MMC (Microsoft Management Console)
framework. ESM uses many different APIs such as CDOEXM and WMI
(Windows Management Instrumentation), but while ISVs could build their
own snap-ins for MMC, the average Exchange administrator still could not
customize ESM. You still had to acquire a lot of knowledge about programming interfaces and languages before you could attempt to automate even
the simplest task. For example, managing public folders has always been a
nightmare for Exchange because there was no way to roll your own administrative utilities when the product failed to deliver.
The split between Active Directory management, which is where you
create user accounts, groups, and so on, and Exchange management, where
you deal with the details of mailboxes, stores, and the configuration of the
organization, further complicates matters. Because of the reliance that
Exchange had on IIS, another set of inconsistencies existed in the split of
management of web protocols between ESM and IIS. As Microsoft
approached the issue in the Exchange 2007 design cycle, what seemed to be
missing was a single way of managing the entire set of objects that live inside
an Exchange ecosystem, preferably something that functioned in an open
and extensible manner.
Microsoft’s response that we see Exchange 2007 is to provide broad and
deep support for PowerShell (previously called Monad1) that works alongside
The name Monad comes from The Monadology
, a book by Gottfried Wilhelm Leibniz, one of the inventors of Calculus.
The idea describes components that could be combined to solve complex problems, which is what the PowerShell developers set out to do—provide commands that can be combined in scripts and programs to get work done.
the new Exchange Management Console to deliver a new administrative
framework. PowerShell is a new Windows shell that supports a powerful
scripting language that allows administrators to build their own tools to
manage Exchange and other Windows components.
PowerShell supports many different system objects on the Windows
platform, including COM objects, WMI objects, and .NET objects. Access
to these objects, plus the ability to extend over time to other objects such as
those now supported for Exchange, give administrators the ability to automate administrative tasks in the same way as has long been common for
other enterprise-class operating systems. Administrators of operating systems
such as OpenVMS and UNIX have been able to write scripts to automate
operational procedures for years, and some pretty complicated applications
have been written completely in scripts. You can argue that the provision of a
comprehensive scripting language is a sign of growing maturity in the ability
of Windows to satisfy the demands of enterprise computing, or you might
regard PowerShell as Microsoft’s acknowledgment that it is impossible to
cover all administrative options for an operating system through a GUI.
There is no way that a developer sitting in Redmond, even though Microsoft
has some very gifted people, can determine every single option that customers will want to perform to manage servers, so providing a way to allow customers to build their own management tools allows Microsoft to provide a
much more complete management framework. This aspect is especially visible in Exchange, where the product has been handicapped by the inability of
administrators to do anything but the options provided by the GUI since the
product first shipped. A few registry hacks and some command line tools do
not provide a way to automate common management operations, but a powerful scripting language certainly does.
To achieve the goal of being able to automate administrative tasks, PowerShell provides a single consistent scripting interface to Windows and applications. This step eliminates the need for a mass of competing and confusing
interfaces from ADSI and WMI for Windows to CDOEXM and WebDAV
for Exchange. Another goal behind the introduction of PowerShell was to
create a programmable scripting language for Windows that delivers similar
capabilities to languages such as Perl and Python on other platforms. We are
only at the beginning of the PowerShell era and it will take time for applications to expose their objects (through COM, .NET, or WMI) that administrators can work with every application that they need through PowerShell,
but a good start has been made.
The promise is that you should only have to learn PowerShell syntax and
structure once to be able to use it across the complete range of applications
that support PowerShell. In fact, you only have to use PowerShell to the level
that makes sense to you. It is possible that some administrators will never use
PowerShell, perhaps because they can do everything that they need or want
4.1 EMS: Exchange’s management shell
to do through EMC. If your server only supports 20 or 30 mailboxes, you
may never need to go into PowerShell mode. Perhaps you will not be able to
because the server that you manage has been locked down to meet security
requirements. Another set of administrators will be happy to write one-line
commands all the time or maybe build some quick and dirty scripts by
recording the commands that they have used to save them from having to
figure out the necessary syntax and commands again. Others will have the
time to investigate the language to a deeper level and end up by developing
complex scripts to automate administrative operations. These scripts may be
complete with error handling and support parameters so that the scripts can
deal with a range of situations. These scripts may end up becoming part of
the formal operations framework used to manage production servers and be
treated in the same way that any mission-critical piece of code should be. A
few will go into the depths of .NET programming and extend PowerShell
with new commands. You can compare the different styles of PowerShell
usage to other shells—one-liners like bash, scripting like Perl, and programming like any of the .NET languages. The great advantage of being able to
employ PowerShell in many different ways is that you can begin with a small
knowledge base and gradually evolve over time as your experience of using
the language increases to a point where you can take on new challenges. No
knowledge is wasted and everything contributes to improvement. Another
point is that it’s easy to share PowerShell one-line commands and scripts on
web sites and blogs, so people can gain from the overall knowledge of the
network community rather than having to do everything themselves.
At the end of the day, PowerShell is just a tool and even the most experienced PowerShell users will use one-line commands because it is often the
best way to get things done. Microsoft refers to the range of usage from oneline scripts to compiled code as the Admin Development Model; not because
it’s limited to system administrators in any way, but because the activities
that PowerShell helps with are critical to administrators. Of course, you only
have to go near PowerShell if the standard tools do not do what you want
them to, and many Exchange 2007 administrators will continue to use the
new EMC for all their administrative duties. However, as discussed earlier in
this chapter, some options that appear in previous versions of the Exchange
management console are not supported in the Exchange 2007 GUI (like
configuring transport settings or exporting the contents of mailboxes), so you
will end up in PowerShell, even if you don’t want to at first.
EMS: Exchange’s management shell
On an Exchange 2007 server, the following components contribute to
form the basic management infrastructure that you can exploit through
scripts and programs or execute interactive commands through a command
line interface:
Chapter 4
4.1 EMS: Exchange’s management shell
PowerShell and PS, the PowerShell runtime engine.
Other components that call PowerShell cmdlets, such as the
Exchange 2007 Exchange Management Console.
The script and command parser, which processes language constructs
such as scripts and predicates.
The Extended Type System (ETS), which provides a standardized
view of objects through an interface that allows users to work with
properties and methods of cmdlets, no matter what differences exist
in their underlying data structures or without knowledge of the
underlying data structure. For example, a script can use information
extracted from the file system or the system registry and combine it
with information retrieved from Exchange. Good examples of ETS in
action are provided by the Sort-Object and Group-Object cmdlets,
which are able to sort objects returned by any PowerShell command
despite the vast difference in the type of object that these cmdlets
might be asked to process.
The session state manager, which manages the data sets used by
cmdlets when they execute.
Namespace providers, which provide file system like interfaces to sets
of stored data such as the system registry. You can get a list of the
available namespace providers with the Get-PSProvider command.
Cmdlets2, which provide the way to gain access to and operate on
objects associated with applications that expose their objects to PowerShell using a supported interface such as COM. Applications like
Exchange 2007 support PowerShell directly with a set of cmdlets,
while other applications such as the Active Directory support PowerShell indirectly because you have to work with their objects through
an intermediate interface like .NET.
The structure described above seems complex, but its value is that PowerShell commands are easy to work with once you get used to the philosophy
behind the language and its syntax, even when you use PowerShell to work
with data gathered from multiple sources. Writing scripts that call PowerShell commands is similar to writing command line scripts on a UNIX 3 or
Linux server, or even like DCL command procedures from OpenVMS.
Understanding the structure of scripts and getting to know how to use the
Cmdlets are also called “commands,” which seems like a more intuitive name for them. The two names are used
According to Bruce Payette in Windows PowerShell in Action, the developers started with the shell grammar defined for
POSIX, which is a subset of the UNIX Korn shell, and then optimized PowerShell for the Windows environment in the
same way that UNIX shells are optimized for UNIX.
4.1 EMS: Exchange’s management shell
full power of PowerShell takes a little time, but there is no doubt that access
to Exchange information is more programmable and easier than ever before,
especially if you want to automate common management procedures or create batch processes.
You can expect that you will write your own scripts, borrow others that
you find in blogs or elsewhere in the net, or learn about them from books.
You will be able to take scripts that come along with products (like those provided with Exchange 2007) and amend them to suit your own purposes.
Over time, you will build up a library of scripts that help you manage Windows and Exchange much more effectively than you have ever been able to
do before, especially for Exchange.
Working with PowerShell commands
Cmdlets are .NET classes and not standalone executables. They are the
smallest unit of execution in PowerShell. While each cmdlet has a specific
purpose (such as to move a mailbox or change a property of a mailbox), you
can “assemble” shell commands together to automate common management
processes through scripts. The default version of PowerShell that you might
run on a Windows server does not include any of the commands required to
work with Exchange data as any application that wants to use PowerShell has
to load its command set into the basic shell. Behind the scenes, when you
start the Exchange Management Shell (EMS), PowerShell starts the core shell
and then loads the set of Exchange commands through a snap-in (an assembly of commands) using the command:
Add-PSSnapin Microsoft.Exchange.Management.PowerShell.Admin
Figure 4.1
Listing the set of
snap-ins loaded
into PowerShell
Chapter 4
4.1 EMS: Exchange’s management shell
The Get-PSSnapin command returns the list of snap-ins that PowerShell
has loaded on top of the core shell. You can use the Remove-PSSnapin command to remove a snap-in from the core.
The latest version of PowerShell and some additional information is
available at
management/powershell/default.mspx. You have to install PowerShell on a
server before you can install Exchange 2007, if only because Microsoft built
EMC on top of PowerShell. You can run PowerShell and EMC on Windows
XP or Vista4 workstations, so you can use these workstations to manage
Exchange through the shell or the GUI, as long as you also install PowerShell
and the Exchange management components on your workstation.
Figure 4.1 shows the list of PowerShell snap-ins generated by the Getcommand on an Exchange 2007 server. You might have other
snap-ins listed if you install other applications that support PowerShell, such
as the Virtual Machine snap-in or those from third-party software vendors
that extend the capabilities of PowerShell, such as PowerGadgets.
The Get-Process command is a good example of one of the standard
set of commands available to work with Windows. This command lists all of
the active processes running on a server. You can then group the output so
that you know the company who generated the software that is running in
the processes. For instance, here’s the result of scanning the processes active
on an Exchange 2007 server. You can see components from the hardware
provider (HP and Compaq), anti-virus (Symantec), and base Windows
(Microsoft), among others.
Get-Process | Group-Object Company
Count Name
----- ----
Symantec Corporation
Hewlett-Packard Company
{ccApp, ccEvtMgr, ccSetMgr, DefWatch,
{cissesrv, cpqnimgt, CpqRcmc, cpqteam,
{csrss, csrss, esmagent, Idle, System,
Microsoft Corporation
{ctfmon, ctfmon, EdgeTransport, explorer,
Apache Software Founda... {rotatelogs, rotatelogs, rotatelogs,
Compaq Computer Corpor... {sysdown}
WinZip Computing, Inc.
You can also find information about the top processes that are consuming most CPUs on a server. For example:
As pointed out in Chapter 3, Microsoft will not formally support using Vista workstations to manage Exchange until at
least service pack 1 for Exchange 2007.
4.1 EMS: Exchange’s management shell
ps | Sort CPU –Descending | Select –First 15
The output from running this command on an Exchange 2007 server is
shown in Figure 4.2 where we can see three interesting processes appear in
the list. The first is the Exchange log replay service, so we know that this
server runs either LCR or CCR to protect its databases. The second is the
Microsoft Search service, which generates the content indexes for databases
on the server. The last is the process that synchronizes Active Directory content from a hub transport server to an Edge server. Because of the presence of
the second process, we know that this server is a multi-role server that supports both the mailbox and hub roles. The server must therefore be running
LCR. We know this because CCR only runs on a clustered mailbox server
that can’t support the hub transport role, so this particular server can’t synchronize with an Edge server. LCR and CCR are explored further in Chapter
9. Of course, as we’ll learn when we get into the details of PowerShell syntax,
you could isolate the Exchange processes that run on a server by including a
filter through the Where command:
PS | Where {$_.ProcessName –Like ‘*Exchange*’} | Sort CPU Descending
Figure 4.2
Output from PS
Getting into the details of how you can build filters and the properties
that you can use is a stretch for now, given what we have covered for PowerShell so far. Moving on, ps is an alias or shorthand for the Get-Process
command. You can define aliases for your favorite PowerShell commands to
cut down on the amount of typing you have to do or rely on the default set. 5
The Get-Alias command returns a list of all of the currently defined aliases,
including the ones that PowerShell creates for you, such as ps and commands
The O’Reilly Monad book by Andy Oakley (ISBN 0-596-10009-4) contains a nice appendix that describes basic Windows
PowerShell commands and their aliases.
Chapter 4
4.1 EMS: Exchange’s management shell
that you will use very often when you view Exchange data, such as ft for
You can create a handy text file that contains all the aliases
with this command:
Get-Alias > c:\temp\Alias.txt
Another example of using a PowerShell command to work with Windows is how to return the current status of the set of services that composes
an Exchange server:
Get-Service *Exch* | Select DisplayName, Status
----------Microsoft Exchange Active Directory Topology
Microsoft Exchange EdgeSync
Microsoft Exchange File Distribution
Microsoft Exchange IMAP4
Microsoft Exchange Information Store
Microsoft Exchange Mailbox Assistants
Microsoft Exchange Mail Submission Service
Microsoft Exchange Monitoring
Microsoft Exchange POP3
Microsoft Exchange Replication Service
Microsoft Exchange System Attendant
Microsoft Exchange Search
Microsoft Exchange Service Host
Microsoft Exchange Transport
Microsoft Exchange Transport Log Search
Microsoft Search (Exchange)
As we will see later on in this chapter when we discuss how to use PowerShell to test different aspects of Exchange 2007, the Test-ServerHealth command provides another way to discover whether all the Exchange services are
running. In Chapter 2, we met an example of how PowerShell is able to query
the Active Directory and how to extract a list of Global Catalog servers. Here’s
an example of how to retrieve information about the current Active Directory
forest by loading information into a variable that you can subsequently interrogate to retrieve precise data. It is a good thing to record this information regularly so that you know exactly the exact state of the forest.
# Active Directory Forest
$Forest =
4.1 EMS: Exchange’s management shell
# Get information on the Forest
# List of AD sites
# Forest mode
# List of all domains
# List all GCs
# List all partitions
It’s somewhat frustrating to have to work with Active Directory objects
in such a clunky manner (through their .NET classes), but Microsoft is
aware of the problem and may include PowerShell extensions for Active
Directory in the “Longhorn” release of Windows server.
You can use PowerShell to launch applications that are associated with a
particular file extension. For example, assuming that Word is installed on a
system, here’s how you can launch Word to edit a specified document:
Invoke-Item c:\Temp\Exchange.doc
You can even ask PowerShell to count the number of words in a Word
Type c:\temp\Exchange.doc | Measure-Object –Line -Word
Another interesting way to use PowerShell is when you want to interrogate the Event Log to check for a particular event. For example, here’s how to
look through the Application Event Log for event 701, which is generated by
the Store after it finishes a background defragmentation run against a database. You can use the same technique for any event. The command shown
here may seem complex initially, but you will soon get used to combining the
Where command (to apply a filter on a data set), the Select command (to
pick out records to work with), and the Format-List command (to format
output data in a list) as you work with Exchange data.
Get-Eventlog Application | Where {$_.EventId
Select -First 5 | Format-List
-eq 701} |
We can also output the details of the event that Exchange provides so
that you can see the name of the database that is processed. In this instance,
Chapter 4
4.1 EMS: Exchange’s management shell
we are more specific that the source of the events that we want to see is ESE
(the Store database engine) rather than other parts of Exchange (like the
transport engine) that generate the same event. In addition, we trim the output of the message property to remove a blank line that you’d see otherwise.
Get-Eventlog Application | Where { ( $_.EventId –eq 701) –and
($_.Source –eq ‘ESE’) } | Select –First 5 | Format-List
TimeGenerated, Username, @{ Expression= {$_.Message –replace
“`n”, “” } ; Label = “Message”}
Exchange shell commands
When we discussed the topics of Email Address Policies and Address Lists in
Chapter 3, we ran into Exchange PowerShell commands because they are the
only way to get some of the management tasks done for these objects in
Exchange 2007. Now we’ll plunge into the details of how to use the commands made available through the Exchange Management Shell to work
with many of the more common objects that an administrator deals with on
a daily basis.
When you install Exchange 2007 onto a server, you add over 370 commands6 to the standard set of PowerShell commands, including commands
to work with the system registry, the file system, variables (including environmental variables), and so on.
Some examples (in no particular order) of Exchange 2007 commands
Get-ExchangeServer: Return a list of Exchange Servers in the organi-
Disable a user’s mailbox.
an account.
Move a user’s mailbox from one database to another.
tion group.
remove an Active Directory permission from
Add a new member to a distribu-
Set a property of a user’s mailbox.
Retrieve properties of a mailbox database.
Use the command Get-ExCommand | measure-object –line to find out exactly how many commands are
added by Exchange. Use the command Get-Excommand > c:\temp\ ExCommands.txt to create a listing of all
the commands.
4.1 EMS: Exchange’s management shell
Get-MailboxStatistics: Return statistics about user mailboxes such
as the total item count, quota used, etc.
Return properties that control how the transport service processes messages.
Launches a browser to bring you to the Exchange development team’s blog.
Note the consistent syntax of verb (Get, Set, Move, Remove, Disable)
and noun (Mailbox, User, etc.). Along with commands that operate on
objects, you find commands that help you to work with data, such as WhereObject, Sort-Object, and Group-Object and Import-CSV and Export-CSV.
Where-Object, Sort-Object, and Group-Object, are commonly shortened
by using their aliases of Where, Sort, and Group. You can type Help followed
by a command name at any time to get help on the syntax of the command,
and you can use the Get-Help command to list all the commands available to
work with a specific type of Exchange object:
Get-Help –Role *Mailbox*
Get-Help –Component *Recipient*
In fact, you’ll find that the Exchange developers have done a nice job to
make help for the EMS commands very accessible. Apart from the default
way of getting help shown above (which displays what the command does
and the syntax you should use), there are other ways of looking at help:
Use Help *mailbox* to get help about commands that work with
mailboxes. The same technique works to find commands that deal
with different types of objects that you’ll need to work with and you’ll
see the actions that you can take with these objects. For example, if a
Get- command exists, you can retrieve properties for the object, but if
there isn’t a Set- equivalent, you won’t be able to set a property, so you
know that this is a read-only object:
Help *database* - database commands
Help *server* - server commands
Help *user* - user account commands
Help *contact* - contact commands
Help *group* - distribution group commands
Help *public* - public folder commands
Help *exchange* - Exchange system commands
Help *transport* - transport commands
Chapter 4
4.1 EMS: Exchange’s management shell
Help *connector* - connector commands
Help *rule* - transport and journal rule commands
Use the –Detailed switch to get more detailed help about a command. For example: Get-Help Get-CasMailbox –Detailed
Use the –Full switch to have EMS return every bit of information it
knows about a command: Get-Help Get-DistributionGroup –Full
Use the –Examples switch to see whatever examples of a command in
use EMS help includes: Get-Help Get-MailboxServer –Examples
Use the –Parameter switch to get information about a selected
parameter for a command: Get-Help Get-Mailbox –Parameter
Server. This switch supports wildcards, so you can do something like
Get-Help Set-Mailbox –Parameter *Quota*
You will probably begin by using the full view of commands to get to
know what each command does. After you learn more about the commands,
you can move on to the default view once you become more accustomed to
working with EMS. Remember that the Exchange 2007 help file contains
information about all the EMS commands. The advantage of using the help
file (which is always present on a server) is that you can use the help file’s
index and search for specific entries.
Most of the time, you’ll probably work with commands by invoking
EMS interactively and then typing in whatever individual commands or
scripts are necessary to get something done. After you become used to working with EMS, things flow smoothly and it is easy to get work done and it is
usually faster to start EMS and issue the necessary commands to change a
property on a mailbox or a server than to start EMC and navigate to the right
place to make the change through the GUI. Executing commands through
EMS is especially valuable if you have to perform management operations
across an extended network link when waiting for the GUI to display can be
painful. If you have a programmatic mind, you can also call shell commands
through C# code, which is how Microsoft invokes them in the EMC console
and other places throughout Exchange, such as setting up servers and databases in the setup program. In the past, the different groups that contributed
to Exchange had to build their own programming interfaces whereas now
everyone uses PowerShell.
You’ll note that shell commands focus on performing tasks rather than
taking the more object-focused approach implemented in the GUI, something that reflects a desire to accommodate the needs of administrators who
think about how to do things rather than how to work with objects. After all,
it is only human nature to think in terms of the task of moving a mailbox to
4.1 EMS: Exchange’s management shell
a different server rather than thinking about how to manipulate the properties of a mailbox object to reflect its new location.
Shell commands accept structured pipelined input from each other in
a common manner to allow them to process data in a consistent manner,
no matter what command provides the data, and there is even a command
to read data in from a CSV format file (and another one to export data to a
CSV file). Programmers therefore do not have to worry about reformatting
data for input to specific commands, so the task of assembling different
commands together into a script to do a job is much easier. Microsoft built
PowerShell around the concept of objects, so its commands accept objects
as input and its commands output in the form of objects that you can then
pipe to other commands as input objects. Even if the output from a PowerShell command looks like plain text, what you see is one or more objects
that you can manipulate in a much more powerful manner than you can
ever work with text output. The implementation is really very elegant.
The pervasive nature of EMS means that administrators can access all
management operations through the command shell for the first time. Previous
versions of Exchange support interfaces such as CDOEXM or WMI, but no
previous interface delivers the same range of functions as is now available
through PowerShell. In addition, because Microsoft has implemented the GUI
options presented in EMC or the Exchange setup program through one or
more shell commands, you can do more with a single line of script that calls a
command than you could do with twenty lines of WMI code. Another advantage gained by providing a set of task-oriented commands is that other programs
can call these commands to perform work rather than writing their own code.
The availability of a powerful interface to administrative functions is
also attractive to third-party software providers who traditionally have written a lot of code to integrate their products with Exchange or provided a separate management utility. If you want to integrate a product in the Exchange
2003 administrative environment, you have a range of APIs to deal with,
including CDOEXM, which includes functions used for provisioning recipient and databases, and WMI, which offers functions to retrieve and report
information about the infrastructure. However, the combination of
CDOEXM and WMI does not deliver the functionality that exists in the
Exchange 2003 version of ESM and you will have to use other interfaces,
such as ADSI, if you want to access configuration information about the
Exchange organization. Another issue for developers is that there was no
good or approved method to integrate a new management feature into ESM.
The result is that the Exchange 2003 administrative vista is not consistent
because it has multiple interfaces, tool sets, and programs. PowerShell is the
unifying theme for automation of administrative procedures and the base
upon which third parties extend the Exchange administrative framework by
combining out-of-the-box commands provided by Microsoft with their own
Chapter 4
4.1 EMS: Exchange’s management shell
code. This, at least, is the theory, and it will be interesting to see how quickly
third-party developers move to adopt PowerShell as their interface of choice.
Command editing
It should already be obvious that you could do a lot of typing to enter commands into PowerShell, make the inevitable mistakes, correct, and try again.
To make life a little easier, PowerShell supports the same kind of command
line editing as CMD does. The different keys that you can use are described
in Table 4.1.
Table 4.1
Command editing keystrokes for PowerShell
Keyboard Command
Pops up a list of the last ten commands that you have used
to allow you to select a command to reuse
Requests PowerShell to attempt to complete a command
based on what you’ve typed
Left/Right arrows
Move the cursor left and right through the current command line
Up/Down arrows
Move up and down through the history of previous
Delete key
Deletes the character under the cursor
Insert key
Toggles between character insert and character overwrite
Backspace key
Deletes the character behind the cursor
Most of these keys are pretty basic and straightforward. The two most
interesting keys are F7 and Tab. F7 pops up a list of the last ten commands
that you executed (Figure 4.3) so that you can both see what you’ve done in
the immediate past and select one of the commands to reexecute. The Up
and Down arrow keys serve a similar purpose in that they navigate through
the command history. At times it’s more convenient to use the Up and Down
arrow because you can retrieve more commands and have the chance to edit
a command before executing it (F7 selects the command and executes it
Tab completion is a wonderful feature that PowerShell inherited from
the Windows command shell (CMD). Tab completion means that you can
partially enter a command and then hit the tab key to have PowerShell try to
fill in the rest of the command. For example, type:
4.1 EMS: Exchange’s management shell
Figure 4.3
PowerShell F7
This isn’t a valid command, but it is the root of several commands, so
when you hit tab, PowerShell completes the first valid command that
matches and inserts:
If you hit tab again, PowerShell moves to the next command that
matches and inserts:
If you hit tab again, PowerShell returns to Get-DistributionGroup
because there are only two valid matches. However, things get even better
because PowerShell supports completion for parameters as well. If we insert –
to indicate a parameter value after Get-DistributionGroup and hit tab, PowerShell starts off with the first parameter and will continue through all valid
parameters. If you tab too many times and pass by the parameter that you
want to use, you can use SHIFT-TAB to go back through the parameter list.
If we add some characters to help PowerShell identify the parameter, it
attempts to complete using that value. For example:
Chapter 4
4.1 EMS: Exchange’s management shell
PowerShell completes Get-DistributionGroup –Ma into
Get-DistributionGroup –ManagedBy
Even better, tab completion is context-sensitive, so it understands the
structure of the object that you are navigating. For example, if you want to
move through the system registry, tab completion understands the hive structure, so you can type a location in the registry and then use the tab key to
move through the available choices from that point. For example, type:
CD HKLM:\Software\Microsoft\Exchange
Now press tab and PowerShell will lead you through all of the registry
locations used by Exchange.
Finally, PowerShell will complete variables and even the properties of
variables (such as their length) in a similar way to the way that the Microsoft
Visual Studio Intelligence feature works. If you type the incomplete name of
a variable and hit tab, PowerShell will attempt to complete it from the list of
known variables. For example, if we fill a variable with details of a mailbox
like so:
$Mailbox = Get-Mailbox –Id Redmond
And then type $Ma and hit tab, PowerShell will complete and return
$Mailbox. This is a very useful feature if you forget the names of variables
that you’ve defined. To see how properties are completed, type:
Hitting tab now will request PowerShell to go through the list of properties beginning with “Di.” For a mailbox, the list is “DistinguishedName” and
Getting at more information about something
Any command like Get-EventLog that retrieves some information about an
object will output a default set of properties about the object (or references to
an object). Sometimes those properties are not the exact ones that you want
to examine, so you will inevitably end up using the Format-List and Format-Table commands to expand the set of properties returned by a command. For example, if you use the Get-Mailbox command to view the
properties of a mailbox, the information returned isn’t all that interesting:
4.1 EMS: Exchange’s management shell
Get-Mailbox –identity Bond
---James Bond
However, if you specify the Format-List command after the Get-Mailbox command, you see a lot more information (an edited version of the output
Get-Mailbox –identity Bond | Format-List
: ExchMbxSvr1\First Storage Group\VIPMBX
: DatabaseDefault
: True
: False
: False
: None
: External
: 14.00:00:00
: True
: \VIP Users
: unlimited
: unlimited
: unlimited
: NormalAccount
: False
: Bond
: False
: /o=XYZ/ou=Exchange Administrative Group
: ExchMbxSvr1
: True
: unlimited
: 64KB
: Belfield
: [email protected]
: False
: {UK Employees, Default Global Address
List, All Users}
: Bond
Chapter 4
4.1 EMS: Exchange’s management shell
: Users/Spies
: James Bond
: {smtp:[email protected]}
: False
: /o=XYZ/ou=Exchange Administrative Group
: [email protected]
: UserMailbox
: James Bond
: CN=James Bond, OU=Exchange
: Users/Spies/James Bond
It’s easy to list every property downwards, but when you have limited
screen estate across, you need to be more selective about the properties that
you want to output, and that’s why it’s often a good idea to use the SelectObject command to select the data that you really need before you use the
Format-Table command. In this case, we use the Select alias for SelectObject, just because this command is used so often and it is nice to have the
chance to use shorthand.
Get-Mailbox –Identity Bond | Select Name, PrimarySmtpAddress, Database | Format-Table
---James Bond
[email protected]
-------ExchMbxSvr1\First Storage Group\VIPMBX
Another way of extracting and then working with data is to direct the
output of a command into a variable, in which case you have a complete picture of the object’s properties in the variable. For example, this command
loads all of the available information about the server called ExchMbxSvr1
into the $Server variable:
$Server = Get-ExchangeServer –id ‘ExchMbxSvr1’ -Status
We can now extract additional information about the server to use as we
please by stating the name of the property that we’re interested in (by the
way, specifying the –Status parameter requests Get-ExchangeServer to provide some additional information about the current domain controller and
Global Catalog that the server is using). For example, to get the domain that
the server belongs to:
4.1 EMS: Exchange’s management shell
$Domain = $Server.Domain
$GC = $Server.CurrentGlobalCatalogs
You can also use a variable as an array and populate the array with a call
to a command. In this example, we populate an array called $Mailboxes with
a call to Get-Mailbox, using a filter to ensure that we only include actual
mailboxes (and not room or equipment mailboxes). We do include legacy
mailboxes (those that have not yet moved to Exchange 2007 servers such as
those that use Exchange 2000 or Exchange 2003). Remove the check for legacy mailboxes to exclude these users:
$Mailboxes = Get-Mailbox –Filter {RecipientTypeDetails –eq
‘UserMailbox’ –or RecipientTypeDetails –eq ‘LegacyMailbox’}
Once populated, you can then navigate through the array as follows:
$Mailboxes[2] etc etc etc.
Or reference specific properties of the objects using the “.” operator.
Some values need a little more processing before you can use them
directly. For example, if you look at the PrimarySmtpAddress property of a
mailbox, you see a value like this:
Length Local
------ ----19
To make the property more useful, you have to convert it to a string. For
example, creating a string from the value as follows:
$Address = $Mailbox[53].PrimarySmtpAddress.ToString()
Gives us something more familiar:
[email protected]
Chapter 4
4.1 EMS: Exchange’s management shell
Using common and user-defined variables
PowerShell includes a number of variables that you will use a lot. $True and
$False are variables that you can pass to shell commands and scripts to check
for true and false conditions. Usually, $True is equivalent to setting a check
box for an option in EMC and $False is equivalent to clearing a check box. If
you prefer numeric values, you can replace $True and $False with 1 (one)
and 0 (zero). Other global variables that you commonly meet as you work
with PowerShell include $Null (no value), $home, which returns the user’s
home folder, and $pwd, which returns the current working folder. Important
Exchange 2007 variables include:
$ExBin: Points to the Exchange 2007 bin directory (usually \Program
Files\Microsoft\Exchange Server\bin)
$ExScripts: Points to the Exchange 2007 scripts directory (usually
\Program Files\Microsoft\Exchange Server\Scripts)
$ExInstall: Points to the Exchange 2007 installation directory (usually \Program Files\Microsoft\Exchange Server)
You can use these variables to access files in these directories. For example, to see a list of scripts provided by Exchange, type:
Dir $ExScripts
Checking that a value is $True or $False is a common occurrence. For
positive conditions, you can shorten the check by just passing the property to
check against and PowerShell will assume that you want to check whether it
is true. For example, let’s assume that we want to find out what mailboxes are
enabled to use Outlook Web Access. We can use this command and as you
can see, there is no mention of $True, but it works:
Get-CasMailbox | Where-Object {$_.OWAEnabled} | Select Name
Note the use of $_ in the last command. $_ is a very important variable
because it points to the current object in the pipeline. For example, if you
create a filter to look for people in a certain department because you want to
update the name of the department, you might do something like this:
Get-User | Where-Object {$_.Department –eq ‘Legal’} | Set-User
–Department ‘Law’
4.1 EMS: Exchange’s management shell
Notice that the Department property is prefixed with $_ to indicate that
we want to check this property for every object that the call to the Get-User
command passes through the pipeline. We actually use $_. as the prefix
because it includes the “.” operator to specify that we want to access a property. If we just passed $_, then the comparison would not work because PowerShell would compare “Legal” against the complete object.
User-defined variables can be integer, decimal, or string—you decide by
passing a value to the variable you want to use. For example:
$Tony = ‘Tony Redmond’
$Figure = 15.16
This obviously creates a string variable while the second variable holds a
decimal value. Variables are case insensitive and case preserving. Using the
example shown above, I can refer to $Tony as $TONY or $tony or even
$ToNY and PowerShell will refer to the same variable. Variables are local
unless you declare them to be global by prefixing them with Global, as in:
$Global:Tony = ‘Tony Redmond’
Once a variable is global, you can reference it interactively and in scripts
that you can. Be careful how you use quote marks in PowerShell because
while it may appear that double and single quotes are interchangeable, there
is a subtle difference that may catch you out. Single quotes represent a literal
string, one that PowerShell will use exactly as you provide it. Double quotes
mean that PowerShell should examine the string and resolve any variable that
it finds inside through a process called variable expansion. Consider this
$n = Now
$n1 = ‘Right now, it is $n’
Right now it is $n
$n2 = “Right now it is $n”
Right now, it is Tue Jan 16 17:59:54 2007
You see the difference a little quote mark makes? Best practice is to use
single quotes whenever you are sure that you want a string variable to stay
exactly as you have typed it and to use double quotes elsewhere. You cannot
mix and match the different types of quotation marks to enclose a variable as
Chapter 4
4.1 EMS: Exchange’s management shell
PowerShell will refuse to accept the command if you try. You will not do any
great harm if you use double quotes instead of single quotes, but it is best to
use single quotes as the default. Moving away from strings:
$Tony = 1
Creates an integer, and to create a decimal variable, we can do something
like this:
$Tony = 17.55
You can even perform a calculation, as in $C = 17*5 or $Sum =
(2*7.15)/4. By the way, do not include hyphens when you name variables,
as PowerShell will interpret the hyphens as parameters. In other words,
$ServerName is a good name for a variable while $Server-Name is not.
Like any good scripting language, PowerShell supports conditional
checking with IF and ELSEIF that you will mostly use in scripts. It’s easy to
generate code that goes through a certain number of iterations with constructs such as 1..100 | ForEach-Object <command…>. We will go through
examples of these constructs as we use more sophisticated PowerShell commands in later chapters.
Another thing that catches the PowerShell novice is how to call a script.
The basic rule is that if the script is in the working directory (the directory
that you are currently in), then you prefix the name with “.\”
If you’re not in the right directory, you can move to where you want to
be with the cd command:
cd c:\Scripts\
Alternatively, you can supply the full path to where the script is located:
One of the few things that PowerShell misses is comprehensive debugging capabilities. About the only thing that you can do is to use the SetPSDebug command before you run a script to step through commands. Don’t
4.1 EMS: Exchange’s management shell
get too excited because if you’ve been used to set debug points in programs or
step backward and forward through instructions to examine the results of
commands, change the value of variables, and so on, you’ll be disappointed
with what you can do to debug PowerShell scripts during development. In
some respects, writing a PowerShell script is a matter of trial and error. See
the help file for more details about the Set-PSDebug command. For now, the
way that I use it as is follows:
Set-PSDebug –Trace 2 –Step
You may have noticed the –Identity parameter in use in some of the commands that we have reviewed. Objects like mailboxes, groups, contacts, connectors, and even Exchange servers have identities that allow you to focus in
and work with on a specific object within a set of objects (think of a pointer
to an item in an array). For example, if you issue the Get-ExchangeServer
command, you retrieve a list of all of the Exchange servers in the organization. If you want to work with one server, you have to tell the shell what
server that is by passing its identity. For example, to work with just the server
named ExchMbxSvr1:
Get-ExchangeServer –Identity ExchMbxSvr1
If you want to, you can retrieve a list of objects and store them in a variable and retrieve the values as you wish. The variable holds the objects as an
array. For example, to populate a variable with a set of mailboxes hosted by a
$Mbx = Get-Mailbox –Server ExchMbxSvr1
To retrieve the different objects in the array, pass the number of the
object that you want to work with, starting from zero. For example, to fetch
the first mailbox in the array:
Being more specific, you can ask for one of the object’s properties. For
example, to get the identity of the first mailbox in the array:
Chapter 4
4.1 EMS: Exchange’s management shell
DistinguishedName :
CN=Tony Redmond Users
CN=Tony Redmond,OU=Exchange
Tony Redmond
You may be surprised by the amount of information returned here for
the mailbox’s identity (it’s all defined in the schema), but it contains all of
the ways that you can navigate to this object via its relative distinguished
name (rdn), distinguished name, GUID, and name. Normally, you’ll just
use the name of a mailbox to find it, but you can use the other methods and
Exchange will find the mailbox. There is no requirement to parse out the
part of the identity that you want to use or trim values before you use them
to find information—PowerShell does it all for you. For example, you can
use an identity to discover the groups that a user belongs to. Here’s the code:
$U = (Get-User –id Redmond).Identity; Get-Group –Filter
{Members –eq $U}
The first command loads the user’s identity into a variable, the second
scans through all groups to discover any that include the user in their membership. As you can see from Figure 4.4, EMC displays similar results when it
displays the “Member Of ” property page for the same mailbox. Scanning the
membership list of groups to discover string matches using the Get-Group
command is never going to be as quick (and will get slower and slower as the
number of groups in the forest grows) because a string compare will never get
close to the backward pointers that the GUI uses to display group membership when it comes to speed of access, so don’t be surprised if scanning for
group membership in this way takes some time to complete.
The great thing about identities is that you sometimes don’t need to
bother about using them! This situation occurs when you pipe information
fetched from one command for processing by another command because the
shell understands that it needs to operate on the current object that has been
fetched through the pipe. Fetching objects with one command and processing them with another is a very frequent thing that you’ll do as you work
with Exchange data. For example, let’s assume that you want to change the
value of the Office property for a set of users who have moved to a new
building. It would be tedious if you had to fetch the identity of each user
individually, retrieve their identity, and then use that to make the change. A
simple pipe does the trick because PowerShell knows that it can use the
4.1 EMS: Exchange’s management shell
Figure 4.4
Using an identity
to discover group
stream of data from one command to identify the objects that it has to process with another. Here’s how we might update the Office property for a
complete set of users without any mention of an identity!
Get-User –Filter {Office –eq ‘Building A’} | Set-User –Office
“Building B”
Working in a multi-domain forest
By default, when you start EMS, Exchange sets the scope for queries performed against Active Directory to be the domain that the server belongs to.
This is fine if you operate a single domain forest, but it is definitely not okay
if you have to manage objects in a multi-domain forest because it means that
any query that you perform will only return objects from the local domain.
For example, let us assume that you have a classic forest implementation
where a root domain has three child domains to support accounts for the
regions Europe, America, and Australia. Your Exchange servers are installed
in the regional domains. If you log into a server installed into the Australian
domain and execute:
Get-User | Where {$_.StateOrProvince –eq ‘TX’} | Select Name
Chapter 4
4.1 EMS: Exchange’s management shell
You would expect this command to return a list of users who reside in Texas.
Naturally, these accounts are in the domain that hosts American users.
Because your scope is set to the Australia domain, the command returns nothing, or maybe some Australian users who have decided to relocate to Texas.
Another interesting example is when you have mailboxes from multiple
domains hosted on an Exchange server that is in one of the domains (or a
completely different domain). In this example, we search for anyone whose
mailbox is on a server in the Americas domain and is located in Sydney.
Get-Mailbox –Server AmericasMbx1 | Where {$_.Office –like
‘Sydney’} | Select Name
The only mailboxes returned by this search will be accounts that
belong to the Americas domain. Any accounts in the Australia domain that
happen to be hosted by the Exchange server in the Americas domain will
be ignored until you set the scope of the query to cover the entire Active
Directory forest.
The scope used by Exchange is set in the ADminSessionADSettings.ViewEntireForest global variable that Exchange defines in the
Exchange.PS1 file in the \Bin directory of the location where you installed
Exchange. EMS runs the commands in the Exchange.PS1 file as a bootstrap
to load commands and define a number of variables that EMS uses to control
different aspects of its environment. You can edit the ViewEntireForest property of the AdminSettingADSettings variable to set the scope to view the
entire forest as follows:
## Reset the Default Domain
$global:AdminSessionADSettings.ViewEntireForest = $True
Setting the proper scope is important if you want to retrieve the right
information from your queries. To reset the scope so that Exchange only uses
objects from the local domain, type:
$AdminSessionADSettings.ViewEntireForest = $False
Creating the proper scope for Active Directory lookups is critical for
some commands. For example, the Update-SafeList command gathers safe
list information from user accounts and aggregates them to pass to an Edge
server so that the anti-spam agents running on the Edge server can use this
information to filter out spam. If you execute Update-SafeList with the
wrong scope, you will synchronize only the safe list data from the in-scope
4.1 EMS: Exchange’s management shell
domain to the Edge server and the anti-spam agents may end up blocking
some valid messages as spam.
If you do not want to set your scope to the entire forest, a workaround is
to specify a Global Catalog server in the remote domain to perform the
query there. Taking our previous example, if we changed it to include a
domain controller in the Americas domain, then we would be able to retrieve
the right data at the expense of missing out on the Australians who want to
live in Texas:
Get-User –DomainController | Where
{$_.StateOrProvince –eq ‘TX’} | Select Name
The natural question at this point is whether changing scope will affect
how you work with EMS. The answer is yes because when you set a forestwide scope, you work with data from across the forest rather than the local
domain. Unless you use parameters to focus in on particular groups of
objects, such as specifying that you want to work with the mailboxes from
one server, you will probably have to wait longer for a response from any
command. This is because you will ask EMS to process commands that deal
with servers, mailboxes, databases, or storage groups across a complete forest
rather than just one domain, but in most cases the wait is worthwhile
because you see the big picture and don’t run the risk of missing something.
As we have just seen, when you start EMS, Exchange runs a PowerShell script
called Exchange.PS1 to load the Exchange snap-in and define a set of variables used by EMS. You can see how EMS calls Exchange.PS1 by examining
the properties of the EMS option that the Exchange installation adds to the
Start menu, as shown in Figure 4.5. Note the property pages to control font,
layout, and colors. These are settings that you can use to change the appearance of the EMS command window to use a more readable font, use a larger
font size, or use different colors.
We have already seen how the $AdminSessionADSettings global variable is set in the Exchange.PS1 file to set the scope for Active Directory
lookups during an EMS session. You can define other variables for your
own purpose in your personal PowerShell profile, which is located in \My
Documents\WindowsPowerShell\Microsoft.PowerShell_Profile.PS1 7.
Windows does not create your personal PowerShell profile automatically, so
you will have to create the file manually by using an editor such as Notepad.
Afterwards, you can edit the file to create whatever variables you need or
Earlier versions of PowerShell used a file called Profile.PS1. Either filename works.
Chapter 4
4.1 EMS: Exchange’s management shell
insert other commands to create your personal working environment. PowerShell uses a global variable called $Profile to point to your profile, so once the
file is created, you can edit it from within EMS by typing the command:
Invoke-Item $Profile
Figure 4.5
Properties of EMS
The contents of a sample profile are shown below. You can see the effect
of these commands in Figure 4.6. The commands contained in your profile
are effective the next time you launch EMS. Note that PowerShell executes
your profile first and will then execute the commands in Exchange.PS1.
# welcome message
‘You are now entering PowerShell:’ + $env:Username
$Global:Start = Now
Write-Host “Session starting at $Start”
Set-Location c:\temp\
4.1 EMS: Exchange’s management shell
Figure 4.6
The effect of a
personal profile
You can scan the Internet to discover the many different customizations
that people have applied through their personal profile such as defining new
aliases that they like to use as command shortcuts, creating a different
prompt, and so on.
PowerShell in batch
If you write a script that uses Exchange PowerShell commands that you want
to execute in batch mode, you have to call the Add-PSSnapin command to
load the Exchange PowerShell snap-in before you attempt to work with
Exchange as otherwise PowerShell won’t know anything about the Exchange
commands like Get-Mailbox, Get-MailboxDatabase, and so on. For example, if you wrote a script to report on mailbox quotas for everyone on the
organization, it might be something like this:
Get-MailboxServer | Get-MailboxStatistics > c:\temp\
It all looks very simple and this one-line command will work interactively, but won’t generate a thing if you run it in batch. In fact, PowerShell
does not provide a command to run a script in batch mode, so you have to
rely on the Windows scheduler to submit a BAT (batch file) for processing.
The batch file calls PowerShell and passes the name of the script as a parameter, so the kind of batch file that we need is:
PowerShell.Exe c:\Script\Get-All-Mailboxes.Ps1
Inside the script file, we still have some work to do because we have to
tell PowerShell about the Get-MailboxStatistics command by loading the
Exchange snap-in:
Add-PSSnapin Microsoft.Exchange.Management.PowerShell.Admin
Get-MailboxServer | Get-MailboxStatistics > c:\temp\
Chapter 4
4.1 EMS: Exchange’s management shell
Adding the Exchange snap-in is not an intuitive thing to do when you
come to design scripts that run in batch, but when you think a little, it’s logical that you should have to create the same environment for the batch script
that you’d have when you work with EMS interactively. As explained above,
when you start EMS, it loads PowerShell and then the Exchange snap-in, so
you have to do the same in your scripts. Note that it’s also a good idea to
establish the exact scope of the forest that you want commands to work for
within scripts.
Execution policies
By this point, you’re probably convinced that EMS is pretty powerful and
that just a few lines of code can affect many objects, so the thought might
have crossed your mind about how to control the ability of users to execute
EMS command. PowerShell supports a concept called Execution Policies to
control how users can execute commands. Execution policies define the conditions under which PowerShell loads files for execution. There are four policies: Restricted, AllSigned, RemoteSigned, and Unrestricted.
By default, Microsoft configures PowerShell to run under the Restricted
execution policy, which is the most secure mode. In this mode, PowerShell
operates as an interactive shell only and you cannot call scripts. In fact, all
you can do is issue PowerShell commands interactively at the shell prompt.
Table 4.2 lists the alternate modes together with the potential trade-off in
security that you may have to make.
Table 4.2
PowerShell execution policies
Execution Policy
Execution Context
Users can issue commands at the PowerShell prompt.
Users cannot call scripts.
Users can run scripts, but the scripts and any configuration files must be signed by a trusted publisher.
RemoteSigned (default for
Users can run scripts.
Scripts and configuration files that are downloaded from
communications applications such as Outlook, Outlook
Express, IE, and Messenger must be signed by a trusted
Users can run scripts.
Scripts that are downloaded from communications
applications can be run after the user acknowledges that
this content may be dangerous. No signature is required.
4.1 EMS: Exchange’s management shell
You can use the Get-ExecutionPolicy command to discover what the
default execution policy is on an Exchange 2007 server. If you attempt to run
an unsigned script that doesn’t comply with policy, PowerShell signals that it
cannot load the script. Scripts are signed with the Set-AuthenticodeSignature command, but you need to get a valid certificate first. The certificate
can be one that you generate yourself or one that you buy in from a commercial vendor, such as Verisign. See the Microsoft documentation for details of
how to generate and apply certificates to sign scripts.
Execution policies are not a way around Exchange permissions and you
will not be able to execute anything through EMS that you can’t do through
EMC. For example, if you don’t have at least view-only permission for a
server, EMS will return an error if you attempt to retrieve any information
that isn’t part of the organization configuration (which you gain access to by
having view-only permission). To see what I mean, assume that you have
view-only permission for a server but not for the server called ExchHub1.
This command works:
Get-ExchangeServer –ExchHub1
But this command does not work:
Get-ExchangeServer –ExchHub1 –Status
The reason is that the first command reads from the Exchange organization configuration in the Active Directory. The second combines the first set
of information with some additional information from the system registry
from the target server. You don’t have permission over that server, so the command fails with an error.
Obviously, running an Exchange server with an unrestricted execution
policy is a horribly bad idea. In fact, you should avoid any deviation from the
default policy unless you have an excellent reason why you need to change.
To change the policy, use the Set-ExecutionPolicy command to update the
default execution policy on an Exchange 2007 server. This command updates
the registry. For example:
Set-ExecutionPolicy –ExecutionPolicy AllSigned
The change to the execution policy will come into effect immediately.
Be sure to test any change that you want to make before putting the change
into production because it may break scripts that you or applications depend
on. Note that any update to the registry can be overridden by group policy,
Chapter 4
4.1 EMS: Exchange’s management shell
which is the way to implement a consistent execution policy across a large
Sending email from the shell
It may seem awfully obvious for an email system to provide some method to
create and send messages when it enhances a scripting language, but EMS
provides no special commands to create and send messages. Everything has
to be done with the basic SMTP commands for Windows that PowerShell
includes. We already reviewed an example of what the SmtpClient .NET
class can do in a couple of lines of script. For example:
$SmtpClient = New-Object System.Net.Mail.SmtpClient
$ = ‘’
$From = ‘[email protected]’
= ‘[email protected]’
$Title = ‘Answer to your questions’
$Body = ‘Status Message follows – Here is some important
$SmtpClient.Send($From, $To, $Title, $Body)
These commands work if you point them to an SMTP server that is
willing to relay messages for clients. Most Exchange servers and especially
Exchange 2007 hub transport servers will restrict the ability of clients to relay
messages, especially unauthenticated clients. However, you can pass credentials by either setting the UseDefaultCredentials property to $True (the
default is $False), which tells Windows to fetch whatever default network
credentials are cached for the current logged-on user or you can pass explicit
credentials by setting the property with the necessary values for username
and password. For example:
$SmtpClient.UseDefaultCredentials = $True
$credentials = New-Object
System.Net.NetworkCredential(‘[email protected]’, ‘Password’)
SmtpClient.Credentials = $Credentials
Including credentials in plain text in a script isn’t very secure. You can do
a much better job by prompting the user for credentials to use:
4.1 EMS: Exchange’s management shell
$Credentials = Get-Credential
This command puts up a standard Windows login dialog for the user to
input their username and password. We can then fetch the data and store it
as follows:
$Username = $Credentials.Username
$Password =$Credentials.GetNetworkCredential().Password
$Credentials2 = New-Object
SmtpClient.Credentials = $Credentials2
So far we’ve sent a pretty simple message. What about if we want to read
addresses for all the mailboxes in a container and send a message to them. We
can use a container such as a distribution group (use Get-DistributionGroupMember to fetch the names) or from an organizational unit as shown
below. This script is one that I use to generate a set of messages to load test
systems. The script differs from the simple examples that we have used to date
by also adding an attachment to the message and by reading in some content
from a text file to become the body of the message. Note that if you want a
copy of the message, you have to include your mailbox in the recipient list as
the code shown here generates and sends a basic SMTP message.
# Spam lots of users
# Get a list of the members of a distribution group
$MbxSet = Get-Mailbox –OrganizationalUnit ‘Exchange Users’
–Filter {RecipientTypeDetails –eq ‘UserMailbox’}
Write-Host "About to spam $($MbxSet.Count) users..."
$Title = ‘Very Important and Critical Message to Important
# Determine who's sending the message
$SourceMailbox = Get-Mailbox -id ‘Redmond’
$SourceAddress = $SourceMailbox.PrimarySmtpAddress.ToString()
# What's the address of our SMTP server (that will relay mail)
$SmtpServer = ""
Chapter 4
4.1 EMS: Exchange’s management shell
# Load some text into the message body
$Item = Get-Item c:\temp\objects.tmp
$MsgBody = Get-Content $Item.Fullname
# Create an attachment
$File = ‘C:\temp\Information.txt’
$MsgAttachment = New-Object System.Net.Mail.Attachment
$MbxCount = $MbxSet.Count - 1
$X = 1
# Create a set of addresses in the format expected by email
$Recipients = 0..$MbxCount | ForEach-Object
$ToRecipients = [String]::Join(",",$Recipients)
Write-Host "Spamming $($ToRecipients)..."
1..100 | ForEach-Object {
$Subject = $Title + " " + $_
Write-Host "Generating Message $_ )"
$Msg = New-Object System.Net.Mail.MailMessage
$ToRecipients, $Subject, $MsgBody)
$SmtpClient = New-Object System.Net.Mail.SmtpClient
$SmtpServer, 25
Of course, if you do not want to get involved in scripting, you can simply create a message in the proper format (containing all the necessary fields)
and put it into the SMTP pickup directory on a hub transport server.
Exchange will find the file there and assuming that it does not contain any
invalid or illegal lines, Exchange should send the message for you.
The lesson from all of this is that you do not need standalone programs
such as MAPISend or BLAT that administrators have used with previous versions of Exchange to generate and send email to report the status of jobs, that
various events have occurred, or any other message that they needed to generate as part of operations. The scripting capabilities of PowerShell and the
ability to access and process Exchange data through the shell are much more
4.2 Learning from EMC
powerful than anything we’ve had in the past, short of breaking out the
MAPI programming manuals and writing some heavy-duty code, which
seems excessive if you only want to generate a message.
Now that we have an insight into how PowerShell can work with standard Exchange objects and other important concepts such as variables and
identities, we can plunge into the detail of how PowerShell dramatically
enhances the management framework for Exchange 2007.
Learning from EMC
As we know, EMS commands provide the foundation for the options offered
through EMC. It is therefore logical to assume that we can gain an insight
into how shell commands function by working with EMC – at least for the
tasks that you can perform through EMC. Let’s assume that you want to create a mail-enabled account. This option is available through EMC and a
browse through the Exchange 2007 help file reveals that the New-Mailbox
command creates mail-enabled accounts. You also know that you need to
populate a number of fields in the Active Directory object for the new
account. These fields include:
Name: The common name (CN) of the account.
Database: The distinguished name of the mailbox store to host the
new account.
UserPrincipalName: The name of the user in SMTP address format.
OrganizationalUnit: The organizational unit within the Active Directory into which to place the new account.
Password: The password to assign to the new account.
Note that equipment mailboxes and room mailboxes have simpler
requirements in terms of creation because they are disabled Active Directory
objects. However, these mailboxes have their own distinct set of properties
that you can manipulate that don’t exist for regular mailboxes.
The Exchange 2007 help file describes how you can provide the necessary values to populate these fields when you call New-Mailbox. However, the
syntax of New-Mailbox can be hard to understand, so it’s nice to be able to get
a hint about how to start and that’s where EMC comes in. If you create a
mailbox with EMC, you provide the necessary values through the GUI.
Afterwards EMC creates the mailbox by calling New-Mailbox.
A really neat feature of EMC is that it displays the code that it uses when
it completes a task and prompts you to use the CTRL/C keystroke to copy
Chapter 4
4.2 Learning from EMC
Figure 4.7
PowerShell code
from EMC
the commands. The Exchange help file is another great source of examples of
PowerShell code and you also can cut and paste the code from the help file.
The sample code in the help file is more generic than the code that EMS generates inside your own organization, but it provides a useful starting point for
the more esoteric commands.
Figure 4.7 shows the actual code generated to create a mailbox for a user
called “Mary Jones.” You can start up any editor and paste the commands to
begin the process of automating the creation of new mailboxes. We now have
some code to work with. You might assume that you can simply edit the code
to add the appropriate values for a new user but there is one minor challenge
that stops us doing this. Passwords have to be secure and they would not be
secure if you could enter them in plain text within a script. PowerShell allows
you to declare a variable as a secure string but you can’t just assign a value to
it in a script by specifying a simple text string that remains readable in the
script.8 For now, the easiest way to provide a password is to use the ReadHost command to prompt for a password when the script runs to create the
It is possible to use the ConvertTo-SecureString command to convert a plain text string to a secure string, but
we still have the security problem of the plain text value in the file, so overall it’s best to prompt the user to provide the
4.2 Learning from EMC
new account. Later on in this chapter we’ll discuss how to use more complex
code to compose a secure string based on some data read in from a load file
and then to use the secure string for a password when you perform a bulk
creation of new accounts.
Providing a single password to a prompt or encoding strings fetched
from a load file during a bulk load operation are not ways around password
policies that the Active Directory may enforce. If you input a password that
doesn’t comply with policy (too short, not containing the right character
mix, etc.), New-Mailbox will fail when it attempts to create the new account
in the Active Directory. The complete code that EMS generates to create a
new account is therefore:
$password = Read-Host "Enter password" -AsSecureString
New-Mailbox -Name:'James Bond' -Alias:'Bond'
-OrganizationalUnit:’ Users/Users'
-Database:'CN=Mailbox Store 1,CN=First Storage
Administrative Group (FYDIBOHF23SPDLT),CN=Administrative
Groups,CN=XYZ,CN=Microsoft Exchange, CN=Services,
-UserPrincipalName:'[email protected]' -SamAccountName:'Bond'
-FirstName:'James' -Initials:'' -LastName:'Bond'
-ResetPasswordOnNextLogon:$True -Password $password
As you will see as we investigate different examples, the code that EMC
generates to create a new mailbox is a tad verbose in the way each parameter
is fully spelt out and objects like the target database are identified precisely.
You normally don’t have to be quite so exact because EMS is pretty good at
interpreting input to know what you mean. For instance, if there is only one
database on the local server that’s called “VIP Mailboxes,” then that’s all you
have to specify unless you really want to spell out the full distinguished name
or GUID of the database.
Creating a customized script is not the way to do the job if you only
have one or two accounts to create. Using PowerShell becomes much more
compelling when you need to process lots of objects at one time or you need
to automate the provisioning of user accounts, perhaps by linking the creation to a feed from an HR system.
Similar commands are used to create equipment, room, shared, and
linked mailboxes with slightly different parameters being required to identify
these mailbox types. For example, when you create a room mailbox, you
specify the –Room parameter; an equipment mailbox takes the –Equipment
Chapter 4
4.3 Using EMS to work with mailboxes
parameter; and you must pass the –LinkedMasterAccount parameter to create a linked mailbox. You can discover the exact format of these commands
by creating a mailbox of the required type with EMC and examining the
commands that the GUI generates.
Using EMS to work with mailboxes
The shell provides many commands to maintain mailboxes and distribution groups. While we have already explored how to use PowerShell to
report mailbox data and to create a new mail-enabled account, many other
examples of PowerShell commands in use are dotted throughout the book.
An exhaustive description of PowerShell and the Exchange 2007 Management Shell would require a separate thousand-page book to describe adequately. Instead of attempting that task, I want to look at some specific
examples that illustrate the power of EMS and investigate how you can use
shell commands to automate common Exchange administrative tasks. The
aim here is to help you make the cultural transition from a point where we
look for an option in the GUI to do everything to an appreciation of how
you can use EMS to get your work done. The structure of the EMS commands look similar to other scripting languages and those who have ever
had the chance to administer a UNIX or Linux server will be at home with
PowerShell commands.
Before starting, we should note that you have to connect to a server
that supports the mailbox role before you can work with mailboxes, recipients, or distribution groups and the account that you log into must possess
at least Exchange Recipient Administrator permission for the server that
you want to use.
Creating a new mailbox with a template
When you create a new mailbox with Exchange 2003, you can use the Copy
option in the AD Users and Computers console to copy an existing user
account and create a new account that inherits all of the Exchange properties.
You cannot do this any longer because Microsoft has removed the Exchange
add-ons from the AD Users and Computers console. Furthermore, the EMC
console does not offer administrators the opportunity to create a new mailbox from a copy.
Instead, you have to use the –TemplateInstance parameter to point to
an existing mailbox when you create a new mailbox with the New-Mailbox
$Password = Read-Host ‘Enter Password:’ -AsSecureString
4.3 Using EMS to work with mailboxes
New-Mailbox –Alias ‘DPR’ –OrganizationalUnit ‘Exchange Users’
–DisplayName ‘Redmond, Deirdre’ –FirstName ‘Deirdre’
–LastName ‘Redmond’ –Database ‘VIP Mailboxes’
–UserPrincipalName ‘[email protected]’ –Name ‘Deirdre Redmond’
–TemplateInstance (Get-Mailbox –id Redmond) –Password
As you can see, you still have to specify many other parameter values, so
it is not just a case of pointing to a mailbox and copying everything from that
mailbox. Later on, when you examine the new mailbox, you will find that
specifying another mailbox as a template tells Exchange to copy selected
properties, but not others. The properties that Exchange copies include:
Mailbox quota limits
Protocol settings (for example, what version of Outlook can connect
to the mailbox)
Language setting for Outlook Web Access
Outlook Web Access feature segmentation
Spam Confidence Level settings
Address list membership
The set of 15 Exchange custom attributes
Maximum send and receive size
For some companies, the change in behavior caused by the move from a
simple copy operation to the requirement to use a template with the NewMailbox command will be an annoyance. For others, especially companies
that depend on the ability to copy mailboxes in every detail as part of their
user creation process, it will provoke some unexpected change. The good
news is that it is possible to script the complete creation process to generate
an exact replica from a template mailbox; the bad news is that you have to
build the script to meet your company’s needs.
One small bug in using a template mailbox with the New-Mailbox command is that Exchange does not create a primary SMTP address for the new
mailbox. Microsoft will fix this Exchange 2007 bug in SP1. For now, you
have to update the mailbox after you create it to add the address.
Chapter 4
4.3 Using EMS to work with mailboxes
Setting and retrieving mailbox properties
First, here is an example of a single line of script that accomplishes a task that
you would need many lines of ADSI code to do in Exchange 2003. We want
to update a user’s mailbox to give “Send on Behalf ” permission to another
user. The command is:
Set-Mailbox –identity XYZ\Redmond –GrantSendOnBehalfTo
[email protected]
This command uses the Set-Mailbox command to look for the user
account that belongs to XYZ\Redmond (we could also pass a UPN or an
SMTP address as the identity to find the account) and set the Send on Behalf
permission to the user account identified with the SMTP address
[email protected] UPNs are probably the easiest way to identify mailboxes,
especially if they match users’ SMTP addresses. Note that you can shorten
parameters as long as they remain unique. For example, when you create a
new mailbox, you can use –id for –identity or –org for –organizational unit.
The shell will prompt you for a value if you fail to provide one for a mandatory parameter. Some parameters like –identity are positional so you do not
need to specify them. In other words, the following command is as good as
our earlier example because Set-Mailbox assumes that the first parameter
passed is the identity of the mailbox to work with.
Set-Mailbox ‘Jane Bond’ –GrantSendOnBehalfTo ‘James Bond’
My own preference is to be specific about the identity parameter, so I
tend to use it all the time. If you give someone the ability to send mail for
another user, you will also need to give them the appropriate Active Directory permission if you want to allow them to access your mailbox and work
with its contents.
Add-AdPermission –id ‘Jane Bond’ -ExtendedRights Send-As
-User ‘James Bond’
Knowing the name of the available parameters that an individual command supports can be difficult. As we learned earlier in this chapter, PowerShell makes it easy to discover what parameters are available for a command
by allowing you to use the tab key to cycle through the available parameters.
This feature is especially useful for commands that have many parameters,
such as those that deal with mailboxes. For example, if you type:
4.3 Using EMS to work with mailboxes
Set-Mailbox –
Now press the tab key, EMS starts to go through the available parameters. Moving on to a much simpler command, if you need to disable a mailbox, you can do it with a one-line command:
Disable-Mailbox ‘[email protected]’
And if you have to disable a whole set of mailboxes quickly, you can do
it with Disable-Mailbox too. Here we disable all of the mailboxes on the
“MbxServer1” server:
Get-Mailbox MbxServer1 | Disable-Mailbox
Now that we have a few disabled mailboxes, we can enable selected ones
again with the Enable-Mailbox command:
Enable-Mailbox [email protected] -database ‘Mailbox
We can also process mailboxes for a complete server:
Get-Mailbox –Server MbxServer1 | Enable-Mailbox
Get-Mailbox is a very useful command that gets used a lot. For example,
to list all the users on a specific server:
Get-Mailbox –Server MbxServer1 | Format-Table
Or just one database:
Get-Mailbox –Database ‘VIP Mailboxes’ | Format-Table
You can be more specific about a database and identify that it comes
from a different server:
Get-Mailbox –Database ‘ExchMbxSvr1\Storage Group 1\VIP
Chapter 4
4.3 Using EMS to work with mailboxes
Or even just the mailboxes that belong to a certain organizational unit
on a specific server:
Get-Mailbox –Server ExchMbxSvr1 –OrganizationalUnit
Another useful trick is to search for mailboxes using Ambiguous Name
Resolution (ANR). In other words, you provide EMS with a value and it
finds any mailbox that might possibly match the value in its name or alias.
For example:
Get-Mailbox –Anr ‘Red’
Another special case is where you simply want to search based on name.
Microsoft did some work in EMS to allow you to pass a command like this:
Get-Mailbox *James*
This command performs a wildcard search for any user that has “James”
in their name or SMTP address! You can even pass wildcard characters to
specify what fields you want output, as in:
Get-Mailbox *Ja* | Format-Table Name, *depart*, *quota*
This command lists the name, department, and all the quota properties
of any user who has “Ja” in their name or SMTP address. You probably won’t
use this kind of command often, but it’s interesting to see just how much can
be done in a few characters of code (older programmers who worked on Digital operating systems may remember the late lamented Teco language, which
could do some of the same tricks).
Unfortunately, you can’t use Get-Mailbox to discover what mailboxes
exist in the databases within a specific storage group. However, we can solve
the problem with a combination of the Get-MailboxDatabase command to
interrogate the databases on a specific server and a loop through the databases to return the mailboxes for the storage group that we are interested in:
Get-MailboxDatabase -Server MailboxServer1 | Where
{$_.StorageGroup.Name -eq ‘VIP Storage Group’} | ForEach-Object
{Get-Mailbox -Database $_.Name} | Select Name, Alias, Database
4.3 Using EMS to work with mailboxes
By eliminating the –Server parameter, you could execute a command to
look for mailboxes that belong to a storage group with a specific name that
occur on any server in the organization. For example, you could look for
mailboxes from the “First Storage Group,” the default name given to storage
groups on all Exchange servers. Such a command is likely to take a long time
to execute and you will see an error if a server is unreachable for any reason.
Even if the $Error variable is always available to check if things go wrong, the
degree of error checking and handling in one-line PowerShell commands is
normally not great! As a great example of how elegant PowerShell can be, one
of the book’s reviewers pointed out that a faster solution to the problem is
provided by:
Get-Mailbox –Server (Get-StorageGroup ‘VIP Storage
Group’).Server | Select Name, Alias, Database
In this case, we rely on the fact that the Get-StorageGroup command
can return the name of the server where the storage group resides and can use
that as input to Get-Mailbox. Another example is where you want to find out
how many mailboxes are allocated to a server or database. Here we use the
Group-Object command to collect the mailboxes according to the unique
values in the property that we specify:
Get-Mailbox | Group-Object ServerName
Get-Mailbox | Group-Object Database
The ability to group objects is very useful because you can apply it to
any property that an object returns. For example, we might want to discover
how many mailboxes have been assigned a managed folder policy:
Get-Mailbox | Group-Object ManagedFolderMailboxPolicy
----{Gary Olsen, Dung Hoang Khac, Administrator, Liffey
Business Planning
{Aric Bernard, Conference Room 2, Tony Redmond, Jane
Clean out calendar fol... {Leixlip Conference Room}
We can see here that 39 mailboxes don’t have a managed folder policy
assigned (you can find more details about how to create and apply these policies in Chapter 8). EMS reports the names of some of the mailboxes but
truncates the list, so we need to generate a complete list if we want to discover the names of the mailboxes that don’t have a managed folder policy
assigned. We can execute another command using Get-Mailbox for this purChapter 4
4.3 Using EMS to work with mailboxes
pose. In this case, we eliminate room and equipment mailboxes, but you can
use a similar filter to identify these mailbox types if you want to apply specific managed folder policies to them:
Get-Mailbox | Where {$_.ManagedFolderMailboxPolicy –eq $Null
–and $_.RecipientTypeDetails –eq ‘UserMailbox’} | Select Name
A problem with any command that sets out to process a large number of
mailboxes is that you might blow the limit for the total number of items that
Get-Mailbox returns. As with all EMS commands that return items, the
default set is 1,000 items (this is the same as the default filter size applied to
EMC) and it’s easy to find servers or mailbox databases that support more
than 1,000 mailboxes. If you know that you need to process more than 1,000
mailboxes, you can increase the limit to a set value:
Get-Mailbox –ResultSize 5000
Or, if you are very brave or just don’t know how many items are likely to
be retrieved:
Get-Mailbox –ResultSize Unlimited
Better practice is to restrict the number of mailboxes that are returned
by applying a suitable filter. For example, the example given above to return
counts of mailboxes per database will scan all of the servers in the organization. You’ll get a much quicker result if you scan just one server by changing
the command to include the –Server parameter:
Get-Mailbox –Server ExchMbxSvr1 | Group-Object Database
----{[email protected], [email protected]
{[email protected], [email protected], k...
{[email protected], [email protected],...
{[email protected], [email protected], wen...
{Test Mailbox 1, Test Mailbox 2}
4.3 Using EMS to work with mailboxes
You can export data that you retrieve with shell commands in CSV or
XML format, which makes it easy to then load the output into a program
like Excel (CSV) or an XML reader. For example:
Get-Mailbox –id Bond | Export-CSV c:\temp\Bond.csv
Get-Mailbox –id Bond | Export-CliXML c:\temp\Bond.xml
You can use the ability to export data in CSV or XML format to report
information about a mailbox using the Get-MailboxFolderStatistics command. This command goes through the folders in a mailbox and reports
information such as the number of items in the folder, the size of the folder,
and the timestamp for the newest item in the folder. For example:
Get-MailboxFolderStatistics –id Bond | Select Identity,
FolderSize, ItemsInFolder, NewestItemReceiveDate | Export-CSV
The default operation for Get-MailboxFolderStatistics is to report
on every folder in a mailbox. You can restrict processing to a single folder, but
only if it is one of the default folders or a custom-managed folder (see Chapter 8). For example:
Get-MailboxFolderStatistics –id Redmond –FolderScope Inbox
1/15/2007 6:25:28 PM
10/18/2006 1:17:20
1/15/2007 5:14:29 PM
Once you have the data in CSV format, you can edit it using Excel or
another similar tool as shown in Figure 4.8. Another example where CSV
output comes in handy is where we want to generate a quick list of all the
users on a server:
Chapter 4
4.3 Using EMS to work with mailboxes
Get-Mailbox -Server ExchMbxSvr1 | Select LastName, FirstName,
Phone | Export-csv c:\temp\UserList.csv
Figure 4.8
CSV output from
You can also use Get-Mailbox to return a quick count of all of the mailboxes assigned to different databases within the organization. As scanning all
the servers in the organization might take some time, you may want to
restrict the scope of the command to a single server by specifying the –server
Get-Mailbox –Server ExchMbxSvr1 | Group-Object Database |
Format-Table Name, Count
The same approach that we took to retrieve mailbox information with
the Get-Mailbox command applies if we need to remove mailboxes. When
we want to remove just one mailbox, we use the Remove-Mailbox command
and specify their UPN. For all delete and disable operations that affect mailboxes and other mail-enabled objects, EMS will prompt you to confirm the
operation before it proceeds.
4.3 Using EMS to work with mailboxes
Remove-Mailbox ‘[email protected]’
And to do the same thing to all mailbox-enabled users on a specific
Get-Mailbox –Server MbxServer1 | Remove-Mailbox
It should now be obvious that you will use the Get-Mailbox command
frequently to fetch information about mailboxes and as we know, Set-Mailbox is its counterpart when it comes to updating a property of a mailbox.
Let’s review some other examples of Set-Mailbox in action. For example, we
can assume that I want to set a very large quota on my mailbox:
Set-Mailbox –id ‘[email protected]’
–UseDatabaseQuotaDefaults:$False –IssueWarningQuota 5000MB
–ProhibitSendQuota 5050MB
–ProhibitSendReceiveQuota 5075MB
After we set quotas for mailboxes, we can check to see if any user has
exceeded a quota using the Get-MailboxStatistics command:
Get-MailboxStatistics | Where
{"IssueWarning","ProhibitSend","MailboxDisabled" -Contains
$_.StorageLimitStatus} |
Format-List Displayname, Itemcount, TotalitemSize
When we discussed the topic of how Exchange 2007 controls mailbox
quotas in Chapter 3, we encountered a one-line PowerShell command that
identified mailboxes that Exchange had disabled because they have exceeded
quota. Now that we know a little more about PowerShell, we can develop the
command a little more to give us more information about disabled mailboxes
on a server and to tell us what we need to do to correct the problem.
$Mbx = Get-MailboxStatistics –Server ExchMbxSvr1 | Where-Object
{$_.StorageLimitStatus -eq ‘MailboxDisabled’}
Foreach-Object ($U in $Mbx)
$M = $U.DisplayName
$PSquota = (Get-Mailbox -id $M).ProhibitSendQuota.Value.ToKB()
$MbxSize = $U.TotalItemSize.Value.ToKB()
$Increase = $MbxSize - $PSQuota + 100
Chapter 4
4.3 Using EMS to work with mailboxes
Write-Host $M " has quota of $PSQuota KB and has used $MbxSize KB allocate at least an extra $Increase KB"
This script loops through all of the mailboxes on a server to detect the
ones that are disabled. The mailbox quota and the value for the ProhibitSendQuota property are retrieved and converted to kilobytes (Exchange
allows you to pass quota values in KB, MB, and GB, but it returns the total
size of the mailbox in bytes). We then output a message to tell the administrator what they have to do. You can see the effect of running the script in
Figure 4.9.
Figure 4.9
Checking for
disabled mailboxes
Speaking of quotas, we might want to restrict the ability of a user to
send and receive very large messages. This might be dangerous for someone
who works for a marketing department, who seems to love circulating large
PowerPoint attachments for everyone to admire. In any case, here is how we
restrict the maximum sizes for sending and receiving messages to 5MB.
Set-Mailbox [email protected] -MaxSendSize 5MB
-MaxReceiveSize 5MB
Many users like to create rules to process incoming messages as they
arrive into their mailbox. In previous versions of Exchange, the rules had to
fit into a 32KB space. In Exchange 2007, Microsoft increases the available
space for rules to a maximum of 256KB with the default limit increased to
64KB. You can check on the quota for any mailbox with the Get-Mailbox
Get-Mailbox –id Redmond | Select Name, RulesQuota
If the user needs more headroom to store rules, you can increase their
quota with the Set-Mailbox command. Exchange will signal an error if you
attempt to increase the quota past 256KB:
Set-Mailbox –id Redmond –RulesQuota 128KB
4.3 Using EMS to work with mailboxes
A common administrative task is where you want to add a new forwarding address to a mailbox.
Set-Mailbox [email protected]
–ForwardingAddress [email protected]
In this case, we have told EMS to add a new forwarding address to a
mailbox and to ensure that mail is copied to both the original mailbox and to
the forwarding address. Sometimes when you change mailbox properties,
you want to add something to the property rather than simply overwriting
existing data. Adding a new SMTP address to a mailbox is a good example of
this kind of processing. All mailboxes that are under the control of an email
address policy will have their primary SMTP address automatically configured by Exchange to ensure compliance with the policy, so you cannot
change the primary email address unless it complies with policy. It doesn’t
make sense to change the primary address anyway because Exchange will
only update it when it enforces the policy. However, you can add a new
address to the secondary SMTP addresses:
$MbxData = Get-Mailbox ‘[email protected]’
$MbxData.EmailAddresses += ‘[email protected]’
Set-Mailbox – id ‘James.Bond[email protected]’ –EmailAddresses
This code is definitely not an example of a one-line command, but you
probably will not be doing complicated manipulation of mailbox properties
interactively except when you test individual commands. It is much more
likely that you will manipulate mailbox properties in scripts if you want to do
anything complicated. With this in mind, we can look at these commands
and see that we:
Retrieved the properties of the [email protected] mailbox and stored
them in the $MbxData variable. We can now access all of the properties by referencing them in the variable, as in $MbxData.EmailAddresses.
Appended a string containing the new SMTP address ([email protected])
to the existing addresses.
Updated the mailbox with the new set of email addresses with the
Set-Mailbox command using the collection of addresses held in the
$MbxData.Emailaddresses property.
Chapter 4
4.3 Using EMS to work with mailboxes
A somewhat more elegant way of doing the same job is to pipe data
returned from Get-Mailbox to return the existing email addresses appended
with the new address back into Set-Mailbox:
Set-Mailbox ‘[email protected]’ -EmailAddresses ((GetMailbox ‘[email protected]’).EmailAddresses + ‘[email protected]’)
This simply proves that there is more than one way to skin a cat or even
to set email addresses. Across all these examples you can see how pipelining
allows us an elegant way to combine fundamentally simple commands to
perform more complex operations.
If you are in doubt about the type of data contained in a variable or
property, you can use the Get-Member command to return details of the type.
This is valuable information when you want to know what kind of methods
you can use to process the data. For example, if we want to know about the
type of information contained in the email addresses for a mailbox so that we
understand how to update it, we can do:
$MbxData = Get-Mailbox –id ‘[email protected]’
$MbxData.EmailAddresses | Get-Member
PowerShell will then list all the information that we need to know about
the different types of proxy addresses supported by Exchange.
Other ways of interacting with mailboxes
There are other commands that interact with mailboxes that we will mention
in passing here. The Get-MailboxPermission command returns the permissions (such as delegate access) that Active Directory accounts holds over a
mailbox. In this example, we’re scanning for any mailboxes that Bond’s
account has access to.
Get-MailboxPermission –User Bond | Format-List User,
Identity, AccessRights, Deny
We use the Set-CasMailbox command to work with mailbox properties that interact with Client Access servers. For example, if you want to
determine the options of Outlook Web Access that a mailbox can use such
as the option to use the premium client, or to enable a specific feature of
Exchange like ActiveSync. For example, here’s how to enable a mailbox to
use ActiveSync:
4.3 Using EMS to work with mailboxes
Set-CasMailbox [email protected] –ActiveSyncEnabled $True
EMC has a catch-call category of “recipients” that it uses if you click on the
Recipient Configuration node. The Get-Recipient command is also available to you to retrieve information about every type of recipient and it
exposes a fuller set of properties about a recipient. The normal use is to call
Get-Recipient and specify the type of recipients that you want to view:
Get-Recipient –RecipientType UserMailbox
Get-Recipient –RecipientType DynamicDistributionGroup
Among the set of properties that are returned are some that you can’t
retrieve with other commands. For example, you can view the hashed values
for the safe sender list stored for a user mailbox that Exchange synchronizes
to Edge servers for use in anti-spam filtering by doing:
$Hash = Get-Recipient –id Redmond
You may not find many uses for the Get-Recipient command, but it’s
nice to know that it’s there!
Moving mailboxes
We already discussed how to use the Move Mailbox wizard that is available
through EMC to move mailboxes in Chapter 3. Moving mailboxes is a very
common administrative task and the Move-Mailbox command offers administrators some additional flexibility to script mailbox moves. In addition to
the options presented through the wizard, the Move-Mailbox command
allows you to specify these additional parameters:
GlobalCatalog: Specifies the name of the Global Catalog to be used
during migration.
DomainController: Specifies the name of the domain controller to be
used during migration.
MaxThreads: Define the number of threads to be used to move mailboxes simultaneously (default is 4).
ValidateOnly: Run the validation code to check that a mailbox can be
Chapter 4
4.3 Using EMS to work with mailboxes
ReportFile: Change the directory and/or file name for the XML
report generated for mailbox moves.
AllowMerge: The default is for Exchange to create a new mailbox in
the target database and import the content from the source mailbox
into the new mailbox. However, we can specify this parameter to
force Exchange to merge the content from the source mailbox into an
existing mailbox.
BadItemItem: Specify the number of “bad items” that Exchange can
ignore during the mailbox move. If Exchange encounters more than
the specified value, Exchange aborts the mailbox move. Exceeding a
reasonable limit (such as between 5 and 8 items) is a good indication
that there is a problem with the source mailbox that we should investigate before attempting to move it again.
ExcludeFolders: The default is for Exchange to move everything from
the source mailbox, but we can decide that a mailbox move should
ignore specific folders.
IncludeFolders: You can also specify that only certain folders should
be extracted and moved to the target mailbox.
StartDate and EndDate: We can also apply a filter so that Exchange
only moves items from the source mailbox that match a specified date
IgnoreRuleLimitErrors: Exchange 2003 mailboxes have a 32KB limit
for rules, which is increased to 64KB for Exchange 2007 mailboxes.
If you attempt to move a mailbox from an Exchange 2007 server to
an Exchange 2003 server, the rules limit may be exceeded, so this
parameter allows you to move a mailbox without its rules and so
avoid an error condition.
The basic syntax for the Move-Mailbox command is straightforward. To
move a mailbox immediately, we simply identify the mailbox to move and
the target database and server. In this example, we want to move the user
with the SMTP address “[email protected]” to a database on
server ExMbxSvr1:
Move-Mailbox –id ‘[email protected]’ –TargetDatabase
‘ExMbxSvr1\VIP Mailboxes Database’
In a migration scenario, it could be pretty boring to sit waiting for each
mailbox to move before typing the command to move the next—and it’s
hard to be sure that you process all the mailboxes that you need to. Pipelining comes in handy here as we can take all the mailboxes from a specific
4.3 Using EMS to work with mailboxes
database and move them to our new Exchange 2007 server in a single command. As always, make sure that you have enough disk space on the target
server to receive all the mailboxes that you are just about to transfer. Here
we use the Get-Mailbox command to read a list of mailboxes from a server
and then pipeline that data to the Move-Mailbox command so that it can
perform the move.
Get-Mailbox –server Exchange2003MbxSvr | Move-Mailbox
–TargetDatabase ‘ExMbxSvr1\Exchange 2007 Mailboxes’
Another variation on the move mailbox theme that comes into play in
disaster recovery situations is when you want to move information about
mailbox properties in the Active Directory to point a database on another
server (or another location on the same server). This is a nice change in
Exchange 2007 because it allows you to quickly transfer mailbox information
to another server in disaster recovery scenarios when, for instance, you have
experienced a database outage and you want to allow users to continue to
send and receive email. The advantage gained by having a single command to
replace the many varied scripts that administrators had to write to deal with
moving mailboxes after a database outage in previous versions of Exchange is
obvious. In an Exchange 2007 environment, you can use the Move-Mailbox
command to update the configuration data for the mailboxes to point to
another server so that Exchange can allow users access on that server. Later
on, when the failed database is available again, you can use the Recovery
Storage Group option to load the database and merge information back into
the user mailboxes (see Chapter 10 for more information about the Recovery
Storage Group). Alternatively, you can use the new database portability feature to move the production location for a failed database to a new server (see
Chapter 5 for more information on database portability). For example, to
move pointers to user mailboxes from the ExchMbxSvr1 server (which we
assume has experienced a failure) to the ExchMbxSvr2 server, you could use
this command:
Get-Mailbox -Server ‘ExchMbxSvr1’ | Move-Mailbox
-TargetDatabase ‘ExchMbxSvr2\Mailbox Database’
You might also have to move users based on their department or other
organizational information such as their location. In these cases, it’s a good
idea to understand how many mailboxes you might actually move and their
mailbox sizes before you actually perform the move. You can do this by specifying the –ValidateOnly parameter. For example:
Chapter 4
4.3 Using EMS to work with mailboxes
Get-User | Where {$_Department –eq ‘Technical Sales’} | Move-Mailbox
–TargetDatabase ‘ExchMbxSvr2\Mailbox Database 2’ –ValidateOnly
Get-Mailbox –Filter {Office –eq ‘Dublin’} | Move-Mailbox
–TargetDatabase ‘ExchMbxSvr1\Dublin Mailboxes’ –ValidateOnly
The –ValidateOnly parameter generates quite a lot of information, but
you can capture it by piping the output to a text file where you can review
the details including mailbox size, etc. An (edited) example of the information that you can expect to see is shown below:
: Users/ATG/Martha Lyons
: CN=Martha Lyons,DC=xyz,DC=com
: Martha Lyons
: Lyons
: /o=xyz/ou=Exchange Administrative Group
: [email protected]
: ExchMbxSvr1\First Storage Group\VIP Mailboxes
: ExchMbxSvr2.xyzcom
: ExchMbxSvr2\Technical Mailboxes
: 25411KB
: False
: IntraOrg
: Validation
: 12/18/2006 1:38:46 PM
: 12/18/2006 1:38:47 PM
: 0
: This mailbox can be moved to the target database.
When you’re happy that everything is ready to go, you can remove the
–ValidateOnly parameter and execute the remaining command to perform
the move.
When you move mailboxes between forests, you have to provide more
information to allow Exchange to authenticate itself correctly. In this
example, we want to move a mailbox from the “ABC.Com” forest to our
4.3 Using EMS to work with mailboxes
Exchange organization. The first thing we have to do is to collect the credentials that Exchange will use to identify itself to the source forest. We can
do this with the Get-Credential command, which prompts with a standard Windows dialog to collect a username and password that we can store
in the $AbcCredentials variable. We then call the Move-Mailbox command
and pass the name of a Global Catalog server in the other forest to help
find the mailbox that we want to retrieve and the credentials to identify
ourselves to the remote forest. The complete command looks like this:
$AbcCredentials = Get-Credential
Move-Mailbox –id ‘ABC\Smith’ –TargetDatabase ‘ExchMbxSvr2\VIP
Mailboxes’ –SourceForestGlobalCatalog‘’
–SourceForestCredentials $AbcCredentials –BadItemLimit 9
–NTAccountOu ‘Exchange Users’
Accessing another user’s mailbox
The Add-MailboxPermission command allows you to grant permissions to
users to access the mailboxes of other users. For example, to grant full access
to user Bond to user Smith’s mailbox:
Add-MailboxPermission –id Smith –AccessRights FullAccess –
User Bond
You obviously need permission to be able to log onto another user’s
mailbox, either by creating a new MAPI profile to allow Outlook to open the
mailbox or by selecting the Outlook Web Access option9 (Figure 4.10) to
serve the same purpose. You need to grant full access to a mailbox if you want
the user to be able to open the mailbox through Outlook Web Access.
Figure 4.10
Opening another
mailbox with
Outlook Web Access
You can also open another user’s mailbox with Outlook Web Access by passing the URL https://server-name/owa/[email protected] For example: https://ExchMbxSvr1/owa/[email protected]
Chapter 4
4.3 Using EMS to work with mailboxes
The FullAccess permission grants a user to work with the contents of
any folder in the target mailbox, including the ability to delete items, but it
does not include the right to send messages on behalf of a user. Any attempt
to send a message on behalf of the user will result in the error illustrated in
Figure 4.11. If you want to grant that permission, you either have to use
EMC to grant the Send on Behalf permission through the Delivery Options
settings for the mailbox or use the Set-Mailbox command:
Set-Mailbox –id Smith –GrantSendOnBehalfTo Bond
The reason why the two-step process is needed to allow someone to
send email on another user’s behalf is that the first operation ( Add-MailboxPermission) establishes a right for the specified user within the Store. The
second step (Set-Mailbox) establishes an Active Directory right. Another
way of accomplishing the second step is to use the Add-AdPermission command to add the extended Send As right to the user who you want to grant
control over the mailbox. Note that you may need to wait for the Store to
clear its cache of user information before these rights become fully effective.
Add-AdPermission –identity Smith –AccessRights ExtendedRight
–ExtendedRights Send-As –User Bond
You can use the Get-MailboxPermission command to check the permissions on a mailbox:
Get-MailboxPermission –id Smith
Or to find out what permissions exist on a mailbox for a specific user:
Get-MailboxPermission –id Smith –User Bond | Format-List
As you look at permissions, you will note that some are explicit and
some are inherited, or granted to an account for some reason, such as membership of the Exchange Organization Administrators group. The mailbox’s
owner receives their permissions through “SELF.” You may also see an
explicit deny, meaning that a user has been blocked from a permission. You
can use the Remove-MailboxPermission command to remove an explicit permission from a mailbox:
Remove-MailboxPermission –id Smith –User Bond –AccessRights
4.3 Using EMS to work with mailboxes
Figure 4.11
Unable to send
email because you
don’t have the right
Different commands and different properties
From our discussion, you might assume that Get-Mailbox is top of the list of
the Exchange administrator’s all-purpose commands. However, mailboxes
only exist as attributes of Active Directory user objects, so it should come as
no surprise that separate commands exist to retrieve and set the basic properties of user objects (including those that are not mail-enabled) and the properties that transform user objects into mailboxes.
Get-User and Set-User operate
on the broad set of user object
Get-Mailbox and Set-Mailbox operate on a set of properties that
relate to mailboxes.
Get-CasMailbox and Set-CasMailbox operate on a limited set of
properties that control access to the protocols managed by the Client
Access server.
Get-UmMailbox and Set-UmMailbox operate on the properties used by
the Unified Messaging server.
It can be hard to know when to use one of these commands instead of
another. For example, you might assume that the Department property is
under the control of Get-Mailbox because the EMC GUI displays this inforChapter 4
4.3 Using EMS to work with mailboxes
mation when you look at the properties of a mailbox. This isn’t the case
because Department is a property that non-mailbox-enabled Active Directory objects can possess, so you will not see it displayed if you type:
Get-Mailbox | Select Name, Department | Format-Table
EMS will execute the command, but you will see many blank values
listed in the “department” column even if you are quite certain that you have
populated this property. This is because Get-Mailbox does not include the
Department property in its property list and so treats it as a custom variable.
If you type:
Get-User | Select Name, Department | Format-Table
You now see a lot more information because Get-User is able to work
with any user objects (not just those that are mailbox-enabled) and it knows
about the department property. The rule is that you can use a Set- command
to manipulate any property that you see listed by its Get- counterpart. In
other words, if the Get-Mailbox command can retrieve a value for a property,
then you can use the Set-Mailbox command to set a new value for that
You will use the Get-Mailbox and Set-Mailbox commands to work with
mailboxes most of the time. However, you need to use the Get-CasMailbox
and Set-CasMailbox commands to access the properties that control the protocols managed by the Client Access server, such as a user’s ability to use POP3
or IMAP4 to get to their mailbox, the features available to mailboxes through
Outlook Web Access, and the settings that control how mobile devices access
mailboxes through ActiveSync. You cannot use the Get-Mailbox and SetMailbox commands to work with the properties that control protocol access.
In addition, you cannot use the Get-CasMailbox and Set-CasMailbox commands to work with the general set of Exchange properties for mailboxes. It
can be sometimes confusing to know exactly what you can do with what command, but you quickly get used to picking the right command.
Mail-enabled contacts are Active Directory objects that point to someone
outside our organization that we want to communicate with, such as users of
a different email system within the same company or users in an external
company that we do business with. Contacts have SMTP email addresses
and appear in the GAL. Providing that you know the email address of the
contact, creating it through EMS is trivial.
4.4 Working with distribution groups
New-MailContact –Alias SeanD –Name ‘Sean Doherty’ –Org Contacts
–ExternalMailAddress ‘[email protected]’
If you have an existing contact in Active Directory, you can enable it for
mail with the Enable-MailContact command:
Enable-MailContact –id ‘JFS’ –ExternalEmailAddress
‘[email protected]’
To disable a mail contact, use the Disable-MailContact command:
Disable-MailContact –id ‘JFS’
Disabling a mail contact leaves them in the Active Directory but
removes the properties that allow users to email the contact and allow
Exchange to list the contact in the GAL. If you want to remove a contact
completely, use the Remove-MailContact command:
Remove-MailContact –id ‘JFS’
Use the Set-MailContact command to perform subsequent operations
to update existing contacts. For example, to set a contact so that they always
receive MAPI rich text format messages:
Set-MailContact –id ‘Sean Doherty’ –UseMAPIRichTextFormat
Note that the New-Contact and Get-Contact commands are also available. These commands operate on the basic contact objects in the Active
Directory and do not manipulate the properties used by Exchange.
Working with distribution groups
Creating a new distribution group is a two-step process. First, we need to create the new group and ensure that it has a type of “distribution” rather than
“security” so that Exchange allocates an email address to the new group.
Chapter 4
4.4 Working with distribution groups
New-DistributionGroup –alias SeniorExec –name ‘Senior
–Type Distribution –org ‘’
–SamAccountName SeniorExec
If you already have a Windows group and want to mail-enable it for use
with Exchange, you use a command like this to populate the group with the
properties that make it available for use with Exchange:
Enable-DistributionGroup –id SeniorExec
Once we create the new group it is immediately functional in that you
can send email to it, but the message will go nowhere because the group does
not contain any members. We therefore have to add members to make the
group useful. The basic operation is to add members one at a time.
Add-DistributionGroupMember –id SeniorExec –Member Bond
The last parameter is the identification of the user that you want to add
to the list. The default is to use the account alias, which is unique to Windows, so whatever user whose alias is “Bond” will be added to the list. However, it is easy to make mistakes with aliases when you work with large user
communities, so it is usually a better idea to pass the UPN as this parameter.
Add-DistributionGroupMember –id SeniorExec –Member
‘[email protected]’
You can obviously cut and paste the required Add-DistributionGroupMember commands to add the members of the group or you can import the
membership in from a prepared CSV file and loop through the data to add
each member using the technique outlined for bulk mailbox loads on page
285. Another trick that you can use when you have multiple members to add
to the group is to create a table and pipe the table as input to the Add-DistributionGroupMember command. For example:
“John Smith”, “Amy Cater”, “Ollie Harding” |
Add-DistributionGroupMember –id ‘Senior Exec’
As you would expect, you use the Get-DistributionGroup command to
retrieve properties of a group and the Set-DistributionGroup command to
4.4 Working with distribution groups
set the properties of a group. For example, to set one of the custom attributes
on a group:
Set-DistributionGroup –id SeniorExec –CustomAttribute1 ‘VIP’
In the same way that we encountered differences between using the SetMailbox command to set Exchange-specific properties on an Active Directory user account and the Set-User command to set basic Active Directory
properties, the Set-Group command is sometimes required instead of SetDistributionGroup. For example, to change the user who is responsible for
managing a group, we need to use Set-Group because this action creates a
link to another Active Directory account who will manage the group. You
can assign a manager to any group even if it is not mail-enabled.
Set-Group –id SeniorExec –ManagedBy Bond
Taking this example a little further, let’s assume that you want to assign
the responsibility for managing all of the groups that belong to a department
(such as the Legal department) to a user. We can do this with a combination
of three commands:
Get-DistributionGroup | Where {$_.Name –Like ‘*Legal*’} | Set-Group
–ManagedBy Bond
Another situation is where we want to discover what groups a user
belongs to. This is easy to do in the Active Directory when you view the
properties of an account because the Active Directory maintains pointers to
the groups in the MemberOf property (and the MemberOf property is not
available through the Get-User command). It is not quite so straightforward
in PowerShell, but you can scan as follows:
$User = (Get-User –id Bond).Identity ; Get-Group –Filter {Members –eq $User}
Once you have groups populated, you can use them as the basis for
some management operations. For example, let’s assume that you want to
assign a managed folder policy (see Chapter 8 for information about how
managed folders and managed folder policies work) to users. You can do this
individually, but if all the target users are in a group, you can use the group as
the basis for the operation. In this example, we use the Get-DistributionGroup command to select the group that we want to use, the Get-DistribuChapter 4
4.4 Working with distribution groups
tionGroupMember command to expand the group into individual members,
and then the Set-Mailbox command to assign the managed folder policy.
Get-DistributionGroup –id ‘Senior Executives’ | GetDistributionGroupMember | Set-Mailbox –ManagedFolderMailboxPolicy
‘Senior Leader Records Management’
Deleting a member from a group is easy too.
Remove-DistributionGroupMember –id SeniorExec –Member Bond
One common request is to create a group that contains all the mailboxes
on a particular server. Administrators use groups like this to let users know
when a server is going to be unavailable for some reason, such as when a service pack or hot fix needs to be applied. We can create and populate such a
group with a couple of commands. First, we create the new group:
New-DistributionGroup -Type Distribution
-SamAccountName ‘ExchMbxSvr1Users’
-Name ‘Mailboxes on server ExchMbxSvr1’ -OrganizationalUnit ‘Groups’
To add all the mailboxes to the group, we call the Get-Mailbox command to fetch the names of all of the mailboxes on the server and pipe these
names to the Add-DistributionGroupMember command. This is exactly
what we did before to add a single user to a group only now we do it for a
whole set of mailboxes.
Get-Mailbox –Server ExchMbxSvr1 | Add-DistributionGroupMember
-Identity ExchMbxSvr1Users
You might not include mailboxes in the group that you don’t want to
ever receive messages, such as those used for room and equipment resources.
With this point in mind, we should amend our original command to add a
filter to exclude these mailboxes:
Get-Mailbox –Server ExchMbxSvr1 –Filter {RecipientTypeDetails –eq
‘UserMailbox’ –or RecipientTypeDetails –eq ‘LegacyMailbox’}| AddDistributionGroupMember -Identity ExchMbxSvr1Users
4.4 Working with distribution groups
It is great to have the new group created and ready to go, but you know
that the accuracy of the group’s membership immediately starts to degrade as
you add new mailboxes to the server or move mailboxes to another server.
Adding or moving mailboxes do not automatically update group membership in the same way as mailbox deletes do. Recall that group membership is
formed by a set of backward pointers to the underlying Active Directory
objects that form the group, so if you delete a mailbox and its user account,
the Active Directory automatically removes the pointer and so maintains
group membership. Therefore, if you want to keep the group up to date you
either have to:
Integrate group membership updates into the process that you
use to move mailboxes between servers (you need to remove the
mailbox from one group and add it to another) and the process
that you use to create new mailboxes.
Regenerate the groups on a regular basis. You can do this by creating a script to delete the group and recreate it using the commands
that we have just described. Note that this approach may cause
some problems with Active Directory replication if you delete a
group that someone else is updating elsewhere in the forest.
Use a dynamic group.
The last option is the most interesting because it requires least work on
the part of an administrator.
Working with dynamic distribution groups
When we discussed dynamic distribution groups in Chapter 3, we reviewed
the wizard that Microsoft provides in EMC to create and maintain dynamic
groups and the limitations that Microsoft has imposed on the queries that
you can create to generate dynamic groups through EMC to ensure that the
filters generated to create these groups work and deliver great performance.
All of the dynamic groups generated by EMC have precanned filters because
the filter created by EMC is constrained to using a limited number of properties. The net result is that you can only edit these groups through EMC
and if you want to create a dynamic distribution group that uses any other
property than those supported by EMC in the filter, then you have to create
and maintain the group through PowerShell.
As we think about creating groups through the shell, it is good to
remind ourselves of the three principles that guide the creation of any
dynamic distribution group:
Chapter 4
4.4 Working with distribution groups
What organizational scope will the recipients come from? We
will state this in the RecipientContainer property of the query.
What types of recipients will be included in the query? We will
state this in the IncludedRecipients property.
What other properties are involved in the filter? We pass these
values in the RecipientFilter property. Because we are creating a
custom filter through the shell, essentially we have made the decision that we need to use more than the set of properties supported by the EMC wizard.
First, we start by reviewing how to create a new dynamic distribution
group using the New-DynamicDistributionGroup command. Here is the
one-line command to create a group that contains a recipient filter that
includes anyone who works for HP.
New-DynamicDistributionGroup -Alias ‘CUserDL’ -Name ‘Company Users’
-RecipientFilter {Company -eq ‘HP’ –and RecipientTypeDetails –eq
‘MailboxUsers’} -RecipientContainer ‘HP.COM’
-OrganizationalUnit Groups
This isn’t a very exciting query because it could easily have been generated by the EMC wizard. We haven’t done anything very intelligent in terms
of creating a complex query. Note that the RecipientContainer parameter
specifies the objects that will form part of the dynamic group based on their
position in Active Directory. In this case, we have specified the root of the
domain, meaning that any object under this point will be included. After creating the new dynamic distribution group, you can determine whether the
query used is precanned or custom by using the Get-DynamicDistributionGroup command:
Get-DynamicDistributionGroup –id ‘Company Users’ | Format-List
DisplayName, RecipientFilterType, RecipientFilter,
: Company Users
: ((Company -eq 'HP' -and RecipientTypeDetails -eq
'UserMailbox') -and -not(Name -like 'SystemMailbox{*') -and -not(Name
-like 'CAS_{*'))
LdapRecipientFilter :
4.4 Working with distribution groups
Even though we created this dynamic distribution group through the
shell, it shows up as a precanned filter, just like all of the filters generated by
the EMC wizard. The reason is that Exchange optimizes the filters as much
as possible and has recognized this filter as precanned because it does not use
any properties outside the set of conditional properties that precanned filters
support. As you will recall from the discussion about creating dynamic distribution groups through EMC in Chapter 3, the conditional properties that
are available for filtering when you use EMC are:
The fifteen custom attributes (ConditionalCustomAttribute1 to
The reason why this set of special conditional properties exists is very
simple. A dynamic distribution group is a recipient object itself and you can
set properties for recipient objects, including the properties that you might
want to use when you create a filter. The potential for confusion is obvious,
so the Exchange developers added the special filtering properties to make it
obvious when you want to set a property of the group and when you want to
set a property used in a filter.
To tweak the filters programmatically to take them past the point that
you can achieve with EMC, you can create a dynamic distribution group
and create a filter using whatever properties you need and then pass the filter
in the RecipientFilter property. Note that if you create a custom filter and
want to pass it in RecipientFilter, you cannot combine it with a precanned
filter like –IncludedRecipients. The reason is that Exchange will compute
the filter for you and then write it into the RecipientFilter property, so you
can’t pass one set of values for Exchange to compute and then pass another
value (that might be completely different) in RecipientFilter. For example,
you can’t do this:
New-DynamicDistributionGroup -Alias ‘CUserDL’ -Name ‘Company
-RecipientFilter {Company -eq ‘HP’}
-RecipientContainer ‘HP.COM’
-IncludedRecipients ‘MailboxUsers’
-OrganizationalUnit Groups
-ConditionalCompany ‘HP’
Chapter 4
4.4 Working with distribution groups
The rules of thumb for building filters for dynamic distribution group
are therefore:
The easiest way to build a filter is to use the special conditional properties, if only because you break out the filter for each property into a
separate parameter. For example:
Set-DynamicDistributionGroup –id ‘UK Employees’
–ConditionalStateOrProvince ‘England’, ‘Wales’, ‘Scotland’
–ConditionalCompany ‘HP’, ‘HP UK’, ‘Hewlett-Packard’,
‘Hewlett Packard’
–ConditionalCustomAttribute1 ‘Employee’
–IncludedRecipients ‘MailboxUsers’
In this example, the query that Exchange uses to build the group
is: Any account from England, Wales, or Scotland who belongs to the
company called HP, HP UK, Hewlett-Packard, or Hewlett Packard,
and who has the value “Employee” in custom attribute 1. This type of
query that has to check multiple values for a property is evidence of a
certain lack of standards in the population of mailbox data. Remember the caveat that while dynamic distribution groups are great ways
to address changing populations, they can be slow for the transport
service to resolve against the Active Directory if more than a couple
of hundred addresses result or you add many conditions. This is one
case where simpler is better.
As shown in the last example, you can restrict a dynamic distribution
group to send messages to a certain set of recipient types by specifying the –IncludedRecipients parameter. You can then select from one
or more of the following values: AllRecipients, MailboxUsers,
Resources, MailContacts, MailGroups, MailUsers, or even None
(which would be a kind of non-op).
If you use –RecipientFilter to state a filter to use with a dynamic distribution group, then you can include a broader range of properties
within the filter. You can combine properties to create a filter that is
as complex as you like. For example:
-RecipientFilter {Company –eq ‘HP’ –and Office –eq ‘Dublin’ –and
Department ne ‘Sales’ –and ServerName –eq ‘ExchMbxSvr1’ –and
CustomAttribute15 –eq ‘Employee’}
4.4 Working with distribution groups
You cannot combine –RecipientFilter and the special filter properties
in a dynamic distribution group.
Moving on to work with something that is of immediate interest to
most administrators, the command to create a new dynamic distribution
group with a recipient filter that includes all the mailboxes on an Exchange
2007 server is:
New-DynamicDistributionGroup -Name ‘Mailboxes on ExchMbxSvr1’
-Alias MbxSvr1 -RecipientContainer ‘xyz’ -OrganizationalUnit Groups
-RecipientFilter {ServerName -eq 'ExchMbxSvr1'}
Figure 4.12
Filter for a
You cannot use EMC to edit a custom recipient filter created for a
dynamic distribution group. As you can see in the group properties shown in
Figure 4.12, you can see the custom filter, but you cannot do anything else.
The only way that you can edit the group’s custom recipient filter is to create a
new recipient filter and update the group with the Set-DynamicDistributionGroup command.
Chapter 4
4.4 Working with distribution groups
There is no equivalent to the Get-DistributionGroupMember command
for dynamic distribution groups, so if you want to discover the membership
of a dynamic group, you have to invoke a command such as Get-Mailbox
and pass it to the appropriate filter.
Advanced group properties
Apart from membership, Exchange allows you to set many other properties
for groups that you can do when you create the group or afterwards. For
example, the “Advanced” property tab of the GUI offers options to set these
Hide group from Exchange distribution list: Users can always send to
the group if access to the group is not restricted to a certain list of correspondents and they know its SMTP address. This option removes the
group from the GAL and any other Exchange address lists.
Send Out of Office notifications to originator: You can suppress
OOF10 messages for messages sent to the group by setting this property to be false. The default is to allow notifications to flow back to
anyone who sends to the group.
Send delivery reports to group manager: Before you can set this property, you have to specify what account manages the group. The
default is that the group will not have a manager.
Send delivery reports to message originator: This is the default behavior, meaning that Exchange will return non-delivery and other
reports back to the message originator.
Do not send delivery reports: Set this property to suppress any
reports for messages sent to the group.
Before we start to work with properties of the group, it is a good idea to
check out the current values of the properties and the names of the properties
that we want to work. The easiest way to do this is with a command like this:
Get-DistributionGroup SeniorExec | Format-List
Along with the set of regular properties, you will notice that distribution groups support the set of 15 customized attributes that are available to
OOF means “Out of Facility.” Logically, we should be using OOO for out of office, but OOF has been around since before
Exchange appeared (the term was used on Microsoft’s previous Xenix email system) and will not change now.
4.4 Working with distribution groups
you to store data of your choice. You should also see that the value of both
the RecipientType and RecipientTypeDetails properties is “MailUniversalDistributionGroup” and that the value of GroupType is “Universal.” If you
use the group to assign permissions over objects, it will be a mail-enabled
security universal group, so the value of the RecipientType and RecipientTypeDetails property is “MailSecurityUniversalGroup.” Here is how to set
the properties to hide the group from the GAL and to suppress reports
going back to anyone.
Set-DistributionGroup SeniorExec
We have already seen that some properties need to have data appended
to them rather than simply overwriting the data. You can restrict access to
distribution groups so that only specific users or distribution groups are able
to send messages to them. It is quite common for companies to restrict the
ability of users to send messages to senior executives. You can set restrictions
on who has the ability to send messages to a group by selecting the group and
then accessing the Message Delivery Restrictions option of the Mail Flow
tab. Here’s what we would do to add a new user to the list of accepted senders
for our distribution group through EMS:
Set-DistributionGroup SeniorExec -AcceptMessagesOnlyFrom ((GetDistributionGroup SeniorExec).AcceptMessagesOnlyFrom + ‘James Bond’)
If you want to restrict access to a distribution group, it is easier to use
another distribution group as the control point for the restriction and import
the members of that group into the restriction list. To do this, we use much
the same code but specify the AcceptMessagesOnlyFromDLMembers
parameter and pass the name of the distribution group that we want to use
instead of a specific user.
Set-DistributionGroup SeniorExec -AcceptMessagesOnlyFromDLMembers ((GetDistributionGroup SeniorExec).AcceptMessagesOnlyFromDLMembers + ‘Legal
There are two important points to note here. First, importing members
from another group to add them to the restriction list for a group is a onetime operation. Exchange expands the membership of the group that you
Chapter 4
4.4 Working with distribution groups
specify and adds each of the members of the group to the set of users who
can send messages to the target group. If you make a subsequent change to
the membership of the group that you import, Exchange does not automatically add the new member to the restriction list. For example, if I add the
user “Jack Smith” to the “Legal Executives” group after I use the “Legal Executives” group as the basis for restricting email address to the “Senior Executives” group, Jack Smith is unable to send email to “Senior Executives” until I
add their name separately.
The second point is that Set-DistributionGroupMember will fail if a
user that you import from another group already exists in the restriction
list. You could avoid this problem by setting the AcceptMessagesOnlyFromDLMembers property to $Null first before you specify what the value
should be.
Because PowerShell’s error handling is a tad basic, it’s a good idea to
check that everything went smoothly and that the right users are allowed to
send messages to the group after you make any changes. I like to do something like this:
Get-DistributionGroup SeniorExec | Select Name, AcceptMessagesOnlyFrom |
: Legal Executives
AcceptMessagesOnlyFrom : {Ailbhe Redmond, James Bond, Tony Redmond, Daragh
Note that you can also explicitly prohibit specific individuals from sending messages to a group by setting the RejectMessagesFrom property. For
Set-DistributionGroup SeniorExec -RejectMessagesFrom ((GetDistributionGroup SeniorExec).RejectMessagesFrom + ‘Mary
Working with group membership provides a nice opportunity to explore
pipelining further. For example, let’s assume that we have made the decision
to give all of our senior executives a very large mailbox quota. We have a
group called “Senior Executives” that all the necessary mailboxes are members of, so the logic we need is:
Open the “Senior Executives” group.
Read all the members of the “Senior Executives” group.
4.5 Delegation through the shell
Update each mailbox with the new quota.
This is a good example of an administrative activity that can be a real
pain to execute through the GUI if we have to process a large group. With
EMS we can say:
Get-DistributionGroup ‘Senior Executives’ |
Get-DistributionGroupMember | Set-Mailbox –IssueWarningQuota
5000MB –ProhibitSendQuota 5050MB
–ProhibitSendReceiveQuota 5075MB
Note the –UseDatabaseQuotaDefaults parameter—setting this parameter to “False” tells Exchange to use the specific quotas we have now set on the
mailbox. If you did not set this value, Exchange would continue to apply
whatever default quotas the mailboxes inherited from the database where
they reside.
Delegation through the shell
Chapter 3 discusses the Exchange delegation model that allows different
users to have administrative control over the organization, servers, and recipients and how to use EMC to delegate access to users. You can also delegate
access through EMS. For example, to get a complete list of all users who possess any sort of administrative permission for an Exchange server within the
organization, you issue the Get-ExchangeAdministrator command. I find it
most convenient to pipe the output to a text file as follows:
Get-ExchangeAdministrator | Format-Table > c:\Temp\
As you scan the entries, you find users who have control over the complete organization:
Identity : Administrators/Tony Redmond
: OrgAdmin
Users who have control over recipients:
Identity : Users/Ailbhe Redmond
Chapter 4
4.5 Delegation through the shell
: RecipientAdmin
And others who have administrative control over servers. In this case,
the users have view-only access to the organization (so that they can navigate
through the organization with EMC) and specific control over particular
: Users/Management/Mark Crosbie
ViewOnlyAdmin Users/Management/Mark Crosbie
Naturally, you can also use Get-ExchangeAdministrator to return
information about a specific user:
Get-ExchangeAdministrator –id ‘Tony Redmond’
The Add-ExchangeAdministrator command delegates access to a new
user. For example, to add Jane Smith to the list of administrators for the
ExchMbxSvr1 server, you type the command shown below. After you add the
account to the list of administrators, make sure that you add the account to
the built-in local Administrators group for the server.
Add-ExchangeAdministrator –id ‘Jane Smith’ –Role ServerAdmin
–Scope ExchMbxSvr1
The Remove-ExchangeAdministrator command removes a user’s
administrative access.
Remove-ExchangeAdministrator –id ‘Jane Smith’ –Role
ServerAdmin –Scope ExchMbxSvr1
See Chapter 3 for more information about how to delegate administrative access to an Exchange 2007 server.
4.6 Creating efficient filters
Creating efficient filters
PowerShell supports server- and client-side filters. Any time that you see the
Where command used like all the examples we have discussed so far, you
know that you are executing a client-side filter. Client-side filters are the standard for PowerShell and all PowerShell commands that can filter data support client-side filters. Exchange has different characteristics to standard
Windows management because of the potential number of objects that you
can ask a query to process. It is one thing to ask for all of the processes that
are active on a server and quite another to ask for all of the mailboxes on a
server. This is the reason why Exchange supports server-side filters.
The big difference between the two types of filters is that client-side filters bring all of the requested data set from the server to the client before
applying the filter to generate the final data set, which the command that you
call then processes. The client and server do a lot of work to isolate the necessary data. Server-side filters ask the Exchange server to filter the data set
before passing the resulting filtered data to the client for execution and therefore reduce the amount of data that traverses the network for the client to
process. Only a limited number of commands that deal with users, other
mail-enabled objects, and queues support the –Filter parameter that invokes
a server-side filter and not all of the properties returned by these commands
support filtering. Without filters, these commands can generate a huge
amount of data in any medium to large organization and be quite slow to
execute, so server-side filtering helps by focusing on a specific set of objects
and doing the work on the client. It is worth mentioning that Exchange generates the most efficient queries that it can for OPATH filters. All of the precanned filters generated for dynamic distribution groups, Address Lists, and
Email Address Policies use server-side filters.
For example of server and client-side filtering in action, two methods are
available to find all the mailboxes with “James” in their name. We can use
either of these commands:
Get-Mailbox –Filter {Name –like ‘*James*’} –ResultSize 5000
Get-Mailbox –ResultSize 5000 | Where {$_.Name –like
On the surface, these two pieces of code seem similar, but they are very
different in reality. First, they use different filters. Second, the effect of the filters can generate very different results. If we omitted the –ResultSize parameter, you generate the same query: find all the mailboxes with a name that
contains “James.” The server-side filter will execute faster than the client-side
Chapter 4
4.6 Creating efficient filters
filter, which proves the point that server-side filtering is faster. When we add
the –ResultSize to limit the size of the collection that we process (which you
would probably want to do if you work with a large organization), then the
queries that Exchange generates to locate data are very different.
The first (server-side) filter returns the first 5,000 mailboxes that it
finds that include James in the mailbox name.
The second (client-side) filter fetches data for the first 5,000 mailboxes to the client and then applies the filter to find the mailboxes
that include James in the name. However, the filter only applies to
the set that the client fetched and may not find all of the mailboxes
that we actually want to discover.
Even though we ask the server-side filter to do more work (because it
will have to process significantly more data to find the mailboxes that we
have asked for), it still executes faster! For example, when I executed the two
commands shown above against a very large Exchange organization (170,000
mailboxes), the server-side filter completed in 43 seconds and the client-side
filter in 81 seconds.
Sometimes people make the mistake of assuming that the client-side filter is faster because server-side filters provide the data in one motion after the
server processes all the data. You therefore wait for a while without seeing
anything and then see all the filtered records at one time. By comparison, the
client-side filter fetches and filters data continuously and so you see some
output on the screen as the command finds each matching record. However,
the important indicator of performance is how long each type of filter takes
to complete and server-side filters are always faster.
The commands that you are most likely to use with server-side filters
–Retrieve basic Active Directory properties for any user
account, including mail-enabled accounts.
Get-Mailbox –Retrieve
enabled contacts.
Exchange-specific properties for mailboxes.
–Retrieve Exchange-specific properties for mail-
mail-enabled groups.
–Retrieve Exchange-specific properties for
4.6 Creating efficient filters
Each of the commands that you can use to work with user accounts,
groups, and mailboxes supports a different set of filterable properties. To discover what properties are available for filtering, you can use PowerShell to
query the properties of a returned object. For example:
Get-Mailbox -id Redmond | Get-Member | Where-Object
{$_.MemberType –eq ‘Property’} | Sort-Object Name | FormatTable Name
This set of commands calls a command to return some information
about an object. It then pipes the information returned by the first command
to the Get-Member command, which extracts information about the properties. We sort the properties by name and output them in table format. The
output looks like this:
This method works for the Get-Mailbox, Get-CasMailbox, Get-User,
Get-Recipient, Get-DistributionGroup, and Get-DynamicDistributionChapter 4
4.7 Bulk updates
Group commands. You can use any of the values reported in a –Filter statement. For instance, the all that we just made to Get-Mailbox reports that the
custom attributes are available, so to find all mailboxes that have a value in
the CustomAttribute10 property, we can generate a command like this:
Get-Mailbox –Filter {CustomAttribute10 –ne $Null}
If you look at the filterable properties reported by the Get-Dynamiccommand, you can see that the ManagedBy property is
available for this dynamic distribution group whereas it is not for mailboxes.
Hence, we can execute a filter like this:
Get-DynamicDistributionGroup –Filter {ManagedBy –ne $Null}
When you create a filter, it is best to be specific as possible. You can state
several conditions within a filter. Another example of a server-side filter that
returns all the mailboxes in the Dublin office where the user name contains
“Tony” is shown below. The Get-User command also works with this filter,
but Get-Mailbox executes a tad faster because the server does not have to
process accounts that are not mail-enabled.
Get-Mailbox –Filter {Office –eq ‘Dublin’ –and Name –like
After you have mastered server-side filtering, you will find that you use it
all the time to work with sets of users. For example, let’s assume that you
want to give a new mailbox quota to members of a certain department but
no one else.
Get-User –Filter {Department –Eq ‘Advanced Technology’} |
Set-Mailbox U
– seDatabaseQuotaDefaults:$False –
IssueWarningQuota 5000MB P
– rohibitSendQuota 5050MB –
ProhibitSendReceiveQuota 5075MB
Bulk updates
Anyone faced with the task of bulk updates (either to create a lot of new
mailboxes or other objects, or to modify many existing objects) in Exchange
2000/2003 had quite a lot of work ahead of them because Exchange offered
no good way to perform the work. You could create CSV or other load files
and use utilities such as CSVDE or LDIFDE to process data in the files
4.7 Bulk updates
against the Active Directory, or you could write your own code to use
CDOEXM or ADSI to update the Active Directory. Either approach
involved a lot of detailed work where it was quite easy to make a mistake.
The alternative (to use the ESM or AD Users and Computers consoles to
make the necessary changes through a GUI) was just boring and also an invitation to make a mistake. The root cause of Exchange’s problems with bulk
changes was the lack of a programmable way to automate common management operations, something that changes with the arrival of EMS.
You can combine the Get-User and Set-Mailbox commands effectively
to solve many problems. Here is an example where you need to update the
send quota property on every mailbox for a set of users whose business group
has decided to fund additional storage. You can identify these users by their
department, which always starts with “Advanced Tech” but sometimes varies
into spellings such as “Advanced Technology” and “Advanced Technology
Group.” Conceptually, the problem is easy to solve.
Look for all users who have a department name beginning with
“Advanced Tech.”
Update the send quota property for each user.
With Exchange 2000/2003, we can use the “Find” option in AD Users
and Computers to build a suitable filter to establish the set of users. The
problem here is that you then have to open each user located by AD Users
and Computers to update their quota through the GUI, something that
could become very boring after ten or so accounts. You could also export a
CSV-formatted list of users to a text file, then manipulate the file to find the
desired users, and then process that list through CSVDE to make the
changes, but you have to search for all matching users across the complete
directory first. There is a lot of work to do.
The process is easier in EMS. First, you use the Get-User command
with a suitable filter to establish the collection of mailboxes that you want to
change. The following command returns all users who have a department
name that begins with “Advanced Tech” and then updates the ProhibitSendQuota property to the desired amount (let’s say 1,000MB). Because we have
a collection of user objects established, we can chain on the necessary SetMailbox command to perform the update. Note that some of these users may
not be mailbox-enabled, but error handling is another day’s work.
Get-User | where {$_.Department –like ‘*Advanced Tech*’} |
Set-Mailbox –ProhibitSendQuota 1000MB –
UseDatabaseQuotaDefaults $False
Chapter 4
4.7 Bulk updates
Mergers, acquisitions, and internal reorganizations pose all sorts of
problems for email administrators. EMS won’t solve the big problems, but it
can automate a lot of the mundane tasks that occur. For example, department names tend to change during these events. EMS makes it easy to find
all users who belong to a specific department and update their properties to
reflect the new organizational naming conventions. If only executing organizational change was as easy as this one-line command, which transforms
everyone who works for the “Old Designs” department to now belong to the
“Cutting Edge Design” department:
Get-User | Where {$_.Department –eq 'Old Designs'} | Set-User
–Department ‘Cutting Edge Design’
Note the use of $_.Department. This indicates a value fetched from the
current pipeline object. In this case, it is the department property of the current user object fetched by Get-User.
To verify that we have updated all the users that we wanted to (and
maybe provide a report to HR or management), we can use a command like
Get-User | Where {$_.Department –eq ‘Cutting Edge Design’} |
Select Name, Department | Sort Name | Format-Table > c:\temp\
A variation on this theme is to output the data to a CSV file to make the
data easier to work with Excel or Access or another tool that can read in CSV
Get-User | Where {$_.Department –eq ‘Cutting Edge Design’} |
Select Name, Department | Sort Name | Export-CSV c:\temp\
Things are even easier if you just need to change everyone’s company
name after your company is acquired.
Get-User | Set-User –Company ‘New Company’
You can even do things like only alter the users whose mailbox belongs
to a particular database:
Get-Mailbox –database ‘VIP Mailboxes’ | Set-User –company
‘Big Bucks’
4.7 Bulk updates
–Department ‘Executives’
All of the examples discussed so far depend on you being able to identify
some property that you can use as the basis for a filter. But what about the
situation where you do not have a common property value to check for? In
this case, you can build a simple list of mailbox names (or any other format
that the –identity parameter will accept such as a UPN) and use the GetContent command to read the names one by one and pipe these values to
whatever other command you need to use. For example, here is how we can
use that trick to enable ActiveSync access for a set of users:
Get-Content c:\temp\Users.txt | Set-CasMailbox –
ActiveSyncEnabled $True
Another place where EMS excels is where you want to apply a common
setting across all servers in your organization. For example, let’s assume that
you want to apply a new retention limit for items of sixty days (perhaps mandated by the legal department) to all servers:
Get-MailboxDatabase | Set-MailboxDatabase –ItemRetention
These simple examples demonstrate the value of having a scripting language that supports automation of common management tasks.
Creating sets of mailboxes
We already discussed how to use PowerShell to create a new mailbox, but
what happens when you have a whole set of mailboxes to create at one time,
perhaps to populate a new mailbox server? In this case, you can use a combination of the Import-CSV and New-Mailbox commands to do the job. The
Import-CSV command reads data in from a CSV-formatted file and NewMailbox creates the new mailbox. The ConvertTo-SecureString command
reads the password provided in the file and converts it to a secure string so
that it you can pass it as an acceptable value for a password. A sample input
file in CSV format looks like this:
Id, FullName, LastName, FirstName, Password
MAnderson, Mike Anderson, Anderson, Mike, Testing123
FAnderson, Fergal Anderson, Anderson, Fergal, Testing234
Chapter 4
4.7 Bulk updates
Sometimes it is good to check that the list is populated with the correct
data and can be read as we’d expect. One way to do this is to call the ImportCSV command to process the data to populate a variable and then see what
that variable contains:
$MyData = Import-CSV c:\temp\users.csv
$MyData | Format-Table
Here’s the code to read the CSV file and create the mailboxes (and of
course, the underlying Active Directory accounts):
# Set up variables
$data = Import-CSV c:\temp\Users.csv
# Database where we are going to put the user mailboxes
$mdb = ‘ExchMbxSvr1\Database1'
$ou = ‘ Users/XYZ’
# Import the data and create the new mailboxes
ForEach-Object ($i in $data)
$ss = ConvertTo-SecureString $i.password –AsPlaintext -Force
$upn = $ + ""
New-Mailbox -Password $ss -Database $mdb
-UserPrincipalName $upn -name $i.fullname OrganizationalUnit $ou
-SamAccountName $ -FirstName $i.firstName
-LastName $i.lastname -Displayname $i.fullname
Note that New-Mailbox populates a limited number of attributes in the
new Active Directory object and you may want to call Set-Mailbox afterwards to populate attributes such as mailbox quotas. This script is a little
complex but you only need to write it once (or reuse it from somewhere else).
While the necessary code is much simpler, you can take the same approach to
create a bulk load of mail-enabled contacts. Assume that you have a CSV
load file in the following format:
Name, First, Last, Alias, SMTPAddress
Bob Hope, Bob, Hope, BH, [email protected]
4.8 Reporting mailbox data
Terry Smith, Terry, Smith, TSM, [email protected]
This code is enough to read the load file and create mail-enabled
Import-CSV c:\temp\Newcontacts.csv | ForEach-Object {NewMailcontact
-Alias $_.alias -Name $_.Name -ExternalEmailAddress
-First $_.first -Last $_.last -Org Contacts}
As you can imagine, you can repurpose much the same code for other
Reporting mailbox data
Perhaps one of the more dramatic examples of how the focus for management has changed in Exchange 2007 from the GUI to the shell is the loss of
the ability to view mailbox statistics and logons through the console, which
has been a feature of Exchange management since 1996. The Recipient Configuration node of EMC lists mailboxes, contacts, and groups, but it does not
tell you anything about them in terms of who’s currently connected to a
server, the number of items in their mailbox, and how much quota they use.
To get this information, you now have to use the Get-MailboxStatistics
and Get-LogonStatistics commands to extract and report data rather than
browsing information through the GUI.
The logic behind why Microsoft eliminated the display of user statistics
from EMC has to do with how EMC retrieves information from the server
and the overall change in design philosophy for the management console. To
fetch information about a mailbox, the Exchange 2003 console makes RPC
calls to the database that hosts the mailbox to retrieve information like the
mailboxes in the database plus properties such as the item count, quota used,
deleted item count, and the size of deleted items for each mailbox. This
approach is acceptable if you restrict your view to a single database on a
server, but EMC treats recipients on an organization-wide basis and its
default mode is to display the first 1,000 mailboxes in the organization when
you click on the Mailbox node under Recipient Configuration. EMC makes
an LDAP request to fetch data about mailboxes from the Active Directory
and does not have to access the Store at all until you decide that you want to
view the properties of a mailbox, in which case EMC displays the properties
that it retrieves from Active Directory alongside the mailbox information.
You will notice that a small pause often occurs between requesting the propChapter 4
4.8 Reporting mailbox data
erties and their display, caused by the need to access the mailbox and fetch
the total item count, quota used, and time of last logon. You see the same
pause when you execute a Get-MailboxStatistics command against the
mailbox because this is effectively what EMC has to do to retrieve information about the mailbox from the Store.
Because the design philosophy behind EMC is to treat mailboxes on an
organizational basis, there is no guarantee that the mailboxes that you view
when you open EMC will come from any particular server or database on a
server. EMC normally orders mailboxes alphabetically by their Display
Name, so you fetch mailboxes starting with “A” and continue until you hit
the limit for the number of items that EMC is willing to process. If you want
EMC to fetch mailboxes from your local server, you have to apply a filter.
Although you can argue a good case that EMC should always apply a filter, it
is not currently the default behavior for EMC, so the performance implications of having to make up to 1,000 RPC calls to different databases scattered around the organization to fetch storage details for the mailboxes
before EMC can display anything are unconscionable. Avoiding horrible performance is the reason why Microsoft dropped the ability to browse mailbox
and logon statistics through the console. The official line is that they mitigated the issue somewhat by displaying information about the quota used
and items in a mailbox when you look at its properties and that you can
always use PowerShell to retrieve the information. However, Microsoft has
received a huge amount of negative feedback on this issue, especially from
administrators who do not look after Exchange full-time and those do not
care to learn shell programming. It would not come as a surprise if Microsoft
upgrades EMC so that it can report statistics in a future release.
If you want to retrieve the kind of mailbox statistics that you are able to
view with Exchange 2003 ESM, you use the Get-MailboxStatistics command. Like many other shell commands, the first impression of Get-MailboxStatistics is that it is somewhat verbose in terms of the volume of data
that it generates on anything but a very small server, so if you want to look at
data interactively, you need to provide a filter by database or server to reduce
the amount of information that you have to deal with. In most cases, it is
more convenient to pipe the output to a file and use an editor to go through
the contents. For example, here is what the output from Get-MailboxStatistics looks like when you pipe it through the Format-list command to
show you all the properties that Get-MailboxStatistics can report. In this
example, we extract information for just one mailbox:
Get-MailboxStatistics –id Redmond | Format-list > c:\temp\
: 524
4.8 Reporting mailbox data
Redmond, Tony
1/12/2007 11:59:28 AM
1/12/2007 11:59:22 AM
ExchMbxSvr1\LCR Storage Group\Users
LCR Storage Group
Not all of these properties are necessarily interesting, so we can select
specific properties and use the Format-table command to generate the output as a table. For example, when we extract information for all of the mailboxes in a specific database, we can format the information to make it appear
more attractive:
Get-MailboxStatistics –Database “Mailbox Store 1” | Select
DisplayName, ItemCount, TotalItemSize | Sort ItemCount |
Format-Table @{Expression="DisplayName"; width=30; label
=”Name”}, @{Expression=”ItemCount”; width=10; label =
@{Expression={ [math]::round(
[double](([string]$_.TotalItemSize).split(“B”)[0]) /1KB , 2)}
; width=15; label=“Size(KB)”} > c:\temp\Mailbox-Report.tmp
In this case, we want to output just three pieces of information per mailbox and to sort by item count. We format the output fields with headers that
describe the content better than the default names. We also take the default
output for the TotalItemSize field (normally returned in bytes) and format it
to display in kilobytes. Using the code shown above, we get a report like this:
Chapter 4
4.8 Reporting mailbox data
---Heublein, Alex
Smith, Jane (Techology Co-O...
Bond, James
Walker, Karen
Morrissey, Daragh
Smith, Steve Jnr.
Nemeth, Alan
Crosbie, Mark
Lyons, Martha
Grillenmeier, Guido
-------ExchMbxSvr1\LCR Storage Gro...
ExchMbxSvr1\LCR Storage Gro...
ExchMbxSvr1\First Storage G...
ExchMbxSvr1\First Storage G...
ExchMbxSvr1\First Storage G...
ExchMbxSvr1\First Storage G...
ExchMbxSvr1\LCR Storage Gro...
ExchMbxSvr1\First Storage G...
ExchMbxSvr1\LCR Storage Gro...
ExchMbxSvr1\LCR Storage Gro...
While you review these code examples, the thought may occur to you
that the reports that we can generate using PowerShell commands are not as
pretty as the GUI output from the Exchange 2003 ESM console. This is
true. It is also true that you can extract data from ESM and export it to Excel
to manipulate it further to create colorful reports and graphs. You can do this
work with a couple of mouse clicks much more easily than grappling with
the convoluted formatting requirements of the Format-Table command,
including its ability to generate HTML output. Such a conclusion is natural,
but it ignores the fact that you can write a PowerShell script once to create a
report and then reuse it many times afterwards. You do not see the true value
of the shell when you execute once-off commands; it only appears when you
automate operations through scripts.
Once you have written a script to perform the desired operation, it is
easy to reuse the script time after time. In this case, automation is best when
you can process data for many different databases with one command or process all of the Exchange servers in the organization. Before explaining what is
going on, the complete code for our script is:
Get-ExchangeServer | Where-Object {$_.ServerRole –like
‘*Mailbox*’} | Get-MailboxStatistics | Select DisplayName,
ItemCount, TotalItemSize, Database | Sort TotalItemSize
-Descending | Format-Table @{Expression= ‘DisplayName’;
width=30; label = ‘Name’}, @{Expression= ‘ItemCount’;
width=10; label = ‘Items’}, {Expression={
0]) /1KB , 2)} ; width=15; label = ‘Size(KB)’}, @{expression =
‘database’; width=30; label = ‘Database’} > C:\TEMP\
To explain what the script does:
4.8 Reporting mailbox data
We call the Get-ExchangeServer command to retrieve a list of all
the Exchange servers in the organization.
We impose a filter with the Where-Object command to filter out
any servers that do not support the mailbox role. This command
also removes any legacy Exchange servers because these servers
return “None” in the ServerRole property. The filter is required
because we do not want to access servers that do not support the
Get-MailboxStatistics command (such as legacy servers and
Edge servers).
Get-MailboxStatistics fetches mailbox data from all of the
Exchange servers that pass the filter. We select the display name
for each mailbox, the count of items in the mailbox, the total size
of the items, and the name of the database that hosts the mailbox.
We sort the data by mailbox size.
We format the data into a table. The fields are given headers and
the total size field is formatted to turn bytes into kilobytes.
We pipe the report to a text file.
Proving that the shell is very flexible as well as powerful, EMS allows
you to pipe the mailbox statistics that you extract to a file in CSV format
with the Export-CSV command. In this case, you would amend the script to
do something like this:
Get-MailboxStatistics –Database ‘Mailbox Store 1’ | Select
DisplayName, ItemCount, TotalItemSize | Export-CSV c:\temp\
Everything we have done so far generates simple text output, but proving that the shell can output data in multiple ways, you can use another
function called ConvertTo-HTML to generate HTML format data that is generated by another command. Here’s an example:
Get-MailboxStatistics | Sort TotalItemSize –Descending |
Select –First 10 | ConvertTo-HTML –Property DisplayName,
ItemCount, TotalItemSize > C:\TEMP\Stats.htm
This command calls ConvertTo-HTML to process some sorted and
selected output from Get-MailboxStatistics. We specify the properties that
we want to format and then output the HTML to a temporary file that you
can then view through a browser, as shown in Figure 4.13.
Chapter 4
4.8 Reporting mailbox data
Figure 4.13
HTML output
from GetMailboxStatistics
There are many examples available on the Internet to show how to pipe
the HTML direct to an IE window that you can define as a custom script
and call instead. For example, here’s the code for a script called Out-IE.PS1,
which leverages the fact that you can use the New-Object command to
invoke a new instance of an application through COM. In this case, we’re
calling Internet Explorer:
$ie = New-Object -com InternetExplorer.Application
while ($ie.busy) { sleep 1 }
$ie.visible = $true
Of course, you could also use the Invoke-Item command to call Internet Explorer. This way saves typing:
Invoke-Item c:\temp\Stats.htm
HTML output is a step forward from plain text but it’s not very exciting. Third-party software providers such as PowerGadgets ( have written PowerShell snap-ins that add the ability to generate
4.8 Reporting mailbox data
customizable charts and other graphic output. These add-ons are a useful
way to report data extracted from Exchange 2007 servers. Figure 4.14 shows
a simple example of how the PowerGadgets Out-Chart command charts the
output from the Get-MailboxStatistics command. The command used to
produce this graph was:
Get-MailboxStatistics –Database ‘VIP Mailboxes’ | Select
DisplayName, ItemCount | Sort ItemCount | Out-Chart –Title
‘Total Items in Mailbox’ –Values ItemCount –Label DisplayName
Figure 4.14
Example of
PowerGadgets OutChart command
You can play around with different variations using the Get-MailboxStatistics command to generate different reports. For example, here is how
to select the top ten consumers of mailbox quota on a server.
Get-MailboxStatistics -Server ExchMbxSvr1 | Sort
TotalItemSize -Descending | Select -First 10
In this example, we count all the mailboxes in a database and return the
maximum, minimum, and average size of the mailboxes in the database. The
figures reported are in bytes:
Chapter 4
4.8 Reporting mailbox data
Get-MailboxStatistics –Database ‘VIP Mailboxes’ | MeasureObject TotalItemSize –Average –Maximum -Minimum
If you need to scan a database to discover all the mailboxes that are
larger than a certain size, you can do it by using a combination of the GetMailboxStatistics and Where-Object commands. In this example, we
report all of the mailboxes on a server that are over 100MB.
Get-MailboxStatistics –Server ExchMbxSvr1 | Where-Object
{$_.TotalItemSize –gt 100MB} | Select DisplayName, ItemCount,
The Get-LogonStatistics command produces equally ugly output as
Get-MailboxStatistics, at least when compared to the ability to browse
information about users that are currently logged on to Exchange through
ESM, but once more, you have it in your hands to decide how to extract and
use the data that EMS provides when you ask for it. As an example of how
you might use the Get-LogonStatistics command, assume that you want
to find out what users have logged in to their mailbox on a server since a specific date. A command like this will do the trick:
Get-LogonStatistics –Server ‘ExchMbxSvr1’ | Where
{$_.Logontime -gt ‘1/1/2007’}
Special properties
When we built our report using the Get-MailboxStatistics command, we
found that the TotalItemSize property outputs data in bytes. TotalItemSize is
actually one of the special properties that crop up in Exchange from time to
time, and while a byte size is interesting when you want to be exact, few
administrators actually think about mailbox sizes, message sizes, or quotas in
terms of bytes. For this reason, we usually convert the value TotalItemSize to
make it more understandable. One common method is shown above,
another that involves the use of the ToKB method to transform bytes into
kilobytes is:
4.8 Reporting mailbox data
Get-MailboxStatistics –Database ‘Mailbox Store’ | FormatTable DisplayName, @{Expression={$Size =
$_.TotalItemSize.Value; $Size.ToKB()} ; Label = “Size (KB)”}
Other properties that report sizes in bytes include anything to do with
mailbox quotas. While reporting these properties can take some extra work,
it is much easier to set values because you can specify what you need in bytes,
kilobytes, megabytes, or gigabytes. For example, to set the new default prohibit send quota size for a database:
Set-MailboxDatabase –id ‘VIP Mailboxes’ –ProhibitSendQuota 500MB
Another example of where you might need to execute some special processing is the ServerName property, which you might want to use to list all
the mailboxes on a server:
Get-Mailbox –Filter {ServerName –eq ‘ExchMbxSvr1’}
This is a convenient and quick way to focus in on a group of mailboxes.
Remember that EMS, like EMC, returns a default set of the first 1,000
matching objects for a command, so if you have more than 1,000 mailboxes
on the server, you would have to pass the ResultSize parameter to set the size
of the set you want to process. For example:
Get-Mailbox –Filter {ServerName –eq ‘ExchMbxSvr1’}
–ResultSize 5000
Let’s say that you now want to interrogate a set of servers to get a complete listing of mailboxes. You do not want to use client-side filters because
this would take too long, so you execute a server-side filter to find all mailboxes on servers with a name like “ExchMbx”:
Get-Mailbox –Filter {ServerName –like ‘*ExchMbx*’}
–ResultSize Unlimited
Problem! EMS responds to say that the ServerName property does not
support a text filter. This seems strange—you have just executed another
server-side filter with the –eq operator. What is the issue with the –like operator? Well, the reason is that the ServerName property is a calculated property that Exchange takes from the ServerLegacyDN property that is stored
with the mailbox. The ServerLegacyDN property means a lot to Exchange,
Chapter 4
4.9 Using the shell for other management tasks
but because it identifies the server in X.500 format, it is very long and
unwieldy to process. Microsoft recognized that most administrators would
prefer not to go to the bother of working with the full ServerLegacyDN.
They therefore provide the ServerName property as a convenient shortcut
that you can use in places such as EMC filters. However, shortcuts do not
come with the full set of features, so you cannot use it with the –like operator. Because ServerLegacyDN is a “proper” property, the command that we
actually need is:
Get-Mailbox –Filter {ServerLegacyDN –like ‘*ExchMbx*’}
–ResultSize Unlimited
The good news is that these are the most obvious examples of where you
will run into strange gotchas with Exchange shell commands. Over time,
Microsoft will probably clean things up and these issues will go away.
Using the shell for other management tasks
The influence of the shell permeates throughout Exchange 2007, so it is
impossible to discuss how to manage Exchange through the shell in one
point. Instead, it is more sensible and logical to discuss how to use appropriate shell commands for different management tasks as we meet those tasks in
the remaining chapters of this book. For example, we review details of how to
use shell commands with objects such as databases and storage groups in
Chapter 5, transport and connector settings in Chapter 6, protocol settings
in Chapter 7, managed folder policies and other user management tasks in
Chapter 8, hardware-related tasks in Chapter 9, and other server management tasks in Chapter 10.
To complete an overall picture of the range of commands supported by
EMS, we can say that:
EMS supports all of the traditional management tasks for the Store
such as creating new databases or setting the properties of a database. Because Microsoft has based EMC on the same code as the
EMS, you can execute equivalent shell commands for any option
available through EMC. For example, to set circular logging for a
storage group:
Set-StorageGroup ‘Unimportant Mailboxes’
–CircularLoggingEnabled $True
4.9 Using the shell for other management tasks
You have to perform many of the new management features that
Microsoft has introduced in Exchange 2007, such as the synchronization of Active Directory configuration and recipient information
between hub transport servers and edge servers, through EMS.
There are exceptions to this rule where the options also appear in
EMC, such as setting up Local Continuous Replication for a mailbox database.
EMS is a flexible management tool whenever you want to interrogate
data across a range of servers or other objects. For example, to find all
of the servers in the organization that support the Client Access
server role:
Get-ExchangeServer | Where {$_.IsClientAccessServer -eq
The same technique can be used to check for other server roles in
the organization using the IsMailboxServer, IsHubTransportServer,
IsEdgeServer, and IsUnifiedMessagingServer properties. You can also
check server properties to determine whether the server runs
Exchange 2007 or later (IsExchange2007OrLater) or is clustered
To return a count of all the servers in the organization grouped by
the edition of Exchange that the servers run:
Get-ExchangeServer | Group-Object Edition
Another way to view servers in the organization is by the server
roles that they support:
Get-ExchangeServer | Group-Object ServerRole
It is also useful to be able to execute a quick command to discover
when the databases on a server were last backed up:
Get-MailboxDatabase –Server ExchMBXSvr1 –Status | Format-List
We can also report on what version of software and edition the
Exchange servers in the organization are running:
Chapter 4
4.9 Using the shell for other management tasks
Get-ExchangeServer | Format-Table Name, ExchangeVersion,
EMS is a powerful ally when you need to execute the same command
for a number of objects. For example, let’s assume that you want to
mount all of the mailbox databases that belong to a server called
Get-MailboxDatabase –Server ExchMbxSvr1 | Mount-Database
Another example is where you want to set the same deleted
item retention period on every mailbox database across the entire
Get-MailboxDatabase | Set-MailboxDatabase –ItemRetention
A nice example of how you can use Get-MailboxDatabase to return
information that you might not expect is when you scan a server to
discover the size of mailbox databases and return the information
sorted from largest to smaller:
Get-MailboxDatabase -Server ExchMbxSvr1 | ForEach {GetChildItem $_.EdbFilePath | Select-Object Name, Length} |
-Property Length -Descending
EMS provides separate commands to manipulate the different server
roles that exist within the product:
Set-/Get-MailboxServer works with the properties of mailbox servers
Set-/Get-ClientAccessServer works
with Client Access servers
works with hub transport and Edge
Set-/Get-UMServer works
with UM servers
The most exciting prospect offered for administrators by EMS comes
through the ability to automate common procedures that you could
not have even thought about in previous releases of Exchange. You
4.10 Command validation
can now create a set of commands to perform an operation once, capture the commands in a PowerShell script, and use the script whenever you need to perform that work.
Apart from speeding things up, if you create the necessary commands in
a script to automate common procedures, you eliminate some of the potential for administrator error and ensure consistency of application of policy by
different administrators across the organization. Of course, you have to get
the commands right to perform any particular operation in the first place,
but it is easier to spend the effort to write and test a script once and reuse it
everywhere than to have multiple administrators all attempt to do the same
thing in slightly different ways.
4.10 Command validation
It is not always easy to test a complex command to a degree of certainty that
you know exactly what the command will do. Some EMS commands that
work with mailboxes support the –validate parameter, which allows you to
request EMS to inform you what will happen if the command executes. You
can see a list of the commands that support –validate by typing:
Get-ExCommand | Where {$_.Definition -Like ‘*Validate*’}
The most important command that you will want to validate is Movebecause you do not want to go ahead with the move of one or
maybe multiple mailboxes, all of which could be quite large, to another
server if things are not exactly right. The value of a GUI is that it can incorporate checks to stop you doing silly things but you are on your own when
you work with the command shell, so it is nice to have a check. For example:
Move-Mailbox –id ‘Jane Smith’ –Targetdatabase ‘New mailboxes’
EMS will pause to tell you that it is about to move the mailbox from its
current location to the target database and ask you to confirm that this operation should proceed. If you confirm, nothing will happen apart from
Exchange performing checks to ensure that it could move the mailbox. As
you can see from Figure 4.15, Exchange completes the command by issuing a
status message, which in this case indicates that everything is okay if you
want to move the mailbox. Apart from stopping you from making a mistake,
Chapter 4
4.10 Command validation
using the –validate parameter is a way to check your syntax before starting an
operation such as moving mailboxes that can take some time to complete.
A much larger group of commands support the –Confirm parameter,
which prompts you with “Do you want to perform this action” before proceeding. To see the list of commands that support –Confirm, type:
Get-ExCommand | Where {$_.Definition -Like ‘*Confirm*’}
For example:
Get-User *Bon* | Set-User –department –eq ‘Flying Tonight’ –
Figure 4.15
Remember that you can use command validation with non-Exchange
commands as well. For example, adding the –confirm parameter to the StopProcess command might stop you killing a process like the Information
Store when you really do not want to.
Stop-Process 5468 -Confirm
4.10 Command validation
The –Whatif parameter is available to allow you to “ask” PowerShell
what will happen if you execute a command. For example, if you type the
command to move a mailbox and append –Whatif
Move-Mailbox –id ‘Bond’ –TargetDatabase ‘ExchMbxSvr1\
Database1’ –Whatif
EMS will tell you that the operation could take a long time to complete
and that the mailbox will be inaccessible until the move is complete. The current release of PowerShell does not support the –WhatIf and –Confirm
switches for commands that you include in scripts, so you have to be sure
about what your code does before you incorporate it into a script, especially if the code deletes objects or otherwise affects the system. Jeffrey
Snover, Microsoft’s PowerShell architect, acknowledges that the lack of
support for confirmation switches may be a problem and supplies a solution in the PowerShell blog at
Another form of validation is to record everything that you do when you
work inside EMS, just in case you have to account for something that you
did later on. To start capturing information:
Start-Transcript c:\Temp\LogofAdminActions.tmp –Append
EMS signals that all actions are now being captured in the file that you
pointed to. To stop EMS capturing actions:
The captured information looks like this:
Windows PowerShell Transcript Start
Start time: 20070112223353
Username :XYZ\Redmond
Machine : ExchMBXSvr1 (Microsoft Windows NT 5.2.3790 Service Pack 1)
Transcript started, output file is c:\temp\LogOfAdminActions.tmp
[PS] C:\Documents and Settings\>get-mailboxdatabase
---VIP Mailboxes
-----------First Storage Group
Chapter 4
4.11 Working with remote servers
ATG Mailboxes
Rooms and Resources
Mailbox Database
LCR Storage Group
First Storage Group
First Storage Group
[PS] C:\Documents and Settings\>get-mailboxstatistics
----------Thompson, Blair
Redmond, Alan
[PS] C:\Documents and Settings\>Stop-Transcript
Windows PowerShell Transcript End
End time: 20070112223444
********************** **********************
4.11 Working with remote servers
The default set of PowerShell commands that support Windows operate on
the local server and cannot execute remotely. For instance, you cannot use
the Get-Process command to return the list of active processes on another
computer. In the eyes of some, this is a weakness that the PowerShell developers should address in a future release, but in the meantime, many
Exchange commands can work with remote servers. The reason why is probably because Exchange has always embodied the concept of remote administration, whether it was the ability of an administrator to manage all the
servers in a first-generation Exchange site, or the ability of an enterprise
administrator to work with all of the servers in an organization in Exchange
2000 and 2003.
Clues that show you when you can work with remote servers include the
presence of the –Server parameter, where you can pass the name of the
remote server that you want to retrieve information from or set a property
on, or the presence of a target for data to move to or to be used as a source.
Examples include Get-MailboxServer, Get-TransportServer, Get-MailboxStatistics, Export-Mailbox, Set-Mailbox, Move-Mailbox, and so on.
The ability of EMS to work with remote servers out of the box is a
strength of the product that should not be overlooked.
4.12 Working with non-Exchange 2007 servers
4.12 Working with non-Exchange 2007 servers
Another interesting aspect of PowerShell is that you are not restricted to
working with Exchange 2007 servers. Recall that previous versions of
Exchange support WMI and that you can use WMI to access information
about an Exchange server, such as the list of mailboxes on a server. PowerShell supports access to WMI objects, so you can put the two together to
access some information about Exchange 2000 and 2003 servers from a PowerShell script. Access to legacy Exchange data is limited because WMI can’t
access some of the information on these servers, nor can it perform some of
the operations that are commonplace on Exchange 2007 servers, such as creating a new mailbox or a storage group. Nevertheless, you can still do some
useful work. For example:
get-WMIObject -class "Exchange_mailbox" -namespace
"root/MicrosoftExchangeV2" -computerName "ExchServer01"
–Credential "XYZ\Smith" | Format-Table MailboxDisplayName,
@{e={[math]::round ($_.Size / 1KB , 2)} ; Label ="Size"}
This example uses PowerShell to access the WMI object and read information about Exchange mailboxes from the ExchServer01 computer. The
command passes a set of credentials for the domain that owns ExchServer01.
PowerShell then pipes the output from the WMI query through the formattable command to generate a formatted table, dividing the size of the mailbox (in bytes) by 1KB to return the size in kilobytes. The point about using
PowerShell in this way is that you should not restrict yourself to looking at
PowerShell exclusively for Exchange 2007.
Apart from working through WMI, a small selection of the standard
EMS commands work with legacy Exchange servers. For example, here is an
edited version of the details that the Get-ExchangeServer returns about an
Exchange 2003 server:
Get-ExchangeServer ExchSvr2003 | Format-List
C:\Program Files\Exchsrvr
/O=xyz/OU=Exchange/nn=Configuration /
: False
Chapter 4
4.13 Testing Exchange 2007
: False
: False
: False
: False
: No
: False
: False
: Version 6.5 (Build 7638.2: Service Pack 2)
: None
: False
IsExpiredExchange2007TrialEdition : False
: 00:00:00
: True
: 0.0 (6.5.6500.0)
: {top, server, msExchExchangeServer}
: 9/18/2006 5:23:44 PM
: 2/21/2003 3:46:28 PM
The Get-Mailbox and Get-CasMailbox commands work for mailboxes on
legacy Exchange servers but other commands such as Get-MailboxStatistics
and Get-MailboxDatabase do not. Because you cannot be sure that all of the
EMS commands will work against legacy servers, the best idea is to follow the
guideline of managing legacy Exchange servers with legacy tools and managing
Exchange 2007 servers with Exchange 2007 tools.
Because PowerShell supports many different providers through a consistent interface, you can write code that mixes and matches data from multiple
sources to automate reports, provisioning of servers, and other administrative
operations. While it is true that you will encounter some limitations (such as
the fact that using WMI to work with Exchange 2003 servers has not the
same potential as using PowerShell to work directly with Exchange 2007
servers), there is still a lot of useful work that PowerShell can do in a multiversion Exchange environment.
4.13 Testing Exchange 2007
Exchange 2007 provides a set of shell commands that you can use to verify
that everything is working correctly on a server. This is the first time that
Microsoft has provided commands to help administrators check that
Exchange is working properly and as you look at the comprehensive suite of
4.13 Testing Exchange 2007
test commands available in Exchange 2007, you can’t help reflecting that it
has taken Microsoft a long time to fill in this particular gap. You need to have
Exchange administrative permissions over a server before you can execute
these commands.
The first step in testing a system is to check that all the necessary components are present and are at the right revision level. The Test-SystemHealth command helps by scanning a server to verify that it is running the
correct versions of software (related to Exchange 2007) and to detect some
other conditions that may cause Exchange 2007 a problem. Before it runs,
the Test-SystemHealth command checks the Microsoft web site for any
updates that it should know about to be able to test a server accurately.
Figure 4.16 shows the result of running Test-SystemHealth on a server
that runs a beta version of Exchange 2007 and you can see how the command reports this issue. More importantly, Test-SystemHealth reported that
a driver is present (for HP Remote Insight Manager) that does not meet the
Microsoft standards for some reason. There may be a good reason why the
driver is not compliant, but it is always best to check out errors that
Exchange flags like this.
Figure 4.16
The results of the
From the perspective of Exchange, the most basic test that you can perform is to determine whether all the services that collectively make up the
different server roles in Exchange 2007 are running on a server. Use the
Test-ServerHealth command for this purpose. As you can see, Exchange
responds with the server roles that are installed on the target server and a list
of services that are running plus any that are not currently operational.
Test-ServerHealth –Server ExchMbxSvr1
Chapter 4
4.13 Testing Exchange 2007
RequiredServicesRunning ServicesRunning
----------------------- -------------------------------True
Client Access True
Hub Transport True
Client connections
The Test-MapiConnectivity command verifies that clients can make MAPI
connections to a database. If you run this command on a mailbox server,
Exchange attempts to make a connection to every mailbox database on the
server and reports how quickly it can make the connection. You can specify
the name of a server or a mailbox database to focus in on a specific issue. If
you do not specify the name of a mailbox to connect to, then Exchange connects to the special system mailbox in the database. This command runs a
quick check against a selected mailbox server:
Test-MapiConnectivity –Server ExchMbxSvr1
-------VIP Mailboxes
new Mailboxes
Latency(MS) Error
----------- ----5
4.13 Testing Exchange 2007
The Test-PopConnectivity and Test-IMAPConnectivity commands
allow you to check that a Client Access server is able to service incoming
requests from POP3 and IMAP4 clients, including their ability to make
LDAP requests to search the Active Directory. In this example, note the use
of a pipelined call to the Get-Credential command to force Exchange to
prompt for credentials to log onto the XYZ\Redmond account.
Test-PopConnectivity –ClientAccessServer ExchCAS1
–MailboxCredential(Get-Credential XYZ\Redmond)
Test-IMAPConnectivity ClientAccessServer ExchCAS1
–MailboxCredential(Get-Credential XYZ\Redmond)
MailboxServer Scenario
------------- -------ExchMbxSvr1
Test IMAP4
Result Latency(MS) Error
------ ----------- ----Success
The Test-OWAConnectivity command tests that Outlook Web Access
connections are functioning as expected, including the ability to log onto a
mailbox with Outlook Web Access or that a Client Access server provides the
correct URLs to allow clients to connect. You can test a specific server or all
servers (mailbox and Client Access) in an Active Directory site. For example,
this command tests the ability to log onto a mailbox on a specific server for
the stated user. You will be prompted by EMS to provide the credentials
before the command executes:
Test-OWAConnectivity –MailboxCredential(Get-Credential Xyz\Redmond)
–MailboxServer ExchMbxSrv1
ClientAccessServer MailboxServer URL
------------------ ------------- --ExchMbxSvr1
Scenario Result
Latency Error
-------- ------ ------- ----Logon
Success 5031.12
Mail Flow
The Test-MailFlow command allows you to test whether the transport service is functioning correctly by logging on to the system mailbox in the first
mailbox database on a server to create and send a test message from one
server to another or to an external email address. The test also reports the
Chapter 4
4.13 Testing Exchange 2007
time taken to send the message and whether the test used an external address.
Figure 4.17 shows the resulting message, including the disclaimer applied by
the transport system through a transport rule. You can find out more about
how to create and manipulate transport rules in Chapter 8.
Test-MailFlow –TargetEmailAddress [email protected]
Figure 4.17
Message generated
by the TestMailflow
The Test-OutlookWebServices command tests whether the set of web
services (AutoDiscover and Availability) that Outlook 2007 and Outlook
Web Access clients depend on to locate services such as an OAB distribution
point and free and busy data are working. In this example, we test that the
services are running on a target Client Access server and state that we want to
use the email address “[email protected]” to test the Outlook provider (the service that handles requests from Outlook clients). In addition,
we want to use the Availability service to check the free and busy data for the
user “[email protected]”
Test-OutlookWebServices –ClientAccessServer ExchCAS1 –Identity [email protected] –
TargetAddress [email protected]
Type Message
---- ------Information About to test AutoDiscover with the ...
Information Testing server
Information Found a valid AutoDiscover service c...
Information The SSL certificate on ExchCAS1.xy...
4.14 PowerShell for Exchange administrators
Information Contacted AutoDiscover at https://Ex...
Success Successfully contacted the AS servic...
Error When contacting http://ExchCAS1...
Error Error when contacting the OAB servic...
Success Successfully contacted the UM servic...
Success Successfully tested AutoDiscover.
Among the commands that you can use to verify different aspects of the
transport system are Test-IPAllowListProvider and Test-IPBlockListProvider, which tests the connection to an external third-party provider of
an IP allow or block list that you may want to use on a hub or Edge server. In
both cases, you pass the name of the provider and an IP address of a suspected problem server to test. The result of the test will be a true or false to
indicate that a connection to the provider is possible and another true or false
if the lookup for the IP address was successful.
Test-SenderID is a useful command if you use checks against Sender ID
records in DNS to verify that servers are not sending spam. For example:
Test-SenderID –PurportedResponsibleDomain
If you deploy Exchange edge servers, you can run the Test-EdgeSyncommand on a hub transport server to verify that any edge
servers that synchronize their copy of Exchange configuration and recipient
data with Active Directory (in their local copy of ADAM) are up to date.
We will review details of how Edge synchronization works in Chapter 6.
Miscellaneous test commands
If you have deployed Exchange Unified Messaging, you can use the Testto check that everything is working. Likewise, if you have
deployed Outlook Anywhere, you can use the Test-OutlookAnyWhereConnectivity command to test that it is functioning correctly.
The Test-ExchangeSearch command verifies that Exchange content
indexing and retrieval is working properly. See Chapter 5 for details.
4.14 PowerShell for Exchange administrators
After all of these examples of EMS in action, what can we conclude about
PowerShell from the perspective of an Exchange administrator?
First, PowerShell is simply something that you have to learn. At the
start, you may want only to execute a couple of one-line commands, but
gradually (unless you work with a very simple server), you’ll find that EMS
Chapter 4
4.14 PowerShell for Exchange administrators
becomes part of your daily working life and that your command-line skills
increase so that you do something more complex, such as writing your first
script. You’ll check out web sites to find tips from other administrators and
sample scripts that you can adapt for your own environment. Eventually,
because PowerShell is just so useful, it will be a natural way for you to solve
many of your day-to-day administrative problems.
Second, PowerShell is powerful—maybe too powerful at times. If you
have permission over the Exchange organization or even just a server, you can
wreak havoc with a PowerShell command that does something that you
didn’t expect, perhaps because you made a mistake with syntax and moved
hundreds of mailboxes to a server that couldn’t accommodate the load when
you really wanted to move just ten. The –Confirm and –Validate parameters
are there to provide administrators with protection, but only to confirm that
you have typed commands correctly and have not missed out any important
parameters. They will not stop you doing something stupid, but they may
cause you to pause before you commit to a command. Human nature being
what it is, some won’t use these parameters to check commands before execution, just like some drivers persist in thinking that seat belts are for wimps.
Take your time and master PowerShell before you rush into anything that
could cause damage to your organization. Learn the commands, learn the
parameters, and test, test, and test again.
To help you to get used to PowerShell and to understand the power that
it can have over your environment, it’s a great idea to deploy a test lab that
accurately reflects the characteristics of your organization. If you run clustered servers or a unified messaging server, then they should be in your test
environment. It is not difficult today to create a set of virtualized test servers
that run on a single large computer or even on a powerful laptop, and this
will make all the difference to you when you test how PowerShell commands
work with Exchange data. It is always better to screw up in a test environment than to inflict damage to a production system.
Third, you are going to accumulate scripts over time. You will develop
some and find others on blogs and web sites that you then adapt for your
own purposes. PowerShell scripts are like programs and should be treated as
such. Make sure that only administrators use them and that you exercise
some degree of version control (and testing, as described above) over the
scripts that you plan to use for operational control. It is not great to discover
that you cannot find the script that you wanted to use because someone has
deleted or moved the file, so take the time to store copies of important scripts
in a safe place.
Fourth, PowerShell is great, but let’s realize that it is in its early days.
Like the first version of any product, PowerShell will evolve over time and
remove some of the problems that you can see today, such as syntax inconsistency in places. Another annoyance is that some commands cannot out-
4.14 PowerShell for Exchange administrators
put to a file. For example, you can execute Get-Mailbox > c:\
Mailbox.txt to create a list of mailboxes, but you cannot do Get-SendConnector > c:\Send.txt or Get-ReceiveConnector > c:\Receive.txt to
create a list of send or receive connectors. However, if you pipe the output
of these commands through the Format-Table command (Get-SendConnector | Format-Table > c:\Send.txt), the output appears. It’s all very
frustrating at times.
Last, the Exchange developers have done a great job of embracing and
extending PowerShell for Exchange 2007. You can also use PowerShell to
manage Windows and there is evidence that a solid ecosystem of third-party
products will grow up around PowerShell to help fill in any gaps that exist.
However, not every Microsoft product supports PowerShell today and you
can expect that legacy versions of Microsoft products will not support PowerShell in the future, so you can predict that Microsoft’s fragmented administrative model will continue to exist until all of the product development
groups figure out how they will incorporate support for PowerShell. Even
then, to get to a more integrated administrative model, you will have to
upgrade to the product versions that support PowerShell, and we all know
how long that can take.
The net is that PowerShell marks the start of a new scripting experience
for Exchange administrators. It will do great things when used correctly, but
don’t expect PowerShell to address all the sins of Microsoft’s curiously deficient administrative past.
Chapter 4
This page intentionally left blank
The Store
At its core, Exchange is a database application, albeit one that is highly tuned
for the kind of transactions that a messaging system generates. The Information Store, or simply “the Store,” lies at the heart of Exchange. Without a
well-configured and managed Store, an Exchange server cannot provide a
reliable and robust service to users.
Exchange 2007 introduces a number of changes for the Store, including
an increase in the number of supported storage groups, internal changes that
result from the move to the 64-bit platform, and database portability
between servers. In this chapter, we review the core components of the Store
and discuss how you can manage these components most effectively. Some of
the other important changes, specifically continuous log replication, are in
Chapter 9.
Introducing the Store
While Exchange is a database application, it is important to stress that the
kind of transactions that Exchange users generate are very different to those
of other, more structured databases. For example, the kind of transactions in
a banking database where each transaction is broadly similar in terms of
data and contains fields such as date, account number, transaction type, narrative, and amount. In the case of Exchange, a transaction can be radically
different to the preceding transaction because one message might contain a
simple “Hi” and goes from one user to another, and the next is a message
sent to a large distribution list and has three paragraphs of text in the message body and a large PowerPoint attachment. Microsoft designed the Store
to be able to deal with widely different transactions in a scalable and robust
manner and this aspect of Exchange has contributed a great deal to the
product’s success.
Since Exchange 4.0, the Store has used a database engine called ESE
(Extensible Storage Engine), which is a variant of Microsoft’s JET (Joint
Engine Technology) generalized database engine that provides the specialized
5.1 Introducing the Store
features required by Exchange. The different layers in the Store take the following responsibilities:
ESE organizes low-level database structures. ESE is a multi-user
ISAM (Indexed Sequential Access Method) database with full Data
Manipulation Language (DML) and Data Definition Language
(DDL) capabilities.
The Jet implementation of the database engine organizes the ESE
database structures into pages and trees. The specific implementation
of Jet for Exchange was referred to as “Jet Blue” up to the version
shipped with Exchange 5.5 (ESE97) to distinguish it from other variants such as “Jet Red.” From Exchange 2000 on, Exchange’s database
engine is simply “ESE.” All versions are designed to optimize the
storage of loosely-formatted, semi-structured data, which is a tad different from the very structured approach to data preferred by other
databases, such as SQL.
The Store organizes its database pages into tables that hold messages
and attachments. The Store uses a hierarchical arrangement with
mailboxes holding folders and folders holding the messages and
attachments belonging to messages. The hierarchical model is not
unique to Exchange and many other messaging implementations
have used it, so this model is very much the traditional approach.
Other Microsoft products use JET technology. The Active Directory is
the best and closest example to Exchange as its internal structure and
arrangement is very close to the Store. Access is another example of JET technology in action, but you should not assume that the database engine used
by Access is anywhere close in functionality to the version used by Exchange.
There is, after all, a world of difference in the technology required to drive
personal databases and that are needed by very large messaging servers. During the evolution of Exchange, Microsoft has used different versions of ESE.
For example, the version used in Exchange 2000 and 2003 is “ESE98.”
While the versions changed, the basics of the underlying structure of the
Store remains the same, with some important differences in Exchange 2007
that we will come to shortly.
ESE arranges database pages into a relatively shallow B-tree (balanced
tree), a relatively common database construct (Figure 5.1). Exchange uses a
variant called B+ tree, which provides a higher degree of efficiency by limiting the extent of the internal hierarchy within the database. The pages are of
fixed size (4KB in all versions of Exchange prior to Exchange 2007, 8KB
from Exchange 2007 onwards—the same size as Microsoft uses in the Active
Directory database). In the B-tree, the first two pages are the database header,
5.1 Introducing the Store
followed by a single page (logical database page 1) at the root level that contains pointers to the pages that make up the internal structure of the database. Each page has a 40-byte header that contains the page number, a count
of free bytes available in the page, and a checksum. The page checksum plays
an important role in maintaining the overall integrity of the database. The
page header also contains information about adjacent pages to allow ESE to
navigate to these pages quickly.
Figure 5.1
The B-tree
structure within an
ESE database
Within the internal structure of an ESE database, there are pages that
contain pointers to the leaf pages, which hold data as database records. An
individual page can contain up to 400 pages to leaf pages. The design of the
database structure allows ESE to access any page quickly with a minimum
number of referrals within the database (from root to internal to leaf ). If ESE
needs to fetch data from contiguous pages, it can do so by using the forward
and backward links stored in the pages. Interestingly, ESE does not support
the notion of data locality within its databases, as the ESE does not divide
into pages indexes and data pages. In addition, ESE arranges database pages
in a random manner so that the Store does not write new data in any particular order and it can assign data to any available page in the database. These
facts make it difficult for storage arrays to cache critical data, such as an
index, and so can make it more challenging to scale up an Exchange server to
cope with large numbers of users that generate many I/O transactions. The
move to the 64-bit platform offers ESE more headroom to cache data and so
may make it easier to scale up larger servers.
Microsoft designed the Store to meet the ACID (Atomicity, Consistency, Isolation, and Durability) test for database transaction integrity.
Atomic means that the Store only applies transactions to the database if they
Chapter 5
5.1 Introducing the Store
are complete and intact. A transaction is a series of modifications to the database that leaves the database in a consistent state. These modifications are
called operations. When you send a message, the Store effects your command
by executing the operations to assemble all of the MAPI properties for the
message, such as the subject, recipients, author’s email address, and so on,
and whatever body parts are required, and posting the message in what you
perceive to be a single logical step. Behind the scenes, the Store hands off the
message for transport to its eventual recipients and then it updates the Sent
Items folder in your mailbox with a copy of the message. An operation is the
smallest change that an application can make to an ESE database. Operations include insert, delete, replace, and commit. A database is in an inconsistent state if ESE does not execute all of the operations required to make up a
transaction, as in the case when a sudden power outage causes a server to fail.
In such a circumstance, the Store replays as many operations as possible from
the contents of a database’s transaction logs, but it will ignore any transaction
that is incomplete. The Store maintains consistency by insisting that transactions must be complete and it achieves isolation by not making a database
change visible to users until it is complete. The presence of the transaction
logs and the dual-phase commit for transactions into the Store databases
achieves durability, the final part of the ACID1 test.
ESE uses dual-phase commit for transactions. The first phase writes the
data for the updated or deleted pages that contain the data that forms the
transaction from the volatile cache2 in memory into a transaction log file.
These pages are referred to as “dirty” or being in a dirty state, and the vast
majority of transactions in Exchange affect more than one page. The second
phase then updates pages in the database using the data held in the database
cache (memory) and writes the updates to disk. Both phases can occur
simultaneously and enable ESE to commit transactions to disk rapidly, even
under load. Note that ESE never commits a transaction to the database
unless it is able to commit the entire transaction; if ESE committed a partial
transaction then it would both fail to meet the atomic part of the ACID test
and run the risk of introducing corruption into the database. The presence
of the data in the transaction logs allows Exchange to recover data to a failed
database after an administrator recovers it from a backup copy by replaying
the transactions stored in the log files into the recovered database until the
database is up to date.
At the logical level, the Information Store uses ESE to organize database
pages into a series of tables that organize data in a form that Exchange can
ACID means that the ESE is a transactional-based database that implements atomic, consistent, isolated, and durable
transactions. See Transaction Processing: Concepts and Techniques, Gray and Reuter, 1983.
The msExchESEParamLogBuffers parameter (an attribute that Exchange maintains for each storage group) determines the
size of the volatile cache. See Microsoft Knowledge Base article 328466 for more information on how to tune the parameter. The size of the cache influences the I/O profile of the Store; a larger cache encourages fewer large I/O operations
while a smaller cache generates a larger number of small I/O operations.
5.1 Introducing the Store
consume. For example, all of the mailboxes in a mailbox store have entries in
a mailbox table. The entry for an individual mailbox acts as a pointer to a
separate table that holds entries for all of the folders in the mailbox. When an
Outlook client connects to Exchange, it accesses the mailbox table to locate
the pointer to its folder table and then reads all of the folder information to
build the folder hierarchy for display to the user. The Outlook client can
then access the pointer in the folder table to the inbox folder and use that to
read information about all the messages in the inbox folder. If the user then
decides to read a message, Outlook takes the pointer for the message and uses
it to extract the MAPI properties for the message and to populate the message form that it displays to the user. If the message has attachments, Outlook uses the pointers to the entries in the attachment table to fetch the
attachments and display them in the message form. ESE maintains different
tables for any custom views that users create. Unlike the standard tables
(mailboxes, folders, etc.), users can create views3 at any time and can order a
view by any field that they include in the view. Exchange supports nested
folders, or subfolders within folders, with pointers providing the necessary
link from a child folder back to its parent. From the description above it is
easy to imagine how many different tables that ESE has to manage in a typical Exchange mailbox database. There will be at least one table per mailbox,
and it is likely that there will be tens or even hundreds of tables per mailbox.
As you might expect, the tables that store messages and attachments occupy
over 90% of the typical Exchange database. The dynamic nature of views and
the sheer number of tables in the database pose a major performance challenge for any database, and one that Exchange has been able to address over
the years, but not without some pain.
For example, views are an excellent user feature and they work very well.
However, once a user has a large mailbox that has more than a few thousand
items in some of the critical mail folders (inbox, sent items, calendar, deleted
items), then it becomes very performance intense for ESE to generate a new
view for the folder. Multiply this requirement by several thousand users
accessing a server concurrently and you understand the challenge. The fact
that mailboxes are increasing steadily in size as users store more information
compounds the performance challenge. The advent of cached Exchange
mode in Outlook 2003 addressed some of the problem by offloading activity
to the client, but the need still exists for ESE to perform a huge amount of
processing on the server. For example, no matter how many client-side operations Outlook takes care of, the server still has to handle the insertion of
new messages as they arrive into user mailboxes to generate appropriate notifications, update the views, and so on.
Experience demonstrates that the best Exchange administrators have a
good knowledge of the Store and of how ESE works. It is logical to assume
Only MAPI clients support views.
Chapter 5
5.2 Differences in the Exchange 2007 Store
that anyone who attempts to administer a database application will understand the fundamental underpinnings of the application, but not always the
case in practice.
Differences in the Exchange 2007 Store
When Microsoft began the design process for Exchange 2007, their original
plan was to use this version to change the Store over to use the “Yukon” version of SQL (as implemented in SQL 2005) as the database engine. The plan
(the “Kodiak” project) was very logical because it had the potential to deliver
great savings to Microsoft as they standardized on a single database platform.
With a single database platform for multiple applications, Microsoft would
simplify development and eliminate a lot of cost that they incur in testing,
packaging, and support. They would also be able to take advantage of some
of the higher transactional capability that SQL delivers, especially on highend systems. However, the engineering complexity of moving Exchange from
ESE to SQL proved too much to bite off and Microsoft decided to continue
with ESE in Exchange 2007. One major issue is that ESE accommodates all
the quirks and foibles of the way that MAPI organizes data into properties
and allows users to create their own views of data, such as sorting by received
date, author, or item size. If the Store had only to deal with simple clients like
those that use the IMAP protocol, it would be much easier to swap database
engines. However, MAPI clients like Outlook can create different views for
each folder and on a server that supports thousands of mailboxes, each of
which can hold thousands of folders, the database might end up by having to
deal with millions of folder views. As we can see from the experience with
ESE, handing thousands of dynamic views is not an insurmountable problem, but it is challenging for a database like SQL that is designed to deal with
the kind of predictable transactions that exist in traditional database applications. Note that changing folder views is a feature executed on the client
when Outlook operates in cached Exchange mode, so there is less of an
immediate impact on the server as the new view can be synchronized in the
background. New views created by Outlook Web Access clients impact the
server more heavily because the change occurs immediately, resulting in more
work for ESE to create the view and return the result to the client—and a lot
of processing results if the folder contains a lot of items. Microsoft’s recommendation is to keep less than 5,000 items in a folder, but users aren’t often
aware of this point and probably couldn’t care less.
All these reasons underline that there is no point in going ahead with a
changeover in underlying database technology if you cannot guarantee that
you can achieve at least comparable performance and functionality, and this
was not possible in the Exchange 2007 timeframe. Microsoft could have persisted with the project to make Exchange 2007 use SQL, but only with the
prospect of slipping eventual product availability to an unknown date and
5.2 Differences in the Exchange 2007 Store
with no real guarantee of achieving the necessary performance, even on a 64bit platform. The decision to stay with ESE is therefore very understandable,
but we may yet see a move to SQL in a future version of Exchange, once
Microsoft has had the chance to figure out how to solve the technical issues.
Because of the complexity involved in making the transition from ESE, it is
unlikely that the changeover will occur in the next version of Exchange, but
you never know after that.
Instead of the radical transformation required to move the Store to use
SQL, Microsoft took on the technical challenge of how to improve the ESEbased code to take advantage of 64-bit servers. Clearly, the extended address
space on a 64-bit platform compared to a 32-bit platform provides an enormous increase in the memory available to any application, so the question is
how Exchange could take maximum advantage of the available memory.
Microsoft’s answer in Exchange 2007 is to trade I/O operations for memory
and so address a performance bottleneck that has always afflicted Exchange,
the need for huge amounts of I/O capacity before a powerful server can handle the load generated by large numbers of concurrent mailbox connections.
In addition, Exchange 2007 offers some hope that we can reduce the expense
of deployments by increasing the ability of low-end storage configurations to
provide the necessary power to service Exchange. In effect, you do not always
have to deploy a SAN to provide storage to Exchange 2007, as a lower-priced
and less complex storage configuration may be able to provide the necessary
I/O capacity and throughput. However, you should not read this to be a general recommendation to move away from SAN deployments for Exchange
because there are many good reasons to continue to use SANs, such as
increased resilience to hardware failure, more configuration options, and the
ability to consolidate storage requirements for multiple applications into a
single infrastructure.
Are 64 bits that important?
Exchange has been down the 64-bit path before as Exchange 4.0 through 5.5
all supported the Digital Alpha 64-bit platform from 1996 to 2000. Indeed,
Digital engineers did a lot of work to help Microsoft solve some major scalability problems in the Exchange 5.5 Information Store. However, Microsoft
made the decision that Windows 2000 would not support Alpha and the
consequence was that Exchange 2000 and subsequent releases reverted to 32bit only.
The Alpha platform was the fastest system of its time, but it was handicapped by the fact that Microsoft merely recompiled programs like Exchange
to run on Alpha as Microsoft did not want to incur the additional cost of
maintaining two different code bases for 32-bit and 64-bit versions. For this
reason, the Alpha version of Exchange could never take full advantage of the
Alpha processor, and while the CPU delivered raw speed, the overall package
Chapter 5
5.2 Differences in the Exchange 2007 Store
was handicapped by the fact that the Exchange code never attempted to maximize its use of Alpha by, for example, caching more of the Store or directory
databases in memory. Alpha enthusiasts regretted the situation and Alpha’s
speed advantage was gradually eroded by Intel’s 32-bit CPUs, but it is fair to
ask whether Exchange could have actually taken full advantage of a 64-bit
platform at this point in its evolution. After all, many companies were still
migrating from products such as Microsoft Mail and Lotus cc:Mail, storage
was more expensive and delivered less capacity than is available now, mailboxes were smaller, and the number of mailboxes supported by a server was
generally fewer than today. All in all (and with the benefit of hindsight), the
32-bit versions of Exchange 5.5, 2000, and 2003 provided the necessary
combination of speed, performance, and supportability for the state of messaging technology as it existed.
Across the board, many of the parameters that dictate the performance
that a messaging server like Exchange can deliver have changed in the last
decade. For example:
Message size is greater: Many organizations that migrated to
Exchange 4.0 in 1996 came from “green screen” email systems where
the average message size was in the 2KB–4KB range. Today, in the
intensely graphic world we live in, the average message size (including
attachments) is closer to 100KB and can go higher as users cheerfully
shuttle large PowerPoint presentations, MP3 music files, digital photos, and other attachments around.
Mailbox size is greater: While administrators attempt to limit mailbox size so that backup and restore times stay reasonable, the demand
from users for more space grows. The fact that storage is much
cheaper today than it was even three years ago means that it is often
cheaper to deploy more disks than to ask users to spend the time necessary to clean up their mailboxes. The net result is that the average
mailbox size has grown from 25MB in 1996 to well over 100MB in
most organizations today. Indeed, multi-gigabyte mailboxes are not
unknown and the largest mailbox is now well into the tens of
gigabyte range.
Client features encourage users to consume disk space in a wasteful
manner. For example, Outlook’s default behavior is to incorporate
the complete text of a mail thread in a message. This is sometimes
useful because you can scan all of the previous messages to see the
flow of a discussion, but it also generates larger messages. If Outlook
had some intelligent filter that allowed it to extract the essential
points in a debate and present them then we could move away from
ever-expanding messages, but software engineering has not cracked
that problem yet.
5.2 Differences in the Exchange 2007 Store
Consolidation is a continuing theme: The advent of Exchange 2000
began a phase of consolidation as administrators attempted to remove
complexity from their organization by reducing the number of servers. Consolidation has been helped by cheaper network costs, so
Exchange servers deployed in branches have gradually been pulled
back into centralized datacenters. The result is that today’s Exchange
servers support a larger number of mailboxes than they have in the
past. Storage is tied to consolidation as there has been a move to use
larger and more complex storage systems to provide the consolidated
servers with the I/O performance and raw storage capacity that these
servers require. In addition, storage networks have become more
complex as administrators seek to deploy more resilient configurations such as clustered servers, both the simple variety as well as
stretched or geographic clusters.
We access mail more often and through more varied ways: In 1996,
the average user probably accessed Exchange from a desktop PC and
may have spent four hours working with their email. Today, we
increasingly access email using laptop PCs more than desktop PCs,
so we can be online and access our mailboxes for longer. We also use
mobile devices like PDAs, Blackberries, and SmartPhones, so there
are more ways to get to our inboxes and we stay connected more. A
further consequence is that servers have to support more concurrent
authenticated connections than ever before. For example, as I type
this document, I am connected to Exchange via Outlook and via my
Windows Mobile 6.0 SmartPhone. Many of these devices constantly
poll the server, which creates a different type of workload than normal user activity and another consequence is that because we are
more connected, we generate more messages. The introduction of
cached Exchange mode in Outlook 2003 has also increased connections as Outlook now maintains multiple connectivity threads to the
server for background synchronization operations. Each connection
increases demand for kernel memory, which becomes a limited commodity on a 32-bit server. The problem caused by multiple client
connects is well illustrated in Microsoft Knowledge Base article
912376, which describes why Exchange 2003 SP2 runs out of kernel
mode memory when the paged pool is exhausted due to too much
memory being allocated to client access tokens. Another issue arises
because Windows 32-bit servers have limited capacity to support
HTTPS connections and top out at between 17,000 and 20,000
connections due to kernel memory exhaustion. This limits the number of clients that can connect to a server using RPC over HTTP clients (each client will consume at least 10 TCP connections and may
consume more to handle the RPC threads that allow Outlook to
accept and send messages at the same time).
Chapter 5
5.2 Differences in the Exchange 2007 Store
Table5.1 summarizes how different configuration guidelines have
altered since Microsoft shipped 4.0 in 1996. This data comes from various
articles that I have written about Exchange over the period and the data are
not specific to any one organization, so your mileage will vary. However, they
are an illustration of the growing pressure on Exchange in terms of dealing
with more mailboxes, bigger messages, and more data than ever before.
Together with the need to support more client connections—and more intelligent clients—should we be surprised that the 32-bit Exchange architecture,
especially in the Store, is creaking and groaning?
Table 5.1
Bigger mailboxes from bigger messages
Exchange 4.0
Exchange 5.5
Exchange 2000
Exchange 2003
User mailboxes per
Up to 4,000
Average mailbox size
Average message size
Up to 256MB
Up to 512MB
Larger mailboxes and message sizes also mandate a need for additional
storage on servers. If you consider following some of the commentary from
Microsoft and look at increasing mailbox quotas to something like 1GB, you
might also look at removing PSTs. I am on record that I do not like PSTs
because these files are insecure and prone to failure. It is much better to move
the data from PSTs to server mailboxes and take advantage of the greater
resilience and security that servers deliver compared to client-based storage.
Through product briefings from late 2005 onwards, Microsoft communicated that Exchange 2007 would be the first true 64-bit version of
Exchange. Their dilemma was whether to continue with a 32-bit version of
Exchange. Keeping two different code bases increases Microsoft’s engineering
expenses as they develop, test, debug, package, and translate Exchange. From
an engineering perspective, if Microsoft had retained a 32-bit version 4, they
would not have been able to address some of the problems that prevent the
Store scaling up, such as the virtual memory fragmentation issue that limits
cluster state transitions in Exchange 2000 and 2003 or increasing the number of databases that a single Exchange server can support. Windows allocates
virtual memory in contiguous address blocks. Every time the Store mounts a
storage group and its databases, it requests 10MB of free virtual memory to
As discussed in Chapter 1, Microsoft provides a 32-bit version of Exchange 2007 for testing purposes. Microsoft does not
support this version for production use.
5.2 Differences in the Exchange 2007 Store
perform the operation. If the largest block of free virtual memory on the
server is less than 10MB, the Store cannot mount the storage group, even if
Windows is able to provide 10MB composed of smaller noncontiguous
address blocks. As a server comes under load, it allocates virtual memory to
all the processes that request memory, and eventually you get to a point
where large chunks of contiguous virtual memory are unavailable. The only
solution is to reboot the server and start the process of allocating memory to
processes again.
In addition, 32-bit Exchange servers experience some problems with
kernel mode memory, including providing the memory required to handle
delegate access to calendars and mailboxes or the number of client connections that arise when users access Exchange through a variety of different
devices. One indication of when a 32-bit Exchange server is under strain is if
you see errors such as paged pool allocation failures in the event log during
times of peak load (such as 9AM to 11AM every morning when users first connect to their mailboxes). Virus scanners, which are an absolute necessity for
Exchange servers, also use kernel mode memory and decrease the amount
available to other processes. Full-text indexing and archiving can also occupy
resources that Exchange could otherwise dedicate to clients.
Over time, we have seen that Windows increasingly takes up more kernel memory as it loads more drivers. For example, Windows 2003 SP1 has
10% less kernel memory available for applications because Windows enables
IPSec by default and loads additional storage drivers. Another problem that
is sometimes seen with Exchange 2000 and 2003 is the kernel mode exhaustion that can happen if you attempt to serve more than 10,000 mailboxes
with a single public folder server (by making the public folder server the
default public store for these mailboxes).
Microsoft struggled with virtual memory issues in Exchange since they
released Exchange 2000 and they issued a number of patches to address
problems with components such as client access tokens that consumed virtual memory (for an example of the kind of virtual memory management
problems experienced by Exchange 2003, see Microsoft Knowledge Base
articles 912376 and 912480). The 64-bit version of Windows supports eight
terabytes of user-mode memory and eight terabytes of kernel-mode memory,
which means that an Exchange 2007 server can handle many more concurrent client connections and practically eliminates any of the current issues
that Exchange 2003 experiences with virtual memory fragmentation and kernel mode exhaustion. Greater ability to accommodate load and the growing
inability of the 32-bit Windows platform to deal with the demands of applications like Exchange is probably the most compelling reason to move to the
64-bit platform.
You could argue that the easy answer would have been for Microsoft to
retain the 32-bit version because this would allow customers to continue to
Chapter 5
5.2 Differences in the Exchange 2007 Store
use their existing hardware to move to Exchange 2007. Microsoft could have
made Exchange 2007 the last version to support 32-bit hardware and use it
to prepare the user community to move in the release that follows Exchange
2007. On the other hand, the next major release of Exchange might not
appear until 2010 and today’s server hardware is ready to run 64-bit Windows and Exchange, and indeed, it is difficult to buy servers that only run
32-bit Windows now. While the initial reaction of administrators to
Microsoft’s decision to support only 64-bit platforms with Exchange 2007
was to take a sharp breath, after everyone had the chance to think through
the issues a little more, the community concluded that Microsoft’s decision
was logical and broadly welcomed it.
Trading memory for I/O
For years, the Exchange development team has worried about the number of
I/O operations per second (IOPS) that users generate, especially on large
servers. One problem is that ESE for Exchange 2003 uses a 1GB cache that
cannot be extended on 32-bit systems.5 The Store maps its cache into virtual
memory and its objective is to prevent expensive disk activity by allowing the
Store to access commonly used and recently accessed data from in the cache
instead of having to go to disk. The limitations of 32-bit Windows restrict
Exchange 2003 to a maximum of 3GB of addressable user-mode virtual
address space, given that Windows takes about 1GB itself. In this respect, it
is interesting to look at an end-user application like Outlook, which will
cheerfully use as much memory as you care to throw at it (and many desktop
and laptop computers can provide 1GB for Outlook), while we support
thousands of Exchange users in an equivalent amount of memory on server
computers. Exchange 2003 does a reasonable job of managing the memory
that is available to it through code such as Dynamic Buffer Allocation, but
on heavily loaded systems there never seems to be enough memory to accommodate all demands.
The buffer manager, a component of the Store, manages the database
pages in the cache using an algorithm called Least Recently Used Reference
Count (LRU-K). ESE flushes pages from the cache if it calculates that the
pages have not been used recently, while it retains pages that processes
(users or internal Store processes) have accessed. Due to the relatively flat
nature of the page hierarchy within an ESE database, ESE needs no more
than four I/Os to navigate from the root page to the page that contains
data that you need. These I/Os are eliminated if the pages are in the cache,
so a large cache is a good thing.
The theoretical size of the cache on an Exchange 2003 server is 2.1GB. However, Microsoft recommends that you tune
the cache back to closer to 1GB (896MB)—see Microsoft Knowledge Base article 266768.
5.2 Differences in the Exchange 2007 Store
Another issue in the Exchange 2003 Store is that JET organizes data in a
series of 4KB pages and has done so since Exchange 4.0 shipped in 1996. At
that time, the average message size in corporate messaging systems was usually around 4KB, so the Store could pack entire messages into a single page.
Now, the average size is much larger (in some organizations, it has grown to
over 100KB), and it is impossible for the Store to hold any but a small percentage of messages in single pages. Even taking the message body alone (the
text of the message without attachments), because users often reply and
include previous text, the average Exchange 2003 message will not fit into a
single page. For example, if you send a message whose body text is more than
4KB, ESE cannot fit it into a single page in an Exchange 2003 database and
must therefore split the content across two pages—and those pages are not
necessarily contiguous within the database. Splits also occur for large attachments, so ESE must divide a 1MB PowerPoint attachment across 250 pages
on an Exchange 2003 server or 125 pages on an Exchange 2007 server.
Retrieving a large attachment therefore generates a lot of work for the Store
and if the data is not in the cache, it can generate many I/O operations. To
help address the problem of how to manage ever-increasing message sizes,
Exchange 2007 increases the Store page size from 4KB to 8KB. Microsoft
believes that the change will allow the Store to hold more than 50% of all
messages in a single page, with a subsequent decrease in the number of read
and write operations that the Store generates for each message. Changing the
page size is not an earth-shattering update for Exchange because other database applications (such as Oracle 9i and SQL Server) use this page size and
Microsoft already had plenty of experience with it in the Active Directory
database. In addition, because you cannot upgrade Exchange 2000 or 2003
servers to Exchange 2007, Microsoft does not have to convert existing databases to the new format.
While ESE attempts to locate data in the most efficient manner, it does
not help performance to have ever-increasingly large messages split across
multiple pages, and on Exchange 2003 servers, many pages in the database
are extensions for other pages. Because of the page splits, we end up with a
pattern of random access across the entire database, which means that cache
optimization techniques such as those implemented in RAM or storage controller caches, do not help as much as they could to speed access to Exchange
data. The overall result of the original 4KB page size is that the Exchange
2003 Store processes more I/O requests to read and write messages across
multiple pages. To compound matters, we know that messages are unlikely to
get smaller as attachments grow larger, users compose message bodies with
graphical editors like Word, and Microsoft starts to push unified messaging
technology so that we have voice messages in our mailbox as well. Because of
all these factors, we end up with larger mailboxes, larger databases, and a
huge demand for I/O, with the typical design figure for Exchange 2003
being 0.5 to 1.0 IOPS per concurrent active client. The problem with a
Chapter 5
5.2 Differences in the Exchange 2007 Store
growing demand for I/O is that while storage density has increased dramatically in the last decade so that 1TB disks are on the horizon, we have not seen
a similar increase in the ability to process I/Os.
All of this is fine if you only need to run a small server because current
server and disk technology can easily support several hundred Exchange
mailboxes at a very cost-effective price. However, once you start to scale up to
servers that support more than a thousand mailboxes, sheer storage capacity
is not the limiting factor or most costly item. Instead, the critical area
becomes the capability of the storage subsystem to process the I/O demand
generated by users. If the storage subsystem cannot handle the I/O demand,
then users see degraded performance because of bottlenecks in the Store. You
can certainly design and deploy storage subsystems that are capable of dealing with 4,000 concurrent users on an Exchange server, as long as you are
willing to spend a lot of money. The expense of providing and managing sufficient storage that can handle the I/O load generated by very large Exchange
servers is one of the reasons why Exchange 2003 “tops out” at around the
4,000 mailboxes per server mark. There are other important reasons that
limit the number of mailboxes that you want to host on a single server,
including the time required to perform backup and restore procedures, but
storage expense is a real constraint to building larger Exchange servers.
Microsoft is more than aware of the I/O problem, if only because they have
gone through a massive consolidation process for their own internal
Exchange deployment.
The interesting thing about moving to a 64-bit platform is that
Microsoft believes that the 64-bit platform holds the key to solving many of
the longstanding problems in the Exchange Store. With more addressable
memory, there should be far less of a problem with fragmentation of virtual
memory, so Exchange 2007 supports more storage groups and databases (see
Table 5.2). Email databases are big and getting bigger all the time and on the
32-bit platform the Store has to do a lot of processing to move data pages in
and out of its cache as users make requests, which then results in I/O operations as the Store fetches data from the database and commits page updates
from the cache. A larger cache means that more pages can stay longer in the
cache (database professionals say that there is less risk of “data eviction,”
which is a nice way of thinking about what’s going on). Reading a page from
disk to find the next item in a user’s inbox takes at least 1,000 times longer
than retrieving the data from memory, so Exchange becomes more responsive
to requests. The net is that the extended memory model achieved by running
Exchange on the 64-bit Windows platform allows the Store to cache far more
user data than it can today, with the result that Exchange 2007 generates (on
average) a quarter of the same I/O operations as an Exchange 2003 server
does for the same number of users.
5.2 Differences in the Exchange 2007 Store
Table 5.2
Storage group and databases in Exchange 2003 and 2007
Maximum Number of
Storage Groups
Maximum Number of
Maximum size of
Exchange 2003 Standard
2 (1 mailbox, 1 public
16GB, then 75GB
after SP2
Exchange 2003 Enterprise
20 (maximum of 5 per
storage group)
Limited only by hardware
Exchange 2007 Standard
5 (maximum of 5 databases
in a storage group)
Limited only by hardware*
Exchange 2007 Enterprise
50 (maximum of 5 databases per storage group)
Limited only by hardware
* While the database size is unlimited, Microsoft recommends that no database should exceed 100GB unless you use LCR/CCR,
in which case their recommended maximum is 200GB. In real life, there are many examples of production databases that exceed
250GB comfortably. It all depends on the degree of risk you are prepared to take.
As an example of what caching data means to a server, Microsoft calculated that if a Exchange could cache the contents of every user’s inbox, the
rules that typically process messages arriving into the inbox, and the most
current 50 messages, they could reduce the number of I/O operations significantly because the Store could then provide this data from memory instead
of constantly going to disk. One calculation from the Exchange engineering
group shows that the 32-bit JET cache can store 250KB per user on a server
that supports 4,000 mailboxes. By comparison, on a 64-bit Windows server
equipped with 16GB of RAM,6 the expanded ESE cache can hold 3.5MB
per user for the same number of mailboxes, a 14-times increase in the
amount of data that the Store can access from memory rather than having to
go to disk. Typical user access to Exchange concentrates on a reasonably
small number of folders such as the Inbox, Sent Items, and Calendar—something that is even truer for mobile clients. The ability to cache these folders
in memory will lead to much improved performance all round, no matter
what client you use. Note that the extended cache does not automatically
mean that the Store will generate 14 times fewer I/Os, but it is obviously a
good thing to trade expensive storage I/O for relatively cheap memory access.
Consider this fact: Accessing data through an I/O operation (disk read) usually takes something like 14 milliseconds and incurs some CPU cycles; reading a data page from memory takes 50 nanoseconds and no CPU. It’s an easy
trade-off to swap memory for disk I/Os.
User mode virtual address space on a 64-bit Windows server can theoretically grow as large as 8TB. However, the actual
space available to Exchange depends on the available RAM. Large Exchange 2007 servers will have lots of RAM, so they
will have large caches and so be better able to handle I/O loads by trading memory for I/O.
Chapter 5
5.2 Differences in the Exchange 2007 Store
I/O coalescing is the ability to group read or write operations to contiguous pages together so that you deal with everything in a single I/O operation rather than generating multiple I/Os. As we have discussed, optimizing
I/O operations is a major concern for Microsoft when they engineered
Exchange 2007. The Exchange 2003 version of the ESE engine coalesces up
to 64KB of data in a single I/O, while the additional size of the ESE cache
enables Exchange 2007 to handle up to 1MB (128 pages), which contributes
to a reduction in the IOPS requirement per user. Another change is an
increase in checkpoint depth, which governs the amount of dirty pages that
the Store maintains in its cache. Checkpointing is just a way to manage
changed pages efficiently until a database can commit the pages to disk.
Exchange 2007 uses a checkpoint depth of 20MB (of transaction logs) per
storage group. Making the checkpoint depth deeper increases the chance that
users will update these pages multiple times before the Store eventually
flushes them into the database and so allows ESE to combine multiple transactions into one write operation. Batching write operations allows Exchange
to eliminate some unnecessary writes and drives a further reduction in I/O
operations. For example, you would probably agree that you update your
inbox and calendar folders frequently. If the Store can cache the pages that
represent the contents of these folders (especially the pages that hold the top
of your inbox and this week’s appointments, which are highly likely to
change) and update the pages in memory rather than constantly going to
disk all the time, it’s easy to see how extra checkpoint depth can help performance. Multiply the effect by the number of concurrent users accessing a
server and you can realize that this can have a big impact. In passing, it is
worth noting that an application-specific cache, such as the one that
Exchange 2007 uses to optimize database writes, is very different to the
application-independent data cache that you find on intelligent storage controllers. Hardware caches take care of data from multiple applications and do
not attempt to optimize write activity at an application level.
All of the changes made to optimize the Exchange 2007 Store mean that
you will probably install more memory on an Exchange 2007 server than you
will for previous versions. While it is too early to say just how much physical
memory you need to install on an Exchange 2007 server to achieve optimum
performance, Microsoft provides guidance of 5MB per active concurrent
user plus 2GB for Exchange, so a server supporting 2,000 concurrent users
needs 10GB for the users plus 2GB = 12GB. As this is merely guidance, you
should discuss memory requirements with an Exchange performance expert
to arrive at accurate memory sizing for any particular server. Along the same
lines, it will take time to test systems to get real production data to verify
exactly what impact 64-bit Exchange makes on IOPS. Educated feedback
from those involved with Microsoft engineering put the figure at between
0.3 and 0.25 IOPS/user, according to data gathered from some of Microsoft’s
own systems7. These systems had been operating at 1 IOPS/user on a 32-bit
5.2 Differences in the Exchange 2007 Store
platform, so these results are very positive. After all, if you can reduce the I/O
demand to a third or quarter of that current for Exchange 2003, the theory is
that you can reduce your storage costs. It is interesting to note that larger
mailboxes increase the number of I/Os generated per mailbox for clients that
do not operate in cached Exchange mode, which is a good reason to insist
that all Outlook clients are configured to use cached Exchange mode if you
want to use large mailbox quotas. The increase in I/O demand is likely to be
due to Exchange having to cope with more data in the mailbox (more folders
and more items in each folder), more user activities (if a user can store more
in their mailbox, they will do things like subscribe to online mailing lists),
and the side-effect of having more items in mailboxes such as content indexing. Benchmarks that seek to test I/O performance may be based on small
mailboxes that contain relatively few items; a different I/O pattern is likely
when your users behave like human packrats and each has a mailbox containing 200 folders with up to 5,000 items per folder.
Experience with early deployments demonstrates that it is difficult to
achieve the full amount of theoretical cost savings in storage that better I/O
performance in Exchange 2007 might enable. You are hardly likely to jettison all your existing storage and replace it with lower cost alternatives.
Instead, you will keep the same storage infrastructure and if the I/O demand
drops, then you may be able use the storage to host more servers or to handle
the increased workload of new applications, such as unified messaging. Consolidation and extended storage lifetime rather than the installation of
cheaper storage solutions are where the real cost savings will be produced
through the changes that Microsoft are making in Exchange 2007.
The decrease in storage costs
IDC predicts that the storage industry will ship 15 exabytes of storage in
2006–2008. To put this amount into perspective, it is more than the industry has shipped in aggregate since it started. Servers have more storage than
ever before because users want to store more data than ever before and the
decrease in storage costs has allowed administrators to configure servers with
more and more storage.
Exchange 2007 trades memory for I/O operations to drive down the
overall IOPS figure for a server. While you may have to provide more storage to satisfy user demands, you may be able to take advantage of cheaper
storage subsystems to support larger servers. With Exchange 2003, if you
want to deploy large Exchange servers that support thousands of mailboxes,
you incur much of the capital cost for the project to buy the necessary stor-
Your mileage will vary from Microsoft’s experience depending on user performance characteristics and the other software
that is active on the server alongside Exchange. For example, anti-virus software can exert a significant performance load
on a server.
Chapter 5
5.3 No more streaming database
age technology that can supply the I/O requirements to handle the desired
number of concurrently active mailboxes. The fact is that while storage
technology has seen huge advances in capacity (measured in both cost per
GB and the average size of disks—500GB or larger disks are available now)
since 1996, storage technology has failed to drive performance at anything
like the same rate. The disks that we have today are bigger and less prone to
failure, but their I/O performance is not much better than those that we
used ten years ago. The upshot is that if you want to provide lots of I/O
capacity, you have to invest in upscale storage technology, and that comes
with a big price tag.
In some cases, the costs of the storage for large deployments is much
higher than the servers, so much so that Microsoft says that 80% of the hardware costs of many Exchange deployments is due to storage. Of course, costs
vary greatly from deployment to deployment, but there is no doubt that
SANs are expensive to deploy and operate. They are more complex, they
require more expertise, but they offer more features and functionality than
simple direct attached storage does. The question for administrators is
whether you want to trade functionality, performance, and complexity for
cost and simplicity. Other factors such as resilience to failure also come into
the equation.
Any step that eliminates cost for an implementation is welcome and
companies will be interested in investigating how they can take advantage of
Exchange 2007’s reduced I/O demands to reduce costs. However, I do not
imagine that companies that operate SANs for Exchange 2003 will dump
their SANs and rush to deploy low-cost storage for Exchange 2007 as the
need still exists to deliver a balance of cost versus robustness and resilience for
production environments.
No more streaming database
Up to Exchange 2000, the product only used the EDB or MAPI property
database. Exchange 2000 introduced the concept of the STM file, or streaming database. Exchange 2000 also introduced ExIFS, the Exchange Installable File System, which it used to access data in the STM. The idea behind a
streaming database was very logical. Up to that point, the vast majority of
content generated by Exchange users was through MAPI clients and was very
suited to storage as MAPI properties in the EDB. For example, the title of a
message is a MAPI property, as is its author, the recipients, the message body,
and even pointers to any attachments that the message might have.
Exchange 5.5 was the first version of Exchange to support a wide array
of clients that preferred Internet protocols such as POP3, IMAP4, SMTP,
and MIME, so Exchange had to do an increasing amount of work to convert
these protocols into MAPI properties before the Store was able to accept the
5.3 No more streaming database
data. Microsoft therefore decided that they would give the Store the ability to
handle Internet protocols in their native format and that the Store would do
no conversion unless it was necessary, such as in the case when an Outlook
client requested a message that an IMAP4 client had originally created and
sent. MAPI clients continued to use the Store and the EDB database in
much the same way that they always had, but messages generated in Internet
formats by IMAP4 and POP3 clients went into the STM file, with just a few
basic MAPI properties (such as message subject and author) being “promoted” into the EDB. On Exchange 2000 and 2003 servers, every EDB
database has a partner STM file and the Store provides the software layer to
mask the complexity of format translation and the different storage paths.
The implementation of the dual database was fine and did not cause any
problems, but the advantages of increased processing power and memory
availability generated by the move to the 64-bit platform for Exchange
allowed Microsoft to review the situation to decide whether Exchange still
required the STM file.
The engineering answer was that Exchange no longer requires the STM
file as the upgraded Store is capable of handling Internet formats as easily as
it handles MAPI. In addition, the power of current hardware platforms
means that extra CPU cycles are available to handle the processing load of
format conversion. The major advantage gained by the elimination of the
STM file is simplification of the file structure within the Store. This advantage is not apparent if you have a small server with just one database, but it is
clearer on larger servers that support 20 databases or more. As you cannot
upgrade an Exchange 2003 server to Exchange 2007, you never see an STM
file on an Exchange 2007 server. The move mailbox operations that transfer
content from older Exchange servers to Exchange 2007 mailboxes takes care
of the data transformation from STM to EDB format en route. A consequence of the elimination of the STM is that Microsoft has also retired
ExIFS. This decision is a good one because ExIFS never delivered on the
promise that it (or Microsoft developers) held out to provide fast access to
data in the Store for applications, including the ability to map mailbox data
through the DOS prompt, something that became a favorite party trick at
Microsoft demonstrations. Some administrators took Microsoft at their word
and attempted to backup Exchange databases through ExIFS only to find
that while backups occurred, they did not contain information that
Exchange could use to restore a database.
One potential drawback of the elimination of the STM file is the
requirement for a high-performance TEMP location that the Store can use
to convert MAPI content to native Internet content for consumption by
IMAP4 or POP3 clients. This is only a problem if you have a server that
supports large populations of these clients and servers that support a predominantly Outlook client base do not have to concern themselves about
the issue.
Chapter 5
5.4 Tables and items
Tables and items
When a client makes an authenticated connection to a mailbox in the Store,
it builds a hierarchical representation of all of the folders in the mailbox from
the tables that hold the data. The major tables that clients interact with to
read and send messages are:
Table 5.3
Folder ID
The mailbox table: Each row in the table represents a mailbox.
The folders table: Each row represents a folder in a mailbox and each
folder has a unique identifier. One table holds details of every folder
in the database. A parent/child scheme supports nested folders so that
a subfolder maintains a pointer to its parent folder. For example, in
Table 5.3, the Articles folder in TonyR’s mailbox is a child folder of
the Publicity folder and the Exchange folder is a child folder of the
Microsoft folder. The count of new items in the table allows a client
to bold the folder name to give the user a visual indication that there
is new content to be read in the folder.
The message table: Each row represents a message in a folder. Again,
one table holds every message in a database.
The attachments table: Each row represents an attachment to another
A set of message/folder tables: The Store holds a separate message/
folder table for every folder in a database.
Sample contents of the Folders table in a mailbox database
Folder Name
Folder Owner
Count of items
Count of
New Items
Parent Folder ID
Deleted Items
5.4 Tables and items
Public folder databases also organize into folders and rows. However, the
major client interaction is with mailboxes, so this is what we investigate here.
Pointers link data in different folders together. The use of pointers is the
basis of the single instance storage model, which means that the Store holds
only a single copy of any unique content (such as a message body or an
attachment) and uses pointers to link all of the mailboxes that have an interest in the content. The benefit is that you do not have to duplicate content
across multiple mailboxes. The Store manages the pointers so that as users
delete items from their mailboxes, the Store reduces the “usage count” for
pointers to a piece of content until the content is no longer required. At this
point, the Store erases the content from the database. Single instance storage
was a big feature for the first generation of Exchange but its importance has
decreased since Exchange 2000 introduced support for multiple databases on
a server. Obviously, if Exchange delivers a message to multiple mailboxes that
exist in different databases, it must create the content in each database, and
so eliminating some of the benefit of the single-instance storage model.
Table 5.4
Contents of a message/folder table
Kevin Laahs
Exchange 2007
Training for
New cafeteria
Ken Ewert
LCS is great!
New brand
The Store holds header information for each folder in a separate table,
called a message/folder table (Table 5.4). The header information is composed of a set of MAPI properties, such as Size, From, Received, and so on.
Maintaining this information in a separate table allows the Store to support a
separate sort order or view for each folder. When a client requests information about a folder from the Store, they read information from the message/
folder table.
The message identifier in a row in a message/folder table links the message properties with the content, which the Store holds in the Message table
(Table 5.5). Note that the message body content is always in RTF (Rich Text
Format), even if you use Word as your message editor.
Chapter 5
5.4 Tables and items
Table 5.5
Contents of the Message table
Message Body Content
Tony Redmond
We have just started to train
Tony Redmond; Kevin
Training for administrators
Tony Redmond
LCS is a new way of contacting…
All employees
Our new brand is ready for
Figure 5.2 illustrates the interaction between the different tables in the
Store to populate the Outlook user interface. The folder tree, including
nested folders, comes from data in the folder table. The messages in the
folder displayed in the right hand pane come from the data in the message/
folder table for the folder (in this case, the folder is “Tony’s Inbox”). When
the user clicks on a message to read it, Outlook combines the message header
data with the content, including an XLS attachment, to build the message
form that the user sees.
Figure 5.2
How Outlook
combines data from
different Store
tables into its UI
5.5 Storage groups
Storage groups
Up to Exchange 2000, administrators dealt with a single mailbox database
and a single public folder database per server. Exchange 2000 introduced the
concept of storage groups, or an instance of the database engine running
within the Information Store process that can manage up to five databases
within the group, all of whom share a single transaction log set. With the
Enterprise Edition of Exchange 2000 and 2003, you can deploy up to 4 storage groups on a server, or a maximum of 20 databases.
Microsoft’s original design for Exchange 2000 envisaged that a server
could support many more databases, but memory restrictions forced the
developers to limit Exchange to 4 storage groups. As we have seen, the additional memory that is available for the Windows 64-bit platform has eased
this restriction quite considerably to a point where Exchange 2007 now supports up to 50 storage groups per server, each with its accompanying set of
transaction logs. Each storage group occupies some virtual memory. Apart
from the obvious issue of how to manage large numbers of storage groups
especially in terms of the number of storage LUNs required to accommodate the groups, there is no technical reason why Exchange 2007 could not
have supported 100 storage groups or more. After all, the headroom made
available by the move to the 64-bit platform makes a lot more memory
available to the Store, and it could have used the memory to accommodate
more storage groups. In short, Microsoft settled on 50 as a reasonable compromise between manageability and functionality, but the architecture
allows them to increase the number of supported storage groups in a future
version of Exchange, if demand justifies such a change.
It is still possible to allocate multiple databases to a storage group, but
experience with Exchange 2003, especially in terms of easier disaster recovery, demonstrates that it is better to deploy fewer databases per storage
group and Microsoft’s best practice is to deploy one database per storage
group. Some mailbox servers in organizations that support legacy servers
will continue to support a public folder database in one of the storage
groups on the server.
The situation was not always so simple. Immediately Exchange 2000
introduced the ability to support multiple databases divided across multiple
storage groups. The debate erupted as to which was the better approach: to
deploy multiple databases in a single storage group, or to spread the databases
across multiple storage groups. Memory management overhead and some
bugs in Exchange 2000 meant that Microsoft’s recommendation at that time
was to avoid creating more storage groups than you absolutely needed.
Exchange 2003 fixed the software problems and Exchange 2007 provides
even more headroom so now you can select whatever option for storage
group deployment that seems right to you.
Chapter 5
5.5 Storage groups
The debate still rages and it is hard to set any hard-and-fast rules because
every deployment differs in its needs and constraints. Here are some points
to take into account as you consider how to deploy storage groups within
your organization.
Adding complexity to a deployment is always a bad thing unless it is
necessary. For example, CCR and LCR require storage groups that
host just one mailbox database, so if you decide to deploy continuous
log replication, you are more likely to deploy additional storage
groups. Extra storage groups mean extra sets of transaction logs to
manage and deploy. In particular, you have to ensure that you deploy
the logs and databases across available storage in such a way that user
activity does not generate excessive I/O to any spindle and that a failure will not compromise more than one storage group.
Splitting user load across multiple databases usually results in smaller
databases, which are easier to manage, especially if you have to restore
from tape after a failure. Plan on a restore speed of 20GB/hour to
establish the SLA that you can meet; you will probably do better in
practice as tape technology improves, but it is good to know the
worst case. However, the availability of Volume ShadowCopy Services and support for hot snapshots means that it is easier to restore
very large databases, so the database size may not be a concern to you
if you deploy the necessary technology to support hot snapshots.
The more databases you deploy, the fewer the users affected by any
database failure unless hardware causes the failure and affects multiple databases. You can compare supporting all users in one or two
very large databases to putting all of one’s eggs into a single basket.
Everything is okay until a database failure.
User mailbox quotas are growing, so the size of your databases today
will be larger in the future. Plan on 300% growth over three years
and you will not go far wrong, and then use that figure as the basis for
Sometimes, depending on the storage technology that you use, you
simply have to spread the I/O load across multiple streams, including
multiple transaction logs. In this case, you have to deploy multiple
storage groups as each storage group has its own transaction log set.
Having additional transaction log sets should not increase the time
required to replay transactions after a failure as Exchange generates
roughly the same number of transaction logs whether you have one
or four storage groups. There is some additional complexity in restoring transaction logs, but only if you have to restore logs for several
storage groups.
5.5 Storage groups
Some people believe that Exchange can take a long time to replay
transaction logs in case of failure because the logs contain interleaved
transactions from all of the databases in the storage group; restore
performance is therefore faster if the transaction logs only contain
data for one database. This was certainly the case for Exchange 2000,
but from Exchange 2003 SP1 onwards log replay is rapid—the
improvement went from a minute per log to 10–15 logs per minute
on many systems.
The backup and restore solution that you deploy may influence how
you use storage groups. For example, if you deploy a VSS-based solution, you have to dismount all of the databases in a storage group to
perform a restore, even if you want only to restore a single database.
This is a significant difference when compared to how traditional
streaming backup and restore programs work.
Different people have different views on the best approach to take to
storage groups, mostly based on their own experience. In the case of
Exchange 2007, we can now configure many more storage groups on a server,
so the debate is different. For example, consider the situation where you have
an Exchange 2003 server that supports ten mailbox databases of up to 80GB
each. You have 500 users assigned to each database, so the server supports
5,000 users. In this scenario, you have to manage the databases in four storage groups, so two of the storage groups have three databases and the other
two support two databases. You also have four sets of transaction logs to
manage. On an Exchange 2007 server, you could assign each database to its
own storage group, or you can opt to create 20 storage groups and have one
40GB database in each group, or 50 storage groups that each has a 20GB
database, or indeed 20 storage groups where each storage group supports two
20GB databases. The design decision is whether to shrink the database size
by creating more storage groups and if so, at what point the number of storage groups (and their associated transaction log sets) becomes unmanageable.
Some argue that it makes perfect sense to limit each storage group to one
database because you then have isolation of all data belonging to the database
in the storage group, while others believe that Exchange manages interleaved
transactions in the logs well enough to support multiple databases in each
storage group.
Another way to look at storage groups is to determine how many mailboxes of what size you want to deploy and then to calculate how many storage
groups you need to create to support the mailboxes. We can include the maximum reasonable size for a database in the equation (as determined by how
large you are willing to let a database grow in terms of backup times).
Microsoft now recommends that you configure storage groups with just one
database, so that is another factor to take into account. Table 5.6 shows a
Chapter 5
5.5 Storage groups
rough calculation of the number of single-database storage groups required to
support different sizes of mailboxes for different user communities using databases of different sizes. It is no surprise to see that the larger the mailbox, the
fewer mailboxes we can support in a database, and the more storage groups we
need. What may be a surprise is how quickly we get into a situation where the
server has to support twenty or more storage groups. Microsoft’s white paper
covering their internal deployment of Exchange 2007 mentions that they
configure their “mailbox I” server type with 28 storage groups and their “mailbox II” server type with 42 storage groups, which backs up the assertion that
we will see a big increase in the number of storage groups as companies deploy
Exchange 2007.
Table 5.6
Storage Groups and mailbox sizes
Maximum Database Size in Storage Group
1,000 × 250MB
2,000 × 250MB
3,000 × 250MB
4,000 × 250MB
1,000 × 500MB
2,000 × 500MB
4,000 × 500MB
1,000 × 1GB
2,000 × 1GB
4,000 × 1GB
1,000 × 2GB
2,000 × 2GB
4,000 × 2GB
Remember that an Exchange 2007 server can support up to 50 storage
groups. Looking at our table, we therefore conclude that we have to be prepared to let databases grow to 100GB or more if we want to offer even 1GB
mailboxes to users. We can also see that the total number of storage groups
requires exceeds the maximum supported by Exchange as we go past 2,000
mailboxes on the server. The other point of concern is the number of disks
that are required to support so many storage groups. We can use mount
5.5 Storage groups
points to get past the disk letter limitation, but the prospect of managing so
many disk locations for databases and transaction logs is not attractive. Of
course, we can configure storage groups to host multiple databases, but then
we cannot use continuous log replication. What you gain on the swings, you
lose on the roundabouts.
Optimizing disk I/O is another factor to take into the equation.
Exchange 2007 trades memory for I/O and has a radically different I/O profile than Exchange 2003. If you use a single database per storage group then
the full transaction log checkpoint depth (the size of the in-memory database
cache) is available to that database. The Store therefore has a higher likelihood
that it will find a page to update in the cache where it holds dirty database
pages because the pages stay in the cache until the checkpoint forces the Store
to write the pages into the database. If multiple updates can be applied to a
page in memory, the Store needs only to write the last page update to the
database and so generates a single I/O instead of the multiple I/Os that it
would otherwise need if it were forced to commit more updates to disk to
accommodate the transactions for multiple databases.
We have a lot of experience of multi-database storage groups from operating Exchange 2000 and 2003, but there is no doubt that Microsoft is heading to a position where one database per storage group becomes the rule. The
challenge is to balance your database storage requirements with management
and to deploy enough storage groups to support users without going completely overboard.
Creating a new storage group and database
Creating a new storage group is easy, but a little bit of planning goes a long
way. The first thing to consider is the disk location where you will put the
storage group. Best practice is to:
Keep the transaction logs and database files for a storage group on
separate disks so that a failure on one disk will not compromise all of
the files.
If a server supports multiple storage groups, put the storage groups
on different disks.
Keep storage groups away from drive C: whenever possible. In most
cases, the default location suggested by Exchange is under the \Program Files\Microsoft\Exchange Server\Mailbox directory, so you
need to move away from here.
Chapter 5
5.5 Storage groups
Other considerations also come into play when you plan the position of
a storage group, such as the need to balance I/O operations across available
disks, the overall configuration for SAN deployments, the requirements of
other applications, and so on. The key point is for you to understand where
you want to put the storage group before you start rather than making a key
decision halfway through its creation. You can move a storage group after it is
created, but it is human nature to leave things alone, which is a good enough
reason to get things right from the start.
Figure 5.3
Storage group and
database files
immediately after
After you decide on the physical location for the storage group, you
should create whatever directories are required to hold the transaction logs,
system files, and databases. It is a good idea to use a consistent naming
standard for this purpose. For example, you could use a simple convention
of naming the directory “SG” plus a number, as in “SG1,” “SG2,” “SG3,”
and so on. If you think it a good idea to make an explicit reference to
Exchange to make it obvious what application uses the directory, you
might use Exchange as a prefix, as in “Exchange SG1” and “Exchange
SG2.” Some administrators prefer to include the Exchange server name in
the names for storage groups and databases, as they believe this to be useful
if they ever have to mount databases in the Recovery Storage Group and
because it makes the server that owns the databases obvious when you work
with many databases through shell commands. This naming convention
results in names such as “ExchMbxSvr1-SG1” and “ExchMbxSvr1-SG1-
5.5 Storage groups
DB1.” Selecting the right naming convention is largely a matter of what
works for you and is often influenced by legacy servers and databases, but it
is a good thing to consider as you create new Exchange 2007 servers, storage groups, and databases.
Under the root directory for the storage group, it is a good idea to create
a set of subdirectories to hold the transaction logs and databases. Figure 5.3
illustrates a typical on-disk structure for a storage group containing a single
mailbox database. This situation is immediately after Exchange has created
the database, so you can see the initial set of transaction logs in the \Logs
directory and the newly created database in the \Data directory.
Once you know where all the files are to go, creating a new storage
group and adding a database is easy with the EMC wizard. It is more interesting to look at the shell commands that you can use to script the creation of
a storage group and a mailbox database. First, you use the New-StorageGroup
command to create the storage group:
New-StorageGroup -Server ‘ExchMbxSvr2’ -Name 'Critical
-LogFolderPath 'D:\Exchange SG4\Logs'
-SystemFolderPath 'D:\Exchange SG4\Data'
With the storage group in place, we can create a new mailbox database
with the New-MailboxDatabase command.
New-MailboxDatabase -StorageGroup ‘ExchMbxSvr2\Critical
-Name 'VIP Mailboxes'
-EdbFilePath 'D:\Exchange SG4\Data\VIP Mailboxes.edb'
If we want to use local continuous replication for the database (and
it is the only database in the storage group), then we would also specify
the –HasLocalCopy and –CopyEdbFilePath parameters. See Chapter 9 for
more information on local continuous replication. If you want to create a
public folder database instead of a mailbox database, you need to specify the
–PublicFolderDatabase parameter.
Working with storage groups and databases
After you create a new mailbox database, you can mount it with the MountDatabase command:
Chapter 5
5.5 Storage groups
Mount-Database -Identity ‘Critical Mailboxes\VIP Mailboxes’
The storage group and mailbox database are now ready for you to populate with mailboxes. You can check its status with the Get-MailboxDatabase
Get-MailboxDatabase –id ‘Critical Mailboxes\VIP Mailboxes’
If you are unhappy with a property of a database, you can change it with
the Set-MailboxDatabase command. For example:
Set-MailboxDatabase –id ‘Critical Mailboxes\VIP Mailboxes’
–ProhibitSendQuota 1995MB
If you are completely unhappy with the database, you can dismount it
and then delete it.
Dismount-Database –id ‘Critical Mailboxes\VIP Mailboxes’
Remove-MailboxDatabase –id ‘Critical Mailboxes\VIP Mailboxes’
The apparent inconsistency in the naming of commands that work with
databases is because the Mount-Database and Dismount-Database commands work with mailbox and public folder databases. There is a separate
Remove-PublicFolderDatabase command to delete public folder databases,
just like there are commands to create (New-PublicFolderDatabase), and
fetch and set properties (Get-PublicFolderDatabase and Set-PublicFolderDatabase). In any case, after you delete a database, you have to complete
the clean-up operation by deleting the physical files on disk.
Of course, after we delete the database, we may want to clean up completely by deleting the storage group. We can do this with the RemoveStorageGroup command:
Remove-StorageGroup –id ‘ExchMbxSvr2\Critical Mailboxes’
Other notable commands that you may want to use in scripts include
Move-DatabasePath, which moves database files from one disk location to
another, and the Clean-MailboxDatabase command, which checks a mail-
box database for any disconnected mailboxes that exist in the database that
are not properly marked. The command then corrects their status. A discon-
5.6 Transaction logs
nected mailbox is a mailbox that exists in a database but does not have an
associated Active Directory account. You can connect a disconnected mailbox to an Active Directory account with the Connect-Mailbox command.
The Move-StorageGroupPath command is available if you need to move the
location of transaction logs or system files.
Transaction logs
Exchange has always featured transaction logs as the method that the Store
uses to capture transaction data before the Store commits transactions to its
database. Transaction logs capture all transactions—new items, modifications
to existing items, and deletions—for both mailboxes and public folders.
Because of the asynchronous way that the Store saves transactions to the log
while batching transactions for efficient committal to the database, it is
entirely possible for users to read and write messages in memory without the
Store ever going to disk to fetch data. Fast, efficient, and secure access to data
is the major advantage delivered by the write-ahead logging model used by
Exchange, and it is important that every administrator understands how the
model works.
The Store never commits an item to a database unless it first saves the
data to a transaction log. This means that until the Store commits data to a
database, the only durable place that the data exists is in the transaction log.
Of course, the data is in the Store’s cache, but the cache can quickly disappear through hardware failure. The major function of the transaction logs is
therefore to allow the Store to use the data in the logs to replay transactions
and so preserve data by rebuilding a database from the combination of a
backup copy and the transaction logs. Obviously, this is only possible if you
take care to preserve the transaction logs in the same way that you take regular backups of the databases.
Every time the Information Store service starts up, the Store automatically checks the databases as it mounts them to verify that the databases are
consistent. A flag in the database header indicates whether the database is
consistent or inconsistent, depending on whether the Store was able to shutdown the database cleanly the last time it shutdown. A clean shutdown
flushes all data from the Store’s cache and commits any outstanding transactions into the database to make the database consistent. An inconsistent
database is one that has some outstanding transactions that the Store has not
yet committed. If the Store detects that a database is inconsistent, it attempts
to read the outstanding transactions from the transaction logs to replay the
transactions into the database and so make the database consistent. This
operation is referred to as a soft recovery and while it happens less frequently
today than with earlier versions of Exchange, soft recoveries occur from time
to time, usually after a software or hardware failure has caused Exchange to
Chapter 5
5.6 Transaction logs
terminate abruptly. In most instances, you are unaware that the Store has
performed a soft recovery and you will not find out unless you check the
event log to see whether the Store replayed any transactions after it mounted
a database. If you are unsure whether a database is consistent, you can run
the ESEUTIL utility with the /MH parameter to check the database header.
The only way that you can be sure that a database is consistent is to perform a controlled shutdown of the Information Store service (which makes
all databases consistent) or to use ESM to dismount a specific database. At
this point, the Store makes sure that it commits any outstanding transactions
that exist within its cache before it shuts down the database. Taking a full
backup creates a somewhat similar situation in that the database after the
backup is consistent. However, because the database is still online, the Store
will continue to commit transactions immediately after the backup finishes,
so the database is only consistent for a very short time.
ESE deals with transaction logs as if they formed one very large logical
log, which the Store divides into a set of generations to be more convenient
to manage. On Exchange 2000/2003 servers, transaction logs are 5MB
whereas they are 1MB on Exchange 2007 servers. Each log represents a single generation that ESE assigns a sequential generation number to for identification purposes. Obviously, a single message that has a number of large
attachments can easily span many log files. ESE manages the split of data
across the logs automatically and is also able to retrieve data from several
logs to form the single message and its attachment if the need arises to
replay the transaction into a database. On a busy server, millions of transactions might flow through the logs daily, and it is common to see hundreds if
not thousands of logs created each day. Apart from the activity generated by
users, transaction logs capture background and maintenance activity such as
the generation of NDRs, mailbox moves, the import of mailbox data from
other messaging systems, and so on. Any operation that causes data to flow
in and out of a database is captured in a transaction log. If you want to estimate the number of transaction logs that a server will generate daily, you
can use the rule of thumb of five 5MB of data (or five logs) per active user
per eight-hour working day. This rule assumes that you remove transaction
logs through daily full backups and has stood the test of time. You will not
go far wrong if you use it.
Each storage group has its own set of transaction logs and the Store
interleaves the transactions for all of the databases in the storage group
within the transaction logs. Thus, the first transaction in a log may belong to
mailbox database 1, the next belongs to mailbox database 2, the next to mailbox database 1, and the next to mailbox database 3. You cannot separate out
the transactions for one database from a transaction log—the Store does this
automatically when it needs to replay transactions into a database. Transaction logs are tied to their storage groups in two ways. First, ESE writes a
5.6 Transaction logs
unique identifier (the signature) into each log as it creates the file. The log
identifier must match the signature of the storage group before ESE can use
the contents of a log to recover transactions. Second, ESE records the path to
the directory where the database is located in the transaction logs. You can
find information about identifiers and locations by running the ESEUTIL
utility with the /ML parameter to dump the header information from a
transaction log as shown in Figure 5.4.
Initiating FILE DUMP mode...
Figure 5.4
Output from
dump of
log header
Base name: e01
Log file: c:\temp\e010000aa15.log
lGeneration: 43541 (0xAA15)
creation time: 01/03/2007 08:50:58
prev gen time: 01/03/2007 08:50:57
Format LGVersion: (7.3704.10)
Engine LGVersion: (7.3704.10)
Signature: Create time:12/01/2006 21:00:37 Rand:85169237 Computer:
Env SystemPath: H:\SG2-Logs\
Env LogFilePath: H:\SG2-Logs\
Env Log Sec size: 512
852, 42600, 24960, 42600,
Using Reserved Log File: false
Circular Logging Flag (current file): off
Circular Logging Flag (past files): off
1 H:\Data\DB2\DB2.edb
dbtime: 105576739 (0-105576739)
objidLast: 161066
Signature: Create time:12/01/2006 21:00:37
Rand:85161050 Computer:
MaxDbSize: 0 pages
Last Attach: (0x1B53,9,86)
Last Consistent: (0x1B51,6B8,149)
Last Lgpos: (0xaa15,7FF,0)
Integrity check passed for log file: c:\temp\e010000aa15.log
The Store creates file names for transaction logs using a naming scheme
that combines the prefix for the storage group with a sequential log number
in hex, so you can have over a million log files before the Store has to reuse
file names within a storage group. For example, E00 is the prefix of the
default storage group. The current transaction log for the default storage
group is E00.LOG. Exchange 2007 changes the naming convention for log
files. Exchange 2000 and 2003 use an eight-character name composed of the
storage group prefix and the log generation number, resulting in names such
as E0007E71.log. On an Exchange 2007 server, the names are ten characters
Chapter 5
5.6 Transaction logs
long (such as E00000002A.log). The difference is accounted by the requirement to avoid the potential that log file names are recycled and overwritten.
In reality, a low potential for this situation existed beforehand, but it did
increase when the number of log files expanded by five due to the reduction
in size from 5MB to 1MB. The extra characters allow Exchange to create
over four billion logs before any possibility exists for name reuse. Figure 5.5
illustrates the different file size and naming convention used by Exchange
2007 (left) and Exchange 2003 (right).
Figure 5.5
Transaction logs
from Exchange
2007 and
Exchange 2003
When you create a new storage group, ESE allocates it a separate prefix
to use for transaction log naming (you cannot assign your own prefix), so the
second storage group uses E01, the third E02, and so on. Figure 5.6 shows
that the log file prefix for the “LCR Storage Group” is E02, so we know that
this is the third storage group created on the server. Note that if you create a
storage group and then remove it, Exchange may not reuse its log file prefix.
Assigning a different prefix to the logs for each storage group allows you to
keep all the transaction logs from every storage group in the same directory
and avoid any problems that might occur if the storage groups used the same
naming convention. However, cluttering up a single directory with thousands of log files is not a good idea. It is best practice to place the transaction
logs for each storage group into a separate directory.
Despite the growing size of messages and message volume, Exchange log
files have been 5MB since Exchange 4.0. Microsoft changed the log file size
downwards to 1MB for Exchange 2007 to accommodate the requirements of
log file shipping and the continuous replication features that are now available. In particular, to ensure that any data divergence that might occur as a
result of an interruption in log replication can be quickly resolved and that it
is less likely that data will be lost if a log file cannot be copied or is corrupted
through hardware or software failure. Any variation from the expected file
size is an indication that the file might be corrupt for some reason. If you see
unexpected log sizes, you should stop the Information Store service and
check the event log, database, and disks for errors.
In all cases, you should take care to place the transaction logs on separate
physical disk devices to their databases (or two disk logical units on a SAN).
Apart from the performance issue caused if too much I/O goes to one device,
you need to ensure that you lay out important Exchange files in a way that
5.6 Transaction logs
you create maximum resilience to failure. You can have an accident and lose a
database through disk corruption or an administrative failure, and if so, you
can proceed to recover the database from a backup and replay all the transactions from the log files. However, if you lose a database and then find that
you cannot recover the transactions, users end up by losing data and you may
end up by losing your job. With this in mind, be sure to maintain clear separation in storage terms between databases and their transaction logs.
Figure 5.6
Storage group log
file prefix
Circular logging
The Store supports the concept of circular logging, which means that instead
of creating a new transaction log for every 1MB of transaction data, the Store
reuses a set of transaction logs to save disk space. In other words, as the Store
commits all of the transactions in a log file to the database, that transaction
log file becomes a candidate for reuse. Under anything other than heavy load,
the Store uses a set of transaction logs. However, if the server or the Store
comes under heavy load, the Store may not be able to commit transactions as
quickly as it normally does into the database, and in these circumstances, the
Store may create and use additional log files. In all cases, the Store only uses a
limited number of logs, so you can be sure that the server will never run out
of disk space because of log usage.
Chapter 5
5.6 Transaction logs
Because all of the databases in a storage group share a common set of
transaction logs, you cannot set circular logging for a specific database.
Instead, you enable circular logging by selecting a storage group and updating its properties as shown in Figure 5.7. You can also effect the change with
the Set-StorageGroup command:
Set-StorageGroup –id ‘ExchMbxSvr1\First Storage Group’
–CircularLoggingEnabled $True
The reason why circular logging exists is to restrict the amount of disk
space that the Store consumes to hold transactions. This reason was very
important in the time when storage was expensive and disks were small. On a
server running Exchange 4.0 or 5.0 in 1996, the prospect of running out of
disk space was very real and administrators paid a lot of attention to services
that consumed storage because if less than 10MB is available on the disk that
hosts transaction logs, the Store service shuts down. However, given that
most server computers have much larger amounts of disk space available
today, the need to keep such a close eye on available disk space is not as necessary. Exchange does a good job of removing obsolete transaction logs if you
take regular backups and most companies assign the necessary disk space
(with a good buffer) to handle the storage of at least three days worth of
transaction logs just in case backups do not happen. Experience of Exchange
in production proves that the prospect of losing valuable data through the
unavailability of a transaction log and being unable to restore a storage group
completely after a disk outage is a more frequent occurrence than exhausting
space on the disk that hosts transaction logs.
This does not mean that you do not have to monitor the space consumed by transaction logs because there are occasions when the Store can
generate many more logs than normal. For example, if you decide to move a
large amount of mailbox data from one server to another, the act of moving
the mailboxes generates transactions on both the source and target servers
(usually a transaction log for every megabyte of mailbox data that moves),
resulting in the generation of a large number of logs on both computers. For
this reason, it is a good idea to take a full backup of both computers after the
move is complete to remove the transaction logs and to ensure that you can
restore the servers to a postmove state, just in case a problem occurs.
The golden rule is never to enable circular logging for a storage group
on anything other than a test server or a storage group that holds transient
data that you can afford to lose. For example, you can safely enable circular
logging on a storage group on a front-end server that does not host mailboxes. Never use circular logging for a mailbox store unless it hosts only
test mailboxes.
5.6 Transaction logs
Figure 5.7
Enabling circular
Creating new transaction logs
If you enable circular logging for a server, the Store uses a small set of transaction logs to capture database transactions. The set varies in number depending on the transactional load on the server, but it is usually between four and
six logs that the Store switches in and out to become the current log as it
commits transactions into the database. Circular logging is not a good idea
on a production mailbox server but is common on test servers and those that
handle purely transient traffic, such as Edge servers.
When circular logging is disabled, the Store continuously creates new
transaction logs as it fills the current log with data. The Store performs the
following steps to create a new log and switch it to become the current log.
First, the Store advances the checkpoint in the checkpoint file to indicate
that it has committed the transactions in the oldest log into the database.
Next, the Store creates a temporary file called <storage group prefix>tmp.log
or E00tmp.log for the default storage group. The Store switches this file into
use to become the current log when the Store closes off the current log and
ensures that the Store can continue to write transactions into a log without
pausing. If the Store fails to create the temporary file, it usually means that
no more disk space is available on the disk that holds the transaction logs. In
this case, the Store will proceed to an orderly shutdown of the Information
Chapter 5
5.6 Transaction logs
Store service and uses the two reserved log files to hold transaction data that
it flushes from its cache during the shutdown.
When the temporary log file is created, the Store initializes its log header
with the generation number, database signature, and timestamp information.
When the current log file is full, the Store stops writing transactional data
and renames the current log file (always E00.log for the default storage
group) to incorporate the generation number in its file name (for example,
E000007E71.log). The Store then renames the temporary log file to be
E00.log to make it the current transaction log and then resumes writing
transaction data into the current log file, flushing any transactions that have
accumulated in the short time required to switch logs.
The Store continues with the process of creating the temporary log file
and switching it to take the place of the current log file until the Information
Store service shuts down. The number of logs created on a server varies
according to message traffic and other activity, such as mailbox moves and
public folder replication. Busy and large servers may generate tens of
gigabytes of transaction logs daily, so this is obviously an important factor to
take into account when you size storage for servers. However, the space occupied by transaction logs is released when the files are deleted after successful
full backups. The logic here is that you do not need the transaction logs any
more if you have a full backup because the database is consistent and you
have a backup copy if a problem arises. The backup copy also contains the
transaction logs that are necessary to allow the Store to make the database
consistent if you need to restore it.
Reserved logs
In Exchange 2000/2003, every storage group has two special log files called
RES1.log and RES2.log that the Store uses to reserve some storage in case
it is unable to create new transaction logs for any reason, usually space
exhaustion on the storage device that holds the transaction logs. Running
out of disk space was a reasonably common occurrence in the early days of
Exchange because disks were small and expensive, but such an event is a
much less common experience for today’s Exchange administrator. Most
administrators now monitor free disk space on all disks used by Exchange
and take action to release space if less than 250MB is available. Exchange
2007 renames the reserved logs (which now only occupy 1 MB each) to
E00res00001.jrs is the first reserved log file for the default storage group).
5.6 Transaction logs
Transactions, buffers, and commitment
After a client submits a message to Exchange, an ESE session that is responsible for the transaction follows a well-defined order to apply the transaction to
commit the new message to the Store. The same order is followed for other
transactions such as deletes and moves. First, ESE obtains a timestamp using
the internal time (called a “db-time” held in an 8-byte value) maintained in
the database header. In order to modify a page, ESE must calculate a new dbtime based on the current value in the header. Once it has calculated the new
db-time, ESE writes the records that make up the transaction into the current transaction log. After this operation completes, the session can go ahead
and modify the appropriate pages in the database. Page modifications occur
in an in-memory cache of “dirty pages,” so ESE may first have to fetch the
necessary pages off the on-disk database.
Eventually, ESE flushes the modified pages from the in-memory cache
and writes them into the database. The database write cannot occur before
the transactions are first committed into a transaction log to ensure that data
is always protected. If you look at the steps that make up a complete transaction, you see that the last step is to commit the transaction for an individual
ESE session to disk. For example, the last step shown in Figure 5.8 is a commit command for ESE session 8. Other prior steps begin a transaction for
session 8 and insert some data to replace an existing page. The commit is a
synchronous operation, so no other transaction can occur for that session
until the write to disk is complete. Enabling write-back caching on the disk
that holds the transaction logs improves performance by allowing the write
to complete in the controller’s memory and so release the synchronous wait.
The controller is then responsible for writing the data to disk.
Figure 5.8
Data in a
transaction log
Chapter 5
5.6 Transaction logs
The delivery of a single new message causes ESE to modify many different pages since many tables (Inbox, message folder table, and so on) are
updated. ESE also updates the index for each table and if the message contains a large attachment, its content will be broken up into several long value
chunks, all of which generate log records and the Store generates a set of log
files to hold all the pages that it updates with the new content. On the other
hand, if you delete an item that has a very large attachment, Exchange only
needs to capture the page numbers of the pages that now contain deleted
data and the actual content does not appear in the logs. As you can see from
Figure 5.8, records for different transactions are interspersed throughout a
log file, so replaying a transaction is quite a complex matter. These records
are low-level physical modifications to the databases in a storage group. Each
transaction log contains a sequential list of operations that the Store performs
on pages in memory. The log captures details of when a transaction begins,
when it is committed, and if the Store needs to roll it back for some reason.
Each record in the log is of a certain type. Record types include begin (a
transaction is starting), replace (some data in a page is being updated), delete
(data is removed), insert (data is added), and commit. In addition to interleaved transactions from multiple databases in a storage group, transactions
from multiple sessions are interleaved throughout a transaction log. This
means that the “begin” record type also identifies the session that performed
a transaction. You can think of a session as a thread running within the Store
process. The session forms the context within which ESE manages the transaction and all of the associated database modifications. Each session could be
tied back to a particular client, but the database has no knowledge of individual clients (MAPI or otherwise), as all it sees are the threads that operate on
its contents.
Regretfully, there is no tool provided to interpret a log file. Figure 5.8
illustrates how a set of transactions might appear in a log. In this example,
the first transaction in session 8 (or thread 8) is replacing a record in the database. Every physical modification to a database is time stamped. ESE uses
timestamps later if it has to replay transactions from a particular point in
time. The page number and an offset within the page are also recorded. The
length of the data to be replaced is then noted and is then followed with the
actual binary data that is inserted into the page. The next transaction is a
record delete. The set of insert transactions demonstrates that transactions
from multiple sessions are intermingled within a log. Sessions write data into
the log file as they process transactions. Any dump of a log file from even a
moderately busy server will record transactions from scores of sessions.
When the Store replays transactions from logs to make a database consistent, it has to interpret the contents of all the different transaction logs that
have accumulated since the last good backup to find the transactions that it
requires and then assemble the different records for those transactions from
the logs and replay them into the database.
5.6 Transaction logs
Transaction log I/O
ESE always writes transactions in sequential order and appends the data to
the end of the current transaction log. All of the I/O activity is generated
by writes, so it is logical to assume that the disk where the logs are located
must be capable of supporting a reasonably heavy I/O write load. In comparison, the disks where the Store databases are located experience read and
write activity as users access items held in their mailboxes. You should
never place transaction logs on a compressed drive, as Exchange will have
to decompress them each time it needs to access the content, which only
slows down processing.
On large servers, the I/O activity generated by transaction logs is usually managed by placing the logs on a dedicated drive. This solves two
problems. First, the size of the disk (today, usually 100GB or greater)
means that free space should always be available. If you accumulate 100GB
of logs (100,000 individual log files) it means that either your server is
under a tremendous load or you haven’t taken a full online backup in the
recent past. Full online backups remove the transaction logs when they successfully complete. Second, in all but extreme circumstances, a dedicated
drive is capable of handling the I/O load generated by transaction logs. I
cannot think of a reason why a dedicated drive could become swamped
with I/O requests from log activity. In any case, if such a load was ever generated on a server, the I/O activity to the Store is probably going to be of
more concern than the log disk.
Of course, having a dedicated drive for log files is a luxury that you
might not be able to afford. But the logic applied to justify the drive—reserve
enough space for log file growth and keep an eye on I/O activity—should be
remembered when you decide where the logs should be stored on your system. For example, it’s a bad idea to locate the logs on the same drive as other
“hot” files, such as a Windows page file. Also, never place the logs on the
same drive as a database. Keeping the logs with their database may seem like
a good idea, but it risks everything. If a problem afflicts the disk where the
stores are located, the same problem will strike down the transaction logs and
you will lose data.
Protecting transaction logs
Apart from the fundamental first step of separating the transaction logs away
from their databases through placement on different physical volumes, best
practice is to deploy the necessary hardware to provide the Store databases
with maximum protection against disk failure.
You also need to protect transaction logs, but they usually do not need
RAID-5, as the overhead generated by RAID-5 will slow the write activity to
Chapter 5
5.6 Transaction logs
the logs. However, it is possible that a specific storage controller will dictate
that you should use RAID-5 on the drive that you use for transaction logs for
a reason particular to that controller, which illustrates the need to consider
the performance characteristics of your hardware and the way that the hardware handles read and write operations before making a firm decision. In the
same vein, using RAID 0+1 to protect transaction logs is also overkill. In
most cases, all you need to do is to allocate two disks for the transaction logs
to create a mirror set. You can place transaction logs on an unprotected drive,
but if you do this, you must understand that any problem on that drive may
render the logs unreadable. In this situation, if you then have to restore a
database for any reason, you will not be able to replay transactions from the
logs into the restored database and data will be lost.
As discussed earlier, the default behavior for ESM is to create transaction
logs and databases on the same volume. Exchange places the default storage
group and its transaction logs on the same volume when you first install a
server. You should therefore review log placement on a regular basis with an
aim of assuring both maximum protection and performance.
I may seem a touch paranoid when I discuss protection for the databases
and transaction logs, but I look at it a different way. Consider how much
money it costs to recover data if a drive fails. Now factor in the cost in terms
of loss of productivity, the strain on everyone involved, and the sheer lack of
professionalism that exists in any situation where you might compromise
data integrity. With these points in mind, I think I am right to be paranoid
about protecting data!
Transaction log checksum
Every transaction log contains a checksum that ESE validates to ensure that
the log data is consistent and valid. Microsoft introduced the checksum to
prevent logical corruption occurring as the Store replays transactions into a
database during a recovery process. The checksum also prevents administrators replaying a selective set of logs back into the Store after a restore, something that used to be possible up to Exchange 5.5. Selective replays are a bad
idea because the Store interleaves transactions from all databases in a storage
group into a single set of logs, so it is possible to miss parts of a transaction if
the Store does not replay the complete set.
ESE uses a type of “sliding window” algorithm called LRCK (Log
Record Checksum) to validate checksums for a selected group of records in a
log to ensure log integrity. ESE reads and verifies these checksums during
backup and recovery operations to ensure that invalid data is not used. If
ESE detects invalid data through a checksum failure, it logs a 463 error in the
system event log. If ESE fails to read the header of a transaction log and is
unable to validate the checksum, it signals error 412 in the application event
5.6 Transaction logs
log. Transaction log failure inevitably leads to data loss, as the only way to
recover from this error is to restore the last good backup. All of the transactions since that backup will be lost.
Maximum database size
Microsoft makes the point that the 64-bit larger memory model of Windows
and the changes made in the database engine enables Exchange 2007 to support larger mailbox quotas. One interesting aspect of the introduction of
continuous replication is Microsoft’s assertion that you can double the size of
the mailbox databases that you operate from Microsoft’s usual recommended
100GB8 maximum size for databases that are not on an LCR server to
200GB. If you allocate 1GB mailbox quotas, these limits mean that you can
only support 100 or 200 mailboxes per mailbox database, which does not
seem a lot. Of course, you may run into other problems if you deploy very
large mailboxes, such as the number of storage groups that you need to use.
Once you run the Enterprise Edition of Exchange 2007, the software
does not enforce any limit for a mailbox database size. In fact, the theoretical
limit for a database on an Exchange Enterprise server is 16TB, so there is
plenty of room to grow from 100GB. The operational requirements that surround the deployment of very large mailbox databases means that it is unusual
to see a mailbox database that exceeds 300GB, but there are instances of
Exchange 2003 servers that support mailbox databases that are larger than
1TB. Microsoft’s suggested limit of 100GB for a standard mailbox database is
a somewhat arbitrary value because the limits are influenced by:
The desire of the Exchange engineering group for customers to operate manageable databases on production servers. It is reasonable for
an engineering group to tell customers to remain within well-defined
limits for supportable database sizes because it reduces the possibility
that customers will deploy very large databases. In addition, if a bug
or hardware failure affects a database, users lose less data when databases are smaller.
Experience gained from the internal deployment of Exchange within
Microsoft and the way that Microsoft IT manages their Exchange
servers and storage. It is natural to take lessons such as the time
required for backup and restore operations from a well-functioning
deployment and represent them as best practice.
The focus that the Exchange engineering group had a focus on
enabling low-cost storage as a suitable platform for Exchange 2007 to
Some consultants use a lower 75GB maximum for a database.
Chapter 5
5.6 Transaction logs
drive down cost. If you deploy low-cost storage, you are using storage
that is less resilient than a Storage Area Network, so it makes sense to
limit database sizes. It is reasonable to create a 50GB database on
direct attached storage and less of a good idea to create a 200GB on
the same platform.
The larger the database, the longer it will take you to perform offline
operations such as defragmentation (with ESEUTIL). While you
should not need to perform these operations very often, you also do
not want to have to spend days to process a 300GB database.
Experience gained with problems within the database with Exchange
2000 and 2003. For example, in a very large mailbox database, ESE
has to manage a huge fee page lookaside list (the list of free pages that
are available for new items), and this caused some performance and
stability issues if the database is stressed.
It is true that the basic problem with large databases is that the larger the
database, the longer it takes to perform online and offline backups and
restore operations. Restore operations usually take much longer than backups. In addition, Exchange may struggle to perform the nightly background
maintenance operations such as the removal of deleted items past their retention date within the maintenance window for these tasks. Even with these
caveats, you can support very large mailbox databases through rigorous management practices.
Microsoft doubles their recommendation for maximum database size to
200GB when you use LCR to protect a server because LCR allows you to
recover much more quickly from a database failure by swapping the local
replica in to take the place of the failed database. However, even using continuous replication, Microsoft is careful to set expectations as to what practical maximum size of a production database should be because of the time
required to reseed the database replica if this is required. For example, if a
failure occurs on the drive that supports the replica, you need to reseed the
replica before LCR can restart, and this requires you to take a block copy of
the production database (using a file level copy or with the Update-StorageGroupCopy command, which uses the ESE streaming backup functionality to
copy the database). Copying a very large database can take a long time even
on fast storage9, hence a big reason for Microsoft’s preference that you limit
the size of production databases.
Continuous replication provides a solution for hardware failure that
affects mailbox stores, but only if you place the replicated files on a different
A good estimate is to expect to be able to copy 25MB/second for a single database, with speeds up to 100MB/sec for
multiple parallel database seeds on a gigabit network.
5.7 Database portability
device that is not affected by the failure. One assertion that I have heard is
that the use of continuous replication removes the need to perform daily full
backups because another copy of the database is always available and up to
date. Conceptually, this position may be accurate in terms of the technology,
but it absolutely fails to consider the requirement that companies have to
preserve and protect business critical data, which often mandates the need to
take full daily backups followed by the removal of the backup media to an
offsite location.
The big problem with Microsoft’s recommendation for maximum database sizes is that there are many examples of deployments that routinely use
larger databases with both Exchange 2000 and Exchange 2003. Given good
management practices in design, deployment, and operation, especially
hosted on SAN-based storage, there usually are not problems with very large
mailbox databases, so there is no need to reduce mailbox database sizes as you
migrate from Exchange 2003 to Exchange 2007.
In summary, Microsoft’s suggested limits for maximum database size are
reasonable to use for planning purposes and you should have good reasons
before you decide to use higher limits.
Database portability
Database portability is a new feature in Exchange 2007. It means that you
can move a database from one Exchange 2007 server and remount it on
another Exchange 2007 server in the same organization or in a different storage group on the same server. You can then reconnect users to mailboxes in
the moved database. Database portability is only available for mailbox databases—you cannot move a public folder database between servers as this
would cause havoc for the replication mechanism. Databases in all previous
versions of Exchange are static and tied to the storage group and server where
they are created. You cannot move databases from one Exchange 2003 server
to another, except in the very limited circumstances permitted when a database is mounted in the Recovery Storage Group to allow an administrator to
recover mailbox contents. Users cannot access a database mounted in the
Recovery Storage Group, so you cannot consider this database as portable.
You also cannot move a database from an Exchange 2003 server to an
Exchange 2007 server because of the different database formats (such as the
change in page size made by Exchange 2007).
Database portability is important because it allows you to recover from a
server failure by moving database files (and their mailboxes) to another server
and so restore service to users. Database portability also allows you to split up
the databases in a storage group by moving the mailboxes from one database
to a database in another storage group. You may want to do this to split up a
storage group that contains several databases into a set of storage groups that
Chapter 5
5.7 Database portability
each contains one database so that you can use LCR with one or all of the
storage groups to improve database resilience. See Chapter 9 for more information about LCR. Of course, you can move mailboxes from one database
to another with the Move-Mailbox command, but this approach generates a
lot of processing for the server and a pile of transaction logs. It is easier and
faster to move the physical files from one disk to another than to move mailboxes, especially if the two locations are on the same server. Database portability is a situation where operating Exchange 2007 on a SAN is better than
on direct-attached storage as it is much easier to remap the LUN that holds a
database and redirect it to another server.
Database portability depends on some additional capabilities introduced
in the Store in Exchange 2007 and the ability of the Move-Mailbox command
to move configuration data for mailboxes, which is also a new feature in
Exchange 2007. To move only the configuration data for mailboxes, you use
the –ConfigurationOnly parameter with the Move-Mailbox command. For
Move-Mailbox –id Redmond –TargetDatabase ‘ExchMbxSvr2\New
Database’ -ConfigurationOnly
Exchange maintains pointers in the Active Directory to associate a user
account with the database where its mailbox is located. When you move configuration data, it means that you move the pointer that ties a mailbox to a
database to a different database. The next time an Outlook 2007 client
attempts to connect to the mailbox, the AutoDiscover service will redirect it
to the database that you have moved the mailbox to. Older Outlook clients
should be able to switch databases automatically, but you will have to reconfigure the profiles of IMAP4 and POP3 clients to update them with the new
database location.
Of course, you have not physically moved the mailbox (yet), just a
pointer. Of course, after you use the Move-Mailbox command to move configuration data, the Active Directory has to replicate the changes around the
organization. If something such as a network outage stops the Active Directory from being able to replicate the data, users will not be able to connect to
their mailboxes on the new server until replication completes.
You can use these steps to move mailboxes in a database to a different
server or to move the database from one disk to another on the same server:
If it is possible, take a full backup, just in case. Apart from anything else, this step will reduce the number of transaction logs
that you have to copy from the old to the new location.
5.7 Database portability
Create a list of all of the mailboxes and their contents (number of
items and mailbox sizes) from the database that you plan to
move. The easiest way to do this is with the Get-MailboxStatistics command.
Create a new database on the target server (or in the desired location on the same server). You must give the new database the
same name as the database that you want to move. You are going
to overwrite this database with the database that you move, so
this step is to ensure that the configuration data in the Active
Directory matches the name of the moved database. For the same
reason, make sure that the properties of the new database include
the “this database can be overwritten by a restore” setting.
Dismount the database in the target location and delete the .EDB
Move the user mailboxes, remembering to specify –ConfigurationOnly.
Dismount the database that you want to move plus any other
database that is in the storage group, including a public folder
database if present. All of the databases in a storage group share a
common set of transaction logs, so if you do not dismount all of
the databases, you will not be able to copy the logs.
If you are unsure that the database has shut down cleanly (for
example, the server crashed unexpectedly), you can run the
ESEUTIL utility with the /R switch to commit transactions from
any log files that are needed and put the database into a clean
Copy the database files including transaction logs and content
indexing catalog files to the new location.
Mount the database in the new location.
Check that all the mailboxes that you wanted to transfer have
moved across. Use the Get-MailboxStatistics command to
generate a report on the mailbox contents (see below).
The more paranoid (or wise) administrators would now take
another full backup on the basis that you can’t be too sure about
things and a good full backup is always a reassuring thing to have.
You can now clean up the old database files by first removing
them through EMC or with the Remove-MailboxDatabase command, and then deleting the files.
Chapter 5
5.7 Database portability
You probably will not move just one mailbox to access the moved database. Instead, it is more likely that you will want to move all of the mailboxes
from the database on the old server to the new server. For example, this command moves the configuration data for all of the mailboxes belonging to
“Database1” on server ExchMbxSvr1 to use the same database name on
server ExchMbxSvr2:
Get-Mailbox –Database ‘ExchMbxSvr1\Database1’ | Move-Mailbox
–TargetDatabase ‘ExchMbxSvr2\Database1’ –ConfigurationOnly
After moving the mailbox database and logs and remounting it in the
new location, you can check that the contents are in place. Get-MailboxStatistics should show you that all of the mailboxes are now in place in the
moved database. If you took a list of the mailboxes before the move, you can
now compare that list to the report generated after the move.
Get-MailboxStatistics –Database ‘ExchMbxSvr2\Database1’
Figure 5.9
Mixed transaction
logs after a move
It is entirely possible that the prefix used by the storage group that you
move the database to may differ from that used by the source server. In this
case, you will see that new transaction logs created after you mount the database take the prefix used by the new storage group rather than persisting with
the prefix used on the old. This is quite logical because transaction logs
always use the prefix of the storage group that creates them. Figure 5.9 shows
what the prefix switch means in action. You can see that the directory contains a set of transaction logs created with the E0 prefix, which means that
5.8 MAPI connections and logons
they belong to the default storage group. This set of logs terminates at E00
and E00000895 at 11:43AM. The next transaction log starts at E030000040,
created at 12:01PM. You can see that the prefix changes from E0 to E3 and
that the log name has gone backwards in terms of the sequence number. All
of this is logical because the new logs have taken the prefix of the storage
group that the database now belongs to, but it can be a little confusing the
first time you see this happen.
Zero database pages
If you view the properties of a database through the Exchange 2003 GUI,
you can set a checkbox to have Exchange zero out deleted database pages.
This means that the Store writes zeros into deleted pages to overwrite any
data that exists in the pages and remove the possibility that someone could
dump the database to work out what those pages contain. Zeroing out pages
in this manner adds a small amount of overhead to Store processing, so
Exchange does not enable this feature for databases by default. Generally,
only administrators of highly secure Exchange servers opt to use page zeroing
and this is the reason why Microsoft removed the option from the Exchange
2007 GUI.
If you want to zero out deleted database pages in Exchange 2007, use
the Set-StorageGroup command as follows:
Set-StorageGroup –id ‘VIP Storage Group’ –ZeroDatabasePages
As the name of the command implies, all of the databases in the storage
group now zero out deleted pages.
MAPI connections and logons
It would be simple if one MAPI connection from a client like Outlook was
reflected in a single link to the server. However, MAPI is a reasonably complex protocol and the work done by a single client is usually represented
through multiple connections, a fact that can be somewhat confusing to the
unwary administrator.
When a MAPI client connects to Exchange, it establishes a MAPI session (otherwise known as a connection). The session is an object that contains information about the user including the mailbox that they access and
their security credentials. After it establishes a session, the client is able to log
on to the session. A client is able to establish multiple logons to a server
within a single session and there can be multiple active sessions per user. For
Chapter 5
5.9 The Deleted Items cache
example, a user is connected through Outlook and also has a session established through the BlackBerry Enterprise server so that new messages can be
delivered to their handheld device. Each logon provides the connection to a
mailbox, public folders, or another user’s mailbox (to access their calendar,
for instance). Remember that remote clients may logon to a server to access
a folder on the server, which increases the total sessions that a server has to
support. All of the logons are terminated to end the session when the client
exits. The ability to maintain several distinct connections to a server within a
single session is the reason why you see Exchange report multiple connections from a single Outlook client. In general, if you want to understand the
connections that exist on a server, you can look at them through the following relationship:
Number of logons >= Number of sessions >= Number of users
accessing the server.
With this statement in mind, Table 5.7 describes the major Performance
Monitor counters that you can use to assess the current workload of an
Exchange server.
The Deleted Items cache
The Deleted Items cache is a temporary resting point where items remain
between the time that a user deletes them from their mailbox and the time
that the Store removes the items from a database. The deleted items retention
period is the time that the items remain in the cache. The ability to recover
deleted items from the cache has saved me on innumerable times to retrieve a
message that I deleted that suddenly became important again.
When a user deletes an item, their client moves the item into the
Deleted Items folder. The item remains in the Deleted Items folder until the
user makes an explicit choice to empty the folder or the client automatically
empties the folder the next time it exists. Outlook controls this behavior
through the option shown in Figure 5.10. Best practice is to encourage users
to set the option to empty their Deleted Items folder automatically, if only
to help keep under their mailbox quota, but some users like to keep items in
the Deleted Items folder for quite a period, perhaps because they want to
review these items before they make the decision to delete them finally.
Note that you will not be able to recover an item if you move it into a PST
and then delete it from that location (either by using the Shift-Delete key
combination when working with items in a folder or by emptying the PST’s
Deleted Items folder). This is because PSTs delete items immediately and
finally and do not have the ability to keep items until a retention period
expires as the Store can.
5.9 The Deleted Items cache
Table 5.7
MAPI Connections and logons
Performance Counter
MSExchangeIS Mailbox\Active Client
The number of logons that have issued a
MAPI request within the last 10 minutes.
As a client can maintain multiple logons,
this number may be more than the number of mailboxes on a server. The number
is incremented for each logon and decremented after a logon has been inactive for
10 minutes.
MSExchangeIS Mailbox\Client Logons
The number of client processes, including
system processes, that have initiated
MAPI logons to sessions. A single Outlook client normally maintains several
MSExchangeIS\Active User Count
The number of unique users that have
logged onto the server and been active in
the last 10 minutes. If the server maintains public folder replicas, remote users
may connect to these replicas and increase
the active user count. The count also
includes remote users who connect to calendars on the local server.
MSExchangeIS\Active Anonymous User
The number of anonymous users (those
who do not authenticate) who have connected to the server and been active in the
last 10 minutes.
MSExchangeIS\User Count
The number of users currently connected
to the Store, including users connected to
public folder replicas and those who are
logged on to other users’ mailboxes.
MSExchangeIS\Anonymous User Count
The number of anonymous users currently connected to the Store.
MSExchangeIS\Connection Count
The number of current connections to
the Store. A user may maintain multiple
connections if they use different protocols
such as MAPI and IMAP.
MSExchangeIS\Active Connection Count
The number of connections that have
been active in the last 10 minutes. Generally, the active count is less than the total
connection count.
Chapter 5
5.9 The Deleted Items cache
Figure 5.10
Outlook 2007
option to empty
Deleted Items
Up to Exchange 5.5, the story stopped here. Once deleted, items
stayed deleted. However, users make mistakes and sometimes they delete
items when they should not. When this happened, administrators had a
simple choice: Either tell the user that they could do nothing to recover the
deleted item or go ahead and restore the mailbox database on another
server, find the item, export it to a PST, and provide it to the user. Given
these options, most administrators chose not to restore databases and the
user suffered—unless they held a sufficiently high position in the organization to be able to convince the administrator to change their mind. Since
Exchange 5.5, Exchange servers have supported the option to maintain a
cache of deleted items that remain in the Store until a set deleted retention
period expires, at which point the Store removes the items from the database. This is a great feature for both users and administrators alike because
it saves work for everyone.
To allow deleted items recovery, the Store uses a flag (a MAPI property) to enable “soft” or two-phase deletes. When a client empties their
Deleted Items folder, the Store initially hides the items from view by setting the delete flag. This step has the effect of placing the items into the
Deleted Items cache. The cache is logical rather than physical and does not
5.9 The Deleted Items cache
exist in a separate area in the Store. Instead, the items remain in the
Deleted Items folder but you cannot see them because the Store does not
reveal their existence unless you use the “Recover Deleted Items” option.
The Deleted Items folder does not exist for public folders, but an administrator can still recover a deleted item for a public folder from the cache by
opening the public folder that it was deleted from and then using the
Recover Deleted Items option.
By default, Exchange 2007 applies a 14-day retention period to all new
databases, which means that the Store keeps items in the cache and then
expires them, which then allows the regular nightly background maintenance
tasks to remove the items from the database. Because you cannot upgrade
older servers to run Exchange 2007, every database begins with the default
retention period. Microsoft sometimes mentions that they selected the 14day period because they wanted to reduce the number of times that administrators had to recover items for users by restoring backups, but in most cases,
large companies have operated 7- to 10-day retention periods since Exchange
2000, so the change to 14 days will not have much impact.
Increasing the deleted items retention period has an impact on database
size. The Store has to retain deleted items for longer and these items occupy
pages. To calculate the expected impact, you can estimate based on the average daily message traffic received by users. If this is 20MB and you have
2,000 mailboxes on a server, then the database grows by 20MB × 2,000 =
40GB daily. Users will delete some of these messages. If we assume that they
keep half, then the actual database growth is closer to 20GB and 20GB
moves into the deleted items cache, so we end up with 280GB of cache storing items deleted in the last 14 days. This amount stays constant because
deleted items move in and out of the cache, but you have to account for this
overhead in terms of overall database size and backup times. If you assign
1GB mailbox quotas, this amounts to 14% additional overhead. All of these
calculations are highly specific to the quotas and behavior that apply within
an organization, but they may help you to understand the potential impact.
If you want to change the deleted items retention period for a database,
you do so through EMC by selecting the database and modifying the limit
through the Limits property tab, as shown in Figure 5.11. You can also set
the retention period for deleted mailboxes here. By default, Exchange 2007
retains deleted mailboxes in a database for 30 days after an administrator
deletes them, just in case someone makes a mistake and removes a mailbox
early. Thirty days is an appropriate period for most companies. You can use
the following command to find out what deleted item settings exist for user
mailboxes within your organization.
Get-Mailbox –ResultSize UnLimited –Filter
{RecipientTypeDetails –eq ‘UserMailbox’ –or
Chapter 5
5.9 The Deleted Items cache
RecipientTypeDetails –eq ‘LegacyMailbox’} | Select
DisplayName, DeletedItemFlags, RetainDeletedItemsFor,
Note the use of the ResultSize parameter to ensure that we can handle
more than 1,000 mailboxes and the filter applied so that we do not check
room or equipment mailboxes. A scan like this on a large organization can
take some time to complete and we probably want to pipe the output to a
text file for review. The kind of information you will see is like this:
----------Bernard, Aric
Morrissey, Daragh
Redmond, Tony
Laahs, Kevin
Doyle, Owen
Kirby, Jill
DeletedItemFlags RetainDeletedItemsFor
---------------- --------------------DatabaseDefault
RetainUntilBackup 60.00:00:00
The last mailbox in this report has flags set to keep deleted items for 60
days and an explicit flag set to keep deleted items in the database until you
backup the database. Naturally, you can manipulate these properties through
the Set-Mailbox command if you find that you want to set certain values for
a mailbox:
Set-Mailbox –id ‘Doyle, Owen’ –RetainDeletedItemsFor
–RetainDeletedItemsUntilBackup $True
–UseDatabaseRetentionDefaults $False
You can also set the deleted items retention period for a database
through the shell with the Set-MailboxDatabase command. For example,
the next command sets a deleted items retention period of 120 days on a
database. It also sets the retention period for deleted mailboxes to 60 days.
Finally, the command sets the flag to instruct the database to retain deleted
items until an administrator takes a successful backup, even if this causes the
items to exceed their retention period. This is a good option to take because
it ensures that a deleted item will be kept as long as is necessary until it is
safely stored in another location that you can recover it from if required.
Set-MailboxDatabase –id ‘VIP Mailboxes’ –ItemRetention
–DeletedItemRetention 60.00:00:00
5.9 The Deleted Items cache
To set the deleted item retention for all databases on a server, add a call
to Get-MailboxDatabase:
Get-MailboxDatabase –Server ExchMBXSvr1 | Set-MailboxDatabase
–DeletedItemRetention 120.00:00:00
Figure 5.11
Setting a specific
retention period for
a mailbox
It is often useful to be able to assign a special deleted items retention
period to one or more mailboxes. Because it takes time to set limits on individual mailboxes, this is not a recommended option and you should only
take it on an exception basis. For example, during a legal discovery action,
your legal department may compel you to be able to recover all messages sent
and received by particular users such as the officers of the company or those
involved in particular deals. You can either group all the affected users into a
single mailbox store or apply the desired retention period to the database, or
you can edit the properties of each mailbox. The latter option is preferable if
the target accounts use mailboxes on a set of distributed servers, perhaps
spread across a wide geographic area. You can set a deleted items retention
period for a mailbox in three ways:
Chapter 5
5.9 The Deleted Items cache
Accept the default retention period inherited from the database
that hosts the mailbox.
Use EMC to modify the deleted items retention period for the
mailbox as shown in Figure 5.12. In this instance, the retention
period is set to be 60 days.
Use the Set-Mailbox command to set the deleted items retention
period. For example, to set the retention period to be 60 days for
the “Bond” mailbox:
Set-Mailbox –id Bond –RetainDeletedItemsFor 60.00:00:00
-UseDatabaseRetentionDefaults $False
Figure 5.12
Setting Deleted
Items retention for
a mailbox
The Store does not calculate the size of items in the deleted items cache
when it applies quotas to user mailboxes. Some users exploit this fact when
they come close to exceeding their quota by selecting large items from their
mailbox, deleting the items, emptying the Deleted Items folder, and then
performing whatever actions they wish. Later on, when their quota frees up,
they can recover the items from the cache.
Cleaning the Deleted Items cache
The background Store maintenance tasks performed by the System Attendant remove items in the cache after their retention period expires. The exact
schedule that controls background processing is set through a database property and it is usually started nightly just after midnight. In database terms,
5.9 The Deleted Items cache
removing items from the deleted items cache are “hard deletes” because the
Store removes the items from its database tables and you cannot recover them
later without performing a database restore. You can get an idea of the
cleanup processing done by the Store by looking for event 1207, which
reports the number of items and their size in the cache before and after scanning for items with an expired retention period. Figure 5.13 shows how the
Store reports the cleanup for a database. In this case, the Store did not
remove anything because no item exceeded the deleted item retention period!
Figure 5.13
Items removed from
the Deleted
Items cache
Recovering items and mailboxes
Pushing work down to users is always popular with administrators and all
MAPI clients from Outlook 98 onwards support the option to recover
deleted items from the cache. Outlook Web Access also supports this functionality. Anyone who uses a client like Outlook Express that does not support deleted item recovery will have to log on with a client that does and
Outlook Web Access is usually the preferred option.
When you use the Recovered Deleted Items option (Figure 5.14),
Exchange provides a list of all the deleted items in your mailbox that have
not yet exceeded their retention period. The items listed were deleted from
the folder that you run the option from, so if you take the option from the
Deleted Items folder, you see the majority of the items that you have
deleted. On the other hand, if you attempt to recover deleted items from the
Chapter 5
5.9 The Deleted Items cache
Inbox, Outlook only lists the items that were deleted directly from this
folder and never went through the Deleted Items folder. You can only
recover items when you connect to a server. If you run Outlook in cached
Exchange mode, Outlook connects to Exchange to recover items. Items
recovered in cached mode take a little time to reappear in the Deleted Items
folder because Outlook has to synchronize the items from the server into
the client-side folder replica.
Figure 5.14
deleted items
By default, the Deleted Items folder is the only folder enabled for item
recovery. However, clients can delete an item in place in any folder (use the
Shift/Delete combination instead of the Delete option), meaning that the
item does not progress through the Deleted Items folder before it enters the
cache. If users are accustomed to delete items in place, you may want to
enable deleted items recovery for every folder to allow users to recover items
in their original folders. Enabling this feature is a good idea for users who
operate in cached Exchange mode because deleting in place avoids the network overhead of synchronizing the items that move into the Deleted Items
folder. It also makes Outlook faster to exit because there is less to delete in
the Deleted Items folder. Note that if you delete an item in place and want to
recover it later on, you have to access that folder and then use the Recover
Deleted Items option rather than attempt to recover the item from the
Deleted Items folder.
To enable deleted items recovery for every folder, create a new
DWORD value at the following registry location on the client PC and set
its value to “1”—this works for all versions from Outlook 98 through Outlook 2003. You don’t need to make this change for Outlook 2007 because
5.9 The Deleted Items cache
it enables the dumpster for all folders, even when it is not connected to
Exchange 2007.
Outlook versions prior to Outlook 2003 do not filter the cache when
you attempt to recover items into any other folder than the Deleted Items
folder. In other words, you see everything in the cache rather than just the
items deleted from the folder that you want to recover items for. Outlook
2003 and 2007 filter items, so you only see the items deleted from the folder
rather than the complete cache.
You can use the Get-MailboxStatistics command to list any mailboxes that have been deleted. For example:
Get-MailboxStatistics –Database ‘Executives’ | Where
{$_.DisconnectDate –ne $Null}
Use the Connect-Mailbox command to reconnect a deleted mailbox to
an Active Directory user account.
Connect-Mailbox –Database ‘VIPMBX’ –identity ‘George Smith’
In this case, the name of the disconnected mailbox is “George Smith.”
We haven’t told Exchange what Active Directory account to connect the
mailbox to, so Exchange searches the Active Directory to discover a suitable
account using the LegacyExchangeDN and Display Name values that it
retrieves from the disconnected mailbox. If Exchange can find a match, it
prompts you to confirm that you want to proceed to reconnect. If Exchange
can’t find a match, you will have to provide the name of the Active Directory
account to use by passing the value in the –User parameter.
EMC also supports an option to reconnect a deleted mailbox. In this
case, you open the Disconnected Mailbox node under Recipient Configuration to see the list of deleted mailboxes that are available to connect (Figure
5.15). You can then select a mailbox and take the connect option to launch a
wizard to select the user account to reconnect the mailbox with. As with the
shell, you can search the Active Directory using the LegacyExchangeDN and
Display Name values to find a match or select a new account.
Chapter 5
5.10 Background maintenance
Figure 5.15
Reconnect a
deleted mailbox
5.10 Background maintenance
All databases have internal structures that require some degree of maintenance and Exchange is no different. You can take a database offline to rebuild
it with the ESEUTIL utility or verify its internal structures with the ISINTEG utility, but these operations require you to deprive users of access to
their mailboxes. As Microsoft designed Exchange to be highly available with
as little downtime as possible, it is obvious that some online maintenance
operations are required to keep the databases efficient while allowing users to
continue working. The Store performs online maintenance as a background
task nightly to ensure logical consistency within the mailbox and public store
databases and to remove unwanted data. As shown in Table 5.8, the Store
performs 12 background maintenance tasks nightly.
Table 5.8
Background maintenance tasks
Mailbox Store
Public Store
Purge indexes
Perform tombstone maintenance
Purge the deleted items cache
Expire outdated public folder content
Expire tombstones in public folders
5.10 Background maintenance
Table 5.8
Background maintenance tasks (continued)
Public folder conflict aging
Event history cleanup
Update server versions
Secure folders
Purge deleted mailboxes
Check for obsolete messages
Database defragmentation
While the Information Store process controls background maintenance,
it actually only executes the first 11 tasks and executes the last by calling ESE
to perform defragmentation, an action the Store takes after at least one of the
11 tasks is complete. The purpose of background defragmentation is to reorganize pages within the database so that they are in the most efficient structure. At the end of the pass, Exchange reports how much free space is
available in the database. Figure 5.16 shows that the database has 262 megabytes of free space. In other words, Exchange can store an additional 262
megabytes of information in this database before it will be forced to extend
the file size of the database. You cannot recover the free space and return it to
the file system without taking the database offline and processing it with the
ESEUTIL utility to perform a complete rebuild. In the bad old days when
Exchange background defragmentation was not as efficient as it is now and
file space was more costly, administrators often rebuilt databases to recover
disk space, but this is not required today.
In concept, the difference between the tasks performed by the Store and
those executed by ESE is similar to how ISINTEG understands the structure
of tables and indices that the Store assembles pages into while ESEUTIL
deals with pages in the database at a lower level. Obviously, background
maintenance can generate a huge amount of updates to Store contents. Like
any other database update, the Store captures the details in transaction logs,
which accounts for why you will see more transaction logs generated during
the maintenance window than you can account for through user activity
(which is usually low at this time) or even through public folder replication.
Note that the “secure folders” task is now obsolete as it only applies to public
folders that are homed in Exchange 5.5 sites. Once you eliminate Exchange
5.5 servers from your organization, the need for this task goes away.
You can take backups at the same time as background maintenance tasks
are active with the exception of background defragmentation, which clashes
with backups because of the way that ESE rearranges data in pages as it
Chapter 5
5.10 Background maintenance
Figure 5.16
defragments the database. If defragmentation is active when you start an
online backup, the Store suspends it until the backup finishes. If the Store
cannot complete all the tasks during the allocated time, it finishes the last
running task and records where processing stopped so that it can pick up
from that point at the next maintenance period.
The event history cleanup task is the only addition to the background
maintenance performed by Exchange 2007. The Store maintains a list of
event notifications such as a new outbound message or new calendar request
that the transport service and the Exchange assistant processes use to understand when they have to take action. Over time, the table that holds these
events accumulates entries for events that have expired, so the event history
cleanup task goes through the table to find and delete expired events.
By default, Exchange schedules background maintenance for between
1AM and 5AM local time. You can set up a separate custom schedule for each
server by selecting the database properties through EMC, and then set the
maintenance window as shown in Figure 5.17. Unlike Exchange 2000 or
2003, Exchange 2007 does not support server policies, so if you want to
apply the same maintenance schedule to a set of mailbox or public folder
databases, you have to use the Set-MailboxDatabase and Set-PublicFolderDatabase commands. Use the Get-MailboxDatabase or Get-PublicFolderDatabase commands to retrieve the current schedule. For example:
5.10 Background maintenance
Get-MailboxDatabase –id ‘VIP Mailboxes’ | Select Name,
Figure 5.17
Setting the
window for a
For example, to set the maintenance window to between 1AM and 7AM
daily for a group of databases on a server, you can use a command like this:
Get-MailboxDatabase –Server ‘ExchMbxSvr1’ | Set-MailboxDatabase
–MaintenanceSchedule ‘Sun.1:00 AM–Sun.7:00 AM, Mon.1:00 AM–Mon.7:00
AM, Tue.1:00 AM–Tue.7:00 AM, Wed.1:00 AM–Wed.7:00 AM, Thu.1:00 AM–
Thu.7:00 AM, Fri.1:00 AM-Fri.7.00 AM, Sat.1:00 AM-Sat.7.00 AM,
Sun.1:00 AM-Sun.7:00AM’
This syntax is useful because it allows you to be extremely precise about
the exact time when maintenance can occur. However, the syntax used to
specify the date/time settings is a little fussy. Always leave a space between the
minutes and “AM” (as in 7:00 AM) and have a hyphen between the first time
and the day for the second time (as in AM-Wed).
Fortunately, Microsoft has provided a shortcut to express maintenance
windows by allowing you to use a number of predefined windows. For
Get-MailboxDatabase –Server ‘ExchMbxSvr1’ | Set-MailboxDatabase
–[email protected]([Microsoft.Exchange.Data.Schedule]DailyFrom1AMto5AM)
Chapter 5
5.10 Background maintenance
In this case, the special “DailyFrom1AMto5AM” expression sets a maintenance window from 1AM to 5AM daily. The complete set of special expressions available to set maintenance windows are:
Background tasks
Now that we understand what happens during background maintenance and
when it happens, we can consider some of the details. Clients like Outlook
make extensive use of the Store’s ability to generate indexes or views on a
dynamic basis. For example, if you decide that you want to view your inbox
by author rather than by date, Outlook requests the Store for this view. If the
Store has previously generated the view and has it cached, the response is very
quick, but the Store is able to process quickly a request for a brand new view.
Over time, the Store accumulates large numbers of views—each folder in
every mailbox can have several views. It is not desirable to retain all of the
5.10 Background maintenance
views because each view occupies space within the database. In addition,
users may not require a view after its first use. The Store assigns an expiry
time to each view and monitors this data in an internal table called the indexaging table. When background maintenance runs, the Store scans the indexaging table to discover views that are older than 40 days and removes any
view that has expired. Of course, if a client accesses a view, the Store resets its
expiry time.
The next task is to perform tombstone maintenance. The Store maintains a list of deleted messages for each folder and a list of “tombstones” to
indicate that a message was deleted. The tombstone list is most important for
replicated folders, such as public folders, because it provides the Store with a
list of message delete operations that it must replicate to every server that
holds a replica of the folder. After successful replication, the Store can clear
the entries from the tombstone list. Background maintenance ensures that
the lists are accurate.
When a client deletes a message, the Store sets a flag to hide the message
from the mailbox (a soft delete). Clients can retrieve messages from the
deleted items cache by clearing the flag, thus causing the message to reappear
in the mailbox. During background maintenance, the Store examines every
message that has the deleted flag set (if enough time is available in the maintenance window, the Store processes the entire contents of the deleted items
cache) to determine whether its retention period has expired. If this is true,
the Store removes the message from the Store (a hard delete) and makes the
pages used to hold the message available for reuse. You can set retention periods on a per-mailbox, per-database, or per–public folder basis, and can control these setting through EMC or with shell commands.
Figure 5.18 shows the limits set on a public folder database. You can
see the deleted retention time is set at 14 days. Exchange also allows you to
set a retention period for content in a public folder either per database or
per folder. The idea here is that you may use some public folders to hold
content that ages out in a systematic way, such as a feed from an NNTP
service. By aging out content from a public folder, you impose a check on
how large the folder grows to. To set a new age limit for a public folder
database to 365 days:
Set-PublicFolderDatabase –id ‘Public Folders’
–ItemRetentionPeriod 365.00:00:00
To set a specific age limit for a public folder:
Set-PublicFolder –id ‘\Exchange 2007\Admin Discussions’
–AgeLimit ‘365.00:00:00’ –UseDatabaseAgeDefaults $False
Chapter 5
5.10 Background maintenance
Figure 5.18
Public Store limits
If you want to set an age limit for a set of public folders from a known
root, you can use a combination of the Get-PublicFolder and the Set-PublicFolder commands. Note the use of the –Recurse parameter to walk down
through the child folders.
Get-PublicFolder –id ‘\Exchange 2007’ –Recurse | Set-PublicFolder
–AgeLimit ‘365.00:00:00’ –UseDatabaseAgeDefaults $False
Background maintenance also checks for deleted folders that have
exceeded their retention period and then removes them. Much the same processing occurs for deleted messages in public folders. The Store also checks
for deleted mailboxes, which have a default retention period of 30 days, and
removes any that have expired. Removing obsolete folders and mailboxes
cleans up the internal table structure of the database and increases efficiency.
The next check is for deleted public folders that have expired. By
default, if you delete a public folder, the Store creates a tombstone for the
folder. This allows the replication mechanism to propagate the deletion properly to servers that hold replicas. Because replication may not occur quickly
in some circumstances, the Store retains the tombstone for 180 days (default
value). If replication has not propagated the tombstone in 180 days, your
5.10 Background maintenance
organization is experiencing fundamental replication difficulties and a few
erroneous tombstones are of little concern. During background maintenance, the Store checks for folder tombstones that have expired and removes
them. However, the Store only removes a maximum of 500 tombstones per
24-hour period.
Public folder replication is somewhat of a black art and it is easy for conflicts to occur; for example, if multiple users modify the same item in different replicas of a public folder or if multiple users attempt to simultaneously
save an item in the same replica. When a conflict happens, the Store sends a
conflict-resolution message to the users that provoked the problem to ask
them to resolve the issue. It is easier for human beings to decide which
change is more important and so resolve the conflict. While the human
beings decide what to do, the Store maintains the different versions of the
items in conflict. However, if the humans fail to respond, the Store examines
each item in conflict and resolves them automatically, usually by accepting
the most recent version.
The next task is to update server versions for the public stores. This process updates any version information that is necessary to maintain the system
configuration folder. The Store incurs no great overhead and you cannot
control the process.
The Store uses a single-instance storage model, meaning that it keeps a
single copy of message content and uses pointers in user mailboxes to the
content. The Store tracks the number of mailboxes that have pointers to
content through a reference count and physically removes the content from
the database when the reference count reaches zero. In any database, there
is the potential for pointers to become unsynchronized, so background
maintenance checks for lingering messages that have a zero reference count
and then removes up to 50,000 of these items per day. The final and most
compute-intensive task is for the Store to invoke ESE to perform online
Tracking background maintenance
Like any other operation in Exchange, the selected level of diagnostics logging
determines how much information the Store writes about maintenance operations into the Application Event Log. You set the level for diagnostic logging
using the Set-EventLogLevel for the “MSExchangeIS\9000 Private\Background Cleanup” and “MSExchangeIS\9001 Public\Background Cleanup”
settings. Change the value from “Lowest” to “Expert.” For example:
Set-EventLogLevel –id ‘MSExchangeIS\9000 Private\Background
Cleanup’ –Level ‘Expert’
Chapter 5
5.11 Fixing failed databases
After applying the new setting, the Store begins to generate events the
next time it runs background maintenance. Except on very large or heavily
loaded servers, many of these tasks complete quickly. The obvious exception
is background defragmentation. This is the most intense activity and can last
a number of hours, depending on whatever other load the server is under,
hardware configuration, and the size of the databases that the Store processes.
5.11 Fixing failed databases
A database failure has always been the source of bad dreams for Exchange
administrators. It’s not just the failure and the work required to restore service to users there’s the inevitable inquiry as to why the failure occurred—was
it a software bug, fault in a disk drive or controller, a lapse in administrative
procedures, or something else that needs to be understood and protected
against in the future. Database failures were reasonably common in the early
days of Exchange (4.0 through 5.5 and even into 2000), but things have
improved significantly as the software matured and, even more importantly,
hardware (especially disks and controllers) has become much more reliable.
At this point, it’s fair to assert that a database failure comes as a surprise
rather than a common occurrence for most Exchange administrators, assuming that normal best practice in hardware maintenance and monitoring is
carried out. Even so, it’s good to be aware of the tools that you can use to
recover from a database failure should the need arise.
Transaction logging is one of the great strengths of the Exchange Store.
With a full set of transaction logs and some good backups, you should be
able to handle the vast majority of failures. Simply swap in some new hardware to replace the dud components, restore the database from backup,
replay the transaction logs to bring the database up to date, and restart services. Of course, there are different variations on the restore theme, including
dial-tone restores (to restart services with blank mailboxes and eventually
recover the missing contents behind the scenes as users continue to work),
but everything revolves around the ability to restore a copy of the failed database from a backup successfully. Troubles truly begin if you discover that you
can’t restore the database from a backup or Exchange refuses to start up with
the backup copy. Not having a good backup to restore from, finding that the
backup media is damaged, or perhaps that the backup failed for some reason
and no one noticed are all reasons to update your resume.
Even with the reliability and robustness of today’s hardware, nothing
quite beats the feeling of confidence that a good backup brings—or the calmness that happens when you know that solid backups are available in an offsite location. But let’s assume that you have managed to restore a copy but
that Exchange won’t mount the database for some reason. What can you do
then? The answer lies in two utilities that have been part of the product since
5.11 Fixing failed databases
Exchange 4.0. ESEUTIL is the ESE database utility and ISINTEG is the
Store integrity checker. Both executables are located in the Exchange program files directory. These are command line utilities that can only run when
a database is not mounted by the Store, so when these utilities run, users
don’t have access to their mailboxes. Microsoft has tweaked ESEUTIL and
ISINTEG over the years and the latest versions provided with Exchange
2007 are the best and fastest yet. Never use a program from one version of
Exchange to work with databases from a different version of Exchange as it is
highly unlikely that the programs will work and you run the risk of doing
some damage to a database.
ESEUTIL operates deep down within the bowels of the ESE database to
fix problems at the page and records level; ISINTEG operates at a much
higher level to fix problems with the logical arrangement in data within the
database that turns tables and pointers into mailboxes, items, and attachments. Another way of comparing the two utilities is to say that ESEUTIL
takes care of physical corruption while ISINTEG fixes logical corruption.
While neither utility is able to fix problems caused by fundamental hardware
failure (fixing a few corrupt pages won’t get around the widespread corruption
that a failed disk controller can cause), the utilities are very good at fixing the
kind of problems that stop Exchange databases working the way that they
should. It’s common to find administrators who think of these utilities as panaceas for any problem that may afflict a database, a situation that couldn’t be
further from the truth. For example, five or six years ago, it was common
practice to run ESEUTIL regularly to compact databases and recover disk
space to the system. In fact, there are even third-party products that wrap
ESEUTIL with a GUI and run it on a schedule to compact databases. This is
certainly an action that was beneficial with early versions of Exchange when
the Store’s internal processing wasn’t particularly efficient or effective at reusing “white space” within the databases, but any version from Exchange 2000
SP2 onwards is very good at maintaining its databases and you should never
need to run ESEUTIL as a matter of course.
The situation you’re in is that you’ve restored a backup copy of the
database but the Store won’t mount it for some reason. Before we panic,
make sure that the database really is not going to work. Sometimes the database is fine but something else isn’t and that’s what stops the database working. The first thing to do is to check to see if Exchange has reported any
problems in the application event log. Perhaps the restore operation didn’t
complete properly and you have an incomplete copy of the database on
disk; perhaps the database is missing some files, such as transaction logs,
that the Store needs to verify that it can replay any missing transactions into
the database. Exchange is pretty good at noting problems in the event log
and you can check the error number and descriptions reported in the log
against the Microsoft Knowledge Base to see if there are any recommended
actions to take.
Chapter 5
5.11 Fixing failed databases
Another action that you can take is to restart the server. This may be a
case of clutching at straws, but a server restart has been known to clear out
random problems. In any case, it doesn’t hurt to put the server back into a
good known state before you take any further action to cure the problem.
After the reboot, we will proceed with the following actions:
Copy database files.
Run ESEUTIL to fix any database corruptions that it can.
Run ESEUTIL to defragment and rebuild the database (optional).
Run ISINTEG to correct any pointers.
Take a full backup.
To start the repair process, make a copy of the database files—the EDB,
the STM (if on an Exchange 2000 or 2003 server), and the transaction logs.
The database utilities don’t offer a sophisticated GUI that protects administrators from mistakes so it’s important that you have something that you can
revert to if required. Put these files in a safe place and then check that you
have enough disk space available on the drive where the database is located
to make a repair. As a general rule of thumb, try to have 120% of the size of
the EDB file available for use when you run ESEUTIL. If you need to
rebuild the database, ESEUTIL will recreate a new version that is likely to
be around the same size as the original. Reserving an extra 20% is just to be
sure that ESEUTIL has enough space to expand into if it needs to.
ESEUTIL allows you to redirect temporary files to another disk through
command line switches, so you can always take this option if you don’t have
enough free space on the disk that holds the database. While it’s easiest if the
EDB and STM files are in the same directory, they don’t have to be and
ESEUTIL has command line switches to allow you to specify different locations for the files.
You’re now ready to run ESEUTIL in /P or repair mode. Here’s an
example of a command to repair a database10:
This command:
Tells ESEUTIL that it is running in repair mode.
See for the full set of command line switches for ESEUTIL.
5.11 Fixing failed databases
Points ESEUTIL to F:\DATA\Mailbox1.EDB as the input database.
On an Exchange 2000 or 2003 server, ESEUTIL assumes that the
STM file is in the same location as the EDB. If not, you can point
ESEUTIL to the right location with the /S command switch.
Instructs ESEUTIL to create E:\TEMP\Fixed.EDB as the output
database. All of the processing performed by ESEUTIL is captured in
a file called <database name>.integ.raw in the same location as the
input database. While this file is unlikely to be of great interest to
you, it may be useful to Microsoft PSS if you encounter problems in
repairing a database and need help to resolve an issue.
On an Exchange 2000 or 2003 server, it is possible to proceed and
repair an EDB database even if its matching STM is unavailable by specifying the /CreateSTM command switch. This forces ESEUTIL to create a
brand new STM and while you will end up with a fully functional EDB,
any content that clients created in the STM will be missing. The exact
impact that this will have on clients depend on the client type. If you use
Outlook and Outlook Web Access, then the demise of the STM is unlikely
to have a major effect because these clients don’t create data in the STM.
On the other hand, if you use IMAP4 and POP3 clients, the reconstruction of the STM will remove all of their data because the STM is the
default storage location for these clients. Exchange 2007 doesn’t use the
same rendering mechanism for MIME protocols as Exchange 2000/2003
do, and the data for all clients is now stored in the EDB, so this situation
doesn’t arise in Exchange 2007.
Figure 5.19
Chapter 5
5.11 Fixing failed databases
ESEUTIL has never been a fast utility in terms of how quickly it processes data (Figure 5.19), but it gets things done pretty rapidly when it performs a repair and you can expect ESEUTIL /P to execute at approximately
1–2GB/minute. Things get slower when you rebuild a database with
ESEUTIL /D and you can expect the utility to do this work at around 20GB/
hour—your mileage will vary depending on CPU and disk speed. Even with
the fastest CPU, multiple cores, and fast disks, ESEUTIL won’t set any speed
records, so be prepared to wait. In fact, the slowness of ESEUTIL is one of the
factors that many administrators take into account when they calculate the
maximum size of a database that they are happy to manage in production. It’s
all very well to generate a massive database that’s 200GB or more, but if it
takes more than eight hours to run a procedure against the database, then you
start to wonder how you’ll ever meet your SLAs.
The repair operation won’t fix every problem that may be lurking in the
database. In particular, the resulting database won’t be as space efficient as it
could be, so you can run ESEUTIL with the /D switch to defragment and
rebuild the database to create its most effective structure. Once again, you’ll
need enough disk space available to hold the output database. This is usually
between 10% and 20% smaller than the original database, but you may not
see this shrinkage as it all depends on how efficient the original database was.
If you want to get an idea about how much the database is likely to
shrink, you can run ESEUTIL /MS to create a “space dump” of the database. A space dump can generate a lot of information, so it is best to pipe
the output to a text file where you can examine it later on. The interesting
data that you’re looking for is at the start of the dump, where you’ll find the
available pages in the database. Figure 5.20 shows that roughly 26% of the
database pages (7,684) are free and will be recovered if the database is
rebuilt. Multiply this value by 8,192 (the size of a page in an Exchange 2007
database is 8KB; for Exchange 2000 or 2003 use 4,096 as these versions use
4KB pages). In this case, roughly 60MB of free space is available and will be
recovered if the database is rebuilt. These figures are never exact, but they
give you a good idea of what is likely to happen.
Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Figure 5.20
Output from
Version 08.00
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating FILE DUMP mode...
Database: vipmbx.edb
******************************** SPACE DUMP ***********************************
PgnoFDP PriExt
Owned Available
5.11 Fixing failed databases
Rebuilding a database is an optional step, but it’s a good idea to do so
just to be sure that the database is in its most efficient state before users are
allowed to access it. To rebuild a database, ESEUTIL /D must read the input
database page by page, verify the checksum and the physical integrity of the
page, and then write it into a temporary output database. Any page that fails
the checks is dropped, which then leads to the potential for data loss. A page
that doesn’t contain any data won’t matter, but a page that holds information
about a message or even details about messages in a folder is clearly a different matter. The upside of running ESEUTIL to rebuild a database is that it
fixes database problems and that it returns disk space to the file system; the
downside is that you may end up with a number of black holes in user mailboxes. To make sure that an administrator understands that some risk exists
in running a repair, ESEUTIL flags a warning as shown in Figure 5.21.
Figure 5.21
A warning from
After all the pages are processed from the input database, ESEUTIL
copies the temporary output database to take the place of the original. If you
have been able to keep all the files on the same disk, the copy happens
quickly but clearly it will be a lot slower if you’ve had to put the temporary
database on another drive, especially if the database is larger than a few
The last step is to run the ISINTEG utility to fix any logical issues that
exist within the tables in the database, such as correct counts for items in a
folder or pointers to items that don’t exist. Run the command as follows:
ISINTEG –S “Server Name” – Fix –Test -Alltests
As you can see from Figure 5.22, ISINTEG lists the databases on the
server and allows you to select the one that you want to process, which must
still be dismounted at this point. However, behind the scenes, ISINTEG
mounts the database to access its content through the Store process and then
dismounts it afterwards. Users aren’t aware that the database has been
mounted and the Store blocks access anyway. ISINTEG processes data rapidly. It’s a verbose utility because it was originally created by Microsoft as an
internal test utility to verify the integrity of a database in the days when there
were more software bugs that caused logical errors in the Store than there are
today. At the end of a successful ISINTEG run, you should see very few or
zero errors reported. If there are errors, you can run ISINTEG again to see
Chapter 5
5.11 Fixing failed databases
Figure 5.22
whether ISINTEG can fix the errors, but if persistent errors continue it may
be an indication that a fundamental corruption is still present in the database. In this case, you can either run ESEUTIL again followed by ISINTEG
to see whether the problem disappears or, if time is pressing and users want
to get to their mailboxes, proceed to mount the database. If users report further problems, you should move mailboxes to another database until you are
able to delete the database. If the problem database hosts public folders, you
can move the replicas to a public folder database on another server, let replication finish, delete the replicas from the problem database, and then finally
delete the database.
The final thing to take care of is to perform a full online backup after
you have remounted the database. All of the transaction logs created before
you started working with ESEUTIL and ISINTEG are nullified if you
rebuild the database and you don’t want to have to repeat all the work that
you’ve just done, so take a full backup immediately before the database is
back online. Apart from anything else, the checks that the backup API does
as information is streamed out of the database will give you an extra feeling
of confidence that you’ve restored the database to good health.
If you run Exchange 2003 on your servers, you have to perform a database repair step by step using the raw utilities. Exchange 2007 is a different
matter because the Toolbox node of EMC includes a Database Recovery
Management that will lead you through the entire process of running
ESEUTIL and ISINTEG. All you’ll have to do is make sure the target database is dismounted before starting the utility, mount it again after successfully running the utility, and complete the job by taking a full backup.
Figure 5.23 shows the results page of the Troubleshooting Assistant after it
performed exactly the same operations as we covered in this section. See
Chapter 10 for more information about the Exchange Toolbox.
5.12 Exchange 2007 content indexing
Figure 5.23
Results of a
database repair
5.12 Exchange 2007 content indexing
Microsoft introduced content indexing (also called Exchange Search) for
mailbox and public databases in Exchange 2000. This implementation used
an early version of the Microsoft Search engine, but it proved to be unsuccessful in production because of the load that indexing generated on the
server. Over time, we have seen increases in server performance, better
knowledge and implementation of search algorithms, and the appearance of
efficient desktop search engines that are capable of indexing mailbox data,
such as Microsoft’s own Windows Desktop Search and the LookOut add-in
for Outlook that Microsoft purchased in 2005, plus competitors such as
Google Desktop. Microsoft includes the ability for Outlook 2007 to use
Windows Desktop Search, but the big issue with desktop indexers is the
potential performance impact desktop indexing can provoke on the server.
To the server, a desktop indexer can seem to be like a voracious user who
keeps on demanding information in a seemingly endless stream of requests
from the desktop to the server. It makes sense that it is more efficient to perform indexing once, on the server, and have clients use that index rather than
generating local indexes, but users will not move away from local indexes
unless they are sure that the server-based equivalent works. Therefore, to
avoid the proliferation of local client-based indexes, we need tremendously
efficient and responsive server-based indexing. This never happened in
Chapter 5
5.12 Exchange 2007 content indexing
Exchange 2003, where indexing was so slow that even Microsoft realized that
it was best to disable the feature by default.
Apart from desktop searches, Microsoft supports Exchange search folders and programming interfaces to allow external programs to search within
the Store. Huge differences exist between Exchange 2007 content indexing
and in-Store searches. Here are some of the major differences. The first three
are the most important in practice—content indexing is faster, more accurate, and can look through attachments:
Content indexing is invariably faster because it uses indexes to locate
requested information while Store searches have to perform serial
scans through everything in the mailbox or folder that you search.
Content indexing allows searches based on words, phrases, and sentences while a Store search uses a stream of bytes. Content indexing
therefore ignores punctuation marks and spaces and is case insensitive
while Store searches look for exact matches of all supplied characters.
Content indexing supports filters for a number of common attachment types so it can index and search the content of those attachments. Store searches look through MAPI properties, which include
message bodies but not attachments. On the other hand, because
Store searches can access MAPI properties, you are not limited to
content searches and can check date ranges and other properties such
as importance, flags, etc.
Content indexing is usually more accurate because it uses word prefix
matches as well, so a search for “star” will also discover items that
contain “starlet” or “stargazer,” but not “filmstar.” Store searches will
find all three instances unless a prefix match is explicitly specified.
Content indexing understands non-English languages, which gives it
the ability to support correct searches against content that contains
double-byte characters such as Japanese and Chinese. The byte
stream searches performed by the Store have great difficulties with
double-byte languages.
While content indexing offers major advantages in search accuracy, customers will enable it only if it responds quickly to user requests and does not
overload the server. These were the major issues with content indexing in previous versions of Exchange, so Microsoft rewrote the entire subsystem for
Exchange 2007 based on Version 3.0 of the Microsoft Search engine, which
SQL Server 2005 also uses. SharePoint Portal Server 2007 also uses Microsoft
Search 3.0 together with some of its own technology.
5.12 Exchange 2007 content indexing
Microsoft also rewrote the software layer that connects the indexing
engine to the Store so that it is much more efficient in how it accesses data.
The net effect is that indexing is smooth and consistent and up to 35 times
faster (Microsoft’s own estimate) in indexing activities. The only real downside is the requirement to allocate between 25% and 30% of the size of the
databases for the index. Because indexing now only takes between 3% and
5% of server CPU when it is running in a steady state to keep the indexes
up to date, Microsoft now enables indexing by default. From a user perspective, searches return within seconds rather than minutes, so there is a
reduced desire to use a desktop search albeit recognizing that desktop search
engines remain useful when PCs operate disconnected from the network.
Exchange 2007 uses two services to perform indexing.
The Exchange Search Indexer is responsible for creating and maintaining the search index. It monitors the Store for new items and
invokes the necessary processing to update the index. For example:
The Indexer adds any new items that arrive into mailboxes
through email delivery or item creation.
The Indexer scans the complete mailbox if a new mailbox is added
to a database. This step ensures that any mailbox moved to a server
has its contents incorporated into the index as soon as possible.
The Indexer scans new databases that are added to the Store.
The Indexer is responsible for locating new items for indexing.
When it discovers items that have not been indexed, it sends sets of
pointers to these items to the MS-Search engine, which is responsible for indexing the content. Exchange creates a separate catalog
folder that contains multiple index files for each mailbox database
and the index files occupy between 5% and 10% of the size of the
mailbox database.
To retrieve the content, the MS-Search engine uses a special protocol handler to fetch the items from the Store and then pours the
content through a set of filters that can understand specific formats
such as Word, PDF, HTML, PowerPoint, Excel11, and so on (at the
time of writing, the new Office 2007 formats are not supported—
Microsoft may address this omission in Exchange 2007 SP1). It is
possible to customize indexing by adding new filters for new formats,
but if you add a new filter (which requires you to buy in the filter
See the registry value HKLM\Software\Microsoft\Exchange\MSSearch\Filters for the set of file types supported by
Chapter 5
5.12 Exchange 2007 content indexing
code or install a new service pack or hot fix), Exchange will only
index new content in that format unless you recreate the entire index.
The output of the filters is plain text content, which MS-Search
then processes using a set of word breakers that identify the individual words in the content. The word breakers are able to handle content in different languages, including Japanese and Chinese. One
possible complication for content indexing arises when documents
are in one language but marked as being in another. For example, if
you create an English-language document on a PC that runs the
Dutch version of Word, Word marks the document as Dutch. Then
content indexing applies its Dutch word breakers and you cannot
guarantee that the words that are discovered will make sense for the
purpose of searching.
If you look at processes running on an Exchange server, you will see two
processes taking up CPU and memory to perform indexing. MSFTESQL.EXE is the core indexer, while MSFTEFD.EXE is a filter daemon that
pours content from the Store through the filters and word breakers. Apart
from a radical improvement in performance, the biggest difference in the
implementation is the move from a “crawl” model (periodic updating of the
index) to an “always up-to-date” model, which means that the Store indexes
new items automatically as users create them in the database. A new message
is usually indexed within 10 seconds of its arrival into an inbox. Of course,
because of the increase in data cache enabled by the 64-bit platform, new
items can be kept in the cache long enough for them to be indexed, so indexing generates no additional I/O, and new items appear in indexes very soon
after they are created.
An Exchange server can encounter various situations that cause a mailbox
database to undergo automated or manual maintenance and some of these situations can provoke the need for Exchange to rebuild the content indexes.
Table 5.9 summarizes these situations. Note that Exchange does not automatically include the catalog folder in online backups taken using the Exchange
backup API, so this is the reason why a full rebuild of the index is required if
you have to restore a database from backup. If you take a file-level backup
with acquiescent databases, you have the choice to include other files along
with the databases and can include the catalog folder. In this scenario, if you
have to restore the databases after a failure, you can restore the catalog folder
too and so avoid the need for an index rebuild.
Performing a full crawl of a server to populate an index from scratch
consumes a lot of CPU and memory and can generate a lot of I/O activity.
Depending on the other work that the server is doing, crawling has the
5.12 Exchange 2007 content indexing
Table 5.9
Effect of database operations on indexing
Online backups
Backup taken with VSS or
streamed to tape.
Re-index required if a database
is restored from a backup.
File level offline
Offline backup to disk or tape.
Index data can be backed up
and restored (without requiring
an update) along with mailbox
Fast recovery
Dial-tone recovery using
blank mailbox database followed by recovery of the failed
Exchange creates a new index
for the blank database and reindexes once the failed database
is recovered.
Log shipping and application
of transactions to local copy of
a mailbox database.
Exchange creates a new index if
you have to switch over to the
passive copy.
Switch to copy of the database
on a passive node in a MNS
Search continues to work after
transition. For lossy failovers,
the index is okay and will be
updated provided that the loss
duration is less than seven days.
Switch to passive node (shared
copy cluster).
Because only the active node is
indexing, the transition causes
Exchange to rebuild the index.
potential to disrupt email throughput. To avoid this problem, indexing
throttles back its processing demands in periods when server load is high.
During normal indexing, the indexer regulates the load that it places on the
server by controlling the number of items that it attempts to process per unit
of time. A monitoring thread keeps track of the load on each mailbox database by measuring the average latency required to retrieve a document from
the database. If the latency crosses a specific threshold (20 milliseconds is the
default value), then the indexing engine reduces the load on the server by
delaying requests to retrieve information from the Store. Before attempting
to fetch a document from the Store, the indexer checks the current processing delay and if it is larger than zero, the indexer goes to sleep for a short time
before attempting the fetch. This kind of throttling only occurs during full
crawls as the immediate updates to maintain the index do not impose significant load on a server.
Interestingly, content indexing does not index items that Outlook’s junk
mail filter detects as spam and moves into the Junk E-Mail folder. However,
if you find that a message has been wrongly detected as spam and you move
Chapter 5
5.12 Exchange 2007 content indexing
it from the Junk E-Mail folder to any other folder, content indexing will
detect the move and include the message in its index.
Using content indexing
Content indexing is configured by default for all databases, so no work is
required to set it up. You can check on the current indexing status for all
mailbox databases on a server with the Get-MailboxDatabase command:
Get-MailboxDatabase –Server ExchMbxSvr1 | Select Name,
The databases that are shown with IndexEnabled = True are obviously
those that Exchange is currently indexing. To disable content indexing temporarily, you use the Set-MailboxDatabase command:
Set-MailboxDatabase ‘VIP Mailboxes’ –IndexEnabled $False
Remember to reenable content indexing later on by setting the IndexEnabled property to “True.” If for some reason you want to disable content
indexing for all databases on a server, you can either set the IndexEnabled
flag to False with the one-line shell command:
Get-MailboxDatabase –Server ExchMbxSvr1 | Set-MailboxDatabase
–IndexEnabled $False
Alternatively, you can stop the “Microsoft Exchange Search Indexer” service. Users will experience a radical reduction in performance if they conduct
online searches when the indexing service is unavailable, especially when
searching through large folders, and they will only be able to search message
subjects rather than content. Outlook Web Access signals the potential performance degradation with a pop-up, as shown in Figure 5.24.
Figure 5.24
Content indexes are
not available
5.12 Exchange 2007 content indexing
Another interesting shell command allows you to test the performance
of a search operation for a specific mailbox (and the underlying mailbox
database). You need to have the permissions to write into the mailbox that
you want to test:
Test-ExchangeSearch ‘Tony Redmond’
Exchange responds to the test by reporting whether it was able to perform a search and the search time. A result of 10 or less for the search time is
acceptable. A reported search time of –1 indicates failure!
Figure 5.25
Building a complex
search with
Outlook 2007
Now that we understand how to enable and test content indexing, we
need to understand when it is used. If the mailbox database that hosts the
mailbox is enabled for content indexing, all searches executed by Outlook
Web Access clients connected to mailboxes in the database use content
indexing. This is easy enough to understand because Outlook Web Access
works in online mode. Things get a little more complicated with Outlook
because it supports online and cached Exchange modes.
When Outlook clients (any version from Outlook 2000) operate in
online mode, the vast majority of searches use content indexing providing
that the mailbox database is indexed and the Exchange Search service is
enabled for the database. In addition, the search query must contain fields
Chapter 5
5.12 Exchange 2007 content indexing
that are actually indexed. Most fields are indexed, so this is not usually a
problem. However, some queries exist that content indexing cannot satisfy,
especially those that invoke complex combinations of NOT, AND, and OR
checks against fields or any query against a field that is not indexed, such as a
date field. In these cases, Outlook reverts to a Store search. Figure 5.25 shows
the Outlook search builder in operation—you can build very complex queries to narrow in on the right information, but it’s also probably as easy to
confuse users.
Outlook 2003 and 2007 clients that operate in cached Exchange mode
use client-side linear scans similar to those performed by the Store. Performance is often better than Store searches, but these clients never use
Exchange content indexing. You can, of course, install an add-in product
that indexes Outlook data and use that to achieve better performance.
Windows Desktop Search uses the same basic indexing and search technology as SharePoint and Exchange, but obviously on a less massive basis.
Windows Desktop Search is included in Windows Vista and can be downloaded and installed on Windows XP clients and controlled through Group
Policy. On a purely unscientific basis, it seems that Windows Desktop Search
functions more smoothly under Vista than Windows XP because it seems to
place less strain on the operating system when it indexes items.
Figure 5.26
Outlook 2007
Search Options
Figure 5.26 shows the options that you can configure with Outlook
2007 to control how search operates. Outlook 2007 clients that operate in
5.13 Public folders
cached Exchange mode do not use server-based content indexing and use
Windows Desktop Search instead. Windows Desktop Search relies on the
cached copy of the mailbox plus any PSTs to include messaging data into its
index. Note that it is possible for a client to generate searches using
Exchange-based indexes and local indexes. For example, if you run Outlook
2007 in online mode and execute a search across your mailbox and a PST,
Outlook uses Exchange content indexing to search mailbox content and
Windows Desktop Search to search the PST.
5.13 Public folders
Public folders have been part of Exchange since the product’s earliest days.
The best definition that I can offer is that a public folder is a special form of
shared mailbox that Exchange manages within a public folder hierarchy.
Unlike mailboxes, which users can only access by connecting to a specific
server that hosts the desired mailbox, Exchange can replicate the hierarchy
and content of public folders to multiple servers within the organization so
that users can access the data from their local server rather than connecting to
a remote server. Exchange synchronizes modifications made to local folder
replicas to ensure that data remains constant throughout the organization.
During the 1995–96 period, Microsoft often referred to public folders
as “public fora” to indicate their intended use as a fulcrum for discussion and
debate. Microsoft also emphasized the potential of public folders to be an
application development platform through electronic forms that operated on
the data stored in public folders. The theory was that you could generate the
electronic forms necessary to automate common tasks such as expense processing by creating a suitable form to provide users with the ability to work
with data in a public folder, taking advantage of the replication facility to
keep information up to date across an organization. Microsoft’s vision of
what public folders could do was sophisticated and many demo applications
probed the possibilities of public folders. In fact, there was even a chess application that users could play across the network by entering moves through an
electronic form. Exchange stored the moves in a public folder and replicated
them between servers so that players could see the latest state of the game and
update their positions. Sadly, the limited nature of the 16-bit Electronic
Forms Designer (EFD), the lack of development from the initial launch, and
the black box that public folder replication became all conspired to block
public folders from fulfilling the grand plan. Public folders slipped back to
take up more mundane tasks such as providing a rudimentary shared document repository, archiving messages that users sent to distribution lists, and
so on.
However, while it was easy to create new public folders and to populate
the folders with data, companies found it more difficult to manage the prolifChapter 5
5.13 Public folders
eration of public folders if they allowed users to create folders without control. The result is that some companies that deployed Exchange early have
ended up with hundreds of thousands of public folders. Exchange management consoles have never provided great insight into what public folders are
used for or what they contain. In some cases, public folders may hold very
important corporate information, but it’s difficult to be sure unless you go
and check every folder (which is a very difficult task if you have thousands of
folders to review). Understanding what data exists in public folders is an
important task for administrators as they contemplate the long-term future
for this repository and that leads on to the next task, which is to decide where
to store the data in the future.
Their relative lack of success has caused Microsoft to consider the future
of public folders many times over Exchange’s history. Microsoft attempted to
reignite the potential of public folders by introducing support for multiple
top-level folder hierarchies (TLHs) in Exchange 2000, with the intention
that the standard hierarchy (called the MAPI TLH) would continue to host
“standard” public folders while programmers would create and use new
TLHs to host applications. Unfortunately, a lack of programming capabilities to exploit public folders did not accompany the new folder hierarchies,
so they never succeeded. Given the lack of success, a lack of tools to manage
public folders effectively, and the advent of other Microsoft solutions for collaboration such as mail-enabled document libraries in SharePoint 2007, you
can see why Microsoft might want to move their products and customers
away from public folders as soon as they can.
Public folders and Exchange 2007
In an Exchange 2007 environment, you only need public folders for two
To support legacy Exchange 2003 or 2000 servers.
To support legacy Outlook clients (anything pre-Outlook 2007).
To give customers some comfort about the future of public folders,
Microsoft states that they will fully support Exchange 2007 for 10 years after
first product release, so you can continue to use public folders with Exchange
2007 until 2016. However, you have to put this statement into context with
Microsoft’s overall support strategy (see
default.aspx?pr=lifecycle), which limits mainstream support to 5 years after
which a product moves into extended support. Exchange is no different to
other Microsoft business and developer products and 10 years is a reasonable
time horizon to plan for an application. However, when all is said and done,
5.13 Public folders
the fact remains that Microsoft is still “de-emphasizing” public folders in
Exchange 2007, which means that Microsoft will continue to look for an
opportunity to move public folders out of one of the next major releases of
Exchange. Originally, Microsoft intended to remove public folders from the
next major release of Exchange (codenamed Exchange 14), probably due in the
2009–10 timeframe. Customer pushback and the lack of mature and easily
attainable migration targets forced a rethink and so it is now likely that we will
not see public folders fully removed from Exchange until a release in the 2011–
13 timeframe, so there is plenty of time to prepare for a migration. Plans may
easily change in this area, so it’s something to keep an eye on.
While it might be desirable, Microsoft cannot eliminate public folders
if they want to maintain a high degree of backwards compatibility for
Exchange. For example, all MAPI clients up to and including Outlook 2003
require a connection to a public folder server to access components that
these clients depend on, including the offline address book, free/busy data,
Outlook security settings, and the organization form library. When you
install the first Exchange 2007 server, the setup program prompts you to
indicate whether you want to support legacy Outlook clients (anything
before Outlook 2007). If you respond that you have legacy clients, the setup
program creates a public folder database and populates the hierarchy with
the public folders that these clients expect. If you say that you have no legacy clients, the setup program does not create a public folder database and
it sets up a block to prevent legacy clients connecting to the Store. Basically,
Exchange checks for the presence of a public folder hierarchy when the
Information Store service starts and if it can’t find the hierarchy, the Store
blocks any attempt by a legacy client to log on to a mailbox. Legacy clients
can’t really function effectively without public folders (you can create and
send mail, but many features are crippled by the lack of public folders), so it
makes sense to block access.
Microsoft designed Outlook 2007 to work in a new manner with
Exchange 2007, so these clients do not require access to public folders
because they incorporate different mechanisms to get to shared data such as
free/busy information. When connected to Exchange 2007 servers, Outlook 2007 clients can use a BITS (Background Intelligent Transfer Service)12 connection to a web distribution point on an Exchange 2007 Client
Access server to fetch a copy of the Offline Address Book; they access free/
busy information through the Exchange 2007 availability web service, and
retrieve Outlook security settings from the registry. As the first step in
phasing out public folders, organizational forms are not available in
Exchange 2007, so it is time to move from these forms to something more
modern, such as InfoPath. All of this means that you have to wait until you
can upgrade your MAPI clients to Outlook 2007 before you can even think
See for more information on BITS.
Chapter 5
5.13 Public folders
about a retirement strategy for public folders. See Chapter 7 for more
information about how Outlook 2007 interacts with Exchange 2007 when
public folders are not present.
Microsoft offers customers some guidance about what to do with public
folders and you can expect the story to mature and develop as we gain experience with Exchange 2007 and people experiment with long-term alternatives
for public folders. At the time of writing, Microsoft recommends that you
migrate to Exchange 2007 and Outlook 2007 as quickly as possible to
remove the dependency that older Outlook clients have on public folder
servers. Microsoft also recommends that customers develop new forms-based
workflow applications with the .NET framework. Existing applications that
use a combination of Outlook forms and public folders will continue to
work as long as you keep the organizational forms library in a system public
folder. Finally, you should keep a close eye on developments as Microsoft figures out how best to move data from public folders to new repositories.
If you decide that Exchange 2007 is a good time to move from public
folders, you can consider Office SharePoint Server 2007 or its predecessor
(for example, a SharePoint site makes a pretty good archive location for messages sent to an Exchange distribution list). This seems to be the logical path
forward from public folders because SharePoint offers much better document
management and search features, and SharePoint has a solid development
plan that has delivered real progress since SharePoint Portal Server 2001. In
addition, SharePoint leverages the .NET development platform, so you can
be confident that any application that you build around SharePoint will use
the latest programming techniques and libraries. Even when you select a new
target platform, you still have to figure out how to move the data that has
accumulated in public folders. One approach is to leave the data where it is
and plan to obsolete the data gradually. In other words, do not create any
new public folders and redirect data away from public folders to a new repository whenever possible. You can also create a review and retirement program
for public folders with the aim of eliminating as many folders as you can in
the short term.
Changes in public folders administration since
Exchange 2003
In some ways, maintaining public folders is a case of Microsoft applying
enough oxygen to the patient to keep it alive until new software rescues the
situation. Some manageability improvements appeared for public folders in
Exchange 2003 SP2, including:
The ability to stop and resume content replication.
5.13 Public folders
The ability to apply delta changes on a recursive basis throughout the
public folder hierarchy.
The ability for an administrator to decide to synchronize the public
folder hierarchy.
Better control over the removal of servers that hosted public folders
from an organization.
Logging of public folder deletions.
It can be argued that most if not all of these features should have been
included in Exchange from the first release and that their omission led to
some of the problems that have resulted in the bad reputation that public
folders enjoy today. In the same timeframe as SP2, Microsoft released the
PFDavAdmin13 tool to help administrators understand the data that their
servers store in public folders. This tool closed a gap in the product that had
existed since Exchange 4.0. PFDavAdmin is a useful tool in planning a
migration because it gives you the hard data to analyze just what content you
have to deal with. Running the tool is not difficult and it completes quickly.
For example, a run against HP’s Exchange infrastructure took an hour to discover and report on 160,000 public folders. This may seem like a large number of folders, but HP is similar to many companies that have run Exchange
since 1996 and deployed public folders as the basis for applications or data
sharing, often because it was a better answer than file shares. The interesting
item that PFDavAdmin allows you to identify is the date when content last
changed in a public folder. This is useful data because it indicates whether
the folder is still in use. If someone has not accessed a folder and updated it
in two years, the folder is probably not in use and you can delete it safely. See
Chapter 10 for more details on the PFDavAdmin tool.
If you decide to move away from public folders and use a tool like
PFDavAdmin to analyze and then clean up the public folder hierarchy, you
can figure out what to do with the remaining public folders. Some advocate
leaving the data in the public folders to decay gently over time by not
encouraging users to use public folders and by preventing creation of new
public folders. The theory here is that you will eventually be able to remove
all of the public folders, providing you are willing to wait long enough. A
more proactive approach is to look for a new repository, and Windows SharePoint Services and SharePoint Portal Server are good solutions that can take
on many of the functions public folders serve today. Forms-based applications that use public folders remain an issue, but Microsoft proposes InfoPath forms as the way forward here. The problem with forms-based
DAV stands for “Distributed Authoring and Versioning” and is an extension to HTTP that Microsoft uses extensively with
Outlook Web Access in Exchange 2000 and 2003.
Chapter 5
5.13 Public folders
applications is the need to recode the forms and application logic and not the
data, which may need a one-time conversion from its current home in a public folder to a new repository. Finally, you need to consider network connectivity for users to the data and applications. Public folders use replication to
get data and applications close to the user, but solutions like SharePoint Portal Server do not have the same replication capability, so you are likely to end
up bringing users over extended network links to a central server. Given that
today’s network connections are probably more capable than those that
existed when you originally deployed the application, this may not be an
issue, but it may still be a problem for users in branch offices at the end of a
hub and spoke network.
Calming replication storms
Public folder replication is a black art at the best of times and lack of knowledge about what happens to replicate content around an organization has led
to many horror stories, such as the administrator who replicated the entire
set of public folders from New York City to Australia for a very large financial
institution. Replication took forever and it took even longer to rehome the
public folders back to the servers where they should have been. Few
Exchange system administrators understand how and when replication
occurs and the Exchange administrative tools, especially the Exchange 2003
ESM console, are weak in this area, perhaps because Microsoft realized that
public folders were not as successful as they had originally anticipated in the
early days of Exchange and that there were other more usable repositories,
such as SharePoint Portal Server. From Exchange 2000 onwards, Microsoft
knew that they would eventually move away from public folders and so never
gave this component the attention that other parts of Exchange have
received. Overall, it is difficult to get a good picture of public folder replication to know when processing is proceeding as expected and when problems
occur. Some companies have put together their own early warning systems
based on data extracted from the application event log and WMI counters,
but it is not a satisfactory situation, especially as the Exchange 2007 management console does not include any ability to manage public folders, so you
have to manage public folders either through EMS or by keeping an
Exchange 2003 server in the organization so as to be able to use its console.
Neither approach is completely satisfactory.
There is a small but significant feature in Exchange 2003 SP2 that can
help administrators to manage the phenomenon known as a “public folder
replication storm.” Replication storms occur when an uncontrolled or unexpected amount of replication activity is generated within the organization so
that network links are swamped with the amount of data that public folder
servers replicate to update the folder replicas on these servers. The typical
cause of a storm is a change that an administrator makes that triggers the rep-
5.13 Public folders
lication of many folders and items in those folders. Sometimes a replication
storm is the result of a planned activity, such as when you add a new public
folder server to the organization and need to replicate a large amount of data
to that server in order to populate its set of folder replicas. Other times the
storms occur by accident, such as when an administrator creates a new replica of a very large folder at 9AM on Monday morning, right in the middle of
peak-time user demand when the servers are busy handling requests by Outlook clients to refresh their caches and process mail created offline. Sometimes replication storms occur without your knowledge and without any
impact to the messaging infrastructure. This is especially true when the public folder servers that participate in replication are connected through highspeed, low-latency network links that can handle the peak demand generated
by a limited replication storm.
You can control public folder replication storms to some degree by taking actions such as starting replication at a time when you know that the
network is quiet and no other major activity is planned. Storms only cause
problems when they are unplanned, especially if a large amount of data is
channeled across an unreliable or slow network link or a link that is high
latency. Bugs in Exchange have been known to cause replication storms in
the past, but Exchange 2000 SP2 and later versions are pretty reliable in this
respect and most storms are now caused by administrative error, such as the
scenario around the creation of a large public folder described above. Of
course, you can argue that if Exchange offered better management facilities
for public folders (including the ability to monitor and throttle replication
as required) then administrators would not make mistakes, and while this
assertion is probably true, even the best administrator makes mistakes sometimes. For example, in a moment of weakness, you might create a new replica of a very large folder on a server that’s at the end of a high-latency link.
The intention is good—you probably want to allow users of that server to
enjoy local access to the folder data—but the effect might be a replication
storm. Exchange 2003 SP2 includes a public folder replication stop/start
feature to allow you to address the problem. You can stop replication across
your entire organization, fix the problem by reversing the changes that you
made (in this case, remove the folder replica from the remote server), and
then resume replication.
Microsoft implemented the ability to stop public folder replication as
an organization-wide setting. This is logical because you want to stop replication on all servers. The options to stop and restart public folder replication are therefore revealed through the organization object in ESM. As
shown in Figure 5.27, select the organization object and right click to see
the option to stop public folder replication. This is the normal state of
affairs as ESM only displays the option to resume replication when a stop
operation is in effect. As we will see in a little while, EMS provides comChapter 5
5.13 Public folders
Figure 5.27
Exchange 2003
ESM option to stop
a replication storm
mands to allow you to stop and restart replication through the shell, which
has the same effect as the ESM commands.
If you click on the “Stop Public Folder Content Replication” option,
ESM displays a warning to ensure that you understand the effect of the
action that you are about to launch, which is to stop content replication
between all of the public folder servers in the organization. It is critical to
underline the organization-wide impact of your action before you proceed.
Clearly you don’t want every administrator in the organization clicking on
this option just to see what it does, so you need Exchange Organization
Admin permission to be able to execute the option to stop a replication
storm. If an unprivileged administrator attempts to execute the option, ESM
issues an “LDAP Provider issue” message, which is ESM-speak to say that the
user doesn’t have the right permissions.
Assuming that you have the right permission, ESM proceeds by setting
an organization property to indicate that public folder content replication is
disabled and then sends a “cease and desist” message to every server that hosts
a public folder database in the organization. The cease and desist message
stops servers responding to backfill requests from other servers. Each backfill
request asks a server to provide some data to update a public folder replica on
a remote server, so if the server stops responding to backfill requests, it has
the effect of nullifying any replication requests that the server would normally handle. Servers still send out backfill requests to request other servers
to provide backfill updates for their replicas, but the receiving server never
answers the requests. Of course, it is only as the servers receive and action
5.13 Public folders
this message that replication gradually decreases and eventually stops. Note
that changes to the public folder hierarchy are still replicated around the
organization. This is because hierarchy updates go to every server that hosts a
public folder database, are relatively tiny in terms of the data that these
updates contain, do not contain folder content, and are necessary to allow
administrators to adjust the replica lists to stop the replication storm recommencing when replication restarts.
Servers that run versions prior to Exchange 2003 SP2 do not understand
the “cease and desist” message or the organization setting that prevents content replication, so these servers continue to respond to backfill requests. The
non-SP2 servers also continue to broadcast details of changes that occur to
the public folder replicas that they host. Your ability to suppress a public
folder replication storm effectively is therefore linked to the percentage of
Exchange 2003 SP2 servers in your organization—the higher the percentage,
the more effective you will be.
You can verify that Exchange 2003 SP2 servers that host public folders
are not responding to replication requests by checking for event 3118 in the
application event log on these servers. The Store logs event 3118 whenever a
backfill request arrives that the Store cannot respond to because of the organization setting to suppress public folder replication.
A replication storm is bad but you do want public folder replication to
proceed when network conditions are stable. Without replication, functions
such as free/busy information won’t work properly because servers will be
unable to replicate data between each other. After replication has been
halted, you can start replication again by clicking on the organization object
in ESM and selecting the “Resume Public Folders Content Replication”
option. You can only see this option after you have successfully issued a stop
replication command. ESM will then display a dialog to ask you to confirm
that you want to proceed to restart replication and if you confirm that replication should resume, ESM sends a message to public folder servers to tell
them to start replication again. Servers resume replication according to their
usual replication schedule, so if you have configured servers to replicate data
once a day at midnight, they will not resume replication until that time
arrives. Before you restart replication, you should ensure that you have
addressed any problems that caused the original replication storm.
When replication begins again, the Store logs event 3119 in the application event log to indicate that replication has been reenabled. Replication
should quickly settle down into its normal pattern in small organizations,
but large distributed organizations that have more than a few public folder
servers may take several hours to resume normal service. It is conceivable
that very large organizations or those that depend on high-latency or lowbandwidth links to reach some servers may take some days before replication settles down.
Chapter 5
5.13 Public folders
The fact that we have an option to control replication storms is nice, but
you can’t ignore the fact that public folders have a short expected lifetime left
within Exchange. With this fact in mind, perhaps you should use the introduction of Exchange 2007 to implement a policy that prohibits the creation
of any new replica folders (which will effectively halt the potential for future
replication storms) and direct the data that might be held in public folders to
another repository.
Managing public folders with Exchange 2007
Compared to the ability to manage public folders in the Exchange 2003
ESM, the options presented by the first version of the Exchange 2007 EMC
are sparse and limited. You can create a new public folder database if a server
does not already have one (you can only have one public folder database per
server). You can mount and dismount a public folder database or move database files, and you can amend some of the properties of the public folder
database as shown in Figure 5.28 to determine a replication schedule and set
limits on the database. But there’s no ability to create new replicas or to navigate through the folders in the database or to examine how much storage
each folder occupies or to see the items in the folder.
Figure 5.28
Public folder
database properties
in Exchange 2007
If you want to gain an insight into the content of the public folders
rather than just knowing what folders exist in the public folder hierarchy, you
have to resort to Exchange 2003 ESM or use an Outlook client to see the
information. Figure 5.29 shows what you can expect to see from Outlook
when you view folder sizes. Note that Microsoft provides no interface to
access public folders in Outlook Web Access 2007, nor can you access public
folders using IMAP or POP clients.
5.13 Public folders
Figure 5.29
Viewing public
folder sizes from
Outlook 2003
It is common knowledge that Microsoft has deprecated the use of public
folders within Exchange, so the lack of support for managing public folders
through EMC shouldn’t come as a complete surprise, nor should the inability to work with public folders through Outlook Web Access 2007.
Microsoft plans to introduce support for public folder management in
Exchange 2007 SP1 and this will be a welcome relief to administrators. Until
then, if you operate a mixed organization where you have some Exchange
2003 servers, you can continue to use ESM to manage public folders. You
can even argue that you should keep one Exchange 2003 server for this purpose until you migrate off public folders to another repository and there is
some sense in this approach. However, if you move to a pure Exchange 2007
organization and retain public folders, you have to be prepared to manage
public folders through the shell. With this point in mind, we need to examine how to perform some of the most common public folder management
operations with EMS. Microsoft provides some sample scripts with the
Exchange 2007 installation kit14. You can use these scripts to work with public folders or as the basis to develop scripts that meet your unique needs.
Table 5.10 lists the sample scripts and their intended use.
While it is useful to have some sample scripts to work with, you still
need to understand the basic EMS commands that work with public fold14.
The default location for the example scripts is C:\Program Files\Microsoft\Exchange Server\Scripts.
Chapter 5
5.13 Public folders
Table 5.10
Sample public folder management scripts
Adds a server to the replication list for a
public folder, including all the child folders under the folder.
Removes a server from the replication list
for a folder, including all the child folders
underneath. If the server is the only one in
the replication list, the script leaves the list
Replaces a server in the replication list for
all public folders with a new server
(includes system folders).
Replaces a server in the replication list for
a folder with a new server, including all
the child folders underneath. If the server
that you want to replace is not listed,
nothing is changed.
Adds a user to a folder’s permission list
including all child folders. If the user is
already listed, their permission is updated
to the set specified in the script.
Replaces a user with a new user on a
folder’s permission list, including all child
folders. Folders that do not contain an
entry for the specified user are not altered.
Replaces the permissions of a user with a
new set of permissions, including all child
Removes a user from a folder’s permission
list, including child folders. The “Default”
and “Anonymous” permissions cannot be
removed, although an attempt to remove
them reduces their permissions to
ers. The most basic command is to create a public folder database on a
server that hosts the mailbox role. You also have to have the necessary permissions to manage the target server. If a server does not have a public folder
database, you can create the database with the New-PublicFolderDatabase
5.13 Public folders
command. This command creates a new public folder database in the
default Storage Group.
New-PublicFolderDatabase ‘Public Folders’ –StorageGroup
‘First Storage Group’
To validate that the new database exists, you can use the Get-Publiccommand. Type the command without a parameter to see
all the public folder databases that exist within the organization. By default,
Exchange does not return the names of public folder databases that reside on
Exchange 2003 servers, so if you want to see this information, you have to
specify the –IncludePreExchange2007 parameter as follows:
Get-PublicFolderDatabase –IncludePreExchange2007
You can pass the name of a specific public folder database to retrieve more
information about the database. The best way to do this is to include the format-list command. In this instance, we pass the display name of the public
folder database as you see in EMC (see Figure 5.30), so the command is:
Get-PublicFolderDatabase –id ‘Public Folders’ | Format-List
If you want to specify a public folder database on a server, include the
server name in the –identity parameter.
Get-PublicFolderDatabase –id ‘ExchSvrPF1\Public Folders’ |
Now that we have a public folder database, we can populate it with folders using the New-PublicFolder command. For example:
New-PublicFolder ‘\Exchange 2007 Discussions’ –Path
‘\Discussions’ -Server ‘ExchSvrPF1’
This example creates a new public folder called “Exchange 2007 Discussions” under the “Discussions” top-level public folder and hosts the initial
replica on the server called ExchSvrPF1. The root folder must exist on the
target server. After you create the public folder, you can check its presence
with the Get-PublicFolder command.
Chapter 5
5.13 Public folders
Get-PublicFolder –id ‘Exchange 2007 Discussions’
Alternatively, you can check all of the child folders that exist under a
root folder. For example:
Get-PublicFolder –id ‘\Exchange 2007 Discussions’ –Recurse
In this example, we are working with a top-level public folder immediately under the root. It is unlikely that you will be in this situation as toplevel public folders probably exist already and you will not be in a hurry to
create more. It is more likely that you will work with folders some way down
the public folder hierarchy and will therefore have to specify the full path to
the folder in order to work with its properties. For example:
Get-PublicFolder –id ‘\Top\Second\Third\Fourth\Fifth’
Get-PublicFolder and other commands that manipulate folder properties execute reasonably quickly because they interrogate the public folder
hierarchy. Exchange replicates a copy of the public folder hierarchy automatically to every server in the organization that hosts a public folder database, so you are always working with local data when you retrieve
information about a folder. The ability to scan through a complete copy of
the hierarchy comes in handy if you need to locate a public folder but you
do not know where exactly it is. For instance, assume that you need to find a
public folder called “Critical Documents.” You can use a command like this
to scan from the root folder downwards to locate every folder that matches
the folder name.
Get-PublicFolder \ -Recurse | Where {$_.Name -eq ‘Critical
Like many of the EMS commands, you could consider the output of
be pretty verbose. Here is an edited extract that shows
the presence of two child folders under the \Discussions top-level folder:
Get-PublicFolder to
Exchange 2007 Discussions
5.13 Public folders
\Discussions\Exchange 2007 Discussions
Exchange 2007 User Discussions
\Discussions\Exchange 2007 User Discussions
If a folder has many child folders, it is better to be selective in the data
you output:
Get-PublicFolder –id ‘\Exchange 2007 Discussions’ –Recurse | Select Name,
---Exchange 2007 Discussions
Exchange 2007 Admin Discussions
Exchange 2007 User Discussions
Exchange Management Console
Public Folders
Routing and Transport
Windows 64 platform
Discussions\Exchange 2007 Admin...
Discussions\Exchange 2007 User ...
Discussions\Exchange Management...
Discussions\Public Folders
Discussions\Routing and Transport
Discussions\Windows 64 platform
In very large organizations, it is possible that a call to Get-PublicFolder
that uses the –Recurse or –GetChildren parameters to force EMS to list all of
the folders that exist under the specified root will result in the output of a
Chapter 5
5.13 Public folders
huge number of folders. By default, EMS limits the maximum number of
folders returned to 10,000, but you can override this by passing the desired
number of folders to output through the –ResultSize parameter. From the
output reviewed so far, you may have noticed that the Get-PublicFolder
command does not return information about the servers that host the databases where the public folders are stored. You can get at the data, but only if
you dig through the full set of folder properties. Given that the name of most
public folder databases is “Public Folders” and they are usually located in the
default “First Storage Group,” you can see that it sometimes can be difficult
to find the public folder that you really want. It would have been nice if
Microsoft had given the Get-PublicFolder command the ability to be more
obvious about where the folders are located, but maybe this option will come
in the future.
Exchange uses two types of public folders: system and nonsystem.
Exchange uses system folders to store information such as the Offline
Address Book and free/busy information. You can view system folders by
specifying the Non_IPM_Subtree root. For example:
Get-PublicFolder -Identity \NON_IPM_SUBTREE -Recurse |
Format-List Name
As we examine the properties of public folders, we may find that we
need to change some of those properties. For example, we may want to mailenable the folders (so that users can create new items in the folder by email,
something that often happens when you use a public folder as a repository
for discussions). We may also want to impose quotas on the folders. To mailenable a public folder, use the Enable-MailPublicFolder command:
Enable-MailPublicFolder ‘\Exchange 2007 Discussions’
You can check that the command executed correctly with this command:
Get-PublicFolder –Identity ‘\Exchange 2007 Discussions’ |
Select Name, MailEnabled
To reverse the process, use the Disable-MailPublicFolder command.
You can use the Get-MailPublicFolder command to view all of the public
folders that are mail-enabled. This command is likely to return many items
and the data includes the set of system folders that are mail-enabled.
Get-MailPublicFolder | Select Name
5.13 Public folders
You can also execute this command for an individual folder if you know
its name:
Get-MailPublicFolder ‘\Exchange 2007\User Debates’ | FormatList
While you should see that the value of the MailEnabled property for
the folder is “True,” one problem with mail-enabling a public folder with
Exchange 2007 is that the details returned by the Get-PublicFolder command do not include the email address. You are therefore in the dark as to
the right address to use to send items to the public folder unless you examine the folder properties with ESM or through the GAL. If you know the
Email Address Policies in effect, you can guess as to what email address
Exchange stamps on the public folder. In this case, it was Exchange_2007_
[email protected]! You may not want mail-enabled public folders to
appear in the GAL, so you would set another property to effect this option:
Set-PublicFolder –id ‘\Exchange 2007 Discussions’
–HiddenFromAddressListsEnabled $True
Remember that you can always use email to post an item to a public
folder even if it does not appear in the GAL by using its SMTP address to
send messages. Some administrators have a practice of adding a mailenabled public folder to a distribution group through Outlook while the
folder is still listed in the GAL and then to hide it to prevent users inadvertently sending new messages to the folder. The hidden public folder
remains in the group thus serving the purpose of acting as a repository for
any message sent to the group.
Public folder permissions
By default, Exchange assigns owner permission to the user who creates a public folder. You probably need to assign permissions to other users to allow
them to add information to the folder and you can do this with the Add-PublicFolderClientPermission command. Here is how to set two different
types of permission on a public folder. The second command assigns the contributor permission to a distribution group, which is the permission that you
need to set if you wanted to use the public folder as a repository for messages
sent to the group.
Chapter 5
5.13 Public folders
Add-PublicFolderClientPermission –id ‘\Exchange 2007
–AccessRights ‘Owner’ –User ‘James Bond’
Add-PublicFolderClientPermission –id ‘\Exchange 2007
–AccessRights ‘Contributor’ –User ‘Legal Executives’
You can check permissions with the Get-PublicFolderClientPermis-
sion command:
Get-PublicFolderClientPermission –id ‘\Exchange 2007
If you make a mistake, you can use the Remove-PublicClientFolder-
Permission command:
Remove-PublicFolderClientPermission –id ‘Exchange 2007
–AccessRights ‘Contributor’ –User ‘James Bond’
Note that EMS prompts you to confirm this action before it will remove
the permission. Apart from assigning permissions for clients to work with
public folders, you may also want to assign permissions for users to manage
public folders and perform operations such as creating new replicas, changing the deleted item retention period, or modify the storage quota. Exchange
provides the Add-PublicFolderAdministrativePermission command for
this purpose. For example, assume that you want to give a user full permission to alter any property related to the Store for a folder:
Add-PublicFolderAdministrativePermission –User Bond –id ‘\
Exchange 2007’ –AccessRights AllStoreRights
The AllStoreRights access option does not allow the user to assign
administrative permissions to another user. If you want to give them full
administrative permission, you use the –AccessRights AllExtendedRights
option. To check what management permissions exist on a public folder:
Get-PublicFolderAdministrativePermission –id ‘\Exchange 2007’
To remove an administrative permission from a public folder, use the
5.13 Public folders
Remove-PublicFolderAdministrativePermission command:
Remove-PublicFolderAdministrativePermission –id ‘\Exchange
2007’ –User Bond –AccessRights AllStoreRights
Deleting public folders
You can delete a public folder with the Remove-PublicFolder command.
EMS will refuse to execute the command if you attempt to remove a public
folder that has child folders, so in this instance you need to specify the
Recurse parameter:
Remove–PublicFolder –Id ‘\Exchange 2000 Discussions’
As this command can wipe away valuable content held in the public
folders, EMS prompts for confirmation before it proceeds to delete the folders. Removing a folder also removes any replicas that exist, although the exact
point when Exchange has removed all replicas from all servers depends on
replication across the organization. You can delete folders on a remote server
by specifying its name.
Remove–PublicFolder –Id ‘\Exchange 2000 Discussions’ –Server
ExchPubPF1 –Recurse
Before you can remove a public folder server from the organization, you
usually have to make sure that replicas of the folders that the server holds
exist on other servers. If you proceed to delete without checking, you may
remove the only copy of a folder that exists within the organization, and that
may be a bad thing. Assuming that you have checked and that it is okay to
delete every public folder on a server, two commands will clean up the server.
The first command is to remove all of the nonsystem folders in the public
folder tree (starting from the “\” root); the second removes copies of any system folders such as the OAB or free/busy from the server. Note that we allow
Exchange to process more than 1,000 folders. We also instruct Exchange to
keep on processing if an error occurs when it deletes a folder.
Get-PublicFolder –id ‘\’ -Server ExchPubPF1 -Recurse
-ResultSize Unlimited | Remove-PublicFolder -Recurse
-ErrorAction SilentlyContinue
Get-PublicFolder –id ‘\Non_Ipm_Subtree’ –Server ExchPubPF1 Recurse
Chapter 5
5.13 Public folders
-ResultSize Unlimited | Remove-PublicFolder -Recurse
–ErrorAction SilentlyContinue
Before you delete the folders, you may want to execute just the Getcommand to check exactly what folders it finds. If there are
too many folders to check on the screen, you can create a report that lists the
folders together with the number of items in each folder, their size, and the
last access time with the following command:
Get-PublicFolder –id ‘\’ -Server ExchPubPF1 -Recurse
-ResultSize Unlimited | Get-PublicFolderStatistics | Select
Name, ItemCount, TotalItemSize, LastModificationTime > C:\
Setting limits on public folders
As we can see from Figure 5.22, public folders can accumulate a lot of data
and we do not want the folders to become a dumping ground where users
can store large files that would otherwise blow their personal mailbox quota.
The default that Exchange 2007 assigns for the maximum size of an individual item that a user can post to a public folder is very large (10MB). In addition, the size that an individual folder can grow to before Exchange starts to
issue warnings and then to prohibit further posts are very large at 1.9GB and
2.0GB, respectively. It is therefore a good idea to restrict these properties for
public folders to more reasonable levels. For example:
Set-PublicFolderDatabase –id ‘PUB’ -IssueWarningQuota 450MB
-ProhibitPostQuota 500MB -MaxItemSize 5MB
But what if we want to allow users to post large items to a specific public
folder? We can set limits that override the database default and double the
limits that apply to a specific folder as follows:
Set-PublicFolder –id ‘\Exchange 2007 Discussions’
–StorageQuota 990MB
–PostStorageQuota 1GB –MaxItemSize 10MB
If you examine the properties of the folder afterwards, you’ll see the new
quotas and that the UseDatabaseQuotaDefaults property is set to False.
5.13 Public folders
Managing public folders on a remote server
You are not limited to executing the Set-PublicFolder or Set-PublicFolcommands on a local server as EMS is quite happy to execute
commands on any server for which you have administrative permission.
Remember that if you want to execute commands for a database on a remote
server, you have to specify the full name of the database, including the name
of the server where the database resides. For example, if I want to set new
default quotas for the “PUB-2” database on server ExchPFSvr2, I would refer
to the database as “\PUB-2.” Here is another example
to prove the point. EMC presents no ability to create a new replica of the
public folder that we have just created and this is an operation that you could
never perform with Outlook, so we have to do it with EMS. A moderately
complex one-line command does the job:
Set-PublicFolder -id ‘\Exchange 2007 Discussions’ -Replicas
((Get-PublicFolder -id ‘\Exchange 2007 Discussions’).Replicas
+ ‘\PUB-2’)
In this case, we retrieve the current set of replicas with Get-PublicFolder and then call Set-PublicFolder to add the new replica to the list and
you can see how we specify the target public folder database to host the new
Of course, creating a set of public folders, altering their properties so
that they match our needs, and setting up appropriate replicas is only the
start of the management challenge. Over time, users will populate the public folders with content using Outlook clients. The next issue that we have
to deal is to understand what content is being stored and how this affects
storage. If you have a limited set of folders in a well-arranged hierarchy, it is
easy enough to browse public folders and investigate what they contain
(Figure 5.31).
Public folder statistics
To get a more comprehensive overview of public folders, you can use the
Get-PublicFolderStatistics command. For example, here’s how to use the
command along with some formatting instructions to create relatively pretty
output, but possibly not as comprehensive or as pretty as that generated by
the Exchange 2003 PFDavAdmin utility.
Get-PublicFolderStatistics | Select Name, ItemCount,
TotalItemSize | Sort TotalItemSize | Format-Table
@{expression="Name"; width=30; label =”Public Folder”},
@{expression=”ItemCount”; width=10; label = “Items”},
Chapter 5
5.13 Public folders
Figure 5.30
Viewing public
folder content
0]) /1K , 2)} ; width=15; label=“Size(KB)”}
Public Folder
Exchange 2007
Exchange 2007 Discussions
On page 388, we discuss changes in Exchange 2003 SP2 to allow
administrators to control replication storms. EMS offers the same facilities.
To suspend public folder replication:
EMS prompts to confirm that you want to suspend replication before
executing the command. After you have completed the changes that are nec-
5.13 Public folders
essary to reduce replication activity and calm the storm, you can resume replication with:
Finally, if you ever need to force replication of the public folder hierarchy, you can do this with the Update-PublicFolderHierarchy command.
In reading about all of these commands, you might conclude that public
folder management is a lot harder in Exchange 2007. In some respects this is
true because if you don’t take the time to understand how EMS works and
the commands that EMS has to work with public folders, there is no doubt
that public folder management will be a nightmare. On the other hand, if
you are shell-literate and willing to experiment, you can automate public
folder operations to a far higher level than was possible with any other version of Exchange. And as most companies will migrate relatively slowly from
Exchange 2003 to Exchange 2007, you will be able to fall back onto the
Exchange 2003 ESM to perform public folder management if you really
want to.
Permissions on top-level folders
For whatever reason, the lack of a GUI seems to have made it very difficult to
allocate the permission to a user (outside Exchange administrators) to create
new top-level public folders. You may argue that you don’t want people to
create any more top-level public folders, but if you do, you can allocate the
necessary permission with a command similar to the one shown below,
which assigns the right to create a top-level public folder on the folder hierarchies object. There may be a more effective or efficient way to do this, but
this way works!
Add-ADPermission -id "CN=Public Folders,CN=Folder
Hierarchies,CN=Exchange Administrative Group
Exchange,CN=Services,CN=Configuration,DC=xyz,DC=com" -User
-ExtendedRights ms-exch-create-top-level-public-folder
-AccessRights readproperty,genericexecute
Exchange 2003 supports the concept of referrals to allow a client to connect
to a remote public folder server to retrieve content. Exchange chooses the
Chapter 5
5.13 Public folders
public folder server to provide the content by reference to the connector costs
that exist between the routing group of the server that hosts the user’s mailbox and the routing group that hosts the public folder replica. Exchange
2003 also uses these costs when it needs to backfill content into a public
folder. Administrators have some control over referrals by disallowing public
folder referrals across a connector or by giving a server a preferred list of public folder servers to use.
Exchange 2007 does not use routing groups, so the Store has changed
to use Active Directory site costs. However, there is no concept of a setting
to disallow public folder referrals across a link. If you want to stop the
Store referring clients to a very remote public folder server, then you need
to set the cost of the site link connector (the Exchange site cost) to a high
enough value that the Store will disregard the link when it comes to compute the available referral paths because the connection is “too expensive.”
In this respect, high enough means 50 or higher, assuming that site costs
are normally 10 or under.
Migrating public folder content
There is no silver bullet available to answer the question of how to migrate
the data that has accumulated in public folders since 1996. Every company
uses public folders differently and every company will need to develop their
own plan.
The first challenge is to understand the public folders that are in the
hierarchy and which ones are actually in use. You can run a program like
PFDavAdmin on an Exchange 2003 server to generate reports about public
folders that are a good first step in understanding the scope of the problem.
However, PFDavAdmin will not tell you why the public folder was created,
who created it, who manages it now, and the business value of the data held
in the folder. These are all questions that you will have to answer through
patient and detailed investigation.
Part of the challenge is to categorize public folders in terms of their use
in order to help determine how and where you might eventually migrate the
data. Common usages include:
Repositories for forms-based applications, sometimes workflow in
nature. These applications are much more common in companies
who deployed Exchange before 2000 because Microsoft placed a
heavy emphasis on how to use public folders with Outlook forms
early on.
Project document repositories. These folders probably store an eclectic set of items in varying formats from mail messages to Word documents to Excel worksheets. Users post items to the folders on an on-
5.13 Public folders
demand ad hoc basis by dragging and dropping from an Exchange
mailbox folder or Windows Explorer.
Archives for discussion groups. Exchange has never offered list server
functionality, but people created their own basic implementation by
using distribution groups as the list and a public folder (automatically
copied on every post by including the SMTP address of the folder in
the list) as the list archive.
If you discount the options to ignore all the data held in some or all
public folders or to leave the data in place until it gradually ages to a point
where it is no longer required, you have to figure out how to move it out of
Exchange. Unfortunately, there is no equivalent to the Export-Mailbox command to process public folder data and export it to another repository.
Microsoft and third parties have written extractors to move data to SharePoint. Now, exploring a move to SharePoint is the route that most companies
will look into, if only because it is a Microsoft solution. SharePoint Portal
Server is quite a good option because it offers some extra functionality to
help manage data, such as better programmability and customization and
radically better search facilities. Aside from moving data to SharePoint, the
drawing board is bare, although there’s certainly a possibility that further
offering will appear to fill the gap as companies realize that the public folder
issue is something that they have to deal with.
The final issue is how to replace the functionality made available
through public folders. Some answers are available in Exchange 2007 and
you can anticipate that Microsoft will respond to customer demand by providing more features to handle public folder migration in future versions of
Exchange. However, we cannot count on any such features today, so what
can you do now? If you have any forms-based applications, you will probably
have to redevelop the application to use a different forms package and a different repository. Replication originally solved the issue of getting data close
to the user at the expense of other problems, such as conflicts caused by users
making changes in multiple places concurrently. Thankfully, better and
cheaper networks have made replication less necessary, so it is now conceivable to run an application from a central server that stores all the data.
Windows Team Services provide a solution for project repositories.
Team Services are simple to deploy and use but suffer from some of the same
management challenges as public folders, because it is not easy to know what
team sites are in use, what they store, and what the data is used for. Nevertheless, Team Services is a more than adequate replacement for project repositories. You might also experiment with Managed Folders to see whether they
can help address some needs in this area.
There is no silver bullet available today to assist you to migrate data
from public folders but it is not yet time to become desperate as there is no
doubt that some keen minds will focus on the problem before Microsoft
Chapter 5
5.15 Backups
ceases support for Exchange 2007. In the interim, the best idea is to stop
using public folders wherever possible, investigate possible routes forward,
and to do a lot of experimentation.
5.14 Removing database size limits
Prior to Exchange 2003 SP2, the limit of a mailbox store in the Standard
Edition of Exchange was always 16GB. This limit had been in place since
Exchange 4.0, so it was obviously outdated and inappropriate in a world
where the average size of a message has grown, average disk size has increased,
and the cost per GB has decreased enormously since 1996. Microsoft’s
response was to increase the limit to 75GB in Exchange 2003 SP2. The new
limit also applies to the version of Exchange included in Microsoft Small
Business Server. It is nice to have a bigger limit for the store, but even a
59GB increase does not match the growth in average message size. In 1996,
the average message in most corporations was around 10KB and now it is
100KB or larger, so just to keep pace Microsoft should have increased the
store limit to 160GB or more.
Once Exchange hits its store size limit, the Information Store service
gracefully halts and you have to reduce the size of the store to allow Exchange
to restart. The easiest way to shrink a store database is to run ESEUTIL to
remove all the white space from the database, but this is only a short-term fix
as it is unlikely that you will recover more than 1GB unless you recently
removed many mailboxes. The long-term fix is to ask users to remove old
messages from their mailboxes, and if they don’t cooperate, to help them by
implementing the Exchange 2003 Mailbox Manager so that their mailboxes
are automatically cleaned even if they don’t want them to be.
Given Microsoft’s focus on developing the Store as an all-purpose repository for anything from email to voicemail, and their view that users will
demand much larger mailboxes than they have today, it was a very rational
decision for Microsoft to remove all software limits for database sizes from
the standard and enterprise versions of Exchange 2007. The only database
limits that you will ever meet now are the amount of storage that you can
make available to Exchange and the operational window for backups and
restores that you want to apply.
5.15 Backups
A very wise person once said that an important difference between email and
word processing applications was that the effect of a word processor failure is
limited to a single user and a small quantity of documents. When an email
system collapses, everyone connected to that system is affected. Usually, peo-
5.15 Backups
ple at the top of a company notice the problem fastest (and complain quickest). Having a large server go offline because of a hardware problem creates
lots of extra work for administrators and users alike. Not being able to
recover data because no backups exist or discovering that you cannot restore
the data for some reason are fundamental lapses in system management discipline that can lead to instant dismissal for an administrator. There is simply
no excuse for not taking backups, and no excuse for not ensuring that
backup media is valid and restorable.
Email systems depend on many different hardware and software components to keep everything going. If any element fails to operate in the required
manner, data corruption can occur. If the hardware suffers a catastrophic failure, or some disaster such as an electricity outage afflicts the location where
the computers are, you will need to know the steps necessary to get your
users back online as quickly as possible. In all these instances, system backups
are a prerequisite.
In their purest sense, backups are snapshots of a system’s state at a certain point in time. All of the data available to the system should exist in the
backup and should be restorable to exactly the same state if required. You can
write backups out to many different forms of magnetic or other media,
although the most common media for small to medium organizations is
some form of high-density magnetic tape while large servers may first backup
to disk and then move the backup media to tape.
Exchange has always supported online backups, which generally proceed without causing any discomfort or impact to clients connected to the
server. There is no reason to take Exchange offline just to perform a backup.
In fact, taking Exchange offline is not a good thing to do. First, it reduces
overall system availability because users cannot connect to their mailboxes.
Second, each time you bring the Store service online, the Store generates
public folder replication messages to request updates from replication partners. Finally, the Store calculates checksums for each page before streaming
them out to media during online backups, and if a checksum does not
match, the Store will halt the backup operation. Best practice is always to
perform online backups, and to perform daily full online backups if possible, mainly because it is much easier to restore from a single tape than have
to mess around with incremental backups. Microsoft believe that the introduction of continuous log replication technology in Exchange 2007 (see the
section on LCR and CCR in Chapter 9) reduces the requirement to take
full daily backups to a point where you only need full weekly backups, but
this is only true if a) you deploy continuous replication and b) your auditors
and data retention experts agree that you don’t need the daily backups. The
problem is that many organizations routinely move daily backups of critical
data offsite to a secure vault and do so to ensure that they can recover from a
Chapter 5
5.15 Backups
catastrophic failure and that they can meet legislative or regulatory requirements for data security.
Apart from the introduction of continuous replication technology, the
biggest difference in Exchange 2007 that may influence your backup strategy
is the ability to move any database to any live server in the same organization.
This was an impossible task in the past because of the close link between the
server name and databases.
When you install Exchange, the installation procedure enhances
NTBACKUP.EXE, the standard Windows backup utility so that it can process online backups of Exchange databases. These enhancements are:
Support for the transactional nature of the Store: In particular, the
capacity to deal with storage groups, mailbox stores, transaction logs,
and transactions that might still be queued in memory.
The ability to perform online backups: In other words, to copy
online databases to tape without having to shut down Exchange services. Users continue to work during backups.
Extension of the NTBACKUP.EXE user interface to allow system
administrators to select which servers and databases they wish to
backup or restore.
NTBackup also includes support for hot backups taken through Windows Volume Shadow Copy Services.
Other commercial backup products
NTBackup is a basic backup engine that is very suitable for small to medium
organizations. Larger organizations, especially those that want to use a single
backup product to deal with many different types of data, tend to consider
other commercial backup products.
Exchange provides two DLLs that are for use by backup utilities to
access Exchange data. These are ESEBACK2.DLL, which interfaces with the
Store to perform a backup or to restore files, and ESEBCLI2.DLL, which
provides the interface to the backup client. Exchange-aware backup utilities
load these DLLs when they start. The DLLs include the API necessary for
backup utilities to process Exchange data. The following functions illustrate
the flow of a backup operation:
5.15 Backups
HrESEBackupPrepare: Establishes an RPC connection to Exchange
and returns a list of the available databases available for backup.
HrESEBackupSetup: Used by the backup utility to tell Exchange
which storage group it wishes to backup. Each storage group uses its
own backup set. You can perform a restore operation of a single database by selecting it from the list held in a backup set.
HrESEBackupOpenFile: Opens the selected database for read access.
Both EDB and streaming file are included.
HrESEBackupReadFile: Reads the database in 64K chunks. This
function also performs checksum calculation and verification.
HrESEBackupCloseFile: Closes the database.
HrESEBackupGetLogAndPatchFiles: Get a list of all the log file names
to include in the backup set. ESE backs up these files using the
HrESEBackupOpenFile, HrESEBackupReadFile, and HrESEBackupCloseFile functions.
HrESEBackupTruncateLog: Delete any obsolete transaction logs after
a full backup. Truncation only happens after ESE successfully backs
up all selected databases. If you do not backup all of the databases in
a storage group for any reason, ESE cannot delete the logs.
HrESEBackupInstanceEnd: End the backup for the instance (storage
group or selected database).
HrESEBackupEnd: Disconnect from the RPC connection to
Exchange provides a similar set of functions to restore databases from
backup sets, including HrESERestoreOpen, HrESERestoreAddDatabase, and
HrESERestoreOpenFile to support the restore of a single database or a complete storage group. While it is technically possible to run multiple concurrent restore operations, best practice suggests that it is safer to concentrate on
a single restore at a time, just to reduce the potential for error. The Exchange
backup API is comprehensive and is utilized by all of the major vendors of
backup software, as shown in Table 5.11. Use the web address to get information about the latest product features (including the precise version that
supports Exchange 2007) to help you to decide which product best meets
your needs.
It can be difficult to justify the additional expense of buying in a package to replace a standard utility, especially when the standard utility does the
job in an adequate manner. The cost of deploying, managing, and supporting the new utility across multiple servers also has to be taken into account.
It is a mistake to use different products to backup Exchange within an orgaChapter 5
5.15 Backups
Table 5.11
Selected third-party backup products for Exchange 2007
VSS Aware
Web Address
Backup Exec
Networker Module for
Microsoft Exchange
Data Protector
Computer Associates
ArcServe Backup Agent
for Exchange
BEI Corporation
nization as this makes it much more difficult to move data between servers.
Due to the different ways that software manipulates data, it is highly unlikely
that one backup package will be able to read a tape created by another. Once
you allow diversity, you create a potential situation where you cannot restore
a backup for one reason or another. Moreover, Murphy’s Law dictates that
this situation occurs just after a server hosting thousands of users has crashed
and you cannot bring it back online quickly. For this reason, you should
standardize on a single backup product and use the same backup media
(tapes and drives) to gain maximum flexibility. You never know when you
will need to move data onto a different server and it is embarrassing to discover that you cannot because the target server is not equipped with a drive
that can read the backup media.
A specialized package often provides very useful features that are missing
in the standard utility. In practice, third-party backup products usually provide four major enhancements over NTBackup:
Ability to take VSS hot backups
5.15 Backups
In addition, third-party backup products may provide features to help
you manage clusters and continuous log replication. Specialized backup
products tend to support high-end features such as unattended backups,
automated loading and unloading via tape libraries, and network backup to
very large storage silos (juke boxes or similar). Such features are not critical to
small or medium servers, but you need to consider them for any server that
hosts large databases.
It is important to understand that while third party backup software can
provide some valuable advantages, it is another piece to fit into your system
configuration matrix. New releases of the backup software must be qualified
against existing versions of Exchange, and then against new versions of
Exchange. You should also check the backup software against service packs of
both Exchange and Windows. The Exchange development team cannot
check their product against every third-party product or extension, so it is
entirely possible that the installation of a service pack will introduce a problem that only becomes apparent when a backup operation is attempted. On
the other hand, even worse, a problem that suddenly appears during a
restore. The complex interaction between Exchange, Windows, and any
third-party product adds weight to the argument that you should test all
combinations of software products (including patches) before they are introduced to production servers.
Creating a backup strategy
All hardware systems can expect hardware failures to occur over time. When
it is your turn to experience a failed server, you will be glad that backups
exist. System failures come in three broad categories.
Disk or drive controller failure.
Other noncritical system component failure, for example, the video
monitor for the server develops a fault.
Critical system component failure, for example, the motherboard or
other associated component experiences a fault that you cannot
quickly rectify.
Any failure situation requires some fast but calm thinking in order to
make the correct decisions to get everything up online again. If you cannot
bring the system back online, can you substitute another similar system and
restore application data? If it is a particular system component, is a replacement available or can you change the system configuration to work around
the problem? Having backups around will not replace making the right decision, but they are a great safety net.
Chapter 5
5.15 Backups
The MTBF (Mean Time between Failures) rate for disks improves all
the time and new storage technologies guarantee an ever-increasing resilience
to unexpected failure. The advent of hot snapshots through Windows Volume ShadowCopy Services and the introduction of continuous log replication in Exchange 2007 creates new tactics to consider for backup strategies.
These developments do not mean that you will never experience a disk failure, but it does mean that you are less likely to have one over any particular
period or that you can recover from failures more rapidly than before. A high
MTBF is no guarantee that the disk will not fail tomorrow, so the challenge
for system administrators is to have a plan to handle the problem when it
arises. Without a basic RAID array or more sophisticated storage technology,
if something does go wrong with a disk you will have to stop and fix the
problem or swap in a new drive. Once the hardware problem is fixed you will
have to restore the data, and of course, this simple statement assumes that
you have copies of all the application files that were stored on the faulty
drive, and possess the capability to move the data onto the new drive. The
value of good backups is instantly obvious at this point! Of course, none of
the protection afforded by storage technology is viable if you do not deploy
the right hardware, such as hot-swappable drives and the best controller you
can buy.
The database-centric nature of Exchange poses a particular challenge for
restore operations. There is no getting away from the fact that if a restore is
necessary it is going to take much longer than you may realize. The length of
time that it takes an administrator to backup or restore databases from a large
production Exchange server may come as a surprise to some. It is easy to forget just how long it takes to move data to backup media. Databases have the
capacity to expand quickly, so your backup times will grow in proportion.
After a server has been in production for a number of months, you might
find that it is time to consider looking for faster or higher-capacity media to
help reduce backup times.
Remember too that it is not just the Exchange databases that you have
to backup to ensure that you can recover a system. The Active Directory
database, other Windows data, the IIS metabase, Exchange binaries, user
data, and other application data all contribute to the amount of data and
length of backup operations. In line with best practice, most Exchange servers are not domain controllers or Global Catalogs, so you may not have to
restore the Active Directory after a failure on an Exchange server.
It is also important to apply the correct levels of service packs to Windows and Exchange on the recovery server before commencing any restore
operation. The internal database schema has changed a number of times in
Exchange’s history. In general, you cannot restore a database from a backup
taken with a higher software revision level. In other words, do not expect to
be able to restore a backup taken with Exchange 2007 to a server running
5.15 Backups
Exchange 2003. This rule does not hold true for all combinations of service
packs and versions, but there is enough truth in it to make a strong recommendation that you should maintain servers at the same software levels in
order to ensure that you can always restore backups.
You must account for all of the factors discussed here in the operational
procedures you employ at your installation. Restores are equally as important
as backups, but all the attention goes on backup procedures and backup products. Administrators usually only look into the detail of restores when a disaster happens, which is exactly the wrong time to begin thinking about the
issue. Consider this question: How long will it take to restore one gigabyte of
data from your backup media using the backup software you employ? Now,
how long will it take to restore 3GB, or maybe 10GB, or how about a fullblown 100GB mailbox store? And what about a public store? Do you have
procedures in place to handle the restore of the different permutations of the
Active Directory? And the data for all the other applications you may want to
run on a server such as the IIS metabase?
In most circumstances, even when suitable hardware is waiting to be
hot-swapped into your production environment, the answer for how long it
takes to restore a server is counted in hours. Even when the restore is complete, you may have other work to do before Exchange is available for use
again. For instance, the Store must replay transaction logs to roll forward
messages and other items that are not fully committed to the database. The
Store does this automatically, but applying masses of transactions at one
time delays the start-up operation (depending on processor speed and other
load, a set of 100 logs might take between 20 and 30 minutes to process on
a slow server). Users will be yelling at you to get the system back online as
quickly as possible, so it’s going to be a time when stress levels are rising,
which just adds to the piquant nature of the task. These are the facts of system management life. You must be prepared and able to conduct a wellplanned restore of essential data in order to be successful. No one wants to
explain to a CIO why the messaging system was unavailable for one or two
days while staff fumbled their way through a restore. Knowing how to properly use and safeguard backup media is an important part of a backup strategy. Think of the following questions: After you perform backups, who will
move the media to a secure location? Is the secure location somewhere local
or remote? When will you use the tapes or other media for backup purposes
again? How can you recover the backup media quickly if a disaster occurs?
Backups and storage groups
Exchange permits you to take a granular approach to backup and restore
operations as it is possible to backup or restore a single database instead of
the entire Store. Figure 5.32 shows how you can select individual databases
Chapter 5
5.15 Backups
within a storage group for backup using the Windows Backup Wizard.
When one or more databases from a storage group are processed, all of the
transaction logs belonging to the storage group are also included in the
backup media to ensure that the backup includes all transactions, including
those that are not yet fully committed to the database. Therefore, it does not
make sense to backup individual databases unless you have good reason to do
so. Always process backups on a storage group level whenever possible. Note
that NTBackup lists all of the servers in the organization and you can perform a remote backup across the network, if you choose to do so. However,
this is not a good option unless you have a very capable high-speed link
between the source database and the target backup device.
Figure 5.31
Using the Windows
Backup Wizard to
select databases
Exchange reserves an additional special storage group for restore operations. When you restore a database, the Store overwrites the failed database
with the database from the backup set and moves the transaction logs from
the backup set into the temporary directory. The Store can then replay transactions from the logs, commit changes to the database, and make the database consistent and up to date. When the database is ready, the restore
storage group turns over control to the normal storage group and operations
recommence. This technique ensures that the other databases on a server
continue to be operational when you have to restore a failed database.
5.15 Backups
Backup operations
The exact process to take a backup depends on the software that you use. To
illustrate the process, this section describes how to take a full backup of an
Figure 5.32
A full backup is
about to start
Exchange storage group. You can compare the steps outlined here with the
way that the software that you use behaves when you use it to perform a
The backup agent (NTBackup or a third-party utility) performs its
own processing to prepare for the backup. This includes establishing
the type of backup (incremental, full, or differential) and the target
media (tape or disk).
The backup agent makes a function call to ESE to inform the storage
engine that it wishes to begin a backup. ESE logs event 210 to indicate that a full backup is starting.
Chapter 5
5.15 Backups
The current transaction log is closed and the Store opens a new transaction log. The Store then directs all transactions that occur during
the backup to a new set of logs that will remain on the server after the
backup completes.
The backup process begins (Figure 5.33). The backup agent requests
data and the Store streams the data out in 64K chunks. ESE writes
event 220 to the application event log (Figure 5.32) as it begins the
backup of each database, noting the size of the file. A backup to disk
uses shared memory and is registered by Exchange 2007 as event 907.
As the Store processes each page, it reads the page to verify that the
page number and CRC checksum are correct to ensure that the page
contains valid data. The page number and checksum are stored in the
first four bytes of each page, and if either is incorrect, the Store flags a
-1018 error to the Application Event Log. The backup API will then
stop processing data to prevent you from taking a corrupt backup.
This may seem excessive, but it stops administrators blithely taking
backups of databases that potentially contain internal errors. There is
no point in taking a backup if you cannot restore the data, and restoring a corrupt database is little use to anyone.
ESE logs event 221 as the backup of each database completes.
After all the pages from the target databases (a complete storage group
or individually selected databases) are written to the backup media, the
backup agent requests ESE to provide the transaction logs, which are
then written to the backup media. ESE then logs event 223.
ESE only writes the set of transaction logs present when the backup
starts to the backup media. If this is a full backup, ESE then deletes
these logs (and notes the fact in event 224, which also notes the file
names of the logs that are deleted) to release disk space back to the
system. It is quite safe to do this as the transactions are committed to
the database and are available in the logs in the backup set. Of course,
if a situation occurs where a backup copy of a database is corrupted in
any way (perhaps due to a tape defect), then you will need copies of
every log created since the last good full backup to be able to perform
a successful restore. For this reason, some administrators take copies
of transaction logs to another disk location before a backup starts.
Some may regard this behavior as a touch paranoid; others point to
the failure rate for tapes and consider it a prudent step to ensure full
The backup set is closed and normal operations resume. Successful
completion is marked in event 213.
5.15 Backups
When it completes processing, the backup utility usually reports the
details of its work. For example, an NTBackup report of the successful
backup of an Exchange storage group is shown below:
Backup Status
Operation: Backup
Active backup destination: File
Media name: "Storage Group 1 created 3/21/2007 at 11:37 AM"
Backup of "ExchSvrMbx1\Microsoft Information Store\Storage
Group 1"
Backup set #1 on media #1
Backup description: "Set created 3/21/2007 at 11:37 AM"
Media name: "Storage Group 1 created 3/21/2007 at 11:37 AM"
Backup Type: Normal
Backup started on 3/21/2007 at 11:38 AM.
Backup completed on 3/21/2007 at 11:53 AM.
Directories: 3
Files: 6
Bytes: 9,221,460,426
Time: 15 minutes and 22 seconds
Figure 5.33
A backup in
Chapter 5
5.15 Backups
Incremental and differential backups only copy transaction logs to
backup sets. Incremental backups copy the set of logs created since the last
backup, while a differential backup copies the complete set of logs created
since the last full backup. Thus, to restore Exchange databases you need:
The last full backup, or
The last full backup plus all incremental backups taken since then, or
The last full backup set plus the last differential backup.
Obviously, a full backup is the easiest way to restore because there are
fewer tapes to handle and less opportunity to make mistakes. However, if you
rely exclusively on full backups, it is possible that you will miss some log files
you need to recover back to a time before the last full backup. For this reason,
it is best practice to take differential backups between full backups, leading to
a backup schedule where you might take a daily full backup at 6PM, with a
differential backup at lunchtime when system load is often lower.
Exchange 2003 and 2007 both stamp databases with the date and time
after each successful backup. You can view the timestamp by selecting a database and viewing its properties through EMC. Exchange records times for
the last full and last incremental backups, as shown in Figure 5.35 (in this
case, the database is obviously backed up by a regime that favors full backups
instead of incremental backups). Apart from anything else, the existence of
the timestamp gives you confidence that a successful backup exists as well as
telling you how old it is.
Another way of checking on whether backups are being taken properly
is to scan the organization with the Get-MailboxDatabase command, specifying the –Status parameter so that Exchange includes details of each database’s current status. For example:
Get-MailboxDatabase –Status | Select Name, StorageGroup, Mounted, LastFull
-------------4/9/2007 7:00:18 PM
4/9/2007 7:19:15 PM
4/9/2007 7:23:12 PM
4/10/2007 12:00:...
4/8/2007 1:00:14 AM
4/8/2007 12:30:1...
4/10/2007 1:40:1...
5.15 Backups
Figure 5.34
Last good
backup timestamp
True 4/10/2007 2:30:1...
True 4/9/2007 3:59:49 PM
Assuming that we executed this command around March 10, the status
looks acceptable for the first four servers (we might ask why two of the three
databases on server ExchMbxSvr2 lag two days behind in backups), but we
can see that no one has taken a backup of servers ExchMbxSvr5 and
Checkpoint file
The checkpoint file (E00.CHK is the file used by the default storage group)
maintains a note of the current log file position so that ESE knows the last
committed transaction written to the databases in a storage group. ESE
maintains a separate checkpoint file for each storage group. If you enable
Chapter 5
5.15 Backups
continuous replication for a storage group, ESE maintains a checkpoint file
for the database replica.
The primary role played by the checkpoint file is during “soft” recoveries, when databases have closed down in an inconsistent manner through
some system failure that has not resulted in a corrupt database. When the
Information Store service restarts, ESE opens the checkpoint file and
retrieves the location of the last committed transaction, and then begins to
replay transactions from the logs from that point. While this is convenient,
the checkpoint file is not mandatory for replays to occur, as ESE is able to
scan the complete set of transaction logs to determine the point from which
replays should commence. ESE does not write the checkpoint file to a
backup set because it serves no purpose during a “hard” recovery. In this scenario, you restore databases and transaction logs to different locations, so the
ESE instance that controls the restore depends on the log information contained in the database header to work out what logs it should replay. In addition, ESE checks that all necessary log generations are present and available
before beginning to replay transactions and will not proceed if gaps exist.
Other Windows components such as DHCP, WINS, and the Active Directory that use ESE also maintain checkpoint files.
Figure 5.35
Selecting a
database for restore
Restoring a database
The Store is responsible for restore operations and ESE uses a reserved storage group to enable online restores. The following steps occur to perform a
database restore using NTBackup, which streams the data out of a backup set
5.15 Backups
to recreate the database and any required transaction logs in a recovery location on disk:
The database or storage group affected by the failure is dismounted
by the administrator or is rendered unavailable by a hardware problem, such as a disk failure. In this case, you must fix the hardware
problem before a restore can proceed. The Information Store service
must be running.
If you have a corrupt database in place, you probably want to overwrite the file on disk. Alternatively, you may prefer to take a copy of
the corrupt database (EDB, STM, and associated transaction logs)
before the restore overwrites the file and move it somewhere safe for
later investigation. A database property controls whether the Store
will allow the backup agent to overwrite a database. Either access the
store’s properties and set the necessary checkbox, or simply delete the
corrupt file before beginning the restore.
The backup agent initializes and displays details of the backup sets
that you can restore from the available media. Figure 5.36 shows
NTBackup in operation. The backup set is from a full backup operation because both databases and transaction logs are available. Incremental and differential backup sets only hold transaction logs. We
can now select whatever data we need to restore.
Figure 5.36
Setting parameters
for the restore
Chapter 5
5.15 Backups
The backup agent notifies ESE that it wishes to begin a restore operation. The Store then launches an ESE recovery instance to manage
the special recovery storage group. The recovery storage group only
exists during restore operations.
The backup agent begins to stream data out of the backup set into
the necessary databases, using direct Win32 file system calls to copy
the files into the appropriate locations. This operation proceeds
under the control of the Store, which uses the special reserved storage
group for this purpose. You restore differential and incremental sets
first, followed by the most recent full backup. If you follow best practice, and always take full backups whenever possible, restore operations are much simpler. Figure 5.37 shows how you direct the backup
set to the target restore server. Note that because this is the last
backup set that must be restored, we can set the “Last Backup Set”
checkbox to tell the backup agent that it does not have to look for
any other backup sets and can complete the recovery operation once
the databases have been restored. For NTBackup, this checkbox is a
critical part of the restore operation because ESE will perform none
of the fix-up processing to make the database consistent until you tell
ESE that it is okay to proceed by checking this box. However, if a
database proves to be inconsistent after a restore, you can use the
ESEUTIL utility to try to correct any errors15. Note that the user
interface of some third-party backup utilities do not include the “Last
Backup Set” checkbox because the utility controls the entire restore
operation, including the decision when to begin post-restore fix-up
processing. You can also check the “Mount Database After Restore”
checkbox to get the restored mailbox store into operation as quickly
as possible after the restore is complete.
ESE restores the EDB and streaming files to the production directory,
but the transaction logs are not, as they might overwrite logs that
contain transactions that ESE later needs to replay. You must therefore provide a temporary location to hold the transaction logs from
the backup set for the duration of the restore. Make sure that the
temporary location (for example, C:\TEMP\BACKUP) has enough
disk space to accommodate all of the log files in the backup set.
The usual indication that you have to take some action to fix a database is when you cannot mount it following a restore.
You will probably see error 544 signaled by Exchange when you attempt to mount the database with the Mount-Database
command. This error should also appear in the application event log and it indicates that you need to perform a hard
recovery before Exchange can mount the database. You should have a restored database and a set of transaction logs
accumulated since the time of the last backup on disk, so you can perform the necessary recovery by replaying the transactions from the logs into the restored database and so make the database consistent and fully up to date. You can perform the hard recovery with the ESEUTIL /CC /T<storage group instance> command or by running the database
recovery utilities in the Exchange 2007 toolbox.
5.15 Backups
ESE creates a file called restore.env in the temporary location to control the processing of transaction logs during the restore. The
restore.env file contains information about the databases, paths, signatures, and other information used by the restore and replaces the
older registry key used by previous versions of Exchange to signal that
a restore operation was in progress. ESE creates a separate file for each
restore operation. You can dump the contents of restore.env with the
ESEUTIL utility.
The next step is to process any transaction logs required to bring the
database up to date. During this process, ESE validates the log signature and generation sequence to ensure that the correct transaction
logs are present and are available to recover transactions. If a log signature does not match the database signature, ESE returns error –
610, and if it discovers a gap in the log generations, ESE signals error
–611. If either of these errors is encountered the recovery process is
stopped. ESE has not yet replayed any transactions into the database,
so you have a chance to fix the problem and start the restore again.
Fixing the problem may need you to check the signatures on the
databases and logs to ensure that they match, or discover why a gap
exists in the log generations.
ESE actually reads two sets of logs during the restore. First, it processes the
logs held in the temporary location, and then the logs in the normal location
that have accumulated since the backup. Transaction logs contain data for all of
the databases in a storage group, so if you are restoring a single database, ESE
must scan all the data in the logs to isolate the transactions for the specific database and then proceed to apply them. ESE ignores all of the other data in the
logs. This phase of the operation may be the longest of all, depending on the
number of transaction logs that have to be processed.
Once the data in the transaction logs has been applied, the recovery
storage group performs some cleanup operations, exits, and returns control
to the primary storage group (the storage group that is being restored), which
is then able to bring the newly restored databases back online. ESE also
deletes the files in the temporary location. ESE notes details of all of the
recovery operations in the Application Event Log.
In the case of differential or incremental restores, only the transaction
logs are loaded into the temporary location and processed there. At the end
of a successful restore, ESE writes event 902 into the application event log.
ESE also records the details of the restore operation in a text file (the file is
named after the backup set). To be certain that everything is okay before you
allow any users to connect to the store, check that ESE logs event 902 and
that ESE has recorded no errors in the text file.
Chapter 5
5.15 Backups
Given the growing size of databases, the focus for Exchange backup
and restore operations is moving from the traditional streaming approach
as used by programs such as NTBackup to the snapshot or cloning
approach offered by backup programs that support Windows Volume
ShadowCopy Services (VSS). Streaming backups are all that small to
medium servers require, but if you work with databases that are larger than
50GB, you should consider moving to VSS-based backups, if only because
they deal with large volumes of data much more efficiently and faster than
streamed backups.
The future of streaming backups
Companies have used NTBackup and other streaming backup utilities to
take online backups of Exchange servers since 1996. Even large companies
like the flexibility of being able to use NTBackup to take a backup to disk
and then complete the process by moving the backup file from disk to tape
or to a tape SAN. Microsoft’s future direction for streaming backups is not
certain because there the new Windows Backup utility that is part of Longhorn cannot generate streamed backups. In addition, the Longhorn version
of NTBackup may be restricted to only being able to read and restore from
existing backup sets and will not be able to generate new backups.
The logic is that Microsoft wants to move towards block-level backups
that are taken with VSS. This means that you will only be able to backup
complete volumes in the future. VSS-based utilities will create backups as
images of the source volumes in the form of VHD (Virtual Hard Disk)
files. The meaning of these changes is that you will not be able to store
backup sets on the same volume as the original data or on any other disk
except a volume that you dedicate for backup purposes. While medium to
large enterprises will be able to find the required disk resources easily
enough, this evolution adds extra cost to small deployments.
While it is true that Microsoft never intended that NTBackup to
become an enterprise-capable backup utility (which is one of the reasons
why so many other backup products exist), the fact still remains that
NTBackup is a popular choice for Exchange deployments. The move to
VSS-based backups will require companies to change their procedures for
backup and restore operations and to spend time to select and test software
to use. They will experience additional cost if they decide to use a thirdparty backup product and they will have to invest in some extra disk. None
of this seems like a good idea, but you could argue that streaming backups
are yesterday's technology and that it is time to move on to something
more modern. In any case, this is an area to keep an eye on as Longhorn
matures and its effect is felt by Exchange 2007.
5.16 Moving from the Store
5.16 Moving from the Store
The Store provides a great repository for the messages and other items generated on an Exchange server, but an email system does not exist purely on its
ability to act as a repository as it has to be able to transfer messages to other
servers and different email systems. The Store has a great connection to the
Exchange transport system, which is our next stop.
Chapter 5
This page intentionally left blank
Exchange Transport and Routing
While the Store is the heart of Exchange, the Transport engine is its lungs. In
this chapter, we review how Exchange 2007 transports and routes messages
from origination to destination.
The evolution of routing
In terms of flexibility, the way that messaging systems route email has evolved
greatly over the last ten years. Mapping developments against the history of
Exchange, a rough timeline is as follows:
Pre Exchange: Message routing is quite inflexible and administrators
need to sit down and map out the path of messages from one server
to another within an organization to understand how messages flow.
Routing messages to another company or to a different email system
within the same company usually requires a connector to link incompatible systems and rewrite email addresses. Satisfactory message
interchange is often only possible in 7-bit plain text format and
attachments are usually unsupported.
Exchange 4.0–5.5: The X.400 protocol provides the basis for message routing. Messages flow between servers in the same site automatically, but administrators have to create paths between sites with
connectors. The routing topology, even within the same organization,
is still inflexible and a server outage can have a huge impact on reliable message routing. Connectors are required to link to other email
systems. It is easier to pass rich text format messages between different email systems, but interoperability remains a challenge.
Exchange 2000–2003: SMTP provides the basis for message routing. Messages flow within the same routing group automatically and
a great deal of flexibility exists within the organization because
Exchange servers send link state information to each other to update
6.2 Change through experience
servers about problems along routing paths. Servers can then change
the routing topology to accommodate changing conditions. Connectors are still required, but less traffic flows across connectors, as
SMTP has become the lingua franca of the email world. Greater support for SMTP makes message interoperability much easier.
Exchange 2007: SMTP remains the foundation for message routing.
Exchange 2007 drops its own routing topology, adopts the topology
that Windows uses to replicate Active Directory information, and
favors direct connections between email servers whenever possible.
Exchange 2007 introduces the concept of hub transport servers that
are responsible for the routing of all messages within an Exchange
organization. Exchange 2007 also introduces the Edge server, a specialized version of a hub transport server that Microsoft has designed
to operate in the DMZ and be the first port of contact for incoming
messages. While they offer the best degree of integration with an
Exchange organization and deliver strong SMTP performance and
good protection against spam and viruses, Edge servers are optional
and you do not need to deploy them unless you do not have another
way to protect your organization against the polluted mail streams
that arrive in from the Internet.
The bottom line is that message flow is more integrated with the operating system today than ever before, leading to less administrative overhead
(because you have only one infrastructure to manage) and faster transmission (less overhead to get in the way). At the same time, the adoption of
SMTP everywhere makes it easier to connect different email systems
Change through experience
The first generation of Exchange servers supported SMTP, but only through
a separate connector (the Internet Mail Connector, or IMC). The second
generation made SMTP the default protocol for email transmission and supported SMTP on every server. Instead of using sites1 as the basis for the routing topology, Exchange 2000 introduced routing groups, each of which
included a server that acts as the routing group master. To build the routing
topology, you link the routing groups together with connectors. A server
inside each routing group is nominated as the routing group master and is
responsible for tracking the current state of the topology by exchanging link
state information with other routing group masters. These specially encoded
messages contain link state updates that provide the routing group masters
An Exchange site (4.0–5.5) is a collection of servers connected together by Windows RPCs.
6.2 Change through experience
with new data about the network, such as connector outages or new connectors. The concepts of link state routing, OSPF (Open Shortest Path First),
and link state propagation used by Exchange 2000/2003 provide a routing
topology that is much more flexible than the fixed nature of the GWART
(Gateway and Routing Table) that Exchange 5.5 generates nightly.
There is no doubt that Exchange 2000 and 2003 servers are capable of
routing very high volumes of messages and that link state routing works, but
administrators encountered three problems in production. First, the propagation of link state information is advantageous in networks that experience
changing conditions because it keeps the mail flowing around potential bottlenecks. Knowing that a connector is unavailable can prevent messages accumulating on a queue when an alternate route is available. If your network is
stable, you don’t gain any advantage from link state updates and incur the
overhead to propagate the information between servers.
In complex networks with many routing groups, especially those built
on hub and spoke networks where connections to the spokes might not be
consistently available, the overhead created by link state updates can create
network bloat. When an outage occurs in a leaf and spoke network no alternative network routes exist, because once a connection to a spoke is unavailable, Exchange cannot transfer email to that leaf node, no matter how many
link state updates flow between servers. From Exchange 2003 onwards, you
could turn link state routing off for these networks by making a change to
the registry.
The second issue is the need for administrators to build and maintain
routing groups to create the routing topology. This is not a major problem
because the majority of work often occurs at the start of a messaging
project and then the work reverts to simple maintenance, but it is another
administrative task that only delivers value to large enterprises. Companies
that run one or two Exchange servers may not even understand that routing groups exist!
The last problem is the ability of administrators to troubleshoot routing
problems. Because link state routing implies that routes between servers can
change at any time, you are never quite sure about how messages flow. This is
not an issue when everything is working well, but it can be if you have a
problem and you do not quite know where to start debugging the issue. The
lack of good administrative tools in Exchange 2000 and 2003 to debug message flow and reveal the current routing topology compounded this problem.
The issues that administrators discovered with the Exchange 2000/2003
transport engine are irritating rather than disastrous and Microsoft could
have fixed or ameliorated the problems by taking steps such as tuning the
propagation of link state information or providing better administrative
tools. However, developments elsewhere meant that Microsoft had to change
part of the core of the transport engine. Microsoft took the decision to
Chapter 6
6.2 Change through experience
rewrite the complete transport engine and base it on the .NET framework
rather than to upgrade the older code base used in Exchange 2000/2003. The
immediate impact of this change is that if you have any customized code created for Exchange 2000/2003 (such as event sinks for the transport service),
you will have to rewrite it for Exchange 2007. The same issue arises if you
have code that modifies SMTP attributes in the IIS metabase because
Exchange no longer depends on IIS for the SMTP protocol. Exchange 2007
now includes all of the code necessary to support a comprehensive SMTP
service in its own code base.
Microsoft cites the following advantages about the Exchange 2007
transport engine:
The code base is now managed code, so it is more secure and robust.
For example, managed code generates no buffer overflow conditions
for hackers to exploit. The APIs used in the .NET framework are
more consistent and easier for programmers to work with, so the code
base is easier to maintain and extend.
The routing topology created with routing groups and connectors is
not needed by many Exchange organizations. Everyone has to deploy
the Active Directory before they can install Exchange and the Active
Directory has its own topology created from sites and site links. Eliminating the separate Exchange routing topology simplifies matters
and removes a task from the administrative workload.
SMTP is the default transport protocol for Exchange 2007. The type
of networks deployed today permit point-to-point connections
between servers, especially when you use an efficient protocol like
Point-to-point connections create a much simpler routing infrastructure because you have a full-mesh network where any hub transport
server can contact any other hub transport server in another Active
Directory site to transfer messages. It also leads to better scalability.
The Exchange 2003 transport engine has problems when there are
more than 100,000 messages in a queue. You might not hit this problem in a small to medium enterprise, but it can be an issue for large
enterprises and hosted deployments.
Administrative tools such as queue viewers are easier to write for simpler routing environments.
The new transport engine has the ability to detect and handle “poison” messages. These are badly formed or illegally-formatted SMTP
messages that can cause the previous routing engine to stop.
6.2 Change through experience
It made sense for Exchange 2000 to rely on IIS for the SMTP service
because it allowed Microsoft to ship Exchange 2000 faster. Over
time, the complications of managing the interaction between IIS,
Exchange, and Windows to control a relatively simple service surpassed the original advantage, so creating an Exchange-specific
SMTP service that only runs on hub transport and Edge servers simplifies matters all around. Exchange 2007 servers disable the standard
IIS SMTP service and replace it with an upgraded SMTP service that
runs inside MSExchangeTransport.EXE. If they need to, Microsoft
can upgrade the Exchange SMTP engine without affecting IIS and
potentially other applications.
Compliance and message tracking was another big issue that Microsoft
could not address in an adequate manner with their old code base. In an
Exchange 2000/2003 environment, both the Store (local delivery) and the
transport engine (remote delivery) can handle messages, so it is extremely
difficult to introduce software to monitor and trace the path of messages
and be able to satisfy the requirements of auditors and legislators. Many
companies in the financial industry developed transport events to impose
Chinese Walls (otherwise known as ethical walls) between different parts of
the companies (brokers and traders, for instance) or to process messages to
ensure that they complied with legislation. Code like this works very well
for any message going to an external address, but because the Exchange
2003 Store handles local delivery and Exchange does not invoke transport
events for these messages, it is difficult for Exchange to comply with some
regulatory requirements. To be fair to Microsoft, no one could have predicted the growth in the need for compliance in email systems when they
designed Exchange 2000 and every other email system has faced a struggle
to achieve compliance. The net result is that it made a lot of sense for the
developers to rewrite the transport engine in Exchange 2007 and to deliver a
much simpler and straightforward SMTP-based routing environment that
incorporates new functionality to add support for compliance through
transport and journal rules.
Hidden administrative and routing groups
Of course, it is difficult to move from one transport architecture to another
without leaving some traces behind. In the case of Exchange 2007, the
designers had to support backwards compatibility with routing and administrative groups to allow Exchange 2007 servers to coexist inside an organization with Exchange 2003 servers. The solution was to create a hidden
administrative group and a hidden routing group to hold Exchange 2007
servers as they were installed into the organization. You can see the hidden
groups if you examine the details of the Exchange organization through the
Chapter 6
6.2 Change through experience
Active Directory or look at them through ESM. A minor but important
detail was to come up with a suitable name for the hidden administrative and
routing groups, and the Exchange designers decided to call them:
Exchange Administrative Group (FYDIBOHF23SPDLT)
Exchange Routing Group (DWBGZMFD01QNBJR)
On the surface, it looks as if the designers used the same algorithm as is
used to generate the alphanumeric keys for Windows licenses, but the Trivial
Pursuit answer is that they used a simple Caesar cipher2. Thus, the actual
meaning of the names are:
All of this goes to prove that at least some humor was applied during the
design of Exchange 2007.
After you install Exchange 2007 into an Exchange 2003 organization, the
installation procedure updates the ESM console to prevent you renaming the
special administrative and routing groups. Some administrators have asked if it
is possible to change the name of the special administrative and routing groups
because the names, while cute, are somewhat clumsy, especially if you want to
use LDAP to interrogate the directory. It is obviously much easier (and less
likely to make a mistake) to type in a name like “Ex2007Admin” instead of
“Exchange Administrative Group (FYDIBOHF23SPDLT).” The special
When Caesar sent sensitive messages around, he used a simple code created by shifting letters one position forward or
backwards in the alphabet to encode the message. You reverse the process to decrypt the message.
6.3 Exchange 2007 transport architecture
names aren’t just in a few places as they are scattered across the Active Directory
in use in LegacyExchangeDN properties for many objects. Microsoft’s answer
is that it is impossible to change the names because they are hard-coded in
some places like the setup program, which expects to install servers into the
special administrative group.
Note that the creation of the hidden administrative and routing groups
to support Exchange 2007 servers will result in a full OAB download for any
client like Outlook 2000 or XP that uses OAB format 2 or 3.
Exchange 2007 transport architecture
You have to consider the change in the transport architecture alongside the
introduction of server roles in Exchange 2007 and Microsoft’s desire to split
functionality up so that a server only runs the code that it absolutely must in
order to perform its role. As discussed in Chapter 1, previous generations of
Exchange servers are complete messaging servers in that Microsoft designed
an architecture that ran the complete system on a single computer. You can
still keep things simple by deploying multiple roles on an Exchange 2007
server, but the overall design philosophy is to move toward specialized server
roles, and the hub transport3 server is critical to the ebb and flow of messages within the Exchange organization. In large deployments, hub transport servers are operated as separate servers, but in smaller deployments, it is
more common for a server to support both mailbox and hub transport
server roles. These servers may also take on the Client Access server role to
provide a complete messaging service within an Active Directory site. Note
that you cannot deploy the hub transport role on a clustered server.
Figure 6.1
Message flow
through the
Exchange 2007
transport system
Hub transport servers perform the same work as bridgehead servers in previous versions of Exchange.
Chapter 6
6.3 Exchange 2007 transport architecture
Figure 6.1 illustrates a simplified view of message flow between
Exchange 2007 mailbox and hub transport servers. The major points that
you need to understand from this diagram are:
Exchange 2007 mailbox servers do not transport mail. Instead, when
a mailbox server has a message that it needs to send to either a mailbox on another Exchange server or a recipient in another email system, the Microsoft Exchange Submission service that runs on the
mailbox server informs a hub transport server that it has a message to
be fetched. The Submission service indicates the database, mailbox,
and message identifier to point to where the new message is waiting.
The transfer of the message between mailbox and hub transport
server is the responsibility of the Store driver. Note that the mailbox
server performs the same processing for remote and local deliveries
because if it did not then Exchange 2007 would have the same problems with compliance systems. In effect, outgoing mail goes into an
“outbox” folder in the Store where it is fetched by the Store driver
and moves the message to the hub transport server, where the message ends up on the submission queue. The transport service retrieves
the messages from the submission queue, categorizes them, and then
routes the mail by placing the messages onto the appropriate delivery
queues for onward transmission.
Mailbox and hub transport servers within an Active Directory site
communicate together using server-side RPCs. The need to accommodate Exchange RPCs requires the servers to be connected by
LAN-quality bandwidth as otherwise you run the risk of RPC timeouts and failure.
If multiple hub transport servers are available in the site, the Mail
Submission Service selects one on a load-balancing basis. This provides a level of fault tolerance for the transport service. If a server
hosts both the mailbox and hub transport role, Exchange uses the
local hub transport server to process messages.
The Active Directory Topology Service is responsible for providing
the transport service with information about the current site topology
by reference to an Active Directory domain controller/Global Catalog server. The interface used is RLAPI.
Outgoing messages leave via SMTP to other hub transport servers,
through a routing group connector to legacy Exchange servers, an
SMTP connector to other SMTP email systems, or to a purpose-built
gateway for transmission via other means, such as FAX.
The transport service applies message policies when the categorizer
has expanded the set of addresses for messages. The policies are
6.3 Exchange 2007 transport architecture
applied through transport rules that can perform processing from the
mundane (apply disclaimer text to outgoing messages) to complex
(block messages from travelling between different groups of users).
The transport service also invokes any journal rules that you define to
capture message traffic at this stage.
Message policies are applied when the categorizer has expanded the
set of addresses for messages. The policies are applied through transport rules that can perform processing from the mundane (apply disclaimer text to going messages) to complex (block messages from
travelling between different groups of users). Journal rules to capture
message traffic are also invoked at this stage.
A new version of the Exchange Virus Scanning API (VSAPI) is available to scan incoming and outgoing messages on hub transport and
Edge servers.
All of the hub transport servers in the organization share a common
configuration that dictate settings such as the maximum message size
that Exchange can transport, the maximum number of recipients on
an incoming message, and the connectors that are available to
Exchange including connectors hosted by Edge servers. The configuration settings, including details of transport and journal rules, are
held in the Active Directory. Edge servers operate as isolated entities
that are loosely connected to the organization but don’t share configuration settings. If you deploy Edge servers, you have to configure
each Edge server separately.
Exchange 2007 determines available routing paths for messages by reference to the Active Directory topology and does not use the link state routing
algorithm employed by Exchange 2000 and 2003. Link state update data is
not exchanged between hub transport servers to keep them abreast of potential connector outages. The routing topology created from routing groups
and routing group connectors (and less commonly, SMTP and X.400 connectors) is replaced by simple point-to-point connections between Exchange
2007 hub transport servers. If a point-to-point connection between hub
transport servers is unavailable for some reason, Exchange determines how
close to the eventual destination it can transport a message by reference to
the Active Directory site topology, but only includes sites that include an
Exchange hub transport server. To determine the best routing path,
Exchange 2007 uses the costs assigned to the links that connect Active Directory sites. From this discussion it’s obvious that a well-designed and properly
functioning Active Directory site topology is now a critical success factor for
Exchange 2007.
Chapter 6
6.3 Exchange 2007 transport architecture
The critical role of hub transport servers
In order to go deeper into the Exchange 2007 transport architecture, it’s
important to fully understand the work that the hub transport server and the
components that collectively form the Exchange transport services do in an
Exchange 2007 organization. We will get to the details of many of these
operations, such as setting up SMTP connectors, as we move through the
remainder of this chapter.
Hub transport servers are responsible for:
Mail Flow: Hub transport servers process every message sent inside
an Exchange 2007 organization before the messages are delivered to
mailboxes or routed outside the organization. There are no exceptions to this behavior; Exchange always directs messages through a
hub transport server to determine and execute routing. Obviously,
this means that you must install a hub transport server inside every
Active Directory site that contains a mailbox server.
Message Submission: Messages flow into hub transport servers in
four ways. The first is through SMTP submission via port 25 through
a receive connector (or port 587 in the case of POP3 and IMAP4 clients). Connectors handle messages flowing in from other hub transport servers or an Internet connection or through a routing group
connector that allows messages to flow in from legacy Exchange servers in a mixed organization. The second is when applications place
properly formatted SMTP messages in the Pickup or Replay directories. The third method is via the Store driver, which handles outgoing
messages that the driver fetches when MAPI clients connected to
mailbox servers send messages. The Store driver fetches the messages
from the outbox folders in user mailboxes in MAPI format and converts them to TNEF format before submitting them. The last is when
an agent creates a message. In all instances, the message ends up on
the Submission queue for processing by Transport services. Edge servers accept messages only through a receive connector.
Message Categorization: The Categorizer performs all address resolution for recipients by checking the addresses for internal recipients in message headers against a Global Catalog server. It expands
distribution groups, including resolving the queries necessary to
provide the membership of dynamic distribution groups, and
checks these addresses against the Global Catalog. Once it knows
the set of addresses it has to deal with, the categorizer resolves the
best routing path for messages and performs any content conversion that is necessary (for example, to ensure that a user receives
6.3 Exchange 2007 transport architecture
messages in a preferred format). The categorizer is also responsible
for message bifurcation, if this is required for efficient message
delivery. All messages that move through a hub transport server pass
through the categorizer. Before the categorizer places a message
onto a delivery queue, it checks to see whether the message has to
be processed by a transport or journal rule and then performs whatever processing is required.
Message Delivery: When a hub transport server determines that a
message is to be delivered to an Exchange 2007 mailbox, it either
notifies the Store driver if running on the same server to effect delivery to a local mailbox (a local mailbox is a mailbox on any server in
the same Active Directory site as the hub transport server) or it transfers the message via a send connector to another hub transport server
in a different Active Directory site, which then takes responsibility for
onward delivery to the destination mailbox. Messages going to nonExchange recipients are delivered through an SMTP connector,
routed to an Edge server, or routed to an external gateway that has
responsibility for transmission to a foreign email system. You don’t
have to do anything to configure the connectors between hub transport servers in different Active Directory sites, because all hub transport servers know about each other through the configuration
information held in Active Directory and the connectors are automatically configured by Exchange. You do have to create SMTP connectors for outgoing communications. All communication between
hub transport servers is via SMTP and is automatically secured with
TLS and Kerberos. Again, you don’t have to do anything to configure
the secure connections.
Protection: Hub transport (and Edge) servers are the only Exchange
2007 servers that can be configured with anti-spam agents to filter
and cleanse the messaging stream before messages are routed to recipients. Hub transport servers can host anti-virus software to provide a
further level of protection. Mailbox servers can also host anti-virus
software. We will discuss how to deploy anti-spam and anti-virus
software on hub transport and Edge servers later on in this chapter.
Message Recovery: The hub transport server has a facility called the
transport dumpster that is used in conjunction with mailbox servers
running Cluster Continuous Replication (see Chapter 9). The dumpster is used to recover messages that were in transit when a cluster
outage occurs so that they can be redelivered to the mailbox server.
Messaging Records Management: Hub transport servers provide a
single point of contact for Exchange to be able to apply transport
and journal rules to messages as they flow through the organization.
Transport rules can apply disclaimers, take selective copies of
Chapter 6
6.3 Exchange 2007 transport architecture
messages, prevent groups of users communicating with each other,
and so on. Journal rules allow Exchange to take full copies of message traffic and move the data to another repository, such as an
HSM system. There’s a lot more detail about Messaging Records
Management explored in Chapter 8.
While you can describe the Store as the heart of Exchange, the message
flow capability provided by hub transport servers are the lungs because they
provide the means for Exchange to function and get work done as a messaging system.
Receive connectors
All versions of Exchange support SMTP connectors to allow bidirectional
email communications with other messaging systems that support SMTP.
Exchange 2000 made the fundamental switch in focus from X.400 to SMTP
and since then, SMTP has been the major method of communication for
Exchange. Exchange 2003 servers support SMTP virtual servers and SMTP
connectors that you configure to use the virtual servers to send and receive
messages. Exchange 2007 servers take a different approach.
Figure 6.2
Receive connectors
and properties
You can configure a receive connector to accept accepting incoming
messages from other Exchange 2007 servers, other SMTP servers, or POP3/
IMAP4 clients. Each receive connector is a “listener” that monitors a specific
port for incoming connections, so you’d configure a connector for SMTP
traffic to listen on port 25 and a connector to handle POP3/IMAP4 clients
6.3 Exchange 2007 transport architecture
on port 587. Unless you want to support unauthenticated connections, the
normal approach (and a change from the default configuration used by
Exchange 2000 and 2003) is to force POP3 and IMAP4 clients to authenticate before they can connect to port 587 and use an Exchange 2007 receive
connector. As we will see later on, you can also restrict the range of IP
addresses that a connector will accept connections from.
Receive connectors are scoped to a particular server and establish what
incoming messages that hub transport or Edge server is willing to accept.
When you install the hub transport role on a server, the installation creates
two receive connectors to allow the hub transport server to communicate
with other Exchange 2007 servers and to support clients. The two connectors are the “default” receive connector for interserver communications and a
receive connector for clients. The properties of a connector establish the connections that it can handle. As you can see in Figure 6.2, the default connector on this hub transport server can handle connections from:
Anonymous users
Exchange users
Exchange servers
Legacy Exchange servers
You can use the Get-Connector command to view details of the connector properties. For example:
Get-Connector –Server HPQBOX-ES90
-------HPQBOX-ES90\Default HPQBOX-ES90
By default, receive connectors on hub transport servers are configured to
accept authenticated connections from other Exchange servers, including
legacy Exchange servers in the same organization. In this respect, the default
configuration differs from the configuration shown in Figure 6.2 because the
“Anonymous users” permission group is excluded. Because of its outward facing role, the receive connector on an Edge server is configured automatically
to accept anonymous connections.
A permission group is a predefined set of permissions (in this case, for
the connector) that is granted to well-known security principals such as a
Chapter 6
6.3 Exchange 2007 transport architecture
user or group. Exchange 2007 uses permission groups to make it easier to
configure access to a connector. You can find details of all of the permission
groups and the permissions that each includes in TechNet online. For now,
Table 6.1 describes the AnonymousUsers and ExchangeUsers permission
Table 6.1
Example permission groups
Permission Group
Security Principal
Included Permissions
Anonymous User
Authenticated User
The default configuration for hub transport servers works well if all the
hub transport servers need is to receive incoming SMTP traffic on port 25
from other Exchange servers. However, if you want to receive incoming messages from other SMTP servers inside the company, including servers in
another Exchange organization, then you have to either update the receive
connector to include the “Anonymous users” permission group by selecting
its checkbox on the Permission Group property page, as shown in Figure 6.2
or create another receive connector to handle the connections. If you don’t
configure a receive connector to allow anonymous connections, users who
send messages from the other email systems will receive nondelivery notifications with an error code of 5.7.1, which means that the sending server was
unable to authenticate itself to Exchange and so the connection was terminated. Figure 6.3 shows the error message generated by an Exchange 2007
organization for a message that it couldn’t send to a user in another Exchange
2007 organization, both within the same company.
It might seem logical to you that the receive connector on hub transport
servers should be able to accept anonymous connections. Microsoft took a
deliberate decision to force administrators to upgrade the connectors if they
wanted to support anonymous connections to emphasize their philosophy of
“secure by default” and the role of the Edge server, which is explicitly
designed to accept and process incoming messages from external servers.
With security in mind, Microsoft’s recommendation is that you create a specific receive connector to handle connections from external SMTP servers. In
6.3 Exchange 2007 transport architecture
Figure 6.3
A receive connector
won’t handle
reality, it’s quite acceptable to update the default receive connection to permit
anonymous connections. Allowing anonymous connections effectively
means that “it’s okay to accept messages from other SMTP servers that aren’t
part of this Exchange organization.” If you prefer, you can use the SetReceiveConnector command to do the same thing:
Set-ReceiveConnector -id “Default HPQBOX-ES90”
–PermissionGroups AnonymousUsers, ExchangeUsers,
ExchangeServers, ExchangeLegacyServer –Comment “Updated
Connector to allow Anonymous Connections”
Updating the default receive connector to allow anonymous connections is the easy solution to the problem of accepting incoming messages
from external SMTP servers. However, this approach means that you use the
same configuration for internal and external messages. You may want to treat
external messages differently, for instance, you may allow users to send
10MB messages internally but want to restrict incoming messages to 5MB.
In this circumstance, you can create a separate receive connector and customize it to meet your requirements.
To create a new receive connector, select the hub transport server that
you want to host the connector in EMC and then select the “New Receive
Connector” option. The wizard that creates the connector first prompts for a
Chapter 6
6.3 Exchange 2007 transport architecture
Figure 6.4
Creating a new
receive connector
for external SMTP
name and usage type. The usage type determines the permission groups that
Exchange assigns to the connector. Table 6.2 lists the usage types defined by
Exchange 2007 and their meaning for a connector.
Table 6.2
Receive connector usage types
Usage type
Default Permission Groups
A connector to support IMAP4 and
POP3 clients. This type cannot be used
on Edge servers because Edge servers
don’t support client connections.
A connector tailored by the administrator to meet specific needs. For
example, to connect two Exchange
2007 organizations together.
None (needs to be determined
by the use of the connector)
None (needs to be
determined by the
use of the connector)
A connector to link Exchange 2007
and Exchange 2003/2000 servers.
Exchange Server
A connector to handle SMTP traffic
from external (non-Exchange) servers.
A connector to secure SMTP communications with a partner organization.
Basic plus TLS
None or externally
Requires TLS with
a remote domain
The connector illustrated in Figure 6.4 is of the Internet usage type and
will listen on port 25. The shell command used to create the connector is
shown below. Note the value passed in the –Usage parameter:
6.3 Exchange 2007 transport architecture
New-ReceiveConnector -Name 'SMTP from the Internet' -Usage
'Internet' -Bindings '' -Fqdn '' -Server
We’re not quite done when the connector is created because its settings
might not be fit for our purpose. We can check the complete configuration
with the Get-ReceiveConnector command, which outputs a lot of information that is not revealed by EMC:
Get-ReceiveConnector –id ‘SMTP from the Internet’ | Format-List
0.1 (8.0.535.0)
SMTP from the Internet
Chapter 6
6.3 Exchange 2007 transport architecture
We can now see that TLS is the authentication mechanism selected for
the connector and that it will accept messages from anonymous senders. This
is an appropriate setting if the external servers that we want to communicate
with support TLS, but we may want to start off by removing TLS as this will
allow any SMTP server to connect. Other parameters that we can tweak
include the maximum message size, the number of recipients allowed per
message, and the number of connections that we are willing to accept. If the
external SMTP servers generate older format messages that don’t support
extended SMTP, we may want to disable 8-bit MIME support, binary
MIME support, and chunking. All of these changes can be done with the
Set-ReceiveConnector command. Here’s an example that tunes the maximum message size down to 5MB, allows 250 recipients per message (if more
recipients are on a message, Exchange will accept for the first 250 and it’s up
to the sending server to resend for the remainder), and turns off TLS and any
other authentication. We also add a comment so that other administrators
understand the use of the connector and update the FQDN that is sent back
to a server after it makes a successful connection.
Set-ReceiveConnector –id ‘SMTP from the Internet’
–MaxMessageSize 5MB –MaxRecipientsPerMessage 250
–AuthMechanism None –Comment ‘Connector to support external
SMTP servers’ –fqdn ‘’
Figure 6.5
Updating a receive
6.3 Exchange 2007 transport architecture
If you’re concerned about opening up a receive connector to accept connections from any SMTP server, you can restrict the connections that it will
accept by defining a set of IP addresses that you’re willing to accept messages
from. Do this by removing the standard mask (–
and then adding the IP addresses that identify the servers that you want to
accept messages from. You can do this with individual IP addresses, IP
address ranges, an IP address together with a subnet, or an IP address in
Classless Interdomain Routing notation (for example, Enter
the IP address data through the Network property page for the receive connector, as shown in Figure 6.5.
The command to restrict access to a range of IP addresses for the default
connector on the hub transport server called ExchHub1 is:
Set-ReceiveConnector –id “ExchHub1\Default ExchHub1”
–RemoteIPRanges “”
Send connectors
Every hub transport server operates a default send connector. You will not see
any trace of this connector if you look for it through EMC, but you realize
that it must exist because Exchange 2007 is able to send messages off the
server. In fact, the default send connector is an integral part of the transport
system and is present in memory. You can get details of the default send connector with the Get-TransportServer command:
Get-TransportServer –Server ExchHub1 | Format-List
Like many other EMS commands, the output generated is verbose, but
provides an excellent insight into the properties that you can control for the
default send connector. An edited version of the output from Get-TransportServer is shown below.
Get-TransportServer –Server ExchHub1 | Format-List
Chapter 6
6.3 Exchange 2007 transport architecture
: 2.00:00:00
: 00:01:00
: True
: 30.00:00:00
: 250MB
: 10MB
: C:\Program Files\Microsoft\Exchange
MessageTrackingLogSubjectLoggingEnabled : True
OutboundConnectionFailureRetryInterval : 00:10:00
: None
: 64KB
: 100
: C:\Program Files\Microsoft\Exchange
: True
: 2
: 00:03:00
: 30.00:00:00
: 250MB
: 10MB
: C:\Program Files\Microsoft\Exchange
: C:\Program Files\Microsoft\Exchange
: 7.00:00:00
: 50MB
: C:\Program Files\Microsoft\Exchange
: 30.00:00:00
: 250MB
: 10MB
: C:\Program Files\Microsoft\Exchange
: 6
: 00:05:00
Use the Set-TransportServer command to update properties. As we
will see in Chapter 10, one common use of this command is to change the
properties that control the generation of message tracking logs:
Set-TransportServer –id ExchHub1 –MessageTrackingLogPath ‘L:\
Apart from the default send connector, custom send connectors are all
SMTP-based. You can create new send connectors to connect Exchange
2007 to the Internet (or to transfer messages to foreign SMTP-based email
6.3 Exchange 2007 transport architecture
systems) on a hub transport server in much the same way you’d create an
SMTP connector hosted on an Exchange 2000/2003 bridgehead server, but
there are some differences:
Send connectors can be configured on hub transport or Edge servers
and you configure and manage the two sets of connectors separately.
It’s likely that the Edge servers will be synchronized with a hub transport server to be able to route incoming messages for an organization.
In this case, you configure connector settings on the hub transport
server and synchronize details out to the Edge server through the
EdgeSync process. In fact, you can’t configure connectors on an Edge
server if it has established synchronization with a hub transport server.
Unlike receive connectors, which are created automatically when you
install a hub transport server and remain bound to that server, send
connectors are created at the organization level and are available for
any server to route messages across. However, before a server can use
a send connector to route messages, you have to add it as a valid
source to the send connector’s properties. Figure 6.6 shows that two
servers are able to send messages via the “HPQBOX to the Internet”
send connector. Both EMC and EMS flag warnings if you add a hub
transport server that is not in the same Active Directory site as the
hub transport servers that are already registered. The warning is to
notify you that you may have created a nonoptimum routing because
the transport service always uses deterministic routing and always
picks one Active Directory site to route messages. Therefore, the
transport service cannot load balance messages across two sites.
Instead, the transport service selects the first Active Directory site in
the connector’s source hub transport list and always routes messages
via that site.
If you know that the new source server has adequate network connectivity to support routing of messages from other Active Directory sites across
the connector, it is acceptable but not best practice to add it. Note that even
if a hub transport server is not registered as a valid source, it can still route
messages across the connector, but it must route the messages via one of the
valid source hub transport servers.
In most cases, a single custom send connector to handle external messages will suffice, but you can configure connectors to handle messages for
specific domains. When you configure a custom send connector, you need to
think about the settings to apply to the connector:
Chapter 6
6.3 Exchange 2007 transport architecture
Figure 6.6
Source servers for a
send connector
Like a receive connector, the usage of the send connector determines
the permission group for the connector: Internal means that the connector will connect to other Exchange servers; Internet means that
anonymous connections will be accepted; Custom means that you
will define permissions manually.
The address space that a connector supports allows you some control
over what messages flow across the connector. If you use an address
space of “*”, then the connector can handle any message to any
domain4. You may want to establish a connector that routes messages
to a specific domain, in which case, you would create a new connector and define the appropriate address space on that connector. Figure 6.7 shows a connector being configured to send messages to a
domain called However, let’s assume that you had
the need to connect to a specific domain in and want
to be sure that the messages go through a certain connector. You can
accomplish your goal by creating a connector that hosts the exact
address of the domain that you want to reach (for example, After you create the connector and estab-
Exchange 2007 doesn’t support SMTP address spaces of “*.*” or “@*.*”. Always use “*” if you want a connector to be
able to route messages to any SMTP domain.
6.3 Exchange 2007 transport architecture
lish the address space, Exchange has two available routes that it can
use for messages to the default SMTP connector and the one specifically created for this domain. Exchange
always routes messages across a connector with the most precise
address space, even if choosing this connector incurs a higher routing
cost, so any message sent to will go across the default SMTP
connector and any addressed to will always
travel across the connector that you’ve just created. This is not new
behavior in Exchange 2007 as this is the way that routing has worked
since Exchange 5.0.
Whether to use DNS MX (“mail exchange”) records to route email or
to send email through one or more smart host servers. A smart host is
a server that is able to route SMTP traffic on behalf of other servers.
Typically, this is a service offered by your ISP or a server that acts as a
central point for routing within a company.
The hub transport servers that are associated with the connector. You
can decide to create an association with an Edge subscription at this
point so that the connector is published through a hub transport
server to become known to an Edge server.
Figure 6.7
Creating a new
SMTP send
If you opt to use a smart host to route SMTP messages to external destinations, Exchange asks you to specify what kind of authentication mechanism to use when it connects to the smart host. It may be that you have an
SMTP server that’s set up as an open relay for general use within the company, in which case you won’t have to use authentication (the case shown in
Figure 6.8) or you may have to provide a username and password for basic
authentication. The smart host can be another Exchange server, in which
case you’d use Exchange Server Authentication, or it might support IPSec for
encrypted communication between the two servers.
Chapter 6
6.3 Exchange 2007 transport architecture
Figure 6.8
settings to route via
a smart host
The shell command to create the send connector that we have just
explored is:
New-SendConnector -Name 'SMTP to SystemX' -Usage 'Custom'
-AddressSpaces ';1' -DNSRoutingEnabled
$False -SmartHosts ''
-SmartHostAuthMechanism 'None' -UseExternalDNSServersEnabled
$False -SourceTransportServers ‘ExchHub1’
We can see the properties of the connector with the Get-SendConnector
command. As we experienced when working with receive connectors, you
have the opportunity to tweak many more settings through EMS than you
do through the GUI.
Get-SendConnector –id ‘SMTP to SystemX’
: {;1}
6.3 Exchange 2007 transport architecture
Microsoft MTA
SMTP to SystemX
SMTP to SystemX
Exchange Routing Group (DWBGZMFD01QNBJR)
If we need to change any settings, we can do it with the Set-SendConcommand. Let’s assume that we want to reduce the maximum message size that flows across the connector to 5MB. The command is:
Set-SendConnector –id ‘SMTP to SystemX’ –MaxMessageSize 5MB
–Comment ‘SystemX mailboxes are limited to 5MB’
Linking Exchange 2003 and Exchange 2007
During a migration project, you will have to decide how to connect Exchange
2007 servers into the existing routing infrastructure. Exchange 2007 servers
install into a special hidden routing group that is exposed to Exchange 2003
servers to allow them to communicate with Exchange 2007. When you install
the first Exchange 2007 server with the hub transport role, you are asked which
routing group you want to connect to so that the installation program can create a routing group connector between the Exchange 2007 routing group and
the routing group that you nominate.
Chapter 6
6.3 Exchange 2007 transport architecture
The simplest and best approach is to select a hub routing group to act as
the point of connection between Exchange 2003 and Exchange 2007.
Remember that this connector will act as the message interchange between
Exchange 2003 and Exchange 2007 and a lot of traffic will flow across the
connector during the migration, so you need to select a routing group that is
both appropriate for this role and one that can handle the load. The servers in
the routing group need to have solid, reliable, and fast network connections
with the rest of the Exchange 2003 organization. Picking a non-optimum
routing group isn’t a disaster—at least not immediately—but it could have
ramifications as you ramp up the message load across the connector as more
and more users migrate to Exchange 2007. For example, you may decide to
connect the first Exchange hub transport server with a routing group that is
not part of the core network, perhaps even a small routing group that occupies
a network spoke. Things will work just fine with a small group of users sending mail from Exchange 2007, but as traffic ramps, it doesn’t make sense to
funnel all messages from Exchange 2007 down into a network spoke and back
up again for delivery to other routing groups.
On the Exchange 2007 side, it is equally important that the hub transport server at the other side of the routing group connector is able to handle
the traffic, so you should also install this server into a core network site.
Figure 6.9
Exchange 2003
ESM displays the
special routing
Exchange 2007 installs its servers into a special administrative group
that includes a routing group. You can’t see these objects through EMC, but
they do appear through ESM (Figure 6.9). However, as noted earlier, you
can’t work with these objects through ESM as Exchange applies a block to
6.3 Exchange 2007 transport architecture
stop you doing anything except viewing their properties. For example, if you
click on a routing group connector that is created in the special routing
group to link Exchange 2007 and legacy servers, you can view all the properties, but you can’t change anything (Figure 6.10).
The question therefore is how do you create such a specialized connector?
The answer is in the New-RoutingGroupConnector command, which is
designed to handle all the work to establish bidirectional connectivity between
Exchange 2007 and legacy servers. Because all of the details of the Exchange
2007 servers are available to ESM, you can go through the process of setting
up a routing group connector using ESM to connect a bridgehead in another
routing group with a hub transport server in the special Exchange 2007 routing group, but you will also need to add the name of the legacy bridgehead
server to the ExchangeLegacyInterop security group (Figure 6.11) afterwards
as otherwise messages won’t flow across the connection. The New-RoutingGroupConnector command takes care of all the details, so it’s the recommended way to accomplish the task. For example:
New-RoutingGroupConnector –Name “RGC Exchange 2007 <->
Legacy” –SourceTransportServers “ExchHub1”
–TargetTransportServers “E2003Svr1” –Cost 50 –Bidirectional
Figure 6.10
Properties of the
special RGC
Chapter 6
6.3 Exchange 2007 transport architecture
This command creates a new routing group connector that links the
Exchange 2007 hub transport server called ExchHub1 with the Exchange
2003 bridgehead server called E2003Svr1. The cost of the link is set to 1
because we want this to be the default connection between the two parts of
the organization (we might create another routing group connector between
two other servers with a higher cost to act as a backup) and we’ve told
Exchange to create the necessary connectors on both sides. Afterwards, you
can check the properties of the connector with the Get-RoutingGroupConnector command (an edited version of the output is shown below):
Get-RoutingGroupConnector –id “RGC Exchange 2007 <-> Legacy” | Format-List
: RG-InterOp
: 50
: {E2003Svr1}
: /o=xyz/ou=Exchange Administrative Group (FYDI
RGC Exchange 2007 <-> Legacy
PublicFolderReferralsEnabled : True
: Exchange Routing Group (DWBGZMFD01QNBJR)
: {ExchHub1}
: 0.1 (8.0.535.0)
: RGC Exchange 2007 <-> Legacy
: RGC Exchange 2007 <-> Legacy
: {top, msExchConnector,
: True
Note that the cost of the connector is set reasonably high at 50 rather
than the default cost of 1 that is usually assigned to a new routing group connector. The logic here is that you do not usually want to use the routing
group connector between Exchange 2003 and Exchange 2007 as a backbone
connection as it’s best to keep messages within their native environment as
much as possible and only route them across to Exchange 2007 when their
final destination is on an Exchange 2007 server. Of course, a time will come
during the migration when Exchange 2007 is performing the majority of
routing or you take the decision to make Exchange 2007 the backbone mail
router for the organization, and in this case you can reduce the connector
6.3 Exchange 2007 transport architecture
Figure 6.11
The Exchange
security group
cost to allow Exchange 2003 servers to transfer messages across the connector
more often.
Remember that all of the Exchange 2007 servers appear to be in a single
routing group, so we’ve just created two connectors between that routing
group and the rest of the Exchange 2003 routing topology. Because we have
two routes that Exchange 2003 can take into Exchange 2007, we have created a situation where any outage might convince the Exchange 2003 servers
to route messages in a non-optimal manner and potentially even create a
loop. You can assume that no connectors will ever experience an outage, in
which case you don’t have to do anything, or you can take the pragmatic view
that software and networks tend to meet problems from time to time. In this
case, you want to stop link state updates propagating so that the Exchange
2003 servers never attempt to recalculate routes and will always use the
topology that you’ve just created.
There really isn’t anything more than that to setting up bidirectional message flow between Exchange 2007 and legacy servers, unless you still have some
Exchange 5.5 servers deployed. In this case, you’ll have to continue to route
messages from Exchange 5.5 through an Exchange 2003/2000 bridgehead
server on to Exchange 2007 as there is no way to establish a direct routing
group connection between Exchange 2007 and Exchange 5.5.
Chapter 6
6.3 Exchange 2007 transport architecture
Multiple routes into Exchange 2003
As long as you only have a single routing group connector between Exchange
2003 and Exchange 2007, you do not have to worry about link state information. Exchange 2003 will continue to send link state updates between its
routing groups but Exchange 2007 will simply drop the updates and ignore
them. The two sides will operate independently of each other and maintain
separate routing topologies. Operating a single connector will be sufficient
for small to medium organizations but things often become more complex
with larger organizations because of the need to handle bigger volumes of
traffic and to route messages efficiently within geographical network segments. In this situation, you probably need to configure additional routing
group connectors between appropriate Exchange 2003 servers and Exchange
2007 hub transport servers.
Decommissioning Exchange 2003 routing groups
Eventually, you will come to a point where you can start eliminating routing
group connectors because you have decommissioned Exchange 2003 servers
and emptied some routing groups. Another challenge now awaits you: how
to decommission servers without creating islands of link state replication. If
you remove connectors between Exchange 2003 routing groups and force
the Exchange 2003 servers that are left behind to route messages via
Exchange 2007, mail will flow but link state updates cannot because
Exchange 2007 ignores these updates. The upshot is that, in terms of routing topology, the Exchange 2003 servers will exist in a moment frozen in
time and will never be aware of any topology changes that occur elsewhere
in the organization. This may not be a bad thing if you are decommissioning servers fast and don’t expect the servers to have to function in the mode
for long, but in most cases (especially in large organizations), you can’t
switch over to Exchange 2007 quickly enough to avoid the potential that
messages will be routed ineffectively (or conceivably end up in a black hole
because Exchange can’t figure out where to send them). For this reason, any
time you remove a routing group connector, make sure that the Exchange
2003 servers that are left behind have an alternative connector that they can
use to receive link state updates.
Handling Exchange 2003 link state updates
during migration
You don’t have to care about link state updates if you are in a pure Exchange
2007 organization, but you do have to pay attention to them during migrations from Exchange 2003. Link state updates are a concept that Microsoft
introduced in Exchange 2000 where servers that act as routing group masters
6.3 Exchange 2007 transport architecture
in an organization communicated information about the routing topology
with each other so that servers would know when problems existed in the
topology, such as a connector being down or otherwise unavailable, and
would then be able to adjust the way that they routed messages to avoid the
problems. Dynamic routing was a great idea and it made a lot of sense as a
natural evolution from the static GWART (gateway and routing table) used
by Exchange 5.5. The problem with using a fixed view of the routing topology is that servers can send messages to a destination where they become
stuck until a connector is fixed and brought back into service.
Exchange 2007 doesn’t use link state information to decide how to route
messages. As we know, Exchange 2007 uses the underlying network as
defined in Active Directory sites to perform least cost routing between sites.
Microsoft’s view is that this approach puts routing decisions closer to the network layer so any problems become easier to diagnose. Instead of trying to
understand how changing network conditions and connector outages may
have caused Exchange to reroute messages, all you have to do is understand
how the network connects sites together (and sites are IP subnets, so you can
go down to that level) and you can figure out how messages flow. It’s also easier to estimate where messages are because you know that Exchange 2007
will queue them at the closest point of failure instead of attempting to
reroute them dynamically across nonoptimum routing groups. Collectively,
these steps make it much easier to understand message routing in complex
networks that span hundreds of Exchange servers. In smaller networks, where
you only operate a few servers, everything should run automatically and you
probably won’t care too much after you connect your sites together.
If you want to stop link state propagation, you have to make a registry
change on every Exchange 2003 bridgehead server. To do this, go to
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\RESvc\Parameters and add a new DWORD value called SuppressStateChanges. Set the
value to 1 (one) and then restart the Exchange Routing Engine, the SMTP
service, and the Exchange MTA service. Servers will continue to send link
state updates to each other but they will ignore any connector outages that
might force a topology change. If a connector experiences an outage,
Exchange will either reroute via another available connector or queue messages at that point until the connector is available again.
Foreign connectors
Foreign connectors link Exchange to other messaging systems. In the past,
installing foreign connectors was a necessary evil because different email systems used different protocols. Microsoft provided the Exchange Development Kit (EDK) to allow third parties and their own engineers to build
connectors for email systems such as Lotus Notes, Novell GroupWise, Lotus
cc:Mail, Fax, Short Message Service, and so on.
Chapter 6
6.3 Exchange 2007 transport architecture
The growing acceptance of SMTP as the de facto standard for email
exchange plus the addition of other Internet-driven standards such as iCal
permit much better out-of-the-box connectivity between email systems today
and have eliminated the need to operate many bespoke gateways. Microsoft
has removed support for gateways developed with the Exchange 2003 EDK,
so if you use one of these gateways, you will have to get an updated version
that can connect to the Exchange 2007 transport system. Microsoft has new
versions of their interoperability and connectivity tools for major systems such
as Lotus Notes and Novell GroupWise, and you can expect that the other
software vendors who have provided tools in the past will upgrade their products to support Exchange 2007. Note that connectivity is just one part of the
interoperability picture when you have to support multiple email systems. It’s
relatively easy to pass messages between the different systems, but achieving
the finer details such as directory synchronization and support for message
that contain meeting requests takes more effort.
The Exchange transport service uses Windows permissions to allow different
operations to proceed depending on where they originate. The permissions
model extends to the different SMTP verbs or events that occur during
message processing. For example, the transport service checks permission to
establish whether a sender is allowed to submit messages from a particular
email domain or submit a message of a certain size. The transport service can
perform authentication on a session basis to establish the permissions that are
available, which allows it to assign one set of permissions to messages
submitted by anonymous Internet senders and another set of permissions to
internal users.
Accepted domains
Accepted domains are email domains that you define to be those that you are
willing to accept email for and establish what recipients Exchange 2007
deems to be internal and what it deems to be external. In previous versions of
Exchange, we configured SMTP address spaces on connectors, but in
Exchange 2007, if you specify an accepted domain, you create an entry in the
Exchange organization’s configuration that says that the domain is one of
three types:
An authoritative domain is one that is hosted by the Exchange
organization. When you install the first Exchange 2007 hub
transport server in the organization, the installation procedure
automatically adds the FQDN of the forest root as an authorita-
6.3 Exchange 2007 transport architecture
tive receive domain, meaning that this Exchange organization is
able to receive and process incoming email sent to that domain.
You need to specify other authoritative domains if you want
Exchange to process incoming messages sent to these domains.
Any recipient that belongs to an authoritative domain is deemed
to be an internal recipient.
An internal relay domain is one that you are willing to accept
messages for recipients who do not have mailboxes, but are registered as mail-enabled contacts.
An external relay domain is one that you are willing to accept
messages and then relay them to a separate server for onward
routing and delivery.
You create additional domains on a hub transport server (at the organizational configuration level) and synchronize them to Edge servers through
EdgeSync. Figure 6.12 illustrates how to create a new accepted domain.
Behind the scenes, EMS creates the domain with:
New-AcceptedDomain -Name:'HPQBOX' -DomainName:''
In this case, we declare that our Exchange organization is willing to
accept messages sent to the domain, so presumably there are
some users in the organization who have email addresses. It is
reasonably common for an organization to support multiple email domains.
For example, an organization might include users who have joined the company through acquisition and you have allowed those users to retain the
email addresses from their old company (as secondary email addresses), so
that they can continue to receive messages from correspondents who only
have those addresses.
Edge servers reject incoming messages (with a DSN of 550 5.7.1 Unable
to Relay) that are addressed to domains that they do not recognize, so it is
clearly important to understand what domains you support and then to populate those domains out to the Edge servers.
Transport storage
Exchange 2003 stores messages in file system directories. Exchange 2007 uses
a Jet database called mail.que (similar to those used by the Active Directory
and the Store) as its repository, writing messages from memory into the database in 20MB batches or as required. For example, a message that causes a
Chapter 6
6.3 Exchange 2007 transport architecture
Figure 6.12
Creating a new
accepted domain
problem during delivery such as those regarded as “poison” (see section later
in this chapter), are written into the database immediately.
As shown in Figure 6.13, the default location for the mail.que database
is stored in \Program Files\Microsoft\Exchange Server\TransportRoles\Data\
Queue. If you look at the number of transaction logs that the transport service generates on a server, you can get some sense of how busy the server is.
In the case illustrated here, we know that the server is lightly trafficked
because it has only generated 104 (hex value of 67 plus the current log) 5MB
logs since the server was installed.
The vast majority of the transactions that flow through the mail queue
database are transient, so the database uses circular logging and 5MB transaction logs instead of the 1MB used by the Store. The current transaction log is
trn.log and, just like the Store databases, Exchange uses two reserved logs to
ensure that processing can continue even if disk space runs out. Given the
amount of available storage on most Exchange servers today, administrators
do not normally have to worry too much about the disk space occupied by
mail.que or its transaction logs unless you operate a very busy hub transport
server that handles a huge volume of messages. An Edge server that handles
thousands of messages per hour will have a large message queue database,
6.3 Exchange 2007 transport architecture
Figure 6.13
Transport queue
especially if you want to build in some capacity to handle an outage situation
when the Edge server cannot communicate with the hub transport servers
behind the firewall. For example, if your average message size is 100KB and
you want to hold a queue of up to 100,000 messages, then the message queue
database will be approximately 100GB. Your mileage will vary depending on
the average message size and the volume of traffic.
When the server has sufficient memory, the transport service stores
incoming messages in memory and commits them to the transaction logs,
using a similar mechanism to that employed by the Store to work with its
transactions. If memory becomes scarce and a back-pressure situation exists,
the transport service throttles back on its memory demands by caching only
the first 128KB of a message in memory and storing the remainder in the
database. In all cases, the transport service writes the message data into the
transaction log. When the transport service needs the complete message, it
streams the data out of the database and performs any content conversion
that is required using the TEMP directory (the directory pointed to by the
%TEMP% variable). For this reason, it is best for performance to colocate
the message queue database on the same disk LUN as the TEMP directory.
On the other hand, you should separate the message queue database and its
transaction logs.
The really interesting thing about the mail queue database is that you
can take it from one hub transport server and load it on any other Exchange
2007 hub transport server to have that server attempt to process any queued
mail in the database. Providing that the addresses on the messages are valid,
the hub transport server will route them to their final destination. This is a
valuable weapon to have in disaster-recovery scenarios, where you may have
problems on a hub transport server that require you to take it out of service
because you can move the mail queue database to the replacement server to
ensure that any waiting messages are processed.
Chapter 6
6.4 Routing ABC
Routing ABC
Before we look at different routing scenarios, we have enough detail to
present a picture of how the major components link together to establish
message flow in an example Exchange organization. Before we begin, let’s
recap some points:
The Active Directory Topology Service provides information to the
Exchange transport service about sites and site links. Exchange uses
this data to calculate the least-cost path for a message and to understand whether any hub routing sites occur along the path and if it
needs to bifurcate a message at any point.
Hub transport servers make point-to-point connections with other
hub transport servers to transfer messages as close as possible to the
final destination.
If a hub transport server is unavailable in the destination site, then
the message goes to a hub transport server in the closest site (as dictated by routing costs). If Exchange can’t reach a hub transport
server in the closest site, then it “backs off ” and attempts to transfer
the message to a hub transport server in the next closest site, and so
on until the message is eventually on its way. If it is impossible to
transfer a message to another site, Exchange queues it until a connection is possible.
Messages that arrive into an intermediate hub transport server stay
there until Exchange can transfer them to their final destination or to
a hub transport server in a closer site.
Figure 6.14 shows Exchange 2007 servers installed into four Active
Directory sites (Dublin, Paris, London, and Prague). Recall that you don’t
need to configure the send and receive connectors that allow messages to flow
between the hub transport servers between sites as this is done automatically
when you install the hub transport role on a server. Each of the sites support
some mailbox servers and the Dublin site hosts two other connectors: one is
a custom SMTP connector to route messages onwards to a smart host for
delivery to Internet addresses; the other links Exchange 2007 to the legacy
servers in the rest of the organization through a routing group connector. In
this example, Dublin and Paris are hub routing sites. Now that we have a
mental picture of how to link everything together, let’s see how routing works
in practice.
The simplest routing situation for the transport service is where a message sent on a mailbox server in Paris is addressed to a mailbox in Dublin.
6.4 Routing ABC
Figure 6.14
How the routing
pieces fit together
The message travels from the mailbox server to the Paris hub transport server
and is then transferred using a point-to-point SMTP connection over to the
Dublin hub transport, which then delivers it to the mailbox server that hosts
the destination mailbox.
Slightly more complex, a message from a mailbox in London to a mailbox in Dublin or Prague must travel through the hub transport server in Paris
because it is a hub routing site en route to Dublin. The same is true for messages generated in Prague, which flow through Paris en route to Dublin or
London. If the Paris hub transport server is unavailable, then messages queue
up because they cannot be transferred. If Paris was not defined as a hub routing site, then the hub transport server in London could make direct connections to the hub transport server in Prague or Dublin.
All messages to Exchange 2003 must flow through the Dublin hub
transport server because it is the only hub transport server that is associated
with the routing group connector. All messages flowing in from Exchange
2003 travel through the Dublin hub transport server, which then makes
the necessary connections to Paris to transfer messages that are not
addressed to recipients in Dublin. Remember that Paris is a hub routing
site, so messages from Exchange 2003 that are destined for mailboxes in
Prague and London flow through Paris. The same situation exists for Inter-
Chapter 6
6.4 Routing ABC
net traffic, which all travels through the send connector that the Dublin
hub transport server supports.
If Dublin supported two hub transport servers, each of which supports
the Internet and the routing group connector, then the transport service load
balances any messages flowing into Dublin across the two hub transport servers. If one of the hub transport servers goes offline for some reason, the transport service diverts all traffic through the online hub transport service. The
component called the Edge Connection Manager running within the transport service is responsible for establishing connections to remote servers,
such as hub transport servers in remote Active Directory sites and smart
SMTP routing hosts. It is the Edge Connection Manager that figures out
how to load balance connections so that messages flow as quickly as possible.
For example, if Paris has 60 messages queued to go to Dublin, it divides the
number of waiting messages by 20 to calculate the number of connections
that it should use. Dublin has two hub transport servers available, so Paris
will establish two connections to one hub transport server and one connection to the other. Each connection delivers the messages one at a time off the
queue and will disconnect when the queue empties. The number of messages
that is actually handled by a connection depends on the size of the message
and its attachments, the speed of the connection, and how quickly the other
connections process messages off the queue.
The number of connections that a hub transport server can establish at
any time can be controlled by a number of transport server parameters:
Get-TransportServer –id ExchHub1 | Select Max*
The important parameters here are:
MaxPerDomainOutboundConnections limits the number of connections that can be established for a single queue.
MaxOutboundConnections limits the total number of outbound
connections that can be established by the server.
6.4 Routing ABC
If you have good reason, use the Set-TransportServer command to set
a different value:
Set-TransportServer –id ExchHub1 –MaxOutBoundConnections 1200
Resolving multiple paths
Exchange 2007 routing is deterministic in that it a