6.0.0-git
2024-03-29
Last Modified 2012-11-06 by Michael Rubinsky

ActiveSync

Support for allowing Horde to sync with mobile devices using support for Microsoft Exchange/ActiveSync. This functionality has been available and stable since Horde 4.

People

Michael Rubinsky

Description

The Horde ActiveSync implementation provides over the air synchronization of contacts, calendar, task, and (since Horde 5) Email data to devices such as the iPhone/iPod Touch, Android and any other system supporting Exchange ActiveSync. Horde 4's implementation supports Microsoft ActiveSync protocol versions up to 2.5 -the version implemented by Microsoft Exchange 2003 while Horde 5 supports up to 12.1 - the version implemented by Microsoft Exchange 2007sp1.

As of Horde 5, ActiveSync support passes Microsoft's Remote Connectivity Analyzer - though you must disable provisioning on the account you use for testing since the analyzer doesn't respond to the 449 Header that is sent when Provisioning is required.

Server Setup

ActiveSync support is stable, but there are still devices on which it has not been tested yet. If you have used a device with Horde's ActiveSync support that is not listed below, please feel free to let us know how it went.

To activate the server, it needs to be enabled in Horde's configuration, on the ActiveSync tab. The SQL tables that horde uses are created as usual from the Horde configuration screen.

You will also need to configure your webserver to redirect the URL /Microsoft-Server-ActiveSync to your horde/rpc.php file. How you do this depends on your webserver and it's configuration. For Apache, something like:

Alias /Microsoft-Server-ActiveSync /var/www/horde/rpc.php

Note: It has been reported that when running PHP via mod_fcgid on Apache that the Alias directive will not pass the correct URL to the fcgid-script handler. This can be worked around by using a RewriteRule instead (adapted from http://maurus.net/weblog/2010/10/26/running-z-push-1-4-2-with-apache-and-fastcgifcgid/):

    RewriteEngine On
    RewriteRule ^/Microsoft-Server-ActiveSync /horde/rpc.php [PT,L,QSA]

There has also been a report from that the Authorization headers are not correctly passed when using mod_php with Apache. These are known issues and are should actually already be taken care of by the Horde_Controller_Request object. However, if you are still having issues with ActiveSync complaining about no Authorization errors, you can try the following configuration:
RewriteRule .* - [E=HTTP_MS_ASPROTOCOLVERSION:%{HTTP:Ms-Asprotocolversion}]
RewriteRule .* - [E=HTTP_X_MS_POLICYKEY:%{HTTP:X-Ms-Policykey}]
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]

None of these issues have been reported using lighttpd/fastcgi.

In order for the Autodiscovery service to work, you should also create an alias/rewrite rule for the URL /autodiscover/autodiscover.xml to horde/rpc.php as well:

Alias /autodiscover/autodiscover.xml /var/www/horde/rpc.php

Note that the Autodiscovery service attempts to contact domain used in the email address, not necessarily the sub domain that Horde is installed at so e.g., if your Horde install is at https://horde.example.com but your users use user@example.com then you will need to redirect requests for https://example.com/AutoDiscover/AutoDiscover.xml to https://horde.example.com/AutoDisocover/AutoDiscover.xml:

For lighttpd:

$HTTP["host"] =~ "(^|www\.)example\.com$" {
    url.redirect = ("^/(?i)AutoDiscover/AutoDiscover.xml$" => "https://horde.example.com/autodiscover/autodiscover.xml")
}

alias.url = ("/Microsoft-Server-ActiveSync" => "/var/www/horde/rpc.php",
             "/autodiscover/autodiscover.xml" => /var/www/horde/rpc.php");

Note that Autodiscovery support is VERY flaky in most clients that we have tested.

Application Configuration

No additional steps are normally necessary for synchronization of the supported applications. However, each application that supports synchronization also has a user preference to determine which shares will be synchronized. For example, in Kronolith the user's default calendar is *always* synchronized, but the user can choose to add any additional calendars he/she owns. Whenever the user changes one of these settings, the device is always automatically re-paired with the server to pick up the change.

Users can also view all their paired ActiveSync devices by visiting their ActiveSync Devices preferences. This is located within Horde's Global Preferences. From here, it is possible to force a complete re-sync, or to request a remote wipe of a provisioned device (see below).

Email Support

Email synchronization has been added in Horde 5. Since in some installs this could increase server load or may be otherwise undesirable, it is possible to deactivate email support via Horde's configuration, on the ActiveSync tab.

ActiveSync email support requires an IMAP server POP3 is not supported. It will use the same server that IMP is configured to connect to. It is recommended that this server support the QRESYNC server extension for performance reasons, though it will work without this. It can also help performance if an IMAP proxy is used.

The only flags supported by ActiveSync are the seen and flagged for follow up flags. Flag changes will be synchronized, but flag changes alone will not trigger a SYNC for performance reasons. The only thing that will trigger a SYNC is the arrival of a new message (technically, an increase in the NEXTUID value). Once this SYNC is triggered though, all message changes are taken into account - including any flag changes.

Since ActiveSync does not support the deleted flag, messages in a mailbox with this flag are ignored when syncing. Deleting a message on the device will do one of two things; If the user has enabled a Trash mailbox then the message will be moved to that mailbox. Otherwise, the message is immediately expunged. This is in accordance with the ActiveSync protocol specs.

Forwarding a message will always attempt to put the main message text body in-line and keep any original attachments. It seems that a number of devices cannot view message/rfc822 attachments, so any messages that have been forwarded as an attachment may not be viewable in the ActiveSync mail client.

Autodiscovery

Some devices attempt to use Exchange's Autodiscovery service to make it easier for both the user to setup the account and for the administrator to make drastic changes like moving the server to a new URL. Horde attempts to support this as best it can. For this to work, you must create the URL alias as described above, and Horde must be able to figure out the Horde username based on the email address the user provided to the device. The configuration screen provides multiple options for this. In the worst case, if Horde cannot authenticate based on the provided information from the Autodiscover request, the device will fall back to requiring manual configuration.

ActiveSync Versions

Horde 5 adds support for ActiveSync versions 12.0 and 12.1 - the version shipped with Exchange Server 2007 and 2007sp1. This adds among other things: HTML email support, flagged for followup, more atomic policy settings, additional search sources, local wipe rules, and WBXML based provisioning (instead of the XML used in 2.5). There are a few things to mention regarding 12.0 support:

  • It seems some devices, most notably older Android devices, will not automatically detect the new version support - even if the device is removed from Horde. You must actually remove the account from the device and configure it again. So, if you are not seeing 12.0/12.1 only features, be sure to try re-adding the account to the device.
  • It seems that iOS devices do NOT support 12.0. The device will communicate using version 2.5 with a 12.0 server. The next version they support is 12.1 - what ships with Exchange Server 2007sp1. So, if you are seeing 2.5 only features on an iOS device, be sure the Horde server has been configured to support 12.1. If it has, then see the first point above.

Administration

Administrators can view all of the ActiveSync devices paired with the server. This is the ActiveSync Devices link located under the Administration menu. From here an administrator can request a remote wipe, or force a re-provisioning of any device.

@TODO: Explain various setup configuration options and security policies (hearbeat etc...)

Provisioning/RemoteWipe

Provisioning allows devices to be more tightly registered with a particular server. It enables the server to be able to send policy settings to the device. These policy settings include things like requiring a PIN to unlock the device, the complexity of the PIN required, the number of failed login attempts allowed etc... Additionally, it enables devices to be remotely wiped so that if a device is lost or stolen, the user or administrator can request the device to be wiped.

As of Horde 5, provisioning is enabled via the permissions interface. In order to enforce any security policies on a device, it most be provisioned. However, not all devices support this and some will downright refuse to work if it's enabled. There are three choices for provisioning support. None, Force, and Allow. Choosing None will disable provisioning and any enforcement of security polices or remote wipe. Force will only allow devices that are successfully provisioned to connect to the server. This means devices that don't properly support provisioning, such as some older Android versions, will simply not work. The third choice, Allow will enforce provisioning on the devices that support it, but will also allow devices that don't support it to connect to the server. Once provisioning support is added, security policies can also be added via the permissions interface.

Users can initiate a remote wipe, as well as view/manage their partnered devices in the ActiveSync user preference.

Clicking Wipe in the Horde interfaces for device management flags the server to send the wipe command to the device the next time it synchronizes. The next time the device attempts to request a command other then PING or OPTIONS, it will be wiped. The ActiveSync preference page shows the status of all the user's devices. If the status is listed as Pending, and you wish to cancel the wipe request, you may do this by clicking the Cancel Wipe button. You should see the status be reset to Provisioned. After it is wiped, the status will be shown as Wiped, if you wish to allow the device to connect to your server again, you need to explicitly remove the device as a sync partner by clicking the Remove button. If you do not remove this entry, the device will continue to be wiped each time it reconnects to the server.

What works

Contacts, Calendar, Task and Email syncing are all working. Note that not all devices support Tasks. Of the tested devices, iOS (versions < 5.0) and Android are lacking native Task applications. The TouchDown client, Moxier Mail, and Windows Mobile both support Tasks.

Client searching of an "Exchange Global Address List" (GAL) is supported, provided that Turba (or other application providing the contacts api) is configured to provide a GAL. See the Turba configuration for details.

Provisioning along with remote wipe is also working on devices that actually support it.

This code has been reported to work on the following devices:
Device Version(s) Provisioning GAL Searching Notes Verified EAS Versions
BlackBerry PlayBook 2.1.0.1088 ? ? Emails, contacts and calendars are synced, but the PlayBook often resets the sync when connecting to 2 accounts 12.1
Google Nexus 7 4.1.2 ? ? Only searching emails works. Email folders and calendars are empty 12.1
HP WebOS 2.1.0 Yes ? Contacts, Calendar, Tasks are working, for SSL with a private certificate you have to trust the certificate in the browser 2.5
HTC Desire Z / HTC Desire HD 2.2 Yes Yes Contacts and Calendar via native ActiveSync, SSL ok 2.5
HTC Desire S 2.3.3 ? ? Contacts and Calendar via native ActiveSync, SSL ok 2.5
HTC Magic Android 2.2.1 unbranded ? ? Contacts, Calendars 2.5
iOS Devices (iPhone, iPad, iPod) 3.1.3 -> 4.3.5 Yes, with Bugs. Certain versions of iOS - 4.3(8F190) for one, go into a provisioning loop due to a bug in iOS (it continues to send the OLD X-Ms-Policykey value after it receives a new one). Yes Contacts, Calendar and Email 2.5, 12.1
iOS Devices (iPhone, iPad, iPod) 5.x Yes Yes Contacts, Calendar, Email, and basic support for Tasks via the Reminders App. 2.5, 12.1
iOS Devices (iPhone, iPad, iPod) 6.0 Yes Yes Broken email push, issues with meeting invitations and responses. 2.5, 12.1
Motorola XT910 Android 2.3.6 Yes ? Contacts, calendar data tasks and remote wipe work. SSL works as well. ?
Motorola Droid 2.0.1, 2.1, 2.2, 2.3 Broken support before 2.2, works with 2.2 and above. Native support in 2.2 and later, earlier versions can use the Corporate Directory app in the Marketplace. Contacts, Calendar and Email. On Froyo/2.2 SSL connections will NOT work with a self signed certificate even if the "Accept All Certificates" checkbox is selected. See http://www.google.com/support/forum/p/android/thread?tid=45e6836618212fdf&hl=en (A (Free) Level One certificate from http://www.startssl.com/ seems to work well here). 2.5, 12.0
Motorola Milestone 2.1, 2.2 See Motorola Droid above. See Motorola Droid above. Contacts, Calendar, and Email 2.5, 12.0
Moxier Mail 2.15.1 (Android) Yes Yes Contacts, Calendar with minimal recurrence support and Email. DOES NOT RESPECT SERVER SIDE STATE RESETS - so changing sync prefs, clearing state on server will require a manual resync on the device! 2.5, 12.0
Nokia E5-00 ? ? ? Contacts & calendar via RoadSync. Calendar works native client but contacts do not seem to work. 2.5
Nokia E90 MfE 3.0 ? ? Contacts verified to work. 2.5
Nokia N900 Maemo 1.3 No ? Client fails to send standard post variables, but Horde works around this. Contacts verified to work. Earlier versions of firmware are either completely broken, or only support ActiveSync version 12.1. 2.5
Samsung Galaxy Gio 2.3.6 Yes Yes Contacts and calendar data works. SSL works as well. 2.5
Samsung Galaxy Nexus 4.0.2 (ICS) Yes, full support. Yes, native support via the search functionality. Calendar, Contacts, and Email works. Be sure to ENABLE email syncing. Disabling email syncing - even if enabling calendar and contacts - seems to prevent the initial folder sync required for the account to be initially set up on the phone. 2.5, 12.0, 12.1
Samsung Galaxy S2 2.3.4 ? ? GT-I9100 - Contacts and calendar data works. SSL works as well (self-signed certificates also). 2.5
Samsung Galaxy S2 4.0.3 (ICS) Yes, full support. Yes, native support via the search functionality. GT-I9100 - Calendar, Contacts and Email works. Be sure to ENABLE email syncing. Disabling email syncing - even if enabling calendar and contacts - seems to prevent the initial folder sync required for the account to be initially set up on the phone. SSL works as well (self-signed certificates also). 2.5, 12.0
TouchDown For Droid Version 6.5.0002 Yes Yes Contacts, Calendar (recurrence/exceptions mostly work - minor bugs still being worked out), Tasks, and Email. 2.5, 12.1
Windows Mobile 6.1 ? ? Contacts, Email 2.5
Windows Mobile 6.5 Yes, full support. Yes Contacts, Calendar, Tasks, Email. 2.5, 12.0, 12.1
Windows Phone 7.0 Yes, with limited security policy support. Yes Contacts, Calendar, Tasks, Email 2.5, 12.0, 12.1
Windows Phone 7.5 Yes, with limited security policy support. Yes Contacts, Calendar, Tasks, Email. Some devices may require a Deleted items folder to be enabled. If error 8004010F is displayed on the device, this is the likely culprit. See http://social.msdn.microsoft.com/Forums/en/os_exchangeprotocols/thread/86e10e2d-bc4d-43dc-b6b7-f02630ff052b for more info. 2.5, 12.0, 12.1

Setting up the device

It's beyond the scope of this page to go into detail for each individual device. In general, you will need to create a new account on the device. The account type should be something like Microsoft Exchange or ActiveSync. Some devices use Corporate. You will need to enter your normal Horde username and password in the appropriate fields. In the field for the server address, you should enter the root of the webserver or virtual host that hosts Horde. For example, if you host horde at http://host.example.com/horde then you should enter host.example.com. You can ignore any reference to a domain entry. If the device requires the domain entry (some Windows Mobile devices do this) you may safely enter any value.

A special note for the iPhone/iPod (and possibly others) - if you do not use a SSL enabled site you may receive errors about not being able to find the ActiveSync server. If this happens, just continue, or save, or whatever your option is to continue. On the iPhone, after everything is completely set up, you must go back into the account settings and disable SSL.

After the connection particulars are entered, you should choose to enable the folders that you want sync'd.

What to do if you have problems (or How to help us help you)

First off, you should check the list of known issues to see if your problem is expected or not. You can also check the Horde bug tracker to see if your issue has been reported already.

If you are not even able to get past the initial setup page on your phone: you should first check to be sure you do not have SSL enabled on the phone when you're server is not serving SSL. The iPhone/iPod will not let you turn this off until after you save the configuration, so you must continue through all the errors and go back into the settings to disable SSL. You should also make sure that you have not enabled Provisioning support if your phone does not support it.

If the configuration went well, but you are not seeing any contacts/calendar items appear on the device: Some clients require a manual refresh or folder selection after setup when not using the "Automatic Discovery" facility of Exchange. With TouchDown, for example, after setup you must select the folders you want sync'd under the Advanced settings tab.

If all else fails and you can't figure out the issue, we will be happy to try to help you work it out, but you should be able to check/provide us with the following:

  • Be sure to check the notes in the chart above for your specific device. This might be a known issue, or a workaround may be known.
  • Check the web server error logs and see if there are any PHP errors being logged.
  • Configure Horde to send ActiveSync log messages to a separate logfile. This is configured on the ActiveSync tab of Horde's configuration screen.
  • If you are able to, it would also be useful to run a wireshark session to capture the network communication.
  • In some cases, it might be useful for us to see the affected device's state records in the database.

Using tshark (command line wireshark) to obtain a network capture

If you want to sniff the traffic on your server, and wireshark is not available becuase there is no windowing system, you can use the tshark application instead. The following command will capture http traffic on port 80, and will ignore most requests we are not interested in. It's worth mentioning that for the capture to be useful, you MUST not setup SSL on the device. Depending on your user's rights, you may need to run this as sudo:

tshark 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' -w /path/to/capture/file

Debug logging on device.

On Android devices, it is possible to enable debug level logging of the ActiveSync conversation as well:

To reach the Debug logging screen:
pre-Honeycomb: Go to the Account screen in the Email application and type debug.
Honeycomb (tablet): Go to the Account Settings screen using the action bar, then tap Email Preferences repeatedly until Debug appears in the account list. Tap Debug.
All phones: From the dialer, dial *#*#36245#*#* (the numbers correspond to EMAIL)
All devices: Go to the account creation screen (method differs depending on OS version) and enter d@d.d for email address and debug for password.

Here's what the checkboxes mean:

  • "Enable extra debug logging" - If checked, this turns on basic logging (no passwords or other content is logged); most useful for debugging crashes.
  • "Enable sensitive information debugging" - If checked, this causes POP/IMAP passwords to be logged. Exchange passwords are NEVER logged.
  • "Enable Exchange parser logging" - If checked, logs a great deal of the CONTENT of mail, contacts, and calendar entries (excluding the text of emails other than perhaps the first line). This information helpful for debugging things like missing folders, missing calendar entries, etc.
  • "Enable Exchange SD card logging" - If checked, puts all of the Exchange related logging onto SD card in a file named emaillog.txt. This file grows continuously until deleted (or the checkbox is unchecked). This can be helpful in finding issues that are sporadic, as the log is essentially unlimited in size.

Horde_ActiveSync vs. Z-Push

The protocol handling in Horde_ActiveSync was based on Z-Push. The code that handles the protocol level is essentially the same, though it has been heavily refactored and cleaned.

Z-Push comes out of the box with a number of backends. The only one that is really fully functional is the "ICS" backend which connects to a Zarafa server. In addition to the ICS backend, Z-Push also provides a number of other backends - all of which extend what they call the "Diff" backend.

The diff backend is a very inefficient way of determining what needs to be synched. It uses file based storage - depending on the Z-Push version it uses either a single file or a directory of files for each device. These files contain, along with some basic device state information, a list of every UID that is on the device.

To determine what has changed, Z-Push essentially polls whatever storage backend e.g., the IMAP server, every $timeout seconds to get the full list of message IDs on the server. It then iterates over all the UIDs that are known to be on the device and stats every single one of these UIDs against the server to get the modification time, flags etc.

Like mentioned above, a number of backends are based on this Diff backend. Out of the box, if you are not using a Zarafa server you have the following options:

  • An IMAP backend - obviously only for syncing emails. This uses the PHP IMAP functions for fetching the complete list of message IDs and stating each and every UID - on each and every PING loop - by default something like every 10 seconds or so.
  • A vCard backend that syncs contacts against a directory of vCards on the server.
  • A Kolab backend - I'm not sure how complete this is, but this may be the backend that's used by Kolab's ActiveSync support.
  • A Maildir backend that - from what I can tell - syncs email by using filesystem based commands instead of going through an IMAP server.
  • A SearchLdap backend that - from what I can tell - is used as a Global Address Book source for searching for individual contacts (not for syncing the entire address book).

As of the time Horde_ActiveSync was written, you could only use one backend at a time - so, unless you were using Zarafa (or maybe Kolab) you could sync email or contacts. Since then they have started a "combined" backend that is supposed to wrap any number of backends. Last I checked it wasn't complete yet.

The main differences between 1.5.x and 2 are the versions of EAS that are supported. 1.5.x supports only up to Exchange 2003sp1 (same as Horde 4). Version 2 is supposedly going to support up to EAS 14 (Exchange 2010) - though I believe only up to 12.1 (Exchange 2007) is working...and I don't believe that is even fully functional.

The main differences

Some of these are specific to using Horde data as a backend to Horde_ActiveSync:

  1. Modularity. Separate classes for maintaining device state and for obtaining message diffs. If not using Horde as a backend, all that is needed is to write a new class that extends Horde_ActiveSync_Driver_Base. This allows the backend, itself, to determine the best way to calculate diffs...like using Horde's History system. In Z-Push, diff generation and state management are tightly coupled.
  2. Efficiency. The history based diff engine is orders of magnitude more efficient than Z-Push's file based diff backend.
  3. Unless using Zarafa, Z-Push contained no message-specific logic. For example, no code for dealing with appointment related issues such as timezones, recurrence series etc. Horde_ActiveSync contains support for these things out of the box.
  4. At least at the time of writing, Horde allowed better configuration of things such as heartbeat/timeout intervals. We also actually allow configuring available security policies. Z-Push had basic support for provisioning and for turning on or off the requirement for a device PIN - but contained no facility for configuring any of the other options without editing code. Also, if using Horde_ActiveSync as part of a typical Horde install you get all the ease of configuration that our administration interface provides.
  5. I don't have hard data on device compatibility as it compares to Z-Push, but I do know that we have fixed some device specific issues in the past that - at least at the time of this writing - were not working with Z-Push. Certain Nokia devices come to mind that use MfE. Z-Push lists these as "unknown" compatibility but they work, at least for contacts/calendar with Horde. Remote wiping iOS devices is also problematic in Z-Push.
  6. Email sync with Horde_ActiveSync uses the vastly more efficient Horde_Imap_Client library and takes full advantage of IMAP servers with QRESYNC capabilities. Z-Push uses the very poor performing c-client PHP extension.
  7. Horde's email support is more complete than Z-Push's IMAP based implementation - with support for version 12.0 style email store searching, follow-up flag synchronization, S/Mime signatures and encryption, and more.
  8. Horde_ActiveSync currently supports almost the full feature set up to Exchange 2007sp1 - Z-Push has code in development that works to some extent with 2007 and plans to look at supporting up to 2010.
  9. Horde_ActiveSync supports multiple users per device - when the device supports it. AFAIK, Z-Push only supports a single user account per device.

Resources

http://z-push.sourceforge.net
https://developer.berlios.de/project/showfiles.php?group_id=8963

Useful information/examples:
http://www.scribd.com/doc/6601589/W11-Server-Active-Sync
http://en.wikipedia.org/wiki/Exchange_ActiveSync - Good description of differences between AS versions.

Another AS implementation in PHP - there is some good implementation information there for the taking.
http://www.tine20.org/wiki/index.php/Developers/Getting_Started/Working_with_GIT

Not much here at all yet, but worth keeping an eye on:
http://code.google.com/p/libeas/

Some links to very detailed articles. Mostly more recent EAS versions though.
http://paulrobichaux.wordpress.com/2011/08/09/advice-to-exchange-activesync-developers/
http://blogs.msdn.com/b/exchangedev/

TouchDown Client:
http://www.nitrodesk.com/dk_touchdownFeatures.aspx

Android SDK:
http://developer.android.com/sdk/index.html

iPhone/iPod Touch EAS:
http://manuals.info.apple.com/en_US/Enterprise_Deployment_Guide.pdf

Windows Mobile Emulators:
The cellular network emulator:
http://www.microsoft.com/downloads/info.aspx?na=47&p=2&SrcDisplayLang=en&SrcCategoryId=&SrcFamilyId=38c46aa8-1dd7-426f-a913-4f370a65a582&u=details.aspx%3ffamilyid%3dA6F6ADAF-12E3-4B2F-A394-356E2C2FB114%26displaylang%3den

Device Images:
http://www.microsoft.com/downloads/details.aspx?familyid=38C46AA8-1DD7-426F-A913-4F370A65A582&displaylang=en

How to set it up:
http://www.devx.com/wireless/Article/40981/1954