6.0.0-git
2021-10-22
Last Modified 2012-05-03 by Michael Rubinsky

ActiveSync

Horde 4 has support for allowing Horde to sync with mobile devices using the mobile device's support for Microsoft Exchange/ActiveSync. The code that Horde's ActiveSync implementation uses for handling the protocol communication is based in part on the Z-Push library (http://z-push.sourceforge.net).

People

Michael Rubinsky

Description

The Horde ActiveSync implementation provides over the air synchronization of contacts, calendar, task, and Email (Horde 5) data to devices such as the iPhone/iPod Touch, Android and any other system supporting ActiveSync. Horde 4's implementation supports Microsoft ActiveSync protocol versions up to 2.5 -the version implemented by Microsoft Exchange 2003 while Horde 5 will support up to 12.0/12.1 - the version implemented by Microsoft Exchange 2007sp1.

Server Setup

ActiveSync support is stable, but there are still devices on which it has not been tested yet. If you have used a device with Horde's ActiveSync support that is not listed below, please feel free to let us know how it went.

To activate the server, it needs to be enabled in Horde's configuration, on the ActiveSync tab. It's already enabled by default. The SQL tables that horde uses are created as usual from the Horde configuration screen.

Next, you will need to configure your webserver to redirect the URL /Microsoft-Server-ActiveSync to your horde/rpc.php file. How you do this depends on your webserver and it's configuration. For Apache, something like:

Alias /Microsoft-Server-ActiveSync /var/www/horde/rpc.php

Note: It has been reported that when running PHP via mod_fcgid on Apache that the Alias directive will not pass the correct URL to the fcgid-script handler. This can be worked around by using a RewriteRule instead (adapted from http://maurus.net/weblog/2010/10/26/running-z-push-1-4-2-with-apache-and-fastcgifcgid/):

    RewriteEngine On
    RewriteRule ^/Microsoft-Server-ActiveSync /horde/rpc.php [PT,L,QSA]

There has also been a report from that the Authorization headers are not correctly passed when using mod_php with Apache. These are known issues and are should actually already be taken care of by the Horde_Controller_Request object. However, if you are still having issues with ActiveSync complaining about no Authorization errors, you can try the following configuration:
RewriteRule .* - [E=HTTP_MS_ASPROTOCOLVERSION:%{HTTP:Ms-Asprotocolversion}]
RewriteRule .* - [E=HTTP_X_MS_POLICYKEY:%{HTTP:X-Ms-Policykey}]
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]

None of these issues have been reported using lighttpd/fastcgi.

In order for the Autodiscovery service to work, you should also create an alias/rewrite rule for the URL /autodiscover/autodiscover.xml to horde/rpc.php as well:

Alias /autodiscover/autodiscover.xml /var/www/horde/rpc.php

For lighttpd:

alias.url = ("/Microsoft-Server-ActiveSync" => "/var/www/horde/rpc.php",
             "/autodiscover/autodiscover.xml" => /var/www/horde/rpc.php");

Application Configuration

No additional steps are normally necessary for synchronization of the supported applications. However, each application that supports synchronization also has a user preference to determine which shares will be synchronized. For example, in Kronolith the user's default calendar is *always* synchronized, but the user can choose to add any additional calendars he/she owns. Whenever the user changes one of these settings, the device is always automatically re-paired with the server to pick up the change.

Users can also view all their paired ActiveSync devices by visiting their ActiveSync Devices preferences. This is located within Horde's Global Preferences. From here, it is possible to force a complete re-sync, or to request a remote wipe of a provisioned device (see below).

Email Support

Email synchronization has been added in Horde 5. Since ActiveSync email support is inferior to many native email clients, this functionality is disabled by default. If you would like to enable Email support, you need to select this in the ActiveSync tab of Horde's configuration screen.

ActiveSync email support requires an IMAP server, not POP3. It will use the same server that IMP is configured to connect to. It is recommended that this server support the QRESYNC server extension for performance reasons, though it will work without this. It can also help performance if an IMAP proxy is used.

The only flag supported by ActiveSync is the seen flag. Flag changes will be synchronized, but flag changes alone will not trigger a SYNC (again, for performance reasons). The only thing that will trigger a SYNC is the arrival of a new message (technically, an increase in the NEXTUID value). Once this SYNC is triggered though, all message changes are taken into account.

Since ActiveSync does not support the deleted flag, messages in a mailbox with this flag are ignored when syncing. Deleting a message on the device will do one of two things; If the user has enabled a Trash mailbox then the message will be moved to that mailbox. Otherwise, the message is immediately expunged. This is in accordance with the ActiveSync protocol specs.

Forwarding a message will always attempt to put the main message text body in-line and keep any original attachments. It seems that a number of devices cannot view message/rfc822 attachments, so any messages that have been forwarded as an attachment may not be viewable in the ActiveSync mail client.

Autodiscovery

Some devices attempt to use Exchange's Autodiscovery service to make it easier for both the user to setup the account and for the administrator to make drastic changes like moving the server to a new URL. Horde attempts to support this as best it can. For this to work, you must create the URL alias as described above, and Horde must be able to figure out the Horde username based on the email address the user provided to the device. The configuration screen provides multiple options for this. In the worst case, if Horde cannot authenticate based on the provided information from the Autodiscover request, the device will fall back to requiring manual configuration.

ActiveSync Versions

Horde 5 adds support for ActiveSync version 12.0 - the version shipped with Exchange Server 2007. This adds HTML email support, flag or followup, more atomic policy settings, additional search sources, local wipe rules, and WBXML based provisioning (instead of the XML used in 2.5). There are a few things to mention regarding 12.0 support:

  • It seems some device, most notably older Android devices that will not automatically detect the new version support - even if the device is removed from Horde. You must actually remove the account from the device and configure it again. So, if you are not seeing 12.0 only features, be sure to try re-adding the account to the device.
  • It seems that iOS devices do NOT support 12.0. The device will communicate using version 2.5 with a 12.0 server. The next version they support is 12.1 - what ships with Exchange Server 2007sp1. Support for this version is planned for an upcoming Horde 5.x release.

Administration

Administrators can view all of the ActiveSync devices paired with the server. This is the ActiveSync Devices link located under the Administration menu. From here an administrator can request a remote wipe, or force a re-provisioning of any device.

@TODO: Explain various setup configuration options and security policies (hearbeat etc...)

Provisioning/RemoteWipe

Provisioning allows devices to be more tightly registered with a particular server. It enables the server to be able to send policy settings to the device. These policy settings include things like requiring a PIN to unlock the device, the complexity of the PIN required, the number of failed login attempts allowed etc... Additionally, it enables devices to be remotely wiped so that if a device is lost or stolen, the user or administrator can request the device to be wiped.

Users can initiate a remote wipe, as well as view/manage their partnered devices in the ActiveSync user preference.

Clicking Wipe in the Horde interfaces for device management flags the server to send the wipe command to the device the next time it synchronizes. The next time the device attempts to request a command other then PING or OPTIONS, it will be wiped. The ActiveSync preference page shows the status of all the user's devices. If the status is listed as Pending, and you wish to cancel the wipe request, you may do this by clicking the Cancel Wipe button. You should see the status be reset to Provisioned. After it is wiped, the status will be shown as Wiped, if you wish to allow the device to connect to your server again, you need to explicitly remove the device as a sync partner by clicking the Remove button. If you do not remove this entry, the device will continue to be wiped each time it reconnects to the server.

What works

Contacts, Calendar and Task syncing are working. Note that not all devices support Tasks. Of the tested devices, iOS (versions < 5.0) and Android are lacking native Task applications. The TouchDown client and Windows Mobile both support Tasks.

Client searching of an "Exchange Global Address List" (GAL) is supported, provided that Turba (or other application providing the contacts api) is configured to provide a GAL. See the Turba configuration for details.

Provisioning along with remote wipe is also working on devices that actually support it.

This code has been reported to work on the following devices:
Device Version(s) Provisioning GAL Searching Notes Verified EAS Versions
HP WebOS 2.1.0 Yes ? Contacts, Calendar, Tasks are working, for SSL with a private certificate you have to trust the certificate in the browser 2.5
HTC Desire Z / HTC Desire HD 2.2 yes yes Contacts and Calendar via native ActiveSync, SSL ok 2.5
HTC Desire S 2.3.3 ? ? Contacts and Calendar via native ActiveSync, SSL ok 2.5
HTC Magic Android 2.2.1 unbranded ? ? Contacts, Calendars 2.5
iOS Devices (iPhone, iPad, iPod) 3.1.3 -> 4.3.5 Yes, with Bugs. Certain versions of iOS - 4.3(8F190) for one, go into a provisioning loop due to a bug in iOS (it continues to send the OLD X-Ms-Policykey value after it receives a new one). Yes Contacts and Calendar 2.5, 12.1
iOS Devices (iPhone, iPad, iPod) 5.0 Yes Yes Contacts and Calendar, and basic support for Tasks via the Reminders App. 2.5, 12.1
Motorola Droid 2.0.1, 2.1, 2.2, 2.3 Broken support before 2.2, works with 2.2 and above. Native support in 2.2 and later, earlier versions can use the Corporate Directory app in the Marketplace. Contacts and Calendar data. On Froyo/2.2 SSL connections will NOT work with a self signed certificate even if the "Accept All Certificates" checkbox is selected. See http://www.google.com/support/forum/p/android/thread?tid=45e6836618212fdf&hl=en (A (Free) Level One certificate from http://www.startssl.com/ seems to work well here). 2.5, 12.0 (no 12.1)
Motorola Milestone 2.1, 2.2 See Motorola Droid above. See Motorola Droid above. Contacts and Calendar 2.5, 12.0
Moxier Mail 2.15.1 (Android) Yes Yes Minimal recurrence support in calendar. DOES NOT RESPECT SERVER SIDE STATE RESETS - so changing sync prefs, clearing state on server will require a manual resync on the device! 2.5, 12.0
Nokia E5-00 ? ? ? Contacts & calendar via RoadSync. Calendar works native client but contacts do not seem to work. 2.5
Nokia E90 MfE 3.0 ? ? Contacts verified to work. 2.5
Nokia N900 Maemo 1.3 No ? Client fails to send standard post variables, but Horde works around this. Contacts verified to work. Earlier versions of firmware are either completely broken, or only support ActiveSync version 12.1. 2.5
Samsung Galaxy Nexus 4.0.2 (ICS) Yes, full support. Yes, native support via the search functionality. Initial testing with Calendar and Contacts works. Be sure to ENABLE email syncing (even though Horde doesn't support it. Disabling email syncing - even if enabling calendar and contacts - seems to prevent the initial folder sync required for the account to be initially set up on the phone. 2.5, 12.0
TouchDown For Droid Version 6.5.0002 Yes Yes Contacts, Calendar (recurrence/exceptions mostly work - minor bugs still being worked out), and Tasks. 2.5, 12.1
Windows Mobile Device 6.1 ? ? Contacts reported to work. 2.5
Windows Phone 7 7.0 ? ? Contacts, Calendars 2.5

Setting up the device

It's beyond the scope of this page to go into detail for each individual device. In general, you will need to create a new account on the device. The account type should be something like Microsoft Exchange or ActiveSync. Some devices use Corporate. You will need to enter your normal Horde username and password in the appropriate fields. In the field for the server address, you should enter the root of the webserver or virtual host that hosts Horde. For example, if you host horde at http://host.example.com/horde then you should enter host.example.com. You can ignore any reference to a domain entry.

A special note for the iPhone/iPod (and possibly others) - if you do not use a SSL enabled site you may receive errors about not being able to find the ActiveSync server. If this happens, just continue, or save, or whatever your option is to continue. On the iPhone, after everything is completely set up, you must go back into the account settings and disable SSL.

After the connection particulars are entered, you should choose to enable the folders that you want sync'd. Contacts, Calendar and Tasks are supported, but your device also needs to support the requested folders.

What to do if you have problems (or How to help us help you)

If you are not even able to get past the initial setup page on your phone: you should first check to be sure you do not have SSL enabled on the phone when you're server is not serving SSL. The iPhone/iPod will not let you turn this off until after you save the configuration, so you must continue through all the errors and go back into the settings to disable SSL. You should also make sure that you have not enabled Provisioning support if your phone does not support it.

If the configuration went well, but you are not seeing any contacts/calendar items appear on the device: Some clients require a manual refresh or folder selection after setup when not using the "Automatic Discovery" facility of Exchange. With TouchDown, for example, after setup you must select the folders you want sync'd under the Advanced settings tab.

If all else fails and you can't figure out the issue, we will be happy to try to help you work it out, but you should be able to check/provide us with the following:

  • Be sure to check the notes in the chart above for your specific device. This might be a known issue, or a workaround may be known.
  • Check the web server error logs and see if there are any PHP errors being logged.
  • Configure Horde to send ActiveSync log messages to a separate logfile. This is configured on the ActiveSync tab of Horde's configuration screen.
  • If you are able to, it would also be useful to run a wireshark session to capture the network communication.
  • In some cases, it might be useful for us to see the affected device's state records in the database.

Using tshark (command line wireshark) to obtain a network capture

If you want to sniff the traffic on your server, and wireshark is not available becuase there is no windowing system, you can use the tshark application instead. The following command will capture http traffic on port 80, and will ignore most requests we are not interested in. It's worth mentioning that for the capture to be useful, you MUST not setup SSL on the device. Depending on your user's rights, you may need to run this as sudo:

tshark 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' -w /path/to/capture/file

Debug logging on device.

On Android devices, it is possible to enable debug level logging of the ActiveSync conversation as well:

To reach the Debug logging screen:
pre-Honeycomb: Go to the Account screen in the Email application and type debug.
Honeycomb (tablet): Go to the Account Settings screen using the action bar, then tap Email Preferences repeatedly until Debug appears in the account list. Tap Debug.
All phones: From the dialer, dial *#*#36245#*#* (the numbers correspond to EMAIL)
All devices: Go to the account creation screen (method differs depending on OS version) and enter d@d.d for email address and debug for password.

Here's what the checkboxes mean:

  • "Enable extra debug logging" - If checked, this turns on basic logging (no passwords or other content is logged); most useful for debugging crashes.
  • "Enable sensitive information debugging" - If checked, this causes POP/IMAP passwords to be logged. Exchange passwords are NEVER logged.
  • "Enable Exchange parser logging" - If checked, logs a great deal of the CONTENT of mail, contacts, and calendar entries (excluding the text of emails other than perhaps the first line). This information helpful for debugging things like missing folders, missing calendar entries, etc.
  • "Enable Exchange SD card logging" - If checked, puts all of the Exchange related logging onto SD card in a file named emaillog.txt. This file grows continuously until deleted (or the checkbox is unchecked). This can be helpful in finding issues that are sporadic, as the log is essentially unlimited in size.

Horde_ActiveSync vs. Z-Push

Horde_ActiveSync was based on Z-Push. The code that handles the protocol level is essentially the same, though it has been heavily refactored and cleaned.

Z-Push comes out of the box with a number of backends. The only one that is really fully functional is the "ICS" backend which connects to a Zarafa server. In addition to the ICS backend, Z-Push also provides a number of other backends - all of which extend what they call the "Diff" backend.

The diff backend is a very inefficient way of determining what needs to be synched. It uses file based storage - depending on the Z-Push version it uses either a single file or a directory of files for each device. These files contain, along with some basic device state information, a list of every UID that is on the device.

To determine what has changed, Z-Push essentially polls whatever storage backend e.g., the IMAP server, every $timeout seconds to get the full list of message IDs on the server. It then iterates over all the UIDs that are known to be on the device and stats every single one of these UIDs against the server to get the modification time, flags etc.

Like mentioned above, a number of backends are based on this Diff backend. Out of the box, if you are not using a Zarafa server you have the following options:

  • An IMAP backend - obviously only for syncing emails. This uses the PHP IMAP functions for fetching the complete list of message IDs and stating each and every UID - on each and every PING loop - by default something like every 10 seconds or so.
  • A vCard backend that syncs contacts against a directory of vCards on the server.
  • A Kolab backend - I'm not sure how complete this is, but this may be the backend that's used by Kolab's ActiveSync support.
  • A Maildir backend that - from what I can tell - syncs email by using filesystem based commands instead of going through an IMAP server.
  • A SearchLdap backend that - from what I can tell - is used as a Global Address Book source for searching for individual contacts (not for syncing the entire address book).

As of the time Horde_ActiveSync was written, you could only use one backend at a time - so, unless you were using Zarafa (or maybe Kolab) you could sync email or contacts. Since then they have started a "combined" backend that is supposed to wrap any number of backends. Last I checked it wasn't complete yet.

The main differences between 1.5.x and 2 are the versions of EAS that are supported. 1.5.x supports only up to Exchange 2003sp1 (same as Horde 4). Version 2 is supposedly going to support up to EAS 14 (Exchange 2010) - though I believe only up to 12.1 (Exchange 2007) is currently working.

The main differences

Some of these are specific to using Horde data as a backend to Horde_ActiveSync:

  1. Modularity. Separate classes for maintaining device state and for obtaining message diffs. If not using Horde as a backend, all that would theoretically be needed is to write a new class that extends Horde_ActiveSync_Driver_Base. This allows the backend, itself, to determine the best way to calculate diffs...like using Horde's History system. Support is also built in to allow using a different method of determining diffs based on the collection we are syncing. So, for example, if we ever implement email we would need a way of determining diffs that is not based on on the History system. This implementation allows us to easily add a new diff system and use multiple diff systems at once e.g., use History for contacts/calendar/tasks and some other diff system for email. In Z-Push, diff generation and state management are tightly coupled.
  2. Efficiency. The history based diff engine is orders of magnitude more efficient than using Z-Push's file based diff backend.
  3. Unless using Zarafa, Z-Push contained no message-specific logic. For example, no code for dealing with appointment related issues such as timezones, recurrence series etc. Horde_ActiveSync contains support for these things out of the box.
  4. At least at the time of writing, Horde allowed better configuration of things such as heartbeat/timeout intervals. We also actually allow configuring available security policies. Z-Push had basic support for provisioning and for turning on or off the requirement for a device PIN - but contained no facility for configuring any of the other options. Also, if using Horde_ActiveSync as part of a typical Horde install you get all the ease of configuration that our administration interface provides. I think some of this has improved as part of implementing more recent EAS versions in Z-Push.
  5. I don't have hard data on device compatibility as it compares to Z-Push, but I do know that we have fixed some device specific issues in the past that - at least at the time - were not working with Z-Push. Certain Nokia devices come to mind that use MfE. Z-Push lists these as "unknown" compatibility but they work, at least for contacts/calendar with Horde.
  6. Email sync with Horde_ActiveSync uses the vastly more efficient Horde_Imap_Client library, Z-Push uses the c-client PHP extension.
  7. Horde's email support is more complete than Z-Push's IMAP based implementation - with support for version 12.0 style email store searching, follow-up flag synchronization, and more.
  8. Horde_ActiveSync currently supports up to Exchange 2007 - Z-Push has code in development that works to some extent with 2007 and plans to look at supporting up to 2010.
  9. Horde_ActiveSync supports multiple users per device - when the device supports it. AFAIK, Z-Push only supports a single user account per device.

Todo

  • Implement EAS 12.1 (Exchange 2007sp1).

EAS 12.1 adds binary encoded request variables (reduced bandwidth), additional security policies, and a revamped SYNC command that does away with the need for an additional PING request (reduced bandwidth as well as reduced battery usage).

Resources

http://z-push.sourceforge.net
https://developer.berlios.de/project/showfiles.php?group_id=8963

Useful information/examples:
http://www.scribd.com/doc/6601589/W11-Server-Active-Sync
http://en.wikipedia.org/wiki/Exchange_ActiveSync - Good description of differences between AS versions.

Another AS implementation in PHP - there is some good implementation information there for the taking.
http://www.tine20.org/wiki/index.php/Developers/Getting_Started/Working_with_GIT

Not much here at all yet, but worth keeping an eye on:
http://code.google.com/p/libeas/

Some links to very detailed articles. Mostly more recent EAS versions though.
http://paulrobichaux.wordpress.com/2011/08/09/advice-to-exchange-activesync-developers/
http://blogs.msdn.com/b/exchangedev/

TouchDown Client:
http://www.nitrodesk.com/dk_touchdownFeatures.aspx

Android SDK:
http://developer.android.com/sdk/index.html

iPhone/iPod Touch EAS:
http://manuals.info.apple.com/en_US/Enterprise_Deployment_Guide.pdf

Windows Mobile Emulators:
The cellular network emulator:
http://www.microsoft.com/downloads/info.aspx?na=47&p=2&SrcDisplayLang=en&SrcCategoryId=&SrcFamilyId=38c46aa8-1dd7-426f-a913-4f370a65a582&u=details.aspx%3ffamilyid%3dA6F6ADAF-12E3-4B2F-A394-356E2C2FB114%26displaylang%3den

Device Images:
http://www.microsoft.com/downloads/details.aspx?familyid=38C46AA8-1DD7-426F-A913-4F370A65A582&displaylang=en

How to set it up:
http://www.devx.com/wireless/Article/40981/1954