Horde Syncing Roadmap

Things that need to be worked on

Bugs/Limitations

• The client cannot split data in multiple messages or deal with such splits from the client
  • Synthesis support has to be completed for tasks/calendar/address book
  • History retrival has to be fixed to narrow results to meaningful entries
  • The various vcard/vcalendar import/export functions really need some work. Recurring events are not dealt with at all yet.
  • Initial SlowSync of Evens does millions of datatree reads and still takes way too much time. Somethings wrong with that.
  • when server has a matching anhor it will always do a delta sync, even when the client requests a slowsync

Possible Extensions

• Allow customizations which calendars/addressbooks should be synced. Maybe allow merging of more than one data source (think of private and shared calendar)
  • Proper conflict resolution. current policy is: client wins. (As the client send it changes first.)
  • Aligning Client and Server categories
  • Fancy features like 'delete old entries from client calendar/todo list' etc. to save space on client.

Other stuff:

Protocol Issues

Once the current code makes it to CVS people should give it a try and check if it works. If not, the xml in /tmp/sync/, the horde log and the php error log should be helpful. If you can provide a patch to make it work: great!

Otherwise mail the offending xml and/or logs to the list. I'll have a look at it.

As many phones/pdas/whater should be thrown at the code. I'm especially interested to see the <devinf> device information section that the phones sends in its initial packet. This gets stored in /tmp/sync/syncml_client_0.xml when you start with an empty /tmp/sync/ dir.

Make sure to modify the <cred>...</cred> part in the SyncML header if it exists. It contains your username and password.

I'll try to come up with a brief "how to help with debugging" and add it to the installation docs. Otherwise we might get lots of "just installed the horde rpm and SyncML doesn't work with my XXYY phone! pls hlp!!!! I'm new to linux!!!!!" messages :-)

Data Issues

The second main area where I expect problems are the various implementations of the text/x-vcalendar,vcard,... formats flying around. Basically something like

BEGIN:VNOTE

VERSION:1.1

BODY: Here's a quick note

END:VNOTE

SyncML itself doesn't understand these things: it just passes them around between the phone and the external horde api "as they come".

If you have tricky stuff like recurring events, inline photos or other exotic data in the file (or got them "treated" by Outlook), horde or your phone might quickly get a hickup. So we need to test as many of these formats as we can get our hands on.

This is more of a general horde issue than about SyncML. You can test the behaviour by trying to import/export files with this ical/vcard data into kronolith/nag/turba using their respective import/export menu entries. That should use the same codebase.

Right now as I'm writing this, I've noticed Jan creating a framework/iCalendar/tests/charset1.phpt test file. Excellent!

Code Issues

Of course the SyncML code is far from being finished. Here are a few things that need work besides the stuff directly coming fomr I or II. (relevant modules are mentioned in brackets)

1. refactoring of current codebase (SyncML)

The current codebase is somewhat confusing and hard to deal with. I'll try to refactor it to make it easier for other people to jump in. Any help is appreciated. Especially suggestions about the best way to parse XML subtrees into PHP objects.

1. better error handling (SyncML): currently no real error handling is in place.
   1. Turba contacts support (Turba): I haven't looked at it yet. it might just work or not. Depends on whether VCARD in turba support is fine or not. You can just try to sync vcard data and see if you run into problems.
   2. better tasks import(Nag): currently nag imports only task name, description and priority. See the two @TODO lines in nag/lib/api.php. This should be extended to import the full info (alarm, due dates etc.). Export should be checked as well.
   3. Collision handling: (SyncML, Nag, Mnemo, Kronolith, Turba). My idea for collision handling is: while running SyncML: keep the existing entry and add the collision as a new one somehow marked as "collision" of entry guid #xxx. (stored in DataTree). Then kronolith etc. should have a page to display the colliding items side by side. There you can select which one to keep (and/or maybe edit manually). The "losing" entry gets deleted, which is then automatically synced as any normal delete during the next sync session. Should be rather straightforward to implement and requires minimal effords on the SyncML side.
   4. Get support for additional Sync Types (SyncML). Currently only SlowSync and TwoWaySync are supported. These are the only two that are mandatory in SyncML and the only two my phone supports. Don't know if we need the others. Some code is already there.
   5. Make use of devinf information (SyncML): In the inital SyncML packets both sides exchange a <devinf> block with information about device capabilites. This info should be stored and used.
   6. Horde api enhancements (HORDE API):

a) currently the horde api only supports listBy($action,$ts) to retrieve $actions since timestamp $ts. The SyncML protocol requires that the ending timestamp of the sync timeframe to be exchanged _before_ the actual syncing starts. So we need an additional parameter:

listby($action,$ts,$ts_end)

b) the SyncML works as follows:

1) the clients sends its changes to the server

2) the server sends its changes to the client.

so when in step 2), the horde api is called with a request like "give me all changes in horde since the last sync", you get the changes induced by the client in step 1) as well. You have to somehow "tag" them to avoid echoing (and thus duplicatinging) them back to the client. Simply storing the guids in the session is not sufficient: the changes are made _after_ the end timestamp (see a)) of the current sync so you'll dupe them in the next sync. My current implementation deals with this as follows: directly after a client induced change is done in horde, state.php's gettsforAction is called to find out the biggest=latest ts for the given guid. If the horde api would provide me with this info directly (maybe just store it in a static var and provide a getLastTS() method in Driver), that would be great and eliminate costly and redundant DataTree calls. Similar in listBy where you get a list of guids but still need to retrieve the latest History timestamps for these guids manually using the DataTree so you can check out if the changes are a result of a client request or not. If listBy would return an assoc array $guid->ts that would help a lot (and solve a) as well).

Testing Suite

Nobody has access to all the SyncML clients all the time. So there's a always the risk that you supply a fancy enhancement patch that works fine for you but breaks down everything for everyone else. So it would be great to have a "testing suite" (in JUNIT terms) which simulates sync runs for all supported phones. (Maybe done by sending specifically crafted XML files based on the logging data of various phones and checking the results). Before a commit of a new patch a test run is done with this suite to ensure it doesn't break things up. Just an idea (dream) for now, suggestions how this could be done are welcome.