Correctly Adding Contacts and Groups to Lync Using UCMA 3.0

Recently I have been revisiting one of our products, BuddyList Migrator, which migrates IBM Sametime buddylists into Microsoft Lync contact lists.  While I was doing this, I came across an unexpected, and, so far as I can tell, undocumented nuance involved in using the UCMA ContactsGroupServices class to add contacts to a Lync user’s contact list.  In the interests of serving the UCMA development community, I’d like to share what I’ve learned.
To follow along with this, you will need to have a Lync 2010 server to play with, and a development environment with the UCMA 3.0 SDK installed.  You will also, ideally, need to setup a Lync TrustedApplication,so that your application can login in behalf of users without needing to supply credentials, although for experimentation's sake, you could skip that step.

Application Background

In our application, we need to generate contacts for a Lync user based on their existing SameTime buddylist.  SameTime buddylists are stored in a Lotus Notes database, VPUserInfo.nsf.  The buddylists in this database refer to each buddy in a user’s list by the buddy’s SameTime Notes ID, which, in most cases, is completely unintelligible to Lync.  However, we can cross-reference this Notes ID with the user’s email address, by looking the user up in the Notes address book database, Names.nsf.  Once we have the buddy’s email address, we can use this to find the buddy’s Lync address in Active Directory to add them as a contact in Lync.
This process of exporting SameTime buddylists is outside the scope of this discussion.  We use another internal Java application to perform this process for us, which outputs an XML representation of each user to be converted’s buddylist.  The actual format is slightly more complex, but for the purposes of illustration, we can view this file as similar to this:

<xmlBuddyList>
    <ownerEmailAddress>john.doe@foo.com</ownerEmailAddress>
    <buddylistGroups>
        <group>
            <groupName>Group 1</groupName>
            <groupUsers>
                <userEmailAddress>Bill.Smith@foo.com</userEmailAddress>
                <userEmailAddress>Mary.White@foo.com</userEmailAddress>
                <userEmailAddress>Jim.Ford@foo.com</userEmailAddress>
            </groupUsers>            
        </group>
        group>
            <groupName>Group 2</groupName>
            <groupUsers>
                <userEmailAddress>John.Green@foo.com</userEmailAddress>
                <userEmailAddress>Hannah.Hayes@foo.com</userEmailAddress>
                <userEmailAddress>Betty.Adams@foo.com</userEmailAddress>
            </groupUsers>            
        </group>
    </buddyListGroups>
</xmlBuddyList>

Once we have the user’s buddylist in this logical XML representation, we can start the process of pushing the buddylist into the user’s Lync contact list.

Lync Initialization

To do much of anything useful with the Lync UCMA library, one needs to first connect to the Lync server, and obtain a UserEndpoint object for the user.  After working on a number of UCMA projects, I’ve developed this simple utility class to abstract out the details:

class UCMAWrapper {
    public bool IsInitialized { get; private set; }
    private CollaborationPlatform _collaborationPlatform;
    private readonly Dictionary<string, UserEndpoint> _userEndpoints = 
        new Dictionary<string, UserEndpoint>();

    public void Start(string appName, int port, string gruu) {

        string localhost = System.Net.Dns.GetHostEntry("localhost").HostName;
        // for a TrustedApplication
        var settings = new ServerPlatformSettings(
            appName, 
            localhost, 
            port, 
            gruu, 
            CertificateHelper.GetLocalCertificate()
            );
        _collaborationPlatform = new CollaborationPlatform(settings);
        /* for a non-trusted application
            *
        var settings = new ClientPlatformSettings("myapp", SipTransportType.Tls);
        _collaborationPlatform = new CollaborationPlatform(settings);
            * 
            */
        try {
            _collaborationPlatform.EndStartup(
                _collaborationPlatform.BeginStartup(null, null)
            );
        } catch (RealTimeException ex) {
            // handle error
        }
        IsInitialized = true;
    }
    // More...

CertificateHelper is a utility class I created to encapsulate grabbing the correct certificate from the local machine store for the TrustedApplication.

In this class, _collaborationPlatform is our connection to the Lync server.  We will need this connection object in order to create UserEndpoint objects, which will allow us to login to Lync on behalf of users and perform different actions.  We also create a Dictionary to contain the UserEndpoints that we have created, indexed by the user’s Lync SIP address, as creating and connecting a UserEndpoint is a relatively expensive operation, and in situations where we will be reusing the same endpoint repeatedly, it is worthwhile to maintain a previously created endpoint and reuse it, rather than dispose and recreate it.

Once we have created our CollaborationPlatform and successfully connected to the Lync server, we can create UserEnpoints that allow us to perform actions on behalf of the user and access their contact lists.  We’ll use the following method to create and connect a UserEndpoint, reusing our cached instance if we already have one:

public UserEndpoint GetUserEndPoint(string uri) {
    string proxyServerFqdn = "lync server fqdn or ip address";
    const int tlsPort = 5061;

    if (uri == null) {
        return null;
    }
    if (_userEndpoints.ContainsKey(uri)) {
        return _userEndpoints[uri];
    }

    var settings = new UserEndpointSettings(uri, proxyServerFqdn, tlsPort);
    _userEndpoints[uri] = new UserEndpoint(_collaborationPlatform, settings);

    try {
        _userEndpoints[uri].EndEstablish(_userEndpoints[uri].BeginEstablish(null, null));

    } catch (Exception ex) {
        _userEndpoints[uri] = null;
    }
    return _userEndpoints[uri];
}

NOTE: You’re going to see me using the End(Begin()) method pairs in a synchronous fashion here.  Microsoft doesn’t really recommend doing it this way, but I have found that the recommended asynchronous way results in a proliferation of callback methods and wait handles that can introduce irritating bugs if you mess up your state management and makes the code much more difficult to follow.  I’m also going to strip out most of the error-handling and logging code that you would want in a production application. 

Lastly, we need some methods to clean up our UserEndpoints and the CollaborationPlatform we created when we are finished with them.

public void Shutdown() {
    while (_userEndpoints.Keys.Any()) {
        CloseUserEndpoint(_userEndpoints.Keys.First());
    }
    try {
        _collaborationPlatform.EndShutdown(
            _collaborationPlatform.BeginShutdown(null, null));
    } catch (RealTimeException ex) {
    }
}

private void CloseUserEndpoint(string uri) {
    if (!_userEndpoints.ContainsKey(uri) || _userEndpoints[uri] == null) 
        return;
    _userEndpoints[uri].EndTerminate(
        _userEndpoints[uri].BeginTerminate(null, null));
    _userEndpoints.Remove(uri);
}

Managing Contacts

Once we have created a UserEndpoint whose contact list we wish to add to, we will add contacts and groups using the endpoints ContactGroupServices object.  Once again, there is a fair amount of plumbing in doing this correctly, so we will create another wrapper class to handle this for us:

public class ContactManager {
    private readonly ContactGroupServices _contactGroupServices;
    private ContactList _contactList;
    private UserEndpoint _userEndpoint;

    public ContactManager(UserEndpoint endpoint) {
        if (endpoint == null) {
            throw new ArgumentNullException("endpoint");
        }
        _contactList = new ContactList();

        _userEndpoint = endpoint;
        _contactGroupServices = _userEndpoint.ContactGroupServices;
        //Log.Info(("Subscribing to contact updates"));
        _contactGroupServices.NotificationReceived += OnNotificationReceived;
        try {
            _contactGroupServices.EndSubscribe(
                _contactGroupServices.BeginSubscribe(null, null));
        } catch (InvalidOperationException ex) {
                
        }
    }
    // More...
}

This class manages an in-memory version of the user’s contact list in the _contactList member.  We create a ContactManager by passing in the UserEndpoint that we have created.  Next, we store the ContactGroupServices member of the endpoint in another member variable.  We’ll bind an event handler to the ContactGroupService’s NotificationReceived event, which fires whenever we subscribe to the user’s existing contact list, using the End/BeginSubscribe methods, and whenever we modify the contact list using the ContactGroupServices methods.

Our event handler for the NotificationReceived event extracts any new contacts and groups from the notification and adds them to our in-memory contact list.  Ignore the _groupWaitHandles dictionary for the moment; its reason for existence will become clearer once we cover adding a new group to a contact list.

private void OnNotificationReceived(object sender, ContactGroupNotificationEventArgs e) {
    foreach (var contact in e.Contacts.Select(item => item.Item)) {
        if (_contactList.Contacts.All(c => c.Name != contact.Name)) {
            _contactList.Contacts.Add(
                new ContactInfo {
                    Data = contact.ContactData,
                    Name = contact.Name,
                    GroupIds = contact.GroupIds,
                    Extension = contact.ContactExtension,
                    Uri = contact.Uri
                });
        }

    }
    foreach (var group in e.Groups.Select(item => item.Item)) {
        if ( _contactList.Groups.All(g => g.Name != @group.Name))
        _contactList.Groups.Add(
            new GroupInfo {
                Data = group.GroupData,
                Name = group.Name,
                ID = group.GroupId
            });
        if (_groupWaitHandles.ContainsKey(@group.Name)) {
            _groupWaitHandles[group.Name].Set();
        }
    }
}
private readonly Dictionary<string, AutoResetEvent> _groupWaitHandles =
    new Dictionary<string, AutoResetEvent>();

NOTE:  Properly, we should be checking the value of the Operation member of each contact and group received as part of the notification.  For our particular scenario, (migrating a SameTime userbase to Lync) we are operating in an environment where users are not actively logged into the Lync server while our tool runs.  Some care should probably be taken to keep the memory list in sync with the server if you are running in an environment where users or other tools may also modify contact lists.

As you can see, our in-memory _contactList member is essentially just a persistent version of the data returned to us by the NotificationReceived event.  We need to maintain this data because the Lync server does not always send us out a full listing of the user’s contact list; only when we subscribe to the contact list using the ContactGroupServices End/BeginSubscribe methods do we receive the full state of the contact list.  Otherwise, we only receive the modified items.  The structures for the _contactList are:

public class ContactList {
    public List<GroupInfo> Groups { get; set; }
    public List<ContactInfo> Contacts { get; set; }
    public ContactList() {
        Groups = new List<GroupInfo>();
        Contacts = new List<ContactInfo>();
    }
}
public class ContactInfo {
    public string Data { get; set; }
    public string Extension { get; set; }
    public string Name { get; set; }
    public string Uri { get; set; }
    public int[] GroupIds { get; set; }

    public ContactInfo() {
        Data = Extension = Name = Uri = "";
        GroupIds = null;
        Extension = "<contactSettings></contactSettings>";
    }
    public class GroupInfo {
        public int ID { get; set; }
        public string Name { get; set; }
        public string Data { get; set; }
    }
}

Adding Groups

Once we have created our ContactManager and received the current state of the user’s contact list via the NotificationReceived event, our next step will be to add the user’s SameTime private groups as groups to their Lync contact list.  Basically, the process is as follows:

1. Determine if the group that we are adding already exists in the user’s contact list


2. If it does not, add a new group.


3. Wait for the notification that the group has been created on the Lync server, or that the operation has timed out.

public void CreatePrivateGroup(string grp) {
    try {
        var groupName = grp;
        var groupID = GetGroupID(groupName);
        if (groupID == -1) {
            var gp = new GroupInfo { Name = groupName, ID = groupID, Data = "" };
            AddGroup(gp);
        }
    } catch (Exception ex) {
    }
}
public void AddGroup(GroupInfo group) {
    if (_contactList.Groups.Find(g => g.Name == group.Name) != null) {
        return;
    }
    try {
        _contactGroupServices.EndAddGroup(
            _contactGroupServices.BeginAddGroup(
            group.Name, 
            group.Data, 
            null, 
            null
        ));
        _groupWaitHandles.Add(group.Name, new AutoResetEvent(false));
        //HACK - I can never actually receive the notification that a group was added
        _contactGroupServices.EndUnsubscribe(
            _contactGroupServices.BeginUnsubscribe(null, null));
        _contactGroupServices.EndSubscribe(
            _contactGroupServices.BeginSubscribe(null, null));
        //ENDHACK
        var success = _groupWaitHandles[group.Name].WaitOne(20000);
        if (!success) {
        }
        _groupWaitHandles.Remove(group.Name);
    } catch (Exception ex1) {
    }
}
public int GetGroupID(string groupName) {
    foreach (var g in _contactList.Groups.Where(
        g => g.Name == groupName)) {
        return g.ID;
    }
    return -1;
}

This is where the previously mentioned _groupWaitHandles dictionary comes into play.  If we only wanted to add groups, we could simply call End/BeginAddGroup, and return, then move onto the next group.  However, we are going to be adding contacts into these newly created groups.  Lync associates the contacts with groups by using the integer ID field, which is auto-generated on the server, so, while we could take a best guess at what it might be, we don’t really know how to assign contacts to the group until we receive the notification that the group has been created, with the accompanying new ID.  This is why we need the dictionary of wait handles.


You’ll notice the End/BeginUnsubscribe-End/BeginSubscribe block outlined with HACK comments.  For whatever reason, I have not been able to get the Lync server to actually fire a notification when a group is created as it should.  Unsubscribing and then resubscribing forces the full contact list push, which ensures that we obtain the new group’s ID before moving on to try to add contacts to it.

Adding Contacts

Compared to adding groups, adding contacts is simple.  The only tricky thing here is to ensure that we add the group ID for each group the contact is a part of to the ContactAddOptions structure that we pass into the End/BeginAddContact calls.

public void AddContact(ContactInfo contact) {
            
    if (_contactList.Contacts.Find(
            c => c.Uri == contact.Uri) != null) {
        return;
    }
    try {
        var contactAddOptions = new ContactAddOptions {
            ContactData = contact.Data,
            ContactExtension = contact.Extension,
            ContactName = contact.Name,
        };
        contactAddOptions.GroupIds.AddRange(contact.GroupIds);
        _contactGroupServices.EndAddContact(
            _contactGroupServices.BeginAddContact(
                contact.Uri, contactAddOptions, null, null
            )
        );

    } catch (InvalidOperationException ex) {
    }
}

Putting it Together

With these utility classes created, the process of moving a user to Lync from SameTime becomes relatively straightforward.

1. Read in the xml buddylist document.



2. Resolve the user’s email to their Lync SIP address using Active Directory.



3. Obtain a UserEndpoint for the user.



4. Create a ContactManager object for the user.



5. Create the groups for the user.



6. Add the contacts to each group, resolving the contact email addresses to Lync SIP addresses using Active Directory.

After the running through this process, you will see the user’s SameTime contacts and groups moved over to Lync (click for larger images):

SameTime buddylist xml	Lync Contact List

We’ve got all the user’s groups and contacts moved over.  Great, we’re done!


Not so fast…


Let’s switch over from the Groups tab in the Lync client to the Status and Relationship tabs:





Nothing… Hmm…


When one adds a contact using the Lync client, contacts will be sorted into these views automatically.  Yet none of the contacts we’ve created using our UCMA application are to be found.  Why is that?

The Mysterious Tilde Group

After digging through the underlying SQL databases for several hours, and pursuing a number of other blind alleys, I found a clue in the bowels of the Lync RTC database.  Poring through the ContactGroup table, I noticed that a large number of the rows included a value of 0x7E for the DisplayName column.  After checking out the OwnerId keys for a number of these rows, I realized that there appeared to be a row like this for each user that has actually created contacts using the Lync client.  Taking one of the users, I cross-referenced with the ContactGroupAssoc table, and discovered that this 0x7E group appeared to contain all of the contacts in all of the user’s other groups.


Hmm, that is interesting, I thought to myself.


Since 0x7E, and all the other values in that column, looked like hex, I grabbed one of the entries for my personal Lync account, and discovered, after running it through a hex-to-string converter, that it matched up with the name of one of my contact list groups.  So what is 0x7E?  Turns out, it converts to “~”.  Looking at my Lync client, I didn’t see any groups displayed named “~”.  Curious.





Next, I took one of our test accounts that I had run our conversion tool on, and tested the effects of various actions in the Lync client.  I found that if I moved a contact from one of the SameTime imported groups into one of the other groups (Other Contacts or Pinned Contacts), that contact would then appear in the the Status/Relationship views, even after I moved the contact back to its previous group.  The Lync client was clearly doing something under the covers when I moved a contact into another group, so I pulled up the database again and found the particular contact in the ContactGroupAssoc table.  In addition to the group that the contact was actually in, I now found another row, linking it to another group that turned out to be the ~ group for this user.  Bingo…


I modified the code for our application, so that it added all the imported SameTime contacts to this “~” group in addition to the imported SameTime groups, reran the import for a test user, and voila, the Status and Relationship tabs lit up.

Conclusions

Now, be forewarned, that all of what follows is based solely on what I have been able to deduce, by examining this situation, the Lync databases, and the behavior of the Lync client.  I wasn’t able to find any mention of this tilde group in the UCMA API docs, and the one example I found of a similar situation involved a completely different method of adding contacts than the one presented here.

• The Lync client creates and maintains a special contact group, named “~”.



• This group is not displayed in the Groups pane of the client UI, but is used to populate the Status and Relationship views, filtered by presence/privacy list information maintained eleswhere.



• The Lync client silently updates the ~ group whenever you add, remove, or relocate contacts using the Lync client.



• Adding a contact to a contact list with UCMA does not modify the ~ list, unless you explicitly add the new contact to the ~ list, as well as the group that it actually belongs to.



• The ~ group is only created once the user has actually logged into the Lync server using the Lync client, and possibly only after contacts have been added using the Lync client.

So, if you’re working with contact lists in UCMA, make sure you add new contacts to the magical ~ group!