Earlier this week we had problems with MSDTC between a developer machine running Windows XP SP 2 and Windows Server 2003, and after testing also from developer machines running Vista. As I always do when having problem with MSDTC, we downloaded DTCPing and adjusted a suggested setting, but we still did not manage to get DTCPing through to a positive result. We kept ketting "Access denied", and after trying almost everything (including rebooting) some desperate googling began. One of the developers somehow managed to find something mentioning "RestrictRemoteClients". Continuing and googling on this brought him to "The Enterprise Single Sign-On Service and associated BizTalk Server 2004 services fail after you install Windows XP Service Pack 2 (SP2)" knowledge base article, and if it was me I would have stopped reading immeadetly. Fortunately it wasn't me :). Down on the bottom of the article it explains how to set the RestrictRemoteClients registry setting and after doing that DTCPing worked like a charm and gave us positive result.
Ironically this setting was not needed for the distributed transaction to work, but only for DTCPing, so we spent a couple of hours trying to solve an problem that really did not exist (just for the debugging tool used).