May 082010
 

Despite its pitiful adoption in the developer community, I am implementing Transactional NTFS (TxF) transactions using the Microsoft.KtmIntegration.TransactedFile class. This allows me to reap the benefits of TransactionScope and distributed transactions for file operations (e.g. creates, updates, deletes). This is the only missing piece for typical transactional business applications. With the “KTM” and “KtmRm for Distributed Transactions” services, available only on Vista, Windows 7, and Windows Server 2008, file operations will roll back if the TransactionScope is not completed.

There’s just one problem. Transactional NTFS does not work with file shares. I can?t remember the last time I put a “C:FileStore” reference in a config file. A friendly share like \serverFileStore is always preferred, especially since DFS came about. Attempting to use a share results in the following error message:

The remote server or share does not support transacted file operations

Don’t read this as “your remote server” or “your remote share”, but rather “all remote servers and shares”. As mentioned in this MSDN article, TxF is not supported by the CIFS/SMB protocols. The error was probably written with the expectation that one day some remote servers and shares would support TxF. I emailed Microsoft about it and received a response fairly quickly. The response was simply:

“We understand the need and have plans to eventually support TxF over SMB2, but we?re not there yet and are not ready to announce if or when this will be supported. When it is the documentation will be updated.”

I’m not getting my hopes up, but Windows Server 2011 looks to be our only hope before .NET changes beyond recognition and TxF is a distant memory. Until then, I wrapped up all of my TxF code in a WCF service and install that service on the server with the FileStore folder.

MSDN article – When to Use Transactional NTFS

      http://msdn.microsoft.com/en-us/library/aa365738(v=VS.85).aspx

TxF Sandbox – Sample Projects (including Microsoft.KtmIntegration.TransactedFile)

      TxFSandbox.zip

Sep 032009
 

The 2009 Jacksonville Code Camp was a great success. Many thanks to Bayer, Brandy, and everyone else that made it happen. The bar has been set really high for future Jacksonville code camps, and for the rest of Florida too.

My session on Transactional WCF Services went well. Many great questions and compliments after the session. If you attended and have any unanswered questions, please email me.

You can download the session files below. It contains staged versions of all of the transaction modes we discussed. It also contains a tracing solution and tracing result files to view the client and host tracing files in Client/Service mode. Also see my previous post on using the Service Trace Viewer. It also contains a few demo projects that we didn’t get to in the one-hour session.

Files/Solutions included in Session Archive:

  • PowerPoint slides
  • Transaction Promotion Code Snippet
  • Testing database backup
  • Testing SQL script (query and cleanup between tests)
  • IDesign ServiceModelEx Project (used by all included Solutions)
  • Code Demo Solutions

Code Demos include:

1. TransactionScope – Shows how single/multiple resource managers affect which Transaction Manager is chosen to handle the scoped transaction. Also gives first look at transaction promotion detection.
2a. Mode None – WCF transaction mode with which no transactions are created or flowed from the calling client.
2b. Mode Service – WCF transaction mode with which no transactions are flowed from the calling client, but a transaction is created for your service operation.
2c. Mode Client – WCF transaction mode with which a transaction is required to be flowed, and the service will only use the client transaction.
2d. Mode Client/Service – WCF transaction mode with which a client transaction will be flowed and used by the service, if available. If no client transaction is flowed, a transaction will be provided automatically for the service operation.
3. Explicit Voting – Shows how explicit voting with a session-mode service is performed using OperationContext.Current.SetTransactionComplete().
4a. Testing Various Resource Managers – Shows how a client can use a single TransactionScope to call several services (some transactional, some non-transactional), a database stored procedure, and an IDesign volatile resource manager Transactional<int>.
4b. Testing Services – Provides a host project for a transactional service and a non-transactional service used in 4a.
5a. Tracing – Same as 2d. modified with the additional app.config settings in the client and host projects to allow for service tracing to .svclog files.
5b. Tracing Results – Stored results from executing 5a. in case you don’t want to load the database and actually run the projects. The .stvproj file can be opened directly in the Service Trace Viewer. On the “Activity” table, click on the activity “Process action ‘http://services/gotjeep.net/GpsTrackServiceContract/SubmitTrack'” then click on the “Graph” tab. You will see that the client and host activities where the arrow moves from client to host (send and receive message, respectively) show the OleTxTransaction in “Headers.” The next activity in the host reads “The transaction ‘5bd25b08-848c-409d-9163-6303b9138382:1’ was flowed to operation ‘SubmitTrack’.”

 

Download the session files:
TransactionalWCF.zip (854 KB)

Jan 292009
 

“Do not anticipate trouble, or worry about what may never happen. Keep in the sunlight.” 

– Benjamin Franklin

 

It’s obvious that Mr. Franklin didn’t have to deal with bad data, bad code, or even the occasional hiccup. In the real world, there is “trouble” and we need to not only anticipate that trouble, but also worry about what may never happen. Simply “keeping in the sunlight” won’t maintain data or application state integrity. In the real world we need transactions!

Since .NET 2.0, we’ve had the privilege of using System.Transactions.TransactionScope to manage our transactions with very few headaches. One of the headaches that almost everyone experiences is MSDTC. One of the oldest and most elusive topics on the web. There are tons of blog and forum posts directing our fellow developers to check their firewall settings for every MSDTC problem. The latest MSDTC hiccup I have seen comes in the beautifully packaged error message:

The flowed transaction could not be unmarshaled. The following exception occurred: Communication with the underlying transaction manager has failed.

The what could not be what? You can read some MSDN documentation on the topic which will probably cause you more pain. If you are seeing this error message, there’s only a few things that may be wrong.

  1. MSDTC Settings
    • Check the MSDTC settings on the machine that is initiating the transaction. If “Allow Outbound” is not checked, then it won’t allow the transaction in progress to be flowed to the next machine in the transaction chain. Check the box and restart MSDTC… it should work.
  2. Un-Trusted Domains
    • I have seen this error when you are trying to flow transactions between machines that are in un-trusted domains. Machines in different domains that do not trust each other block the antiquated, yet necessary, WINS resolution between the two machines. MSDTC relies on WINS resolution. I have fixed this problem before by adding host file entries on both machines pointing to the other machine. I can’t guarantee that this will work in all cases. Both of those machines are no longer under my control.
  3. Imaged Servers
    • The most recent, and most blogged about problem is surprisingly caused by two machines created from the same image. Cloning or imaging servers is quite common. Building a server from scratch is not a fun activity, so we build one, create an image, and put that image on every server we want to build after that. Once again, MSDTC is standing in our way because of the way it detects the sending and receiving application’s unique identifier. Each machine has a GUID in the registry that identifies it uniquely as a participant in an MSDTC transaction. Imaged machines have the same GUID. I’ll talk about the detection and resolution of this for the remainder of this post.

Running DtcPing.exe between the two machines will actually tell you that the machines are using the same GUID. Output window text from DtcPing shown below.

DTCping log file: C:DTC PingTHRESHER4160.log
Firewall Port Settings:
Port:5150-5250
RPC server is ready
Please Start Partner DTCping before pinging
WARNING:the CID values for both test machines are the same
Please send following LOG to Microsoft for analysis:
Partner LOG: SCORPION6128.log
My LOG: THRESHER4160.log

Tucked away in the last step of a Microsoft Knowledge Base article titled “How to troubleshoot MS DTC firewall issues” is a reference to this problem. Use regedit.exe to look at the registry on both machines. Locate the HKEY_CLASSES_ROOTCID key in the registry.

Find your MSDTC CID in the registry

Find the CID key that has a description value of “MSDTC”. If they are the same, the transaction cannot flow.

WARNING: Back up your registry before making any changes!

Solution 1 – Replace the offending CID keys/values on one of the machines: In this case, you will need to find all keys/values with GUID 28b81f1c-2afb-4ee2-ad85-5bc62dad1647 in your registry and replace it with a new GUID (using GuidGen). There is likely to be 3 places this GUID appears. It is also important to note that the offending GUID appears in the DtcPing log file generated during the ping test.

Solution 2 – Use msdtc command line tool to re-install MSDTC: The commands are simply:

msdtc -uninstall
msdtc -install

After making the registry changes or running the msdtc command line tool, you must restart your server for the changes to take effect.

Dec 052008
 

Unless you are working on a extremely simple or read-only application, transactions are a must. Using the System.Transactions namespace is the easiest and most efficient way to maintain system consistency when dealing with multiple calls or multiple resources. Although System.Transactions arrived in .NET in the 2005 product, it is still a relatively unknown part of the framework. System.Transactions was designed by the Indigo team in preparation for WCF. It is not compatible with Win98 or WinME, but most people are incompatible with Win98 and WinME so it works out just fine.

Before System.Transactions, we only had access to System.Data.SqlClient.SqlTransaction or a true SQL transaction using BEGIN/ROLLBACK/COMMIT TRAN. Using SQL transactions, you are stuck with only being able to update DB records as part of your transaction. If you wanted to change a cached value in your app in addition to the SQL updates in the same transaction then you would be out of luck. This also required a lot of transaction code in your stored procedures, writing stored procedures that can be called independently or part of transaction made for very messy stored procedures and often led to multiple stored procedures that served the same purpose.

Using the SqlTransaction class was also messy. The most important restriction is that you need to have all database calls inside the same SqlConnection. This does not work well for a well-designed N-tier application. The other problem is that you need to handle your own non-DB transaction logic inside the same SqlTransaction and conditionally commit/rollback as necessary. This all tends to lead to several try-catch statements within one SqlTransaction. Handling the plumbing to manually add each SqlCommand to the transaction gets old quickly too.

Using SqlTransaction

   27 string connectionString = ConfigurationManager.ConnectionStrings[“Testing”].ConnectionString;

   28 using (SqlConnection con = new SqlConnection(connectionString))

   29 {

   30     SqlTransaction tran = null;

   31     try

   32     {

   33         con.Open();

   34         tran = con.BeginTransaction();

   35         using (SqlCommand cmd = new SqlCommand(“usp_ErrorLog_Insert”, con))

   36         {

   37             cmd.Transaction = tran;

   38             cmd.CommandType = System.Data.CommandType.StoredProcedure;

   39             cmd.Parameters.AddWithValue(“Message”, “Testing 1”);

   40             cmd.Parameters.AddWithValue(“UserID”, 5150);

   41             cmd.ExecuteNonQuery();

   42         }

   43 

   44         using (SqlCommand cmd = new SqlCommand(“usp_ErrorLog_Insert”, con))

   45         {

   46             cmd.Transaction = tran;

   47             cmd.CommandType = System.Data.CommandType.StoredProcedure;

   48             cmd.Parameters.AddWithValue(“Message”, “Testing 2”);

   49             cmd.Parameters.AddWithValue(“UserID”, 5150);

   50             cmd.ExecuteNonQuery();

   51         }

   52 

   53         tran.Commit();

   54     }

   55     catch

   56     {

   57         if (tran != null) tran.Rollback();

   58     }

   59     finally

   60     {

   61         con.Close();

   62     }

   63 }

System.Transactions liberated us from the mundane SqlClient code and repetitive try-catches. simply wrapping your old ADO.NET with a using (TransactionScope) { } is all you need to do. You will typically add a transactionScope.Complete() statement as the last line in the TransactionScope using block is really all you need. Any exception thrown before this point will break out of scope, implicitly aborting the transation. Much better.

System.Transactions uses the LTM (Lightweight Transaction Manager) when dealing with single resources or machines. The transaction is automatically promoted to MSDTC (Microsoft Distributed Transaction Coordinator) when another resource is enlisted in a transaction. A lot of people struggle with MSDTC because it is difficult to setup, requires special firewall considerations, and doesn’t really work well for smart client applications since you have to install MSDTC on every client machine.

I’ll show one transaction performed three different ways and show what happens with the LTM and MSDTC for each of them. I will also demonstrate an excellent reason to migrate to Enterprise Library.

1) Executing two ADO.NET SqlCommands in different SqlConnections

  122 using (TransactionScope scope = new TransactionScope())

  123 {

  124     string connectionString = ConfigurationManager.ConnectionStrings[“Testing”].ConnectionString;

  125     using (SqlConnection con = new SqlConnection(connectionString))

  126     using (SqlCommand cmd = new SqlCommand(“usp_ErrorLog_Insert”, con))

  127     {

  128         cmd.CommandType = System.Data.CommandType.StoredProcedure;

  129         cmd.Parameters.AddWithValue(“Message”, “Testing 1”);

  130         cmd.Parameters.AddWithValue(“UserID”, 5150);

  131         try

  132         {

  133             con.Open();

  134             cmd.ExecuteNonQuery();

  135         }

  136         finally

  137         {

  138             con.Close();

  139         }

  140     }

  141 

  142     Console.WriteLine(“Local Transaction ID: {0}”,

  143         Transaction.Current.TransactionInformation.LocalIdentifier);

  144     Console.WriteLine(“Distributed Transaction ID: {0}”,

  145         Transaction.Current.TransactionInformation.DistributedIdentifier.ToString());

  146 

  147     using (SqlConnection con = new SqlConnection(connectionString))

  148     using (SqlCommand cmd = new SqlCommand(“usp_ErrorLog_Insert”, con))

  149     {

  150         cmd.CommandType = System.Data.CommandType.StoredProcedure;

  151         cmd.Parameters.AddWithValue(“Message”, “Testing 2”);

  152         cmd.Parameters.AddWithValue(“UserID”, 5150);

  153         try

  154         {

  155             con.Open();

  156             cmd.ExecuteNonQuery();

  157         }

  158         finally

  159         {

  160             con.Close();

  161         }

  162     }

  163 

  164     Console.WriteLine(“Local Transaction ID: {0}”,

  165         Transaction.Current.TransactionInformation.LocalIdentifier);

  166     Console.WriteLine(“Distributed Transaction ID: {0}”,

  167         Transaction.Current.TransactionInformation.DistributedIdentifier.ToString());

  168 

  169     scope.Complete();

  170 }

This writes the following to the command line:

Local Transaction ID: e90f47f4-df80-496b-a9c0-0c45b2f452c4:2
Distributed Transaction ID: 00000000-0000-0000-0000-000000000000
Local Transaction ID: e90f47f4-df80-496b-a9c0-0c45b2f452c4:2
Distributed Transaction ID: 1fad8108-ddae-496a-a7da-ce92df175e40

You’ll notice that the first command creates a transaction using LTM as indicated by the Local Transaction ID. After the second command is executed, the transaction is promoted to DTC as indicated by the Distributed Transaction ID. This is expected because there are two distinct SqlConnections. Even though the connection string is the same, TransactionScope treats these ADO.NET objects as unique resources.

This has additional implications when connection pooling comes into play. After I close the first connection, it is returned to the pool and is available for use. If this connection is requested for use, it will no longer be available to commit or abort this transaction, and you will see the dreaded MSDTC error “Communication with the underlying transaction manager has failed.”

2) Executing two ADO.NET SqlCommands in the same SqlConnection

   69 string connectionString = ConfigurationManager.ConnectionStrings[“Testing”].ConnectionString;

   70 using (TransactionScope scope = new TransactionScope())

   71 using (SqlConnection con = new SqlConnection(connectionString))

   72 {

   73     using (SqlCommand cmd = new SqlCommand(“usp_ErrorLog_Insert”, con))

   74     {

   75         cmd.CommandType = System.Data.CommandType.StoredProcedure;

   76         cmd.Parameters.AddWithValue(“Message”, “Testing 1”);

   77         cmd.Parameters.AddWithValue(“UserID”, 5150);

   78         try

   79         {

   80             con.Open();

   81             cmd.ExecuteNonQuery();

   82         }

   83         finally

   84         {

   85             con.Close();

   86         }

   87     }

   88 

   89     Console.WriteLine(“Local Transaction ID: {0}”,

   90         Transaction.Current.TransactionInformation.LocalIdentifier);

   91     Console.WriteLine(“Distributed Transaction ID: {0}”,

   92         Transaction.Current.TransactionInformation.DistributedIdentifier.ToString());

   93 

   94     using (SqlCommand cmd = new SqlCommand(“usp_ErrorLog_Insert”, con))

   95     {

   96         cmd.CommandType = System.Data.CommandType.StoredProcedure;

   97         cmd.Parameters.AddWithValue(“Message”, “Testing 2”);

   98         cmd.Parameters.AddWithValue(“UserID”, 5150);

   99         try

  100         {

  101             con.Open();

  102             cmd.ExecuteNonQuery();

  103         }

  104         finally

  105         {

  106             con.Close();

  107         }

  108     }

  109 

  110     Console.WriteLine(“Local Transaction ID: {0}”,

  111         Transaction.Current.TransactionInformation.LocalIdentifier);

  112     Console.WriteLine(“Distributed Transaction ID: {0}”,

  113         Transaction.Current.TransactionInformation.DistributedIdentifier.ToString());

  114 

  115     scope.Complete();

  116 }

This writes the following to the command line:

Local Transaction ID: e90f47f4-df80-496b-a9c0-0c45b2f452c4:1
Distributed Transaction ID: 00000000-0000-0000-0000-000000000000
Local Transaction ID: e90f47f4-df80-496b-a9c0-0c45b2f452c4:1
Distributed Transaction ID: becac9c9-e15f-4370-9f73-7f369665bed7

This is not expected because both commands are part of the same connection. Of course I am closing the connection to simulate an N-tier app where the data access layer is maintaining it’s own SQL access, opening and closing its connection as it should. If I did not close the connection, you would not see a Distributed Transaction ID after the second command.

3) Executing two Enterprise Library commands

  176 using (TransactionScope scope = new TransactionScope())

  177 {

  178     Database db = DatabaseFactory.CreateDatabase(“Testing”);

  179     DbCommand cmd = db.GetStoredProcCommand(“usp_ErrorLog_Insert”);

  180     db.AddInParameter(cmd, “Message”, System.Data.DbType.String, “Testing 1”);

  181     db.AddInParameter(cmd, “UserID”, System.Data.DbType.Int32, 5150);

  182     db.ExecuteNonQuery(cmd);

  183 

  184     Console.WriteLine(“Local Transaction ID: {0}”,

  185         Transaction.Current.TransactionInformation.LocalIdentifier);

  186     Console.WriteLine(“Distributed Transaction ID: {0}”,

  187         Transaction.Current.TransactionInformation.DistributedIdentifier.ToString());

  188 

  189     Database db1 = DatabaseFactory.CreateDatabase(“Testing1”);

  190     DbCommand cmd1 = db.GetStoredProcCommand(“usp_ErrorLog_Insert”);

  191     db1.AddInParameter(cmd1, “Message”, System.Data.DbType.String, “Testing 2”);

  192     db1.AddInParameter(cmd1, “UserID”, System.Data.DbType.Int32, 5150);

  193     db1.ExecuteNonQuery(cmd1);

  194 

  195     Console.WriteLine(“Local Transaction ID: {0}”,

  196         Transaction.Current.TransactionInformation.LocalIdentifier);

  197     Console.WriteLine(“Distributed Transaction ID: {0}”,

  198         Transaction.Current.TransactionInformation.DistributedIdentifier.ToString());

  199 

  200     scope.Complete();

  201 }

This writes the following to the command line:

Local Transaction ID: 6737b756-2d5b-4eff-902d-15f9ccd5c26f:3
Distributed Transaction ID: 00000000-0000-0000-0000-000000000000
Local Transaction ID: 6737b756-2d5b-4eff-902d-15f9ccd5c26f:3
Distributed Transaction ID: 00000000-0000-0000-0000-000000000000

Whoa! How cool is that? No DTC promotion. Enterprise Library is intelligently deciding to keep the connection open when it is part of the transaction. This will save a lot of wasted time as the promotion to DTC adds a noticeable delay. If I wasn’t using Enterprise Library already, I’d switch now.

Useful links:

Oct 302008
 

Don’t be so quick to blame the service or MSDTC when you see the error “Communication with the underlying transaction manager has failed.”

Symptom:

An error message that reads something like:

System.Transactions.TransactionManagerCommunicationException: Communication with the underlying transaction manager has failed. —> System.Runtime.InteropServices.COMException (0x80004005): Error HRESULT E_FAIL has been returned from a call to a COM component.

Solutions:

 

“Check your firewall settings” is what you will find in almost all forum posts and on MSDN. You need port 135 open bi-directionally for RPC’s end point mapper (EPM). You also need ports 1024-5000 open bi-directionally if you have not specified your own port settings for RPC in the registry. If you have your own ports specified in the registry, then those need to be open bi-directionally.

 

WHAT ?!? It may also be your code causing the issue. If you are using TransactionScope, you have to be mindful of every method called within the using braces. Looking at the code below, you will see two service calls and a seemingly innocuous ShouldContinue() method checking to see if the second operation should be called.

using (TransactionScope scope = new TransactionScope())

using (MyServiceClient proxy = new MyServiceClient())

{

       proxy.DoOperationOne(someID);

      

       if (ShouldContinue()) // uh, oh! What if this has an ADO.NET connection that is opened and closed inside it?

       {

               proxy.DoOperationTwo(someOtherID);

       }

}

If ShouldContinue() opens and closes an SqlConnection, the TransactionScope object has no means by which to commit or rollback this part of the transaction. This will cause the error “Communication with the underlying transaction manager has failed.”

 

1. If you do not need to results of DoOperationOne() to feed ShouldContinue(), then do that logic before the TransactionScope using block.

2. If you do need the result of DoOperationOne() to feed ShouldContinue(), then you can wrap the internals of ShouldContinue() with a TransactionScope using block specifying TransactionScopeOption.Suppress. This will not add the resource access contained within the block to the ambient transaction.

3. Use an intelligent data access library like Enterprise Library that manages your connections for you. It won’t close the connection if enlisted in a transaction.

 

Look at your code before you involve your network dudes. This is more common when integrating legacy code with new service calls.