Logging Analytics

On my current project I have found an urgent need to improve the general instrumentation.

This started by adding a few useful trace statements that will show how long certain key operations will take.

Typically this will involve reading some logs (or in fact querying the WADLogsTable).

I have just realised that without some useful log reporting queries then I am just setting myself up for a lot of manual work.

Indeed on a large scale system there is no point in adding logging unless you have something that can read and display these in a useful form.

I am wonding about the analysis side. I have seen lots of articles about writing logging but very little on reading logging.

I am thinking of having a set of classes that act like reports to perform prebaked queries on WADLogs. These could expose this data as restful web services allowing other applications (powershell scripts, excel, reporting services…) to display the data.

Here is an article on logging analytics.

Querying Azure Table Storage

Here is how to generically query Azure Table Storage without having to define the entity that is in the table (which you may not know and can vary).

CloudStorageAccount storageAccount = CloudStorageAccount.Parse(diagnostics);
CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
CloudTable cloudTable = tableClient.GetTableReference(“WADLogsTable”);
TableQuery query = new TableQuery();

query.FilterString = filter;
var items = cloudTable.ExecuteQuery(query).ToList();

See my previous post for how to set the filters.

Azure SDK 2.0

Azure SDK 2.0 has been released.
Here are the the release notes.

Breaking Changes:

  1. Windows Azure Diagnostics no longer requires 1.7 storage
  2. .NET Framework minimum version is now 4.0
  3. ServiceRuntime, Configuration and Caching assemblies are now built against the .NET Framework 4.0 runtime. So upgrade all apps to target .net 4.0
  4. Windows Azure Connect– The Windows Azure Connect preview is being retired June 30.
  5. Hosted Web Core Support- Windows Azure web roles now require a <Site> element in the service definition file (CSDEF) for your role.
  6. CSUpload.exe warning- CSUpload.exe now emits a warning recommending that you use Windows Azure PowerShell cmdlets to upload VHDs to Windows Azure.
  7. Service Bus client library – the Message Buffer feature, and all APIs related to MessageBuffer have been removed. You should instead use Service Bus queues.

Here are the actual code breaking changes.

Truncating Azure Logs

The following demonstates how to delete from WADLogsTable from before a given date.
It may not be the most efficient code but it does work.

This has been tested against Azure SDK 2.0

///

/// TruncateDiagnostics(storageAccount, DateTime.Now.AddHours(-1));
///

///
///
public void TruncateDiagnostics(CloudStorageAccount storageAccount, DateTime keepThreshold)
{
try
{

CloudTableClient tableClient = storageAccount.CreateCloudTableClient();

CloudTable cloudTable = tableClient.GetTableReference(“WADLogsTable”);

TableQuery query = new TableQuery();
query.FilterString = string.Format(“Timestamp lt datetime'{0:yyyy-MM-ddTHH:mm:ss}'”, keepThreshold);
var items = cloudTable.ExecuteQuery(query).ToList();

Dictionary batches = new Dictionary();
foreach (var entity in items)
{
TableOperation tableOperation = TableOperation.Delete(entity);

if (!batches.ContainsKey(entity.PartitionKey))
{
batches.Add(entity.PartitionKey, new TableBatchOperation());
}

batches[entity.PartitionKey].Add(tableOperation);
}

foreach (var batch in batches.Values)
{
cloudTable.ExecuteBatch(batch);
}

}
catch (Exception ex)
{
Trace.TraceError(string.Format(“Truncate WADLogsTable exception {0}”, ex), “Error”);
}
}

Minimal Azure Table Storage Sample

CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString"));
CloudQueueClient cloudQueueClient = storageAccount.CreateCloudQueueClient();
CloudQueue cloudQueue = cloudQueueClient.GetQueueReference("test-queue");
cloudQueue.CreateIfNotExists();

Please rememeber that queue names are quite fussy. You need to use lowecase names with hypens as seperators.
It does not like brackets or underscores.
This is especially important if you use the environment name in the queue name to allow shared storage accounts across a dev team.