Friday, November 20, 2015

N-Gram Extraction

For reporting purposes we do some simple word analysis. I've been looking to expand on that by analyzing the frequency of 2 and 3 word pairs. So I've been evaluating some n-gram extraction algorithms.

Wikipedia defines an n-gram as follows:

In the fields of computational linguistics and probability, an n-gram is a contiguous sequence of n items from a given sequence of text or speech. An n-gram of size 1 is referred to as a "unigram"; size 2 is a "bigram" (or, less commonly, a "digram"); size 3 is a "trigram"

Initially I had been using string.split() to break the text into words and process each one but found the overhead to be too high (lots of string copying). So I changed the approach to loop through each character in the text and build up words. To get the word pairs (or triplets) my first thought was to use Queues. First I had to have a queue that was a fixed size. It needed to drop items from the front when new ones were added to the end. From StackOverflow I found a simple example to do just that:

With that in place I could do something like the following:

That worked but still had some unnecessary overhead with a StringBuilder for each word and having to join the words back together to get the bigram. So I rewrote the code using pointers into the text and copying out the words (or word pairs) as I found them. This approach proved to be the fastest.

I also took one more pass and made the code more generic. Rather than using specific variables for each pairing I used an array to hold on each word location. This code was slower than the previous due to the loops and array access. For looking for larger number of n-grams (say 4+) I think this would be the better code to use.

Monday, July 27, 2015

Now more dynamic than ever

The dynamic keyword was introduced in C# 4 and I remember reading about it back then. Most of the articles (like this one) highlighted using it for late binding of method calls. Useful for calling external libraries in a different language or interfacing with old COM code but not something I would normally need in managed C# land.

Well the other day as I was reading about the Visitor pattern in C# I came across this article (Farewell Visitor) which used dynamic dispatch to simplify the visitor code. Very neat use of dynamic to essentially dynamically cast a variable to the underlying object type before calling a matching overridden method.

To see a quick example of this in action past the following code in LinqPad and check out the results.
void Main() {

 var shapes = new List();
 shapes.Add(new Shape());
 shapes.Add(new Circle());
 shapes.Add(new Square());
 // Only the Shape method gets called each time
 foreach (var element in shapes) {
 // Using dynamic the matching object method gets called
 foreach (dynamic element in shapes) {

public void PrintShape(Shape shape) {
 Console.WriteLine("I am a Shape");

public void PrintShape(Circle circle) {
 Console.WriteLine("I am a Circle");

public void PrintShape(Square square) {
 Console.WriteLine("I am a Square");

// Simple shape and two subclasses
public class Shape {

public class Circle : Shape {

public class Square : Shape {

Friday, June 5, 2015

Use the Snipping Tool in Delay Mode

Over the years I've used a number of screen capture tools. They are great for demonstrating how to steps or documenting a bug. In the past, I would recommend Snagit as the best capture software but then in Windows Vista Microsoft added the Snipping Tool. Snagit remains my recommendation for as one of the best full featured capture tool (with video capture and image markup) but for quick screen capture the Snipping Tool is great.

One of the items that appears to be missing from the Snipping Tool is a delayed mode so you can capture a menu or dropdown list after it is expanded. Well it turns out you can do a delayed capture with the Snipping Tool even if it's not obvious how to trigger it.

First start the Snipping Tool. If it's in capture mode hit Cancel so the window looks like:

Next open the menu or drop down the list you want to capture then press the "Ctrl" + "Print Screen" buttons on your keyboard. This will trigger a new capture with the screen locked as it is so you can get an image of the menu.

In Windows 10 it looks like the delayed mode will become easier to do with an explicit UI button to trigger it (info from Neowin)

Monday, November 3, 2014

Creating Word documents with Open XML SDK

Creating a simple Word document and adding content is relatively easy using the Open XML SDK (2.5). Here's a basic Hello World example:

public static void CreateWordDocument(string filename) {
    // Create a document by supplying the filename. 
    using (var wordDocument = WordprocessingDocument.Create(filename, WordprocessingDocumentType.Document)) {
        // Add the MainDocumentPart, root Document and the Body.
        var mainPart = wordDocument.AddMainDocumentPart();
        var document = mainPart.Document = new Document();
        var body = document.AppendChild(new Body());

        // Add a Paragraph and a Run with the specified Text
        var para = body.AppendChild(new Paragraph());
        var run = para.AppendChild(new Run());
        run.AppendChild(new Text("Hello World"));

That's great for creating a simple document but what about when you want to add some real content and style it. Once you try and do that you quickly learn that what you are creating with the Open XML SDK truly is an empty document. This is not like opening Word and selecting "New Blank document" which has a bunch of preset styles available. To start having your document look like a real Word document (with things like styles and bullet lists) you have to start adding in classes like StyleDefinitionsPart and NumberingDefinitionsPart (and lots of additional stuff). Even when you add in those additional elements it's hard to make a document look as a good as Word out of the box.

An easier solution is to create your own template and use that as the basis for your new document. That way you will get a bunch of stuff already defined. The simplest way to do this is to open Word and create a new blank document and then save this as a Word template (*.dotx). You can also go into Word and customize the styles and add items like a header page and use that in your template. Once you have a template the code to create that basic Hello World example can be altered like so to use it:

public static void CreateFromTemplate(string templateFilename, string documentFilename) {
    // WordprocessingDocument.Create will overwrite an existing file. 
    // If we are using Open we have to delete the file first 
    // if we want to copy that behavior.
    if (File.Exists(documentFilename)) {

    // Copy the template to the output file name.
    File.Copy(templateFilename, documentFilename);

    // Now open the copied file
    using (var wordDocument = WordprocessingDocument.Open(documentFilename, true)) {
        // We need to change the file type from template to document.

        // MainDocumentPart, root Document and Body already exist just access them
        var mainPart = wordDocument.MainDocumentPart;
        var document = mainPart.Document;
        var body = document.Body;

        // Add a Paragraph and a Run with the specified Text
        var para = body.AppendChild(new Paragraph());
        var run = para.AppendChild(new Run());
        run.AppendChild(new Text("Hello World"));


It's a little more verbose but now you have a document with lots of predefined style ready to add content.

Friday, October 10, 2014

Adding a formatted hyperlink to a Word document using Open XML

I was looking for a way to insert a hyperlink into a Word document I am creating using the Open XML SDK (v2.5). I found a couple of examples on the web and got them to work. The link that was created was clickable but it wasn't styled like a hyperlink. After some trial and error I figured out the issue. Since I was creating the document from scratch it has no styles defined in it. So I created a link in word and copied the hyperlink style.

Here is the example code:

const string outputFilename = @"c:\temp\linked.docx";
using (var wordDocument = WordprocessingDocument.Create(outputFilename, WordprocessingDocumentType.Document)) {
	var mainPart = wordDocument.AddMainDocumentPart();

	// Create the document structure and add some text.
	var doc = mainPart.Document = new Document();
	var body = doc.AppendChild(new Body());

	// Start a paragraph with some text
	var para = body.AppendChild(new Paragraph());
	var run = para.AppendChild(new Run());
	run.AppendChild(new Text("This is a formatted hyperlink: ") {
		Space = SpaceProcessingModeValues.Preserve // Need this so the trailing space is preserved.

	// Create a hyperlink relationship. Pass the relationship id to the hyperlink below.
	var rel = wordDocument.MainDocumentPart.AddHyperlinkRelationship(new Uri(""), true);
	// Append the hyperlink with formatting.
		new Hyperlink(
			new Run(
				new RunProperties(
					// This should be enough if starting with a template
					new RunStyle { Val = "Hyperlink", }, 
					// Add these settings to style the link yourself
					new Underline { Val = UnderlineValues.Single },
					new Color { ThemeColor = ThemeColorValues.Hyperlink }),
				new Text { Text = "Click Here"}
			)) { History = OnOffValue.FromBoolean(true), Id = rel.Id });


Wednesday, May 7, 2014

BASIC turns 50

The other day the BASIC programming language turned 50. BASIC (an acronym for Beginner's All-purpose Symbolic Instruction Code) was designed in 1964 by John Kemeny and Thomas Kurtz at Darmouth. On May 1, 1964 at 4 a.m. ET, John Kemeny and John McGeachie ran the first BASIC programs to be executed successfully from terminals by the DTSS system (see Dartmouth BASIC). I think it's fascinating to know the actual birth date for a programming language as important as BASIC.

This is the language that was on all of the early home computers (Apple II, PET 2001, TRS-80 and TI-99/4A) and essentially launched Microsoft as a company (see Altair BASIC). The TI-99/4A was my first home computer and I remember typing in line after line of BASIC programs from the computer magazines at the time (and saving to cassette tapes but that is another story). It's heritage lives on in Visual Basic and VB for office apps, although thankfully now with objects.

As a birthday present I wrote a little Happy Birthday program. I tried to capture some of the 'spaghetti' problems of the language with the un-needed GOSUB but I don't think I got it. Probably needed more GOTOs. It was fun to take a trip down memory lane but glad to return to more 'modern' languages.

10 CLS
20 LET X = 50
30 LET Y = 4
40 LET N = "BASIC"
60 FOR F = 1 TO Y
62 IF F = 3 THEN LET S = B + N ELSE LET S = B + "YOU"
63 GOSUB 100
70 PRINT "HAPPY " + X + "TH " + N
99 END
100 REM Subroutine
999 END

Thursday, December 5, 2013

Var-ious Thoughts

I started my programming career as a Smalltalker (before Java or C# even existed) and enjoyed the "everything is an object" dynamically-typed variables. In the late 90's, I worked on the VisualAge for Java product (which eventually became the Eclipse platform) using Java. Then I switched jobs and ended up using VB6. It wasn't as bad as I feared because VB6 at least had objects, classes and interfaces (but no inheritance). Still, I was happy when I convinced my bosses that we should switch to C#. It was nice to be back using a fully object-oriented language.

Coming from a Smalltalk background I had been reluctant to embrace a strongly-typed language (let alone strongly-type collections). I found after awhile I liked the some of the benefits like having the compiler catch type exceptions and having the type declaration be a way of self-documenting the code. With the introduction of the var keyword in C# 3.0 I was concerned that we were taking a step back to the VB days. I understood the scenarios with anonymous types they needed to add it for but didn't want to lose documentation benefits. I've come to realize that when you are declaring a variable and then instantiating and assigning an object to it 'var' can save a bunch of typing. Just don't overuse it when the type declaration is not readily apparent (ie don't make me think or have to search to figure out the type).

Here are my general guidelines for using 'var':

  1. Don't use 'var' when the type of variable is not readily apparent.
  2.   var s = SomeMethod();
  3. Use 'var' when declaring a variable and assigning an obvious type to it.
  4.    var collection = new List<string>();
  5. Use 'var' when required to (e.g. with anonymous types).
  6.   var anon = new { Name = "Joe", Age = 34 };