LINQed IN - Troy Magennis' View on .NET, C# and Software Development

Server Virtualization - Go Green!

Troy.Magennis — Tue, 13 May 2008 16:18:19 GMT

The company I work hosts a decent size data center to support its business (a website hitting around 1.5Mil hits a day, and countless business applications to keep the company in business). Around the end of last year our backup-power (UPS') and air-conditioning were hitting the alarming 90% level (this is understated!).

The issue wasn't the number of production servers, but rather the pre-production servers for Development, Testing and Staging. Replicating the rather large interconnected complex environment took hundreds of servers, all at 5% CPU utilization most times during the month!

Our Savior: Server virtualization

Just by converting as many pre-production servers to virtual instances and hosting these on decent machines, we saw a reduction to below 45% capacity. The additional benefit is that many of these machines were hosted on poor, unstable hardware previously - virtualized, these are running on solid hardware and in many cases perform better than they did.

If every company house that has software development and testing server resources virtualized their pre-production environments imagine how much less power our server rooms would consume. That has got to be better for the financial operating cost and also better for the environment in general!

Troy.

.NET 3.5 Enhancements Training Kit

Troy.Magennis — Thu, 17 Apr 2008 12:40:00 GMT

A comprehensive set of hands on labs and training material has been release for a number of .NET 3.5 Enhancements by the Visual Studio & .NET Framework evangelism team. Nice work guys.

"The .NET Framework 3.5 Enhancements Training Kit includes presentations, hands-on labs, and demos. This content is designed to help you learn how to utilize the .NET 3.5 Enhancement features including:

ASP.NET MVC,
ASP.NET Dynamic Data,
ASP.NET AJAX History,
ASP.NET Silverlight controls,
ADO.NET Data Services
ADO.NET Entity Framework."

You can download it from here.

Troy.

Software Architect - The Test Kitchen for your Company

Troy.Magennis — Thu, 10 Apr 2008 10:52:25 GMT

I've become a fan of America's Test Kitchen, a TV and book series about cooking, where they not only show how to cook a great Pork roast, they will try it 10 different ways and see which one works best. They end simply by describing the best way (decided by a set of tasters normally) and publish that recipe with a pre-amble about all of the disasters.

It occurred to me that when describing what a Software Architect does to non-technical people, I might start by saying "I'm the test kitchen for software development in my company; I try to solve a problem in numerous ways and then describe the one that worked best."

I see a lot of Architects unable to get their hands dirty in code; How do they really understand a new technology unless they are able to cook the perfect integration pattern in their favorite kitchen: Visual Studio 2008! I think what America's Test Kitchen does right is they document the process really well - what worked, what didn't and why. I know my Red Wine Sauce has improved immensely by using their books; If I could just communicate to my team why my Service Software Factory was designed the way it is!

Troy.

AForge - Image, Math, Vision, Neuro and Computer Learning Libraries

Troy.Magennis — Wed, 09 Apr 2008 21:20:57 GMT

I'm probably the last to know, but I had a real need for some solid imaging and graphic library source code. I was incredibly impressed with the AForge library by Andrew Kirillov.

Get the source and library here: AForge on code.google.com

AForge.NET is a C# framework designed for developers and researchers in the fields of Computer Vision and Artificial Intelligence - image processing, neural networks, genetic algorithms, machine learning, etc.

At this point the framework is comprised of 5 main and some additional libraries:

AForge.Imaging – a library for image processing routines and filers;

AForge.Neuro – neural networks computation library;

AForge.Genetic – evolution programming library;

AForge.Vision – computer vision library;

AForge.Machine Learning – machine learning library.

Releases are frequent; The code clearly written; Good samples - all round a great project. Nice work to all those involved.

Troy.

Tips for WinForms Memory and Painting Debugging

Troy.Magennis — Wed, 09 Apr 2008 21:04:00 GMT

Every now and then I come across a Windows Forms application that has memory leak, or graphic resource leak issues. In managed environments like .NET, some developers overlook that many components wrap un-managed finite resources. I worked with a fellow developer Mike Zhang today, and because of the difficulties in replicating the issue, we decided that a full pass through the entire source code fixing these issues was our best shot at solving a niggling production issue. After today, here are our top 4 tips and the first steps to ensuring you have a well behaved Windows Forms .NET application.

Begin Update / End Update (Blank, unpainted regions of the screen)

Many Windows controls have a BeginUpdate and EndUpdate methods. These inhibit any Paint events being called when a lot of data is being added to a ListView for example. Big trouble occurs if an EndUpdate isn’t called matching a BeginUpdate! Most general case, is an exception occurs after the begin, but before the EndUpdate gets called.

To Fix: Do a solution wide search for “BeginUpdate”; Go through the result list (yes, EVERY one of them) and ensure that a the EndUpdate is wrapped in a Finally block.

BAD

   1:  this.BeginUpdate();

   2:  // lots of work, including work that may throw an exception!

   3:  // ...

   4:  this.EndUpdate();

GOOD

   1:  this.BeginUpdate();

   2:  try

   3:  {

   4:       // lots of work, including work that may throw an exception!

   5:  }

   6:  finally

   7:  {

   8:      // EndUpdate called even in the case of an Exception in the lots of work above.

   9:      this.EndUpdate();

  10:  }

Pen, Brush and Graphic Resources (Resource leak)

Even in .NET, a lot of managed code uses unmanaged resources. These managed wrappers implement the IDisposible pattern, where the Dispose method nicely releases all unmanaged resources. However – What if Dispose doesn’t get called? Either because Garbage Collection in .NET is happens on its own non-deterministic timetable, some external reference keeps an object alive, or an exception occurs and Dispose is never called. This manifests itself as areas that don’t draw properly or have an enclosed red cross.

To Fix: Do a solution wide search for “Graphic ”, “Pen “ and “Brush “; Go through the result list (yes, EVERY one of them) and ensure that the scope of the Brush, Pen or Graphic is within a using clause, or a try/finally block.

Read: "using" statement reference on MSDN: http://msdn2.microsoft.com/en-us/library/yh598w02.aspx

BAD

   1:      Graphics g = Graphics.FromImage(sourceImage);

   2:      Pen pen = new Pen();

   3:      // lots of work, including work that might fail

   4:      g.DrawRectangle(pen, 10, 10, 100, 100);

   5:      throw new Exception();

   6:      pen.Dispose(); // never called

   7:      g.Dispose(); // never called

GOOD

   1:      using(Graphics g = Graphics.FromImage(sourceImage))

   2:      {

   3:          using(Pen pen = new Pen())

   4:          {

   5:              // lots of work, including work that might fail

   6:              g.DrawRectangle(pen, 10, 10, 100, 100);

   7:              throw new Exception();

   8:           } // calls pen.Dispose() in all cases

   9:      } // calls g.Dispose() in ALL cases

Manual Event Wiring Up and Object Lifetime (memory and resource leak)

This is without a doubt the most common (and some say only common) memory leak on .NET applications. If your component subscribes to an event in ANOTHER component or class, you must –

a) Unhook yourself in your Dispose method for the form or class you subscribed FROM

b) Avoid multiple hookups to an event if your form is opened, or called multiple times

Tips:

a) Hookup events using the Forms designer when possible; This automatically handles events in the proper way

b) To be sure multiple hookups don’t accidently occur, remove yourself from an event before hooking your event up (this is safe, even the first time) –

BAD
ValidationGlobalManager.OnValidate += new My ValidationHandler(MyValidationFunction);

GOOD
ValidationGlobalManager.OnValidate -= new My ValidationHandler(MyValidationFunction);
ValidationGlobalManager.OnValidate += new My ValidationHandler(MyValidationFunction);

c) If you hook up an event outside of InitializeComponent (i.e. NOT through the forms designer), then YOU are responsible for unhooking your event in the Dispose() method of your form.

OnPaint and OnDraw Event Handlers - Call the base. implementation in all cases (unpredictable drawing behavior)

Many controls allow you to implement your own OnDraw and OnPaint event handlers. However, the base.OnDraw(); or base.OnPaint(); methods MUST be called under all conditions, even in the case your code fails with an exception.

To Fix: Do a solution wide search for “OnDraw ” and “OnPaint“; Go through the result list (yes, EVERY one of them) and ensure that

a) the base call is first, so it always runs, OR

b) the base call is protected with a try/finally block

Mike and Troy.

LINQ Reference Mug and Mousemat

Troy.Magennis — Fri, 08 Feb 2008 21:05:01 GMT

I put together a mug and mouse-mat design with LINQ Reference information. I wanted to have a list of the standard query operators and the new C# Query Expression Syntax on hand at all times. My only concern is spilling coffee over the keyboard when looking up the "Join" syntax reference!

They take information from the HookedOnLINQ Wiki website, and you can buy them for $14.99 if you think they might be useful.

Buy a LINQ SQO and Query Expression for $14.99 + shipping

Buy a LINQ SQO Mousemat for $14.99

I don't have a VB equivalent yet, and I'd be interested if anyone has ideas for improving them or can point out errors or omissions.

Troy.

CodeCamp Seattle Notes

Troy.Magennis — Thu, 31 Jan 2008 22:28:18 GMT

I was fortunate enough to attend Code Camp in Seattle last weekend. It was great to see the event attended by around 300 keen developers.

I took notes throughout some sessions, and posted full details the HookedOnLINQ.com website.

By far, the Parallel LINQ session was a standout. I was surprised just how cleanly the ParallelFx team have committed to making multiple processor development mainstream. This is a library and feature set that is important to watch.

Troy.

Choosing between LINQ to SQL and LINQ to Entities

Troy.Magennis — Wed, 30 Jan 2008 20:59:01 GMT

I've been fielding questions on this topic and struggled for a cohesive answer. Should some people wait until LINQ to Entities and the ADO.NET Entity framework is shipped or go LINQ to SQL now?

I saw on Kirk Allen Evan's Blog a pointer to a MSDN Article on this very question.

Introducing LINQ to Relational Data

Summary: This article introduces two implementations of LINQ, LINQ to SQL and LINQ to Entities, and the key scenarios for which each was designed. (19 printed pages)

- Troy.

Technorati tags: LINQ, LINQ to SQL, LINQ to Entities

Prioritizing and Assigning Scale to Feature Lists and Bugs Efficiently

Troy.Magennis — Wed, 16 Jan 2008 15:14:00 GMT

Sorting, prioritizing or ordering features and business objectives (or tasks, or bugs, etc) with a large group of stakeholders can be painful. Often the loudest voice in the room gets there way; some people just sit in the corner too scared to raise their objections. After sitting through many of these, I thought I’d post some ideas for improving the experience, accuracy and interaction –

Do a quick read through of all the items to set the room to a common definition and understanding of the terms and items being discussed
Clearly post the definition of each scale on a poster/whiteboard in clear view. If you are the moderator stand near it to re-affirm the definitions during discussion
People can assign the boundary conditions quicker than the middle territory. Often a quick pass through the list solely to assign the top rating or the bottom rating (in, out, will, won’t, etc). This process also warms the group up and gets the blood flowing. It is important that if there is ANY dissent, that item be skipped
The moderator should keep an eye on if someone isn’t contributing and focus a question for opinion in that person’s direction when the domain would suit (especially if it is an “obvious” yes or no). This is also a way of making sure a dominant personality gets slowed down in driving an agenda
People get better at the process through experience. At regular intervals, revisit a subset and ascertain the group still feels the same way
After each break (at least every 1 hour), re-evaluate the last few items to “re-synch” people’s barometer. Depending on the time of day or the day of the week (or the amount of caffeine here in Seattle) – people can change their votes

The prioritization process is absolutely central to any agile process. Making sure you get accurately reflected priorities with large group can be very challenging.

Troy.

Lean Architecture - How much up-front architecture is needed in a Agile project?

Troy.Magennis — Sun, 13 Jan 2008 19:43:00 GMT

How much up-front design or architecture to do in an agile project is always a contentious issue. Having the job title with "Architect" in it makes me personally biased; Having a background as a developer and product manager in successful agile XP gives me a huge respect for agile development practices and somewhat balances my development approach. This posting tries to define my current view on how to balance good up-front architectural planning (minimal, but where it counts) and how to assess whether more or less architectural oversight needs to be applied during a project through intervention.

Many agile scenarios promote that a project be split in a set of features; These features prioritized into milestones by the "Customer" and then these features be worked in in almost a serialized fashion. The focus is on completing the current feature, whilst promoting that refactoring be the tool of choice when a later feature adds or alters a previously signed-off complete and closed item.

I can see some issues with this when followed blindly. My definition of "Refactoring" centers on improving currently functional code with lessons that emerge throughout a project. What I see in many projects is that "Refactoring" is the title given to an almost rewriting of a previously closed feature because the number and scope of changes to keep it operating in the face of the new feature. This is bad, and it is seriously affecting development velocity.

Here is my proposed compromise, that allows features to be built in (semi)-ignorance of future features, whilst still allowing for intervention if a pattern of serious re-writing (or refactoring if you want to water the terminology down) begins to affect velocity -

Up-front architecture that should always be carried out as iteration 0 of any project includes anything that WILL BE NEEDED for EVERY feature (or requirements that will cause a registration of almost every feature that has already been completed). This includes technology choices, communication and integration patterns, operational requirements like logging, deployment strategy and localization requirements can be used as a starting list.
At regular intervals (e.g. 1/4, 1/3, 1/2, 2/3, 3/4) of the way through features in a project analyze the task breakdowns of the latter features to determine if there is a class of task that might be eliminated by an architectural building block or by more holistic design/planning

It is the second bullet point I want to discuss further in this posting.

My boss David Anderson run a pretty tight Lean Software software development department. A lot of the lean principles are facilitated by using a Kanban tracking system (and its electronic counterpart being the data captured in Microsoft TFS). The data captured is very thorough and used as a management tool in determining progress, bottlenecks and general project health. My goal is to use the feature and tasks completed so far to assist in determining where architectural intervention or more up-front planning could avoid that class of task being necessary in future work.

Lets look at an example - In a current project that is very integration heavy with a large ERP/CRM system, that our lack of solid planning for the interface specifications (which weren't ready when we started, but are available now) is cause way too much rework as we move through the features in a linear fashion. It way pay dividends in team velocity to introduce a feature right now that looks at the spectrum of interface specifications we now have and strawman the entire message / web service interfaces. If this reduces the number of tasks in future feature that would have revisited completed interfaces and added that extra field, or changed that database schema - improved velocity is inevitable.

But there is a balance - striving for perfection in this planning intervention would also be catastrophic to feature completion velocity. Going overboard with architecture is just as evil as sticking ones head in the sand and simply working longer hours or hiring more developers to close the backlog gap.

Be interested if other teams have analyzed task types to look for patterns or work that might be eliminated by a targeted architectural focus.

Troy.

Technorati tags: Software Architecture, Software Development, Kanban, David Anderson

How Many Languages Do You Need to Build a Website?

Troy.Magennis — Thu, 10 Jan 2008 22:59:00 GMT

Answer: > 6

I was looking at a prototype web project today. The one point that struck me was the number of languages and different technologies that were used in assembling a solution:

Cleint Side -

HTML for the page templates
CSS for the stylesheets
JavaScript for the client side interactive code (and understand the browser compatibility differences when coding)
AJAX toolkit of your choice (still JavaScript, but you need to learn the intricacies of the one you are using)
Flash or Silverlight for embedded interactive animations, streaming video, or rich media element

Server Side (this is technology choice dependent) -

Ruby or C# or Java for the server side code
ORM Tool of choice (might be in the same language as above, but configuration file formats are a language/skill of its own)
SQL - at some stage you are going to have to create database tables, script domain data or debug an ORM generated query! Face it, you need to know at least some SQL!

Developers not only need to have some understanding of these languages, but they also have to know how to debug and deploy each technology. What is the chance of hiring an off-the-street developer who knows all of these skills used in a project today in 12 months time?

It seems like the web development tier is undergoing an explosion of little point tools. Pick an AJAX toolkit, pick an ORM solution, pick a MVC pattern to use. Many of our choices today in leveraging a tool to "expedite" initial development incurs a cost in maintainability, and some won't make the distance and be around or upgradable in a few years time.

I'm simply not sure whether this is a problem at all. Whilst I think there are benefits in designing a solution without external dependencies and using as few technologies and languages as possible, I'm not sure how important it is.

Some other questions I have as I fall asleep -

1. Does Script# help? (write C#, generates JavaScript)

2. Does Silverlight offer hope? (write C# and have it hosted in either JavaScript or a cross-platform runtime)

3. Does using LINQ to SQL or the ADO.NET Entity Framework and/or Astoria offer enough ORM/Rest features whilst using C# as the development language?

4. Does Volta help us at all? "First, design and build your application as a .NET client application, then assign the portions of the application to run on the server and the client tiers late in the development process. The compiler creates cross-browser JavaScript for the client tier, web services for the server tier, and communication, serialization, synchronization, security, and other boilerplate code to tie the tiers together."

5. Should we use Flash totally using Adobe Flex and go with the time tested solution?

Failing Gracefully - Limited Functionality Mode on Failure

Troy.Magennis — Mon, 03 Dec 2007 12:23:00 GMT

I was impressed on a project a few years ago (CommSee Case Study) that the architecture allowed as much of the application to run (albeit in a limited fashion) even when the database server was in-accessible for some reason (other services relying on web-services still operated); They called it Limited Service Mode. For real-time customer facing systems, there must be some functionality that is available when a resource has failed. It is all too easy to design a system that fails completely leaving end-users with no sign-of-life when operating conditions degrade.

What are some of the aspects to consider when desiging in this sort of feature:

Offline Mode - Can the application be built to operate offline and synchronize when network connection of database functionality resumes?
If data is retrieved and sent to web-services, can the server-side start queuing these requests, and catch-up when the DB or SOA bus is restored?
If it is a public website, at least the home page and contact us page should display, even if it is only in one language (hopefully the language which gets the majority of hits)
Not all actions should be allowed; If data or financial loss could occur, then that feature isn't a candidate for use when in limited functionality mode

It is an interesting thought experiment; When you are writing your next feature have a think about how you could improve its end-user experience in the case of a network, database or other failure mode.

Troy.

Parallel Extensions to .NET 3.5 Framework CTP Released

Troy.Magennis — Sat, 01 Dec 2007 12:21:00 GMT

Microsoft released the first public CTP of their Parallel Extensions Framework for .NET. This framework reduces the entry barriers and improves the developer experience when writing Multi-threaded applications. Why is this so important? Because, the laptop or desktop you are reading this post on probably has a dual-core processor. The one you get for Christmas will probably have a 4 core processor, and your IT department is probably getting sales calls from their reps offering then 8, 16, 32, 64 core or multi-processor machines.

I've had a play previously with OpenMP which is a standard syntax for marking up C++ code that then executes parallel (supported since visual Studio 2005, .NET 2.0), but if we wanted these benefits in managed code you are faced with rolling your own threading implementation, (and then getting it working :-) ) Possible, but by no means as simple as it could be.

These extensions will make it easier for everybody to write thread-safe, parallel operating code across as many processors or cores a machine has. You have got to love that.

Parallel Computing Microsite on Microsoft.com

December 2007 Parallel Extensions CTP for Visual Studio 2008 Download

Parallel Computing Blog for these Parallel Extensions

Go download them - What you can't multi-task at work???

Troy.

Building a Event Log Viewer using LINQ

Troy.Magennis — Tue, 27 Nov 2007 23:39:00 GMT

Now that LINQ is available to all developers with the release of Visual Studio 2008, it is going to take some time to realize all of its uses. I now find I examine a .NET class library object and see if any properties implement an IEnumerable collection interface.

Like the EventLog component does. You drag this component onto a form and you immediately can read and write to and from a Windows Event Log. I immediately wanted to see if I could use LINQ to filter and query the entries.

My final aim is to make an RSS publisher for certain event log entries that match a filter. But for this example, I just wanted to display and filter the event log sources into a grid.

The full code can be Downloaded here or a full step-by-step available on the HookedONLINQ Wiki site here.

The two important LINQ query bits look like this:

        private void MainForm_Load(object sender, EventArgs e)
        {
            // retrieve a list of all event log sources, and populate the drop-down combo box
            eventLog.Log = "Application";
 
            var sources = (from EventLogEntry es in eventLog.Entries
                           orderby es.Source
                           select es.Source).Distinct();
 
            comboBoxSource.DataSource = sources.ToList();
        }

        private void comboBoxSource_SelectedIndexChanged(object sender, EventArgs e)
        {
            // retrieve all the event log entries matching the source selection in the combo-box
            var query = from EventLogEntry el in eventLog.Entries
                        where el.Source == comboBoxSource.Text
                        orderby el.TimeGenerated descending
                        select new
                        {
                            Time = el.TimeGenerated,
                            Source = el.Source,
                            Message = el.Message
                        };
 
            dataGridView.DataSource = query.ToList();
 
        }

You can see that is pretty simple queries. To do the same using loops and if statements would have been considerable more work though. Keep an eye out for any type that exposes a list property, and keep LINQ in mind when using that property. If you are seeing lots of complicated for loops and if statements - you should refactor and use a LINQ query instead.

Troy.

Does Continuous Integration Scale to Enterprise Projects?

Troy.Magennis — Mon, 26 Nov 2007 16:41:00 GMT

I've often heard and seen large teams struggle when implementing some agile project practices in large enterprises. I recently joined Corbis who were using many agile practices (introduced by my boss, David Anderson who wrote Agile Management book), but Continuous Integration and Automated Builds weren't on the list.

We discussed why CI was needed for a while, and then successfully implemented it. During this process, I kept careful notes on the barriers we encountered and how those barriers were overcome. On David's urging (I mentioned he was by boss right and does my reviews), I put together this draft article (attached to this post in PDF format). I'm keen to hear feeback on it.

Here is the Introduction -

Introduction

Continuous Integration and an Automated Build process are common practices employed by many high-functioning agile software teams. Many small teams are reporting high productivity improvements, but success stories of Continuous Integration and Automated Builds for Enterprise Scale software projects are much rarer. Some agile pundits have stated that there is a scale limitation for agile practices and these practices are in-appropriate for Enterprise Scale (or high staff count) projects.

This article aims to explore if and why the measurable benefits of Continuous Integration and Automated Build processes begin to decline as application size, and team size grows.

Our conclusion is that there is no inherent reason why Continuous Integration and Automate Build processes won’t scale to any size team. In fact, this article concludes that the problems these techniques solve are a higher pain point for Enterprise Scale applications and become more essential than ever. In addition, this article provides guidance on how to scale agile practices and how to make smart architectural choices up-front to ensure smooth adoption of Continuous Integration and Automated Builds leading to smoother projects and improved success rates.

Interested in your thoughts....

Troy.