MOCK white box testing

Last night I was introduced to white box testing using nMock and POCMock tools. Mock testing, in a nutshell, provides the required framework to simplify white box testing. Scenarios can be defined that exercise specific code path in your application. Mock'ed classes and methods isolate part of the application code to return predetermined values or throw exceptions. This provide the benefit of:
- exercising hard to reproduce situations (such as specific date or network conditions)
- throw exceptions
- isolate part of your system

Mocking complements unit testing by allowing developers to "mock" part of the application while unit tests are being run: unit tests continue to provide pass/fail indication. I I used to do mock testing by replacing certain parts of the system/application I was building and replace it with a "stub", a fake functional equivalent. MOCK takes this idea and formalizes it, giving developers a structured way to mock part of their system, in the same way unit testing frameworks do (such as nUnit or soon, visual studio team system ;-).

However, it seems that MOCKing is much more immature than unit testing is. From what I gather, there are only 2 frameworks out there: nMOCK (open source) and POCMOCK (commercial app, see Furthermore,
MOCKing is much more complex to setup and maintain than unit testing because the code is test case dependant. Each mock must validate parameters, then return values specific to those parameters. MOCKs can even throw exceptions!

Then again, I thought that MOCKs are especially useful in the following scenarios:
- web services: I was burned once designing a system that relied heavily on web services for its business logic. When these WS changed, unit tests could catch the changes but they prevented the rest of the application from passing. Having MOCKs in there would have allowed me to continue testing and even develop by completely de-coupling the web service provider
- database: all my web systems have a database running it. I always endup having some fancy database values for my tests and development. Sometimes I wish I could just have a MOCK that returns predefined values so that I can develop/test in peace.

MS diagrams are not UML

MS diagramming is *not* using UML but rather DSM, a new, proprietary diagramming notation. VSTS will be able to import/export UML but do not expect more than this. I find this unfortunate as it will require us to pick one direction and not be able to leverage best-of-breed solutions.

Rational vs MS Team System

How is Rational going to react with the arrival of MS VSTS. Here are some thoughts:
- java? Obviously, if you have a client with java project or mixed projects, it seems clear that those cies would remain with RAtional suite.
- rational tools are stable and available now. VSTS will come out in 1.0 version in 2005. Probably not until 2006 that bugs will be ironed out. From now until then, it will be a long and bumpy road. any project that starts until then should consider rational tools, unless you are willing to experiment. In which case, ensure you have Microsoft support (ie. the client must be big and the project meaningful to VSTS team)
- let's face it, MS VSTS 1.0 is not going to be great for projects with large teams. I expect performance issues, probably conflicts, painful use in distributed delivery projects, etc. VSTS will probably be great for small teams (<10-20 people total I would think) in the same location. Then again, such small teams cannot afford Rational suite so there is VSTS market target...
- IBM is planning a complete revamp of their rational products in Q404, with much better performance for tools like XDE, which will become integrated into eclipse and written from scratch to use EMF. Round trip engineering will be easier and much faster.
- finally, VSTS is a first cut that focusses on the most burning SW development issues. It deals mainly with communicating tasks to various team members but leaves the soft eng aspects of the work behind. For example, the diagramming tools are very basic and do not yet support code generation a-la-MDA. This is where developer productivity will increase the most by allowing them to focus on design rather than code (plumbing). Rational is already targetting those applications and there are numerous open source tools available on the market.

Microsoft snubs Rational?

Given that IBM bought Rational, Microsoft is now shunning Rational tools and methodology (RUP) from TechEd. Compuware (test tools) and Borland (requirements management) tools have been presented in keynotes and breakout sessions. RUP is not shipped with VS.net05 team system but rather Agile methodology. I also expect to hear a *lot* more about MS Solution FRamework (MSF) over the next year. Will MSF contend with RUP for the top methodlogy?

What does that mean for Capgemini which has aligned itself to RUP and Rational tools internally for managing its own projects? If history can be our guide, it is safe to assume that delivering Microsoft projects using Rational tools and RUP methodology is going to be very difficult in the future and that Capgemini must have a strategy to support MSF and VSTS in Microsoft projects.

Visual studio team system (VSTS): first reactions

Microsoft has announced a new product called visual studio team system (VSTS) at TechEd Tuesday. Here is what I was able to understand about it so far.

VSTS is designed for all the actors involved in a software development life cycle.
- project manager: can create "work items" which represent tasks, bugs, new features to implement, risks, etc. They can manage this work item database using excel spreadsheet (to view/edit a list of tasks or risks or requirements), ms project (where they can timeline the WI) or in a sharepoint portal (where they access metrics)
- infrastructure architect: can draw network and server configuration and assign contraints
- solution architect: can draw application diagrams that link to infrastructure and work items
- developer: can perform round trip engineering with a class diagram tool
- tester: can perform unit, and load testing with full instrumentation

At its core, VSTS has a "work item" database, basically a "bug tracking" SQL database, but more flexible to allow any kind of WI to be defined and stored in there. All actors can get assigned to work items and update their status, create new ones, etc. from within Visual Studio or Excel or Project. So Microsoft is adressing project management issues in VSTS.

From a software engineering standpoint, VSTS provides diagramming tools and round-trip engineering capabilities. But also, they are *rewriting* their version control software (VSS remains but will be phased out) from scratch to support multiple files checkin assigned to one work item. Also, a build tool is made available outside of The nightly build process becomes part of the default installation and provide build report and metrics, allowing build master to drill down into build results. Unit testing and load testing tools are also available (great for nightly build). Functional testing is mainly provided by 2rd party providers (compuware).

Microsoft does not require one methodology to use VSTS but rather expects multiple methods to be made available as project templates in VSTS. They will ship an "agile" and a "formal" project template. Accenture has also announced that they will come out with a "delivery suite" when VS05 launches.

At this time, VS05 is scheduled for first half of 2005, basically a year away. First beta is expected in the fall. Microsoft has just come out of stealth mode, which means that they are just starting to validate their product feature set with the end users. I expect there will be lots of changes to the feature set as the feedback starts to trickle into Microsoft. There are numerous focus group discussions at TechEd where Microsoft VSTS program managers are polling attendees on what and how they plan to use VSTS. More on this later.

Nevertheless, VSTS opens lots of new opportunities for "addons" such as tools (Borland and Compuware have been demoed) as well as methodologies and a host of metrics and customizations of the tools themselves to provide "templates" of all kinds (reports, default projects, etc.).

To be a software engineer at TechEd feels lonely...

I came to teched as a software engineer, looking for information on how Microsoft is gearing up to battle IBM, now that it has purchased Rational almost one year ago. Specifically, does Microsoft plan to offer tools that bridge the gap between analysis, design, development and project management? Rational has a great methodology out there (RUP) with a number of products that support its implementation. Microsoft has nothing, yet...

Microsoft is maturing, not looking at individual products and new features, but rather at integration and more uniform product features. They want to provide the following to businesses and individuals using their tool
1- manage cost
2- keep businesses running
3- deliver new business value
This however makes TechEd a conference for sysadmins and IT managers, not so much for developers and project managers. They focus on their 600M installed base: this is where the money is for Microsoft and a large number of the integrators and professionals out there. This is great for software integration and operation projects as it provides an engineered view into their software tool, in an integrated fashion.

This leaves lots of place for IT consultants like Capgemini to fill-in the holes left behind on all software development projects. Microsoft has always invested heavily into tools for developers, to simplify their life and increase their productivity. This focus continues this year with the planned visual studio 2005. When it comes out in 2005, there will be basic team management functionality.

Visual Studio 2005 Team System

Microsoft has just given an extensive demo of the team system product integrated to vstucio05. First impression is the following:
- it provides the basic tasks tracking features that we expect to have project managers informed of developer actual work with an integrated work items in vstudio
- provide an automated nightly build system with reporting capabilities
- analysis and testing components (for unit and load and functional tests)
- some diagramming features

Lot is provided which is great but it focusses on the "plumbing" and leaves the design and business analysis work untouched. Then again, if that is all that remains we will be better off than we are now.

This is good news, not great news given that these new features are 1 year away and probably won't be very stable in the beginning...

More vstudio details in the days to come.

do more with less: have they forgotten about architects and project managers?

"Do more with Less" is the conference theme. No one is sure what that means tough...

Steve Balmer's keynote focussed on "plumbing", applications that power businesses (great plains), websites (sharepoint) or offices (office). He demoed new web service integration tools that allow easier integration of WS into office application. The demo allowed users to view client portfolio data from a link in outlook, then open the client portfolio as web service calls, then place a trade and move the resulting trade fill to the email, along with client positions. The focus is very much on how easy it is to build applications.

More and more, the focus will also be on how easy it is to deploy and manage applications. This is reflected in tools like 2005 which will provide a GUI build tool (MSBuild) which can be deployed outside of vs05 so that apps can be built without installing VS first (as is the case now). There are also class designer and application designer coming up in VS05 where you can draw the deployment architecture of an application: hit a button the web app goes to the web server and the db schema to the database.

So if I understand correctly, "do more with less" applies to coders and operations, not to architects and software engineers?


This is my first blog and I am using the TechEd 2004 conference to give it a try.

I am here at TechEd 2004 to investigate the software engineering tools & methodologies that Microsoft plans to use to competite with Rational. In particular, I will track architecture sessions.

Welcome to this Software Engineering blog

This blog has been setup to discuss software engineering tools, technologies, and methodologies, as well as issues that arise when using them.