Archive for the ‘Induction into the dream’ Category


Meaningless Numbers

November 24, 2009

For those of you who enjoy performance metrics and statistics, I bring to you the first semi-serious attempt at producing some for an ESA application.  Personally, I am a realist, who thrives on theory but trusts only empirical data, and isn’t happy until the two meet.  And I’m pretty happy at the moment.  Theory says that a stateful ESA server should be many times more efficient than a stateless server given that there is no state get/set overhead.  So when these first practical tests are performed and the numbers look good, I am happy.

Test PC is an AMD triple core, 4G memory running server 2008 x64.  Server test is simply incrementing a number, but that of course comes through as a discrete method request via a web service as normal.

Test harness is a WPF application which pretends to be some number of distinct users and then runs a preset script indefinitely and short random intervals.  I ran 3 instances of this each pretending to be 100 users, very fast typing users, as they pushed through between 2 and 5 transactions/second.

I took screenshots of task manager at the start, after 300 users logged in, after 100 then 200 then all of them became busy.  Some are memory sorted, the rest are cpu sorted.  A final shot was taken after stopping, but not logging out the 300 users.  The test harnesses also log a time-stamped entry every 200 transactions.

Here they are

Now for the transaction rates – they are all slightly varied as the request intervals have a small random component.   They also ran for different lengths of time as indicated above – 100 went usy, then 200 then 300 then they stopped).
Tester 1 (time=0:20, transactions=5400, rate=16200/minute)
Tester 2 (time=0:39, transactions=13200, rate=20300)
Tester 3 (time=1:06, transactions=22000 rate=20000)

Total transaction rate was 56000 transactions / minute and the heart rate had just started to go up.  And thats just one AppServer!  One day soon I will set up a dedicated machine with a load balancer and maybe 4 appservers and another PC running the test harnesses and see if I can break 1 million transactions / minute on pretty standard hardware.

You know what I’m gonna say.



Abstract Friday (Lazy Friday)

November 20, 2009

Too lazy today to bother drawing/cutting pictures to illustrate yesterdays post.  Sorry.  Instead a quick quiz – to separate the abstract thinkers from the concrete thinkers.

Who can tell me the difference between a number and a numeral?

A number is an abstract concept, while a numeral is a concrete representation (instance) of that concept.

When people are modelling (OOAD type activities) the problem and solution space for a system they are developing, some can go straight to a class model while others need concrete assistance, such as screens and reports and the business classes are re-engineered from those.  Both of these methods are equally valid – its all about the final product, not how you got there (complete opposite of life which is all about the journey eh). 

In our quest for achieving an assembly line paradigm for software development, abstraction is king.  Anything concrete is tough to reuse.   Development starts with OOAD and modelling, and so necessarily needs to be abstract, and should be done by someone familiar with the business domain to help identify reusable patterns from now and before.

Which brings to me to the real point of this post.  If I titled it ‘Multiple Inheritance (MI) vs Single Inheritance (SI)’ no one would read it.  After many years of debating MI vs SI I start by stating: MI is like sight or hearing.  If you have it and lose it its devastating, but if you’ve never it (born blind or deaf) had you dont know what you’re missing.

Another word: orthogonal.  Look it up.  MI is oh so useful during the analysis and design phase – an astute analyst will pick out the 3 or 4 important orthogonal aspects to a business domain and, using MI, arrive at a succint and elegant model.  You might have things that go at different speeds and things that are different shapes.  So if I want a fast circle I inherit from the ‘fast’ class AND from the circle class.  A slow square from slow and square.  The concepts of shape and speed are orthogonal.  I so wish .NET had true MI, and I am willing for the compiler to spit at the slightest hint of a diamond pattern (ignoring object of course).

If you model and so draw diagrams using MI but can not implement in MI, it gets real messy, so I dont do it.  I have to think different without MI and that bothers me.  I grew up – spent my formative years with it – so having lost it hurt.  And still hurts.

Have a great weekend.


Finally! I’m Ready… hit me with the heavy stuff

November 18, 2009

and this is heavy, especially if you are unclear about anything so far.  Deeply and fundamenally unclear that is, not just unclear on how to employ this wonderful stuff.  Thats what this post is about – the final ingredient of the secret sauce.

As proof that the ESA’s utility, we had to build a demo system.  Hello World wasn’t gonna do it.  As it turns out, the ESA gets better the more complex and larger the problem gets – it works fine on small projects, but it works wonderfully on large complex projects, because you just dont need to write much [new] code.

So we built an ERP system.  It doesn’t get much more complex than that.  Right from a G/L, AP, AR, stock management, sales, purchasing… the works.  As a demo system, it’s great because it uses multiple devices (customer and supplier portals are in a browser, main staff use Windows Forms and warehouse and delivery users are on PDA’s running Windows Mobile), it has highly complex algorithms (for pricing, warehouse management & picking etc), it must perform well, be available 24 x 7, be very secure, both permissions [authorisation] and system-wise and easily customisable on a per client basis.  Easy.

So whats the secret sauce… get on with it!

Well, as you can imagine, we ended up with over 150 model [business] objects such as Person, Company, Order, Sales Order, Purchase Order, OrderLine, ItemPriceList, GLAccunt,Debtor,FinancialPeriod etc with relationships between them.  There are 3 types of relationships (well 2 really) association (1-1 or 1-many) and inheritance.  Thats it.  So an Order has many OrderLines, SalesOrder inherits Order as does PurchaseOrder.  You get the picture.  If you’re familiar with old fashoined E-R (entity-relationship) diagrams you’ll feel right at ease looking at our master object model.  Patience… almost there.

On the other side of the page, we had our Business Process Diagrams with actors and use cases.

If you were to instantiate any of the model objects, without limits and through all references, you would instantiate the entire model, consume all available memory, time and space and collapse into a singularity.  Not good.

So each use case needs some data – some set of Fields from various model objects.  The Create Financial Year use case needs the FinancialYear and FinancialPeriod objects, but not the sales order, Create Sales Order needs SalesOrder, OrderLine, ItemForSale, Promotion, and PriceList (but not PurchaseOrder or FinancialYear or StockTake).  So how about if we lassoo a subset of our master object map  and use that subset to fulfil a use case.  Thats it.  Go for a short walk to digest that.

So our modelling tool lets you capture both BPM diagrams and Class diagrams and associate a subset of the class diagram for each UseCase.  If the rules change between customers, we simply subclass the required model object and use the new class for that customer.  Screens are thin, so painting them is a quick design exercise.  All the while of course we are designing very carefully – the model object class design is crucial – it has to accurately reflect the real world – problem space.

Tomorrow I’ll throw in some diagrams and real world examples of this.


Quick review before we hit the heavy stuff

November 17, 2009

Sure you’ve spent some time reflecting on each little chapter in the journey, but there’s nothing like a good solid recap of everything we’ve learned to make it crystal clear.

We started our  journey with a paradigm shift for software development – away from the factory pattern to the assembly line pattern – and realised that if we achieved this then it will be rewarding beyond what it did for manufacturing as we actually consume no raw materials.  We can in fact achieve code normalisation which is not 5% or 33.7% better than today but orders of magnitude.

We went on the rediscover that design is important and good [hard] design should be rewarded with easy implementation.

There was a little aside – the realisation that true and complete separation of the presentation layer from the functional layer presented us the opportunity to be technology agnostic at the client, and in fact invent a new class of UI – the rich/thin client.

The journey continued with relationships – which can be made easy and reusable when you consider set functionality as well as singleton functionality when designing your classes.  We touched on a real world example where ‘advanced’ OO techniques were used to solve what could have been a major unit of work into a simple, managable one.

Finally, all these wonderful business objects needed a home, and we gave them a load balanced stateful home.

Recap of the recap: you can now develop code in c#/visual studio that is orders of magnitude more productive than before, house it in a highly secure and scalable server, and deploy to mutiple rich/thin client devices simultaneously.

Any questions?


Zen and the Art of State Maintenance

November 11, 2009

Oh goody, another religious post!  Turn up your anger dial now.  Although you should really be getting used to some far out concepts by now.

I could never ever understand how the world came to favour the stateless model.  Maybe its just that I never built ‘web apps’.  I built applications with centralised server-based logic and thin clients, but still field-at-a-time interactions, not page-based, and certainly not in a ‘browser’.  Isn’t ‘browser’ a great name for a tool that is used for interactions?  What would you call a tool that is just used for looking – browsing??

So my server sessions were stateful.  Someone logged in, created a session, and then interacted with it.  They changed values, invoked methods, ran reports etc.  No big deal.  If they never logged out, that was a problem.  So I built some session sweeping logic to take care of those.  Easy.  But then one day things got slow – too many users.  Time to make a load balancer and allow multiple Session Managers.  Can’t remember what I did after lunch that day.

No cookies, no state farms.  Just a token or handle to identify which session was yours.  And minimal network traffic – only the deltas (changes) between client and server are exchanged.  So user types 20 characters and tabs out of a field.  20 bytes go upstream, hits some business rules and 52 bytes come back down.  Often no database access even, unless they hit a key Field or invokes a method that requires database access.  I’ve never bothered to count CPU cycles for a typical request/response but I can tell you it will be several orders of magnitude less than a stateless server model.

Need a picture to help illustrate?


Expect an exam tomorrow.  Better review everything now.


Multi-context inheritance and other space-time phenonema

November 10, 2009

So who here really understands relativity?  Has anyone actually read some of Einstein’s papers – those meant for the general public – not the heavy scientific ones?  He was able to convey through simple text and diagrams many of the concepts he was on about.  I encourage anyone slightly curious to read them.  One hint: remember its the inertial frame of reference, not visual.

Why did I start with that?  Because its mind blowing, mind twisting and utterly complex and overwhelming.  Until you get it, at which point it becomes intuitive.  Just like complex or multi-context inheritance.  This is where the real beauty and elegance of all the hard work we’ve done to date comes to fruition.   Lets start with a diagram


Some years ago, a construction company that was doing quite well decided to splurge and buy several of the subcontractors companies – the best ones.  So they bought a plumbing company, electrical, fencing, floor covering etc.  And then panicked because they had several sets of customer data, general ledgers and worst of all many different types of Work Orders, workflows, etc.  There are plenty of G/L’s, AP/AR systems but custom integrated work order processing would be a problem.  Until you invent multi-context inheritance.  Keeping in mind the key catchphrase – “design is everything” its pretty easy.  Identify the common aspects of all Work Orders and Work Order Lines, abstract them, and inherit as applicable any number of concrete implementations of this.  The abstract classes contained about 1000 and 3000 lines of code (Order/Order Line) and none of the subclasses had more than 200 lines of code (and of course metadata descriptions).  The screens / UI were easy and looked like they had been tirelessly built from scratch, custom for each company – perfect fit at very low cost.


Relationships are hard. (All jokes aside)

November 9, 2009

Haha!  Just kidding.   Relationships (in OO computing) are easy.  Relationships in the real world – well thats another story altogether. 

OK, you still claim they are hard?  Thats because you haven’t followed OBA and OSA (see OO 101)  to their ultimate completion.  When you model a class – eg perform that analysis and design stuff on some real world problems – are you modelling for a single instance or are you modelling for set operations?  Or both?  What do I mean?  Consider a Sales Order.  You have modelled a class called SalesOrder, which has a relationship to a Customer (1-1) and to SalesOrderLine (1- many).  So far so good. 


Somewhere of course is a Field or Property called OrderTotal (and tax etc).  Lets fill out details to help clarify…


Note that normally the SalesOrderLine would have a 1-1 relationship with a ItemForSale class but for simplicity’s sake we are selling a fixed set of items from a simple enumeration.

When it comes time to implement this, where do you put the iterator that loops through the SalesOrderLine collection for a SalesOrder and obtains the Total?  This is the big one.  Remember, we are normalising our code base by practising OBA and OSA to the extreme.  While this at first will seem odd (and thus difficult) in the end (and not too far away) this will seem so easy and natural.

So the answers to the above questions are: both – we must model single and set rules and operations in a class.Which means the answer to the last question is SalesOrderLine.  The iterator and totalling code goes into SalesOrderLine.  So far so good, until you start coding this.  At some point you will want to do this:

class SalesOrder


SalesOrderLine OrderLines[];

and its game over.  How do you put the iterator and code for a collection inside a collection?  Hmmmm….  Lets try again.

class SalesOrder

SalesOrderLine OrderLines; //single instance – no collection

decimal OrderTotal { get { return OrderLines.getTotal(); } }

How does that feel?  I remember the day I first did this and it felt wierd, but good.  Then I coded the rest of it, made it work and felt great and took the rest of the day off.  I had cracked it!  Well in theory at least.  Now many years later the whole story is complete.  All of these objects / classes descend from one very very smart class (the prototypical business object), all relationships between these classes are implemented as single instance (although modelled as per usual) and all relationships are navigable both ways.  At runtime, the exact direction of the relationship is injected from metadata and the world adjusts itself accordingly.

That very smart class – the prototypical business object – has a wealth of functionality including the ability to iterate through both in-memory collections (ot itself) or persisted data.  And all of the Fields modelled inside those business objects are instances of other smart classes (Strings, numbers, dates etc) which form our DSL.  Here’s a lovely class diagram to illustrate.


Tomorrow eh.


Object Oriented 101

November 6, 2009

I often ask people what is the classical [computing] definition of an ‘object’.  Polymorphism and Encapsulation is what I get back a lot, but thats not what I am after.  Those are features of object oritented technology, but an object is simply defined as Data + Methods (the older crowd normaly gets this right).

The reason I ask this is because ts a simple concept – Data + Methods.  Easy.  No big deal.  So why is there so complexity in OO these days?  Yes, some things are complex and hard, but we must try to make as much as possible simple and easy.  Thats why I like Data + Methods.  James Martin, the old codger/guru of the CASE world of the 80’s, also likes this simplicity.  So in his book, ‘The Principles of Object Oriented Analysis and Design’, the core concepts he tries to convey is to think Data + Methods.  He used the terms OSA and OBA for the actual tasks performed – Object Structure Analysis (Data) and Object Behaviour Analysis (Methods).

But that was the 80’s.  And we’re a lot smarter now aren’t we.  Just look around you at the wonderful world of software that OO has brought us.  Except that a lot of the world still runs on COBOL, IBM mainframes and AS/400s.  We’ve had 20 years of OO concepts and languages and not that much to show for it.  Yet.

Yes it was the 80’s and he didnt get it perfect, but James Martin was on the right track.  Lets see where time has taken OSA and OBA, and how it fits in with todays world of agile, DSL’s and UML.

No one is going to disagree that agile development is a good thing, just to the degree that you practice it.  Tick that box.  I think I like DSLs  -Domain Specific Language)- but might try a variant – something like DST – Domain specific Types.  Where a Type is of course a class which has – yes – Data + Methods.  So those methods may appear to be a DSL – you can create domain specific methods in Types.  For example, I might have a class of MyStringClass which has some data (or properties) and methods, such as ToLower(), various substring() variants etc.  Now to build a CRM system I will be sending a lot of emails, so I will sublcass MyStringClass into MyEmailAddressClass and add methods such as IsValidEmailAddress, GetDomainFromEmail etc.  Anywhere I need an email address Type as a member variable, I will use this new Type.  Easy.

Small aside here and Coffee time: you need to be alert for this.

Extrapolating from the above, it will become ‘obvious’ in time that using this approach will lead to code normalisation.  Yes, everyone loves to normalise their database (or object structure) but still copies and pastes (denormalises) their code.  </Aside>

Finally, UML is certainly not a bad thing, in fact certain aspects are great, but I am finding less and less glowing reports of full UML adoption.  We love class diagrams and Business Process Diagrams, which when used cleverly in tandem turn out to be a major aspect of our ‘secret sauce’.

The good news: only two more OO lessons and you will have reached enlightenment.


Another wise musing

November 5, 2009

Not so much a pearl of wisdom, but a truism.  That nobody truly pays heed to.

Lets take the universally accepted statement that its best practice to separate the presentation layer from the functional layer.  I dont even need to go into why thats best practice.

How many people really do that?  Easily?  Or with great pain and effort?  So many models to choose from MVC, MVVM etc.  And everyones implementation of these is bit different in any case.

To do that fully and properly, you must use total abstraction.  In your implementation anyways – design and modelling generally require the use of visual objects to help denote their meaning.  Most people just can’t go straight to the abstract.  But once modelled, why cant we have an abstract model of that design, and somehow bind or couple those abstract widgets to presentation objects.  That would achieve the decoupling and allow for functional reuse.  Easily too.

So functionally we have [named] abstract numbers, dates and strings etc, all happily working and executing rules, with concrete presntation objects – Text Boxes, grids etc magically bound [by name?] to those abstract objects.  So we could replace the presentation layer quite dramatically (even change technology!) without affecting or functional code.  Why didn’t I think if that?

While we’re dreaming of this lovely world where software development is easy, using the assembly line paradigm, having complete separation of presentation and functional layers, we may as well go completely nuts.  Everyone loves RIA’s – Rich Internet Applications – but these are still ‘fat’ and Thin is In.  You cant have 1000’s of lines of ActionScript inside a Flex/Flash application deployed to 1000’s of desktops trying to remote to some data somewhere without serious issues.  Soooooo, lets make a rich/thin client.  All the goodness and usability of a rich client, yet all the robustness, centralisation and security of a thin client.

Got it yet?  Of course you have – its easy!  Its a simple, natural outcome of the above – completely separating the functional from presentation layer means you can have rich/thin clients, lots of them, all at the same time, without affecting any of your functional [server-based] code.

Dont wake up yet.  In fact, you’re not even dreaming.  Or are you?



Life wasn’t meant to be easy

November 4, 2009

Twas a wise old man who coined that phrase.  But its bit of a philosophical issue really – if life was always easy it would not be rewarding – no satisfaction, just as if life was all hard it would really get you down.  So sometimes hard is good.

Software development seems to be all hard though.  Analysis, design, coding, testing, reworking, report writing, etc – none of this is ‘easy’. 

Here comes one of my all time favourite observations: another wise phrase is “necessity is the mother of invention”.  Nope.  Necessity may be the father of invention – a shallow, real time but temporal use of a counter edge to open your beer when no opener is handy.  Laziness however, is deep, primal and gestating.  Laziness is the mother of invention.  Why do the same thing over and over again when you can make a tool that does it for you?

Following me?  Nope?  OK – motto time.  “Design is Everything”.

Thats it.  Design is hard. [Good] design is rewarding.  I’ve designed the damn thing and it was hard and I’m proud of it.  Why do I have to go and code it now as well?  That should be easy.  Well it is – and if its not – you’re doing it wrong.

Now there is nothing wrong with code.  Code is good; code solves almost everything.  But code should not be hard.  Dont think that the ESA is another one of those fill-in-the-blanks wizards that either rigidly executes your input without code or spits out 10000 lines of code.  No – we have invented  way to capture good design and make it reusable – with code.  And its easy.