Creating an extra high productivity & extra high quality SW development team

Background

We have actually known for a long time much of the concepts and theory of good software development. The journey of 50 year of computing has gradually led us into the point where we understand.

The result of this is model driven process that ends up to a 3-tier application implementation. The layers are: Application layer, Domain layer and Persistence layer. The key element in this approach is the unidirectional association between the application and the domain layer. Here the application layer is strongly associated with domain layer but the domain layer has no references out of it at all! The development process should start with abstract domain modeling. This phase won’t take more than 10 working days in average of a team to build. The development starts from just implementing this model. Then it continues incrementally with design and implementation of the application GUI and workflows. As the work proceeds the details are also added to domain layer that started with skeleton model. The experience from several projects is that the number of classes increases very little but addition of attributes and behavior is of course substantial.

Great opportunity for a top high quality & high productivity team

Currently most of the development of operational software for companies is done based on very old and proven bad paradigms. In practice majority of development is done with a waterfall approach. This is bound from different directions like, wrong idea of the nature of the work, planning and working tradition, buying, selling and agreement traditions. This all lead to situation were a carefully selected team of skilful professionals would have an outstanding advantage in competing on the arena.

The structure and the working process have been proposed and even used many time in the past. I will assembly here ideas from Frederick Brooks: Mythical Man-Mont, Grady Booch: Object Solution, Ken Schwaber & Mike Beedle:  Agile Software Development with Scrum and Kent Beck: XP.

The good principles and best practices for such team could be added up as follows:

  • Small team with member each high professional skills
  • A Scrum-like working team yet somewhat different duties and responsibilities between team members. One could call it “surgical team” like Brooks did. In software the “surgeon” would be call chief programmer & architect.
  • There could be two teams. One for domain & total architecture and another for GUI & work flow control details. The interests of these teams are clearly more loosely connect than inside these teams.
  • The working process should start with clearly separated time boxed abstract domain modeling. Then the implementation process should start with generating the implementing code from this abstract model as starting point and guide lines for the incremental and iterative development work.
  • Professional skills tend to be badly underestimated. Somehow in IT-field the dominating assumption has been (as perhaps is still) that most of software development is routine work, which one can do with short introduction to work. The truth is complete opposite of this! The software implementation is difficult design work from modeling until the last line of coding. It is mostly problem solving and successful people have both good skill and enough routine work behind them to be good at it.The ideal small team could consist of following roles:
    • Chief domain developer. This fellow is the “surgeon”. He has the main responsibility of the overall success.
    • Domain co-developers (1 – 3). These as the assistants of the chief developer. The domain developers discuss and make the key design decision together. One of these is the deputy chief developer.
    • Application developer who designs and implements the work flow control
    • GUI designer
    • Case dependent additional roles
  • The team is really functional after a few ramp-up projects as each member learns to fit and function in this particularly team. To replays any team member requires a fully professional candidate and a learning period with assistance of the leaving member or the rest of the team.
  • One open question (that I cannot provide an answer now) is whether such teams should practice their performance  in advance outside any customer projects.

Really good customer is a must

All this things doesn’t work without the final link in the chain. This is the buyer, the one that the whole thing is done for and the one that should be the most interested in the result.

Participating in such a process require real interest and committing to the process. This means that the organization appoints the most knowledgeable key people in the project and at least some of those fulltime and that the management of the company is actively participating in the process.

In an agile development process very important decisions must be made all along the process. Actually the development of an operation management system is always at the same time a huge opportunity and a steep learning process for the whole organization.

Choosing the customer (at least at early state of working this way) is actually more crucial than creating the best possible team.

Advertisements

Dynamics of object creation

A fundamental starting point is that all the classes in a domain model are connected. This means that there is always an association path from one class to another.  In most cases the creating objects are directly associated with the new object. On the other hand domain layer objects don’t have any reference outside the domain. All domain objects must be created by a domain object even if the stimulus to do so comes almost always from outside (from application layer).

When an application usage starts, some entity object exists. So creation of these objects is out of the scope of the application. Of course technically every object of an application must be created, but this is not the issue. So a set of objects exists and are in a way given to the application. In most cases for example the home organization, its roles and persons exists.

Now when we start to analyze object creation, the first fundamental base finding, that by the way is easily passed by beginners, is that an object cannot ever create itself. The next issue is can an object from the same class create an object from the same class. This is true from some classes and not true for others. For instance object in Persons class can create a new person. Now actually this needs two objects from person class and 9 months – but serves as an example anyway. An example of entity class that cannot create an object from same class is the class House.

When we take into account the Coad’s color categories then we can be more precise. My first examples were bots entity-type classes. So we can say in general that some entity class object can create object from the same class.

Role-type classes are actually objectified associations. There is even no theoretical aspect why a role object should create similar objects. The same applies to Description-type as well as to Event –type.

Typical creation patterns

Let look at different dependencies and situations at object creation. Let’s go through Coad’s categories what are the creation patterns and their justification. The bottom line here is to set the behavior into an object that “knows” most or best what to do. This seldom corresponds strictly reality instead some level of abstraction (read imagination) is needed.

Creating Entity-class objects

As I already mention some entities can create similar entities. Here are some typical patterns to create entity objects. One example of these abstract ways is the first very common entity creation pattern.

Creating entities from descriptions

Here the idea is that we have a description object, that describes an entity. The responsibility of creation is put in this description. Let’s take a couple of examples: The drawings (the design) of a house will create (read build) the house. A recipe will create (read cook) a meal and so on. In reality this description in most cases is unable to do this, but the abstract way of thinking this is that the description is the actual performer but it uses something of someone in reality as an instrument to do so. In the case of a meal the instrument is the cook.

Creating entities  from events

Similar to description are event. The production of entities from can also be modeled with and event object. In these cases the responsibility of entity creation is typically but into the event. For instance a cook event can also be considered to be responsible of creating a meal. This is an alternative solution instead of the previous one.  The need to have the event in the model instead of just a method is the need to know something about this event itself. Simple example of cook event could be the duration.

Creating Description-class objects

This is very similar to entity case. The domain stories often starts from a description, so they are not actually created within the domain but rather outside. Of course the application layer have to have a functionality to crate object that are starting points of the domain behavior. This is however another story.

Creating Role-class objects

In most cases a entity object will create a role for another entity. This creation of a role enables this entity to participate in some activity. For example a role of developer have to be created for a person by the organization entity before it can be associated for instance I a project task.

Creating Event-class objects

This is perhaps most flexible category. Entities, role are typical creators of events, but event can be created by other events and descriptions as well.

About understanding and communicating an object model

A model by definition is a simplification of reality. The main reason for model is always a simplification of existing or a design of something to be. Sometimes as an extreme a model can be one-to-one, but calling that a “model” is actually somewhat misleading.

The reason to have a simplification of something in most cases is the fact that the total amount of detail would be unmanageable and not understandable and for this most general reason we use models.

As we model concrete 3D things like houses we can have the model in 3D also, but the size of the model is far smaller than the size or the real thing. The downsizing evidently will disregard or destroy some amount of detail and simplify the model compared with reality . Another way of modelling these things is a mapping done from reality (current or future) to some kind of language. In the case of building this language consists of geometrical figures, which maps to realty. For instance a floor plan consists of together joined rectangles and a set of other graphical signs like door and windows.

To understand meaning of a model implies that the person in question is able to “see” the corresponding reality trough the model. If we use a mapping from reality to not self-evident language (like in case of house plans or maps), then the prerequisite for understanding the model is the knowledge of the discreption language.

Understanding is not a two-value function (on ; off) but a complex multi value thing. Thus it is very risky to say that a person doesn’t or a group do don’t understand the model. Rather we should consider to what extent someone understands the model.

It is obvious that the more complex the model is the more difficult it is to understand all aspects of the model. Even the “understanding” concept is difficult. The complexity of a model depends on two independent sources: 1) the domain in concern intrinsic complexity and 2) the chosen abstraction level of the model.

The complexity of the domain cannot directly be influenced. Indirectly this can be influenced by limiting size of reality to be modelled. The level of abstraction is completely in hands of the modellers. The semantic descriptiveness of a class and its level of abstraction has the following dependency with ease other:

This means that when chosen the level of abstraction of a class this with determine the semantic content or richness of the concept. So the lower the level of abstraction is the more semantic meaning the class name (&definition) will have. Of course it is good to have as much semantics as possible but the contradicting demand is to reduce the amount of less important details to promote understanding.

To balance the model between these demands the most effective choice is somewhere in between the extremes (the orange line in the figure). A very important aspect of OO-model creation is that towards the end of modelling a given domain the emphasis is on simplifying the model as much as possible without losing its semantic weight or value. This means that we approach a point where leaving something away will have a dramatic effect on the model collapsing it totally. Here the model has reached its complexity minim (at a given semantic level). If someone then demands to reduce the complexity of the model being too complex to understand this will mean no model at all!

The abstract object model consists of a class model and a set of collaboration diagrams. Class model contain all the classes ( concepts) found during the domain analysis and all the relationships between the classes. This class model can be represented in one or several class diagrams. Even if the model is represented in more than one class diagram the model itself is always one and connected. It is moderately ease to show that this model if it is correctly done is in the complexity minim of all models.

I will continue with understanding the model. First we have several different methods to model reality. Some of them are structurally simpler than OO-models like for instance use cases which have hardly any structure at all. Then we can move the whole model (ie. all classes) up and down the semantic-abstraction triangle. If we chose very high abstraction level we loosed a lot of detail and we will have a simple model. So simple models are far easier to understand than the models that have more structure, semantics and more fine grain behaviour defined.

The best OO-model in the middle of the triangle is most effective model. Thus it leads to most effective implementation as an application.

The model include also a set of collaboration diagrams describing a few cases with the deepest and most important sequences, which describes some instances of  creation of the most important event object of the business cycle. These diagrams consists of objects not classes. For this reason the are very concrete and far easier to understand. Sometime we can draw additional static object diagrams to explain complex structure without any messaging.

Why use cases are evil?

The previous post about databases and OO wiht headline saying …also eval actually refered to this text – so their order is wrong. Anyhow here it is now for you.

Well this is not the whole truth and first this is not only use cases but all procedural abstractions of behavior. These come with many names in addition to use cases: activity models, process models, rule models etc. Another fact is that they are not intrinsically mean. Actually they are just one way of abstracting behavior. The reason which makes then mean in context of modeling for application development is that they yield ten times worse solution.

All the way from the early days of OOA (late 80’s and early 90’s ) it was crystal clear that the description of procedural and object behavior are strikingly different.  There is no right or wrong here. They just are the same thing from completely different viewpoints. They are orthogonal abstraction of the same thing.

The most significant thing that created object paradigm was just this difference. This is a paradigm issue and thus the difference has a strong antagonistic nature. This means that you must choose and there is now way of mixing these together.

The big step in deeper understanding simulation modeling was that object collaboration yield to a magnitude simpler model than the corresponding procedural one. What is the cause of this difference? The root of explanation comes from microbiologist Stuart Kauffman. It is question of how to manage complexity. His findings were that the most complex organisms (and organism are always both structure and behavior at the same time) are in the middle ground or gray area between chaos and rigid order.  The conclusion was that you need both ingredients to accomplish.

In case of our models (and also programs) “wild” or “free” behavior (in programming a method or equivalent) represents the flexible chaotic side and class definition, class boundary with behavioral encapsulation and objet collaborations represent the order ingredient. This mixture is actually the trigger that collects the jackpot. This way we are able model 10 times more complex simulation than in the realm of pure chaos. The fundamental aim to avoid the repetition of the same behavior in the model over and over again is gained by the classes definition -or concept- that ultimately limits the objects possible behaviors. In other words the number or trick that a single object can is limited and this of course reflects to implementation so that the number of methods (if designed right) is limited. Thus the object model is the  only way (that at least I know) that can achieve this.

We can ( and actually have several times) come across a subset of reality that we would like to reflect  with application but which is too complex for us. There is of course a limit what we can do with this. The real limitation at the bottom of this all is our computer. Currently it is an implementation of deterministic Turing machine. We know the limitation of this from mathematics.

When we want to exceed these limitations and we would like to create learning applications we must go to nondeterministic Turing machines. The fist difficulty here is that we don’t have the hardware yet. We human stars to understand enough of brain structure and behavior then we can copy this to a machine. On other consequence from this is of course the fact that these system cannot be programmed. They must be taught! The final inevitable consequence is these application can do mistakes like humans!

Confusions about abstraction

Confusions about abstraction

What the concept abstract really means ?

The term abstract is widely used, but the meaning in many cases is vague or fuzzy. To calibrate the term let’s try to define what we exactly mean with the concept (the word)

The next is a quote from English Wikipedia.  The definition of the word abstraction:

Abstraction is the process or result of generalization by reducing the information content of a concept or an observable phenomenon, typically in order to retain only information which is relevant for a particular purpose. For example, abstracting a leather soccer ball to a ball retains only the information on general ball attributes and behavior.

Abstraction uses a strategy of simplification, wherein formerly concrete details are left ambiguous, vague, or undefined; thus effective communication about things in the abstract requires an intuitive or common experience between the communicator and the communication recipient.

Abstraction is reducing the information content

So here the point is simplification by reducing the amount of details.  The purpose if this is to emphasize the important aspects from the chosen point of view.

To say that abstraction is simplification is true, but this expression could be interpreted too strongly. The simplification is in most cases achieved be reduction of details. So there is still room to move within the simplification. The amount and nature of simplification depends on the situation and change case by case.

From pure theoretical (mathematical)point of view we can say that abstraction is a mapping  f ( L) -> R, where L is the source set (in our case a subset of reality)and R is the result set (the model of reality). The mapping  f will abstract (simplified ) the complexity of L if the result set is more simple than the source set. In most general form this means reduction of the number of detail. This implies that the mapping f is homomorphism but is not isomorphism. Otherwise the reduction would be 0-reduction. Homomorphism means that many source points should map to one result point. This way a reverse mapping is not possible. This implies that the mapping genuinely destroys information. This lost information cannot be returned in any way just from the model.

A terrain and a map of this terrain is a good example of this kind of modeling.

Simplification is a double-edged sword. When the point of view and usage is strait forward and simple normally there are no problems. When we consider my geographical map example if we need direction with car in southern Finland or street map for waking in Helsinki city center, then the scaling of  the map is no big issue. The situation changes a lot when the “map” (model) is used in several different points of view and interests.

Common confusions of “abstract”- concept

Non-concrete is abstract
Abstract-concept is often used fuzzily as synonym for non-concrete and/or difficult. Sometimes people say that mathematic or geometry are “abstract” as such. This does not however conform to the previous definition. In other words there happens neither simplification nor reduction of detail. Both of these mainstream fields of mathematic are axiomatic system with rules of manipulation.   Completely other story is then the fact that mathematic and geometry can be used as instruments in abstraction but in such a situation the mathematic is only vehicle or tool to create the mapping between sets. In the same way we can say that chess game is abstract. This is false again. The game is most concrete thing in the world with it’s board and chess pieces and game rules. Following the same deduction also the programming languages are not abstract but well defined games in their own world.

Programming is abstract

The world of computer programming is defined by the physical structure of von Neuman machine. The next question is then do programming languages form layers of abstraction above each other? Many people say that Cobol is more abstract than assembler. My opinion is NO in my strict use of the concept abstract. My argumentation is following. As our current programming languages are all deterministic the code cannot truly simplify anything that effects the decisions made in the path of execution. This implies that all deterministic programs are isomorphic with each other. This means that when we have a function f from programming language a to b which gives the mapping then there always exists e reverse function f-1 which gives the reverse mapping from program in b back to original program in a. The real thing that some call abstraction is only compression! Here is a small simple example of that compression.

Example:  for-loop java 4 ja 5

void cancelAll(Collection<TimerTask> c) {
    for (Iterator<TimerTask> i = c.iterator(); i.hasNext(); )
        i.next().cancel();
}

and

void cancelAll(Collection<TimerTask> c) {
    for (TimerTask t : c)
        t.cancel();
}

Simplification always boils down reducing the number elements – the less important ones.  When the program is deterministic all those elements (attributes and their values) that control the flow of execution control must be present regardless the chosen programming language.  So in every equivalent program exactly the same if– statement must be present in one form or an other

UML is  abstract

UML is one-to-one mapping between a well defined set of concept and their corresponding graphical signs. This mapping is isomorphic between the graph and the garpth “verbal structure” which can be any programming language. This way UML is a transformation algoritm between a diagram  and a description (which can use for instance Java language).

About real simplification

Let’s return real simplification. When we model the reality we face all the time the question of how much do we simplify. When we are doing the mapping between the realty and the model we have to choose the scale. The following diagram illustrates the two sides of the decision. The y-axis is the amount of abstraction (or simplification) and x-axis describes the amount of semantics within a concept. When you pick up a point in y-axis and decide the level of abstraction the at the same time you get the amount of semantic value, which is the width of the triangle at the point. So the higher we are the smaller amount of semantic the concept gives.

Abstraction Triangel

There is a shaded  area in the center of the y-axis.  This shows optimal (read the best possible) level of abstraction and the read thread indicates that different individual concepts (read classes) can be at different levels of abstraction. This actually means that the benefit that we get from abstraction increases to maxim somewhere in the middle from 0-abstrction to total abstraction, which is single point with no content.  See the next diagram:

A common myth is that the higher the level of abstraction is the better. As this discussion shown this is not the case in contrary. As we can see from the parable at first there is a clear increase of clarity will follow but then at some point the raising of the level will reach a point where the simplification starts to corrupt the most essential parts of the information and finally the mapping will collapse to zero.  So both ends of this graph are area of danger.

First example of big crumpling at the left end of the graph was IBM ambitious attempt with objects. The project was called San Francisco (3200 classes, 24600 methods)  at the end of previous millennium.  The attempt was more or less to produce a model that would cover all possible businesses. As you can see from the figures above the level of abstraction was far too low. The model was finally constructed with huge effort. The trouble saw that it saw totally useless with that amount of information.  I am still quite frequently running into attitude where really big (read detailed) models are bolt and to be proud of but sadly the trough is almost the opposite.

The second extreme is of course at the other end of the function. These are models with very general concepts and only a few needed. Usually these models are technically correct but semantically completely empty. So they look nice but don’t contain any real value to develop applications.

This is the point where I can return to Grady Booch, when he asked his audience: “When is a domain model ready?” His answer to this question was that it not ready when all the possible classes have been added to the model but it is ready when you cannot remove a single class from the model without totally collapsing it!

My experience is that such a model typically consists of 30 – 60 classes. So even here the famous rule of Albert Einstein: “Everything should be made as simple as possible, but not simpler.”  is completely valid!

By the way the modeler can decide the number of classes in the model even without knowing anything about the target reality. This is of course done by either lifting or lowering the level of abstraction of several classes of the model.

Abstraction within programming

The level of abstraction of the classes is not directly reflected in the absolute number of classes but rather in the relative number of classes and methods together. This means that the higher the level of abstraction in the model is the more complicated are the implemented methods and vise verse.

This way the logical 3-tier architecture can lift the abstraction level for the GUI programmer by encapsulating the lower level details inside the object boundary into the method implementation.

Stubborn confusion with behavior

All the way from the early days of OOA (late 80’s and early 90’s ) it was crystal clear the description of procedural  and object behavior are strikingly different.  There is no right or wrong, they just are the same thing from completely different viewpoints. They are orthogonal abstraction of the same thing.

The most significant thing that created object paradigm was just this difference. This is a paradigm issue and thus the difference has a strong antagonistic nature. This means that you must choose and there is now way of mixing these together.

The big step in deeper understanding simulation modeling was that object collaboration yield to a magnitude simpler model than the corresponding procedural one.

This is a simple fact of reality and it can never change. This means that it is logically impossible that ever anywhere could come any new information that could change this.

Somehow most of IT community lost this crown gemstone of OO. First came use cases as Ivar Jacobsson introduced them. He had actually good intentions but he was misinterpreted and his idea was turned upside down. Then came components and after this activity diagrams. Now the newest names for procedural description are processes and SOA!

I saw somewhere an article where the writer praised “the blessing” of the fact that we “get rid of spreading the behavior and we can collect that all in one place”. This very thing was just the reason or drawback which was cured be bringing the structure and behavior together and distribute responsibilities between object. In this way we got two layers of behavior with first layer encapsulated within the objects. Then the second layer is the collaborations between objects.

We can actually prove this reduction of complexity today with the huge help from microbiologist Stuart Kauffman. Our practical experience tells us that the simplification ratio in average case is 1: 10.      

My motto.: The Use Cases (by whatever name) are evil so retreat  to collaboration.

 

Unambiguous language and DDD

The real purpose of OO-modeling is  – as modeling as whole – to create abstract simulation of the world. The closer the model follows the structures of reality and the corresponding behavior of the associated parts the better the model simulates reality.

For this reason it is vitally important that all the concepts are clearly define and unambiguous.

I have been developing Smalltalk and Java application since 1994. All of these have been built starting from scratch an abstract domain model and finally implementing this as the middle tier of and application. In my experience the most crucial thing here is right model of the world itself. This means that the business understanding is crystallized in the model. This means that the understanding, the language it is expressed is equivalent with the model. These are the same thing. Thus you don’t actually need a separate glossary because your class and attribute definitions is that glossary!

Yes! This is the most crucial thing of all. When this is right everything else goes smoothly if this is wrong there is very little to be done to correct the situation. More of this at my web-site and of course in Grady Booch excellent book:  Object Solutions ISBN:

Finally a quote from that book:

 The center of gravity of an object-oriented information system should be its domain model.

·         Get this model right, and your task will be much easier.

·         Get this model wrong, and abandon all hope.