In my previous post I described a “complete antmining algorithm”.

This post is much shorter: for bees it is the same as for ants, with one major difference : bees fly ! The do not use pheromones to mark their path, they dance in the hive to indicate direction and distance. So a bee, leaving the hive already knows where he is heading to. They observe the dancing bees in the hive and choose a distance/direction combination. If they find food they return home, otherwise they search randomly.

Other difference : they cannot continue searching until forever : after a certain time and nothing found, they return to the hive to find information where to find food.

In some of my previous posts I referred to data mining algorithms based on ant behaviour : Ant mining. However, these algorithms only follow a limited set of ant behaviour. For example in the Antminer software of Martens and his coworkers at the university of Leuven each step an ant can take immediatly leads to a new rule, whereas in the real ant world, an ant has to take a lot of steps before it finds some food. Other software looks more like locust mining than ant mining (see my post “ants or locusts” in the data mining category).

Here I want to discribe how I see an antmining algorithm. I do not know if it is effecient in terms of performance, but I am sure I must be efficient in terms of finding qualitatively good algorithms.

To start, you need an ant heap : the nest. let’s put in in the center of a 19*19 grid. Then there is the food : scattered around randomly on the grid, let say 20 rules. Each and every ant starts from the nest and begins a random walk: it randomly choses one of the 8 squares around the nest (later on this randomness will be influenced by pheromone levels). But time goes by for every ant at the same time, so in turn each ant chooses a square. Here comes into play one of the parameters : howmany ants can be on the same square at the same time ?

When each ant has taken his step, the “food level”of the heap, in our data mining algorithm the rule effectiveness, must be evaluated. Until now, there is no rule yet, so the random rule still plays. Let us call this each-ant-step-plus-rule-evaluation the “heap step”, which also means one unit of time.

In the second heap step each ant has to decide : return to the nest or look further for food. He wants only to return to the nest when he has found some food. In our algorithm this ‘food’ is a selection rule : a logical rule of a variable having a particular value (this is simple for categorical variables, but continuous variables have to be binned and then fuzzified). And not only a selection rule, but one that improves the model. Did I say that each ant, when it leaves the nest ‘knows’ the food level of the nest: the selection rule that was in place when he left. So in order to know if the rule he found is beneficial, the ant has to perform a local evaluation: is the new rule an enhancement to the ‘nest rule’ he remembers ?

And so we go on and on.

And now the pheromones : they are placed whenever an ant is walking from one square to another. These pheromones influence the random choice of next squares by giving a greater probability to squares with higher pheromone levels. The influence level is a parameter to be set at the beginning and is a characteristic of the ant heap.

In short :

at each ‘heap step’ each ant takes one step. Before the ant step, the ant evaluates the rule he has found (or no rule) and searches further or returns to the nest. At each heap step also the heap rule is evaluated. When an ant returns to the nest with a rule, in the heap rule evaluation it is accepted or rejected (added or not to the heap rule).

So far (and very short) for an ant heap. Now we all know that the ant world consists of many ant heaps en natural selection works on the heap level. This means that for a full-blown antmining algorithm we must have a lot of ant heaps and select the best ones, let them reproduce (duplicate, mutate (=slight changes to the parameters)) and start all over again.

Must be fun to develop and test for a graduate student. Unfortunately I have to be productive doing datamining, so I do not have the time to try this out. If you do : let me know how it works !

Enjoyed this post ? Then you might also be interested in the following :
Ant Colony Optimisation : ants or locusts ?
Ant Colony Optimisation : list of source codes
Swarm Intelligence in Data Mining : Studies in Computational Intelligence
Swarm versus intelligence

                                     

In some previous posts I encountered some discussions of what is a swarm, what does it take to speak of swarm behaviour ? For example there was the question : do stock traders show swarm behaviour ?

Working further on that question, I took the liberty to establish a table with some swarm characteristics on the one side and some swarm types on the other, indicating which characteristics are present in which swarm type.

Here is the table (click on it to see the entire table). Any comments are welcome.

Apparently ants have been cheating on us all the time. The “emerging” patterns in ant colonies are usually seen as altruistic collaboration in favor of the swarm.

According to the article Royal corruption is rife in the ant world recent investigation suggests that ants are cheaters !

In this presentation, Stephane Cheikh very elegantly tells about his experience in his company to start with web 2.0 tools.
“Its is about changing peoples behaviour”.
“Start small”
Just this second point makes me wonder : How small ? Right now I have started with a personal ‘wiki on a stick’ (smaller is impossible) to organise the information I work with. It is just wonderful how quickly you can organise things and how easy you can find everything back.
I intend after some time to share this experience with my team members (I am not their boss !).
Perhaps after a while we can ‘infect’ another team and another, just by letting them experience the benefits.

Just like Luis Suarez we could stop sending documents by email but in stead send them the link to our wiki-page on the subject.
I wonder if this is going to work.

Untill now I read, think and also write a bit on knowledge management and enterprise 2.0.

The more I read and think about it, the more I miss it in my job. I work in a company where everybody still keeps all information scattered around on word documents, spreadsheets and powerpoint slides. If you know where it is saved, you can find it.

Looking for something like quick wins I found Wiki on a stick , which is described as a personal wiki.
However, I took it to my job and saved it on our fileserver. Although not perfect, it is fairly easy to use and I even think it might be useful for a small team. It requires no software (it is one single totally self-adapting javascript/html file) and all you need is a browser (it works fine with mozilla firefox at home and MS internet explorer at the office).
So no administrator access needed and anybody can use it. No expensive knowledge management team required. You just copy a “blank” wiki on a stick to a place where your teammembers have access and you have a team wiki up and running and within the firewall of your company. The beauty of it is that you can go on the way you were. Just type in short descriptions and overviews on your wiki and link them to the oldfashion documents and folders. The first advantage is : you do not have to search anymore. The second : while you commit yourself to keep your wiki up to date, you allways have a good view of what’s going on in your team or your projects.
This is a really quick win, and a cheap one !

When there is a crowd, people go towards it to see what is happening. It may be interesting or there may be nothing to see. They do not know in advance. The crowd only dissipates when you can see from a distance that a lot more people are moving away from it than to it. So people know upfront that the thing is over.

Ants do not rely on crowds to know whether or not there is food. They use pheromones.

On the web people use ratings or page ranks to know if a webpage or site is worth surfing to. These ratings are local. There are the equivalence of a crowd watching some street artist. Only when you are already there, or the address is clickable, can you see what others think of it. Not before! This whould pose a problem for ants : how to get there ?

In the ant world pheromones are placed on the way between the food source and the ant hill, when the ants return home loaded with food. Other ants just have to follow the largest pheromone trails to find the food source. Pheromones dissipate and as such dissapear when the food source is depleted. Can we use things like pheromones on the internet ?

NO, for two reasons :

First : ant food gets depleted after a while, while the only thing that augments when shared is information/knowledge. So on the internet the information sources are not eaten, but copied and stay there, available for anyone forever.

Second : where could you lay pheromone trails on the internet ? There are no roads to follow. You just jump to where you want to be, you do not follow a road ! So Surfers on the internet behave like locusts. They jump from one place to the other without touching the ground in between.

One more thought about depletion : even on the internet the information gets depleted. Although it stays there, it becomes obsolete, newer information is more interesting, so it is necessary for googles and yahoo’s and diggs to let their page ranks evaporate, and give a higher weight on more recent links, clicks, rss feeds and whatever shows interest in some web page.

“Exploring Failure-Factors Of Implementing Knowledge Management Systems In Organizations” by Peyman Akhavan , Mostafa Jafari, Mohammad Fathian, Iran University of Science and Technology In the journal of knowledge management Practice contains this interesting list of the ten most important failure factors of knowledge management system implementation :

1. Lack of familiarity of top management with dimensions of KM and its requirement

2. Selecting an unsophisticated and inexperienced person for leading KM team

3. Improper selection of knowledge team members

4. Wrong planning and improper forecasting for the project

5. Lack of separate budget for knowledge management project

6. Organizational culture

7. Lack of support and commitment of top management

8. Resistance against the change

9. Inability of KM team for distinguishing organizational relations

10. Nonconformities between current systems and new systems

This list, although every item is reflecting a great deal of reality, is nothing new under the sun: it is exactly the same list that shows the most important failure factors of any IT project ! In fact, if you take a closer look at the list and you omit the words “knowledge management” that appear here and there, you end up with a perfectly good list of why projects fail.

Another example of taking old wine, pouring it in a new sack and selling it as new wine.

 

I stumbled upon this 1998 article of Liam Fahey and Laurence Prusak called “The eleven deadliest sins of Knowledge Management.” Although the article is 10 years old, I think it is still up to date. Here the list of sins :

1: Not developing a working definition of knowledge
2: Emphasizing knowledge stock to the detriment of knowledge flow
3: Viewing knowledge as existing predominantly outside the heads of individuals
4: Not understanding that a fundamental intermediate purpose of managing knowledge is to create shared context
5: Paying little attention to the role and importance of tacit knowledge
6: Disentangling knowledge from its uses
7: Downplaying thinking and reasoning
8: Focusing on the past and the present and not the future
9: Failing to recognize the importance of experimentation
10: Substituting technological contact for human interface
11: Seeking to develop direct measures of knowledge

Luiz Suares at IBM has been busy trying to avoid e-mail at work and replacing it with the social softwares. Apparently after three weeks it has already been a success. I quote :”instead of me going ahead and replying to each and everyone of them, I don’t. I go out there into IBM’s various social computing spaces and provide the answers over there to them“, but read more about it in his post.

Olivier Zara in his book Collective intelligence management goes the same direction but not so far as to want to get rid of e-mail completely. He wants to limit e-mails to information “that can be erased once it has been read.” Meaning that everything that has to be kept should be shared otherwise, in “a virtual information and collaboration marketplace” (discussion forums, communities of practice, e-coaching). This is an environment where each person decides on his own push and pull : what do you want to search for yourself, what do you want to be delivered automatically, what do you want to share …

I go on reading is this amazing book.

If you give information, based on sound scientific reseach, will scientifically educated people believe you ?

Not necessarily !

This post is a very good illustration of it. I copy the key points here. But be sure to read at least the first halve of the original post.

– Pharmaceutical companies give misleading and biased information.

– Physicians prescribing habits are influenced by drug company interactions.

– Pharmaceutical reps and pharma sponsored lectures are often the number one source of continuing medical education.

– Small gifts of food, pens, and other paraphernalia create an obligation that alters prescribing behavior.

– The conclusions in published studies in peer review journals are in part determined by who provided the funding, and the more pharmaceutical funding, the more like the results will be in favor of the pharmaceutical company’s products.

– Physicians often do not know when they are being manipulated.

– Physicians deny that these interactions actually alter their practice. To quote one abstract “Although each physician is likely to consider himself or herself immune from being influenced by gift giving, he or she is suspicious that the “next person” is influenced.”

I assume in knowledge management there is the same bias : how the information is presented, by whom etc. can play an important role on whether or not it is believed or used.

What will be the role of Web 2.0 in scientific research ?

Scientific American will publish an article on this subject, but first releases it on the web for scientists to comment it. So the article itself will be a bit “Science 2.0”.

Here is the article.

In earlier posts I mentioned two major obstaclef for enterprises to adopt web 2.0 technologies : management afraid to lose control, and people not wanting to give away information for free.

These items among others are discussed in length it this article.
One interesting point though is the big difference between knowledge management tools and web 2.0 tools : their price : I quote : “see an almost immediate ROI given their low entry cost” compared to “document and knowledge management systems that take years to develop, deliver, integrate and gain return from“.
Since ROI is one of those magic words managers will listen to, perhaps this is the way of getting an approval to at least do some tests in your enterprise.

Yesterday I wrote about the selfishness of people and the necessity of establishing a written and signed collaboration contract, otherwise people do not give away information or cooperate unless its advantageous to themselves.

How come swarm intelligence works in nature ? Do ants or bees have a collaboration contract ?

In fact in a beehive there is something weird going on.

I will try to explain.

Half of your genetic material comes form your biological father, the other half from you biological mother. But you get only half of the material of your father and half of the material of your mother, otherwise you would have twice as much as them.

This means your father is only “half” related to you, or you to him. In other words : each one of your children has half of your genetic material.

As Richard Dawkins wrote in “the selfish gene” : genes (= the building bricks of genetic material) , they use people to reproduce themselves (just like an egg uses a chicken to reproduce itself; can you tell what was first ?). Parents look after their children because they carry their genetic material.

I will not go into detail about bee genetics, you can google all that stuff, but the fact is that bee workers are not 50% related to the youngsters but 75% ! This means that it is more advantageous to look after the offspring of the queen and the males than to have offspring themselves, which would be only related for 50%.

To conclude : the fact that they do not reproduce but instead show that marvelous altruistic swarm behaviour is purely driven by genetic selfishness.

My conclusion for human cooperation within an enterprise follows the same reasoning: if you want real cooperation, you have to evaluate your employes not only on the work they produce themselves, but also on the support they give to others. Meaning that one way or another you have to measure their level of cooperation and that they know that cooperating is advantageous to themselves also and not only to the others.

Obviously information has to come from somewere.   Things, happenings, events, feelings, about anything can become information :
By people :

• information capturing :  capturing and remembering information
• thinking alone
• thinking together : use multiple brains as one large brain
• various meeting types : decision meeting, work-meeting, brainstorming …

By computers:

• information capturing : capturing and storing information streams
• data extraction, data selection : fetch data out of a database, googling, …
• report generation : summarize and present data
• data mining : structuring data, finding selection rules, patterns …
• text mining : structuring documents, sorting them out, tagging (example : Silobraker)

Knowledge management is about a lot of things :

• information
• learning
• technology
• sharing
• management

Earlier I wrote about sharing but also about the problems people have because they do not want to give away their information for free (see “information=gold ?”)
About this last problem, Olivier Zara proposes a collaborative contract : a written agreement between the manager and the employee, that “supports intellectual cooperation by making it tangible“.
It is clear for him that knowledge management will not succeed without a culture change. This can be achieved by simultaneously start with three aspects of knowledge management :

1. provide the means to cooperate : intranet, web 2.0 and whatever is needed technically
2. change the “mental framework” by learning, reading, training about knowledge management and cooperation (collective intelligence management techniques)
3. announce, present, co-construct and finnaly sign the collaborative contract

The first of the three is the simplest, the last the hardest and will take the most time.

It is all written in his book Collective intelligence management , which I just began to read and seems very promising.

I suppose more of it will follow in my blog.

The Cynefyn framework of Dave Snowden is meant as a decision taking framework, but I guess we can use it to structure our information to find our ways in knowledge management.
Where to put/find which information ?
Cynefin sees five domains :
1. simple (= known information). This is all that is for sure, not to be questioned. Like enterprise rules, HRM regulations, etc.
==> Should be put on the intranet, for anyone to find what to do or to find the rules to behave normally in the enterprise.
2. Complicated (= knowable info). Information that is not that clear, questionable but after looking and comparing and thinking about it you should nevertheless find what you are looking for.
==> could be on the intranet, but a lot could find its place in blogs, wiki’s and other enterprise 2.0 tools.
3. Complex (= never clear or explainable). Information you only afterwards know if it was the right choice. You cannot come closer than a good –experienced– guess.
==> typical in blogs, discussion groups, where at least you know it is not clear and different people have different views.
4. Chaotic (= impossible to know). You will nowhere find the information. Just take your best shot to do something about the situation without knowing what the result may be. At least after the facts you will probably have learnt something.
==> put it in a blog or discussion group, it may become “complex” or even “simple”.
5. Disorder (none of the above). Extremely simply stated : you may choose one of the above and act likewise.

I know this does not help a lot, but it may be a first aid to do some structuring in all the information.

Two days ago I saw an interesting documentary on TV about the current and future possibilities of robots. Apart from ASIMO (video on youtube , the Honda site ) and its alikes, they documented also on an operation where they planted electrodes in someone’s head to cure depression (see Dailymail) and apparently it is also possible to increase the intelligence of a rat by 75% just by adding a chip into its brain.

So we are on our way to extend our brain capabilities by adding computer power.

Ray Kurzweil stated that within 20 years or so about everyone will walk around with computer stuff wired into their heads and within 40 years our natural brainpower will only be a fraction of our total brainpower.

If I think a bit further on this it is clear that with all that comes also all the peripheral computer equipment : they will plug in some bluetooth communication devices so that we can communicate to our friends directly with our thoughts. So at last telepathy really will exist, and not only telepathy but also access to internet will be possible, database queries, cellphone-like connections via the 5th of 6th generation broadband network etc… We will be able to show our deepest thoughts on a screen to people who are not wired.

No need of PC’s any more as we know them. The cpu will be in our head (of course a 100 times more powerful than our natural intelligence). Perhaps only a screen for visualization and loudspeakers ? No ! your PC uses a screen and loudspeakers to communicate with the user. In the future the communication will be directly with our brain, no need for eyes or ears.

This also means that blind people will be able to see thanks to something like a miniaturized webcam. Deaf people will hear etc.

On the other side it will be a dream for marketing to put personalized adds directly into peoples heads.

Hopefully they also will implement a virtual switch to shut the communication down, otherwise the whole night we would wake up by alerts from new mail, rss feeds, ads or chat programs .

Perhaps total new patterns of cooperation will see the light ? And at the end, will there really emerge something like a human swarm intelligence ?

This reminds me of the story by Fredric Brown. At last there may be a God, but not the way he saw it.

If you enjoyed this post, then you might also be interested in the following : Web 5.0 : the telepathic web The Human Cyborg robotic insects or cyber insects ? Is God the result of evolution? Is Google God ? The end of evolution

 

                                     

I found
this text ; on technology to analyse email traffic to prevent terrorist attact
on newkerala.com and a lot of other newssites.
The point is here that with the technology they want to identify potential insider
threats to the enterprise.

The question is : will this work ?

Email analysis is just like other data analysis : it is datamining, but preceded by a
preparation phase of converting unstructured information (text) to structured
information.
There is ample evidence that datamining/textmining works and can generate a lot of
otherwise non-accessible information.
But as in other targeting analysis, the search for terrorists or other “bad” people
is binary : the datamining analysis tags you as good or bad. The problem with
this tagging is that at the end you get four categories : 1) the ‘goods’ that
really are good, 2)the ‘goods’ that really are bad (the false negatives), 3) the
‘bads’ that really are bad and 4) the ‘bads’ that really are good (false positives).
It is clear that categories 2) and 4) are the problematic ones.
Tagging a bad person as good leaves the possibility of a malevolent attact. You misses
him, just like we miss him if we do no analysis whatsoever.
More problematic is tagging a good person as bad. What are we going to do with
that person ? Punish him in advance ? putting him on a black list ? Anyway it wil
have an impact on his privacy, on his quality of life etc.

And even more problematic is that, with so few really bad persons around (and this
is the problem of mining so-called sparse events), you will tag far more good
persons as bad ones that really are bad persons.
You can play with the threshold in order to get much less false positives. But if
you set the treshold high enough so that the number of false positives becomes
acceptable, it is quite sure that the number of true positives will be very, very
small (close to negligable) AND that the number of false negatives, i.e. the
missed bad persons will be so high that the whole analysis becomes worthless.

So yes, there may be a technology and yes, it might work, but as far as I am
concerned in real life it is probably going to be useless.

Image via Wikipedia

With the Antminer+ algoritm David Martens, Manu De Backer, Raf Haesen,
Bart Baesens and Tom Holvoet at the University of Leuven, Belgium try to classify observations by simulating the behaviour of foraging ants. The paths these fake ants follow lead from nowhere to a decision rule. Each step connects two rules, a rule being something like ‘armlength less then 67 cm’ and the total path is the resulting combined if-statement. If the resulting combined rule is an improvement in the solution space feromones are added to the path so that the probability that the same steps are reused rises. Unused steps see their feromones evaporate. Exactly the way ants searchfor food.
Since each possible rule is connected to each other possible rule, and each rule is a possible value of a categorical variable, the connection space increases exponentially with the number of variables. So the principle is nice, but the usability, e.g. for real world datamining purposes, is extremely limited. A dataminer in a financial institution rather uses 1000-odd variables. If you transform 1000 variables in 5 categories per variable, this gives you (5000 times 5000 minus 5000)/2 connections = 12547500 connections or pheromone levels.

More scalable is the solution by Michelle Galea and Qiang Shen in a chapter in Ajith Abraham, Crina Grosan, Vitorino Ramos : Swarm Intelligence in Data Mining . Although they call it “Simultaneous Ant Colony Optimization Algorithms for Learning Linguistic Fuzzy Rules” what they really describe are locusts. They hop from one rule to another, so on their way there are no pheromones but thin air. In stead the places where they land can contain different pheromone levels, influencing their choice either to jump further or to stay (meaning to grasp the rule). Here the number of pheromone levels is exactly the same as the number of rules (5000 in my previous business example).
The drawback of this locust method is that there is no direct interaction between the different rules, because there is no path connecting them. A locust hops trough the sky and can land anywhere.  But the usage of a swarm of locusts should easily cope with this disadvantage.

I like locusts.  For together with this scalability the authors show the possibility of learning multiple rules simultaneously. And there is also the fact that they use fuzzy rules.

Very promising !

                                     

Sometimes in the halloween period I called it “datascarehouse”. Today it was just a datashithouse. Not only missing data but worse : the by-the-way mentioning that they changed the layout a bit. Some data will from now on be stored in another table and probably differently modelled. So far for my data extraction code and model execution. So far for my planning. Why did’n they tell that a few months ago, so that I had time to do the necessary programming ?

Am I the only victim of datawarehouse orks?

What is the role of management in knowledge management ? Some aspects of it are given in short by Lucas Mcdonnel : “to foster an environment that allows for creative and collaborative sharing”
I would put it a bit stronger : not only providing the environment, but also stimulate a good and productive usage of this environment.
Knowledge management is like a never ending project (which is a contradictio in terminis !). The goals have to be defined, the ways to reach those goals have to be set out and the measurements (Key Performance Indicators) that allow to decide if the goals have been reached have te be put in place.
It is evident that the ultimate goal is to be more productive : sell more. But apparently it is extremely difficult to know or determine which knowledge management item is responsable (or is not) for which increase in products sold.
So why not analyse it properly before with tools like cause-and-effect diagrams (or a lot of other diagram types ) in order to understand which intermediate goals has first to be reached.
And of course for each goal a concrete measurement is necessary, otherwise how could you know where you are ?
Good candidate things to measure are activities, knowledge assets, organizational processes and business benefits (from “Measuring Knowledge Management Projects:Fitting the Mosaic Pieces Together, Alton Y.K. Chua and Dion Goh).

So it’s clear : implementing knowledge management is something more than buying a software.
Knowledge management is like any management process. I always compare it to driving a car : allways keep you eyes open to see where you are (measurements!) and use steering wheel, brakes and accelerator when necessary (act in response to the measurement), and of course, you want to know where you are heading …

Today I found a list of topics of interest for Swarm Intelligence and patterns 2006 (SIP´06).
Apparently what you can do with swarm intelligence is infinite. Or can you do anything and call it swarm intelligence ? like the data management / information management/ knowledge management hypes ? Or is it somewhere in between ?

Here is the list :
– Intelligent Systems Design.
– Advanced Signal and Image processing algorithms.
– Pattern Recognition and Emergent Behaviour.
– Data Categorization, Visualization. Data and Knowledge Extraction /
Representation.
– Feature Extraction and Selection. Unsupervised Learning.
– Information Systems and Knowledge Management.
– Collective Intelligence, Behaviour and Search. Exploring versus Exploiting.
– Artificial Habitats and Information.
– Exploratory Data Analysis. Data-Mining.
– Cognition, Interactivity, Signals and Communication.
– Bottom-up Strategies and Non-Hierarchical Systems.
– Adpative Systems and Self-Configuration.
– Mapping Concepts, Cognitive Maps and Self-Organizing Maps.
– Particle Swarm / Cultural Algorithms.
– Complex Adaptive Systems.
– Stigmergy, Self-Organization, Metamorphosis, Emergence and Co-Evolution.
– Artificial Life as well as other Animal Societies bio-inspired algorithms.
– Flocks, Herds and Schools.
– Artificial Societies and Web-based Communities.
– Wireless Communication, Cellular Systems, Indirect Communication through
artefacts.
– Social Networks and New Media.
– Artificial Immune Systems and Self-Organization.
– Classification, Sorting, Data Retrieval, Clustering.
– Web Mining, Semantic Web, Collaborative Mining, GRIDS, Network security.
– Auto-Catalysis, Positive and Negative Feedbacks, Cybernetics.
– Swarms and Cooperative Robotics.
– Distributed algorithms, self-regulation, self-repair and self-maintenance
ontologies.
– Biomedical, multimedia and e-commerce applications.
– Collective on-line Games. iDesign, Active aLif(v)e Art and e-Artefacts.
– Generative and Computational Art.
– Hybridization with other methods (e.g. Evolutionary Computation and
Neural Networks).

Simon Garnier brought my attention to a paper by Thomas Seeley with the thesis that in biological systems, when things become to complex for one individual to hold control (e.g. too much information to process in time) swarm intelligence takes over, at least when the biological system contains a large number of units. So far so good.
But what about humans ? It seems that humans — and to a lesser extend other rather intelligent species — have found another solution : hierarchy.
Not one CEO of a large enterprise can control every tiny procedure that has to be followed in his enterprise. Does he know how to mine data ? How to drill a hole in a piece of aluminium ? How to invent a new system for electronical control of car brakes ? etc …
And still he is in contol. In control of the people who are in control of the people who are in control of the people who …
How come we do not observe this hierachy in other species ?
Take for example a pack of wolves. Yes, there is a hierarchy, but a linear one. There is nothing like a work breakdown structure, or better : command breakdown structure.
People are intelligent enough to organise a complex reward/punisment system thats supports a complex hierarchy. Apparently this complex hierarchical system approach has outsmarted the swarm approach of other, less intelligent animals.

And we see it everywhere : political systems, economical systems, enterprises, … Humans organise themselves, so that when complexity gets too big for one individual he delegates parts of this complexity in stead of loosing control.

And now, with the web 2.0 tools around things have been moving so fast that the swarm approach is setting foot in humanity. No one has gotten the time to take control and organise the world in a novel hierarchy. There have been, and still are some who try : the most obvious one is of course the semi-monopoly Microsoft. Will someone succeed ? In that case, since there is only one internet he or she will have full control ! Beware !

From simple statistics we know that by taking larger samples, the calculated average lies closer to the real average (standard error of the mean equals standard deviation divided by the square root of the number of observations).

The principle behind this is simple : by taking a small sample, you get a random mistake. By taking a lot of small samples you get a lot of random mistakes. The good thing is that the average of these random mistakes approaches zero as the number of small samples increases.

In data mining this using of multiple samples was invented by Leo Breiman and called bagging (bootstrap averaging) and is a very powerfull source of model improvement. But using this technique you may not forget that each separate sample & associated model has to make a random error. In data mining we call this ‘overfitting’. Without overfitting the gain of bagging largely diminishes.
In the recent Netflix data mining competition, the top results were obtained by averaging the results of a multiplicity of models, each with their own ‘random errors’.

But not only the number of samples is important : the difference between the samples has to be as large as possible. Which means that the diversity within the original data source has to be large enough.

Now the question : is bagging only useful in data mining ?

NO !

The google report of their prediction markets showed that people close together showed similar trading behaviour. So, to obtain the obtimal result, you need multiple ‘samples’ of people, samples form a large original data source : in other words, people from as much different locations of the enterprise as possible.

Now I come to the idea of Jenny Ambrozek in a post on the application gap. Enterprises with geographical dispersed staff are in an advantage here to deliver better solutions to problems due to their larger diversity of thinking.

Persueing this line of thought we arrive at web 2.0 situations. If thinking, and information exchange and storage in an enterprise is as free as possible, not limited by a zillion internal rules or technical obstacles, the diversity of information increases and allow much more diverse idea’s and finally better solutions, products, innovations etc…