Technological Singularities, Accelerating Rate of Change

The Speed of World Transformation

Years ago, I was in a professional crisis. Ray Kurzweil’s The Singularity Is Near was a pivotal factor in my re-commitment to predictive analytics as my own forward path.

In terms of a reading list, this is as good a starting place as any.

Caveat: Only the first one-third of the book is truly useful. The more that Kurzweil goes into his own version as to how the evolution will progress, the more he veers off course.

But definitely read those first several chapters.

Later, I came across this paper from Nagy et al. at Santa Fe: Super-exponential long-term trends in Information Technology. What is useful and valuable about this paper is that it gives a five-year extension to Kurzweil’s work. The Singularity’s figures go up to the early 2000’s. This work extends that timeline with data up to about Year 2005. The exponential (and actually super-exponential) trends identified by Kurzweil continue.

Nagy et al. are careful to point out that their work indicates a “mathematical singularity,” not to be confused with the more broadly-sweeping notion of a “technological singularity” discussed by Ray Kurzweil and others.

Blogposts Relating to Singularity (Technological and Mathematical):

• Going Beyond Moore’s Law
• Modeling Trends in Long-Term IT as a Phase Transition
• Good Reads on Accelerating Change

 

Transitions and Outliers

A lot of value comes from examining the outliers.

A case in point: one of my Text Analytics students applied the Latent Direchlet Allocation (LDA), a clustering method often used in text analytics to create ab initio clusters of related terms. (The Wiki on LDA is actually a good read, if you’re so inclined.)

He found that the longer texts in his collection clustered into a small group. We won’t go into technical reasons why this happened, but about 3% of his data items created this small outlier cluster.

Surprisingly (or not), this small set of outliers gave the first valuable bit of information.

The outliers – these longer text pieces – had significantly more information content than the smaller texts. The writers, obviously, took the time to communicate something. They were not being emotional; far from it. They were actually being as factual and as detailed as possible. Thus, the insight and recommendation was: “These longer texts will contain detailed information on problem situations and what we can do about them.”

This may seem obvious in retrospect, but would not have been apparent if he hadn’t done both the analysis, and the follow-up investigation – the “deep-dive” into outlier investigation.

Outliers provide valuable information. That’s why I’m recommending Maxwell Gladwell’s book, Outliers. This is largely a collection of fun and interesting reads dealing with human performance, not data analytics. However, it will give a sense of why outliers should be given attention during data analysis.

On a different but related theme, Gladwell’s Tipping Point talks about what triggers major society-wide transitions.