Coincidentally at the same time I started listening to an audiobook of Isaac Asimov’s Foundation. I soon observed there was a synchronicity. Author’s note: I don’t believe such coincidences are magical or psychic in origin, as Jung thought, but they are still an interesting mental event and even literary tool.

The sentiment ticker and opinion mining in general would definitely qualify as embryonic Asimovian Psychohistorical tools. Governments and organisations could use them for prediction and other research. A way to formalise events in current affairs and link them with observed sentiment trends would be very powerful. If connections could be found then you may have the beginning of actual equations.

The other subject which is connected is the real, as opposed to SF, subject of Psychohistory. This is a discipline that uses psychotherapy techniques to try and understand the motivation of nations, groups and particularly political leaders. It is fascinating and its primary conclusion is that child rearing is critical for the future of the species. This is because psychological damage to children propagates into damaged maladapted adults who act neurotically and create conflict and perpetuation of their damage in the world. The sentiment mining approach could be a very powerful tool for a modern de Mausian psychistorian because huge volumes of textual data could be sifted for emotion words and phrases which correspond to psychohistorical patterns. Incidentally the baroque violence and depraved imagery of newspaper political cartoons are currently one of the richest veins for sentiment mining by psychohistorians, but an image processor would currently be very hard to make that could do this job.

The next synchronicity was that I watched the Adam Curtis documentary called Century of the Self“. This describes the origins and history of the discipline of Public Relations, which was invented largely by Freud’s nephew, Edward Bernays. He has a lot to answer for in my view. One of the revolutionary tools of PR is the focus group, where consumers describe how they feel and emote about products. Of course the sentiment ticker might turn the whole of the web into one giant focus group – a technology very prone to abuse by spin doctors, subliminal advertisers et al. Curtis makes a tentative conclusion that this psychological manipulation is bad for humanity. After watching I felt an increase in the usual sense that I was being spied on, and even manipulated, by shadowy groups in boardrooms. The democratising effect of the sentiment ticker might however empower minority and progressive groups to fight back. Anyway I think its a cool idea so I decided to let the chips fall where they may, and released it to public domain rather than trying to capitalise personally.

After that I listened to an audiobook of “Demons” by John Shirley. This writer shares an interest in common with me because he is a practitioner of the Gurdjieff Work, a self development methodology where the aim is to become more conscious of yourself and the world. I was in bed with gastric flu and the illness seemed to bring on a painful phase of soul searching and inner struggle. I felt the oppression and injustice of the world so keenly that I eventually had to use a Gurdjieff technique to stop the agonising thoughts. (I was also depressed after hearing that a friend of mine had had her baby taken away from her by the anti-human and utterly mentally hygienic UK Social Services).

My next stop was the site recommended to me by my learned nephew called tvtropes.org. This site is just so amazing that it does indeed blow the mind, witness a very late night spent reading hard when I first found it. It analyses plot devices and common themes in tv, fantasy, science fiction, comics, films, games and any media generally. There is a strong radical current because you eventually get to the idea that media spin, religion and propaganda can all be deconstructed using the same tools the site provides. This is revolutionary intellectual crack, although it does break the spell of bad tv, and thus might ultimately ruin your sleep !

Again the tropes also point to powerful resources available to psychohistorians. The analysis of fantasy was a tool of Freud’s. Tvtropes just gets this to rock even more: wish fulfilment and projection, denial and so many other psychological processes are cast into greater relief by the site’s analysis of our main human group fantasy – stories, plain old stories. The ticker could be combined with this resource also to drill deeper into the public mind.

Next connected in my ramblings was a rereading of some Frank Herbert, allied with a plan I have to collect his complete works. Herbert was just so visionary in multiple areas that the superlatives get boring. Wasn’t the ascension of Muad’dib in Dune so facilitated by the effective PR activities and sentiment analysis of the Bene Gesserit ? Wasn’t the observation of minutiae taught him by his mother a beautiful real time form of sentiment analysis using body language and speech rather than text. (Excuse me if you haven’t read it… again, go there !)

The Consensus Trance was a phrase that appeared, and it started to become of more interest to me as I googled. This phrase was coined by the radical Leary era psychologist Dr Charles Tart. Herbert implied and described it so much. It seemed related to Gurdjieff’s sleep vividly as I lay in bed ill. Herbert too was an observer of trance and psi, and I hope that his view was that people should awake from this sleepwalking cultural state of ignorance into the more full and aware life of the individual. Bernays on the other hand seemed to me contrarily bent on merely learning how the consensus trance of the public can be manipulated for gain, usually by his influential greaseball friends. My disavowal of watching broadcast media seemed more justified to me than ever.

Again the ticker could shed light on this, the half formed unconscious dreams that motivate the people, their defences and how to foster and nurture awareness constructively. How to track and outwit the spin doctors and marketing men, how to wake people (including yourself) up, if possible. My nephew of 20, who turned me onto tvtropes, seemed an example too. His interests show an eclectic mixture of styles and influences which he does not make sham attempts to harmonise into a believable simple image, but instead simply lets be …in full embrace of implied paradox. And there it was – “don’t stereotype yourself”. Trying to act and be “of a piece” is directly playing into the hands of the PR manipulators. Gatherers of psychological information about the public rub their hands when their tool the “demographic” makes everyone fit neatly into a stereotyped social box. This damages society’s knowledge by oversimplifying and also directly erodes the wonderful diversity and living unpredictability of the human race. Live out of the mould and you claim your power… after all Knowledge is power and don’t we the wired have the advantage?

Gurdjieff wanted us to discover our real individuality, and now we can politicise this because the powers that be want us to remain asleep, like some of the people I have mentioned here managed to work out.

Already fanatically averse to watching broadcast TV, I now decided to avoid radio also. I weaned myself off the BBC World Service and dealt with comments by friends and family that I “wouldn’t know what was going on”. Instead I found it a pleasant way to subvert the consensus trance and deconstruct the propaganda that seemed to keep us all asleep. I showed more patience than the usual consumer of edutainment soundbites must have, and drilled into the real machinery of our oppression. Tvtropes already was showing me much, and since it also draws upon computer and role playing games like D&D, I sensed how powerful the communal “Jungian active imagination” process of these games can also be to let people get good at analysing public myths. Stories bear precious psychological veins of ore, and role players seem to be learning to manipulate and reshape story itself for fun and challenge. How much more armed the younger generation of geeks must be becoming to protect themselves from lies and fabrication that Bernays’ followers wish to propagate. If they play as many strategy games as my nephew they’d get pretty good at organising armed revolution too btw. Just as players of Sim City would know more than the jocks at the local council about urban planning, right? Modern writers who follow the path of tvtropes will be similarly wiser about cliche and derivative work, and I look forward to seeing what they come up with, in the light of the new knowledge.

It all tied itself into a glorious tangle and, with the poetic license of slight delirium, I began to see my sentiment ticker as a force for liberation. I might even be in a position of unique central opportunity like Paul Atreides, I thought in a flight of fancy. Eventually the fever subsided and I calmed down. But I am still excited about my ticker and hope it becomes more widely known.

Of course the slowcoach corporate/political lobby who sniff at my references to dubious authors and influences far too risky for their closets will discount this… but hey: that’s all part of the sentiment ticker conspiracy !

Wanting something to listen to I decided to use audiobooks, mostly torrented because I am a cheapskate and avowed supporter of the Pirate Party. The audio version of “Dune” was superbly done. (google to find) John Shirley’s “Demons” was good too. After a while though I became frustrated that there aren’t that many audiobooks that have actually been produced. Most of these are a bit too popularist for my taste so I began to feel audio-impoverished.

Then I realised that there must be ways of using text to speech for a solution. The small Linux utility I found for this purpose was “espeak”. It should come pre installed with many distros, certainly with Ubuntu.

The program takes a plain text file as input and renders it into a WAV with a few options needing to be specified. I used:

espeak -s 120 --split="60" -f some_book.txt -w some_book.wav

the -s 120 switch specifies the speed in words per minute. -split produces a series of output files of length 60 minutes. -f shows the input text and -w the name of the file(s) to output.

to convert to mp3 I used lame, doing a

sudo apt-get install lame

first and then

lame -f some_book.wav some_book.mp3

Theres also the gui based “sound converter” I noticed that this is much slower than lame.

the -f switch specifies lower audio quality which I decided upon based on the assumption that the speech produced by espeak was already quite low-fi and therefore I might as well save time by not wasting space on placing a low-fi voice into high resolution.

Now I have Stephen Hawking reading my bedtime stories Any book in the world that I can obtain as text may now be an instant audiobook, for using while out and about or at home. My little mp3 player has an external speaker which I use when at home, and headphones for runs, walks, shopping etc.

Many people might imagine that a computer voice is not that good to listen to, but now I have to say I disagree. Once you are used to it, and the occasional mispronunciation, it is perfectly acceptable. You learn to tune out noise and non-human inflection and eventually it is no worse than a human reader. After all when we read off paper the words are what we focus on not the typeface etc. My computer voice is just a neutral way of getting words as text into words as audio. I am loving it.

I have the ocean of text available to me to listen in a more leisurely and relaxed way than poring over screens or paper. I’ve already copy-pasted stuff like blog posts and wikipedia, too. I tend to amass about a few hundred KB of text which I put into one text file. as a guideline using my methods a 500KB text file will result in about 12 hours of mp3 audio.

It occurred to me that this should be built into ipods. I know that Kindle has a blind-reader, which has been said to be fairly bad. I’d love an ipod which I could upload a text file to and it would read to me. Should be a definite feature in my view in the future. Espeak for Linux is not a very advanced text to speech anyway. I have heard that the apple one is better, and various windows ones too. I suppose that to be good a reader needs to have a lot of pronunciation rules developed by linguists and perhaps even some semantic parsing to allow for as stab at natural inflection based on meaning, as well as based on pronunciation rules. It would be a good challenge for code. The way different phonemes elide together needs to be coded and perhaps with some training on large audio data it could be done a lot better than espeak. Linux needs a really good app to do this so it can compete with win and mac, in my view. A note was made in an ubuntu suggested features forum to this extent. Let’s go for it people.

The sentiment ticker is a small widget that sits on your computer’s desktop much like the conventional stock ticker. This widget connects to a central server cluster run by the provider. It provides a graphical/numerical display of various indices of market, political and consumer mood and feeling. The central servers poll blogs, tweets, journalistic articles and press releases constantly and perform sentiment analysis on this online content. Sentiment mining is looking for clusters of mood and emotion words in certain contexts. this has been shown to be useful, most notably recently when the Arab Spring was “predicted” (in hindsight) by analysing the sentiments of millions of bloggers, tweeters etc. a statistically significant spike was found in middle east mood just before the spring started.

Background

A stock ticker is a small program that stays open on your screen with real-time updated information on stock prices, and states of the various markets. It usually shows a graph or two and required price information you have set it for. All traders use them perennially.

Data mining is any process whereby useful information is gleaned by a program from large collections of data, in this case the textual content of the web. Sentiment mining is a particular kind of data mining where psychological and linguistic techniques are used to guess about human emotion from text on the web. This can be done to find things like “consumer confidence”, “willingness to spend”, “confidence in a certain stock”, “positivity about the future”.

So the ticker is the program that displays useful and concise information, and the mining is the means whereby this information is obtained. One part of the system mines perpetually and sends the data gained to each ticker belonging to every user whereby it is displayed for the user to take advantage of.

Leading Hedge Funds are using sentiment mining already. The ticker would be the democratisation of this process because it would allow sentiment mined knowledge to be available for a price to any investor who wanted, not just the pioneers and the elite rich enough to pay for a bespoke service.

Elaboration

We know that constant updates on actual market price levels is now seen as essential for financial decision makers. The sentiment ticker will enhance anyone in the industries knowledge of what is happening in the world, in a given country, or in a given industry. Gauging the psychological element in financial processes has always been something leading analysts do, but with the sentiment ticker a valuable extra insight into processes will be available. The data will be available to slice in a number of ways, by region, country, chosen time period or even down to an individual company. Knowing the world’s mood will become an essential metric for so many areas. Macroeconomics will also benefit from the ticker, enabling investors and their associates to be that much more educated about what is buzzing in the “planetary mind”.

The central sentiment server will be based on a set of Natural Language Processing algorithms, most of which are quite simple because they involve measuring the frequencies of emotion-indicator keywords. The implementers will expand and refine the system as the markets build the new service into their operations. A survey of human emotional modes will lead to isolating keywords and key phrases at first. Much of this work has already been started, but the process will no doubt give rise to new insights which will be added as the knowledge base of the ticker service provider develops.

The programming work can be done using mainly open source tools such as Python and the NLTK, which form the leading platform for NLP work. It is recommended that the service be Windows based until the means to hire Linux and Mac developers emerges. The costs of renting servers to form the platform will be low and services like Amazon’s EC2 are cheap and flexible enough to work well with the problem. A spider will constantly collect text from the web and the language processing will run concurrently so that the ticker can update on short intervals. Individual blocks of text will be ranked for importance using a spread of factors such as company position of the author, influence of them and their employer, location etc. The sentiment indices will be calculated using a weighting system that takes advantage of the rank given to the text.

The widget will be a small, attractively designed desktop window that stays active in the background when other work is being done. There may at first be a number of packages offered on a subscription basis perhaps starting with a bare-bones free service and building up to a platinum service with full functionality.

Spoofing

It is anticipated that many people will find it in their interests to try and manipulate the output of sentiment tickers by spoofing. This can be as simple as getting Facebook friends to say they believe a given stock will rise, or as complex as hiring armies of content authors to blog for you. This may even birth a whole industry in the same way as web content farms try to spoof Google’s keyword mechanisms to earn pay-per-click money. This will likely necessitate an arms race between sentiment developers and spammers, the exact unfolding of which will need to be observed and adapted to. This possibility does not invalidate the usefulness of the device, though, in the same way that keyword search on the web still stands in the face of spammers.

Development Issues

Initially it would seem sensible to let Google do a lot of the work for you. Page Rank, Google’s index of the influence of a web page will be a useful way to weight the importance of a sentiment source. Developers will start with a handful of stocks and get recent mentions of them from automatically querying google news, then try to make the system “read” the content and develop a means for assessing confidence in the given stock from the text. Many are quite used to stripping data from unstructured text, and at first just try to do it with keywords. the ambiguity factor will come into it a lot though, eg:

“… investor shows high confidence in Volkswagen…” : positive
“…investors are wrong to place high confidence in Volkswagen…” : negative
“… is an open question whether to place high confidence in Volkswagen…” : neutral

these three snippets might look the same to a system that couldn’t make very good judgements about context. so a way of discerning opinions has to be quite subtle, more so than just reading lots of text and trying to judge based on the occurrence of phrases. If you develop a way to make a good-enough system by testing on, say, 5 stocks, then scaling up to hundreds would not be substantially harder.

It is about looking for patterns in language so developers will need to collect loads and loads of these patterns, with an interpretation for each. Existing mining techniques are developing to resolve this kind of linguistic problem and much progress. When I was making the Essigned skills search we found that automation wasn’t very reliable because our system filled itself up with irrelevant words. so we hacked thousands of words by hand which was punishing work. Much of this has already been done in the Sentiment Mining community.

Language is very subtle, but with big data even getting a “buy” instruction right 51% of the time gives a 1 % advantage over random choice and might even get you a steady profit… This is what the pioneering hedge funds are already discovering.

The best way to proceed is to imagine that you had to solve this problem by hand, so you then read lots of financial news and observe how you are interpreting the text to make judgements. Then, when you see patterns emerge in what you do, develop a heuristic and then turn it into code and let the computer do the work for you. This is how sentiment mining is already moving forward.

The Sentiment Ticker – The Visionary Bit

The markets are a Chaotic Phenomenon (yes, caps). In the early days we knew that because of the Butterfly Effect there was a limit to the accuracy of our predictions. What an individual CEO ate for breakfast might critically alter future outcomes. Now we can move the fine granularity of our perceptions down one stage further because there is more data to trawl. The causal nexus of mass world opinion is a door that is starting to open with sentiment mining. The more honest data that people pour into the vastness of the web, the more of a handle we have on generating good predictions. Yes, the Butterfly Effect still holds, but there has been progress and refinement.

It is in the interests of the burgeoning global mind that people share personal data online. As an aside the online privacy, and data misuse issues still remain, of course. And there has never been a more clear need for ethical practises, fair government and responsible use of data than now.

People still watch conventional media news, like ITN News or BBC, because they want to drill down deeper into world situations for planning and foresight. A service that provides a real time breakdown of sentiment across the planet is a global evolution of the very concept of news, taking us one stage nearer to the retreating goal of perfect information. To be truly “in tune” or to have “one’s ear to the ground” investors need to surf the rising tsunami of information with better tools and finer analysis. Competitiveness accelerates too with greater and more timely information The sentiment ticker provides this because it is a microscope into the previously invisible but nonetheless critical machinations and complex flows of world opinion, and human sentiment.

Once one has “got” the concept nothing less will satisfy for the data omnivore. People around the world are desperate for any opportunity to scope deeper into the maelstrom of what we are doing as a species.

Markets will accelerate and diversify. Customer relations, sales, branding, indeed any enterprise dependent on rich data, will evolve irreversibly. Politics too will change as voter opinion becomes a 3D, graphically rich continuum into which we can gaze, and through which we can interactively test our ideas. These changes will rock our world-views, but it must also be stressed that they will demand greater responsibility, as increased knowledge always does.

The evolving economy places power in the hands of innovators and those who can exploit novelty, but the sentiment ticker will eventually democratise further too, because anyone with a network connection may achieve further knowledge for empowered decision making. Open information liberates intellectual capital into the ownership of humanity, and the species itself becomes wealthier. As Sentiment Developers we hope to facilitate this process while still profiting through a small appropriate return on the investment of our faith and vision in a revolutionary concept.

Parallel Python is a parallel computing module that works either for clusters or just multi core machines. After “import pp” you set up an instance of a parallel server and send computations to it using the methods of the server instance.

Both are very easy to use, so I thought I’d do some time testing and compare them.

My machine here at home is a 6 core AMD Phenom with 4GB of RAM (lucky me!). The cores you get from Picloud are 1.2 GHz Xeons. and each has 340MB RAM. Each job will get one of these processors, unless you set a high CPU option in which case each will get 2.5 and 850MB RAM. For these tests I used the high cpu option.

With parallel computing a lot of the art is decomposing a problem into slices that can run on one core each in parallel, usually with some merge code that combines the results from each job into a final answer. Both these systems work by passing the cores a function and a tuple of its inputs, the people at Picloud reckon that with this functional method you still have the power to do anything that an ordinary serial machine can do, and probably faster.

I chose a very simple calculation to perform – to find the sum of the squares of the first 10,000,000 integers. The range(1,6000000) was split into 10 sub-ranges, one and a bit for each core on my machine, or one for each 2.5 cores on Picloud. There were four tests:

• local machine using parallel python to send one job to each of my 6 cores
• local machine without any parallelisation, just Python computing a function
• Picloud with 6 separate parallel jobs each on high CPU, decomposition done on client
• Picloud with 6 separate parallel jobs each on high CPU using map function
• Picloud without parallelisation, just one high CPU instance for the function

We’ll get to the results in a sec.

The commands used were similar for both systems, and I hope the code is fairly readable.

Here is the code:

import pp
import time
import cloud

a=pp.Server()

def sq(x):
    """add all squares between x[0] and x[1]"""
    tot=0
    for z in range(x[0],x[1]+1):
        tot += z**2
    return tot

inputs = [[1,1000000],
          [1000001,2000000],
          [2000001,3000000],
          [3000001,4000000],
          [4000001,5000000],
          [5000001,6000000],
          [6000001,7000000],
          [7000001,8000000],
          [8000001,9000000],
          [9000001,10000000]]

# test 1
ti=time.time()
answer = sq((1,10000000))
print "local naive method",answer
print "that took",time.time()-ti,"seconds"

#test 2
ti=time.time()
jobs = []
for x in inputs:
    jobs.append(a.submit(sq,(x,)))
answer = 0
for z in jobs:
    answer += z()
print "locally parallelised",answer
print "that took",time.time()-ti,"seconds"

# test 3
ti=time.time()

job3 = cloud.call(sq,(1,10000000),_high_cpu=True)
answer = cloud.result(job3)
print "non-parallel on picloud",answer
print "that took",time.time()-ti,"seconds"

# test 4
ti=time.time()
jobs2 = []
for x in inputs:
    jobs2.append(cloud.call(sq,x,_high_cpu=True))
answer=0

for z in jobs2:
    answer += cloud.result(z)

print "parallel on picloud  using cloud.call()",answer
print "that took",time.time()-ti,"seconds"

# test 5
ti=time.time()
jobs2 = cloud.map(sq,inputs,_high_cpu=True)
answer = sum(cloud.result(jobs2))

print "parallel on picloud using cloud.map()",answer
print "that took",time.time()-ti,"seconds"


and here is the output.

The slowest was the local machine without parallel, so that is not so surprising. My own machine with parallel code was similar to Picloud with badly designed parallel code that didn’;t take advantage of the cloud.map() function. Picloud used like it should be with map() was the best…whahey!!

This is far from complete and I would be interested if anyone wants to duplicate or extend what I have suggested here. I might suggest that if the scale of the task was increased these rankings might change, perhaps reducing any effects of latency and other bottlenecks. Also I didn’t push the RAM envelope.

Another thing worth mentioning is that Python, being interpreted, will be a lot slower than a multiple processor system using c++. I love the language though so I will continue to use both these systems, and dream about doing something original with them. Picloud also includes a simulator so you can test and debug code on the local machine without paying until everything’s perfect.

Cheers Picloud and how about cancelling my 1 cent bill as a token of thanks for the free advert !! and special thanks to Aaron for his feedback.

Cheers also to Parallel Python… it gets the Pythonisms simplicity award.

The general impression I have received about this new idea of “The Big Society” is that no one is quite sure what it is supposed to mean. This maybe because of various errors made by our prime minister, in which he has failed to be a very good logician, I’ll try to communicate this idea.

In computer programming we have a very important concept called abstraction. Many people will be familiar with the contrasts between the particular and the general in practical daily life, so I’ll start with an analogy. In business a greengrocer might broaden her market share by starting to sell sweets as well as veg. She is making her operations more multi-purpose. If she then starts to do a stand of newspapers and other basic provisions then its easy to see that she is generalising away from the specific of veg into a range of different product categories, but ones that can still be run using her initial set of “selling retail goods” skills that she learned as a grocer.

This process is abstraction, in computers we often start with code that processes a unique set of one-off data. But then as the task matures we decide slowly to allow our program to work on different sets of similar data and then perhaps later different formats of more different data… and then later again design for different platforms or different user styles. That way systems are portable and more marketable too. By making our systems more general in what they can handle we save work by enabling the transfer of technologies from one domain to another. This is also the power of mathematics which is comfortable with facts like “3 apples plus 2 apples makes 5 apples” which is plainly amenable to the generalisation that “3 of anything plus 2 of the same thing = 5 of that thing”. indeed numbers are the very most general of all concepts.

In programming we also have another useful concept which i’d like to bring in to this discussion which is that of levels of description. This concept is related to abstraction. It is really about building blocks. So much of our work is about algorithms, and every algorithm has a description. these days the description however becomes amenable to a kind of shorthand. If I give you instructions (like an algorithm) to make me a cup of tea, I would usually leave out all the minor details like “first make sure the kettle is plugged in at the wall” since I would assume you didn’t need that kind of level of task verbosity. indeed you might find it condescending to receive instructions like this because we all assume an ability to solve such simple problems. (On another angle try getting a robot to do this though…)

The computational “jacob’s ladder” of machine code to low level code as the bottom rungs and higher levels of abstraction on the way up is a part of the very fundamentals of how to build complexity in our systems which most computer programmers know a good bit about. I sometimes wonder about trying to start a kind of open conspiracy where us geeks buried up to our elbows in code and data are finally recognised as the ones who should truly be in charge. To me a government is a complex system, and an inefficient bureaucracy perhaps something like a ‘legacy system’ written in some dinosaurish and inefficient langauge like COBOL. Mastering and organising the huge data that comes to you must be one of the parts of government, and correct planning based on this data sounds like a classic AI problem.

Much good human communication is very high level in terms of its abstractions but there are different styles for this and arguments we disagree with are often built upon philosophical “platforms” we dislike and thus may deem “buggy” like bad software, while aficionadoes will defend their own chosen philosophical technology. An individual’s political views are often like this, left and right wing each being like a different operating system, say windows, mac or linux. But the lesson of the sofware analogy is that these turf wars are more futile than trying to build common ground. Standards are everywhere and are usually seen as progress because they overcome the Tower of Babel effect and let us or our systems talk.

Maybe this is why some of us the find the pat language of gurus and charismatics so transparent, unless of course we are under their spell, it is because they utter such powerful sounding (but ultimately insubstantial) generalisations that masquerade as wisdom but being devoid of ay specific examples will soon fall down in the messiness of the real world. This could apply also to the nigh-universally applicable dicta of horoscopes but frankly I find it applies best right now to the words of David Cameron in his talk about the “big society.”

The big society sounds like a kind of ultimate high-level concept, a vision that might guide us. The lesson though from programming is that abstraction is always leaky and thus the grass roots of ordinary processes, think of a kind of societal stream of ones and zeros rolling dwn a matrix like greenscreen, is the only place to start. But this is something that Cameron doesn’t seem to be very good at. Indeed to insulate yourself from the rabble is a part of such a leader’s classy image, and is counterproductive in the same way as a programmer who doesn’t study their data enough.

With establishing new skills, especially in complex areas, finding your own comfortable learning path that leads in to the subject at hand is crucial. what is your personal way or style with which you solve this problem ? Some programmers take existing pre writtencode from libraries on trust, others favour a lower level and build up from the basics, while the others say they are wasting time reinventing the wheel. It’s an open debate but an interesting one because I have always felt that reinventing the wheel may mean you come up with a new original approach. If you want something done properly do it yourself is of course another dictum that casts different political styles into relief. Again we programmers have to deal with these issues daily. I’ve never set up Apache on Ubuntu before for example… so its a trip around the web techie forums for me later, having of course to try and sift the incorrect how-to data from that authored by the real experts, some of whom are modest and don’t blare their knowledge from the rooftops or make public party broadcasts !

Learning to encode so much data in textual or tagged forms has been great fun, with general skills and aha insights being acquired all the time to keep my sense of motivation and accomplishment sweet. I want to follow my open-source government ideas more because we all should have the freedom of information (or the wiki-leakability) at heart when designing the complex code of nation-management, and progress comes through sharing our code also.

I wonder what are the evolutionary algorithms that we and our brains use in all this? Or whether what the brain does is even algorithmic? In writing code we learn to manage big complexity consciously and intentionally but the magical subconscious processes in the brain are doing that all the time without us knowing. Indeed the fashionable current insight is that object recognition (where’s that kettle?) is a much greater and harder act of cognition than equation solving or chess… Our leaders are often past masters at subconsciously guiding themselves up the slippery pole too. The compuational wonder and richness of ordinary life acts against this though, maybe degrees in high falutin economics are less satisfactory as skills than hard nosed experience in running a greengrocer’s. Such intellectual elitism is partly marketing and branding these days, I’d trust the data.

Recognising glyphs or letters is at first a game of geometric pattern recognition to the learning child, but later we view it as linguistic. are programming languages like natural languages, yes there is still logic, semantics and syntax, many similarities. All of what we do can be termed “General Intelligence” which so many boffins around the world are dreaming of implementing in silico as “Artificial General Intelligence” right now.

A database saves memory by reading rows or columns one at a time, must be like this or none of our favoruite webistes would work. SQL – Structured Query Language is again a generalisation of how to phrase questions of any simple type, this lies beyond merely computer databases and is an abstraction of what all data retrieval processes share in common. Code clarifies thinking skills, UML – the Unified Modelling Language is also a generalisation of modelling any process / structure visually in pretty diagrams or textually in a language. The formalisation of good coding style or notation is a generalisation of any social knowledge sharing process where we need common protocols. What should we take on trust – lots usually! But we must also have a “smart trust module “that involves wisdom and skepticism too ! It’s only good news when that kicks in in trying to understand what our government is saying to us.

Was complexity built in the brain in the same way ? kind of “lower animals think in machine code”. and moreover in this kind of complexity you need to take many code objects on trust. the very use of natural language is a social act, our definitions and concepts are part of mental software libraries whose origins lie in past history.

When we or often our children ask some questions like “why do birds fly ?” one hallmark of a credible answer is often to use abstraction. (I had to use this in some Prolog code I recently wrote). In programming we might say that “animal” is a superclass of “bird”, an explanation comes: “because all animals have means of locomotion, and a bird does this by locomoting through the air, that’s why they have wings” instance -> class -> inherited property.

This explanation sounds good because it is bandying around an abstraction and referring to that instead of the actual case under question. How detailed after all do you want your answer to be ? Completeness wants evry relevant fact but so often the constraints of soundbiting do limit us. So many politicians do this generalisation trick, but its more psychology than effective explanation, because we think that the (often academic) categories of knowledge are set in stone and a matter of weighty authority.

Us coders are natural radicals though because we have to respect what works rather than what the theory says will work, and this often means drilling down to the detail of individual cases to get a real feel for the data. This is like grass roots politics too ! I just wish our leaders would do this drilling down more often and sample the raw data of ordinary peoples’ lives a little better before they think they have captured that great abstraction in the sky !


# 'philosopher.py' a basic chatbot learning system
# using pyswip which interfaces python and swi-prolog

from pyswip import *
from random import choice

# start a Prolog interpreter instance
p=Prolog()

# load the ontology specification
p.consult("c.pl")

# construct look up tables of common facts 
properties = list(p.query("property(X,Y)"))
categories = list(p.query("parent(X,Y)"))
cousins = list(p.query("cousin(X,Y)"))

# prepare for saving of new inferences
new_prolog_code = open('d.pl','w')
new_facts = []

def say_property_fact():
    """chooses a random fact about one of a category's
    properties and states it"""
    selected_fact=choice(properties)
    return selected_fact['Y']+'s '+selected_fact['X']

def say_category_fact():
    """chooses a random fact about category membership
    and states it"""
    selected_fact=choice(categories)
    return 'a '+selected_fact['Y']+' is a kind of '+selected_fact['X']

def ask_cousin_property():
    """hypothesises that cousins share properties
    and asks for confirmation, saving new code if yes"""
    cousin_pair = choice(cousins)
    props = list(p.query("property(X,"+cousin_pair['Y']+")"))
    possible_property  = choice(props)
    print 'does a '+cousin_pair['X']+' '+possible_property['X']+'?'
    a= raw_input('y or n')
    if a == 'y':
        new_facts.append('property('+possible_property['X']+','+cousin_pair['X']+').\n')
    else:
        return
        
# uncheck to say some facts as a test
#print say_property_fact() 
#print say_category_fact()

# does a training run and saves new inferences

response = ''

while response == '':
    ask_cousin_property()
    response = raw_input('quit ?')

new_set_of_facts = list(set(new_facts))

for x in new_set_of_facts:
    new_prolog_code.write(x)

new_prolog_code.close()

print "thank you, 'philosopher' has learned something new"



/* c.pl */
/* ONTOLOGY */

/* categories */

parent(thing,person).
parent(person,man).
parent(person,woman).
parent(thing,vehicle).
parent(vehicle,tractor).
parent(vehicle,car).
parent(tractor,massey).
parent(thing,animal).
parent(thing,plant).
parent(thing,place).
parent(plant,flower).
parent(plant,vegetable).
parent(plant,tree).
parent(animal,dog).
parent(animal,cat).
parent(place,house).

/* properties */

property(exist,thing).
property(live,person).
property(live,animal).
property(live,plant).
property(think,person).
property(move,vehicle).
property(plough,tractor).
property(grow,plant).
property(eat,animal).
property(move,man).
property(move,woman).

/* rules */

property(X,Y) :- parent(Q,Y),
                 property(X,Q).

child(X,Y) :- parent(Y,X).

sibling(X,Y) :- parent(Z,X),
                parent(Z,Y),
                X \== Y.

ancestor(X,Y) :- parent(X,Y).
ancestor(X,Y) :- parent(X,Z),
                 ancestor(Z,Y).

cousin(X,Y) :- parent(P,X),
               parent(Q,Y),
               parent(R,Q),
               parent(R,P),
               P \== Q.


# pythonic ultra simple encryption

alphabet = 'abcdefghijklmnopqrstuvwxyz .,;:?!'
coder = dict(zip(alphabet,alphabet[::-1]))

def scramble(message):
    return ''.join([coder[x] for x in message])

text = 'hello, this is my simple text encryption program.'

print "message is:           ",text
c = scramble(text)
print "coded message is:     ",c
print "decoded message is:   ",scramble(c)


and the output looks like this, just to make sure it works

>>>
message is:        hello, this is my simple text encryption program.
coded message is:  y,uurecmyxncxncthcnxtqu,cm,imc,s ohqmxrscqorzo!tf
decoded message is:hello, this is my simple text encryption program.
>>>

I wondered if it could be made any shorter, and I did away with using dictionaries, which trimmed it down.


# another way to do a cipher
# less pythonic but shorter

alphabet='abcdefghijklmnopqrstuvwxyz.,; ?!'

def scramble(message):
    a=''
    for x in message:
        a+=alphabet[::-1][alphabet.index(x)]
    return a


finally I used list comprehensions instead of a for loop and that shortened it even further.


# another way to do a cipher
# uses list comprehensions
# shortest one

alphabet='abcdefghijklmnopqrstuvwxyz.,; ?!'

def scramble(message):
    return ''.join([alphabet[::-1][alphabet.index(x)] for x in message])


this one is thanks to Mark Dufour and uses a method from the string module:


import string
alphabet='abcdefghijklmnopqrstuvwxyz.,; ?!'

def scramble(message):
    return message.translate(string.maketrans(alphabet, alphabet[::-1]))


Which one do you prefer ? Can anyone do better ? Post your results please.


def fac(a,b):
    for x in range(2,int(a**0.5+1)):
        if a%x==0:
            b.append(x)
            a/=x
            return fac(a,b)
    return b

print fac(1260,[])

>>> 
[2, 2, 3, 3, 5]


…I’ll tell you : Because the sheer ease of copying and releasing computer based data has become such that this free sharing of it using IT can only get ever more intense from now on. In the old days the image was of a raincoated and depressed looking guy whose upper lip was sweating gently as he carried his attache case across some border. a sadly unreliable and rather dangerous way compared to merely adding an email attachment ! Now we know it is an angry co-worker who simply copies onto a stick, ftps it, or sends a so-innocent looking email. And the leak is done ! Nearer Singularity I believe secrecy will become rare indeed for governments and Corporations particularly. The same increasing computerisation that is taking us towards what might become a singularity is making itself felt right now in a power shift between monolithic slow moving organisations and fast and light small groups with tech savvy and ideals enough to force restructurings in the world order.

I see wikileaks as an open source process. We have open sourced and de obfuscated the code of what is a huge data-reserve of some very deprecated, very legacy systems. Now all the open source journalism and political hackers can take their turns and try and improve this code, and let’s hope that will carry some of those oft repeated benefits that follow from making the analogy with software.

Of course, though, one can get carried away by mainlining Twitter and build a passionate hope in the one true revolution, but as long as police and military have all the guns what can we do except boot up and keep imagining we are freer, while outside our doors the cop cars whine and the sheepwalkers still shamble.

It has helped my understanding though to start looking out for concrete stages and milestones as connectivity revolutionises everything and Assange’s organisation is a significant phenomena. Naysayers and those who sneer might take notice of the obvious fact that even if every wikileaker was assassinated hundreds more would spring up from among the netizens of the world. The internet itself cannot be taken down either of course, short of a totalitarian dictatorship. this is what some of these types may want. That’s why the time is now.

Sure, some secrecy is still needed in a complex world. Hackers or moles who released DIY Bio/Nano threats are a cliche by now, but this cablegate is of a different flavour. it shows again and again the contempt and arrogance of those in power, as they present to us their brave shining heroism in pursuing all the goals outlined in their propaganda, creating peace, giving jobs to the poor, while the wiki-maginfying glass shows the small print of manipulation and plotting of those ever in quest of more wealth and control of the masses.

Twiter has been a great tool in this. the last time we all got so excited was in the green uprising in iran, where twitter came to life so beautifully. Now we have been shown that a government with that winning confident smile and perfect teeth, the US, has pooped in its own front yard so many times its become a health hazard. A trumped up rape charge is such a seedy and cack handed weapon to throw against Julian, especially since it must be obvious that wikileaks would continue to roll even if he was taken in to custody, thanks to the several hundred mirror sites that decent “netizens” (who may not even want to be “citizens” in the original definition anymore) and the barrage of VPSes now filled with TB of leaked data around the world. So one alleged rapist is more important than all those on the swedish sex offender list who are murderer rapists then ? I knew the tide was turning when I learned of this. We have to turn it back with all the IT tools at our disposal. The Western governments in this have fundamentally lost their credibility now among anyone who groks.

Amazon and paypal have joined the losing side, and I will try and dig out some figures about how many customers, and ultimately bucks (hey who cares about the people anyway, right Amazon?) that these nigh-monopolies have lost. Facebook is getting more than 100 people a minute supporting wikileaks in their page, and twitter seems to be getting about 200 #wikileaks tweets a minute. I will also try and graph some of this in a later post.

It’s like the prisoners dilemma, if only 10 people on the planet get busted for supporting them then those poor brave souls will be heavily punished. But if millions join in a domino tipple of small scale acts of single-click bravery they can do nothing. I just hope the complacent ones busily watching talent shows, reality tv or the weather will appreciate the changes that these brave people they think are so weird are facilitating. Trust in the power of the mass, and allow goodwill to trump the fear. We must all become involved. Monolithic organisations can be out-thought by small and dynamic ones who leverage crowdsourcing. And “Net-Guerilla” tactics have their place too, with no weapon needed except a smartphone. Lets crowdsource a new kind of world !

In the words of Jim Morrisson “They got the guns but we’ve got the numbers”. I’d like to see some DDOS attacks on anti wikileaks organisations. I thought of one on interpol, but then (please note) I decided that I must not do anything illegal. Now of course would I do that ? No, although I do admit it might be on my to-do list once there are so many people refusing the kool-aid that we can outvote the politicians, authoritarians and thuggish bully boys who make these laws.

Singularity means accelerated change. That applies to all sectors and groups, hey, the internet has already revolutionised the sale of patchwork quilts, custom mouse mats, personalised birthday cakes and a million kinds of trivia. Now get ready for some other kinds of more profound change ! It seems so simple to me that in a democracy the people must know. And those people are mostly nice ones who don’t want war, murder and torture. Pervasive computation is spreading like lichen on a rock into all sorts of corners, and the leaks are a direct byproduct of the planet wide connectivity that will be leading us toward singularity.

If you don’t feel up to mirroring wikileaks on one of your servers, then at least download the torrent of the ‘insurance policy’ which is available at https://thepiratebay.org/torrent/5723136/WikiLeaks_insurance and seed it. Windows users google utorrent, linux look for transmission.

If you are a military man then bet on the winners, share everything you’ve got, if a policeman do the same. At first the brave souls will do it, then later there’ll be so many of you that you can’t be nicked. You don’t have to be high-up for it to matter because we can handle the data growing hugely, that’s what all us boffins do with our time anyway. Techno democracy will keep debugging the proprietary code of deprecated and legacy political systems until open source government starts to get trendy. Early adopters will move to transient city states at first, then the mainstream will have to listen, and more effective, fair and ethically acceptable political code will get written. It will belong to everyone like the US Constitution did in Asimov’s “The Stars Like Dust.”

Open Sourcing What You Know about our Governments Is A Gift To Humanity and Posterity

Hell even Thanet Council, where I live in the quiet south east of merry England probably has a few nasty coffins in their cupboards. Let’s go for it and smooth out all of the crinkles in our planetary political system so we can get on with the real fun of becoming a type 1 civilisation that doesn’t destroy its home planet, and gets past these last few extinction-level dangers that this historical period offers us.

The Facebook group will be at 1,000,000 supporters by tomorrow or the next day. There are 600,000,000 people on that whole site with a common culture that already makes them a little like a large country, call it “Facebookia” if you like. If some truly representative and politically inclined movement emerged out of facebook then why shouldn’t their citizens decide their own rules and policies. This is a whisper of the political change that must be on the cards if the Singularity really is near. What if wikileaks got 10,000,000 supporters ? maybe we should demand our own Zion somewhere in the world and start afresh there, a bit of land and place to call home. I’d be up for it personally, I’m sick of feeling like a prole straight out of Orwell’s 1984 in my own country, thank you.

I came across a quote by Darwin the other day in a quotation book where he spoke of facts that didn’t fit with his ontology of causes in nature as being the ones that escaped his memory most easily.

I had,… during many years, followed a golden rule, namely, that whenever a published fact, a new observation or thought came across me, which was opposed by my general results, to make a memorandum of it without fail and at once; for I had found by experience that such facts and thoughts were far more apt to escape from memory than favourable ones”

This gave rise to a strange association, since it made me remember various experiences I have had in my life which could loosely be called paranormal. This post is just a rap about the thoughts that quote gave me. I found had done exactly what Darwin did and conveniently ruled these inexplicable experiences out of my mind, perhaps through believing I was in the grip of madness. One time I played with a Ouija board with some other students at university and the glass emitted a loud popping sound and jumped into the air. Another time I saw a Buddhist monk standing in the street in Camden Town and suddenly seemed to perceive that his mind was linked to a beautiful rippling pool of energy from some kind of higher sphere that he had somehow channelled down into himself.

Aged 19, at the Sri Rangam Temple in Trichy, India, I sensed a powerful ‘field’ of energy. Also another time in my mid twenties I entered Canterbury Cathedral and experienced a similar kind of rippling energy in the air inside the building. I would think twice before mentioning these experiences in the company of hard scientists because I find it a slightly unpleasant feeling to be thought a little mad !

I now feel it is as much a wrong turning to become a “scientific materialism fundamentalist”, as to be any other kind of fundamentalist. We just end up full of hot air when we reach too far into unsupportable and over-bold assertions about the nature of the whole universe based on our tiny and highly provincial habitation of a very small corner of it. These thoughts mingled in my mind with those from another book: “Impossibility” by John D Barrow. They formed a kind of Lovecraftian vision of a strange and ominous universe full of insoluble mysteries. Barrow points out that the insoluble problems of reality outnumber the soluble ones and presumably implies that any general intelligence with less that infinite IQ would find this. Some problems are algorithmically intractable, some cannot even be framed or represented. I don’t even know whether the quantifier ‘some’ can describe an infinite set in that last sentence either ! Is mathematics infinite or will we one day come to its end ? I doubt it, after all it embodies concepts of that very infinity within its own esoteric scribbles.

The sceptical elimination of some paranormal phenomena will clearly work for a lot of cases but I think scientists can become arrogant about eliminating all mysteries when we are indeed sitting on top of, or thinking with, one of the truly great unsolved mysteries -that of the grey matter that is itself reading what I have written here. And one catches a sense that any explanation that was ever offered for consciousness just couldn’t quite sum it up. It was the physicist David Bohm who suggested that the brain, in neural microtubules, might somehow leverage quantum mechanical effects in the generation of consciousness. Recently other voices have joined in, with some thoughts now at large that photosynthesis also leverages quantum phenomena. I feel I am sinning somehow if I allow a crude blanket statement that no anomalous phenomena exist in space-time, the great assumption of all science is that the sun will rise tomorrow -i.e. that processes are regular. But this regularity is an axiom which we take on a kind of faith.

Weirdness may well exist out there, and it might be staring back at us too! What is the personified universe saying to me ? I seem to have to reach beyond my own collection of habitual thoughts bequeathed by the past into a space that is utterly present tense, and inherently unique. I am the anomaly.

After musing for a while again I now have a further statement I wish to make about this:

it is actually evil to fake scientific results

You are emitting a highly condensed and dangerous kind of lie if you do this. Scientists have to have a bold, confident and can-do attitude, and believe in their power to solve problems. It is a psychological error, though, to overstate this and be too sure about things, like some young quantitative Mohammed Ali. The faith that the universe never contains anomalies, and that our limited conceptual architecture has seen right to the limits of possibility in this, is indeed a kind of fundamentalism. It is like someone who has just embarked on the excitement and thrill of reading and thinking, but who resolutely stays with all the conclusions of the very first few books they ever read.

I find some kind of idea that the universe has a mirror aspect – it reflects back our own prejudices onto us, and we assume that is all there is. Maybe the eastern philosophical traditions view this kind of idea as less heretical. I’d love to keep researching all this, and will keep a background module running on it in my small way, but I am a code addict too and will probably shelve the largest questions again in favour of that intricate, almost predestined dance of logic which does answer so many questions for me.

Darwin also said that adaptability is more powerful than intelligence. If we can’t adapt to our own perceptions we are not truly scientific. Yes we must also question them, and knock them down if we can, but I haven’t been able to knock down those experiences. Denying facts and forming models that rule out psi because it’s a mystery is this kind of ‘cooking the books’ which I allege above is a such a great crime. When I had these inexplicable magical experiences I ran off and developed a load of ill-formed ideas about them, became confused and then assumed I was crazy, but it is only now with a little more maturity that I feel I can absorb them properly. Not that absorbing is comprehending, mind you.

Some of you will think I am mad for this, but I have another benefit, which is to have common ground with most of the non-scientist human race who aren’t quite as sure as that minority of material fundamentalists. I have more I can talk about with people from other traditions, which feels great. There is a spiritual fellowship available as well as a scientific one. I don’t need to face those arrogant and lonely arguments defending sceptical materialism I used to have as an undergraduate.

It is also an axiom of science that, to count, experiments must be reproducible. How could you ever demonstrate that this regularity is a sound assumption ? A naive view takes the assumption of regularity and then extends it too far. But without it we can’t even start, except to say “there are sensations” and leave it at that. The singularity that we find at the beginning of the universe may have been a unique event, and maybe there are indeed other such events which defy the models we usually build. And maybe that’s all we can ever find out about them, except when magical intuition grants us brief glimpses.

After all the only thing I truly know is that I know nothing, and frankly I’m not even sure of that !