Sentiment Ticker – The Rambling Bit

I have recently entertained a whole series of connected thoughts inspired by the idea of the sentiment ticker. To summarise this is a device that uses keyword analysis to read the emotions of people who write content online. Because there’s so much online it may be possible to gauge the moods and feelings of whole groups, nations or the world. The technique is in its infancy but leading hedge funds already use it to try to predict stock prices, so where the money goes the rest are to follow, perhaps.

Coincidentally at the same time I started listening to an audiobook of Isaac Asimov’s Foundation. I soon observed there was a synchronicity. Author’s note: I don’t believe such coincidences are magical or psychic in origin, as Jung thought, but they are still an interesting mental event and even literary tool.

The sentiment ticker and opinion mining in general would definitely qualify as embryonic Asimovian Psychohistorical tools. Governments and organisations could use them for prediction and other research. A way to formalise events in current affairs and link them with observed sentiment trends would be very powerful. If connections could be found then you may have the beginning of actual equations.

The other subject which is connected is the real, as opposed to SF, subject of Psychohistory. This is a discipline that uses psychotherapy techniques to try and understand the motivation of nations, groups and particularly political leaders. It is fascinating and its primary conclusion is that child rearing is critical for the future of the species. This is because psychological damage to children propagates into damaged maladapted adults who act neurotically and create conflict and perpetuation of their damage in the world. The sentiment mining approach could be a very powerful tool for a modern de Mausian psychistorian because huge volumes of textual data could be sifted for emotion words and phrases which correspond to psychohistorical patterns. Incidentally the baroque violence and depraved imagery of newspaper political cartoons are currently one of the richest veins for sentiment mining by psychohistorians, but an image processor would currently be very hard to make that could do this job.

Text to Speech – how to make free audiobooks

I have recently tried to break my habit of listening to the BBC World Service at bedtime. This is complex and for a number of reasons. One: it was an addiction which rendered me less flexible in life because when no radio is available you get withdrawals. Two: I am currently concerned that most broadcast media is just propaganda. Three: the particular tone of voice and inflection that newsreaders use became annoying. Four: many of them, including newsreaders, are cokeheads and hypocrites who care as much about the problems of humanity as their idol Attila the Hun, who they are usually well to the right of on the real number line. I learned this after spending time associating with some London based journalists through a friend.

Wanting something to listen to I decided to use audiobooks, mostly torrented because I am a cheapskate and avowed supporter of the Pirate Party. The audio version of “Dune” was superbly done. (google to find) John Shirley’s “Demons” was good too. After a while though I became frustrated that there aren’t that many audiobooks that have actually been produced. Most of these are a bit too popularist for my taste so I began to feel audio-impoverished.

Then I realised that there must be ways of using text to speech for a solution. The small Linux utility I found for this purpose was “espeak”. It should come pre installed with many distros, certainly with Ubuntu.

New Concept – The Sentiment Ticker

The Concept

The sentiment ticker is a small widget that sits on your computer’s desktop much like the conventional stock ticker. This widget connects to a central server cluster run by the provider. It provides a graphical/numerical display of various indices of market, political and consumer mood and feeling. The central servers poll blogs, tweets, journalistic articles and press releases constantly and perform sentiment analysis on this online content. Sentiment mining is looking for clusters of mood and emotion words in certain contexts. this has been shown to be useful, most notably recently when the Arab Spring was “predicted” (in hindsight) by analysing the sentiments of millions of bloggers, tweeters etc. a statistically significant spike was found in middle east mood just before the spring started.

Background

A stock ticker is a small program that stays open on your screen with real-time updated information on stock prices, and states of the various markets. It usually shows a graph or two and required price information you have set it for. All traders use them perennially.

Comparing ParallelPython and Picloud

Picloud is a paid cloud computing platform using Python. After signing up you get two login keys and have to install a Python module on the local machine that you will use to send jobs to the cloud. You enter the keys in a config file and Python connects after it sees an “import cloud” line in your code. The fees run at 5 cents for a processor hour. Not a lot.

Parallel Python is a parallel computing module that works either for clusters or just multi core machines. After “import pp” you set up an instance of a parallel server and send computations to it using the methods of the server instance.

Both are very easy to use, so I thought I’d do some time testing and compare them.

Over-abstracting -> bullshit

Amidst the chaos of so much important activity in the world right now, particularly in the middle east, I sometimes find it hard to reliably concentrate on the immediate task. I am in the middle of working on a complex website, and earlier today had problems. But reflecting on these I was struck by a political analogy. Trying to start a web development project and having a problem setting up Apache is after all like trying to get to a business meeting but having to fix your car on the way – you get this unpleasant feeling of being sidetracked ! But sidetracks are always there in unstructured maelstrom of real life away from the keyboard, where bosses or pupils need attention, and communication is so critical but ever impossible to perfect.

a simple learning program using pyswip

I really enjoyed making this work. the “hypothesising that cousins might share properties” was my own idea.


# 'philosopher.py' a basic chatbot learning system
# using pyswip which interfaces python and swi-prolog

from pyswip import *
from random import choice

# start a Prolog interpreter instance
p=Prolog()

# load the ontology specification
p.consult("c.pl")

# construct look up tables of common facts 
properties = list(p.query("property(X,Y)"))
categories = list(p.query("parent(X,Y)"))
cousins = list(p.query("cousin(X,Y)"))

# prepare for saving of new inferences
new_prolog_code = open('d.pl','w')
new_facts = []

def say_property_fact():
    """chooses a random fact about one of a category's
    properties and states it"""
    selected_fact=choice(properties)
    return selected_fact['Y']+'s '+selected_fact['X']

def say_category_fact():
    """chooses a random fact about category membership
    and states it"""
    selected_fact=choice(categories)
    return 'a '+selected_fact['Y']+' is a kind of '+selected_fact['X']

def ask_cousin_property():
    """hypothesises that cousins share properties
    and asks for confirmation, saving new code if yes"""
    cousin_pair = choice(cousins)
    props = list(p.query("property(X,"+cousin_pair['Y']+")"))
    possible_property  = choice(props)
    print 'does a '+cousin_pair['X']+' '+possible_property['X']+'?'
    a= raw_input('y or n')
    if a == 'y':
        new_facts.append('property('+possible_property['X']+','+cousin_pair['X']+').\n')
    else:
        return
        
# uncheck to say some facts as a test
#print say_property_fact() 
#print say_category_fact()

# does a training run and saves new inferences

response = ''

while response == '':
    ask_cousin_property()
    response = raw_input('quit ?')

new_set_of_facts = list(set(new_facts))

for x in new_set_of_facts:
    new_prolog_code.write(x)

new_prolog_code.close()

print "thank you, 'philosopher' has learned something new"


minimal text encryption

I came up with an idea for a “toy” text encryption program, mainly focussing on how simple I could make it to see what python can do in quite short and readable code. The first one I tried was this:

# pythonic ultra simple encryption

alphabet = 'abcdefghijklmnopqrstuvwxyz .,;:?!'
coder = dict(zip(alphabet,alphabet[::-1]))

def scramble(message):
    return ''.join([coder[x] for x in message])

text = 'hello, this is my simple text encryption program.'

print "message is:           ",text
c = scramble(text)
print "coded message is:     ",c
print "decoded message is:   ",scramble(c)


Pythonic minimal factoriser

I like the succinctness of this bit of code, the recursive termination clause doesn’t use a conditional, and the output of the function contains the input number + the list in it until it terminates when it returns just the list. I challenge anyone with boldness and aplomb to come up with shorter code if they dare !!


def fac(a,b):
    for x in range(2,int(a**0.5+1)):
        if a%x==0:
            b.append(x)
            a/=x
            return fac(a,b)
    return b

print fac(1260,[])

>>> 
[2, 2, 3, 3, 5]


why #wikileaks is one symptom of an approaching singularity

wikileaks and the singularity ? but how ?

…I’ll tell you : Because the sheer ease of copying and releasing computer based data has become such that this free sharing of it using IT can only get ever more intense from now on. In the old days the image was of a raincoated and depressed looking guy whose upper lip was sweating gently as he carried his attache case across some border. a sadly unreliable and rather dangerous way compared to merely adding an email attachment ! Now we know it is an angry co-worker who simply copies onto a stick, ftps it, or sends a so-innocent looking email. And the leak is done ! Nearer Singularity I believe secrecy will become rare indeed for governments and Corporations particularly. The same increasing computerisation that is taking us towards what might become a singularity is making itself felt right now in a power shift between monolithic slow moving organisations and fast and light small groups with tech savvy and ideals enough to force restructurings in the world order.

What is your heresy ? Here’s mine…

This piece is partly in honour of the AI scientist Ben Goertzel who seems from his writings to have an active spiritual sense, and to at least credit the existence of what he calls ‘psi’. (He also has a very cool ‘stoner-geek’ manner when he talks which has prompted various witty comments…) I am in doubt about many things but reading and sensing Ben’s lofty grasp of math and code, I wavered in the old, and not unknown to me, direction of believing in a spiritual reality un-knowably beyond science once more. Yeah sure I should make my own decisions and thus should be able to work this out for myself, but no man is an island, and that’s why I am writing this for others (and thereby considering it more carefully for myself).

I came across a quote by Darwin the other day in a quotation book where he spoke of facts that didn’t fit with his ontology of causes in nature as being the ones that escaped his memory most easily.

I had,… during many years, followed a golden rule, namely, that whenever a published fact, a new observation or thought came across me, which was opposed by my general results, to make a memorandum of it without fail and at once; for I had found by experience that such facts and thoughts were far more apt to escape from memory than favourable ones”

This gave rise to a strange association, since it made me remember various experiences I have had in my life which could loosely be called paranormal. This post is just a rap about the thoughts that quote gave me. I found had done exactly what Darwin did and conveniently ruled these inexplicable experiences out of my mind, perhaps through believing I was in the grip of madness. One time I played with a Ouija board with some other students at university and the glass emitted a loud popping sound and jumped into the air. Another time I saw a Buddhist monk standing in the street in Camden Town and suddenly seemed to perceive that his mind was linked to a beautiful rippling pool of energy from some kind of higher sphere that he had somehow channelled down into himself.

Aged 19, at the Sri Rangam Temple in Trichy, India, I sensed a powerful ‘field’ of energy. Also another time in my mid twenties I entered Canterbury Cathedral and experienced a similar kind of rippling energy in the air inside the building. I would think twice before mentioning these experiences in the company of hard scientists because I find it a slightly unpleasant feeling to be thought a little mad !