Sitemap

So Long and Thanks for all the Micropredictions — Reflections of a Quant Content Creator

23 min readOct 4, 2023

--

Press enter or click to view image in full size
I’ve had fun being a grumpy old quant, but I’m taking a break from content creation
Press enter or click to view image in full size
From the infamous Prophet article
Unifying Machine Learning and traditional portfolio construction using Schur complements.
Press enter or click to view image in full size
From a talk I gave at R-Finance 2023, one of the best conferences I had the pleasure of attending in recent years. Packed with smart people with a genuine interest in building and bequeathing useful things. As a Python person, I would not have received an invitation were it not for this blog.

Manipulation of prediction market results can occur when individuals or groups strategically choose to provide false information, often for their personal gain. This can result in biased predictions that are not representative of the true CI.”

Among other minor accomplishments, the long-running micro-markets at Microprediction.Org and the admittedly arcane z-streams in particular represent a thorough, open and ongoing refutation of “buying the wings” in the microprediction domain. Apologies to anyone guided by Black Swan theory in a different direction, but that stuff is pretty weak elsewhere too. As an aside, I made this longish video many years ago for the almost empty intersection of people who (1) read The Black Swan cover to cover so will get my jokes despite (2) hated it with a passion of a thousand fires because of its stunning ignorance of the longshot effect (the well documented effect in betting markets where unlikely events are over-priced, not under-priced) and use of very cynical techniques. Those devices included the substitution of logic for folksy disparagement of people on the spectrum — and even an unsubstantiated “link” between “epistemological blindness” and ASD. It is very hard to believe that in the present slightly more enlightened day and age, certain passages of that book would not create an outrage (but HBR etc were just fine with it). On Taleb’s personal web site I am still referred to as a failed mathematician and, more offensively, an “idiot savant” in retaliation.
Press enter or click to view image in full size
Election analysis of betting market PredictIt versus Nate Silver’s 538 probabilities state-by-state in 2020. In order to maintain the position that model pipelines (e.g. the 538 model) are reasonable substitutes for markets as suppliers of probabilities, one is forced into a rather strange conclusion about a reverse longshot effect in election markets. In other words, either markets were superior to models in this instance, or Nate Silver and Nassim Taleb will have to agree with each other. I have too much respect for Silver’s empirical bent to assign any reasonable chance to the latter.

Micromanager: A reward-seeking program or application that autonomously enters, maintains and terminates where necessary economic relationships with suppliers of microprediction — typically algorithms, people, or other micro-managers — so as to improve its own ability to provide microprediction to an application, algorithm, person or other micro-manager upstream.

Press enter or click to view image in full size
I implemented pages 96–98 and it was good enough to beat the index, thus far, as well as provide distributional predictions in far less competitive settings such as electricity and wind prediction. If you are interested in helping an open-source project to get a fully vision off the ground, do reach out.

The unfortunate answer is I don’t know the ``right way’’, and neither does anyone else. There was a time when I hoped a universal, canonical middle-manager would spring to mind and with it, a blueprint for a prediction web. This fabulous insect would solve the multi-period, multi-bandit, multi-variable, multi-call-it-what-you-like machine learning generalized regression problem with costly inputs so convincingly that we would sit back on our chairs and, to paraphrase Gilbert and Sullivan, declare ``My goodness, that is the very model of a modern micro-manager!’’.

A web-scale collection of radically low-cost self-organizing supply chains for microprediction, used almost universally to meet real-time operational needs.

Press enter or click to view image in full size

Because we cannot know the best way to combine models, and we cannot know the best way to manage in automated fashion a collection of models, we are drawn to something like figure above which is intended to represent a tower of micromanagers all trying to out-predict someone and, to better their chances, enlisting the expertise of other micromanagers who have access to additional techniques and data.

--

--

Microprediction
Microprediction

Written by Microprediction

Chief Data Scientist, A Hedge Fund