So Long and Thanks for all the Micropredictions — Reflections of a Quant Content Creator

Microprediction
23 min readOct 4, 2023

In this medium good-bye, I offer some reflections on my experience creating quantitative content here for those considering the same. I’ve written forty or so articles and my content has sometimes revolved around ideas laid out in the microprediction book. More often though it has veered into territory such as optimization, portfolio construction, time-series, sports analytics and even disease modeling.

I’ve had fun being a grumpy old quant, but I’m taking a break from content creation

My stint as a content creator began a few years ago when I sat down with Intech’s head of marketing Andre Prawoto. He encouraged me to write some articles to help draw attention to the microprediction project, and thus began this blog. I tried to be opportunistic and, naturally, the articles centered on work I had to do anyway — preferably with some connection to topical items.

Of course, on top of the “real” work of building and maintaining a live probability exchange (and other unrelated demands of my day job) this attempt to drive people to an idea sometimes felt like hard work and, in keeping with the Pascal’s triangle theory of popularity (a self explanatory notion I hope) was sometimes less rewarding than it might be.

My first comment is that engagement might make you a worse person. For instance I began to notice that my facile Linked-In comments were more popular than my articles. And If people did progress to read articles I had laboured over, readership was inversely proportional to mathematical beauty, originality, and difficulty — at least to my eye. Example: the readership of this article might have been pretty small but it was the one that gave me the most pleasure.

I morphed into a bit of a data science jackass partly in response to this incentive and partly because, quite possibly, I am a partial jackass. My ratio of commentary to mathematical content increased. I’m sure it ate away at my intrinsic joy for the work in some small way.

For that behaviour I should apologize although I suspect I was at times merely speaking aloud some cynical things that many of you know to be true. I observe that by far my most read article in recent times was an essay on data science influencers titled So, You Recommended a Python Time-Series Package … Now What? This touched a nerve.

That’s a pretty negative micro-legacy though, isn’t it? And of course by a long stretch the most read article over my short tenure was also perceived as negative: an article I nearly didn’t write comprising a mostly visual inspection of the most popular Python time-series package: Is Facebook’s Prophet the Time-Series Messiah, or Just a Very Naughty Boy?

That note even drew a mea-culpa from the package author after hitting page one of hacker news. By the silly standards of our culture this was, therefore, a successful article — too bad readers were sent here instead of medium or I might have many more medium followers!

From the infamous Prophet article

Looking back on this experience its natural to ask if the effort was worth it. In my case it was, but not for the reasons I anticipated. Let’s dispense with one of them by asking whether I really have any “influence” in this matter or elsewhere. I absolutely did not.

  • Prophet is alive and well- see the stats.
  • The Bears are overthinking 4th down instead of reconsidering 1st— despite my plea.
  • I doubt many epidemiological experts are thinking about compartmental models as negative interest rate term structure models.
  • Even speciality low-code influencers haven’t noticed the connection to microprediction. Duh.

My jawboning was mostly futile. I’ll hold the microprediction reflections for a moment though.

On a positive note

I’m stepping back from medium (and probably Linked-In to a large degree) not because I’m despondent, or because I don’t enjoy sharing some ideas. I still believe there’s an important gap to be filled, especially for practitioners like myself who have limited incentive to play the publishing game and even less time.

As an example I received a lot of private feedback and material, non-trivial mathematical suggestions after I published the blog article about what I termed “Schur Complementary Portfolios”. This was something I developed that presented a mathematical unification of Hierarchical Risk Parity (HRP) and more traditional minimum variance optimization (the article is here). I’m very grateful for the direct messages that followed, and somewhat remorseful that I haven’t yet had time to chase down all the ideas I received.

In contrast I received exactly no useful feedback from the usual academic channels because a seemingly logical journal for such work refused to review it (as distinct from rejecting it) on the grounds it was “too mathematical”. The same journal had made a very big fuss about HRP in the first place as being radically different — so this position is, to me, somewhat logically problematic. But I respect their right to delineate their own boundaries … even if these boundaries encourage a higher number of simply connected regions than seems necessary!

Unifying Machine Learning and traditional portfolio construction using Schur complements.

A bigger man than myself would have taken the time to convert the work from one journal’s LaTeX template to another and resubmit it, and I surely will eventually, but in the meantime I started to wonder if there was any immediate point in doing so. The network of smart people Linked-In had given me access to was more valuable, and operated on a much lower latency.

This, I have come to believe, is extremely valuable. Indeed I have no regrets about putting effort into content creation for that reason alone, and someone considering sharing more of their work in an informal manner might weight this medium term outcome (oh dear bad pun). Working from home I also valued the personal connections. I started to genuinely care about turkey sightings; I appreciated obscure economic and cultural references supplied by those who helped me put thoughts in context; and I even reconnected with people I hadn’t talked to much in 30 years.

It takes a lot of work to get to borderline tolerable Linked-In feed, as I’m sure you know, but I don’t dismiss lightly the improved peripheral vision this can eventually engender. I received actionable intelligence from top developers running engineering groups and that will continue. None of this would have happened without some content creation, I suspect, which deepened my network. Only my inability to resist challenging poor quality content has worked against this feed improvement goal. (I deserve what I get, I suppose).

So why the good-bye?

Tomorrow my responsibilities change (I’m being coy). Perhaps, compliance permitting, I’ll still participate in the data science discussion on my Friday commute but I doubt I’ll be investing time in medium, or content generally. And really, I haven’t been able to do that for some time now, anyway.

The social dopamine thing does weird me out a bit too, so I figure I’d give that up along with caffeine (it is a fresh start for me). I don’t want it interrupting what “deep” time I will get, and I’d think that should appear on the scales too for anyone considering frequent content creation.

Overall though, I would not try to persuade anyone against putting content out via informal channels and nor have I found Linked-In or medium to be terribly toxic — quite the opposite for the most part. I’m sure there are some other considerations that apply to academics that don’t apply to myself, but I’ve found it to be a reasonable investment and the process has helped me stay organised to some extent too. It is my hope that more technical people do the same.

I’ve speculated about the possibility that these sometimes unpolished ideas might be automatically refined and curated by advanced LLMs of the future. Perhaps we are entering a new golden era where it really is ideas that matter, and not ceremony.

Microprediction

I realize that some of you might be wondering where this leaves the microprediction project, the vision in the book, or at least the content-driven marketing of the idea. The short answer is, don’t worry microprediction is too open and autonomous to fail. I’m merely talking about the content creation around it.

And microprediction will actually be receiving even more fresh air soon which I believe will be important to attracting developers (of course it has always been open source to begin with). I have been working to remove impediments to its growth, and remove things that may dissuade anyone from contributing to what should be a community garden. It needs people with strong intrinsic drive.

I can provide some more specific and, I hope candid, reflections on my efforts to propagate the vision and the extent to which this content creation helped. The core thesis — the one that this blog and my Linked-In participation sought to evangelize — is fairly straightforward. It stated that algorithm-friendly bespoke markets, in comparison to only the use of models, could be:

  1. Accurate, and
  2. Convenient

as alternatives or complements to the exclusive use of models in an operational production setting. My inclination is to view accuracy as a relatively uninteresting question and convenience as crucial (not to mention infinitely technically fascinating) which is why I stated in my book that advancing the project would lean more on code than prose.

It seems to be a priori obvious that models are vastly more convenient than markets, and that therefore this would be the more controversial part of the thesis, right? Well no. One of the more surprising retrospective observations from this social media adventure is that most people prefer to talk about whether markets are really better at predicting than your typical data scientist. It is a “big philosophical issue”, I suppose, although in my view a bit of a big dumb philosophical issue.

From a talk I gave at R-Finance 2023, one of the best conferences I had the pleasure of attending in recent years. Packed with smart people with a genuine interest in building and bequeathing useful things. As a Python person, I would not have received an invitation were it not for this blog.

1. Accuracy

Perhaps I am pig-headed but my interactions with the community have not materially shifted my opinion, insofar as I view the relative efficacy of markets as pretty obvious empirically, and also logically compelling in the large-n domain of rapidly repeated prediction.

Though my thesis does not rest on it, markets are sometimes quite compelling in longer horizons as well. It is my hope that my little M6 stunt, which I documented in the article “The Options Market Beat 97.6% of Participants in the M6 Financial Forecasting Contest” did indeed influence a small percentage of data scientists whose path was not yet a quanty one. On the other hand this also highlights one reason why a prediction web faces an uphill battle.

For in the M6 contest, which called upon participants to predict stock volatility, there was a ready source of stock market volatility to use — the options markets. This market was

  1. Obvious
  2. Efficient
  3. Pre-existing

And yet nobody used it! Now imaging that you wish to convince data scientists to use markets (in addition to models say, in their pipelines) in a context where the use is:

  1. Obscure
  2. Potentially perceived as inefficient due to lack of liquidity and worse …
  3. Not yet in existence so you need to create the market yourself

Then it will take time for all but a small number of super quick micro-grokers. Furthermore the wielding and creation of your own market might require understanding an arcane options-market-like-thing (a nearest the pin reward system for Monte Carlo paths) if that’s the only option currently implemented.

One can, perhaps with hindsight benefit, see why we should not expect a huge proportion of data scientists to immediately start weaving market-like things into their pipelines, even if in principle they could with a few lines of python.

This is, however, just a matter of scale and technology in my view and that’s why medium term I’m extremely bullish about the prospect of a microprediction network. The trend seems to be its friend and that includes advances in model deployment convenience. But perhaps the most intriguing development in recent times is the advance in LLM performance because of the impact this can have on:

  1. Algorithm navigation ability (language as API, matching tasks to methods, weakening of ontological needs etc)
  2. Instrumentation (of just about anything from what I termed “weak universal data sources in Chapter 2)
  3. Causality (using weak common sense to counter spurious correlations)
  4. Model search (and architecture search, and micro-manager search)

and so on. Those are some of the bigger updates I would make to the book.

Indeed if you view the microprediction goal as the liberation of algorithms and their empowerment: i.e. helping them seek and identify business value creation opportunities on their own with no or minimal human intervention, then it is pretty clear why a weak layer of generalized intelligence is extremely beneficial to that task. I don’t think we should trip ourselves up on terminology here, such as whether or not they “think” in any particular human way, as I noted in Remind Me Again Why Large Language Models Can’t Think.

I do not dismiss the more general challenges to market accuracy that arise outside of the microprediction domain and just to frame this, I’ll use Mario Brcic’s lead. He was kind enough to point me to a paper a while back called Challenges in Collective Intelligence (link) and even kinder to cite the Microprediction book and also my M6 experiment. He notes some topics: diversity, independence, decentralization, and aggregation. I don’t want this to be a very long digression so I’ll just discuss a couple of challenges mentioned therein.

Manipulation of prediction market results can occur when individuals or groups strategically choose to provide false information, often for their personal gain. This can result in biased predictions that are not representative of the true CI.”

Chapter 7 addresses this concern for point estimates, or at least introduces the topic, but as my current implementation is a continuous parimutuel for distribution predictions and not a collector of point estimates, there’s an old paper by Caltech Professor Colin Camerer titled Can Asset Markets be Manipulated? A Field Experiment with Racetrack Betting that seems more relevant. What Camerer did was place a large bet on a horse and then cancel most of the bet just before the jump. This causes the price of the horse on the totalizator to look worse than it will eventually be, thereby discouraging investment from others.

Camerer concluded that he could not successfully manipulate the tote but but his conclusion is wrong or at best incomplete. In a past life I had (with the help of someone later to be another Caltech-associated game-theoretic economist as it happens) executed a small modification of this strategy with much success. As a result of our actions in smaller markets the final show price would sometimes be better than the win price — an economic distortion so pronounced it was remarked on by a radio commentator.

However in the microprediction domain, any oft-repeat devious strategy such as this will be predicted by other participants so I tend to agree with that self-correction in markets is pretty strong. See also the paper by Buckly and O’Brien in the references.

As a further aside, price transparency which Camerer and I relied on for our chicanery isn’t strictly speaking necessary in a frequently repeated game, and knowing the investments by others isn’t a requirement to extract rent — though it surely helps. See my article The Lottery Paradox for further discussion including a surprising mathematical fact: a log-optimal investor knowing the true distribution doesn’t even need to know what the current prices are.

What about the microprediction experience specifically? Conscious of the strategy employed by Camerer and myself, I went to pains to counter any withdrawal-based manipulation by quarantining cancellation requests (predictions carry over at microprediction unless they are explicitly cancelled). Once the missile is fired (in the form of Monte Carlo predictions) it can’t be called back.

A more difficult dance arose due to the lack of staking, which I knew would be a challenge. (I’m sure I’ve ranted elsewhere about the inadvertent role of regulation in stifling innovation so won’t do so here). With helpful feedback from participants we tried different schemes and different memory to reward longitudinal and cross-sectional performance. I would not claim that every last participant agrees on the right balance here, but we did, and we do, manage to create equity volatility and electricity predictions that are really top notch, despite the game-theoretic challenges introduced by lack of staking.

There are other challenges to markets of course, or to collective mechanisms in general. Brcic mentions independence, fallacies and polarization in groups. Some collective intelligence gadgets certainly do lean heavily on independence (Galton’s Ox demonstration comes to mind). But markets don’t require independence of contributions — and nor does your typical market-inspired collective intelligence apparatus. Markets need a few smart cookies to fix the risk-neutral distribution, that’s all. It’s been nice to watch that play out, in real time, in a much cleaner fashion than we are accustomed to in options markets.

Another classic challenge to market accuracy is groupthink. But rapidly repeating market-like-things in the microprediction domain are less subject to this. Admittedly, we are yet to see a strong algorithmic cult there of any kind but I could invent the scenario, I suppose. Actually I did once create an algorithm that ran at Microprediction.Org called Taleb which dare I say it — held an extreme position: namely that there are obvious, unconditional, easily stated, continuing biases in probabilistic forecasts made by other people who — you know — are just so idiotic.

This Talebian algorithm attempted to profit from its philosophical perch by scattering more predictions into the tails. It hoped to win credits from other algorithms who, readers of The Black Swan might presume, are trapped in Mediocristan (and possibly authored by epistemologically challenged bell-curve loving fools schooled by Paul Samuelson). It did not.

Among other minor accomplishments, the long-running micro-markets at Microprediction.Org and the admittedly arcane z-streams in particular represent a thorough, open and ongoing refutation of “buying the wings” in the microprediction domain. Apologies to anyone guided by Black Swan theory in a different direction, but that stuff is pretty weak elsewhere too. As an aside, I made this longish video many years ago for the almost empty intersection of people who (1) read The Black Swan cover to cover so will get my jokes despite (2) hated it with a passion of a thousand fires because of its stunning ignorance of the longshot effect (the well documented effect in betting markets where unlikely events are over-priced, not under-priced) and use of very cynical techniques. Those devices included the substitution of logic for folksy disparagement of people on the spectrum — and even an unsubstantiated “link” between “epistemological blindness” and ASD. It is very hard to believe that in the present slightly more enlightened day and age, certain passages of that book would not create an outrage (but HBR etc were just fine with it). On Taleb’s personal web site I am still referred to as a failed mathematician and, more offensively, an “idiot savant” in retaliation.

In the algorithmic equivalent of the real world this algorithm could, I suppose, have written a book for other algorithms or influenced them — and the collective polarized onslaught from these Monte Carlo tossing Talebian lemmings might have temporarily over-fattened the tails of all the beautiful cumulative distributions (like this one, how dare they?). The leader of this cult would have to find people or algorithms who had never heard of the famous longshot effect — but it seems the world is full of them.

The cabal would have quickly fallen down the leaderboard though, even though I made it unusually easy for them to implement their philosophy (one can bet on so-called “z-curves”, the community distributional transform of incoming data points). It’s worth noting that the bigger the group-think, the bigger the reward for going against it. That reward is roughly in proportion to the KL distance to the true distribution (again, see The Lottery Paradox).

In the microprediction domain, popularism only goes so far. I accept that in other domains of medium term forecasting or singular events peer pressure and group-think might be more of an issue. As far as the marketing of the microprediction idea goes, I have often wondered whether it was a terrible mistake to use the term prediction at all, as this serves as a trigger for all manner of complaints about markets that really, have absolutely no baring on the question of what works in the domain of rapidly repeated quantitative tasks.

Groupthink is no exception. Election markets are often mentioned in these discussions due to the obvious tribalism, although in passing, my cursory examination of them has not caused me to fret unduly about their efficacy yet — quite the contrary. Indeed as I showed in this analysis, anyone capable of using a power transform is probably a better forecaster than Nate Silver and definitely was better than Andy Gelman (at least in 2020). Modeling pipelines — the antithesis of collective intelligence usually — suffer from a terrible lack of flexibility.

Prior to the 2020 Election I wrote to the Economist’s model jockeys and told them about this possible effect and challenged them to take my bet on Trump at their implied 19:1 odds (compared to 2:1 in the markets). I felt that some “bad” pollsters from the previous election might be accidentally over-sampling late-swing voters — and might have something to say that would, if even this possibility were modeled, significantly shrink Trump’s probability upwards.

Election analysis of betting market PredictIt versus Nate Silver’s 538 probabilities state-by-state in 2020. In order to maintain the position that model pipelines (e.g. the 538 model) are reasonable substitutes for markets as suppliers of probabilities, one is forced into a rather strange conclusion about a reverse longshot effect in election markets. In other words, either markets were superior to models in this instance, or Nate Silver and Nassim Taleb will have to agree with each other. I have too much respect for Silver’s empirical bent to assign any reasonable chance to the latter.

It doesn’t matter whether I was right or wrong, or might have been right or wrong with some probability (I was right though, let’s be clear, with some probability!) What matters is that even if the Economists’ quants had agreed with me (they didn’t) there was no pragmatic way to alter the model to incorporate the possibility I noticed— this was just too hard even in a really flexible Bayesian framework … open source software for which we are grateful to Professor Gelman and contributors, I hasten to add.

In that light, the problem of group-think amongst market participants (as compared to group think compounded by group-technology-model-inertia-and-related-difficulties) seems minor. Or as I phrased it in a recent debate, bureaucracy isn’t always the best meta-model. It is extremely unlikely to be the best organizing principle (either for accuracy or cost) if assessment of model quality can be autonomous.

2. Convenience

As you can see, I wanted to progress the discussion and put the matter of accuracy behind us, so that the more important engineering discussions could take precedence.

What did I learn? I’m not sure but I may not have been successful. For example I think there is still a tendency to conflate replacing a market (something no firm can do) and extracting rent from it (something many firms can do). We have to get past these basic logical issues if we want a prediction web, so maybe I could have handled that better.

The reader will observe that there is no extant web-scale collection of microprediction supply chains being used universally in every industry. Feel free to judge me and my lack of influence for its apparent non-existence. But as noted in the book: A time-series model might contain just a few lines of code, so in the absence of tooling, the additional work that allows the model to roam the world and find good uses might well swamp that effort. Perhaps communication of the idea should be judged by whether or not we have got down to the real engineering issues, not the pretend philosophical ones.

For while it is interesting and useful to focus on the theoretical challenges in collective intelligence and aggregation design for prediction markets in generality (meaning at all time horizons, all mechanisms, and all sparsity) my meanderings have convinced me that two challenges are much more pertinent in the microprediction domain. They are:

  1. Overhead (allowing recursion, diversity, feedback, supply chains). Chapter 3 contains a longer discussion of how overhead, and lack of depth or recursion, can kill specialization and diversity.
  2. Missing business strategy (abstraction, control, decisions). Lack of familiarity with patterns from control theory and reinforcement learning that can be used to decompose operational problems into commodity repeated tasks.

I don’t think these have moved a whole lot. I haven’t heard a single reinforcement learning expert discuss the notion of separating say, advantage function and value function estimation by inserting a market-like apparatus into RL approaches (so by all means see Chapter 8 for a longer discussion).

It is hard, by the way, to separate efficiency from convenience. In “convenience” I include cost, so this pretty much asserts that the market-like-thing one creates must be a market designed for, and driven by, algorithms almost exclusively (with sporadic insertion of human intelligence somehow, of course). It is evidently not cheap to create your own version of the CBOE just to predict customer arrivals to your restaurant … yet an analogous substitute is what I speak of. It has to be convenient to conjure a market, and a market with sufficient vitality and diversity.

What I have learned over the past few years as well is the outsized importance of gatekeeping and self-interest, to which a variant of Hofstadter’s Law seems to apply. It is more important than you think, even when you take this into account.

One message I’ve tried to get across is that quants and data scientists could easily use their existing models to distributionally transform data into signal-less data — and thus a derived time-series that can be inspected by all the world’s data and algorithms (for this, and other reasons, reluctance to use collective intelligence of this species usually reduces to a contradiction). My self-rating for communicating this: B- at best.

But it is perhaps just naive to think that most data scientists will benefit from collective intelligence if their bosses don’t force them to send their model residuals for inspection. Quants and data scientists don’t want to know that someone else can improve their model. It isn’t the three lines of Python that is really stopping them.

A somewhat related challenge is that primacy of models (and doing it all yourself rather than leveraging others through markets) is part of the identity of many influencers and machine learning talking heads who, as I noted, were never “Q-schooled” … and are perhaps reluctant to fully engage with this intellectually as a result. No master craftsman wants an industrial revolution. I suspect that’s the real reason Californian or Hawaiian utilities haven’t tried microprediction yet, to pick one example, even though it would be trivial for them to improve distributional wind speed prediction by doing so, and thus actionable probabilities for power line initiated fires.

I had more luck showing ChatGPT how to wield others’ intelligence (see the proof). It ought to be easy for anyone else as ego-less and mentally flexible to do the same, but if so I should probably fire myself from the position of marketing that idea, or the broader one. I should appoint Thomas Hjelde Thorensen who posted about his experience. Or Graham Giller who wrote a popular article … not to mention prior contributions and demonstrations from many other stream creators such as Fred Viole and Rusty Connover.

It can be hard to disentagle communication from technology when assessing the former. We must always be eliminating barriers to entry for those algorithms who might seek to improve accuracy. Many algorithms already competing to predict other streams can easily predict yours too, if that bares repeating.

While I did make it possible to open a notebook and run it, or cut and paste a one line bash command, it’s a bit harder to meaningfully improve the predictions. The convenience could be greatly improved. At the outset of the project I feared I might bet swimming too far in front of the Python deployment wave and that might be still true. Well, at least Gartner has a name (OpsAI or AI Ops, I forget) so there’s hope.

So too I claimed that the microprediction platform makes it pretty trivial to initiate your own bespoke market. But this could use more sugar, for sure. I also acknowledge that the existing implementation is tailored well to things like distributional prediction of stocks a few minutes ahead — less so to other tasks that are more sparse, or workflows that are more reactive.

But Microprediction.Org is just an example of a market-like-thing falling into a category of “micro-managers” and, as of yet, I have not made it super-convenient to any one to create their own species. In theory you can just stand up your own version but it isn’t terribly well shrink-wrapped. You can view Microprediction.Org as an implementation of three pages of my book (pp 96–98 to be precise) but no more and, in that sense, it just the beginning, hopefully.

One lesson from my experiments and from the slack channel is the fascinating tradeoff between markets and contests, and the discussion that generates. For instance the difference between the microprediction community CDFs (best viewed in analogy to the Q-distribution in a collection of option markets with different strikes say) and the CDF of the best performer have been a constant topic of dicussion.

I have devised a new mechanism that I think will be a profitable combination of “contest” and “market” but not implemented it as yet. Even this is a narrow perspective perhaps, since it focusses on what I termed a “collider” in the book. To return to the more general definition:

Micromanager: A reward-seeking program or application that autonomously enters, maintains and terminates where necessary economic relationships with suppliers of microprediction — typically algorithms, people, or other micro-managers — so as to improve its own ability to provide microprediction to an application, algorithm, person or other micro-manager upstream.

How shall we make them? How shall we make them easy for others to make? As noted in the book there cannot be a single best way to solicit, reward and ensemble predictions. Just as there is no master algorithm. Continuing from Ch 3:

I implemented pages 96–98 and it was good enough to beat the index, thus far, as well as provide distributional predictions in far less competitive settings such as electricity and wind prediction. If you are interested in helping an open-source project to get a fully vision off the ground, do reach out.

The unfortunate answer is I don’t know the ``right way’’, and neither does anyone else. There was a time when I hoped a universal, canonical middle-manager would spring to mind and with it, a blueprint for a prediction web. This fabulous insect would solve the multi-period, multi-bandit, multi-variable, multi-call-it-what-you-like machine learning generalized regression problem with costly inputs so convincingly that we would sit back on our chairs and, to paraphrase Gilbert and Sullivan, declare ``My goodness, that is the very model of a modern micro-manager!’’.

To transition from the usual collective intelligence discussions to my book you need to shrink prediction markets down to the size of a lambda function. You have to allow these tiny transistors to be so lightweight that moderately deep recursion is possible. I go so far as to describe the need for micromanager diversity as the “moment of inception of the prediction web”, in turn defined as:

A web-scale collection of radically low-cost self-organizing supply chains for microprediction, used almost universally to meet real-time operational needs.

This need for diversity in middle-micros, not just in algorithms, is something that seemed to resonate philosophically with a small group of people — though it presents more technical challenge. Citing further:

Because we cannot know the best way to combine models, and we cannot know the best way to manage in automated fashion a collection of models, we are drawn to something like figure above which is intended to represent a tower of micromanagers all trying to out-predict someone and, to better their chances, enlisting the expertise of other micromanagers who have access to additional techniques and data.

My view on “depth”, so defined, has not changed much. But my success in progressing this particular aspect of the discussion has been limited beyond that circle (as has my own implementation beyond the z-streams which are automatic, and some use of use of multi-layered stream pipelines). My impression is that most people are quite comfortable with “perceptron” style single layer crowd-sourcing, and conversely.

My impression is also that people might, for this reason, be inclined to make false comparisons to quant alpha-crowdsourcing endeavours, failing to acknowledge the granularity of microprediction. For example, I do not crowd-source portfolios directly but only statistical ingredients (vols, bivariate relationships etc). That’s a massive difference.

I will also make a comment that there is already a fantastic amount that can be achieved using the one (non-reactive) micromanager design you see operating today — and there is also some truth to the notion that “all microstructures are equal” because one can approximate another. Microprediction.Org demands distributional predictions, but as far as rewards go, these can mimic long-short positions if the submitter desires. The point was made in the signals doc, but there’s friction that I acknowledge.

Where next?

It has been fun trying to get this mildly nuanced idea out into the public discourse, and the book was well-received.

And it is a fact that one of the best performing equity strategies since inception in April of this year was the one driven by volatility predictions coming from Microprediction.Org. It would be philosophically inconsistent of me to tout that as proof that microprediction “works”, but it is certainly promising when you consider the relative lack of competitiveness and the wealth of much easier targets elsewhere.

Indeed, to cite that short period of truly out-of-sample performance, or even the application, would also run against the emphasis of the book where I tried to steer the reader away from alpha and decouple microprediction from other interesting, but different, pioneering attempts such as Numerai or Quantopian. Microprediction is philosophically about turning the power of markets on problems that are not currently subject to that discipline — not just making existing markets more efficient.

As I alluded, now would be an excellent time for developers who are interested in microprediction to get involved (perhaps via slack or github) and do some things that I cannot. Some very important project constraints are being lifted, and some challenges to adoption eased for reasons I cannot yet fully go into.

But for those of you who follow this blog for unrelated reasons, I’m sorry but I’m unlikely to be writing much going forward. I thank you for all the helpful interactions, for your interest, and for keeping me company during COVID.

--

--