Sunday, May 31, 2020

The "Weather-VIX" -- A Volatility IndeX for Weather?

A better way to understand climate change and global warming may be to focus less on quantifying the direction of changes, but on quantifying the volatility of weather extremes of all kinds -- temperature, precipitation, humidity, wind, storms, etc.

Many have noted that "global warming" is not just a matter of warming, and that we might better focus on solving the problem with better messaging. Tom Friedman has referred to it as "global wierding," saying, "The frequency, intensity and cost of extreme weather events all increase. The wets get wetter, the hots get hotter, the dry periods get drier, the snows get heavier, the hurricanes get stronger. Weather is too complex to attribute any single event to climate change, but the fact that extreme weather events are becoming more frequent and more expensive — especially in a world of crowded cities like Houston and New Orleans — is indisputable." Brad Plumer made similar points about the need for more understandable messaging.

I have been suggesting that we track and report a “Weather-VIX” (WVIX) -- much as financial markets track a "Volatility IndeX" (VIX). In financial markets, the VIX is often understood as a "fear index." For weather, it might be seen as a "disruption index."

A Weather-VIX volatility index for our weather, would be a complementary metric to average temperature trends. By tracking the volatility of weather (from day to day), wouldn't we see a very significant and increased volatility in temperatures, precipitation, and wind speed? Unlike the small changes in average temperature, volatility trends might be far more dramatic, and much less easily dismissed as just a natural fluctuation. Refocusing on volatility would also remove silly arguments that extremes of cold refute global "warming" -- of course the warming is not always "global," and is not always consistent at any given time. We can better understand that the weather will not be volatile at any given place at every given time, but tracking volatility in each region would give clearer evidence of increasing overall volatility, and how that varies from region to region.

This WVIX could also be tied to the monetary costs of extremes in both directions -- “WVIX-cost.”

Even if only based on data for the last hundred years or so (and only in locations with good data), we might see that violent and erratic weather is already accelerating to increasingly costly levels. Insurance companies will be among the first to see and quantify this as an actuarial cost, but with a simple WVIX index, we will all be able to understand this effect more clearly.

Monday, May 18, 2020

The Pandemic Reminds Us "Everything is Deeply Intertwingled" – We Need Better Logics for That

A bat catches a cold in Wuhan, and weeks later the whole world coughs. America and China battle a trade war, and then there is a shortage of PPE and ventilator parts from China.  Poor neighborhoods suffer high death rates because of poor health, but even celebrities and heads of state go into ICUs.  The economy craters, and we argue over relief to businesses versus workers based on which is more disposed to misuse what they might be given.  Health officials say flatten the curve, financiers say reopen, and corporations say they don’t dare reopen without testing.  The Federal government is too polarized to fix much of anything, and has forgotten their real job of governing by consensus.  

Modern technologies of global connection -- both physical and virtual -- make the pandemic emerge in weeks instead of years, and make all the butterfly effects far more complex.  That is our new curse, but also our new blessing.  We have global travel and supply chains, global communications and media networks -- a global village composed of local villages.  Techies moved fast and broke society, and now discourse seems too polarized to fix it.

All of these effects are driven by market forces -- however regulated.  Marketplaces of goods and services and marketplaces of ideas.  These marketplaces are driven by complex interplays of top-down structure and bottom-up emergence from billions of actors --and systems of actors.  Technology has made these forces more dynamic and turbulent, but technology can enable smarter and better-regulated marketplaces -- if we re-focus.  We cannot undo this onrushing dynamic -- we need to get smarter about how we use technology to help us go forward.

The pandemic may be the kick in the ass we need to reform society over a wide range of domains and levels.  Seeing the commonalities can help us capture a new synergy.  If rise to that challenge, the future will be bright. If we fail it will be dark.  Many see that, but few focus on the root causes. 

Peter Drucker said “The greatest danger in times of turbulence is not the turbulence, it is to act with yesterday’s logic.” Two new logics can help us correct the failures of our current logics.

Ever-growing intertwingularity

The problem we now face all too urgently is that our lives are all deeply intertwingled, but we fall back to simplistic “fast” thinking with rigid categories and institutions.  Some leaders rise to the challenge and others flail, and we argue over who does which.  The regulation of our marketplace of ideas that “mediates consent” about facts has broken down, as has our social/economic marketplace.  These problems are difficult and complex – but we can get smarter about solving them – the first order and second order effects.  (To get a sense of the range of these issues, see this briefing by Tony O’Driscoll and this McKinsey report. To see how this has reopened old questions, and may provide an opening for new thinking, see this NY Times report on the shifting issues for the 2020 election.)
The symbolic circle of the Tao reminds us of that truth is never entirely black or white, but shades of gray that depend on the light we view it in and the perspective we view it from.  Just how much is subject to argument, discovery, and rediscovery, as reality emerges.  This is age-old, but it is more urgent than ever that we come to grips with it.  2020 will mark a turning point in human history.

For decades our world and our markets have been increasingly stressed, even as we seemed to be progressing.  Tensions of nationality, race, ideology, religion, economics, technology, and governance are raging.  Things fall apart; the centre cannot hold /…The best lack all conviction, while the worst / Are full of passionate intensity.”  It is now urgent that we re-center more wisely on our better convictions.

The Enlightenment has run aground because those who saw the light and had the benefits did not pay enough attention to sharing that.  Liberals turned away from “the deplorables” instead of caring for and raising them up.  Capitalists extracted short-term profits and enriched themselves with stock buybacks -- exploiting workers instead of empowering them.  Factions and political parties fought zero-sum struggles to control the existing pie instead of engaging in win-win cooperation to create and share a larger pie.

The Chinese ideograph for crisis is composed from the characters for danger plus opportunity.  Many retreat in fear of the danger and seek to throw blame and erect walls, but wiser heads look to the opportunity.  Most see opportunity in narrow domains, but some look to the big picture.  We now face an urgent and historic opportunity to refocus on a more enlightened and productive kind of cooperation across the full range of issues.

Those who see and work on these problems in particular domains of concern and expertise can unite in spirit and vision with those in other domains.  We can forge a new Age of Enlightenment – a Reformation of The Enlightenment.  An awakening of interconnection and cooperative spirit is emerging.  Our challenge is to synergize it.  Some elements:
  • Economic and health insecurity for some leads to insecurity for all.  A safety net is needed.
  • Market systems need slack to respond to black swan events.  “Just-in-time” and “lean” are efficient only when not overstressed.  Global supply chains need resilience and redundancy. Too much slack and safety drain our wealth and will, but too little lead to disaster.
  • Moving fast and breaking things can break things that cannot be fixed.  Experience can blind us, but inexperience can kill.
  • Power among local, state, national, and global government must be properly balanced and adaptable to stress.  Power and wealth must be shared fairly among people, factions, and nations, or those left wanting will throw rocks at the crystal palace.  The resurgence of nationalism, factionalism, and the crisis of disinformation are symptoms of perceived unfairness.  Government that is too small is just as bad as too big.

Our modern, high-tech world is far too complex for purely top-down or bottom-up management and governance -- we need a smart and adaptive blend. That requires openness, transparency, trust, and fairness, so even when there is disagreement, there is a common sense of reasonableness and good spirit.

New Logics for Intertwingularity

My recent work has focused on two new ways to deal better with this growing complexity.  These new logics that do not just exhort people to be better and wiser, but better align interests so that virtue is rewarded. 

One relates to failures of our marketplace of ideas – especially our social media and other collaborative systems.  Computer-augmented human collaboration first emerged in the 1960s, and was used for disaster preparedness (natural and nuclear).  It progressed slowly until the Web made it far more powerful and accessible to consumers, but we failed to direct those social media systems to serve us well.  Struggling to find a business model, they hit on advertising. We now recognize that to be “the original sin of the Internet” because it misdirects our platforms to serve advertisers and not users.  Algorithms can help augment human intelligence to make us smarter collectively -- instead of making us stupider, as social media now do.  Systems that elucidate nuance, context, and perspective can empower us to draw out and augment the wisdom of crowds (as explained in detail on this blog) to deal more smartly with our deeply intertwingled world.  That could drive a new Age of Enlightenment in which technology augments the marketplace of ideas in the ways that we have always relied on to mediate consent – an emergent mix of top-down guidance and bottom-up emergence that can lead to new, yet natural, forms of digital democracy.

The other relates to failures of our economic marketplace – how we can shift from the short-term. zero-sum logic of extractive mass-market capitalism to more long-term, win-win forms of market cooperation.  That can restore the emergent, distributed, and human logic of traditional markets that Adam Smith saw as socially beneficial -- before modern mass-marketing alienated producers from consumers and lost sight of broader human values.  Our digital economy now enables new ways to shift from fighting over a current pie to cooperating to co-create a larger pie -- and to share it fairly.  That logic can empower a reformation of market capitalism from within that could actually be more profitable, and thus self-motivating.  We can apply the power of computer-mediated marketplaces to let businesses and consumers negotiate at a human level -- about the values they care about, how to co-create that value, and how to share in the benefits.  We have begun to think in terms of customer journeys, but have been trying to fit customers into segments or personas. Instead, we need to design for segments of one that are custom-fit to each customer, to build relationships with each customer on human terms.

These two logics are interrelated: a flawed economic logic for consumer platform services has been built on advertising revenue (“the Internet’s original sin”). That has warped incentives to favor engagement with junk content that sells ads, rather then the value to users of quality content. An improved logic for value will create incentives for our platforms to facilitate a logic for a better marketplace of ideas.

The brief descriptions of these new logics may sound like just more exhortations, but the posts that they link to provide details of operational mechanisms -- and evidence that their elements have proven effective.  These new combinations of elements can quickly become second nature, because they draw on and re-channel natural behaviors that promise to make them highly self-reinforcing.

Many allied visions for better logics of emergence are finding new relevance in this era of crisis.  We have only to join together and rise to the occasion.  We say that “we are all in this together” – we need to open our minds to really think that way, and to work with new logics and “choice architectures” that make that natural.  With better logics, our instinctive behaviors can once again synergize to flow in increasingly enlightened ways.

For more about the new logic for the marketplace of ideas (and intertwingularity in general), see this list of selected items on the blog.

For more about the new logic for the economic marketplace, see this list of selected items on the blog.

Letter to The Atlantic (Renee DiResta) on "Getting the Message Out" (From Experts on Virus)

Renee DiResta's article in The Atlantic, Virus Experts Aren’t Getting the Message Out (5/6/20), is an insightful analysis of how social media have broken society's ability for "mediating consent" as to what is fact and what is fiction (or worse). 

Here is the letter I wrote in response:

To Renee DiResta’s excellent statement of the problem -- getting quality information on Covid-19 to the public in this time of crisis (which is just part of a much broader problem) -- I would add suggestions for a better solution. She is correct that our public health institutions must adapt to modern modes of communication, and that media should select for authoritative voices, including those who are outside those institutions. She rightly says “Some of the best frameworks for curating good information…involve a hybrid of humans and artificial intelligence…These processes are difficult to scale because they involve human review, but they also recognize the value of factoring authoritativeness—not just pure popularity...the ‘consensus of the most liked.’”

The solution to this critical challenge of scaling is to use algorithms more effectively -- to “augment the wisdom of crowds.”  The crowd gets wiser when the human votes of authoritative likes count for more than those of foolish or malicious likes. This can be done by building on the huge success of how Google’s hybrid PageRank algorithm first augmented the wisdom of the Web-linking crowd. PageRank did not rely on machine understanding of content (still very difficult), but only on the raw power of machine tabulation of human understanding (IBM began with tabulating the 1890 census).

The genius of PageRank is not to rank Web pages by purely human authorities as Yahoo did, nor by pure algorithms as AltaVista did, but by a clever and scalable hybrid of man and machine. It interprets links to a Web page from other sites as equivalent to likes that signal the judgment of human “Webmasters” or authors. But it then augments those judgments: instead of weighting all such links as equal votes of authority, it weights them based on their own authority.  It sees who links to them (one level removed), and recursively, what authority those links should have, based on who links to them (a further level removed).

Social media and other information discovery media could apply much the same method. A “RateRank” algorithm that augments human intelligence in this way could determine whose likes it should rank as authoritative and whose likes it should rank as noise. It could track signals that reflect human judgement – likes, shares, comments, followers, etc. -- and determine reputations for those “ratings” to know which to weight highly as from respected raters, and which to discount as from usually foolish or malicious raters. Certifications of authority from independent rating institutions could also be factored in, but this algorithm would also up-rank emerging or non-mainstream voices that deserve to be heard -- including those that are responsibly contrarian.

Such hybrid algorithms would power a highly adaptive “cognitive immune system” that would help insure -- at Internet scale and speed -- that the most authoritative and deserving messages get out most widely, and that misinformation and disinformation is suppressed. (This need not limit First Amendment rights, since it would limit how dubious content is distributed, but it could still be posted and accessible to anyone who specifically seeks it.)

These proposals for up-ranking quality (details at have gotten attention in the technology and policy community, but media businesses have yet to be receptive. The only apparent reason seems to be that their advertising-driven business model thrives on “elevating popularity over facts” as DiResta notes. But, if the current algorithmic de-augmentation of human intelligence does not change, humanity may never recover.

[This letter was sent to The Atlantic on 5/11/20. I had previously sent a draft to Renee for comment, and she responded that she viewed it as thoughtful and encouraged me to submit it.]