Showing posts with label social network. Show all posts
Showing posts with label social network. Show all posts

Thursday, December 26, 2019

2020 Vision -- The Restoration of the Customer

The Age of the Customer: You Ain't Seen Nothin' Yet

Nearly a decade has passed since Forrester said we were entering The Age of the Customer. That is apparent and has obvious implications. But as the decade of the 2020's dawns, I call out a deeper vision -- The Restoration of the Customer -- that could bring far more fundamental changes in the coming decade.

There are two surprising turns that may be taken in this decade to restore power to customers -- one that can fundamentally change how we conduct business, and one that can fundamentally change how we collaborate.

Those turns might just begin to undo many of the ills of the industrial revolution and of the computer revolution.  Both turns center on a return to enlightened human values:
  • The customer is not just a persona with a bundle of attributes that a business can learn how to manipulate, but a unique human being that has been bred since pre-history to thrive on a cooperative effort to create value and share it.
  • The user is not just a a source of attention that can be engaged to be sold to advertisers, but a customer to be served what they value -- again, a cooperative effort to create value and share it.
What those paying attention see

Forrester put the basic drivers nicely (emphasis added):
In this era, digitally-savvy customers would change the rules of business, creating extraordinary opportunity for companies that could adapt, and creating existential threat to those that could not. ...It requires leaders to think and act differently – in ways that feel foreign, unfamiliar, and counter-intuitive. And honestly, it is simply hard to do. ...These dynamics will endure as new technologies like artificial intelligence and robotics emerge to challenge core notions of what it means to be a company, what it means to build human capital, and what it means to compete and win.
...And, a deeper vision

Here I point to some little recognized ideas on how re-centering on value can change not only the dynamic of commerce, but also a parallel dynamic of customer value that is equally important.
  • First, the commercial dynamic that Forrester describes is just the foundation for reversing how the "progress" of technology cost us the human dimension in commerce -- a dimension that we had when commerce was just the way villagers did business with one another -- with human beings on both sides of an ongoing relationship. 
  • Second, we humans, as "customers" of Web services, have lost control of our experience of the world.  Our central experience of human interaction has been hijacked by platforms who "engage" us in order to profit from bombarding us with advertising and paid propaganda.
First: Back to the future of commerce

Consumers are increasingly alienated from the companies they do business with. Instead of neighbors or shopkeepers, we deal with soulless institutions that we distrust and feel abused by. That has been, increasingly, the price of productivity and material riches. But now technology has advanced far enough to restore the dimension of human values -- if we applied to do so. That does not require that we abandon the miracle of capitalism, but only that we bring it back to the marketplace of human value. Technology now makes it possible for even large faceless institutions to build human interfaces that behave with human values. That will drive institutions to interact with human in ways that are more truly human.

FairPay is a framework for centering on why and how to do that. The key is to recenter on relationships and the creation and sharing of value in ways that are tailored to each individual. Specifics on how to do that are in my FairPayZone blog, some articles written with prominent marketing scholars, and my 2016 book. Some of the best places to begin to understand this are:
Second: Who does it serve? - a course correction in how we experience the world

Social media and other online content services have changed how we experience the world, including how we interact with other people. Computer-mediation began with great hopes, but now it seems we have built a Frankenstein's monster.  As growing calls for change are beginning to focus on new levels of regulation, it is not enough to regulate against specific harms. Instead we must refocus on what we want to regulate for -- who these "services" serve, and what we want these platforms to facilitate. They were supposed to make us happy and smart -- instead they are making us angry and stupid. But technology can reverse that, if we incentivize that.

We can design new architectures for our interactive media that create value for us.  The key is to recognize that each of us is an individual, and we should be able to individualize our services, mixing and matching offerings to make just the service we want for what we are doing now. The most urgent part of that is to shape our media services to give each of us what we value. The Web stated out seeking to do that, and we can return to that vision. It won't be free, but it can be affordable. And we have seen that "free" is not really affordable (because it is not really free). If we do not change direction, our democracies and our civilization will collapse. Some starting points for seeing how:

(Cross-posted with my other blog, FairPayZone.)

Sunday, October 20, 2019

Our Digital Platforms -- What We Want to Regulate For (Not Just Against) [Preliminary Draft]

This is now superseded by an updated and expanded post:
Regulating our Platforms -- A Deeper Vision (Working Draft)

Preliminary Draft:  This post is an initial summary of my comments at this event, as sent on 10/20/19 to some of the speakers and attendees. (I am writing an expanded post to supersede this.)
---

Summary and Expansion of Dick Reisman’s comments on attending GMF 10/17/19 event:

I very much support the proposals for a New Digital Platform Authority (as detailed in the excellent background items[*] cited on the event page) and offer some innovative perspectives.  I welcome dialog and opportunities to participate in and support related efforts. 

(My background is complementary to most of the attendees -- diverse roles in media-tech, as a manager, entrepreneur, inventor, and angel investor.  I became interested in hypermedia and collaborative social decision support systems around 1970, and observed the regulation of The Bell System, IBM, Microsoft, the Internet, and cable TV from within the industry.  As a successful inventor with over 50 software patents that have been widely licensed to serve billions of users, I have proven talent for seeing what technology can do for people.  Extreme disappointment about the harmful misdirection of recent developments in platforms and media has spurred me to continue work on this theme on a pro-bono basis.)  

My general comment is that for tech to serve democracy, we not only need to regulate to limit monopolies and other abuses, but also need to regulate with a vision of what tech should do for us -- to better enable regulation to facilitate that, and to recognize the harms of failing to do so.  If we don’t know what we should expect our systems to do, it is hard to know when or how to fix them.  The harm Facebook does becomes far more clear when we understand what it could do – in what ways it could be “bringing people closer together,” not just that it is actually driving them apart.  That takes a continuing process of thinking about the technical architectures we desire, so competitive innovation can realize and evolve that vision in the face of rapid technology and market development.

More specifically, I see architectural designs for complex systems as being most effective when built on adaptive feedback control loops that are extensible to enable emergent solutions as contexts, needs, technologies, and market environments change.  That is applicable in various forms to all the strategies I am suggesting (and to technology regulation in general).

I cited the Bell System regulation as a case in point that introduced well architected modularity in the Carterfone Decision (open connections via a universal jack, much like modern API’s), followed by the breakup into local and long-distance and manufacturing, and the later introduction of number portability.  This resonated as reflecting not only the wisdom of regulators, but expert vision of the technical architecture needed, specifically what points of modularity (interoperability) would enable innovation.  (Of course the Bell System emerged as a natural monopoly growing out of an earlier era of competing phone systems that did not interoperate.)  The modular architecture of email is another very relevant case in point (one that did not require regulation).

I noted three areas where my work suggests how to add more of that dimension to the excellent work in those reports.  One is a fundamental problem of structure, and the other two are problems of values that reinforce one another.  (The last one applies not only to the platforms, but to the fundamental challenge of sustaining news services in a digital world.)  All of these are intended not as point solutions, but as ongoing processes that involve continuing adaptation and feedback, so that the solutions are emergent as technology and competitive developments rapidly advance.

1.  System and business structure -- Modular architecture for flexibility and extensibility.  The heart of systems architecture is well-designed modularity, the separation of elements that can interoperate yet be changed in at will -- that seems central to regulation as well – especially to identify and manage exclusionary bottlenecks/gateways.  At a high level, the e-mail example is very relevant to how different “user agents” such as Outlook, Apple mail, and Gmail clients can all interoperate to interconnect all users through “message transfer agents” (through the mesh of mail servers on the Internet).  A similar decoupling should be done for social media and search (for both information and shopping).

Similar modularity could usefully separate such elements as:
  • Filtering algorithms – to be user selectable and adjustable, and to compete in an open market much as third-party financial analytics can plug in to work with market data feeds and user interfaces.
  • Social graphs – to enable different social media user interfaces to share a user’s social graph (much like email user agent / transfer agent).
  • Identity – verified / aliased / anonymous / bots could interoperate with clearly distinct levels of privilege and reputation.
  • Value transfer/extraction systems – this could address data, attention, and user-generated-content and the pricing that relates to that.
  • Analytics/metrics – controlled, transparent monitoring of activity for users and regulators.

2.  User-value objectives -- filtering algorithms controlled by and for users.  This is the true promise of information technology – not artificial intelligence, but the augmentation of human intelligence.
·         User value is complex and nuanced, but Google’s original PageRank algorithm for search results filtering demonstrates how sophisticated algorithms can optimize for user value by augmenting the human wisdom of crowds – they can understand user intent, and weigh implicit signals of authority and reputation derived from humans inputs at multiple levels, to find relevance in varying contexts. 
·         In search, the original PageRank signal was inward links to a Web page, taken as expressions of the value judgements of individual human webmasters regarding that page.  That has been enriched to weed out fraudulent “link farms” and other distortions and expanded in many other ways.
·         For the broader challenge of social media, I outline a generalization of the same recursive, multi-level weighting strategy in The Augmented Wisdom of Crowds: Rate the Raters and Weight the Ratings.  The algorithm ranks items (of all kinds) based on implicit and explicit feedback from users (in all available forms), partitioned to reflect communities of interest and subject domains, so that desired items bubble up, and undesired items are downranked.  This can also combat filter bubbles -- to augment serendipity and to identify “surprising validators” that might cut through biased assimilation.
·         That proposed architecture also provides for deeper levels of modularity:  to enable user control of filtering criteria, and flexible use of filtering tools from competing sources -- which users could combine and change at will, depending on the specific task at hand.  That enables continuous adaptation, emergence, and evolution, in an open, competitive market ecosystem of information and tools.
·         Filtering for user and societal value:  The objective is to allow for smart filtering that applies all the feedback signals available to provide what is valued by that the user at that time.  By allowing user selection of filtering parameters and algorithms, the filters can become increasingly well-tuned to the value systems of each user, in each community of interest, and each subject domain.
·         First amendment, Section 230, prohibited content issues, and community standards:  When done well, this filtering might largely address those concerns, greatly reducing the need for the blunt instrument of regulatory controls or censorship, and working in real time, at Internet-speed, with minimal need for manual intervention regarding specific items.  As I understand the legal issues, users could retain the right to post information without restriction (with narrow exceptions) -- if objectionable content is automatically downranked enough in any filtering process that a service provides (an automated form of moderation) to avoid sending it to users who do not want such content -- or who reside in jurisdictions that do not permit it.  Freedom of speech (posting), not freedom of delivery to others who have not invited it.  Thus Section 230 might be applied to posting, just as seemed acceptable when information was pulled from the open Web, while the added service of dissemination filtering might apply both user and governmental restrictions (as well as restrictions specific to user communities that desire such filtering) when information is pushed from social media. 
(Such methods might evolve to become a broad architectural base for richly nuanced forms of digital democracy.)

3.  Business model value objectives – who does the platform serve?  This is widely asserted to be the “original sin” of the Internet that prevents better solutions in the above two areas.  Without solving this problem, it will be very difficult to solve the other problems.  “It is difficult to get a man to understand something when his job depends on not understanding it”  Funding of services with the ad model makes services seem free and affordable, but drives platform services to optimize for engagement, to sell ads, instead of optimizing for value to users and society.  Users are the product, not the customer, and value is extracted from the customer to serve the platforms and the advertisers.  This is totally unlike prior forms of advertising because unprecedented detail in user data and precision targeting enables messaging and behavioral manipulations at an individual level.  That has driven algorithm design and use of the services in harmful directions instead of beneficial ones.  Many have recognized this business model problem, but few see any workable solution. I suggest a novel path forward at two levels:  an incentive ratchet to force the platforms to seek solutions, and some suggested solutions mechanisms that suggest that ratchet would bear fruit in ways that are both profitable and desirable.

Ratchet the desired business model shift with a simple dial, based on a simple metric.  A very simple and powerful regulatory strategy could be to impose taxes or mandates that gradually ratchet toward the desired state. This leverages market forces and business innovation in the same way as the very successful model of the CAFE standards for auto fuel efficiency -- it leaves the details of how to meet the standard to each company. 
·         The ratchet here is to provide compelling incentives for dominant services to ensure that X% of revenue must come from users.  Such compelling taxes or mandates might be restricted to distribution services with ad revenues above some threshold level.  (Any tax or penalty revenue might be applied to ameliorate the harms.)
·         That X% could still include advertising revenue if it is quantified as a credit back to the user (a “reverse meter” much as for co-generation of electricity).  Advertising can be valuable and non-intrusive and respectful of data -- explicitly putting a price on the value transfer from the consumer incentivizes the market to achieve that. 
·         This incentivizes individual companies to shift their behavior on their own, without need for the kind of new data intermediaries (“infomediaries”) that others have proposed without success.  It could also create more favorable conditions for such intermediaries to arise.

Digital services business model issues -- for news services as well as platforms.  (Not addressed at the event.)  Many (most prominently Zuckerberg) throw up their hands at finding business models for search or social media that are not ad-funded, primarily because of affordability issues.  The path to success here is uncertain (just as the path to fuel efficient autos is uncertain).  But many innovations emerging at the margins offer reasons to believe that better solutions can be found. 
·         One central thread is the recognition that the old economics of the invisible hand fails because there is no digital scarcity for the invisible hand to ration.  We need a new way to settle on value and price.
·         The related central thread is the idea of a social contract for digital services, emerging most prominently with regard to journalism (especially investigative and local).  We must pay now, not for what has been created already, but to fund continuing creation for the future. Behavioral economics has shown that people are not homo economicus but homo reciprocans. – they want to be fair and do right, when the situation is managed to encourage win-win behaviors. 
·         Pricing for digital services can shift from one-size-fits-all, to mass-customization of pricing that is fair to each user with respect to the value they get, the services they want to sustain, and their ability to pay.  Current all-you-can-eat subscriptions or pay-per-item models track poorly to actual value.  And, unlike imposing secretive price discrimination, this value discrimination can be done cooperatively (or even voluntarily).  Important cases in point are The Guardian’s voluntary payment model, and recurring crowdfunding models like Patreon. 
·         Synergizing with this, and breaking from norms we have become habituated to, the other important impact of digital is the shift toward a Relationship Economy – shifting focus from one-shot zero-sum transactions to ongoing win-win relationships such as subscriptions and membership.  This builds cooperation and provides new leverage for beneficial application of behavioral economic nudges to support this creative social contract, in an invisible handshake.  My own work on FairPay explains this and provides methods for applying it to make these services sustainable by user payments. See this Overview with links, including journal articles with prominent marketing scholars, brief articles in HBR and Techonomy, and many blog posts.  
·         Vouchers.  The Stigler Committee proposal for vouchers might be enhanced by integration with the above methods.  Voucher credits might be integrated with subscription/membership payments to directly subsidize individual payments, and to nudge users to donate above the voucher amounts.
·         Affordability. To see how this deeper focus on value changes our thinking, consider the economics of reverse meter credits for advertising, as suggested for the ratchet strategy above.  As an attendee noted at the event, reverse metering would seem to unfairly favor the rich, since they can better afford to pay to avoid ads.  But the platforms actually earn much more for affluent users (targeted ad rates are much higher).  If prices map to the value surplus, that will tend to balance things out – if the less affluent want service to be add free, it should be less costly for them than for the affluent.

AI as a platform regulatory issue.  Discussion after the session raised the issue of regulating AI.  There is growing concern relating to concentrations of power and other abuses, including concentrations of data, bias in inference and in natural language understanding, and lack of transparency, controls, and explainability.  That suggests a similar need for a regulator that can apply specialized technical expertise that overlaps with the issues addressed here.  AI is fundamental to the workings of social media, search, and e-commerce platforms, and also has many broader applications for which pro-active regulation may be needed.

---

[*Update:] Here are the cited items, plus two other excellent reports:
---
See the Selected Items tab for more on this theme.

Wednesday, October 10, 2018

In the War on Fake News, All of Us are Soldiers, Already!

This is intended as a supplement to my posts "A Cognitive Immune System for Social Media -- Developing Systemic Resistance to Fake News" and "The Augmented Wisdom of Crowds: Rate the Raters and Weight the Ratings." (But hopefully this stands on its own as well).Maybe this can make a clearer point of why the methods I propose are powerful and badly needed...
---

A NY Times article titled "Soldiers in Facebook’s War on Fake News Are Feeling Overrun" provides a simple context for showing how I propose to use information already available from all of us, on what is valid and what is fake.

The Times article describes a fact checking organization that works with Facebook in the Philippines (emphasis added):
On the front lines in the war over misinformation, Rappler is overmatched and outgunned - and that could be a worrying indicator of Facebook’s effort to curb the global problem by tapping fact-checking organizations around the world.
...it goes on to describe what I suggest is the heart of the issue:
When its fact checkers determine that a story is false, Facebook pushes it down on users’ News Feeds in favor of other material. Facebook does not delete the content, because it does not want to be seen as censoring free speech, and says demoting false content sharply reduces abuse. Still, falsehoods can resurface or become popular again.
The problem is that the fire hose of fake news is too fast and furious, and too diverse, for any specialized team of fact-checkers to keep up with it. Plus, the damage is done by the time they do identify the fakes and begin to demote them.

But we are all fact checking to some degree without even realizing it. We are all citizen-soldiers. Some do it better than others.

The trick is to draw out all of the signals we provide, in real time -- and use our knowledge of which users' signals are reliable -- to get smarter about what gets pushed down and what gets favored in our feeds. That can serve as a systemic cognitive immune system -- one based on rating the raters and weighting the ratings.

We are all rating all of our news, all of the time, whether implicitly or explicitly, without making any special effort:

  • When we read, "like," comment, or share an item, we provide implicit signals of interest, and perhaps approval.
  • When we comment or share an item, we provide explicit comments that may offer supplementary signals of approval or disapproval.
  • When we ignore an item, we provide a signal of disinterest (and perhaps disapproval).
  • When we return to other activity after viewing an item, the time elapsed signals our level of attention and interest.
Individually, inferences from the more implicit signals may be erratic and low in meaning. But when we have signals from thousands of people, the aggregate becomes meaningful. Trends can be seen quickly. (Facebook already uses such signals to target its ads -- that is how they makes so much money).

But simply adding all these signals can be misleading. 
  • Fake news can quickly spread through groups who are biased (including people or bots who have an ulterior interest in promoting an item) or are simply uncritical and easily inflamed -- making such an item appear to be popular.
  • But our platforms can learn who has which biases, and who is uncritical and easily inflamed.
  • They can learn who is respected within and beyond their narrow factions, and who is not, who is a shill (or a malicious bot) and who is not.
  • They can use this "rating" of the raters to weight their ratings higher or lower.
Done at scale, that can quickly provide probabilistically strong signals that an item is fake or misleading or just low quality. Those signals can enable the platform to demote low quality content and promote high quality content. 

To expand just a bit:
  • Facebook can use outside fact checkers, and can build AI to automatically signal items that seem questionable as one part of its defense.
  • But even without any information at all about the content and meaning of an item, it can make realtime inferences about its quality based on how users react to it.
  • If most of the amplification is from users known to be malicious, biased, or unreliable it can downrank items accordingly
  • It can test that downranking by monitoring further activity.
  • It might even enlist "testers" by promoting a questionable item to users known to be reliable, open, and critical thinkers -- and may even let some generally reliable users to self-select as validators (being careful not to overload them).
  • By being open-ended in this way, such downranking is not censorship -- it is merely a self-regulating learning process that works at Internet scale, on Internet time.
That is how we can augment the wisdom of the crowd -- in real time, with increasing reliability as we learn. That is how we build a cognitive immune system (as my other posts explain further).

This strategy is not new or unproven. It is is the core of Google's wildly successful PageRank algorithm for finding useful search results. And (as I have noted before), it was recently reported that Facebook is now beginning to do a similar, but apparently still primitive form of rating the trustworthiness of its users to try to identify fake news -- they track who spreads fake news and who reports abuse truthfully or deceitfully.* 

What I propose is that we take this much farther, and move rapidly to make it central to our filtering strategies for social media -- and more broadly. An all out effort to do that quickly may be our last, best hope for enlightened democracy.

Related posts:
----
(*More background from Facebook on their current efforts was cited in the Times article: Hard Questions: What is Facebook Doing to Protect Election Security?

[Update 10/12:] A subsequent Times article by Sheera Frenkel, adds perspective on the scope and pace of the problem -- and the difficulty in definitively identifying items as fakes that can rightly be censored "because of the blurry lines between free speech and disinformation" -- but such questionable items can be down-ranked.

Monday, October 08, 2018

A Cognitive Immune System for Social Media -- Developing Systemic Resistance to Fake News

To counter the spread of fake news, it's more important to manage and filter its spread than to try to interdict its creation -- or to try to inoculate people against its influence. 

A recent NY Times article on their inside look at Facebook's election "war room" highlights the problem, quoting cybersecurity expert Priscilla Moriuchi:
If you look at the way that foreign influence operations have changed these last two years, their focus isn’t really on propagating fake news anymore. “It’s on augmenting stories already out there which speak to hyperpartisan audiences.”
That is why much of the growing effort to respond to the newly recognized crisis of fake news, Russian disinformation, and other forms of disruption in our social media fails to address the core of the problem. We cannot solve the problem by trying to close our systems off from fake news, nor can we expect to radically change people's natural tendency toward cognitive bias. The core problem is that our social media platforms lack an effective "cognitive immune system" that can resist our own tendency to spread the "cognitive pathogens" that are endemic in our social information environment.

Consider how living organisms have evolved to contain infections. We did that not by developing impermeable skins that could be counted on to keep all infections out, nor by making all of our cells so invulnerable that they can resist whatever infectious agents may unpredictably appear.

We have powerfully complemented what we can do in those ways by developing a richly nuanced internal immune system that is deeply embedded throughout our tissues. That immune system uses emergent processes at a system-wide level -- to first learn to identify dangerous agents of disease, and then to learn how to resist their replication and virulence as they try to spread through our system.

The problem is that our social media lack an effective "cognitive immune system" of this kind. 

In fact many of our social media platforms are designed by the businesses that operate them to maximize engagement so they can sell ads. In doing so, they have learned that spreading incendiary disinformation that makes people angry and upset, polarizing them into warring factions, increases their engagement. As a result, these platforms actually learn to spread disease rather than to build immunity. They learn to exploit the fact that people have cognitive biases that make them want to be cocooned in comfortable filter bubbles and feel-good echo-chambers, and to ignore and refute anything that might challenge beliefs that are wrong but comfortable. They work against our human values, not for them.

What are we doing about it? Are we addressing this deep issue of immunity, or are we just putting on band-aids and hoping we can teach people to be smarter? (As a related issue, are we addressing the underlying issue of business model incentives?) Current efforts seem to be focused on measures at the end-points of our social media systems:
  • Stopping disinformation at the source. We certainly should apply band-aids to prevent bad-actors from injecting our media with news, posts, and other items that are intentionally false and dishonest. Of course we should seek to block such items and those who inject them. Band-aids are useful when we find an open wound that germs are gaining entry through. But band-aids are still just band-aids.
  • Making it easier for individuals to recognize when items they receive may be harmful because they are not what they seem. We certainly should provide "immune markers" in the form of consumer-reports-like ratings of items and of the publishers or people who produce them (as many are seeking to do). Making such markers visible to users can help prime them to be more skeptical, and perhaps apply more critical thinking -- much like applying an antiseptic. But that depends on the willingness of users to pay attention to such markers and apply the antiseptic. There is good reason to doubt that will have more than modest effectiveness, given people's natural laziness and instinct for thinking fast rather than slow. (Many social media users "like" items based only on click-bait headlines that are often inflammatory and misleading, without even reading the item -- and that is often enough to cause those items to spread massively.)
These end-point measures are helpful and should be aggressively pursued, but we need to urgently pursue a more systemic strategy of defense. We need to address the problem of dissemination and amplification itself. We need to be much smarter about what gets spread -- from whom, to whom, and why.

Doing that means getting deep into the guts of how our media are filtered and disseminated, step by step, through the "viral" amplification layers of the media systems that connect us. That means integrating a cognitive immune system into the core of our social media platforms. Getting the platform owners to buy in to that will be challenging, but it is the only effective remedy.

Building a cognitive immune system -- the biological parallel

This perspective comes out of work I have been doing for decades, and have written about on this blog (and in a patent filing since released into the public domain). That work centers on ideas for augmenting human intelligence with computer support. More specifically, it is centers on augmenting the wisdom of crowds. It is based on the idea the our wisdom is not the simple result of a majority vote -- but results from an emergent process that applies smart filters that rate the raters and weight the ratings. That provides a way to learn which votes should be more equal than others (in a way that is democratic and egalitarian, but also merit-based). This approach is explained in the posts listed below. It extends an approach that has been developing for centuries.

Supportive of those perspectives, I recently turned to some work on biological immunity that uses the term "cognitive immune system." That work highlight the rich informational aspects of actual immune systems, as a model for understanding how these systems work at a systems level. As noted in one paper (see longer extract below*), biological immune systems are "cognitive, adaptive, fault-tolerant, and fuzzy conceptually." I have only begun to think about the parallels here, but it is apparent that the system architecture I have proposed in my other posts is at least broadly parallel, being also "cognitive, adaptive, fault-tolerant, and fuzzy conceptually." (Of course being "fuzzy conceptually" makes it not the easiest thing to explain and build, but when that is the inherent nature of the problem, it may also necessarily be the essential nature of the solution -- just as it is for biological immune systems.)

An important aspect of this being "fuzzy conceptually," is what I call The Tao of Truth. We can't definitively declare good-faith "speech" as "fake" or "false" in the abstract. Validity is "fuzzy" because it depends on context and interpretation. ("Fuzzy logic" recognizes that in the real world, it is often the case that facts are not entirely true or false but, rather, have degrees of truth.)  That is why only the clearest cases of disinformation can be safely cut off at the source. But we can develop a robust system for ranking the probable (fuzzy) value and truthfulness of speech, revising those rankings, and using that to decide how to share it with whom. For practical purposes, truth is a filtering process, and we can get much smarter about how we apply our collective intelligence to do our filtering. It seems the concepts of "danger" and "self/not-self" in our immune systems have a similarly fuzzy Tao -- many denizens of our microbiome that are not "self" are beneficial to us, and our immune systems have learned that we live better with them inside of us.

My proposals

Expansion on the architecture I have proposed for a cognitive immune system -- and the need for it -- are here:
  • The Tao of Fake News – the essential need for fuzziness in our logic: the inherent limits of experts, moderators, and rating agencies – and the need for augmenting the wisdom of the crowd (as essential to maintaining the intellectual openness of our democratic/enlightenment values).
(These works did not explicitly address the parallels with biological cognitive immune systems -- exploring those parallels might well lead to improvements on these strategies.)

To those without a background in the technology of modern information platforms, this brief outline may seem abstract and unclear. But as noted in these more detailed posts, these methods are a generalization of methods used by Google (in its PageRank algorithm) to do highly context-relevant filtering of search results using a similar rate the raters and weight the ratings strategy. (That is also "cognitive, adaptive, fault-tolerant, and fuzzy conceptually.") These methods not simple, but they are little stretch from the current computational methods of search engines, or from the ad targeting methods already well-developed by Facebook and others. They can be readily applied -- if the platforms can be motivated to do so.

Broader issues of support for our cognitive immune system

The issue of motivation to do this is crucial. For the kind of cognitive immune system I propose to be effective, it must be built deeply into the guts of our social media platforms (whether directly, or via APIs). As noted above, getting incumbent platforms to shift their business models to align their internal incentives with that need will be challenging. But I suggest it need not be as difficult as it might seem.
A related non-technical issue that many have noted is the need for education of citizens 1) in critical thinking, and 2) in the civics of our democracy. Both seem to have been badly neglected in recent decades. Aggressively remedying that is important, to help inoculate users from disinformation and sloppy thinking -- but that will have limited effectiveness unless we alter the overwhelmingly fast dynamics of our information flows (with the cognitive immune system suggested here) -- to help make us smarter, not dumber in the face of this deluge of information.

---
[Update 10/12:] A subsequent Times article by Sheera Frenkel, adds perspective on the scope and pace of the problem -- and the difficulty in definitively identifying items as fakes that can rightly be censored "because of the blurry lines between free speech and disinformation" -- but such questionable items can be down-ranked.
-----
*Background on our Immune Systems -- from the introduction to the paper mentioned above, "A Cognitive Computational Model Inspired by the Immune System Response" (emphasis added):
The immune system (IS) is by nature a highly distributed, adaptive, and self-organized system that maintains a memory of past encounters and has the ability to continuously learn about new encounters; the immune system as a whole is being interpreted as an intelligent agent. The immune system, along with the central nervous system, represents the most complex biological system in nature [1]. This paper is an attempt to investigate and analyze the immune system response (ISR) in an effort to build a framework inspired by ISR. This framework maintains the same features as the IS itself; it is cognitive, adaptive, fault-tolerant, and fuzzy conceptually. The paper sets three phases for ISR operating sequentially, namely, “recognition,” “decision making,” and “execution,” in addition to another phase operating in parallel which is “maturation.” This paper approaches these phases in detail as a component based architecture model. Then, we will introduce a proposal for a new hybrid and cognitive architecture inspired by ISR. The framework could be used in interdisciplinary systems as manifested in the ISR simulation. Then we will be moving to a high level architecture for the complex adaptive system. IS, as a first class adaptive system, operates on the body context (antigens, body cells, and immune cells). ISR matured over time and enriched its own knowledge base, while neither the context nor the knowledge base is constant, so the response will not be exactly the same even when the immune system encounters the same antigen. A wide range of disciplines is to be discussed in the paper, including artificial intelligence, computational immunology, artificial immune system, and distributed complex adaptive systems. Immunology is one of the fields in biology where the roles of computational and mathematical modeling and analysis were recognized...
The paper supposes that immune system is a cognitive system; IS has beliefs, knowledge, and view about concrete things in our bodies [created out of an ongoing emergent process], which gives IS the ability to abstract, filter, and classify the information to take the proper decisions.

Thursday, April 26, 2018

Architecting Our Platforms to Better Serve Us -- Augmenting and Modularizing the Algorithm

We dreamed that our Internet platforms would serve us miraculously, but now see that they have taken a wrong turn in many serious respects. That realization has reached a crescendo in the press and in Congress with regard to Facebook and Google's advertising-driven services, but it reaches far more deeply.

"Titans on Trial: Do internet giants have too much power? Should governments intervene?" -- I had the honor last night of attending this stimulating mock trial, with author Ken Auletta as judge and FTC Commissioner Terrell McSweeny and Rob Atkinson, President of the Information Technology and Innovation Foundation (ITIF) as opposing advocates (hosted by Genesys Partners). My interpretation of the the jury verdict (voted by all of the attendees, who were mostly investors or entrepreneurs) was: yes, most agree that regulation is needed, but it must be nuanced and smartly done, not heavy handed. Just how to do that will be a challenge, but it is a challenge that we must urgently consider.

I have been outlining views on this that go in some novel directions, but are generally consistent with the views of many other observers. This post takes a broad view of those suggestions, drawing from several earlier posts.

One of the issues touched on below is a core business model issue -- the idea that the ad-model of "free" services in exchange for attention to ads is "the original sin of the Internet." It has made users of Facebook and Google (and many others) "the product, not the customer," in a way that distorts incentives and fails to serve the user interest and the public interest. As the Facebook fiasco makes clear, these business model incentives can drive these platforms to provide just enough value to "engage" us to give up our data and attend to the advertiser's messages and manipulation and even to foster dopamine-driven addiction, but not necessarily to offer consumer value (services and data protection) that truly serves our interests.

That issue is specifically addressed in a series of posts in my other blog that focuses on a novel approach to business models (and regulation that centers on that), and those posts remain the most focused presentations on those particular issues:
This rest of this post adapts a broader outline of ideas previously embedded in a book review (on Neal Ferguson's "The Square and the Tower: Networks and Power from the Freemasons to Facebook," a historical review of power in the competing forms of networks and hierarchies). Here I abridge and update that post to concentrate on on our digital platforms. (Some complementary points on the need for new thinking on regulation -- and the need for greater tech literacy and nuance -- are in a recent HBR article, "The U.S. Needs a New Paradigm for Data Governance.")

Rethinking our networks -- and the algorithms that make all the difference

Drawing on my long career as a systems analyst/engineer/designer, manager, entrepreneur, inventor, and investor (including early days in the Bell System when it was a regulated monopoly providing "universal service"), I have recently come to share the fear of many that we are going off the rails.

But in spite of the frenzy, it seems we are still failing to refocus on better ways to design, manage, use, and govern our networks -- to better balance the best of hierarchy and openness. Few who understand technology and policy are yet focused on the opportunities that I see as reachable, and now urgently needed.

New levels of man-machine augmentation and new levels of decentralizing and modularizing intelligence can make these network smarter and more continuously adaptable to our wishes, while maintaining sensible and flexible levels of control -- and with the innovative efficiency of an open market.   We can build on distributed intelligence in our networks to find more nuanced ways to balance openness and stability (without relying on unchecked levels of machine intelligence). Think of it as a new kind of systems architecture for modular engineering of rules that blends top-down stability with bottom-up emergence, to apply checks and balances that work much like our representative democracy. This is a still-formative development of ideas that I have written about for years, and plan to continue into the future.

First some context. The crucial differences among all kinds of networks (including hierarchies) are in the rules (algorithms, code, policies) that determine which nodes connect, and with what powers. We now have the power to create a new synthesis. Modern computer-based networks enable our algorithms to be far more nuanced and dynamically variable. They become far more emergent in both structure and policy, while still subject to basic constraints needed for stability and fairness.

Traditional networks have rules that are either relatively open (but somewhat slow to change), or constrained by laws and customs (and thus resistant to change). Even our current social and information networks are constrained in important ways. Some examples:
  • The US constitution defines the powers and the structures for the governing hierarchy, and processes for legislation and execution, made resilient by its provisions for self-amendable checks and balances. 
  • Real-world social hierarchies have structures based on empowered people that tend to shift more or less slowly.
  • Facebook has a social graph that is emergent, but the algorithms for filtering who sees what are strictly controlled by, and private to, Facebook. (In January they announced a major change --  unilaterally -- perhaps for the better for users and society, if not for content publishers, but reports quickly surfaced that it had unintended consequences when tested.)
  • Google has a page graph that is given dynamic weight by the PageRank algorithm, but the management of that algorithm is strictly controlled by Google. It has been continuously evolving in important respects, but the details are kept secret to make it harder to game.
Our vaunted high-tech networks are controlled by corporate hierarchies (FANG: Facebook, Amazon, Netflix, and Google in much of the world, and BAT: Baidu, Alibaba, and Tencent in China) -- but are subject to limited levels of government control that vary in the US, EU, and China. This corporate control is a source of tension and resistance to change -- and a barrier to more emergent adaptation to changing needs and stressors (such as the Russian interference in our elections). These new monopolistic hierarchies extract high rents from the network -- meaning us, the users -- mostly indirectly, in the form of advertising and sales of personal data.

Smarter, more open and emergent algorithms -- APIs and a common carrier governance model

The answer to the question of governance is to make our network algorithms not only smarter, but more open to appropriate levels of individual and multi-party control. Business monopolies or oligarchies (or governments) may own and control essential infrastructure, but we can place limits on what they control and what is open. In the antitrust efforts of the past century governments found need to regulate rail and telephone networks as common carriers, with limited corporate-owner power to control how they are used, giving marketplace players (competitors and consumers) a share in that control. 

Initially this was rigid and regulated in great detail by the government, but the Carterfone decision showed how to open the old AT&T Bell System network to allow connection of devices not tested and approved by AT&T. Many forget how only AT&T phones could be used (except for a few cases of alternative devices like early fax machines that went through cumbersome and often arbitrary AT&T approval processes). Remember the acoustic modem coupler, needed because modems could not be directly connected? That changed when the FCC's decision opened the network up to any device that met defined electrical interface standards (using the still-familiar RJ11, a "Registered Jack").

Similarly only AT&T long-distance connections could be used, until the antitrust Consent Decree opened up competition among the "Baby Bells" and broke them off from Long Lines to compete on equal terms with carriers like MCI and Sprint. Manufacturing was also opened to new competitors.

In software systems, such plug-like interfaces are known as APIs (Application Program Interfaces), and are now widely accepted as the standard way to let systems interoperate with one another -- just enough, but no more -- much like a hardware jack does. This creates a level of modularity in architecture that lets multiple systems, subsystems, and components  interoperate as interchangeable parts -- extending the great advance of the first Industrial Revolution to software.

What I suggest as the next step in evolution of our networks is a new kind of common carrier model that recognizes networks like Facebook, Google, and Twitter as common utilities once they reach some level of market dominance. Then antitrust protections would mandate open APIs to allow substitution of key components by customers -- to enable them to choose from an open market of alternatives that offer different features and different algorithms. Some specific suggestions are below (including the very relevant model of sophisticated interoperablilty in electronic mail networks), but first, a bit more on the motivations.

Modularity, emergence, markets, transparency, and democracy

Systems architects have long recognized that modularity is essential to making complex systems feasible and manageable. Software developers saw from the early days that monolithic systems did not scale -- they were hard to build, maintain, or modify. (The picture here of the tar pits is from Fred Brooks classic 1972 book in IBM's first large software project.)  Web 2.0 extended that modularity to our network services, using network APIs that could be opened to the marketplace. Now we see wonderful examples of rich applications in the cloud that are composed of elements of logic, data, and analytics from a vast array of companies (such as travel services that seamlessly combine air, car rental, hotel, local attractions, loyalty programs, advertising, and tracking services from many companies).

The beauty of this kind of modularity is that systems can be highly emergent, based on the transparency and stability of published, open APIs, to quickly adapt to meet needs that were not anticipated. Some of this can be at the consumer's discretion, and some is enabled by nimble entrepreneurs. The full dynamics of the market can be applied, yet basic levels of control can be retained by the various players to ensure resilience and minimize abuse or failures.

The challenge is how to apply hierarchical control in the form of regulation in a way that limits risks, while enabling emergence driven by market forces. What we need is new focus on how to modularize critical common core utility services and how to govern the policies and algorithms that are applied, at multiple levels in the design of these systems (another, more hidden and abstract, kind of hierarchy). That can be done through some combination of industry self-regulation (where a few major players have the capability to do that, probably faster and more effectively than government), but by government where necessary (preferably only to the extent and duration necessary).

That obviously will be difficult and contentious, but it is now essential, if we are not to endure a new age of disorder, revolution, and war much like the age of religious war that followed Gutenberg (as Ferguson described). Silicon Valley and the rest of the tech world need to take responsibility for the genie they have let out of the bottle, and to mobilize to deal with it, and to get citizens and policymakers to understand the issues.

Once that progresses and is found to be effective, similar methods may eventually be applied to make government itself more modular, emergent, transparent, and democratic -- moving carefully toward "Democracy 2.0." (The carefully part is important -- Ferguson rightfully noted the dangers we face, and we have done a poor job of teaching our citizens, and our technologists, even the traditional principles of history, civics, and governance that are prerequisite to a working democracy.)

Opening the FANG walled gardens (with emphasis on Facebook and Google, plus Twitter)

This section outlines some rough ideas. (Some were posted in comments on an article in The Information by Sam Lessin, titled, "The Tower of Babel: Five Challenges of the Modern Internet.")

The fundamental principle is that entrepreneurs should be free to innovate improvements to these "essential" platforms -- which can then be selected by consumer market forces. Just as we moved beyond the restrictive walled gardens of AOL, and the early closed app stores (initially limited to apps created by Apple), we have unleashed a cornucopia of innovative Web services and apps that have made our services far more effective (and far more valuable to the platform owners as well, in spite of their early fears). Why should first movers be allowed to block essential innovation? Why should they have sole control and knowledge of the essential algorithms that are coming to govern major aspects of our lives? Why shouldn't our systems evolve toward fitness functions that we control and understand, with just enough hierarchical structure to prevent excessive instability at any given time?

Consider the following specific areas of opportunity.

Filtering rules. Filters are central to the function of Facebook, Google, and Twitter. As Ferguson observes, there are issues of homophily, filter bubbles, echo chambers, and fake news, and spoofing that are core to whether these networks make us smart or stupid, and whether we are easily manipulated to think in certain ways. Why do we not mandate that platforms be opened to user-selectable filtering algorithms (and/or human curators)? The major platforms can control their core services, but could allow users to select separate filters that interoperate with the platform. Let users control their filters, whether just by setting key parameters, or by substituting pluggable alternative filter algorithms. (This would work much like third party analytics in financial market data systems.) Greater competition and transparency would allow users to compare alternative filters and decide what kinds of content they do or do not want. It would stimulate innovation to create new kinds of filters that might be far more useful and smart.

For example, I have proposed strategies for filters that can help counter filter bubble effects by being much smarter about how people are exposed to views that may be outside of their bubble, doing it in ways that they welcome and want to think about. My post, Filtering for Serendipity -- Extremism, "Filter Bubbles" and "Surprising Validators" explains the need, and how that might be done. The key idea is to assign levels of authority to people based on the reputational authority that other people ascribe to them (think of it as RateRank, analogous to Google's PageRank algorithm). This approach also suggests ways to create smart serendipity, something that could be very valuable as well.

The "wisdom of the crowd" may be a misnomer when the crowd is an undifferentiated mob, but,  I propose seeking the wisdom of the smart crowd -- first using the crowd to evaluate who is smart, and then letting the wisdom of the smart sub-crowd emerge, in a cyclic, self-improving process (much as Google's algorithm improves with usage, and much as science is open to all, but driven by those who gain authority, temporary as that may be).

Social graphs: Why do Facebook, Twitter, LinkedIn, and others own separate, private forms of our social graph. Why not let other user agents interoperate with a given platform’s social graph? Does the platform own the data defining my social graph relationships or do I? Does the platform control how that affects my filter or do I? Yes, we may have different flavors of social graph, such as personal for Facebook and professional for LinkedIn, but we could still have distinct sub-communities that we select when we use an integrated multi-graph, and those could offer greater nuance and flexibility with more direct user control.

User agents versus network service agents: Email systems were modularized in Internet standards long ago, so that we compose and read mail using user agents (Outlook, Apple mail, Gmail, and others) that connect with federated remote mail transfer agent servers (that we may barely be aware of) which interchange mail with any other mail transfer agent to reach anyone using any kind of user agent, thus enabling universal connectivity.

Why not do much the same, to let any social media user agent interoperate with any other, using a federated social graph and federated message transfer agents? We could then set our user agent to apply filters to let us see whichever communities we want to see at any given time. Some startups have attempted to build stand-alone social networks that focus on sub-communities like family or close friends versus hundreds of more or less remote acquaintances. Why not just make that a flexible and dynamic option, that we can control at will with a single user agent? Why require a startup to build and scale all aspects of a social media service, when they could just focus on a specific innovation? (The social media UX can be made interoperable to a high degree across different user agents, just as email user agents handle HTML, images, attachments, emojis, etc. -- and as do competing Web browsers.)

Identity: A recurring problem with many social networks is abuse by anonymous users (often people with many aliases, or even just bots). Once again, this need not be a simple binary choice. It would not be hard to have multiple levels of participant, some anonymous and some with one or more levels of authentication as real human individuals (or legitimate organizations). First class users would get validated identities, and be given full privileges, while anonymous users might be permitted but clearly flagged as such, with second class privileges. That would allow users to be exposed to anonymous content, when desired, but without confusion as to trust levels. Levels of identity could be clearly marked in feeds, and users could filter out anonymous or unverified users if desired. (We do already see some hints of this, but only to a very limited degree.)

Value transfers and extractions: As noted above, another very important problem is that the new platform businesses are driven by advertising and data sales, which means the consumer is not the customer but the product. Short of simply ending that practice (to end advertising and make the consumer the customer), those platforms could be driven to allow customer choice about such intrusions and extractions of value. Some users may be willing opt in to such practices, to continue to get "free" service, and some could opt out, by paying compensatory fees -- and thus becoming the customer. If significant numbers of users opted to become the customer, then the platforms would necessarily become far more customer-first -- for consumer customers, not the business customers who now pay the rent.

I have done extensive work on alternative strategies that adaptively customize value propositions and prices to markets of one -- a new strategy for a new social contract that can shape our commercial relationships to sustain services in proportion to the value they provide, and our ability to pay, so all can afford service. A key part of the issue is to ensure that users are compensated for the value of the data they provide. That can be done as a credit against user subscription fees (a "reverse meter"), at levels that users accept as fair compensation. That would shift incentives toward satisfying users (effectively making the advertiser their customer, rather than the other way around). This method has been described in the Journal of Revenue and Pricing Management: “A novel architecture to monetize digital offerings,” and very briefly in Harvard Business Review. More detail is my FairPayZone blog and my book (see especially the posts about the Facebook and Google business models that are listed in the opening section, above, and again at the end.*)

Analytics and metrics: we need access to relevant usage data and performance metrics to help test and assess alternatives, especially when independent components interact in our systems. Both developers and users will need guidance on alternatives. The Netflix Prize contests for improved recommender algorithms provided anonymized test data from Netflix to participant teams. Concerns about Facebook's algorithm, and the recent change that some testing suggests may do more harm than good, point to the need for independent review. Open alternatives will increase the need for transparency and validation by third parties.

(Sensitive data could be restricted to qualified organizations, with special controls to avoid issues like the Cambridge Analytica mis-use. The answer to such abuse is not greater concentration of power in one platform, as Maurice Stucke points out in Harvard Business Review, "Here Are All the Reasons It’s a Bad Idea to Let a Few Tech Companies Monopolize Our Data." (Facebook has already moved toward greater concentration of power.)

If such richness sounds overly complex, remember that complexity can be hidden by well-designed user agents and default rules. Those who are happy with a platform's defaults need not be affected by the options that other users might enable (or swap in) to customize their experience. We do that very successfully now with our choice of Web browsers and email user agents. We could have similar flexibility and choice in our platforms -- innovations that are valuable can emerge for use by early adopters, and then spread into the mainstream if success fuels demand. That is the genius of our market economy -- a spontaneous, emergent process for adaptively finding what works and has value -- in ways more effective than any hierarchy (as Ferguson extols, with reference to Smith, Hayek, and Levitt).

Augmentation of humans (and their networks)

Another very powerful aspect of networks and algorithms that many neglect is  the augmentation of human intelligence. This idea dates back some 60 years (and more), when "artificial intelligence" went through its first hype cycle -- Licklider and Engelbart observed that the smarter strategy is not to seek totally artificial intelligence, but to seek hybrid strategies that draw on and augment human intelligence. Licklider called it "man-computer symbiosis, and used ARPA funding to support the work of Engelbart on "augmenting human intellect." In an age of arcane and limited uses of computers, that proved eye-opening at a 1968 conference ("the mother of all demos"), and was one of the key inspirations for modern user interfaces, hypertext, and the Web.

The term augmentation is resurfacing in the artificial intelligence field, as we are once again realizing how limited machine intelligence still is, and that (especially where broad and flexible intelligence is needed) it is often far more effective to seek to apply augmented intelligence that works symbiotically with humans, retaining human visibility and guidance over how machine intelligence is used.

Why not apply this kind of emergent, reconfigurable augmented intelligence to drive a bottom up way to dynamically assign (and re-assign) authority in our networks, much like the way representative democracy assigns (and re-assigns) authority from the citizen up? Think of it as dynamically adaptive policy engineering (and consider that a strong bottom-up component will keep such "engineering" democratic and not authoritarian). Done well, this can keep our systems human-centered.

Reality is not binary:  "Everything is deeply intertwingled"

Ted Nelson (who coined the term "hypertext" and was another of the foundational visionaries of the Web), wrote in 1974 that "everything is deeply intertwingled." As he put it, "Hierarchical and sequential structures, especially popular since Gutenberg, are usually forced and artificial. Intertwingularity is not generally acknowledged—people keep pretending they can make things hierarchical, categorizable and sequential when they can't."

It's a race:  augmented network hierarchies that are emergently smart, balanced, and dynamically adaptable -- or disaster

If we pull together to realize this potential, we can transcend the dichotomies and conflicts that are so wickedly complex and dangerous. Just as Malthus failed to account for the emergent genius of civilization, and the non-linear improvements it produces, many of us discount how non-linear the effect of smarter networks, with more dynamically augmented and balanced structures, can be. But we are racing along a very dangerous path, and are not being nearly smart or proactive enough about what we need to do to avert disaster. What we need now is not a top-down command and control Manhattan Project, but a multi-faceted, broadly-based movement, with elements of regulation, but primarily reliant on flexible, modular architectural design.


[Update 12/14/20] A specific proposal - Stanford Working Group on Platform Scale

An important proposal that gets at the core of the problems in media platforms was published in Foreign AffairsHow to Save Democracy From Technology, by Francis Fukuyama and others. See also the report of the Stanford Working Group. The idea is to let users control their social media feeds with open market interoperable filters. That is something I proposed here (in the "Filtering rules" section, above). Other regulatory proposals that include some of the suggestions made here are summarized in Regulating our Platforms -- A Deeper Vision.

---
See the Selected Items tab for more on this theme.

---

Coda:  On becoming more smartly intertwingled

Everything in our world has always been deeply intertwingled. Human intellect augmented with technology enables us to make our world more smartly intertwingled. But we have lost our way, in the manner that Engelbart alluded to in his illustration of de-augmentation -- we are becoming deeply polarized, addicted to self-destructive dopamine-driven engagement without insight or nuance. We are being de-augmented by our own technology run amok.


(I plan to re-brand this blog as "Smartly Intertwingled" -- that is the objective that drives my work. The theme of "User-Centered Media" is just one important aspect of that.)


--------------------------------------------------------------------------------------------

*On business models - FairPay (my other blog):  As noted above, a series of posts in my other blog focus on a novel approach to business models (and regulation that centers on that), and those posts remain my best presentation on those issues: