What Determines the Upper Bound of Software Quality: Human Desire, AI Intelligence, and Shared Knowledge Design

Introduction: What Constrains Software Quality?

Software is fundamentally a tool for solving human problems. Building better tools requires relentless exploration and articulation of problem awareness — understanding what the problem is and what we actually want.

In The Shifting Bottlenecks of Software Development, I outlined how bottlenecks have migrated from computational resources to cognitive resources to context windows, suggesting that beyond these lie "questions that quantitative constraints cannot answer." In The "Specification" Problem in the AI Era, I argued that translation from intent to implementation is non-deterministic, while the reverse direction carries lower translation costs. However, neither of these discussions questioned the quality of intent itself.

This article ventures into a more fundamental question: What determines the upper bound of software quality? Is it human intelligence that constrains it, or AI capability? And when AI surpasses human intelligence, how does that upper bound change?

Is Human Articulation Ability the Bottleneck?

There are at least three reasons why humans struggle to articulate their desires:

1. Cognitive constraints — People often don't even know what they want (latent needs)
2. Vocabulary constraints — They're aware of what they want but lack the right words
3. Context dependency — They can't know what they want until they actually use it; knowledge is embedded in situations

Vocabulary constraints (2) are easily addressed by AI. Inferring intent from ambiguous expressions and converting them to precise language is already somewhat possible with current LLMs.

The real challenges are 1 and 3. Latent needs are pre-articulation problems — you can't verbalize what you're not even aware of. Context dependency only surfaces through the act of usage. These aren't language ability issues; they're structural properties of desire itself.

Can Superintelligent AI "Read" Human Desires?

When AI intelligence far surpasses human capability, could it directly read desires without the mediation of language?

The Optimistic View

A superintelligent AI could infer even unarticulated desires from behavioral data, past choice patterns, and context. Just as Spotify discovers "preferences you didn't know you had," superintelligence extends this to every domain. The articulation bottleneck would vanish entirely.

The Gap Between Revealed and True Preferences

But are desires inferred from observed behavior the same as what a person truly wants?

As behavioral economics demonstrates, human behavior is systematically biased. "What you click on frequently" and "what you truly want" are different things. When social media algorithms "optimized for user desires," engagement went up but happiness went down — this is the gap between revealed preference and true preference.

This gap may not be bridgeable by intelligence alone. An is/ought gap lies at the heart of it — the logical impossibility of deriving "what should be done" or "what should be desired" from observations of "what is being done."

Can Feedback Loops Compensate for Intelligence Differences?

Can iterative feedback after usage produce excellent outcomes regardless of differences in human intelligence?

Partially yes. However, the effective range of feedback is limited to what users can perceive. Anyone can say "this UI is hard to use." But "this data model compromises future extensibility" requires a certain level of knowledge to judge.

This is where superintelligent AI could play a fascinating role: feedback translation. If an AI can infer whether a user's complaint of "it feels slow" stems from an N+1 query problem, missing indexes, or frontend re-rendering, then feedback quality would no longer depend on user intelligence. Humans emit the "signal," but AI handles the "interpretation."

Beyond Individual Optimization

The discussion so far suggests a future where software is ultimately optimized for individuals. AI reads personal desires, translates feedback, and delivers software optimized for each person.

However, not all software can be individually optimized.

Domains Where Shared Data Creates Value

Market data, epidemiological data, recommendation engines — these only become information when data from multiple entities is combined cross-sectionally. Individual data alone sees signals drowned in noise.

In these domains, individual customization has inherent limits. A natural separation emerges: the backend (data collection and analysis) is collective, while the frontend (presentation and interpretation) is individual.

Will Data Design Become Unnecessary?

Here's an interesting thought experiment: if we could retain infinite events and instantly analyze them along any axis, would data design (schema, normalization, indexes) lose its meaning?

Current data design is essentially pre-optimization against computational resource constraints. We predict "queries like this will come later" and determine data structure in advance. If computational constraints vanished, we could keep all events in raw form and aggregate at query time. This could be seen as the extreme form of event sourcing.

Yet even if computational constraints disappear, data "design" persists — not as structural design (schema) but as meaning design (ontology).

• What is "revenue"? Does it include returns? Is it tax-inclusive? — This isn't a computational problem; it's a conceptual definition problem
• Cross-entity data analysis requires shared vocabulary and semantic agreement
• Who can see what is a social constraint that computational resources cannot resolve

Can AI Design Shared Knowledge?

Here a crucial question emerges: what if AI could handle the design of shared knowledge — conceptual definitions and semantic agreements?

As Cognitive Capability: Yes

From a cognitive capability perspective, AI can likely perform ontology design better than humans.

• Analyzing how the word "revenue" is actually used across all organizational communications and detecting semantic drift
• Articulating implicit assumptions of multiple stakeholders and proposing common definitions
• Reverse-engineering optimal conceptual boundaries from how existing data is actually used

This is design based on analysis at a scale impossible for humans. A single person cannot grasp how words are used across an entire organization, but AI can.

But Ontology Design Isn't Just a Cognitive Problem

Consider the question "Should revenue include returns?" This isn't an intellectual capability problem — it's a stakeholder alignment problem. The sales department doesn't want to include returns (it hurts their numbers). The accounting department wants to include them (for accurate financial reporting).

Ontology design has two layers:

The Evolution of Human-AI Roles

What if AI could also handle normative judgments — value decisions and stakeholder alignment? Here we can see a staged evolution of human-AI roles.

Level 1: AI as Tool

AI executes according to human instructions. Humans set objectives, specify means, and evaluate results. Most current AI applications operate here.

The upper bound of software quality depends on human articulation ability and judgment.

Level 2: AI as Translator

AI handles desire reading and feedback translation. Humans need only express vague intentions, and AI converts them into structured designs. AI also interprets feedback.

The upper bound of software quality depends on the clarity of human desires. Differences in articulation ability are compensated, but the resolution of what is desired still depends on humans.

Level 3: AI as Mediator

In shared knowledge design, AI understands each stakeholder's interests and proposes compromises acceptable to all. AI designs the ontology of shared knowledge.

The upper bound of software quality depends on whether humans accept the proposals. Not AI intelligence, but trust in AI and legitimacy become the constraints.

Level 4: AI as Decision-Maker

AI makes design decisions including value judgments, and humans follow. "Revenue includes returns," AI decides.

The upper bound of software quality depends on AI intelligence. Human intelligence is no longer the bottleneck. However, this requires significant delegation of authority from humans to AI.

Level 5: AI as Purpose-Setter

AI proposes: "We should abolish the concept of revenue entirely and use these metrics instead." AI creates value criteria that humans never envisioned.

The upper bound of software quality — can no longer be defined. Since AI updates the very definition of "quality," the concept of an upper bound dissolves.

The Dissolution of the Principal

What lies at the end of this staged progression is a structural transformation of the principal-agent problem.

In the traditional model, humans were the principal (delegator) and AI the agent (delegate). The agent efficiently achieves objectives, but it is the principal who sets the objectives.

From Level 4 onward, however, this relationship becomes ambiguous. Humans already delegate many judgments externally — health decisions to doctors, legal judgments to lawyers, asset management to financial advisors. Delegation to AI is an extension of this pattern.

Going further, human objectives themselves are shaped by environment and experience. By designing the environment, AI can indirectly shape human objectives. Just as recommendation algorithms transform preferences, AI-designed software shapes human desires. The distinction between principal and agent dissolves.

Conclusion: The Upper Bound Moves

The answer to "What determines the upper bound of software quality?" changes depending on the stage of the human-AI relationship.

In the short term, human intelligence is the bottleneck. The ability to articulate desires, the quality of feedback, design judgment for shared knowledge — all depend on human cognitive ability. AI supplements human limitations as a translator, but humans define the upper bound.

In the medium term, trust and delegation become the constraints. Even when AI intelligence is sufficient, social and institutional consensus on "Can we really leave this to AI?" doesn't keep pace. The upper bound is determined not by technical limitations but by the limits of human trust.

In the long term, the question of "upper bound" itself changes meaning. When AI participates in purpose-setting, the very definition of software "quality" becomes fluid. It's not that the upper bound moves — the scale for measuring it changes entirely.

However, at every stage of this progression, one thing remains certain: there are things you can only learn by trying. Context-dependent knowledge only surfaces through action. No matter how intelligent AI becomes, the need for feedback loops doesn't disappear. As long as software is a tool for humans, the cycle of humans using it, reacting, and learning from those reactions is inescapable.

The structure we might call "spec-implementation co-evolution" applies to the human-AI relationship as well. Human desires and AI intelligence co-evolve through interaction. Neither one alone determines the upper bound — it is the quality of the feedback loop between them that determines software's ultimate reach.

Related Articles

• The "Specification" Problem in the AI Era — Translation asymmetry and the SSoT discussion
• The True Nature of "Specifications" — The Requirements, Contract, and Implementation framework