and operating AI products involves making trade-offs. For instance, a higher-quality product may take more time and resources to construct, while complex inference calls could also be slower and costlier. These trade-offs are a natural consequence of the elemental economic notion of scarcity, that our potentially unlimited wants can only be partially satisfied by a limited set of obtainable resources. In this text, we are going to borrow an intuitive triangle framework from project management theory to explore key trade-offs that builders and users of AI products should navigate at design- and run-time, respectively.
A Primer on Iron Triangles
The tensions between project scope, cost, and time have been studied extensively by academics and practitioners in the sector of project management since a minimum of the Nineteen Fifties. Efforts to visually represent the tensions (or trade-offs) between these three quality dimensions have resulted in a triangular framework that goes by many names, including the “iron triangle,” the “triple constraint,” and the “project management triangle.”
The framework makes just a few key points:
- It is necessary to investigate the trade-offs between project (what advantages, latest features, or functionality will the project deliver), (when it comes to monetary budget, human effort, IT costs), and (project schedule, time to delivery).
- Project cost is a function of scope and time (e.g., larger projects and shorter delivery time frames will cost more), and as per the so-called , “you get what you pay for.”
- In an environment where resources are fundamentally scarce, it could be difficult to concurrently minimize cost and time while maximizing scope. This example is neatly captured by the phrase “Good, fast, low-cost. Select two,” which is usually attributed (albeit without solid evidence) to Victorian art critic John Ruskin. Project managers thus are inclined to be highly alert to (adding more features to the project scope than was previously agreed without adequate governance), which might cause project delays and budget overruns.
- In any given project, there could also be various degrees of flexibility in levels of scope, cost, and time which might be considered acceptable by stakeholders. It could subsequently be possible to regulate a number of of those dimensions to derive different acceptable configurations for the project.
The next video explains the usage of the triangle framework in project management in additional detail:
Within the context of AI product development, the triangle framework lends itself to the exploration of trade-offs each at design-time (when the AI product is built), and at run-time (when the AI product is utilized by customers). In the next sections, we are going to look more closely at each of those two scenarios in turn.
Trade-Offs at Design-Time
Figure 1 shows a variant of the iron triangle that captures trade-offs faced by an AI product team at design-time.
The three dimensions of the triangle are:
- Feature scope () of the AI product measured in story points, function points, or feature units.
- Development cost () when it comes to person-days of human effort (PM, engineering, UX, data science), and monetary costs of staffing (experienced developers could have higher fully loaded costs) and IT (cloud resources, GPUs for training AI models).
- Time to market (), e.g., in weeks or months.
We are able to theorize the next minimal model of the triple constraint at design-time:
The event cost is proportional to the ratio of scope and time, and is a positive scalar factor representing productivity. The next value of implies a lower design-time cost per unit scope per unit time, and hence greater design-time productivity. The model matches our basic intuition: as tends to infinity (or tends to zero), tends to zero (i.e., stretching the project timeline or cutting down the scope makes the project cheaper).
For instance, suppose that our project consists of constructing an AI product value 300 story points, in a 100-day timeframe, with a productivity factor of 0.012. Assuming a totally loaded cost of $500 per story point, the minimal model suggests that we should always budget around $125k to ship the product:
The minimal model encapsulates the physics-like core of the design-time triple constraint. Indeed, the model is harking back to the equation taught in class linking distance (), velocity (), and time (t), i.e., d = v*t, which relies on some necessary assumptions (e.g., constant velocity, straight-line motion, continuous measurement of time). In our design-time model, we assume constant productivity (i.e., doesn’t vary), a linear trade‑off (scope grows linearly with time and value), and no external shocks (e.g., rework, reorgs, pivots).
Prolonged versions of the design-time model could consider:
- Fixed costs (e.g., a baseline overhead for planning, governance, infrastructure provision), which imply a lower certain for the overall design-time cost.
- Limited impact of accelerating staffing beyond a certain point. As observed by Fred Brooks in his 1975 book , “Adding manpower to a late software project makes it later.”
- Non-linear productivity (e.g., because of rushing or slowing down in several project phases), which might influence the connection between cost and the scope-time ratio.
- Explicit accounting of AI quality standards to permit transparent tracking of success metrics (e.g., adherence to regulatory requirements and repair level agreements with customers). Currently, the accounting happens not directly by attribution to the productivity factor and scope.
- The connection between productivity and the AI product team’s learning curve, as experience, process repetition, and code reuse make the event more efficient over time.
- Accounting for net value (i.e., advantages minus costs) or return on investment (ROI) reasonably than development costs alone.
- Factoring within the sharing of scarce resources across multiple AI products being developed in parallel. This may involve taking a portfolio perspective of AI products under development at any given time.
Trade-Offs at Run-Time
Figure 2 shows a variant of the iron triangle capturing trade-offs faced by customers or users of an AI product at run-time.
The three dimensions of this triangle are:
- Response quality () of the AI product measured when it comes to predictive accuracy, BLEU/ROUGE rating, or another task-specific quality metric.
- Inference costs () when it comes to dollars or cents per inference call, GPU seconds converted to dollars, or energy costs.
- Latency of inference () in milliseconds, seconds, etc.
We are able to theorize the next minimal model of the triple constraint at run-time:
The inference cost is proportional to the ratio of response quality and latency, and is a positive scalar factor representing system efficiency. The next value of implies a lower cost for a similar response quality and latency. Again, the model aligns with our basic intuition: as tends to zero (or tends to infinity), tends to infinity (i.e., an AI product that returns real-time, high-quality responses will likely be costlier than the same product delivering slower, inferior responses).
For instance, suppose that an AI product consistently achieves 90% predictive accuracy with a median response latency of 0.5 seconds. Assuming an efficiency factor of 180, we are able to expect the inference cost to be around one cent:
Prolonged versions of the run-time model could consider:
- Baseline fixed costs (e.g., of model loading, pre- and post-processing of user requests).
- Variable scaling costs because of a non-linear relationship between cost and quality (e.g., going from 80% to 95% accuracy could also be easier than going from 95% to 99%). This might also capture a type of diminishing returns on successive product optimizations.
- Stochastic nature of quality, which might vary depending on the input (“garbage in, garbage out”). This could be done through the use of the expected value of quality, , as an alternative of an absolute value within the triple constraint model; see this text for a deep dive on expected value evaluation in AI product management.
- Fixed and variable latency overheads. Inference cost might be modeled as a function of latency, accounting for queuing delays, network hops, etc.
- Effects of throughput and concurrency. The fee per inference might be lower for batched inferences (because of a form of amortization of costs across inferences in a batch) or higher if there’s network congestion.
- Explicit accounting for component efficiencies of the AI algorithm (because of an optimized model architecture, use of pruning, or quantization), hardware (GPU/TPU performance), and energy (electricity usage per FLOP) by decomposing the efficiency factor accordingly.
- Dynamic adaptation of the efficiency factor with respect to load, hardware, or type/degree of optimizations. E.g., efficiency could improve with caching or model distillation and deteriorate under heavy load because of resource throttling or blocking.
Finally, the choices made at design-time can shape the situation and forms of decisions that could be made at run-time. As an example, the product team may select to speculate significant resources in training a comprehensive foundation model, which could be prolonged via in-context learning at run-time; in comparison with a traditional machine learning algorithm resembling a random forest, the muse model is a design-time alternative which will allow for higher response quality at run-time, albeit at a potentially higher inference cost. Design-time investments in clean code and efficient infrastructure could increase the run-time system efficiency factor. The alternative of cloud provider could determine the minimum inference cost achievable at run-time. It’s subsequently vital to contemplate the design- and run-time trade-offs jointly in a holistic manner.
The Wrap
As this text demonstrates, the iron triangle from project management theory could be repurposed to supply easy yet powerful frameworks for analyzing design- and run-time trade-offs in AI product development. The design-time iron triangle could be utilized by product teams to make decisions about budgeting, resource allocation, and delivery planning. The complementary run-time iron triangle offers several insights into how the connection between inference costs, response quality, and latency can affect product adoption and customer satisfaction. Since design-time decisions can constrain run-time optionality, it will be important to take into consideration design- and run-time trade-offs jointly from the outset. By recognizing the trade‑offs early and dealing around them, product teams and their customers can create more value from the design and use of AI.
