Performance.
Reliability.
Efficiency.
Three views of the same decision: how much useful work can a system deliver, how consistently, and at what cost?
PRE is my shorthand for questions that recur in infrastructure data science. A faster component matters when it improves the work people actually need done. A cheaper run matters when it still meets the requirements. Reliability belongs inside that calculation.
My experience in hyperscale infrastructure has made the connection between measurement and decision central to how I approach these problems. This is a personal framework for that work; the example below uses invented numbers.
Start with the unit of useful work
For an inference service, that might be a response that meets a quality threshold and arrives within a latency target. For a batch pipeline, it might be a correctly completed job before its deadline. Those definitions are choices. They should be explicit before a throughput chart becomes a recommendation.
Performance
How much work reaches the required standard, and how long does it take?
Reliability
How often does the service meet that standard, including under changing load and failures?
Efficiency
What resources are consumed for each acceptable result?
More output, fewer useful results
Illustrative arithmetic only. These are hypothetical configurations, not Microsoft or Facebook measurements.
| Measure | A | B |
|---|---|---|
| Attempts | 1,000 | 1,200 |
| Meet quality and latency requirements | 90% | 70% |
| Acceptable results | 900 | 840 |
| Cost per acceptable result | $0.111 | $0.119 |
B delivers 20% more attempts but about 7% fewer acceptable results. Cost per acceptable result rises about 7%. A dashboard centered on raw throughput could celebrate the wrong change.
Cost per useful result = total cost ÷ results meeting the agreed requirements
This ratio is a starting point. It does not capture the cost of a particularly harmful failure, nor make two differently valuable requests interchangeable. If the service must meet a minimum reliability target, enforce that constraint before ranking configurations by cost.
Turn the question into an experiment
- Name the decision. What would change: capacity, a serving configuration, a model, or a scheduling policy?
- Define acceptable work. Specify quality, latency, availability, and the populations that cannot be hidden in an average.
- Measure the baseline. Include failed attempts, retries, idle capacity, and the cost boundary used in the comparison.
- Change one decision and observe. Use a comparison that accounts for load and workload mix, with a reversible rollout.
- Check the outcome. Compare useful work and cost, inspect failure cases, and decide whether to keep, revise, or reverse the intervention.
The aim is a decision whose reasoning can be checked. An aggregate improvement that disappears for an important workload is a reason to refine the recommendation.
The same habit at a smaller scale
In Axolotl Tower Defense, the baseline passed and a modest sensitivity test still exposed fragility. It is a smaller-scale reminder to ask what an aggregate number leaves out.
For professional background and a way to get in touch, see About & contact.