How to measure a maintenance software pilot

In this guide

Write down the decision first

Choose the workflow and decision the pilot should support. For a fleet, that could be whether every reported defect receives an owner and a verified resolution. For a hotel, it could be whether room readiness is checked after linked repairs close. Select one primary outcome and record its definition before collecting results.

Record a baseline and eligible population

List the sites, assets, users and dates included. Count the work that actually qualifies for the measure. For on-time completion, define which due dates are used and how cancelled, rescheduled and still-open jobs are handled. Preserve source exports so someone else can reconstruct the numerator and denominator.

Separate adoption from outcomes

A signup is not activation, and an opened page is not completed work. Track the first actual asset, approved procedure, scheduled task and completed work record. Then inspect repeat use. Operational changes such as fewer missed tasks should be measured separately from software usage. Seasonal demand, staffing and changes in asset mix can explain apparent improvements.

Publish evidence with consent and limits

A case study should identify the observation window, actual workflow, data source and limitations. Obtain written permission for any customer name, logo, quote or screenshot. Redact personal and commercially sensitive details. There are no customer benchmark results on this page; the downloadable worksheet is a collection framework for future evidence.

Choose a question before collecting a success story

Maintenance evidence is useful when it supports a specific decision. A pilot might ask whether a new request process produces clearer locations, whether technicians can complete work without follow-up calls or whether a planner can identify due tasks consistently. Each question requires different records. Starting with a broad ambition such as “improve efficiency” makes it too easy to select whichever number looks favourable later.

AI-generated editorial illustration; not a customer photograph or product screenshot.

Write the primary question in a sentence and identify the person who will use the answer. A maintenance manager deciding whether to expand a workflow needs different evidence from an administrator checking whether an import completed correctly. Both are valid, but they should not be combined into one unexplained success score.

Keep a small number of secondary questions for context. For example, a request pilot can measure location clarity while also recording reviewer effort. This helps identify a tradeoff without allowing dozens of unrelated metrics to obscure the main decision.

Define the unit of observation

Decide whether each observation is an asset, a work order, a request, a production run or a user action. The unit determines what can be counted and compared. Ten visits to a page are not ten completed maintenance jobs, and several requests about one fault are not necessarily several distinct equipment failures.

AI-generated editorial illustration; not a customer photograph or product screenshot.

Give each observation a stable reference so it can be traced to its source. If records from different systems are combined, preserve both the source identity and the reviewed mapping. This is especially important when an imported history record uses a different identifier from the current work-order system.

Record the inclusion rule before calculating results. A work-order measure may include only tasks due within a defined period, while an asset measure may include only equipment operating at the selected site. Someone else should be able to apply the same rule and obtain the same eligible population.

Establish the baseline with the same definitions

A baseline describes the process before the change using the measure you intend to evaluate afterward. Preserve the source records and the calculation method. A number recalled from memory can provide context, but it is a weak basis for a precise improvement claim.

AI-generated editorial illustration; not a customer photograph or product screenshot.

Use comparable scope where possible. If the baseline covers one site and the later period covers three, a difference may reflect the population rather than the workflow. Likewise, a quieter operating period can reduce work volume or failures without any product effect. Record those changes so the comparison remains understandable.

If a reliable baseline does not exist, say so. The pilot can still establish current performance and data quality, then use that as the starting point for future observation. Inventing a historical estimate simply to produce a before-and-after percentage weakens the evidence.

Keep adoption measures separate from operational outcomes

Adoption measures describe whether the software is being used. Examples include creating the first asset, completing an initial work order and returning to perform the workflow again. These are useful indicators of whether a rollout has reached actual use rather than stopping at registration.

Operational outcomes describe the maintenance process or equipment context. Examples include requests requiring clarification, overdue work or elapsed time to a defined review decision. Adoption can support an outcome, but an increase in logins does not itself establish better maintenance performance.

Track both without collapsing them. A team may use the application regularly while struggling with unclear procedures, or may complete a small number of high-quality records during an early pilot. The combination helps explain what happened and where further work is needed.

Define on-time completion precisely

For an on-time measure, identify which work orders were due in the observation window and which due date is authoritative. Then define what counts as completion and how cancellations, rescheduling and still-open work are handled. These choices materially affect the result.

An illustrative sample might contain 40 eligible jobs, of which 32 were completed by the defined deadline. The on-time completion rate is 32 divided by 40, or 80 percent. If five late jobs are removed from the denominator after the result is known, the new percentage no longer describes the original population.

Preserve the underlying counts alongside the percentage. A reviewer should be able to see how many jobs were included and inspect the exceptions. The aim is to understand the process, including work that was delayed for a valid reason, rather than improve the number by changing the rules silently.

Measure clarity with a reviewable rubric

Record quality can be evaluated with a simple rubric. For a maintenance request, the rubric might ask whether the location is identifiable, the observation is understandable and the next action has an owner. For a completed job, it might ask whether findings, action and verification are recorded sufficiently for the reviewer.

Define the criteria in plain language and test them on a few examples. Two reviewers should be able to apply the rubric with reasonable consistency. If they disagree frequently, clarify the criterion before treating the score as a stable measure.

Keep the sample and review notes. For example, 18 of 20 records passing a first review represents 90 percent for that sample, but the two failures often contain the most useful improvement information. Describe why they failed and what change would make the record easier to interpret.

State the clock used for time measures

Time to first review, time to assignment, active repair time and elapsed time to restoration are different measures. Name the start and end events explicitly. A report labelled “response time” without those definitions can be interpreted in several incompatible ways.

Choose whether the measure uses elapsed hours or staffed hours and apply the choice consistently. A request arriving outside the published review schedule may look slow in elapsed time while following the agreed operating process. Neither convention is automatically wrong, but the reader needs to know which one was used.

Inspect the distribution as well as the average. A small number of very old items may require attention even when the mean looks acceptable. Use the available records to identify those cases and their reasons rather than relying on a single summary value.

Preserve operating exposure for reliability comparisons

Failure counts need context. A fleet operating twice as many kilometres can record more incidents without necessarily having a worse incident rate per unit of exposure. Similarly, a machine used for fewer hours may show fewer failures simply because it operated less.

Record the relevant exposure and unit for the defined population. For an operating-hours MTBF calculation, preserve the total operating hours and counted failures under a consistent definition. Do not substitute the calendar gap between incidents without explaining the difference in meaning.

When exposure is missing or inconsistent, limit the claim. The team can still report the observed count and the data limitation, but should not present an unsupported rate as a precise reliability result. Improving the source record may be the most useful outcome of the first measurement period.

Compare unequal production runs using the underlying quantities

Production runs can differ in planned duration, product and ideal cycle time. A simple average of their OEE percentages can therefore misrepresent the combined productive contribution. Preserve the quantities needed to reconstruct the aggregate rather than treating every run percentage as equally weighted.

AI-generated editorial illustration; not a customer photograph or product screenshot.

PreventiveHQ's OEE workflow uses the recorded ideal cycle time, good units and planned production time to establish the relevant contribution. The OEE guide explains the formulas and worked examples. Keep the unit conversion explicit, particularly when ideal cycle time is entered in seconds and production time in minutes.

Use an example with unequal run lengths when reviewing the calculation. It is easier to detect an inappropriate average when one run represents much more production exposure than another. A plausible percentage is not enough; the reviewer should understand how the underlying quantities produce it.

Record changes that can explain the result

A before-and-after comparison does not isolate the software's effect by itself. Staffing, production demand, asset replacement, procedure changes and reporting behaviour may change during the same period. Record material changes so that the result can be interpreted with the appropriate context.

AI-generated editorial illustration; not a customer photograph or product screenshot.

For example, a request form may increase the number of recorded defects because reporting became easier. That does not automatically mean equipment condition worsened. Conversely, fewer recorded requests may reflect lower adoption rather than fewer problems. The evidence needs enough context to distinguish those explanations.

Describe the comparison as observational unless the evaluation design supports a stronger causal claim. A useful observational result can still guide a rollout decision. Its value comes from clarity about what was measured and what alternative explanations remain, rather than from overstating certainty.

Keep missing data visible

Missing values should not silently become zeros or passing results. A blank repair duration may mean the time was not recorded, not that the job took no time. An unanswered checklist step may mean it was not inspected, not that the condition was acceptable.

Count missing observations and explain how they affect the measure. If a time comparison excludes records with missing timestamps, state how many were excluded and why. Otherwise, the reported result can appear stronger simply because difficult cases were omitted.

Use missingness as a process-improvement signal. Repeatedly absent units, locations or closure notes may point to a confusing form or unclear responsibility. Fixing that source problem can improve the quality of future decisions even before an operational outcome changes.

Reconcile the calculation with source records

Keep a reproducible calculation sheet or documented method showing the included records, filters and formulas. A reviewer should be able to move from the headline number to the underlying observations. This is particularly important when exports from several operational datasets are combined.

Check counts and representative records against the application for the same scope. If totals differ, identify whether the discrepancy comes from timezones, status filters, duplicate identities or a different export time. Resolve the difference before circulating a result that depends on it.

Preserve the source snapshot used for the published calculation. Operational records may change later, and a fresh export may no longer reproduce the original figure without the original observation context. The evidence pack should make the calculation understandable at the time it was made.

Distinguish observed time savings from financial savings

A pilot can measure a reduction in time spent on a defined administrative task. Converting that reduction into a financial claim requires additional assumptions about labour cost, utilization and whether the saved time changes actual expenditure or capacity. Keep those assumptions separate from the observed duration.

For an illustrative task that falls from ten minutes to six minutes under comparable conditions, the observed reduction is four minutes, or 40 percent of the original task duration. That does not automatically mean payroll expense falls by 40 percent, or that every maintenance activity improves by the same amount.

Report the narrow result accurately and explain its scope. If the team uses the released time for additional work, describe that observed use where evidence exists. Avoid expanding a small task-level finding into an organization-wide savings claim without supporting data.

Build a case study from verified evidence and permission

A customer story should identify the workflow, observation period, data source and result. Use the customer's actual words only with permission and preserve their intended meaning. Obtain authorization for names, logos, quotes and screenshots before publishing them.

AI-generated editorial illustration; not a customer photograph or product screenshot.

Separate the customer's experience from the product's general capability claims. One customer's result does not establish that every future customer will achieve the same outcome. Explain the operating context that helped produce the result, including the team's own process changes and effort.

Label illustrative calculations and generated editorial imagery clearly. They can explain a method but should not be used as substitutes for customer evidence. This page provides a framework for collecting future evidence; it does not report customer benchmark results.

Use a short evidence review before expanding the rollout

At the end of the pilot, review the primary measure, source quality, adoption and remaining exceptions. Ask whether the result is strong enough for the decision at hand. Expanding a small workflow may require less evidence than making a broad public claim about reliability or cost.

Document what remains uncertain and what the next observation period will resolve. Assign owners for data-quality fixes and operational changes. This turns the review into a practical decision rather than a retrospective search for a favourable headline.

Use the integration guide to preserve data meaning across exports and the consultant pilot guide for a structured client handover. A credible evidence process makes the result traceable, the limitations understandable and the next action clear.

Report an inconclusive pilot honestly

An inconclusive result is still useful when the team can explain what was learned. A short pilot may show that technicians can complete the workflow and that records are easier to retrieve, while containing too few failures to assess reliability change. Report those findings separately. Do not turn successful adoption into a percentage reduction in breakdowns without the necessary observations.

For example, a team might review twenty completed jobs and find that eighteen contain the agreed closeout fields. That supports a ninety-percent completeness result for that reviewed sample. It does not show that ninety percent of all work is high quality, that the omitted fields were irrelevant or that the software caused better equipment performance. State how the sample was chosen and what the rubric measured.

If baseline records are incomplete, describe the missing information and collect a prospective comparison using stable definitions. A reconstructed estimate can be useful for internal planning, but label its assumptions and keep it separate from directly observed results. The reader should be able to tell which values came from records and which required judgment.

Close the pilot review with a decision and an owner. Continue measurement if the workflow is promising but outcome evidence is immature. Fix the process if users cannot complete the intended task. Expand only when the observed benefits and remaining risks support that decision. This produces a more credible operating plan than treating every trial as a predetermined success story.