Case file 01
ITOM / Discovery and Event Management
A Discovery environment that ran every night and told nobody anything.
Discovery had run since go-live. Nobody on the operations side used what it produced.
Composite note. No client is identified; the details are typical of this kind of engagement rather than a record of one.
The situation
The schedules were the ones set up at go-live. Since then the network team had re-addressed two sites, credentials had rotated more than once, and a cloud account had been added without anyone telling the platform team. Two of four data centres had MID Servers; the other two were covered by a single MID Server across a slow link.
The Discovery error log held thousands of entries that nobody read, mostly credential failures and timeouts. The CMDB still showed servers retired years earlier as operational. Event Management had been switched on, so every one of those gaps also produced alerts, and the on-call engineer had learned to ignore the integration altogether.
The decisions
- D1
Ownership before tuning.
Before touching a schedule, we agreed who owned credentials, who owned IP ranges, and who would read the error log each week. Without that, the same drift would return within a quarter.
- D2
Baseline against the real environment, not against Discovery.
The network team's address plan and the cloud account inventory became the reference. Schedules were rebuilt from that list, with overlap and duration designed rather than inherited.
- D3
Retire through lifecycle, not deletion.
Servers Discovery had not seen for a long period moved to a retired operational status through a rule, so history and relationships survived and nobody had to argue about a bulk delete.
- D4
Event rules tied to what is mapped.
Alerts were kept only where a CI, a service and an owner existed. Everything else went to a review queue rather than to the on-call phone.
The delivery
A baseline first: coverage, errors, credentials, MID Server health and schedule design, measured against what the teams actually relied on. Then fixes in root-cause order: credentials and MID placement, then schedules and patterns, then the event rules.
Handover was a runbook for the platform team, a weekly check they could run in under an hour, and the ownership list written down where the next person would find it.
What changed
On-call could use the server list without a second source. Alert volume fell to the ones that had an owner and an action. The error log became a weekly task rather than a monument.
The platform team kept the environment in that state because the checks were theirs and the ownership was explicit.
