The correct answer is obvious: “there should be no exceptions, because one-offs usually cost you more than you earn with them,” but as always the reality tends to intervene.
There’s also the interesting challenge of rolling out pilot services: you don’t want to start modifying your orchestration platform until you’re pretty sure the new service will sell, which means you cannot provision the service, so it will never sell. A perfect Catch-22.
However, Rok Papež (a great engineer working on network automation within ARNES – Slovenian academic ISP) described a nice workaround while we chatted after my talk at recent SINOG event: they store the non-automated part of the device configuration (configuring the pilot services) in an extra field in their configuration management database, and append it to the configuration generated from device/service data model through standardized templates.
Now imagine you go one step further: instead of storing non-standard parts of device configuration your database contains configuration snippets for every non-standard (pilot or exception) user/service/device triplet. This approach makes it extremely easy to:
- Identify all pilot services and all exceptions;
- Identify which users have non-standard services;
- Verify whether a user has non-standard configuration before you start the troubleshooting process (and maybe escalate the ticket before wasting time trying to figure out what the deployment engineer had in mind);
- See how the engineers configured those services on individual devices before trying to configure a similar solution for the next customer;
- Verify that the orchestration system generates equivalent per-service configuration once the pilot service becomes part of the standard offering;
- Clean up non-standard parts of the configuration once you migrate the users of pilot services to new standardized services.
Thank you, Rok! You gave me a reasonable answer to a pretty tough challenge ;)