Continuous Self-Maintenance of .NET Services Project
.NET services are distributed, large-scale, and long running. As such,
continuous maintenance is necessary in .NET platforms. And it is desirable that
the maintenance be
as autonomous as is possible, thus involving human operators only when it is
unavoidable. In an ongoing joint project with MSR's Marvin Theimer, we are
framework for continuous self-monitoring and evolution of .NET service
components. We propose to complement that project by building frameworks for (i)
composition of fault-tolerance components and (ii) continuous self-testing of
.NET services. Further, we propose to integrate and demonstrate these three
frameworks in the Herald .NET service being currently developed in MSR.
Why Continuous Self-Maintenance?
The following considerations imply that only machine-enforceable aspects of a
large-scale, long running .NET service can be assumed hold continuously.
- Lack of global developer knowledge. Distributed large-scale .NET
services contain many components that change constantly. Few people— if
anyone— understand all the evolving state guarantees and behaviors that an
entire service is supposed to maintain. Service complexity is traditionally
constrained by requiring that its components interact through well-defined,
narrow interfaces that precisely specify the invariants that should be
maintained before and after their invocation; however, developers are not
always aware of all the invariants and don't always implement the interface
specifications correctly. Since it is difficult, if not impossible, to evolve
a large-scale distributed service in a globally consistent manner; such
services will generally tend to be incorrect.
- In-place updates. Long-running .NET services get updated in-place
and a second complete "test" service is typically not available for a priori
testing of all the possible interactions that an updated component will have
with the running service. It is inevitable that incorrectly implemented
components will be introduced during service updates, which will propagate
errors into the state of a running service.
- Inevitable unanticipated faults. Any large-scale, long-running
service, even if not updated, will inevitably be subject to unanticipated
faults, such as faults that occur due to defects in implementation, only occur
when the service is under load, or do not occur when instrumented for
debugging (" Heisenbugs").
- Incomplete specifications. Incomplete or partial specifications are
unavoidable in service maintenance— especially now that design time scales can
be as low as 12 weeks and much less for those of critical services.
We thus claim that properties that are not machine enforceable will
eventually become invalid in the .NET context. Continuous maintenance by the
system itself is therefore
Requirements for Continuous Self-Maintenance.
We focus our attention on the following desiderata.
- Reliability. Service faults should be detected, diagnosed and
tolerated. Since service aspects that deal with faults are rarely tested
during normal operation, reliability should be ensured proactively if
possible. But most importantly, service function should not degrade as a
result of continuous self-maintenance, even across administrative boundaries.
- Availability. Services that are not functioning as desired should
quickly resume desired function.
- Evolution. If the environment changes so that the maintenance of
services becomes inadequate, the maintenance procedures should be evolved
(automatically if possible and manually otherwise).
- Low-cost. New resources should not be necessary for continuous
maintenance. Instead all existing residual system resources should be
aggressively, with the provision for their quick release whenever the demand
for services grows.
Our Approach to Continuous Self-Maintenance.
Our approach to meeting the above requirements consists of three orthogonal
elements. Various combinations of the three can be used depending on the
particular .NET scenario
- Dynamic Composition of Tolerance Components. For reliability
against faults, our approach is to wrap .NET service components with tolerance
components, namely detectors and correctors . Tolerance
components detect the health of service components, e. g. in terms of data
invariants or timeliness of execution. They may also perform replication,
correction, consistency enforcement, failing which they may notify service
managers so to enable fault diagnosis.
- Continuous Self-Monitoring and Evolution. For availability, our
approach is to continuously monitor .NET services (and their tolerance
components). The frequency of monitoring is dynamic, and depends on the
service state, the evolving service composition, and the resources available
in the system. It enables detection of dependencies both within and across
services; e. g. a service consisting of multiple server instances for
reliability and performance will be able to analyze its degree of replication
by monitoring dynamic, global conditions. It also enables service evolution.
- Continuous Self-Testing. For proactive reliability, our approach is
to continuously testing of .NET services via injected faults. Such testing is
carefully designed to not interfere with normal service operation. At the same
time, it exploits available resources in the system.
Previous and Current Related Work.
Aladdin [2-3]. Jointly with
MSR's Yi-Min Wang and Wilf Russell, we built this system for local/ remote home
networking using only undependable commodity devices networks (phoneline and
powerline) and operating systems (Windows 98). Typical scenarios of Aladdin use
include: remotely sending an email home to shut the garage door and receiving a
video confirmation; receiving a cellphone message in case of a water leak in the
basement; and using a natural language interface at home to switch off the
lights in any part of the home. Its architecture, software services (such as
SoftStateStore), and protocols met the goals of a dependable and extensible
system with low-complexity and good performance. By dependability, we mean that
its tolerance components enable the system to mask common faults, self-stabilize
from rare faults, and notify the homeowner in case of catastrophic faults. By
extensibility, we mean that new scenarios of use can be added to it with little
effort. Outside of the lab setting, Aladdin has been in daily use in a large
single-family home in Seattle for well over a year now. Aladdin was showcased in
a PBS serial aired in California in Summer 2000, as well as in a Seattle P-I
news article; it has led to four patents. Its architecture is compatible with
.NET, and has motivated Wang and Russell's recent work on .NET Alert Services.
Continuous Monitoring Framework for .NET Services [4-5].
This project accommodates both self-monitoring and human-in-the-loop monitoring
of services. Specifically, mechanisms are provided to continuously monitor the
liveness of all services as well as their detectors and correctors, which are
tasked with satisfaction of service-specific data invariants. This includes a
means for dynamically controlling which services and detectors/ correctors get
checked and how often; a means for dynamically detecting conflicting data
invariants that have been posited for the service; a means of analyzing the
"error" behavior of a service; a means of evolving the service and its
associated data invariants; and a means for allowing all residual system
resources to be made available continual monitoring.
Framework for Dynamic Composition of Tolerance Components
- Dynamic Composition. This framework will include mechanisms for
tolerance components to be added, removed, and updated while the .NET service
is running. Distributed implementation of tolerance components, i. e. between
clients and servers, will be enabled. Calls to a component will be dynamically
redirected to its tolerance component, which will adjust component input and
output as necessary to invoke new distributed coordination/ fault-tolerance
protocols. Layered addition of tolerance components will be allowed.
- Consistency Semantics. The framework will ensure that (i) tolerance
components updates are atomic with respect to the ongoing service
interactions; (ii) a tolerance component is removed only if all of its
instances are removed; (iii) a tolerance component update implies update of
all of its instances; and (iv) orphans (unused tolerance components) and
widows (tolerance components that call nonexistent components) are handled
- Multiple Abstraction Levels. The framework will allow composition
of tolerance components per instance, per component, or per group of
- Minimal Modification. The framework will require only minimal
modification of the underlying service component source code, i. e. extending
service components to inherit from a special platform class.
Framework for Continuous Self-Testing of Services
- Dynamic Injection. This framework will include mechanisms for
simulating faults by execution of fault methods (" faultors") by an
environment. It will enable dynamic scripting of faultors. Going further than
the dynamic composition framework, it will also enable dynamic addition of
methods to a component.
- Test Specifications. The framework will provide mechanisms for
testing of temporal event predicates, in addition to global state predicates,
for each faultor group.
- Non-Interference. The framework will include mechanisms to quickly
reconfigure services into functioning and test versions, and to prevent
interference of service interactions during reconfiguration. Dually, quick
release of testing resources and scaling up of functioning services will
supported to deal with increases in service demand. Design patterns will be
included to hide the effects of simulated faults from clients, e. g. via
replication and checkpointing-rollback.
- Discovery Support. The framework will include mechanisms to support
discovery of new faults, invariants, and dependencies. Both frameworks will be
implemented in C#. Browser-based interfaces will be provided to access the
various facilities interactively. Interfaces will allow tracing and reporting
facilities so the current composition status of the system can be monitored.
Integration with ongoing MSR .NET projects.
.NET Services built in the Systems and Networking group at MSR provide a
compelling avenue for demonstrating and validating the proposed frameworks.
Thus, after having consulted with Marvin Theimer, we propose to integrate and
demonstrate the frameworks described above in the context of the Herald global
eventing service that Marvin and his colleagues are building. By maintaining
data invariants for Herald service components at all levels (simple components,
single machines, groups of machines, and system wide), we will be able to
measure the effect on availability of satisfying these invariants. By monitoring
intra-component as well as inter-component dependencies in Herald, we will be
able to implement and compare various replica management policies, thereby
obtaining a better understanding of scenarios where global information is
necessary and how to distribute the cost of repair over the global system. By
injecting faults in Herald, we will be able to collect fault data about long
running .NET services much faster than we have collected fault that really
occurred in the Aladdin home network installations.
1. Arora, A. and S. Kulkarni. Detectors and correctors:
A theory of fault-tolerance components. in International Conference on
Distributed Computing Systems. 1998.
2. Wang, Y.-M., W. Russell, A. Arora, J. Xu, and R.
Jagannathan. Towards dependable home networking: An experience report. in
International Conference on Dependable Systems and Networks (ICDSN'2000).
3. Wang, Y.-M., W. Russell, and A. Arora. A toolkit for
building dependable and extensible home networking applications. in
Fourth USENIX Windows Systems Symposium USENIX-WIN'2000. 2000. Seattle.
4. Arora, A. and M. Theimer, Agenda for fault-tolerant
distributed computing in .NET. 2001, The Ohio State University and Microsoft
5. Arora, A. and M. Theimer, Service evolution
framework. 2001, The Ohio State University and Microsoft Research.