System Failure Prediction.

System失败是在软件层面的，基于log，metrices等作为输入

Instead of investigating the occurrence of physical component failures, future system availability can also be estimated through symptomatic evidence and dependency modeling assumptions at the software level.  

Past approaches for system failure prediction are mostly based on the observation of logs, which constitute the most frequent data source, KPIs and hardware metrics, which are typically used in shorter prediction windows and are more frequently associated with the failure detection problem as well.  

Cohen et al. [31] investigate an approach based on Tree-augmented Bayesian Networks (TANs) to associate observed variables with abstract service states, forecast and detect SLO violations and failures in three-tiered Web services. The system is based on the observation of system metrics, such CPU time, disk reads, swap space, and KPI-related measures, like the number of served requests, all interdependently modeled (in addition to the dependent variable, the predicted state).  

Salfner et al. [123] train HSMM for online prediction of failures in event sequences collected from error logs. One HSMM is trained on failing sequences, a second one on non-failing sequences. 

Chalermarrewong et al. [23] propose a system availability prediction framework for datacenters, based on autoregressive models and fault-tree analysis.
Their autoregressive integrated moving average (ARIMA) model works on time series of workload and hardware metrics,
where thresholds are fixed to detect component-level symptoms (such as high CPU temperature, bad disk sectors, memory exhaustion), which also constitute the leaves of the fault tree. 

Failure Detection 

Anomaly Detection.  

AD可以实现的前提假设

Anomaly Detection is applied in the AIOps context under the assumption that failures generate irregular behavior in IT systems (errors) across a set of measurable entities (or symptoms), such as KPIs, metrics, logs, or execution traces.  

由于训练集很难收集，当前研究的方法都是无监督的，主要是聚类，降维，自编码

Because obtaining labeled examples is time-consuming, anomaly detection systems typically rely on unsupervised learning.
Three are the most prominent techniques used in such context: clustering [11, 127], dimensionality reduction [72, 151] and neural network autoencoders [7, 131, 150, 159].  

At the network level, Lakhina et al. [72] monitor network links via SNMP data to detect and diagnose anomalies in network traffic.
The apply PCA on link flow measurements collected over time to separate traffic in normal and anomalous subspaces.  

Several approaches apply unsupervised learning on univariate time series constructed from KPI and metric observations [127, 150]. The paper proposes to collect a variety of measures at the VM and physical machine level, including resource utilization, operating system variables, and application performance metrics. The paper then proposes three different unsupervised Machine Learning approaches for anomaly detection: k-nearest neighbors (k-NN), HMMs, and k-means clustering.  

Donut [150] performs anomaly detection on seasonal KPI time series using deep Variational Autoencoders (VAE) and window sampling.  

Brown et al. [18] also work on log-based anomaly detection, presenting an word-token RNN language model for raw logs based on five attention mechanisms (reaching AUCROC ≥ 0.963). The paper also proposes to analyze attention weights to provide statistical insights, un- derstand global behavior and improve decision making.  

Internet Traffic Classification (ITC).

A task connected to network failure detection is Internet Traffic Classification (ITC).
ITC allows categorizing packets exchanged by a network system to identify network problems, to optimize resource provisioning and improve Quality-of- Service [43, 97].  

Log Enhancement.

Another task connected to failure detection is log enhancement.
Its goal is to improve the quality and expressiveness of system logs, which are frequently used for detection and diagnosis tasks by IT operators and AIOps algorithms. 

Zhu et al. [168] propose a logging suggestion tool, called LogAdvisor, to learn practical logging suggestions from existing log instances.
In code snippets, several structural, syntactical, and textual features are extracted and filtered based on information gain, from which a decision tree is later trained to suggest logging of snippets as a binary classification problem.  

In Reference [163], an approach for automated placement of log printing statements (Log20) based on information theory is proposed. First, entropy is shown to be an informative measure for the placement of printing statements. Then, a greedy dynamic programming algorithm for placing printing statements is implemented.  

Root-cause Analysis (RCA)

Failure detection is the process of collection of symptoms（症状）, i.e., observations that are indicative of failures.
Root-cause analysis is, instead, the process of inferring the set of faults that generated a given set of symptoms [130].  

Fault Localization.

Fault localization is about identifying the set of components (devices, network links, hosts, software modules, etc.) interested by a fault that caused a specific failure. 

Several approaches [80, 83, 132] address fault localization by correlating abnormal changes in KPI values to particular combinations of attributes (representing, e.g., geographical regions, ISPs, hosts, buckets, etc.).  

Hotspot [132] applies Monte Carlo Tree Search and hierarchical pruning to efficiently examine attribute combinations and measure how they relate to sudden changes in the Page View metric.  

Software Fault Localization (SFL) is fault localization in software components through source code analysis.
An SFL system typically returns a report containing a list of suspicious statements or components. 

Zeller et al. [156] propose Delta Debugging, an algorithm for the determination of state variables causing a change in the outcome of execution runs. 

Incident Triage.

Incident triage（事件分流） is the step in problem resolution dealing with categorizing a reported problem.

The purpose of triage is often the assignment to the correct expert resolution group [126, 158].  

Solution Recommendation.

The approaches here described provide implementations for recommending solutions to occurring problems.
Most solutions are based on past incident history and rely on the annotation of solutions in a previous resolution window.  

Zhou et al. [166] propose similarity-based algorithms to suggest the resolution of repeating problems from incident tickets.
The basic approach consists in retrieving k ticket resolution suggestions using a k-NN approach.  

Recovery.

We define as recovery approaches those methods taking direct and independent actions toward the resolution of a diagnosed problem.  

CONCLUSION