Report
The Report provides performance metrics and interactive charts to enable developers to evaluate the performance of their models. To do so, the fields highlighted in the table below must be filled in.
Field | Description |
|---|---|
Market Data | The market data in which the model is used. |
Model Name | The name of the model. |
Pairs | The pairs that the model is tested with. |
Timeframe | The timeframe in which the model is tested. |
Probability Column | Models report three probabilities for a candle being buy, sell, or no-action. Reports can be generated for each of these three probabilities. |
Each report has six main sections:
AUC Scores
ROC Curves
Precision Recall Curves
Probability Density Function
Complementary Cumulative Distribution Function
Probability Columns Scatter Plot
Each of the aforementioned sections is divided into two parts: Train and Test. The time range used for training the model is called Train, and the rest of the data that is in the market data but not used in the training phase is called Test. Scores and charts are presented for these two ranges separately.
AUC Scores
Area Under Curve (AUC) score of the Precision-Recall plot and Receiver-Operating Characteristic plot are reported for Train and Test periods separately. The higher these two scores are, the better the model can follow the target used for training. The maximum value of AUC score is one. Note that high Precision-Recall and ROC scores DO NOT necessarily mean the model can profit reasonably. This means that the model can imitate the behavior of the target used for training.
ROC Curves
The ROC Curves shows the Receiver Operating Character plot for train and test periods. ROC plot is a graphical representation used in machine learning and statistics to evaluate the performance of a binary classification model. It provides insight into the trade-off between the true positive rate (TPR) and the false positive rate (FPR) across various classification thresholds. To read more, click here
Precision Recall Curves
the Precision-Recall Curve is a visualization used to evaluate the performance of a binary classification model, particularly when dealing with imbalanced datasets where one class significantly outnumbers the other. It focuses on the trade-off between precision and recall across different classification thresholds. To read more, click here
Probability Density Function
A probability density function (PDF) is a function used to describe the probability distribution of a continuous random variable ( In our application, the probability of being a buy, sell, or no-action). In simpler terms, it represents the likelihood of the random variable taking on a particular value within a given range. The PDF itself doesn't give the probability of specific values but rather provides the probability density over a range of values. In this graph, the horizontal axis is the probability predicted by the model, and the vertical axis is the probability density. Using this graph, one can see the likelihood of a candle being the class selected in the Probability Column when the model under test returns a specific probability for that class.
Complementary Cumulative Distribution Function
The complementary cumulative probability density function (CCDF), also known as the survival function, is a concept related to probability theory and statistics. It represents the probability that a continuous random variable is greater than a certain value. The CCDF is often used in reliability engineering, survival analysis, and telecommunications, among other fields, to analyze the probability of exceeding certain thresholds or durations. Using this graph, one can decide about the trade-off between the number of buy or sell signals and the centrality of the model. For example, if the CCDF value for buy probability of 0.8 is 0.05, this means that only 5 percent of the candles labeled as a buy have buy probability of 0.8 or higher.
Probability Columns Scatter Plot
This interactive scatter plot illustrates how buy, sell, and no-action samples are distributed on a plane with two altering axes. Since the axes can have different configurations, there is no unique way of interpreting this chart, but in general, the more separate the buys, sells, and no-actions are, the better the model is.