# Analysis

## The Analysis Box

The Analysis Box (Figure below) is the final component in the learning module pipeline, positioned immediately after the Training Box. It serves as the dedicated component for model evaluation, accepting inputs from:

* Individual trained models (via [**Train Model** ](https://medomicslab.gitbook.io/medomics-docs/tutorials/development/training#train-model-model-training-and-optimization-node)nodes)
* Model ensembles (via [**Combine Models**](https://medomicslab.gitbook.io/medomics-docs/tutorials/development/training#combine-models-combine-models-trained-in-your-scene) nodes)

**Key Characteristics:**

1. **Node-Free Design**: Unlike other boxes, this is a preconfigured analysis terminal that cannot contain additional nodes.
2. **PyCaret Integration**: Implements `plot_model()` with the following parameter controls:
   * **Plot Metric** (`plot` parameter):
     * Sets the evaluation visualization type (default: 'auc')
     * Options include: confusion matrix, feature importance, ROC curve, etc.
   * **Scale** (`scale` parameter):
     * Adjusts output figure resolution (range: 0-1)
     * Higher values increase image quality and file size

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2FeCnj2UCDBYhlW3ebsCVo%2Fimage.png?alt=media&#x26;token=90b505d3-df04-48ee-8575-0ea230a6f116" alt=""><figcaption><p>The analysis box</p></figcaption></figure>

The Analysis Box represents the "Analysis" section of the machine learning workflow:

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2FyFl2iXo7FerCU3LNYPKQ%2FAnalysisBoxUsageNew.png?alt=media&#x26;token=78733a09-98bd-4144-89f6-b058627b3b2e" alt=""><figcaption></figcaption></figure>

## The Analysis Mode

{% hint style="info" %}
If you prefer a quick summary,  jump to the [following section](#summary-of-the-analysis-mode).
{% endhint %}

The Analysis Mode becomes available after a successful experiment execution. When activated, a results panel appears at the bottom of the interface, displaying results for all pipelines in the current scene. This mode provides a detailed breakdown of results organized by pipeline and node.

### **Pipeline Results Structure:**

Each pipeline, identifiable by its customizable name, presents results through the following node-specific information:

* **Dataset Node**: Displays the training data table and all parameters applied through [PyCaret's setup function.](https://pycaret.readthedocs.io/en/stable/api/classification.html#pycaret.classification.setup)

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2FGXzlHweNyCBZI22AhOVp%2FResultsDataset.PNG?alt=media&#x26;token=1c2fcbac-2012-4e8e-8609-759289a1b411" alt=""><figcaption><p>Results for Dataset node</p></figcaption></figure>

* **Clean Node**: Shows the preprocessing parameters configured in [PyCaret's setup](https://pycaret.readthedocs.io/en/stable/api/classification.html#pycaret.classification.setup).

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2FgfjBI43r37VABwicD0G6%2FResultsClean.PNG?alt=media&#x26;token=c31f332e-ce15-4b7a-b87d-17284222b195" alt=""><figcaption><p>Results for Clean node</p></figcaption></figure>

* **Split Node**: Presents detailed split statistics, including sample counts per fold/iteration and class distribution metrics.

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2Fggxz8PEYC2ZYX4qI3eVF%2FResultsSplit.PNG?alt=media&#x26;token=2a53bec5-2083-41d5-b091-3e86dc3ac172" alt=""><figcaption><p>Results for Split node</p></figcaption></figure>

* **Model Node**: Contains the complete set of performance metrics for the model.

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2FKVwhiJyve5CKvBwhTJV0%2FResultsModel.PNG?alt=media&#x26;token=567ce435-09a9-4544-852c-3a38a0a3acc5" alt=""><figcaption><p>Results for Model node</p></figcaption></figure>

* **Combine Models Node**: Provides evaluation metrics for the combined model output.

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2Fh0HXYh9uS4TTCqDFKCY4%2Fimage.png?alt=media&#x26;token=dd2041a2-a6d4-4b13-8480-e94180778853" alt=""><figcaption><p>Results for Combine Models node</p></figcaption></figure>

* **Analysis Node**: Displays the plot selected in the Analysis Box.

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2FPytzigodHZHyv6cMVVNv%2FResultsAnalysis.PNG?alt=media&#x26;token=09cfc50f-0103-430f-8c74-eb68f26aa3dc" alt=""><figcaption><p>Results for Analysis node</p></figcaption></figure>

{% hint style="warning" %}
***PyCaret*****&#x20;ROC (Receiver Operating Characteristic**)**/AUC (Area Under the Curve) plots**

The AUC plots generated by the [*PyCaret* ](https://pycaret.gitbook.io/docs)library are derived from the [*YellowBrick* Python package](https://www.scikit-yb.org/en/latest/api/classifier/rocauc.html), which extends the [*scikit-learn*](https://scikit-learn.org/stable/) API. By default, the plot displays multiple curves:

* The ROC curve per class for each class was computed using the one-vs-rest method (meaning that the considered class is treated as the positive class and all other classes as the negative class).
* The micro-average curve is calculated by summing up all true positives and false positives across all classes.
* The macro-average curve is the average of curves across all classes.&#x20;

We acknowledge that these curves can be a bit confusing, especially with binary classification.&#x20;

While using the[ *YellowBrick* package](https://www.scikit-yb.org/en/latest/api/classifier/rocauc.html) directly, we can set parameters to display only the classic ROC curve. However, we haven't found a way to directly set these parameters through our application with [*PyCaret* ](https://pycaret.gitbook.io/docs)yet. We are currently working on fixing this issue.
{% endhint %}

### **Finalize & Save Model**

This feature, used through the button '*Finalize & Save Model*' for a selected pipeline, performs two critical functions through PyCaret integration:

1. **Model Finalization**: Retrains the selected model on the complete dataset using [PyCaret's `finalize_model()` function](https://pycaret.readthedocs.io/en/stable/api/classification.html#pycaret.classification.finalize_model).
2. **Model Saving**: Saves the finalized model as a [pickle file](https://www.geeksforgeeks.org/python/understanding-python-pickling-example/) via [PyCaret's `save_model()` function](https://pycaret.readthedocs.io/en/stable/api/classification.html#pycaret.classification.save_model). The saved model appears in the experiment's `models` subfolder using the model's classname or the Model node's ID if it has been changed from the default one ('*Model*').

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2Fjf8PqfKxta7NNqBzvJeQ%2FFinalizeAndSave.png?alt=media&#x26;token=6009c3d9-0d65-403f-9cb7-920e7ce252cb" alt=""><figcaption><p>How to finalize and save a model in a given pipeline</p></figcaption></figure>

The process requires no parameter configuration, automatically preserving all training parameters from the original experiment.

Additionally, this button represents the "Final Model" section of the machine learning workflow, as shown in the following figure:

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2FeHXomZjRIFvrzk27kQBk%2FFinalizeAndSaveUsageNew.png?alt=media&#x26;token=183a6dfc-bfe3-432c-b193-e5afa6d3f56e" alt=""><figcaption></figcaption></figure>

### **The Generate Feature**

The Generate functionality exports the complete pipeline configuration as executable Python code in [Jupyter Notebook](https://www.dataquest.io/blog/jupyter-notebook-tutorial/) format. You can generate a Jupyter Notebook using the *Generate* button next to a selected pipeline. The generated notebook, which mirrors the pipeline's structure and parameters, appears in the experiment's `notebooks` subfolder using the pipeline's current name as its file identifier.

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2FqOC4Saetrd2ezHXqco5C%2FGenerePipeline.png?alt=media&#x26;token=b31d3475-546b-4b77-b3f9-2c607a4a9972" alt=""><figcaption><p>How to generate a Notebook for a given pipeline</p></figcaption></figure>

This feature enables:

* Deeper investigation of the training process
* Custom code modifications for performance optimization
* Enhanced reproducibility

Additionally, you can also launch any generated notebook directly from the application by simply double-clicking the file. Conversely, you can right-click and select the **"Open in..."** option to open your notebook in VSCode.

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2Fqiv3OxEUkYFxCHAGCGnD%2FOpenNotebookIn.png?alt=media&#x26;token=0a267354-5343-4181-8ba7-776ae988f4b6" alt="" width="563"><figcaption><p>Different options to open a generated Jupyter Notebook</p></figcaption></figure>

An example of a generated notebook, opened in VS Code, is shown below.

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2F43FQE34wTOHFsSYqJO5d%2FGeneratedNotebook.PNG?alt=media&#x26;token=919d7d0a-54e6-4534-aeb0-0f7421955531" alt=""><figcaption></figcaption></figure>

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2FdUPlHSRcwKilnD0rJ814%2FGeneratedNotebook1.PNG?alt=media&#x26;token=7702b3d5-d687-4044-9c62-2fa19b5f7ac0" alt=""><figcaption><p>Example of a generated Jupyter Notebook</p></figcaption></figure>

{% hint style="warning" %}
Pipeline naming conventions directly affect this feature. Check out the next section for more details.
{% endhint %}

### **Manage Pipelines**

The Manage Pipelines interface serves two primary purposes:

1. **Pipeline Overview**: Displays a structured summary of all nodes comprising each pipeline and their connections.
2. **Naming Control**: Allows pipeline renaming, which simultaneously updates:
   * The notebook filename in the Generate feature
   * All experiment tracking references

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2F9j1EiZ9HfjtdDPAhSxGl%2FManagePipelines.png?alt=media&#x26;token=5eb8021e-84db-4366-9b48-1fc4ad52c5d5" alt=""><figcaption><p>How to access and use the Manage Pipelines panel</p></figcaption></figure>

### **The node's selection box**

In both Analysis and Results modes, a checkbox is available at the top of each runnable node. Use this control to selectively display results for specific nodes, hiding the output of others. A green checkbox indicates that the node is a mandatory component of all pipelines; consequently, its results will always be displayed.

<div><figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2F5mvoe5oFbrKRUuUD82F8%2Fcheckbox_blue.png?alt=media&#x26;token=69916263-a040-4d50-bcc0-a1cf6caf42ea" alt="" width="90"><figcaption><p>Selection box</p></figcaption></figure> <figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2Fu8FwL9WEZvOQ8h7BL0TJ%2Fcheckbox_green.png?alt=media&#x26;token=45841438-f2c3-428d-a61e-2af86a88a4a2" alt="" width="90"><figcaption><p>Independent selection box</p></figcaption></figure></div>

In the following example, only the results of the checked node **Clean2** are displayed, while the other pipelines are hidden.

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2FEhZEaCLFWufOySb1RUZG%2FNodeCheckbox.png?alt=media&#x26;token=5aac6036-003c-4799-a184-9440f982cbfc" alt=""><figcaption><p>Illustration of the node's checkbox usage</p></figcaption></figure>

### The highlighting feature

This feature enhances navigation in both Analysis and Results modes by dynamically applying distinct color codes to selected nodes and pipelines. It highlights the entire execution path of a chosen pipeline, making it easy to distinguish from others. The system uses the following color scheme to indicate status:

* **Orange**: Used for non-executed nodes and the connecting edges of a non-executed pipeline.
* **Green**: Indicates a selected and successfully executed node.
* **Blue**: Highlights all elements (nodes and edges) of the currently selected pipeline.

This functionality is particularly valuable in complex scenes with multiple pipelines, as it simplifies the process of tracking and comparing results. The following figure illustrates these color codes in the context of different user interactions.

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2FKgzyvzfPcT1E7xJQmFhF%2FHighlightFeatureFinal.png?alt=media&#x26;token=c86aad04-046f-4725-8ca1-64a617c5b239" alt=""><figcaption><p>Illustration of the highlight feature for different interactions</p></figcaption></figure>

### Summary of the Analysis Mode

A full breakdown of the Analysis Mode is presented in the following figure:

<figure><img src="https://4288370141-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FUO0RN9PzFLqAgLEwwaSn%2Fuploads%2F65mJGc0k4o6mdDhOZZSS%2FBreakdownAnalysisMode.png?alt=media&#x26;token=6187d1bf-085e-4a20-88c5-9cf4c00ec902" alt=""><figcaption><p>Breakdown of the Analysis Mode</p></figcaption></figure>

On the next page, you will learn more about the new scene type '*Experimental*' and how you can use it as a testing environment for your machine learning experiments.
