Step 8: Challenge
Apr 22 – May 13 | Challenge
This step will require you to download the new learning set that we sent you (MEDomicsLab_TestingPhase_Step8.zip). This set comprises two datasets, combining the learning and holdout sets obtained in Step 4 - Explore Data at Time Point 1 and Time Point 2.
To avoid confusion among all the datasets from the beginning of the Testing Phase, we recommend creating a new workspace for this step containing only the new learning set.
An invitation to access the MEDomicsLab_TestingPhase_Step8.zip data has been sent via email.
Welcome to the final step of the MEDomicsLab Testing Phase! We appreciate your engagement throughout this journey.
In Step 8 - Challenge, you will leverage the knowledge you have gained from the MEDomicsLab platform in the preceding steps of the Testing Phase. You are the master here 👏!
The objective here is that participants design their own model via the use of the Machine Learning (ML) Module. These models will be evaluated by the MEDomicsLab team using a fresh, private holdout set from the MIMIC database.
Considering the insights we have gained during the Testing Phase, we have now updated the rules for this step. In contrast to what we initially planned, there will be a single challenge that involves manipulations in the Machine Learning (ML) Module only.
We are continuously working on enhancing the MEDomicsLab platform, and we would like to inform you that there may still be some missing options and tooltips in the Machine Learning (ML) Module, which we intend to implement in the future.
Recommendations
Before proceeding with Step 8 - Challenge of the MEDomicsLab Testing Phase, we recommend revisiting the documentation related to Step 6 - Create Model.
Step 6: Create ModelInstructions for Step 8 - Challenge
Download the new learning set provided in MEDomicsLab_TestingPhase_Step8.zip.
Create a model similarly to the one we created in Step 6 - Create Model using the Machine Learning (ML) Module. Follow these guidelines for creating your best possible model:
Train, optimize, and test models.
Compare different data combinations.
Choose a learning hypothesis and create a final model.
Once you have trained your best model, send it to sarah.denis@usherbrooke.ca with "Step 8 - Challenge | Submission" as the Subject of your email. You are allowed to send multiple submissions, but please note that we will only evaluate your latest submission.
If you wish, you can let us know in the email the name that we should use for the public ranking of your submission (e.g. MLrocks 😄). If not specified, we will use your full name.
For this ML Challenge, most of your work will therefore be done inside the Machine Learning (ML) Module. However, you are also welcome to use the other capabilities of the MEDomicsLab platform. For example, you may want to use the Exploratory Module to continue to gain useful insights about the dataset prior to the learning step.
However, to facilitate the Evaluation process of the models submitted by the participants, we please ask the following:
Only use the data we provided in the MEDomicsLab_TestingPhase_Step8.zip file.
Therefore, you should not be able to use the Extraction Module or the MEDprofiles package.
Do not change column names in the data we provided (this includes the use of the Groupping/Tagging tool).
Do not merge the time point CSV files together.
Dot not use the "Transform Columns tool".
Do not use the "PCA" utility of the "Feature Reduction tool".
If you use the "Spearman" utility of the "Feature Reduction tool", make sure to keep the subject_id and target columns in your dataset by enabling the "Merge unselected columns in the result dataset" and "Keep target in dataset" options.
Note that you new dataset will be placed in the reduced_features folder and that you will have to move it in your learning folder to retrieve it in the Learning Module.
If these instructions are not followed, we will not be able to evaluate your submission, and it will unfortunately be discarded.
Also, note that to create a model in the Learning Module, your CSV files must be placed under a "learning" folder and have the prefix "TX_" (e.g., T1_new_learning) as in the data we provided you.
Content
Intro 0:00
Rules 0:38
Dealing with Server Errors 4:03
Documentation 5:08
The Optimize node 8:58
General Advice 11:05
After the conclusion of the Testing Phase on May 13, we will evaluate the performance of the models submitted by all participants and establish a ranking through an evaluation using our private holdout set.
Following this, we will make the ranking public and reveal the winners of both the ML Challenge 🎉 and the Bug Finder Challenge 😄 at the wrap-up meeting scheduled for May 17. An invitation to this meeting will be sent out soon.