Original paper

The DOI for the original paper is https://doi.org/10.59490/joas.2025.7963

Review - round 1

Reviewer 1

This article presents the methodology used by the authors to obtain a machine learning model predicting the takeoff weight of commercial flights in 2022. This work was done to compete in the Performance Review Commission (PRC) Data Challenge 2024 in which the authors ranked second.

The article describes the data and the method used to obtain good predictions of aircraft takeoff weight. An analysis of the feature importance and an ablation study are done, showing which parts of the model are important or not. Some parts of the method might not be useful which is fine given the limited time competition context. Readers might think that these parts are useful. In order to not mislead the readers, the “Detailed Remarks/Questions” section below details which parts might be tagged as not (that) useful. Some details are missing and there is one theoretical paragraph that might not be relevant (see “Detailed Remarks/Questions” below), but given this easy to add/correct, I recommend an accept with minor revision.

Questions Related to “2.2 Data Preprocessing” and “2.3 Feature Engineering”

• A1 - ADS-B trajectory data are somewhat “dirty”, the article does not mention any “cleaning” process. I looked at get_traj_features4.R and did not see any but I am not “fluent” in R and maybe I missed something. Is there really no trajectory cleaning ? Maybe the article could state clearly if there is a cleaning process or not, and it should be described in the article if there is one.

• A2 - We do not always have data for the whole trajectory. As a consequence some statistics concerning the efficiency of the vertical profile might not be “accurate”. For instance the total time flown in level flight of two flights might differ only because the ADS-B coverage is not the same. Do you have a way to handle these kind of unbalance due to the ADS-B coverage?

• A3 - Concerning the feature $\mathrm{taxi\_ratio}$ in the duration-based section, it is the taxiing time over the total flight duration, indicating the inefficiency. This ratio is then normalized again by dividing it by the median value to facilitate the comparison across flights. However, looking at feature_engineering.ipynb, it is divided by the median of all the flights, hence it is simply a linear transformation of the feature that will have no impact on tree-based algorithms/models (which are invariant to strictly monotone transformation of their feature) and neural network models in which each feature is typically normalized/standardized beforehand. If I did not miss something on that, maybe the paragraph about the division by the median could be removed as this division by the median do not change anything and might misled the readers into believing that it does something.

• A4 - Concerning the speed per distance, still in the duration-based section, looking at
  feature_engineering.ipynb, it is actually equal to 1/flight_duration. As such, if I did not miss something, the phrase “This feature provides a normalized speed metric relative to the flight distance, assisting in analyzing speed efficiency over varying distances.” in the article should be modified. Furthermore, this transformation is strictly monotone, and hence will have no impact on tree-based algorithms/models, aside from multiplying by 2 the probability to be se considered as a split feature during the tree growing phase.

• A5 - Looking at get_traj_features4.R, feature_engineering.ipynb and catboost.ipynb, for the “first ten points features”, some features are not listed in the article despsite being used like the altitude and the sqrt_tas features. I know that the sqrt_tas features are in the end not used as they are eliminated by the catboot selection process, however they could be listed as they were considered in the selection process. As a side note, the “squared” transformation is strictly monotone, and as such tas_1 and sqrd_tas_1 are basically the same feature, and this might have an impact on their respective importance score.

• A6 - From the “Interaction Features” section alone, the rationale is not clear behind the operation done, how multiplying the airspeed by the specific energy “help us understand how an aircraft’s speed influences its energy efficiency during different stages of flight”. Could you elaborate on this?

Questions Related to “2.4 Predictive Models”

• A7 - As a suggestion, you could change the subsection “Predictive Models” to a section. With this change, you will be able to add two subsections to “Articficial Neural Networks”, one for the classical and one for SAINT. An other benefit is having a new section “Gradient Boosted Trees” containing the three gradient based methods. This section could contain a description of the hyperparameter that are shared between these methods. As a side note, the description of these hyperparameters is made inside Extreme Gradient Boosting, the last method described.

• A8 - Even though you did not use optuna to tune the neural networks, maybe you can have hyperparameters tables for the classical and SAINT neural networks. For SAINT, you performed a grid search on the width, maybe you could show this grid of hyperparameters in a table. For the classical neural network, from the text, it is not clear how the hyperparameters were selected. As a side note the batch size is not mentioned in the text despite being an important parameter of the training phase.

• A9 - For the hyperparameter tuning of CatBoost, I am not sure that changing the learning rate is a good practice. The obtained hyperparameters are searched with a “big” learning rate, thus the obtained hyperparameter could be sub-optimal when considered together with a reduced learning rate. Maybe, the article could provide figures on compute times to help justify this choice.

• A10 - For the hyperparameter tuning of CatBoost, the early stopping is used to train the final model, it is not clear that it was also used in the hyperparameter tuning phase.

• A11 - Concerning the ensemble section, I am not sure the theoretical framework presented is consistent with the way the ensemble is built. The theoretical framework presented is typically used in bagging/random forest: the predictive models are just averaged without a “meta-learning” above them that optimize weights using the training set. As the coefficients are actual constants, the variance decomposition is easy to determine.

  In the article, the predictive models are combined using (a part of?) the training set to optimize the weights. Furthermore the bias-variance decomposition is the expectation over (at least) a random training set, as a consequence the weights are not constants (w.r.t. the training set) and the variance decomposition in the article cannot be made here, it is not that straightforward anymore. The ensemble method used in the article is probably more related to “stacking” ([Breiman 1996]), and a more relevant theoretical framework can be found in the “Super Learner” paper ([Van der Laan et al. 2007]).

• A12 - Concerning the ensemble section, it could detail whether the training set used to learn the models is the same used to compute the optimal weights, or is it split in two (or K) parts to do that?

Questions Related to “3 Discussions”

• A13 - Concerning the Table 6 for the ablation study, the temporal and geographical features are described in their own subsubsections 2.3.2 and 2.3.4 but features such as “aircraft characteristics” and “operational features” are not described. Maybe the authors could add subsubsections for these two groups. Additionaly, some features in these groups are not described in the feature section.

Typos

• A14 - line 303: “hype parameters” to “hyperparameters”.

• A15 - line 430: In “The A320 was the most used equipment”, the word “equipment” sounds weird, maybe “aircraft type” would be better.

Appendices

Some thoughts I had reading the article that can be left unanswered by the authors as they might be unimportant, irrelevant or wrong:

• A16 - We do not always have data for the whole trajectory. As a consequence the first ten observed points are somewhat random: they are the first points inside the ADS-B coverage. The altitude “frontier” of the ADS-B coverage may vary from one location to an other. As such, the altitude of these first points may vary, regardless of the way the aircraft is operated. To illustrate how this can be problematic, let us consider the feature $VerticalRate_1$ which is the vertical rate computed on the first observed point. If you observe $VerticalRate_1\left(Flight_A\right)>VerticalRate_1\left(Flight_B\right)$, maybe $Flight_A$ is lighter than $Flight_B$ or maybe they have a similar weight but $Flight_B$ is higher than $Flight_A$ and consequently have a lower rate of climb. The feature “height relative to the origin airport” can mitigate this issue but the tree based models will have to “chain” two conditions/nodes: one on this altitude and the other on the VerticalRate. Intuitively, this consumes tree nodes (and hence model complexity), and as the tree growing algorithm is a greedy algorithm (it creates one condition/node at a time), it is not certain that the first condition will be selected (because it was the locally optimal choice) as the first condition might be good only if it is followed by the second one.

• A17 - The authors imputed the missing values using the median. I think this is for the best here, but sometimes if the fact they are missing is correlated to the variable to be predicted, it is better to left them as missing values. This way, missing values can be treated separately, and be put on the left or right side of the condition which should help reduce the training loss, see https://github.com/microsoft/LightGBM/issues/2921. In the authors case, the fact that the value is missing is independent of the mass of the airplane, a priori, and hence it should be better to impute them with median. It was just a question that popped in my head during my reading.

Reviewer 2

The authors present an interesting framework for predicting aircraft weight using open data. The manuscript is well-structed and the results are promising. However, I have several concerns regarding the model creation process that require further attention.

• M1 - Sec 2.3.1: While the feature selection is important for accurate prediction, some choices are questionable from an aircraft dynamics perspective. For example, the mean temperature might be less informative than the deviation from ISA, which directly reflects atmospheric conditions influencing aircraft performance. Please justify the choice of using mean temperature. In the same way, the maximum altitude could be more relevant than average altitude for weight prediction.

• M2 - Line 140: How was the wind information obtained? Is it included in the open data sets?

• M3 - Line 157: I do not understand how the authors use the level segments. The manuscript mentions three segments (departure, enroute, and arrival). Is "level segment" a fourth category? Please provide a clear definition and explain its role.

• M4 - Line 180: Why were discrete flight durations used instead of continuous values, while most other features are continuous.

• M5 - Line 184: What is the "speed per distance"? Speed is time-dependent while flown distance is a single value of the flight. Please clarify the definition.

• M6 - Line 194: The manuscript calculates both "mean vertical speed" and "elevation gradient". These feature seems redundant.

• M7 - Sec 2.3: Sec. 2.3 lists various variables, but it is unclear which ones are finally used. You should provide a complete list of selected features and their count.

• M8 - Line 258: Please provide references for FastAI’s Tabluar learner and SAINT model.

• M9 - Sec 2.4.1: More details about ANN structures are needed, such as the number of nodes, layers, etc. Additionally, please specify the hyperparameters and their chosen values?

• M10 - Figure 9 - This figure suggests that the distribution varies across aircraft types. This infers that the estimation model tries to improve the accuracy of yellow aircraft rather than green aircraft. It would be beneficial to calculate the RMSE separately for each type. Relying solely on a single overall RMSE might mask performance discrepancies.

• M11 - Table 5: Table 5 provides RMSE values for different algorithms. Including a baseline comparison, such as RMSE using typical weights in each aircraft type/range or just FAA_Weight, would provide valuable context for evaluating the model performance. Also, it is important to discuss the performance compared to existing methods in the literature.

• M12 - Table 6: The list of ablated components does not align with the sections in 2.3-2.4, which makes it difficult to understand which features are grouped together. Please clarify the mapping between components and feature categories.

Reviewer 3

The paper presents one of the winning solutions to the PRC data challenge. The description and explanation of the paper are very thorough, leaving almost no stone unturned. I recommend accepting the paper with minor revisions.

• S1 - It would be nice to mention the data source, Eurocontrol PRU, in the abstract of the paper.

• S2 - Line 13: Abbreviations can be removed or updated (these are examples from the template).

• S3 - I suggest including a table that provides an overview of all features used for training the model in section 2.3.

• S4 - Structurally, I think section 3.1 on exploratory data analysis could be moved earlier, perhaps after section 2.2. However, this is a personal preference.

• S5 - In section 3.4, it would be helpful to provide more background on the "ablation study," such as what it is and how it provides more insights into the models.

• S6 - The colors and labels in Figure 10 are a bit hard to see. Is there a better way to visualize the results? Perhaps by creating a subplot for each aircraft type?

• S7 - Finally, please create a permanent archive with DOI using Zenodo (or a similar service) for the GitHub repository. This way, the code is archived even when GitHub is no longer accessible.

Response - round 1

We would like to thank the reviewers for the valuable comments and feedback, which have contributed to enhancing the quality and clarity of our manuscript. Detailed responses to each reviewer’s comments are provided below.

Reviewer 1

This article presents the methodology used by the authors to obtain a machine learning model predicting the takeoff weight of commercial flights in 2022. This work was done to compete in the Performance Review Commission (PRC) Data Challenge 2024 in which the authors ranked second.

The article describes the data and the method used to obtain good predictions of aircraft takeoff weight. An analysis of the feature importance and an ablation study are done, showing which parts of the model are important or not. Some parts of the method might not be useful which is fine given the limited time competition context. Readers might think that these parts are useful. In order to not mislead the readers, the “Detailed Remarks/Questions” section below details which parts might be tagged as not (that) useful. Some details are missing and there is one theoretical paragraph that might not be relevant (see “Detailed Remarks/Questions” below), but given this easy to add/correct, I recommend an accept with minor revision.

Questions Related to “2.2 Data Preprocessing” and “2.3 Feature Engineering”

A1 - ADS-B trajectory data are somewhat “dirty”, the article does not mention any “cleaning” process. I looked at get_traj_features4.R and did not see any but I am not “fluent” in R and maybe I missed something. Is there really no trajectory cleaning ? Maybe the article could state clearly if there is a cleaning process or not, and it should be described in the article if there is one.

A2 - We do not always have data for the whole trajectory. As a consequence some statistics concerning the efficiency of the vertical profile might not be “accurate”. For instance the total time flown in level flight of two flights might differ only because the ADS-B coverage is not the same. Do you have a way to handle these kind of unbalance due to the ADS-B coverage?

A3 - Concerning the feature $\mathrm{taxi\_ratio}$ in the duration-based section, it is the taxiing time over the total flight duration, indicating the inefficiency. This ratio is then normalized again by dividing it by the median value to facilitate the comparison across flights. However, looking at
feature_engineering.ipynb, it is divided by the median of all the flights, hence it is simply a linear transformation of the feature that will have no impact on tree-based algorithms/models (which are invariant to strictly monotone transformation of their feature) and neural network models in which each feature is typically normalized/standardized beforehand. If I did not miss something on that, maybe the paragraph about the division by the median could be removed as this division by the median do not change anything and might misled the readers into believing that it does something.

A4 - Concerning the speed per distance, still in the duration-based section, looking at
feature_engineering.ipynb, it is actually equal to 1/flight_duration. As such, if I did not miss something, the phrase “This feature provides a normalized speed metric relative to the flight distance, assisting in analyzing speed efficiency over varying distances.” in the article should be modified. Furthermore, this transformation is strictly monotone, and hence will have no impact on tree-based algorithms/models, aside from multiplying by 2 the probability to be se considered as a split feature during the tree growing phase.

A5 - Looking at get_traj_features4.R, feature_engineering.ipynb and catboost.ipynb, for the “first ten points features”, some features are not listed in the article despsite being used like the altitude and the sqrt_tas features. I know that the sqrt_tas features are in the end not used as they are eliminated by the catboot selection process, however they could be listed as they were considered in the selection process. As a side note, the “squared” transformation is strictly monotone, and as such tas_1 and sqrd_tas_1 are basically the same feature, and this might have an impact on their respective importance score.

A6 - From the “Interaction Features” section alone, the rationale is not clear behind the operation done, how multiplying the airspeed by the specific energy “help us understand how an aircraft’s speed influences its energy efficiency during different stages of flight”. Could you elaborate on this?

Questions Related to “2.4 Predictive Models”

A7 - As a suggestion, you could change the subsection “Predictive Models” to a section. With this change, you will be able to add two subsections to “Articficial Neural Networks”, one for the classical and one for SAINT. An other benefit is having a new section “Gradient Boosted Trees” containing the three gradient based methods. This section could contain a description of the hyperparameter that are shared between these methods. As a side note, the description of these hyperparameters is made inside Extreme Gradient Boosting, the last method described.

A8 - Even though you did not use optuna to tune the neural networks, maybe you can have hyperparameters tables for the classical and SAINT neural networks. For SAINT, you performed a grid search on the width, maybe you could show this grid of hyperparameters in a table. For the classical neural network, from the text, it is not clear how the hyperparameters were selected. As a side note the batch size is not mentioned in the text despite being an important parameter of the training phase.

A9 - For the hyperparameter tuning of CatBoost, I am not sure that changing the learning rate is a good practice. The obtained hyperparameters are searched with a “big” learning rate, thus the obtained hyperparameter could be sub-optimal when considered together with a reduced learning rate. Maybe, the article could provide figures on compute times to help justify this choice.

A10 - For the hyperparameter tuning of CatBoost, the early stopping is used to train the final model, it is not clear that it was also used in the hyperparameter tuning phase.

A11 - Concerning the ensemble section, I am not sure the theoretical framework presented is consistent with the way the ensemble is built. The theoretical framework presented is typically used in bagging/random forest: the predictive models are just averaged without a “meta-learning” above them that optimize weights using the training set. As the coefficients are actual constants, the variance decomposition is easy to determine.

In the article, the predictive models are combined using (a part of?) the training set to optimize the weights. Furthermore the bias-variance decomposition is the expectation over (at least) a random training set, as a consequence the weights are not constants (w.r.t. the training set) and the variance decomposition in the article cannot be made here, it is not that straightforward anymore. The ensemble method used in the article is probably more related to “stacking” ([Breiman 1996]), and a more relevant theoretical framework can be found in the “Super Learner” paper ([Van der Laan et al. 2007]).

A12 - Concerning the ensemble section, it could detail whether the training set used to learn the models is the same used to compute the optimal weights, or is it split in two (or K) parts to do that?

Questions Related to “3 Discussions”

A13 - Concerning the Table 6 for the ablation study, the temporal and geographical features are described in their own subsubsections 2.3.2 and 2.3.4 but features such as “aircraft characteristics” and “operational features” are not described. Maybe the authors could add subsubsections for these two groups. Additionaly, some features in these groups are not described in the feature section.

Typos

A14 - line 303: “hype parameters” to “hyperparameters”.

A15 - line 430: In “The A320 was the most used equipment”, the word “equipment” sounds weird, maybe “aircraft type” would be better.

Appendices

Some thoughts I had reading the article that can be left unanswered by the authors as they might be unimportant, irrelevant or wrong:

A16 - We do not always have data for the whole trajectory. As a consequence the first ten observed points are somewhat random: they are the first points inside the ADS-B coverage. The altitude “frontier” of the ADS-B coverage may vary from one location to an other. As such, the altitude of these first points may vary, regardless of the way the aircraft is operated. To illustrate how this can be problematic, let us consider the feature $VerticalRate_1$ which is the vertical rate computed on the first observed point. If you observe $VerticalRate_1\left(Flight_A\right)>VerticalRate_1\left(Flight_B\right)$, maybe $Flight_A$ is lighter than $Flight_B$ or maybe they have a similar weight but $Flight_B$ is higher than $Flight_A$ and consequently have a lower rate of climb. The feature “height relative to the origin airport” can mitigate this issue but the tree based models will have to “chain” two conditions/nodes: one on this altitude and the other on the VerticalRate. Intuitively, this consumes tree nodes (and hence model complexity), and as the tree growing algorithm is a greedy algorithm (it creates one condition/node at a time), it is not certain that the first condition will be selected (because it was the locally optimal choice) as the first condition might be good only if it is followed by the second one.

A17 - The authors imputed the missing values using the median. I think this is for the best here, but sometimes if the fact they are missing is correlated to the variable to be predicted, it is better to left them as missing values. This way, missing values can be treated separately, and be put on the left or right side of the condition which should help reduce the training loss, see https://github.com/microsoft/LightGBM/issues/2921. In the authors case, the fact that the value is missing is independent of the mass of the airplane, a priori, and hence it should be better to impute them with median. It was just a question that popped in my head during my reading.

Reviewer 2

The authors present an interesting framework for predicting aircraft weight using open data. The manuscript is well-structed and the results are promising. However, I have several concerns regarding the model creation process that require further attention.

M1 - Sec 2.3.1: While the feature selection is important for accurate prediction, some choices are questionable from an aircraft dynamics perspective. For example, the mean temperature might be less informative than the deviation from ISA, which directly reflects atmospheric conditions influencing aircraft performance. Please justify the choice of using mean temperature. In the same way, the maximum altitude could be more relevant than average altitude for weight prediction.

M2 - Line 140: How was the wind information obtained? Is it included in the open data sets?

M3 - Line 157: I do not understand how the authors use the level segments. The manuscript mentions three segments (departure, enroute, and arrival). Is "level segment" a fourth category? Please provide a clear definition and explain its role.

M4 - Line 180: Why were discrete flight durations used instead of continuous values, while most other features are continuous.

M5 - Line 184: What is the "speed per distance"? Speed is time-dependent while flown distance is a single value of the flight. Please clarify the definition.

M6 - Line 194: The manuscript calculates both "mean vertical speed" and "elevation gradient". These feature seems redundant.

M7 - Sec 2.3: Sec. 2.3 lists various variables, but it is unclear which ones are finally used. You should provide a complete list of selected features and their count.

M8 - Line 258: Please provide references for FastAI’s Tabluar learner and SAINT model.

M9 - Sec 2.4.1: More details about ANN structures are needed, such as the number of nodes, layers, etc. Additionally, please specify the hyperparameters and their chosen values?

M10 - Figure 9 - This figure suggests that the distribution varies across aircraft types. This infers that the estimation model tries to improve the accuracy of yellow aircraft rather than green aircraft. It would be beneficial to calculate the RMSE separately for each type. Relying solely on a single overall RMSE might mask performance discrepancies.

M11 - Table 5: Table 5 provides RMSE values for different algorithms. Including a baseline comparison, such as RMSE using typical weights in each aircraft type/range or just FAA_Weight, would provide valuable context for evaluating the model performance. Also, it is important to discuss the performance compared to existing methods in the literature.

M12 - Table 6: The list of ablated components does not align with the sections in 2.3-2.4, which makes it difficult to understand which features are grouped together. Please clarify the mapping between components and feature categories.

Reviewer 3

The paper presents one of the winning solutions to the PRC data challenge. The description and explanation of the paper are very thorough, leaving almost no stone unturned. I recommend accepting the paper with minor revisions.

S1 - It would be nice to mention the data source, Eurocontrol PRU, in the abstract of the paper.

S2 - Line 13: Abbreviations can be removed or updated (these are examples from the template).

S3 - I suggest including a table that provides an overview of all features used for training the model in section 2.3.

S4 - Structurally, I think section 3.1 on exploratory data analysis could be moved earlier, perhaps after section 2.2. However, this is a personal preference.

S5 - In section 3.4, it would be helpful to provide more background on the "ablation study," such as what it is and how it provides more insights into the models.

S6 - The colors and labels in Figure 10 are a bit hard to see. Is there a better way to visualize the results? Perhaps by creating a subplot for each aircraft type?

S7 - Finally, please create a permanent archive with DOI using Zenodo (or a similar service) for the GitHub repository. This way, the code is archived even when GitHub is no longer accessible.