Original paper

DOI for the original paper: https://doi.org/10.59490/joas.2024.7890

Review - round 1

Reviewer 1

The authors investigate the use of OpenSky data for predicting aircraft airborne holding times, focusing on the four holding stacks within the TMA surrounding London Heathrow Airport. Although understanding / predicting airborne hold times would be useful for a variety of purposes (e.g., better en route metering, flow control settings, ATFM regulation settings, etc.) , I struggled to see the methodological advancements in the paper: The authors use standard ML approaches (RNN, LSTM), and furthermore the training process was a bit confusing. I provide further detailed comments below:

(1.) The study leverages data from April to August 2023. However, this misses the majority of seasonal trends, e.g., winter-time schedules. The schedule that is being run will impact the number of flights, etc. seeking to arrive to LHR/LGW. This is particularly true for larger hub airports in Europe, where a significant portion of them are slot controlled, and these slots are re-done per season. It would be great if the authors could comment on some of the limitations of their study, particularly as it pertains to data time frame, generalizability, and how future studies could build on this paper but with more comprehensive data.

(2.) A relatively minor point regarding Figure 1 – is this geographically representative of the TMA surrounding LHR and the local London airports? It seems to be a relatively simplistic approximation with a simple circle. Especially seeing how the authors would like to *predict* airborne holding patterns, it would be interesting if they examine the role that airspace geometry potentially plays in how often an aircraft is asked to hold. For example, perhaps a simpler airspace geometry allows for the definition of specific holding stacks / boxes that can be a bit more organized, compared to the case with more complex airspace. If the authors could touch on within the discussion, or perhaps run a new experiment with more realistic TMA boundaries, that would be great. Thanks!

(3.) I am not quite sure what is the purpose of Figure 2. Unless if this heatmap representation is supposed to provide insights into, e.g., distributions of flight positions that are eventually used for the actual prediction algorithm, what is the advantage of this heatmap view over, e.g., just plotting the trajectories themselves? This figure in its current form would need a legend as well, it is not clear what the different color scales mean. I would suggest that the authors provide a motivation for why this visualization is needed.

(4.) It seems to me that using airport delay as a factor to predict airborne holding times is a bit backwards: Generally airborne holding occurs when there is an unexpected reduction in arrival capacity at the airport, and/or (in rare instances) there is a lack of physical space to continue accepting aircraft on the taxiways, tarmac, etc. Both of these factors can show up in airport delay statistics, but just because an airport has a lot of delays does not necessarily indicate that a lot of airborne holding will happen. It would be great if the authors could better justify their use of delays as a predictor / factor here, rather than something further back in the chain, e.g., current airport arrival rates, etc.

(5.) Figure 3 provides intuition to how airborne holding is detected in the trajectory data. I am surprised that the authors simply used the position of the aircraft in relations to the bounding boxes drawn / visualized to detect holding patterns. Since the authors have full access to the entire trajectory, it seems to me that a much more robust and generalizable approach would have been to detect holding as a function of, e.g., turn rates, heading, etc., i.e., physical parameters of the trajectories, and not just the positions? It is also unclear if the authors consider aspects such as path stretching, additional vectoring, as airborne holding as well. While, strictly speaking, these are more speed control tactics, it would increase the in-air time. This point alos needs to be clarified.

(6.) A minor editing point in Figure 4 – it is relatively clear that the authors modified using, e.g., a paint tool or drawing tool, to add the entry angles into a holding stack. It looks a bit unpolished, especially the uneven/asymmetric arcs representing an angle. I would strongly suggest the authors re-do this figure with more polished modifications.

(7.) This is perhaps just a phrasing / wording comment, but it is quite strong to say "ML algorithms are only capable of handling numerical values." Of course, the authors indicate that the ML algorithms utilized in this paper require some kind of numerical encoding / embedding scheme to be performed prior to ingesting the trajectory data. The authors should clarify that ML algorithms *can* handle non-numerical values, but some kind of transformation must be done first.

(8.) In the paragraph starting on line 150, the authors mention the wake turbulence categories which are used for aircraft separation and sequencing. It would be great if there could be additional discussion on the impact of wake category re-categorization, since I believe RECAT-EU would indeed apply to flights coming into LHR. If the authors could comment on how these additional categories might impact their results, that would be great.

(9.) Table 9 is a bit confusing to me. Why are different window lengths used for the same algorithm when applied to different holding stacks? This seems like it may overfit to a specific holding stack at LHR, which would throw the ability of the ML prediction algorithm to generalize in doubt. Also, I do not see any sensitivity analyses performed on the window lengths – how do we know that for, e.g., the BIG holding stack, that 15 is the optimal window length to use?

Reviewer 2

The paper presents an application of machine learning to predict aircraft holding times at London Heathrow Airport. The methodology is well-structured and includes the use of both LightGBM and LSTM models. 

Specific Comments 1. Introduction

• It might be beneficial to elaborate on how the proposed models could integrate into existing air traffic control systems. For instance, would real-time data feeds like Delay Index be necessary? Is the Delay Index available in real time or only post-operations?

• A clearer explanation of the interaction between air traffic controllers and these predictions would enhance the reader’s understanding. Additionally, discussing the timeframe of predictions would add depth. For example, are predictions limited to an hour in advance, or could they feasibly extend to several hours? Clarifying whether the current look-ahead time is operationally sufficient would provide valuable context.

2. Related Works The literature review could benefit from additional references to contextualize the novelty of this work. For instance, the study presented at last year’s OpenSky symposium—On the Causes and Environmental Impact of Airborne Holdings at Major European Airports (Dalmau, R.; Very, P.; and Jarry, G., 2023).

3. Methodology

• The methodology section would benefit from an introductory paragraph summarizing its structure and objectives, placed immediately after the "3. Methodology" heading.

• Further clarification on the extraction of information from textual METARs is recommended. Did the authors utilize existing tools or develop custom regular expressions for this purpose?

• There appears to be some ambiguity regarding data access. AeroDataBox is cited as a source, but their website indicates the data is not freely available. It would be helpful to clarify the nature of the data access.

• Regarding the detection of holdings, why did the authors choose not to utilize the method available in the Traffic library? A brief comparison of this approach with the one adopted in the study, particularly in terms of predictive performance, would be insightful—even if only discussed qualitatively.

• The use of one-hot encoding (OHE) in LightGBM might deserve reconsideration as it is not recommended by the authors of that algorithm. While the (assumed) rationale of maintaining consistency with neural networks is understandable, exploring alternative encoding methods, such as embeddings, could enhance generalizability.

• A minor but important clarification: using the landing runway to compute wind components during real-time inference might pose challenges, as the runway assignment is only known retrospectively. How is this addressed in the implementation?

• Referring to the task as a "time series problem" is somewhat broad. A more precise term, such as "time series regression" or "regression with time series features," might better describe the methodology. All in all, both regression and time series problems are regression problems.

• The choice of LSTM over simpler alternatives like GRU could be better justified. Is the complexity of the prediction task high enough to necessitate the added sophistication of LSTMs?

• Table 3 introduces separate models for tasks with a single varying parameter. Instead, conditioning the model on this parameter (e.g., look-ahead time, or the holding stack identifier) could allow for improved generalization as the model would be trained with more data.

• LightGBM is used effectively in the study, but it would be helpful to discuss why it was chosen over other GBDT implementations, such as CatBoost or XGBoost. Were these alternatives evaluated?

• Regarding numerical feature scaling, while min-max scaling is effective, ensuring features approximate a Gaussian distribution might enhance neural network performance. Exploring a power transformation followed by standardization could be a worthwhile consideration.

• Dropout and early stopping are used to mitigate overfitting. Was overfitting observed in preliminary experiments? Providing a brief explanation would be helpful.

4. Results and Discussion The explanation of feature importance could be more detailed. For instance, which specific method was used? The reference to "permutation importance" in Section 3.4.2.1 suggests this might refer to features rather than parameters. Clarifying this terminology would improve readability. Additionally, exploring Shapley values might offer more insights into marginal feature contributions. 5. Conclusions and Future Work The conclusions section could place greater emphasis on operational aspects. For instance, what changes, if any, would be required to integrate this model into current operations? Who would use the predictions, and at what point in the decision-making process? Addressing whether the current model is ready for operational deployment—or highlighting areas for improvement—would underscore the practical novelty of the work.

Reviewer 3

L61: Does this approach work well? What are the strengths and limitations of [4]?

L65: What are the inputs of the model? Are they aircraft trajectories, flight parameters, or something else?

L70: Does this result require improvement?

L72: Is the input of the model an image of a trajectory, including the trajectory and the background map?

Related Work Section: This section could be significantly improved. While it conveys that predicting holding patterns using machine learning is an interesting topic, the methods presented lack detailed explanations. It is unclear how these methods were used in developing your model, what their strengths and limitations are, and which gap in the literature your work addresses. Furthermore, predicting holding patterns with machine learning likely involves substantial data collection, processing, and feature extraction, but this is not clearly articulated. Specifically, it is unclear what types of inputs the methods employed, which would provide a more seamless transition to your methodology. Currently, the reader learns about the data you collected, but the rationale behind selecting this specific data remains unclear.

L82: Does ADS-B data refer to traffic surveillance data?

L85: How did you determine the bounding box?

L90: Is presenting a traffic object here essential? Consider whether this detail is necessary.

L111: It would be helpful to present the general idea of this algorithm. Is its sole purpose to detect flights performing holding patterns and classify them according to specific holding stacks?

L114: Is this the description of the algorithm introduced in L111? This is not entirely clear.

L116: Are there no false positives caused by sharp turns or tromboning maneuvers? Is this criterion applied exclusively within the vicinity of a given holding stack?

L123: Why are two separate algorithms needed for right- and left-turn holding patterns? Is the only difference a +20° or -20° heading change? If so, is it necessary to define them as two distinct algorithms? Additionally, since the direction of holding is typically fixed for a specific holding stack, what happens if a stack has a right-turn holding pattern (e.g., Lambourne)? If the holding direction is indeed fixed, could the algorithm simply use the absolute heading difference instead of separate algorithms?

L127: Providing validation results for this algorithm would be helpful. How confident are you in its accuracy for detecting holding patterns?

L130: How would you explain this observation? Why is it relevant to your study?

L133: Are the manipulations mentioned here those explained later in the chapter?

L136: Do you use the term “dependent variable” in subsequent sections?

L140: This is an awkward phrasing. In machine learning, “numerical values” typically refer to “continuous variables.”

L145: Is a score of 30 considered good or bad? Is the score a continuous or integer value? If it’s an integer, do you treat it as continuous or categorical?

L140–149: This paragraph is poorly structured. It’s unclear which variables were processed, which ones underwent one-hot encoding, and how ATMAP processing relates to the calculation of runway headwind/crosswind. Clarify these points and improve the logical flow.

L159: To clarify, you retrieved the aircraft type via the Traffic library, then determined each flight’s wake category and engine type? Aircraft type information is often missing in Traffic—how did you address this issue?

L165: Are these models applied to both datasets, or is each model dedicated to a specific dataset?

L180: First, what is the time length of your input? Are you predicting the average holding time for the next 15 minutes based solely on the previous 15-minute average? Are you using a longer observation window? Second, are you predicting the average holding time for the entire airspace or for specific holding stacks? Lastly, regarding your train/test split, if input-output pairs are correctly aligned (e.g., each input corresponds to the subsequent time period’s output), wouldn’t shuffling the data be possible? If the observation distribution in the last month differs significantly from prior months due to weather, construction, or other factors, how does your approach handle this? While time series sometimes don’t allow random splits, it’s unclear if your work falls into this category.

L190: Is it necessary to detail the grid search procedure and hyperparameter space if the final hyperparameter values are not shared?

L197: Does a window length of 1 correspond to a 15-minute average? Additionally, the inability to shuffle your training dataset seems related to using grid search for determining window length. Consider mentioning this earlier.

L201: Are you predicting multiple future timestamps? If so, this should be clarified earlier.

L208: Are five timesteps used? Does the regression model also has sequential data? If not, what are those 5 time steps?

L209: What is the architecture of your model? How many layers does it have, and what is the size of the hidden state? What do the 50 neurons represent? The current description lacks sufficient detail for replication.

L217: Is the exact same architecture used for both array-like and sequential inputs? Unlike for the LGBM model, did you not search for the optimal observation window? Additionally, learning rate is a fundamental parameter—what value did you use?

L241: It seems the LSTM setup contributes to the poor results. Additionally, LSTMs are primarily used for sequential input data, which does not appear to align with this scenario.

L246: What feature importance analysis was performed?

L255: For clarity, consider converting results into hours and minutes.

L257: For the LSTM model, what is the input window length?

L264: The base version of SMOTE may not be suitable for oversampling time series data.

Results and Discussion Section: This section primarily describes results but provides limited explanations. The discussion could be enhanced by elaborating on the factors contributing to good and poor results and providing perspectives on operational implications. Furthermore, limitations of the study are not adequately addressed.

Response - round 1

We would like to first thank the reviewers for their attentive reading of our contribution as well as the editor who allowed us to make some corrections. The lines mentioned in the responses refer to the latest updated paper. The Github repository has been updated with the latest coding and datasets.

Response to reviewer 1

1. The study leverages data from April to August 2023. However, this misses the majority of seasonal trends, e.g., winter-time schedules. The schedule that is being run will impact the number of flights, etc. seeking to arrive to LHR/LGW. This is particularly true for larger hub airports in Europe, where a significant portion of them are slot controlled, and these slots are re-done per season. It would be great if the authors could comment on some of the limitations of their study, particularly as it pertains to data time frame, generalizability, and how future studies could build on this paper but with more comprehensive data.

2. A relatively minor point regarding Figure 1 – is this geographically representative of the TMA surrounding LHR and the local London airports? It seems to be a relatively simplistic approximation with a simple circle. Especially seeing how the authors would like to *predict* airborne holding patterns, it would be interesting if they examine the role that airspace geometry potentially plays in how often an aircraft is asked to hold. For example, perhaps a simpler airspace geometry allows for the definition of specific holding stacks / boxes that can be a bit more organized, compared to the case with more complex airspace. If the authors could touch on within the discussion, or perhaps run a new experiment with more realistic TMA boundaries, that would be great. Thanks!

3. I am not quite sure what is the purpose of Figure 2. Unless if this heatmap representation is supposed to provide insights into, e.g., distributions of flight positions that are eventually used for the actual prediction algorithm, what is the advantage of this heatmap view over, e.g., just plotting the trajectories themselves? This figure in its current form would need a legend as well, it is not clear what the different colour scales mean. I would suggest that the authors provide a motivation for why this visualization is needed.

4. It seems to me that using airport delay as a factor to predict airborne holding times is a bit backwards: Generally airborne holding occurs when there is an unexpected reduction in arrival capacity at the airport, and/or (in rare instances) there is a lack of physical space to continue accepting aircraft on the taxiways, tarmac, etc. Both of these factors can show up in airport delay statistics, but just because an airport has a lot of delays does not necessarily indicate that a lot of airborne holding will happen. It would be great if the authors could better justify their use of delays as a predictor / factor here, rather than something further back in the chain, e.g., current airport arrival rates, etc.

5. Figure 3 provides intuition to how airborne holding is detected in the trajectory data. I am surprised that the authors simply used the position of the aircraft in relations to the bounding boxes drawn / visualized to detect holding patterns. Since the authors have full access to the entire trajectory, it seems to me that a much more robust and generalizable approach would have been to detect holding as a function of, e.g., turn rates, heading, etc., i.e., physical parameters of the trajectories, and not just the positions? It is also unclear if the authors consider aspects such as path stretching, additional vectoring, as airborne holding as well. While, strictly speaking, these are more speed control tactics, it would increase the in-air time. This point also needs to be clarified.

6. A minor editing point in Figure 4 – it is relatively clear that the authors modified using, e.g., a paint tool or drawing tool, to add the entry angles into a holding stack. It looks a bit unpolished, especially the uneven/asymmetric arcs representing an angle. I would strongly suggest the authors re-do this figure with more polished modifications.

7. This is perhaps just a phrasing / wording comment, but it is quite strong to say "ML algorithms are only capable of handling numerical values." Of course, the authors indicate that the ML algorithms utilized in this paper require some kind of numerical encoding / embedding scheme to be performed prior to ingesting the trajectory data. The authors should clarify that ML algorithms *can* handle non-numerical values, but some kind of transformation must be done first.

8. In the paragraph starting on line 150, the authors mention the wake turbulence categories which are used for aircraft separation and sequencing. It would be great if there could be additional discussion on the impact of wake category re-categorization, since I believe RECAT-EU would indeed apply to flights coming into LHR. If the authors could comment on how these additional categories might impact their results, that would be great.

9. Table 9 is a bit confusing to me. Why are different window lengths used for the same algorithm when applied to different holding stacks? This seems like it may overfit to a specific holding stack at LHR, which would throw the ability of the ML prediction algorithm to generalize in doubt. Also, I do not see any sensitivity analyses performed on the window lengths – how do we know that for, e.g., the BIG holding stack, that 15 is the optimal window length to use?

Response to reviewer 2

1. It might be beneficial to elaborate on how the proposed models could integrate into existing air traffic control systems. For instance, would real-time data feeds like Delay Index be necessary? Is the Delay Index available in real time or only post-operations?

2. A clearer explanation of the interaction between air traffic controllers and these predictions would enhance the reader’s understanding. Additionally, discussing the timeframe of predictions would add depth. For example, are predictions limited to an hour in advance, or could they feasibly extend to several hours? Clarifying whether the current look-ahead time is operationally sufficient would provide valuable context.

2. Related Works

3. The literature review could benefit from additional references to contextualize the novelty of this work. For instance, the study presented at last year’s OpenSky symposium—On the Causes and Environmental Impact of Airborne Holdings at Major European Airports (Dalmau, R.; Very, P.; and Jarry, G., 2023).

3. Methodology

4. The methodology section would benefit from an introductory paragraph summarizing its structure and objectives, placed immediately after the "3. Methodology" heading.

5. Further clarification on the extraction of information from textual METARs is recommended. Did the authors utilize existing tools or develop custom regular expressions for this purpose?

6. There appears to be some ambiguity regarding data access. AeroDataBox is cited as a source, but their website indicates the data is not freely available. It would be helpful to clarify the nature of the data access.

7. Regarding the detection of holdings, why did the authors choose not to utilize the method available in the Traffic library? A brief comparison of this approach with the one adopted in the study, particularly in terms of predictive performance, would be insightful—even if only discussed qualitatively.

8. The use of one-hot encoding (OHE) in LightGBM might deserve reconsideration as it is not recommended by the authors of that algorithm. While the (assumed) rationale of maintaining consistency with neural networks is understandable, exploring alternative encoding methods, such as embeddings, could enhance generalizability.

9. A minor but important clarification: using the landing runway to compute wind components during real-time inference might pose challenges, as the runway assignment is only known retrospectively. How is this addressed in the implementation?

10. Referring to the task as a "time series problem" is somewhat broad. A more precise term, such as "time series regression" or "regression with time series features," might better describe the methodology. All in all, both regression and time series problems are regression problems.

11. The choice of LSTM over simpler alternatives like GRU could be better justified. Is the complexity of the prediction task high enough to necessitate the added sophistication of LSTMs?

12. Table 3 introduces separate models for tasks with a single varying parameter. Instead, conditioning the model on this parameter (e.g., look-ahead time, or the holding stack identifier) could allow for improved generalization as the model would be trained with more data.

13. LightGBM is used effectively in the study, but it would be helpful to discuss why it was chosen over other GBDT implementations, such as CatBoost or XGBoost. Were these alternatives evaluated?

14. Regarding numerical feature scaling, while min-max scaling is effective, ensuring features approximate a Gaussian distribution might enhance neural network performance. Exploring a power transformation n followed by standardization could be a worthwhile consideration.

15. Dropout and early stopping are used to mitigate overfitting. Was overfitting observed in preliminary experiments? Providing a brief explanation would be helpful.

4. Results and Discussion

16. The explanation of feature importance could be more detailed. For instance, which specific method was used? The reference to "permutation importance" in Section 3.4.2.1 suggests this might refer to features rather than parameters. Clarifying this terminology would improve readability. Additionally, exploring Shapley values might offer more insights into marginal feature contributions.

17. The conclusion section could place greater emphasis on operational aspects. For instance, what changes, if any, would be required to integrate this model into current operations? Who would use the predictions, and at what point in the decision-making process? Addressing whether the current model is ready for operational deployment—or highlighting areas for improvement—would underscore the practical novelty of the work.

Response to reviewer 3

1. L61: Does this approach work well? What are the strengths and limitations of [4]?

2. L65: What are the inputs of the model? Are they aircraft trajectories, flight parameters, or something else?

3. L70: Does this result require improvement?

4. L72: Is the input of the model an image of a trajectory, including the trajectory and the background map?

5. Related Work Section: This section could be significantly improved. While it conveys that predicting holding patterns using machine learning is an interesting topic, the methods presented lack detailed explanations. It is unclear how these methods were used in developing your model, what their strengths and limitations are, and which gap in the literature your work addresses. Furthermore, predicting holding patterns with machine learning likely involves substantial data collection, processing, and feature extraction, but this is not clearly articulated. Specifically, it is unclear what types of inputs the methods employed, which would provide a more seamless transition to your methodology. Currently, the reader learns about the data you collected, but the rationale behind selecting this specific data remains unclear.

6. L82: Does ADS-B data refer to traffic surveillance data?

7. L85: How did you determine the bounding box?

8. L90: Is presenting a traffic object here essential? Consider whether this detail is necessary.

9. L111: It would be helpful to present the general idea of this algorithm. Is its sole purpose to detect flights performing holding patterns and classify them according to specific holding stacks?

10. L114: Is this the description of the algorithm introduced in L111? This is not entirely clear.

11. L116: Are there no false positives caused by sharp turns or tromboning maneuvers? Is this criterion applied exclusively within the vicinity of a given holding stack?

12. L123: Why are two separate algorithms needed for right- and left-turn holding patterns? Is the only difference a +20° or -20° heading change? If so, is it necessary to define them as two distinct algorithms? Additionally, since the direction of holding is typically fixed for a specific holding stack, what happens if a stack has a right-turn holding pattern (e.g., Lambourne)? If the holding direction is indeed fixed, could the algorithm simply use the absolute heading difference instead of separate algorithms?

13. L127: Providing validation results for this algorithm would be helpful. How confident are you in its accuracy for detecting holding patterns?

14. L130: How would you explain this observation? Why is it relevant to your study?

15. L133: Are the manipulations mentioned here those explained later in the chapter?

16. L136: Do you use the term “dependent variable” in subsequent sections?

17. L140: This is an awkward phrasing. In machine learning, “numerical values” typically refer to “continuous variables.”

18. L145: Is a score of 30 considered good or bad? Is the score a continuous or integer value? If it’s an integer, do you treat it as continuous or categorical?

19. L140–149: This paragraph is poorly structured. It’s unclear which variables were processed, which ones underwent one-hot encoding, and how ATMAP processing relates to the calculation of runway headwind/crosswind. Clarify these points and improve the logical flow.

20. L159: To clarify, you retrieved the aircraft type via the Traffic library, then determined each flight’s wake category and engine type? Aircraft type information is often missing in Traffic—how did you address this issue?

21. L165: Are these models applied to both datasets, or is each model dedicated to a specific dataset?

22. L180: First, what is the time length of your input? Are you predicting the average holding time for the next 15 minutes based solely on the previous 15-minute average? Are you using a longer observation window?

23. Second, are you predicting the average holding time for the entire airspace or for specific holding stacks?

24. Lastly, regarding your train/test split, if input-output pairs are correctly aligned (e.g., each input corresponds to the subsequent time period’s output), wouldn’t shuffling the data be possible? If the observation distribution in the last month differs significantly from prior months due to weather, construction, or other factors, how does your approach handle this? While time series sometimes don’t allow random splits, it’s unclear if your work falls into this category.

25. L190: Is it necessary to detail the grid search procedure and hyperparameter space if the final hyperparameter values are not shared?

26. L197: Does a window length of 1 correspond to a 15-minute average? Additionally, the inability to shuffle your training dataset seems related to using grid search for determining window length. Consider mentioning this earlier.

27. L201: Are you predicting multiple future timestamps? If so, this should be clarified earlier.

28. L208: Are five timesteps used? Does the regression model also has sequential data? If not, what are those 5 time steps?

29. L209: What is the architecture of your model? How many layers does it have, and what is the size of the hidden state? What do the 50 neurons represent? The current description lacks sufficient detail for replication.

30. L217: Is the exact same architecture used for both array-like and sequential inputs? Unlike for the LGBM model, did you not search for the optimal observation window? Additionally, learning rate is a fundamental parameter—what value did you use?

31. L241: It seems the LSTM setup contributes to the poor results. Additionally, LSTMs are primarily used for sequential input data, which does not appear to align with this scenario.

32. L246: What feature importance analysis was performed?

33. L255: For clarity, consider converting results into hours and minutes.

34. L257: For the LSTM model, what is the input window length?

35. L264: The base version of SMOTE may not be suitable for oversampling time series data.

36. Results and Discussion Section: This section primarily describes results but provides limited explanations. The discussion could be enhanced by elaborating on the factors contributing to good and poor results and providing perspectives on operational implications. Furthermore, limitations of the study are not adequately addressed.

Review - round 2

Reviewer 1

I sincerely appreciate the efforts made by the authors to address every one of my comments. I am satisfied with the authors’ edits and rebuttals. I have no additional comments, and am happy to recommend acceptance.

Reviewer 2

This paper attempts to apply machine learning (ML) techniques, specifically LightGBM and LSTM, to predict aircraft holding times at London Heathrow. While the research is relevant to air traffic management, I have identified several weaknesses and critical points that the authors must address before this paper can be considered for publication.

Introduction

The introduction briefly references the Point Merge System (PMS). The paper would benefit from mentioning other major airports using PMS for better context.

Related Works

The authors state that ML techniques have not been used to predict holding times. They should reframe this to better highlight the gap their work fills, focusing on the lack of predictive tools for holding times, not just the novelty of applying ML.

Methodology

- The 1380 NM by 1320 NM bounding box is excessive. Why such a large box was used to filter the traffic is unclear. The authors must justify this size or provide a more reasonable alternative.

- Figure 2: The inclusion of this figure is questionable. It’s just a heatmap of arrivals at Heathrow. It provides no clear conclusion or insight. Additionally, the term "one day" is vague — which day was used?

- There is no explanation of how METAR data was processed, especially regarding how the weather fields were extracted from textual reports. This must be clarified for reproducibility and transparency.

- The journal emphasizes open science, yet no code or data repository is linked in the paper. The use of AeroDataBox, which requires a paid subscription, is also problematic (but I understand nothing can be done at this point). It would have been nice to create a synthetic dataset just for researchers to reproduce the methodology.

- The method for defining a bounding box around each individual holding stack lacks clarity. It is essential that the authors provide precise details on how these bounding boxes were determined.

- No (scientific) performance metrics provided for the method to detect holdings. Given the reliance on this detection, the authors should validate this method against ground truth data and provide metrics for its performance. Comparing this approach with the Traffic library’s methods is also recommended.

- One-hot encoding (OHE) is not recommended for LightGBM, as per the creators of the model. The authors should acknowledge this and either justify their decision or explore better encoding options for categorical data — or at least warn the readers that OHE is not the right approach when using LightGBM (and other GBDT models like CatBoost).

- The approach to assign a landing runway to each individual flight works because Heathrow has a single runway in use, but this is not generalizable to other airports. The authors must clarify how the model can be adapted to airports with multiple landing runways. Specifically, how would the authors identify the landing runway of each individual flight? One could use the methods provided by the Traffic library (which work very well). The authors could give some guidance to the readers on this matter. 

- The authors use RECAT-EU and Aircraft Type features in the regression problem but do not clearly explain how they are integrated into the time series model. Table 2 makes this clearer, but the explanation should be presented earlier in the text.

- In Table 1, the time of day is treated as a continuous variable, but this leads to issues at midnight. It should be treated as a categorical or cyclic variable to account for discontinuities at midnight.

- The authors specify units (kt, ...) in Table 1, but GBDT models like LightGBM are not sensitive to scaling. This section could be revised to remove unnecessary details.

- The authors justify using LightGBM based on its performance in another paper. However, CatBoost is known to handle categorical features better, and a comparison between LightGBM and CatBoost would be a valuable addition to this paper.

- While the authors justify using LSTM based on its success in another work, the performance improvements over simpler models like GRU are not sufficiently discussed. A comparison between LSTM and GRU models would be valuable, as GRUs can sometimes offer a more efficient alternative to LSTMs.

- The authors use different data splits for regression and time series problems. It would have been more logical to use consistent time frames across both problems to ensure a fair comparison of results in the test set between the regression and time series models.

- Why PowerTransformer was used in the regression and MaxMin scaling in the time series. This is not clear and must be clarified.

- Why not use a 9th Model with 4 heads, each one predicting the average delay of each holding stack in a multi-output setting? This could be included to the "future work" section. 

- The authors use an LSTM with 50 neurons and 4 layers, and dropout of 0.2, but they do not explain why these particular choices were made. These hyperparameters should be optimized, and the authors should explain why such architecture was chosen in the first place.

Results and Discussion

- The inclusion of  table 8 is redundant. The SHAP values could be better visualized in figures. Tables here are unnecessary and distract from the discussion.

- The explanation of SHAP values is too simplistic. The authors should provide more context for readers who may not be familiar with these techniques, explaining how they contribute to the model interpretation.

- The readability of Figure 9 is poor. The authors should try using a line plot instead to improve the clarity of their results.

Conclusion and Future Work

- Rather than building separate models for each look-ahead time, the authors could integrate this feature into a single model. This would improve generalization and make the implementation easier.

- The authors mention future comparisons between encoding techniques, but they fail to do this in the current paper. This is a relatively easy experiment to conduct.

I understand that many of the comments above, such as "The authors mention future comparisons between encoding techniques, but they fail to do this in the current paper. This is a relatively easy experiment to conduct and would provide valuable insights." may imply the need to carry out additional experiments. However, I would like to clarify that performing these experiments is not strictly necessary at this stage. It would be entirely acceptable if the authors explicitly outline in the manuscript what they propose to explore in future work or suggest these points as recommendations for the research community. This would adequately address the identified gaps without overburdening the current scope of the paper.

Reviewer 3

The manuscript greatly improved and is much clearer. The authors carefully addressed all the comments. The reader can clearly grasp the stakes of the study as well as the different model implemented. Here are additional minor comments:

• I find the literature review better explained and we get a clear overview about previous work concerning holding patterns. However, the clear identification of the gaps still needs minor improvements:

  • The gap you identified is the lack of ML methods to predict holding times?

  • Why ML methods might be more adapted to the task than previous methodologies?

  • Is your paper purely exploratory: the main goal is to check if ML methods have the potential to overcome previous work?

  • Why LSTM and LGBM seem to be the obvious choice? Previous work in other field already demonstrated the reliability of those architectures for similar problems?

• When detecting a holding pattern, you compare the heading for two consecutive observations separated by 30s. If the heading is more than 20 degrees, then the holding count begins. However, the proper holding pattern could have been initiated anywhere between those two observations, which might lead to an error up to 30s between the real holding pattern start, and the detected one. Thus, from one sample to another in your training dataset, the detected start can be either the very beginning of the holding, or up to 30s in the holding. The same consideration holds for the end, which can lead to discrepancies up to one minute overall. Thus, your model performance depends on your ML model as much as on your detection algorithm. It might be worth to explain further the validation of your model (e.g. comparing the calculated times with the one manually labeled), and also mention it in the limitations if necessary.

• It might be beneficial to remind the size of each constructed training datasets.

• According to figure 9, the model seems to constantly underestimate the holding time. Do you have an idea about the reason?

Response - round 2

Response to reviewer 1

I sincerely appreciate the efforts made by the authors to address every one of my comments. I am satisfied with the authors’ edits and rebuttals. I have no additional comments, and am happy to recommend acceptance.

Response to reviewer 2

1. The introduction briefly references the Point Merge System (PMS). The paper would benefit from mentioning other major airports using PMS for better context.

2. The authors state that ML techniques have not been used to predict holding times. They should reframe this to better highlight the gap their work fills, focusing on the lack of predictive tools for holding times, not just the novelty of applying ML.

3. The 1380 NM by 1320 NM bounding box is excessive. Why such a large box was used to filter the traffic is unclear. The authors must justify this size or provide a more reasonable alternative.

4. Figure 2: The inclusion of this figure is questionable. It’s just a heatmap of arrivals at Heathrow. It provides no clear conclusion or insight. Additionally, the term "one day" is vague — which day was used?

5. There is no explanation of how METAR data was processed, especially regarding how the weather fields were extracted from textual reports. This must be clarified for reproducibility and transparency.

6. The journal emphasizes open science, yet no code or data repository is linked in the paper. The use of AeroDataBox, which requires a paid subscription, is also problematic (but I understand nothing can be done at this point). It would have been nice to create a synthetic dataset just for researchers to reproduce the methodology.

7. The method for defining a bounding box around each individual holding stack lacks clarity. It is essential that the authors provide precise details on how these bounding boxes were determined.

8. No (scientific) performance metrics provided for the method to detect holdings. Given the reliance on this detection, the authors should validate this method against ground truth data and provide metrics for its performance. Comparing this approach with the Traffic library’s methods is also recommended.

9. One-hot encoding (OHE) is not recommended for LightGBM, as per the creators of the model. The authors should acknowledge this and either justify their decision or explore better encoding options for categorical data — or at least warn the readers that OHE is not the right approach when using LightGBM (and other GBDT models like CatBoost).

10. The approach to assign a landing runway to each individual flight works because Heathrow has a single runway in use, but this is not generalizable to other airports. The authors must clarify how the model can be adapted to airports with multiple landing runways. Specifically, how would the authors identify the landing runway of each individual flight? One could use the methods provided by the Traffic library (which work very well). The authors could give some guidance to the readers on this matter. 

11. The authors use RECAT-EU and Aircraft Type features in the regression problem but do not clearly explain how they are integrated into the time series model. Table 2 makes this clearer, but the explanation should be presented earlier in the text.

12. In Table 1, the time of day is treated as a continuous variable, but this leads to issues at midnight. It should be treated as a categorical or cyclic variable to account for discontinuities at midnight.

13. The authors specify units (kt, ...) in Table 1, but GBDT models like LightGBM are not sensitive to scaling. This section could be revised to remove unnecessary details.

14. The authors justify using LightGBM based on its performance in another paper. However, CatBoost is known to handle categorical features better, and a comparison between LightGBM and CatBoost would be a valuable addition to this paper.

15. While the authors justify using LSTM based on its success in another work, the performance improvements over simpler models like GRU are not sufficiently discussed. A comparison between LSTM and GRU models would be valuable, as GRUs can sometimes offer a more efficient alternative to LSTMs.

16. The authors use different data splits for regression and time series problems. It would have been more logical to use consistent time frames across both problems to ensure a fair comparison of results in the test set between the regression and time series models.

17. Why PowerTransformer was used in the regression and MaxMin scaling in the time series. This is not clear and must be clarified.

18. Why not use a 9th Model with 4 heads, each one predicting the average delay of each holding stack in a multi-output setting? This could be included to the "future work" section. 

19. The authors use an LSTM with 50 neurons and 4 layers, and dropout of 0.2, but they do not explain why these particular choices were made. These hyperparameters should be optimized, and the authors should explain why such architecture was chosen in the first place.

20. The inclusion of  Table 8 is redundant. The SHAP values could be better visualized in figures. Tables here are unnecessary and distract from the discussion.

21. The explanation of SHAP values is too simplistic. The authors should provide more context for readers who may not be familiar with these techniques, explaining how they contribute to the model interpretation.

22. The readability of Figure 9 is poor. The authors should try using a line plot instead to improve the clarity of their results.

Conclusion and Future Work

23. Rather than building separate models for each look-ahead time, the authors could integrate this feature into a single model. This would improve generalization and make the implementation easier.

24. The authors mention future comparisons between encoding techniques, but they fail to do this in the current paper. This is a relatively easy experiment to conduct.

Response to reviewer 3

1. The gap you identified is the lack of ML methods to predict holding times?

2. Why ML methods might be more adapted to the task than previous methodologies?

3. Is your paper purely exploratory: the main goal is to check if ML methods have the potential to overcome previous work?

4. Why LSTM and LGBM seem to be the obvious choice?
5. Previous work in other field already demonstrated the reliability of those architectures for similar problems?

6. When detecting a holding pattern, you compare the heading for two consecutive observations separated by 30s. If the heading is more than 20 degrees, then the holding count begins. However, the proper holding pattern could have been initiated anywhere between those two observations, which might lead to an error up to 30s between the real holding pattern start, and the detected one. Thus, from one sample to another in your training dataset, the detected start can be either the very beginning of the holding, or up to 30s in the holding. The same consideration holds for the end, which can lead to discrepancies up to one minute overall. Thus, your model performance depends on your ML model as much as on your detection algorithm. It might be worth to explain further the validation of your model (e.g. comparing the calculated times with the one manually labeled), and also mention it in the limitations if necessary.

7. It might be beneficial to remind the size of each constructed training datasets.

8. According to figure 9, the model seems to constantly underestimate the holding time. Do you have an idea about the reason?