DOI for the original paper: https://doi.org/10.59490/joas.2023.7035
General Feedback
This paper proposes a methodology to explore future scenarios for air transport by using a formal modelling approach. The topic and the approach are original making this work a contribution the the community. Furthermore, the code to achieve the study is made open source. However, the paper contains some issues more or less significant and have to be addressed before publication. The introduction has to be reorganized to present the existing literature and provide the position of the paper regarding the latter. The real contribution of the paper begins in section 5. Before that, the sections are broad state of the art by topic. I would suggest shifting all to one or two sections and synthesizing the content to give more weight to the methodology proposed (sections 5 to 8).
The conceptualization proposed is interesting. However, the text is supported by too few figures to really appreciate and understand fully the methodology developed. The first figure appears in section 8. There, is much room for graphical representations and it would help a lot the reader to understand the concepts and the methodology. This is a mandatory point for publication. For example, section 6 offers a thorough description of the model with useful examples and would benefit strongly from illustration.
Finally, too few examples of usages and future usages are proposed. For example, could we combine the proposed (qualitative) methodology with existing quantitative approaches that also explore prospective scenarios for air transport?
Detailed comments
Abstract
* Computational resources? How much are we talking about?
1. Introduction
* It is quite compelling to have only one reference in the introduction. In addition, that ref [1] is maybe not the most appropriate to cite “the rise of global warming concerns”.
* Adding citations to support “Numerous methods have been proposed to generate scenarios” (line 28) would be interesting for the reader.
2. Scenarios for aviation future planning
* Also regarding recent academic literature on prospective scenarios four articles would have their place in this section:
- Planès, T., Delbecq, S., Pommier-Budinger, V., & Bénard, E. (2021). Simulation and evaluation of sustainable climate trajectories for aviation. Journal of Environmental Management, 295, 113079
- Grewe, V., Gangoli Rao, A., Grönstedt, T., Xisto, C., Linke, F., Melkert, J., ... & Dahlmann, K. (2021). Evaluating the climate impact of aviation emission scenarios towards the Paris Agreement including COVID-19 effects. Nature Communications, 12(1), 1-10.
- Klöwer, M., Allen, M. R., Lee, D. S., Proud, S. R., Gallagher, L., & Skowron, A. (2021). Quantifying aviation’s contribution to global warming. Environmental Research Letters, 16(10), 104027.
- Dray, L., Schäfer, A. W., Grobler, C., Falter, C., Allroggen, F., Stettler, M. E., & Barrett, S. R. (2022). Cost and emissions pathways towards net-zero climate impacts in aviation. Nature Climate Change, 12(10), 956-962.
* Line 71 – Cite which CAEP report * Line 77 – Cite Airbus GMF as done for Boeing and COMAC
3. Future planning: a focus on scenario generation
* Table 2 is an interesting summary of methodologies. However, the reader would be interested in more details about the tools. What is “Global quantitative forecasting models”? How does it differ from “Specific quantitative forecasting models?”. Examples could be a way forward.
4. Objectives of the work and positioning
* Line 281: “It aims to systematize the reflection preceding the decision” à not sure this is the right word
* Line 297: why the “use of probabilities” is a methodological bias? An explanation for the two previous items should be provided.
5. Initial knowledge
* I could recommend providing a definition for each variable despite their names. For example, the name “Limitation policies” is not sufficient to understand what it refers to.
* An interaction/dependency graph of all variables would be interesting
6. Model and application to a use case
* Line 362: What has been updated? A bit of detail could be useful for the reader.
7. Scenario generation
* Where is the tree depth defined? An illustration of such a tree would be useful for the reader. Why is the tree depth limited (c.f. line 534)?
8. Analysis
* Line 559: To support the statement “This means in particular that increasing the aircraft flight supply is responsible for …”, one or two examples would be welcome.
9. Discussion
* Initial knowledge: the approach proposed aims to provide a tool to explore a large number of scenarios. Why not explore a large set of initial knowledge? Through a Design of Experiment for example. This would resolve the bias mentioned.
* Line 732: This sentence is quite compelling. I think that including uncertainties would provide more insights than “bias” to the studies. Or else why is choosing only 3 levels (Low, Medium, High) and not 5 or 7 for FuelSupply not already a bias?
10. Conclusion and future works * Line 859: How much time and computational resources? A quantification would be interesting.
* Future work: no proposition of future usage is given. Could we for example imagine using the proposed approach (qualitative) with quantitative scenario exploration tools? References
* The bibliography contains 55 references.
* As mentioned before, more academic literature on aviation prospective scenarios could be given.
* [1] à A more recent version is available https://inis.iaea.org/search/search.aspx?orig_q=RN:53125082
11. Code
* The code is made open source through a GitHub repository
* Information for installation, for example, versions of dependencies would be useful
* The code could be better organised and the main script integrated into a Jupyter Notebook for executing the tool
The paper presents a formal agent-based modelling formalism for the development and analysis of future design in air transportation that involves multiple stakeholders. This formal modelling formalism is also applied to a relevant use case that involves four types of agents (Airline, Customer, Government, Fuel supplier) and one external agent (a troublemaker). This forms a significant contribution.
However, there are important issues for improvement:
1. Section 1 does not make clear what the objective of this research is. With the current organization, this becomes clear in section 4.
2. Section 2 provides a valuable overview of future scenarios on air transportation that have been produced by various organizations.
3. Section 3 provides an overview of scenario generation methods. The first paragraph suggests that there are four different approaches. Unfortunately, the subsequent material in Section 3 lacks a consistent use of the names of these approaches. As a result of this, a reader will be lost.
4. Section 4 explains the objective of the paper. This should be done in the introduction.
5. Section 5 specifies a selected use case example (one page). It would be more logical to add this single page as a subsection of section 3.
6. Section 6 specifies the agent-based modelling formalism adopted for the generation of future scenarios for multiple agent types. Only at the beginning of section 6, it is mentioned that an earlier version of this formalism is in [47]. This relation should be explained in the Introduction. Moreover, it should be explained what the improvement is relative to [47].
7. The title of Section 6 is “Model and application to a use case”. The use case is here used as an illustrative example. Hence it would be better to change the title to “Formal model for the generation of future scenarios”.
8. Section 7 applies the formal method to the selected use case example. The scenario generation is found to stop at the moment a critical conflict is identified between agent objectives. Isn’t this a type of outcome that will often happen for a new use case example that involves different agents? Once such a critical conflict is identified, then the agents should be informed about such findings.
9. Section 8 provides an analysis of the generated scenarios (plural), though without being clear which scenarios are addressed.
10. Section 9 promises a discussion of the results. However, it is not clear which results are aimed for: is it the formal method or the results obtained in applying the formal method to the selected use case?
11. Section 9 ends with a paragraph on Validation. This erroneously suggests that validation has been conducted in the preceding sections. This paragraph on Validation should be moved to future works in section 10
We would like to thank you for dedicating your time and expertise to the evaluation of our paper. We appreciate your suggestions and aim in this response letter to address each of your comments and explain the revisions we have made based on your recommendations.
Abstract
1. “Computational resources? How much are we talking about?” We have removed the sentence because many other topics are discussed in the section “Discussion” such as modelling and operational choices and the application to the air transport system. The computational resources used to compute our use case are now detailed in the section “Scenario Generation”.
Introduction
1. “Section 1 does not make clear what the objective of this research is. Which current organization, this becomes clear in section 4” The objective of the paper is now specified in the introduction (l.37). Therefore, the section “Objectives of the work and positioning” has been removed.
2. “The introduction has to be reorganized to present existing and provide the position of the paper regarding the latter”
It is quite compelling to have only one reference in the introduction. In addition, that ref[1] is maybe not the most appropriate to cite “the rise of global warming concerns”.
“Adding citations to support “Numerous methods have been proposed to generate scenarios” (l28) would be interesting for the reader.
The previous reference [1] has been removed. Two new references have been added: reference [1] is now the ICAO 2022 environmental report to illustrate COVID-19 concerns and reference [2] is the last IPCC report summarizing previous reports on climate change.
Three references have been added to support “Numerous methods have been proposed to generate scenarios”. They consist of different complete reviews of the existing scenario methods using various criteria.
This section concludes with a summary of the various issues and expectations present in the foresight field. The presentation of our contribution follows, with the objectives aiming at addressing some of the current challenges, applied to the aviation industry.
Scenarios for aviation future planning
1. “Regarding recent academic literature on prospective scenarios some articles would have their place in this section”
Thank you for the suggestions, they are now included in this section, in the review of forecasting, and foresight scenarios for the air transport system.
2. Cite the CAEP report and Airbus GMF
These reports are now cited in this section.
Future planning: a focus on scenario generation
1. “The first paragraph suggests that there are four different approaches. Unfortunately, the subsequent material in Section 3 lacks a consistent use in the names of these approaches.”
We have enumerated the four approaches to make them clearer. We have specified in the next paragraph that approaches 3 and 4 are not developed further in the paper. We have also changed the order of the words for approaches 1 and 2 to highlight the terms “foresight” and “forecasting” that are used as the following subsection titles.
2. “The reader would be interested in more details about the tools evocate in Table 2. What is “Global quantitative forecasting models”? How does it differ from “Specific quantitative forecasting models?”
Examples of these tools are provided in the subsection “forecasting” along with the two terms used in the question. Global forecasting models are usually built as a combination of many different specific forecasting models (i.e., demographic, economic models, etc.). They are used to build global forecasting scenarios on a wide topic.
Objectives of the work and positioning
1. “Section 4 explains the objective of the paper. This should be done in the introduction.”
“The real contribution of the paper begins in section 5. […] I would suggest to shift all (sections 2,3,4) to one or two sections and synthesize the content to give more weight to the methodology proposed”
2. Line 281: “wrong word”
This section has been removed. The objectives have been shifted to the introduction. The subsection “positioning” has been shifted to the section “Model and use case”.
3. Line 297: “An explanation for “use of probabilities” as a methodological bias, should be provided”
This line has been removed but we have added this topic to the discussion (l.753)
Initial knowledge
1. “It would be more logical to add this single page (section 5) as a subsection of section 3”
It has been shifted as a subsection at the beginning of the section “Model and application to a use case”.
2. Provide a definition for each variable despite their names
A definition has been added for each variable.
3. “An interaction/dependency graph of all variables would be interesting”
The variables are independent from each other (no causal link) so there is no dependency graph of all variables. See assumption: “there is no dynamics specific to the system: a variable value only changes under the action of an agent; therefore, the variables are independent of each other” (l.517).
Model and application to a use case
1. “It is mentioned that an earlier version of this formalism is in [47]. This relation should be explained in the Introduction”
“It should be explained what the improvement is relative to [47]”
The previous paper has been mentioned in the introduction along with global explanations concerning the improvements provided in our paper.
2. “Section 6 offers a thorough description of the model with useful examples and would benefit strongly from illustration”
It seems difficult for us to provide illustrations for formal equations which would really improve the reading. We have however reformulated some examples to improve the clarity and added a table on the notion of favourable and unfavourable decisions. To make things clearer we have also provided illustrations in the next section “scenario generation”.
3. “It would be better to change the title to “Formal model for the generation of the future scenarios”
Thank you for the suggestion, the title has been changed.
Scenario generation
1. “Where is the tree depth defined? An illustration would be useful. Why is the tree depth limited”
The tree depth definition is now clearer (l. 548).
2. “Section 7 applies the formal method to the selected use case example. The scenario generation is found to stop at the moment a critical conflict is identified. Isn’t this a type of outcome that will often happen for a new use case example that involves different agents?” It is not the whole scenarios generation that stops when there is a conflict but only one scenario generation; the whole generation continues by starting a new scenario on the tree. The difference between the generation of one scenario and the generation of the entire set of scenarios has been clarified in the whole section.
3. “There is much room for graphical representations and it would help a lot the reader to understand the concepts and methodology”
We have provided in this section a figure to illustrate the notions of final and potential states and a clearer illustration to improve the understanding of the scenario structure.
Analysis
1. L. 559: one or two examples would be welcome
The example has been rephrased with illustrations and more text to make it clearer.
2. “Section 8 provides an analysis of the generated scenarios, though without being clear which scenarios are addressed”
We have explained at the beginning of the section (l.584) that the scenarios analysed are the ones generated with the initial data on the air transport system and with a tree depth limitation of three.
Discussion
1. “Section 9 promises a discussion of results. However, it is not clear which results are aimed for”
This section doesn’t aim to discuss the results (i.e., the generated scenarios about the air transport system themselves) but it aims to discuss the approach and its issues whether “caused by biases in the formal model, in the initial knowledge used, the questions raised in the analysis, and the methods offered to solve them.” (l.709)
2. “Why not explore a large set of initial knowledge? Through a Design of Experiment for example. This would resolve the bias mentioned”
We are not familiar with the concept of the Design of Experiment. The formalism proposed here could however be used with initial data of any size. We chose a small one for convenience, to illustrate our model. A huge dataset may nevertheless complicate the use of the tools because of the complexity it would induce. Finally, there is no link between having a bigger set of initial data and reducing biases, it mostly depends on how these data are chosen.
3. “Line 732: I think that including uncertainties would provide more insights than “bias” to the studies. Or else why is choosing only 3 levels and not 5 or 7 not already biased?”
In the foresight field, our model belongs to, the uncertainties are almost never included. However, one could imagine including uncertainties in our model. This would require a well-founded and scientifically validated uncertainty modelling. It may also add some complexity to the model. Finally, we are not looking for the most probable scenarios, as we are not doing forecasting but foresight, so uncertainty may be less useful in this field.
4. “Section 9 ends with a paragraph on validation. This erroneously suggests that validation has been conducted in preceding sections. This paragraph on Validation should better be moved to future works in section 10”
We have moved this paragraph containing the questions raised by a potential validation in the future works section.
Conclusion
1. “Line 859: How much time and computational resources?”
This question has been answered in the section “scenarios generation” for our case study, along with the specification of the computer used to generate our scenarios on the air transport system.
2. “no proposition of future usages is given. Could we for example imagine using the proposed approach (qualitative) with quantitative scenario exploration tools?”
We have added a paragraph with the possible usages in the future works section. In fact, it is completely possible to imagine using quantitative data with our approach (l.928). They will however have to be discretized and no numerical treatment will be done with them.
Global Question
1. “The real contribution of the paper begins in section 5. […] I would suggest to shift all (sections 2,3,4) to one or two sections and synthesize the content to give more weight to the methodology proposed”
“The real contribution of the paper begins in section 5. Before that, the sections are broad state of the art by topic. I would suggest shifting all to one or two sections (S. Delbecq)
For section 2, to our knowledge, such a review of forecasting and foresight scenarios about the future of air transport in English hasn’t been done yet, it is a part of our contribution so we have decided to let this section on its own.
Section 3 has been shortened a little bit but it seems important for us to keep a review of the existing literature in the foresight field because it may not be well-known among this journal’s readers. Section 4 has been removed and its content is now shared between the introduction and section 5.
Code
1. “Information for installation would be useful. The code could be better organised and the main script integrated into a Jupyter Notebook.”
New features are being added to the code, the GitHub will be updated soon with the suggestions that were given in the review as well as new functions.
The authors have brought significant modifications to the paper with a better structure and a more precise outline of the contributions.
Most of my remarks have been answered and the modifications did not raise additional ones. I still have minor suggestions to make to the authors:
- I am not convinced that an illustration is not possible for section 4. For example, an illustration of the elements of the system (even with no interactions) considered in "4.1 Initial knowledge" would add clarity to the paper.
- Installation instructions have not yet been integrated into the GitHub repository. Also, the main script could be integrated and documented in a Jupyter Notebook.
However, considering the major modifications brought by the authors and the absence of additional remarks on my side, I think the paper can be accepted as is.
The paper presents a formal agent-based modelling formalism for the development and analysis of future design in air transportation that involves multiple stakeholders. This formal modelling formalism is also applied to a relevant use case that involves four types of agents (Airline, Customer, Government, Fuel supplier) and one external agent (a troublemaker). This forms a significant contribution. Although the paper has been improved, to realize a high-quality journal paper, my recommendation is to further improve the paper as follows:
1. Organization of Section 4 is confusing, which can be resolved as follows:
1a. Title of Section 4 refers to “… of the future scenarios”; this should be improved to “….of future scenarios”. 1b. Moving subsection 4.8 text to other sections in the paper (partly to the Introduction, partly to Section 3, and partly to the Discussion section).
1c. Start Section 4 with a few paragraphs that explain the organization. This text should include: i) 1st paragraph of subsection 4.1; ii) Subsection 4.2 (a single paragraph); iii) Subsection 4.3 (a single paragraph); and iv) Explanation which definitions are from [10]), and which are new.
1d. Change the title of the “Initial knowledge” subsection into “Initial knowledge of selected Use Case”
1e. Making the scenario definition (current subsection 4.7) part of the System definition material (current subsection 4.4).
1f. Current subsections 4.4, 4.5 and 4.6 illustrate the definitions on examples taken from the Use Case. Hence it is better to replace the text “Example:” with “Use Case example:” (11 x).
2. Subsection 5.1 starts now with a specification of Algorithm 1. It would be better to start with a text paragraph and place the algorithm specification below this text paragraph (as it was in the original version).
3. Section 5 should explain where the complete model specification for the selected Use Case is given. If the examples in Section 4 form the complete specification of the model for the selected Use Case, then this should be made explicit.
4. The last paragraph of Section 5 reports that for the selected Use Case, the scenario generation is found to stop at the moment a critical conflict is identified between agent objectives. Here it should also be explained if and how such finding is used as feedback to agents involved in the selected Use Case, and to the originators of the selected Use Case.
5. In Subsection 7.2, both the title and the text refer to “the formal model”. From this wording, it remains unclear whether this subsection aims to discuss the formal modelling approach (given by the definitions in Section 4) or the formal model developed for the selected Use Case. To avoid potential confusion, the distinction between formal model and formal modelling method should be made explicit.
6. Acknowledgement mentions several researchers. Common practice is to also mention their research organization(s), e.g. in brackets behind the name of a researcher.
We thank you for this second evaluation of our paper. We appreciate your new suggestions and explain the revisions we have made based on your recommendations in this letter. The line numbers refer to the revised paper without the highlighted modifications.
Section 4. Formal model for the generation of future scenarios
1. Organization of Section 4 is confusing, which can be resolved as follows:
a. Title should be improved
We have taken into account the recommendation for the title modification.
b. Moving subsection 4.8 text to other sections in the paper
We have written a paragraph at the end of Section 3 (l.295) summarizing this subsection.
A sentence has also been added in the discussion (l.773)
c. Start Section 4 with a few paragraphs that explain the organization
Thank you for the suggestion, the organization of section 4 is now explained at the beginning of the section. Moreover, every change from [10] has been completed by quoting [10] when the present definitions have not changed and a footnote when a definition has been revised.
d. Change the title of the “Initial knowledge” subsection into “Initial knowledge of selected Use Case”
We have made this modification to the title, it is indeed clearer.
e. Making the scenario definition part of the system definition There was an indentation mistake, therefore the system and the scenario definitions are now subsections of the model section. However, the scenario differs from the system so they are in different subsections.
f. Replace the text “Example” with “Use Case example”
Thank you for this suggestion, we have made this modification in sections 4, 5 and 6 when the examples were based on the use case data, some of them however are made apart from the initial use case knowledge so we have left the term “example” only.
Section 5. Scenario generation
1. It would be better to start subsection 5.1 with the text and then the algorithm. Thank you for the suggestion, we have tried a new organization of this subsection to make the reading easier.2. Section 5 should explain where the complete model specification for the selected Use Case is given. If the examples in Section 4 form the complete specification of the model for the selected Use Case, then this should be made explicit.
There is now a sentence at the beginning of section 5.1 specifying that the algorithm is based on general concepts independent from the initial knowledge (l.516). Section 5.2 also specifies that the results are based on the knowledge from the use case as input of the general algorithm (l.564).
3. The last paragraph of Section 5 reports that for the selected Use Case, the scenario generation is found to stop at the moment a critical conflict is identified between agent objectives. Here it should also be explained if and how such finding is used as feedback to agents involved in the selected Use Case, and to the originators of the selected Use Case. The exploitation of the scenarios is made during the analysis phase. A sentence now makes it explicit at the end of section 5 (l.582)
Section 7. Discussion
1. In Subsection 7.2, both the title and the text refer to “the formal model”. From this wording it remains unclear whether this subsection aims to discuss the formal modelling approach (given by the definitions in Section 4) or the formal model developed for the selected Use Case. To avoid potential confusion, the distinction between formal model and formal modelling method should be made explicit.
The first paragraph of section 7.2 has been reworked to be clearer (l.769). The beginning of section 4 (l.317 and l.331) now also makes clear the fact that we have developed a formal model with general concepts, and we have implemented it in a scenarios generation algorithm. As input for this general algorithm, we can use any kind of knowledge formalized to fit the formal general model (in this paper we have chosen as Use Case the air transportation system).
Other suggestion
1. Acknowledgement mentions several researchers. Common practice is to also mention their research organization(s), e.g. in brackets behind the name of the researcher.
Thank you for the suggestion, the modification has been made.
2. Installation instructions have not yet been integrated into the GitHub repository. Also, the the main script could be integrated and documented in a Jupyter Notebook.
We are sorry for the delay, installation instructions and a Notebook are going to be integrated into the Github repository soon