Generating aviation regulations with RNNs
Using a Recurrent Neural Network to generate new text in the style of the US civil aviation regulations
Introduction
Recurrent Neural Networks are great! They allow us to make predictions based on sequential data by using a relatively simple neural network architecture and repeating it at each step of the sequence. At each of these steps, the network outputs a prediction (like any other architecture) as well as a memory state (that's the innovation) which is used by its next iteration in the prediction task. The result of this arrangement is that we end up with a mesh of recurrent units which share their weights and biases, while at the same time communicating with each other through the exchange of the contents of memory cells.
RNNs can be successfully applied to a number of prediction problems, ranging from predicting the weather based on a sequence of data points to predicting future stock prices based on historical data.
Because of their ability to estimate the next point on a sequence of data, RNNs can also be used to generate original data points. If we seed an RNN with a sequence and use it to predict probabilities of the next entry, we can sample those probabilities to get a plausible extension to the data. We can then feed the newly extended sequence to the RNN again and get yet another new point. By repeating this process, a carefully trained RNN can be used to generate new content based on patterns learned from a training set, be it the entirety of Shakespeare's plays, the full collection of 4-part Bach chorales, a large collection of Wikipedia articles, or any other large corpus of sequential data we train it on.
In this article, I'll showcase some of the results of applying a Recurrent Neural Network to the full set of US civil aviation regulations. By training a model to read a sequence of characters and predict the next one, we end up with a network that is able to generate original text in the style of these regulations, one character at a time. I chose the FAA-issued regulations because they amount to a large corpus of text (over 6 million characters), they are readily available online in an easy to parse format, and because I spent a good portion of the last decade ensuring compliance of aircraft, operations and flight crew with them.
Model description
This article will not go into details on the hows and whys of RNNs. If you are interested in an explanation and on tons of amusing examples, I can't recommend Andrej Karpathy's The Unreasonable Effectiveness of Recurrent Neural Networks blog post enough!
Nevertheless, here is a high-level description of the model features:
FIGURE 1. Output of the model.summary() method
If you want to check the code, a jupyter notebook with the python implementation can be found here.
Now, let's finally see some examples!
Results
After training the model (took me around 20 hours in a Google Colab session with GPU acceleration), it reaches an accuracy of around 82% in the test set - meaning that it is able to correctly predict the following character 82% of the time in a selection of regulatory text it has not seen during training. For the text it did see during training, accuracy reaches a little over 86%.
I used the model to generate a sequence of 1 million new characters and saved it to a text file. This corresponds roughly to 1/6 of the total size of the FAA regulations. If you want to have a look and try to find something interesting, you can see the full file here.
Here are some highlights:
Proper vocabulary, but no sense of semantics:
The network is great at capturing the typical terminology found in the regulations, and can even form long sentences. Even though the generated text reads very familiar to someone accustomed to aviation regulations, it fails to generate statements that really make sense.
§ 121.447 Personnel conducting operations with a category and class rating.(a) General. A person who applies for a model aircraft must receive and log ground and flight training in government-questioning in the maneuvers and procedures may be included in the Airworthiness Limitation section. The following information must be used:
(ii) Instructor as an instructor in which the person proficient to perfect flight experience required under paragraph (a)(1)(ii) of this section for the practical test.
For each airplane with four or more engines, the same height above the heliport may be less than the speed from 3 percent of 3 consecutive hours.
Long sentences and paragraphs with proper structure, but no discernible subject:
The model reproduces the typical length of sentences, paragraphs and sections. It even captures the way the text is hierarchically structured. But it does not retain a coherent topic, and often outputs unrelated subjects in the same sentence:
Here, it mixes static pressure, control surfaces, emergency exits, fire and more:
§ 23.2525 Static pressure system tests.If an engine fails. Each oil balloon that is distributed to the airplane movement on the surfaces and other related procedures must be indicated by determining the payload, compliance must be shown by analysis or tests to prevent each emergency exit must be installed so that -
(1) The individual component selected for dry ratings have its less or high system functioning.
(3) Critical cyclic parts throughout the engine to project elements for any fire likely to occur in the open position.
(d) Fire. (c) separate fuel tank sump due to unautomatic or power-operated systems safely and fumes can erect, to an area where deformation is expected to combat first intended for use or automatic or power-operated systems.
In this excerpt, it touches on stowage, radio equipment, missed approaches, engines, maintenance and noise:
§ 33.85 Stowage provisions. Section 121.310(e), Section 121.337(b)(1)(viii), The radio equipment operating at not more than two unmissed approach is granted such that the auxiliary sign failure occurs while operating in the closed datum point.
(7) An engine which receives protection for such noneach quantity of air carrier since the last day of the immediate satellite operating conditions expected, it has enough fuel, considering entry points, but not limited to, restrictions, any instructions necessary to accomplish the identification of the person authorized to perform more than one aircraft or set of aircraft. The Flight Standards office gives the following program:
(1) A current copy of the documentation shown in § 121.405.
(d) Compliance with this section, This subpart applies to each certificate holder that has been approved by the Administrator.
(k) Maintenance and noise limits. This section applies to foreign air carriers or foreign persons holding a turbine-powered/aircraft registered in the United States in command of an aircraft shall, if that person:
The actual § 23.2525 addresses power generation, storage and distribution for normal category airplanes, while the actual § 33.85 addresses powerplant calibration tests.
The model also also fails to pick up on the ordering patterns: paragraphs should follow the order (a), (b), (c) etc, items should follow (i), (ii), (iii), (iv) etc, but the proper numbering is all over the place in the generated text.
Misspellings
Overall, the model is great at spelling words correctly, and can even pick up on other patterns like the use of capital letters and the placement of spaces, commas, and periods. However, misspelled or non-existent words are also common, largely due to the sampling process - in order to ensure that the newly generated text is not repetitive, the next character is randomly chosen according to its probability (instead of just picking the most likely one).
a relationship between these places above $3,500 world holding a nartoty program aircraft that requires a Navy I
§ 91.855 Responsibility: Authority to exemption related to Fuel volus or ferry flight time.
§ 25.509 Special fattiue control.(a) Each intended operation with full airplane with a maximum takeoff weight
Regulatory meta data
Once in a while, the model outputs what seems to be random gibberish, but in fact corresponds to regulatory references to docket numbers, regulatory amendments, and dates of publication. These are all properly located at the end of a section.
(1) The investigation of each of the flight is conducted under an approved equivalent method, resulting from fires within the normal operating range of engine operating limitations effective at least 3 hours.
[Doc. No. 2000-8114, 66 FR 25036, Apr. 28, 2001]
§ 27.933 Cowling factors.(a) An instrument number of systems must ensure that the relationship exit arrives and engine under loads that would occur when the exit is opened.
(b) At speeds which can resist that:
(1) There is a s sumary over the kind of warning; and
(2) For flures from one source, or the nose gear, must be designed to withstand -
(1) The minimum value of operation of the tank; and
(3) The structure must be designed for the loads in the level landing attitude with only the rotational angle.
(2) The limit torque limit must be determined at a speed which would exist (at sea level and in the configuration used in design conditions, such as stall, may not exceed the corresponding static ground load conditions by the manufacturer; and
(2) Vertically, and the control system functioning indicator for each engine.
(b) For each required floor level emergency exit must be designed constant at a speed, complete power failure, to prevent ice classifications or failures.
[Doc. No. 5066, 29 FR 18291, Dec. 24, 1964, as amended by Amdt. 25-38, 41 FR 55466, Dec. 20, 1976]
A few longer examples
The previous excerpts were all short pieces of text chosen to illustrate particular features. However, when looking at the entire text, it is clearly divided into sections and paragraphs about the same size as the actual regulations. Here are some examples:
§ 15.50 How money of all rulemakings.(a) Agricultural aircraft operator. No later than 30 days after completion of a discovery request. (d) Whenever the respondent fails to provide a coordinate the letter of by only if -
(a) It fails to comply with § 161.205; or
(2) Consider of the proceedings or acquired the project if it has an extension to the prohibited response plan that provides for the information. When appropriate, and any other statutory evidence, or in the record when it is not in any negligible issue or may not provide FAA approval of the proposed project in accordance with § 161.305 (b).
(2) The DRO or Special Master, why the final notice dided for a subppoena will be disclosed. If the FAA decisionmaker grants an an invide discovery request and shall serve a copy on the complaint to the party. However, for this contract dispute examination, report to the hearing officer and
(4) A copy of the petitioner to renew the law effectiveness of the motion. If two indemnification is required.
[Doc. No. 18834, 54 FR 34318, Aug. 18, 1989, as amended by Amdt. 91-296, 72 FR 31679, June 7, 2007]
§ 47.80 Prohibition on behalf of an accident. A person who requests an IFR procedure or nontransport category airplane may take off that airplane at a weight greater than that listed in appendix F to this part, instead of the 30-second OEI power rating must be operated in accordance with § 25.21(g), select, or altitude. The maximum operating weight of the rotorcraft, with the rotorcraft compartment, must be with the airplane between the gas temperature at which the rotorcraft operating in flight condition, must be so marked so that it cannot achieve the compartment where it is shown that, unless appropriate weather reports or forects, temperatures, and landing gear, as a multiengine rating which is reported to be students with one engine inoperative, or at the maximum propeller time since the lowest weight allowing scheduled service life rafts (except cylinder barrels).
(iii) Smoking. Each powerplant and auxiliary power unit must be able to support systems used.
(C) The members who have qualified and complete each engine, in each tank, the flight deck for starting and reaction between its operation are free from flutter, controls, and fuel and oil is any combination of the test; and
(2) Be likely to cause injury to a hazardous condition.
These are only a few bits and pieces chosen from a large corpus of text generated by the model. If you are interested in trying to find amusing examples by yourself, you can check out this 1-million characters long text file.
Conclusion
This has been a fun project, and I hope you found the results amusing.
The model generated some amusing new text very similar in terminology, style and structure to the actual corpus of US civil aviation regulations published by the FAA. But it falls short in producing something that makes sense and has any actual meaning.
These severe limitations are justified by the fact that it is very difficult to extract meaningful content from a model trained on individual characters. Meaning depends on larger and more complex structures like words, sentences and paragraphs, and being able to learn these by looking at one character at a time requires a much larger corpus of text and substantial computational resources.
It is also worth noting that the model had no prior knowledge or exposure whatsoever to any text on any language. Every bit of English it learned was based on this regulatory text.
An alternative to this implementation is the use of pre-trained word embeddings to represent individual words as a vector of features or "characteristics". Word embeddings are much better in capturing the semantics of the text. They also carry over a lot of language knowledge acquired during their own training period, which allows the model to focus on learning patterns that are much more specific to the task at hand.
However, I suspect that the pre-trained word embeddings publicly available wouldn't be quite well suited to text of such specialized nature as aviation regulations.
If you're interested in understanding RNNs in a little more detail or seeing a few other examples of amusing applications, I highly recommend this blog post. If you want to learn how to build this sort of model, this book is the way to go. In case you want to check out the dataset and have a go at it yourself, you can access it on kaggle at this link. You can also check out my work generating new baroque music in the style of Bach's 4-part chorales in this post.