Parsing Flight Searches Using GPT-3

Experimenting with GPT-3 this week has been a lot of fun, and startling at times.  In addition to quick experiments, I wanted to see what it would be like to try building something "real".

Let's say we're working for an airline, and they'd like their website's search box to be able to handle searches like "flight from Toronto to Orlando on September 9 for under $300", giving the appropriate search results.

Our goal is to convert that to the following Wolfram Language representation:

<|
    "Input" -> "flight from Toronto to Orlando on September 9 for under $300",
    "Expression" -> Entity[
        "Flight",
        <|
            "From" -> Entity["City", {"Toronto", "Ontario", "Canada"}],
            "To" -> Entity["City", {"Orlando", "Florida", "UnitedStates"}],
            "When" -> DateObject[{2020, 9, 9}, "Day", "Gregorian"],
            "Price" -> LessThan[Quantity[300, "USDollars"]]
        |>
    ]

|>

The part we'll use GPT-3 for is to convert the search string to the following:

 

Entity[
    "Flights",
    {
        "Price" -> "Under $300",
        "From" -> "Toronto",
        "To" -> "Orlando",
        "When" -> "September 9"
    }

]

Once we have that, we can use the Interpreter function in Wolfram Language to convert things like "Toronto" to a fully resolved reference like Entity["City", {"Toronto", "Ontario", "Canada"}].

To "train" GPT-3 to do the mapping, we'll include a long preamble with all of our API requests, consisting of examples. For example, to parse the above search string, we'll send the following to GPT-3, where the red text indicates the search string we'd like parsed, and the highlighted text indicates what GPT-3 returns to us:

Phrase: flights from Toronto to Orlando

Object: flights

Descriptors: {from Toronto, to Orlando}

Attributes: {origin, destination}

Representation: Entity["Flights", {"From" -> "Toronto", "To" -> "Orlando"}]

"""

Phrase: first class flights from Vancouver to Tokyo on July 18

Object: flights

Descriptors: {first class, from Vancouver, to Tokyo, on July 18}

Attributes: {class, origin, destination, date}

Representation: Entity["Flights", {"Class" -> "First", "From" -> "Vancouver", "To " -> "Tokyo", "Date " -> "July 18"}]

"""

Phrase: from Shanghai to Dubai on the evening of September 22, window seat

Object: flights

Descriptors: {from Shanghai, to Dubai, on the evening of September 22, window seat}

Attributes: {origin, destination, when, seat column}

Representation: Entity["Flights", {"From" -> "Shanghai", "To" -> "Dubai", "When" -> "Evening of September 22", "SeatColumn" -> "Window"}]

"""

Phrase: isle seat for flight from Buffalo New York to Miami Florida next Thursday

Object: flights

Descriptors: {isle seat, from Buffalo New York, to Miami Florida, next Thursday}

Attributes: {seat column, origin, destination, when}

Representation: Entity["Flights", {"SeatColumn" -> "Isle", "From" -> "Buffalo New York", "To" -> "Miami Florida", "When" -> "Next Thursday "}]

"""

Phrase: cheap flights to orlando in February

Object: flights

Descriptors: {cheap, to orlando, in February}

Attributes: {price, destination, when}

Representation: Entity["Flights", {"Price" -> "Cheap", "To" -> "Orlando", "When" -> "in February"}]

"""

Phrase: book a flight to rome this Saturday, departing between 9 AM and 11 AM

Object: flights

Descriptors: {to rome, this Saturday, departing between 9 AM and 11 AM}

Attributes: {origin, destination, when, when}

Representation: Entity["Flights", {"Destination" -> "Rome", "When" -> {"this Saturday", "departing between 9 AM and 11 AM"}}]

"""

Phrase: red eye back to Boston tomorrow night

Object: flights

Descriptors: {red eye, back to Boston, tomorrow night}

Attributes: {when, destination, when}

Representation: Entity["Flights", {"Destination" -> "Boston", "When" -> {"red eye", "tomorrow night"}}]

"""

Phrase: flight from Toronto to Orlando on September 9 for under $300

Object: flights

Descriptors: {from Toronto, to Orlando, on September 9, for under $300}

Attributes: {price, origin, destination, date}

Representation: Entity["Flights", {"Price" -> "Under $300", "From" -> "Toronto", "To" -> "Orlando", "When" -> "September 9"}]

The generated text (highlighted above) can be easily parsed and turned into the final representation, at least for this simple case where "When" is a simple date.  For more complex cases, such as "When" -> "tomorrow night", I could image using a secondary OpenAI call to turn that into something like "When" -> {"Date" -> "Tomorrow", "TimeOfDay" -> "Evening"}.

Going through this exercise highlighted some interesting things for me:

1. Latency is a challenge: If I use the highest quality model, "davinci", the above takes about 30 seconds to process, which is far too slow.  Using the next-best model, "curie", is workable, at about 4 seconds, but I found it did introduce some degradation to the quality.  The faster models solve the latency issue, but their quality was too poor.
   
   
2. I'm using some intermediate representations here, such as the "Descriptors" and "Attributes" keys.  I did that because I felt it would help the model, and that appears to be true.  In my various experiments, when I remove them and try to go directly from "Phrase" to "Representation", noticeably more errors seem to be made.
   
   
3. Maximum token count is a problem for more general parsers.  Here we only worry about interpreting flight searches, but if I try to perform more general mappings from natural language to symbolic representation, I need too many examples, and I hit the 2049 token limit. This relates to the latency challenge above -- the more examples you need, the slower the model becomes.

For this particular application, OpenAI's API is quite promising in that it allows you to do the heavy lifting almost entirely by curating some examples. But there are some practical challenges around latency and token limits that might make this challenging to use in production, or for more challenging/general parsing tasks.