CN104484319A - Methods and systems for automated text correction - Google Patents

Abstract

The present embodiments demonstrate systems and methods for automated text correction. In certain embodiments, the methods and systems may be implemented through analysis according to a single text correction model. In a particular embodiment, the single text correction model may be generated through analysis of both a corpus of learner text and a corpus of non-learner text.

Description

Method and system for automated text correction

Divisional applications for the following cases:

Application date: 2011-9-23

Application number 2011800459619

Title of Invention: Method and System for Automated Text Correction

technical field

The present invention relates to methods and systems for automated text correction.

Background technique

Text correction is often difficult and time-consuming. In addition, editing texts is often expensive, especially when translations are involved, since editing often requires the use of skilled and trained staff. For example, editorial translations may require labor-intensive work by staff with a high level of proficiency in two or more languages.

Automated translation systems, such as some online translators, can make some aspects of translation less labor-intensive, but they still cannot replace human translators. In particular, automated systems perform a relatively good job of word-to-word translation, but the meaning of sentences is often incomprehensible due to inaccuracies in grammar and punctuation.

Some automated text editing systems do exist, but such systems are often imprecise. Additionally, prior art automated text editing systems may require relatively large amounts of processing resources.

Some automated text editing systems may require training or configuration to edit text precisely. For example, some prior art systems can be trained using an annotated corpus of learner text. Alternatively, some prior art systems can be trained using a corpus of unannotated non-learning text. One of ordinary skill in the art can recognize the difference between learning text and non-learning text.

The output of standard automated speech recognition (ASR) systems usually consists of utterances where important linguistic and structural information such as ground truth, sentence boundaries and punctuation are not available. Linguistic and structural information improves the readability of transcribed speech text and assists further downstream processing such as part-of-speech (POS) tagging, syntax analysis, information extraction and machine translation.

Prior art punctuation prediction techniques use lexical and prosodic cues. However, prosodic features such as pitch and interruption duration are generally not available without the original unprocessed speech waveform. In some scenarios where natural language processing (NLP) for transcribing spoken text becomes a major concern, speech prosody information may not be readily available. In the evaluation activities of the International Symposium on Spoken Language Translation (IWSLT), only human-transcribed or automatically recognized speech texts are provided, while the original unprocessed speech waveforms are not available.

By convention, punctuation is performed during speech recognition. In one example, prosodic features are used together with language model probabilities within a decision tree framework. In another example, interpolation in the broadcast journalism domain includes finite-state and multi-layer perceptron approaches for the task, where prosodic and lexical information is incorporated. In a further example, a maximum entropy based tagging method is implemented for punctuation in spontaneous English conversations, including the use of lexical and prosodic features. In another example, sentence boundary detection is performed by using conditional random fields (CRF). Boundary detection shows an improvement over previous methods based on Hidden Markov Models (HMMs).

Some existing techniques consider sentence boundary detection and punctuation insertion tasks as latent event detection tasks. For example, an HMM can describe the joint distribution over words and inter-word events, where observations are words and word/event pairs are encoded as hidden states. Specifically, in this task, word boundaries and punctuation are encoded as inter-word events. Training phrases involves using smoothing techniques to train an n-gram language model on all observed words and events. The learned n-gram probability scores are then used as HMM state transition scores. During testing, the posterior probability of an event at each word was calculated using dynamic programming using a forward-backward algorithm. The sequence of most likely states thus forms an output giving a punctuated sentence. Such HMM-based approaches have several drawbacks.

First, n-gram language models are only able to capture surrounding contextual information. However, modeling of longer-range dependencies may be required for punctuation. For example, the method is not able to effectively capture the long-range correlation between the initial phrase "would you" and the ending question mark, which are strong indicative interrogatives. Therefore, special techniques can be used in addition to using latent event language models in order to overcome long-range dependencies.

Examples of prior art techniques include rearranging or duplicating punctuation marks in different positions of a sentence so that they appear closer to the indicated word (eg, "how much money" indicates an interrogative sentence). One such technique suggests copying ending punctuation to the beginning of each sentence before training a language model. Empirically, this technique has demonstrated its effectiveness in predicting question marks in English, as most indicated words for English interrogative sentences occur at the beginning of the question. However, such techniques are specifically designed and may not be widely applicable generally or to languages other than English. Further, direct application of the method may fail in the case of multiple sentences per utterance without clearly annotated sentence boundaries within the utterance.

Another drawback associated with this type of approach is that it encodes an assumption of a strong correlation between the punctuation mark to be inserted and the word it surrounds. Therefore, it lacks robustness to handle situations where noise or out-of-vocabulary (OOV) words occur frequently, such as in text automatically recognized by an ASR system.

Grammatical error correction (GEC) has been recognized as an interesting and commercially interesting problem in natural language processing (NLP), especially for learners of English as a foreign or second language (EFL/ESL).

Despite growing interest, research has been hampered by the lack of large annotated corpora of study texts available for research purposes. It turns out that the standard approach for GEC is to train an off-the-shelf classifier to re-predict words in non-learned text. Learning a GEC model directly from an annotated beginner corpus cannot be implemented well, as does the method of combining learner and non-learner text. Further, the assessment of GEC is already a problem. Prior work evaluates either on artificial test instances as a proxy for actual beginner errors, or on proprietary data not available to other researchers. As a result, existing methods cannot be compared on the same test set, making it unclear where the current state of the art actually is.

The industry standard approach for GEC is to build a statistical model that is able to select the most likely correction from a confounded set of possible correction choices. The way to define a confusion set depends on the type of error. Context-sensitive misspelling correction traditionally focuses on confusion sets that have similar spellings (e.g., {dessert, desert"}) or similar pronunciations (e.g., {there, their}). In other words, words in the confusion set are phonetic similarity. Other work in GEC defines confusion sets based on syntactic similarity, for example, all English articles or most frequent English prepositions form confusion sets.

Contents of the invention

This embodiment demonstrates a system and method for automated text correction. In some embodiments, methods and systems may be implemented by analysis according to a single text editing model. In certain embodiments, a single text editing model may be generated by analysis of a corpus of learned text and a corpus of non-learned text.

According to one embodiment, an apparatus includes at least one processor and a memory device coupled to the at least one processor, wherein the at least one processor is configured to recognize words of an input utterance. The at least one processor is also configured to place words in a plurality of first nodes stored in the memory device. The at least one processor is further configured to assign a word-level label to each of the first nodes based in part on neighboring nodes of the linear chain. The at least one processor is also configured to generate an output sentence by combining words from the plurality of first nodes with a portion of the selected punctuation on the word-level labels assigned to each first node.

According to another embodiment, a computer program product includes a computer-readable medium having code for identifying words of an input utterance. The medium also includes code for placing words in a plurality of first nodes stored in the memory device. The medium further includes code for assigning a word-level label to each of the plurality of first nodes based in part on neighboring nodes of the first node. The medium also includes code for generating an output sentence by combining words from the plurality of first nodes with punctuation selected in part on word-level labels assigned to each first node.

According to another embodiment, a method includes recognizing words of an input utterance. The method also includes placing the word in a plurality of first nodes stored in the memory device. The method further includes assigning a word-level label to each first node of the plurality of first nodes based in part on neighboring nodes of the plurality of first nodes. The method also includes generating an output sentence by combining words from the plurality of first nodes with a portion of the selected punctuation on the word-level labels assigned to each first node.

An additional embodiment of a method includes receiving natural language text input, the text input including grammatical errors, wherein a portion of the input text includes a class from a set of classes. The method may also include generating a plurality of selection tasks from the corpus of non-learned text assumed to be free of grammatical errors, wherein for each selection task, the classifier re-predicts the class used in the non-learned text. Further, the method may include generating a plurality of correction tasks from the corpus of learning text, wherein for each correction task the classifier suggests a class to use in the learning text. Additionally, the method may include training the grammar correction model using a set of binary classification problems including a plurality of selection tasks and a plurality of correction tasks. This embodiment may also include using the trained grammar correction model to predict the class of the text input from a set of possible classes.

In a further embodiment, the method includes outputting a suggestion to change the class of the textual input to the predicted class if the predicted class is different from the class in the textual input. In such an embodiment, the learning text is annotated by the teacher with the assumed correct classes. Classes can be articles associated with noun phrases in the input text. The method may also include extracting feature functions for the classifier from noun phrases in the non-learning text and the learning text.

In another embodiment, the classes are prepositions associated with prepositional phrases in the input text. Such methods may include extracting feature functions for classifiers from prepositional phrases of non-learned text and learned text.

In one embodiment, the non-learned text and the learned text have different feature spaces, the feature space of the learned text includes words used by the author. Training the grammar correction model may include minimizing a loss function on the training data. Training a grammar correction model may also include identifying multiple linear classifiers by analyzing non-learned text. The linear classifier further includes weighting factors included in the matrix of weighting factors.

In one embodiment, training the grammar correction model further includes performing a singular value decomposition (SVD) on the matrix of weighting factors. Training the grammar correction model may also include identifying combined weight values representing first weight value elements identified by analyzing the non-learned text and second weight value elements identified by analyzing the learned text by minimizing an empirical hazard function.

An apparatus for automated text correction is also provided. The device may comprise, for example, a processor configured to perform the steps of the methods described above.

Another embodiment of a method is provided. The method may include correcting semantic collocation errors. One embodiment of such a method includes automatically identifying one or more translation candidates in response to a corpus analysis of the parallel language text performed at the processing device. Additionally, the method may include using the processing means to determine features associated with each translation candidate. The method may also include generating a set of one or more weight values from a corpus of study text stored in the data storage device. The method may further comprise calculating, using processing means, a score for each translation candidate in response to the features associated with the one or more translation candidates and the set of one or more weight values.

In a further embodiment, identifying one or more translation candidates may comprise selecting a parallel corpus of texts from a database of parallel texts, each parallel text comprising a text in the first language and a corresponding text in the second language, using processing means to segmenting the text in the first language, using the processing means to tokenize the text in the second language, using the processing means to automatically align words in the first text with words in the second text, using the processing means to extract from the second text Aligned words in the first text and the second text extract phrases and use processing means to calculate probabilities of matching paraphrases associated with one or more phrases in the first text and one or more phrases in the second text.

In a particular embodiment, the feature associated with each translation candidate is the probability of a paraphrase match. The set of one or more weight values may be computed using a minimum error rate training (MERT) operation on a corpus of learning text.

The method may also include generating a collocation-corrected phrase table with features derived from spelling edit distances. In another embodiment, the method may include generating a collocation-corrected phrase table with features derived from a dictionary of homophones. In another embodiment, the method may include generating a collocation-corrected phrase table with features derived from synonyms. Additionally, the method may include generating a collocation-corrected phrase list with features derived from paraphrases imported in the native language.

In such embodiments, the phrase table includes one or more penalty features for use in computing the probability of a paraphrase match.

There is also provided an apparatus comprising at least one processor and a memory device coupled to the at least one processor, wherein the at least one processor is configured to perform the steps of the method of the claims above. There is also provided a tangible computer readable medium comprising computer readable code which, when executed by a computer, causes the computer to perform the operations in the method as described above.

The term "coupled" is defined as connected, although not necessarily directly, and not necessarily mechanically.

The terms "a" and "an" are defined as one or more unless the disclosure expressly requires otherwise.

The term "substantially" and variations thereof are defined as substantially, but not necessarily all, as understood by those skilled in the art, and in one non-limiting example, "substantially" means within 10% of the stated range, preferably within 5%, more preferably within 1%, and most preferably within 0.5%.

The terms "comprise" (and any other forms of inclusion, such as "comprises" and "comprising"), "have", "include" (and any other forms of inclusion, such as "includes" and "including ") and "contain" (and any other form of containing, such as "contains" and "containing") are open-ended linking verbs. Consequently, a method or apparatus that "comprises", "has", "includes" or "contains" one or more steps or elements addresses those steps or elements, but does not Restricted to processing only those steps or units. Likewise, a step of a method or an element of an apparatus that "comprises", "has", "includes" or "contains" one or more features addresses those one or more features, but does not Limit processing to only those one or more features. Further, a device or structure that is configured in a certain way is configured in at least that way, but it may also be configured in ways that are not listed. Other features and associated advantages will become apparent by reference to the following detailed description of certain embodiments in conjunction with the accompanying drawings.

Description of drawings

The following drawings form part of this specification and are included to further demonstrate certain aspects of the invention. The invention may be better understood by reference to one or more of these drawings in combination with the detailed description of specific embodiments provided herein.

FIG. 1 is a block diagram illustrating a system for analyzing utterances according to one embodiment of the present disclosure;

Figure 2 is a block diagram illustrating a data management system configured to store sentences according to one embodiment of the present disclosure;

Figure 3 is a block diagram illustrating a computer system for analyzing utterances according to one embodiment of the present disclosure;

Figure 4 is a block diagram showing a graphical representation for a linear chain CRF;

Figure 5 is an example token for a training sentence for a linear chain conditional random field (CRF);

Figure 6 is a block diagram illustrating a graphical representation of a two-level factorial CRF;

Figure 7 is an example token for a training sentence of a factorial conditional random field (CRF);

Figure 8 is a flowchart illustrating one embodiment of a method for inserting punctuation into a sentence;

Figure 9 is a flowchart illustrating one embodiment of a method for automatic grammatical error correction;

Figure 10A is a graph showing the accuracy of one embodiment of a text correction model for correcting article errors;

Figure 10B is a graph showing the accuracy of one embodiment of a text correction model for correcting preposition errors;

Figure 11A is a graph showing the F1 measure for a method for correcting article errors compared to a common method using the DeFelice feature set;

FIG. 11B is a graph showing the F1 measure of a method for correcting article errors compared to a common method using the Han feature set;

Figure 11C is a graph showing the F1 measure for a method for correcting article errors compared to a common method using the Lee feature set;

Figure 12A is a graph showing the F1 measure for a method for correcting preposition errors compared to a common method using the DeFelice feature set;

Figure 12B is a graph showing the F1 measure for a method for correcting preposition errors compared to a common method using the TetreaultChunk feature set;

Figure 12C is a graph showing the F1 measure for a method for correcting preposition errors, compared to a common method using the TetreaultParse feature set;

Figure 13 is a flowchart illustrating one embodiment of a method for correcting semantic collocation errors.

Detailed ways

The various features and advantages are explained more fully with reference to the non-limiting embodiments illustrated in the drawings and detailed in the following description. Descriptions of well-known starting materials, processing techniques, components and devices are omitted so as not to unnecessarily obscure the present invention in detail. It should be understood, however, that the detailed description and specific examples, while indicating embodiments of the invention, are given by way of illustration only and not limitation. Various substitutions, modifications, additions and/or rearrangements within the spirit and/or scope of the underlying inventive concept will become apparent to those skilled in the art from this disclosure.

Certain elements described in this specification have been labeled as modules in order to more particularly emphasize their implementation independence. A module is "a self-contained hardware or software component that interacts with a larger system," Alan Friedman, "The Computer Glossary" 268 (8th ed., 1998). A module includes machine or machine-executable instructions. For example, a module may be implemented as a hardware circuit comprising custom VLSI circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, or the like.

A module may also include software-defined units or instructions that, when executed by a processing machine or device, transform data stored on the data storage device from a first state to a second state. An identified module of executable code may, for instance, comprise one or more physical or logical blocks of computer instructions, which may be organized as an object, procedure, or function. However, the executables that identify the modules need not be physically together, but may include separate instructions stored in different locations that, when logically connected together, comprise the modules and, when executed by a processor, implement the statement data conversion.

Indeed, a module of executable code may be a single instruction, or many instructions, and may be distributed over several different code segments, among different programs or across several memory devices. Similarly, operational data may be identified and illustrated herein within modules, and may be embodied in any suitable form and organized within any suitable type of data structure. Operational data may be aggregated into a single data set, or may be distributed across different locations, including across different storage devices.

In the following description, numerous specific details are provided, such as examples of programming, software modules, user selections, network transactions, database queries, database structures, hardware modules, hardware circuits, hardware chips, etc., to provide a thorough understanding of the embodiments understand. One skilled in the relevant art will recognize, however, that the invention may be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the invention.

Figure 1 illustrates one embodiment of a system 100 for automated text and speech editing. System 100 may include server 102 , data storage device 106 , network 108 and user interface device 110 . In one particular embodiment, system 100 may include a memory controller 104 , or memory server, configured to manage the transfer of data between data storage device 106 and server 102 or other components in communication with network 108 . In an alternate embodiment, memory controller 104 may be coupled to network 108 .

In one embodiment, user interface device 110 may be referred to broadly and is intended to encompass suitable processor-based devices such as desktop computers, laptop computers, personal digital assistants (PDAs) or tablet computers, access Smartphone or other mobile communication device or manager device to network 108. In further embodiments, the user interface device 110 may be connected to the Internet or other wide or local area network to access web applications or web services hosted by the server 102 and provide a user interface to enable a user to enter or receive information. For example, a user may enter input utterances or text into the system 100 through a microphone (not shown) or keyboard 320 .

Network 108 may facilitate data transfer between server 102 and user interface device 110 . Network 108 may comprise any type of communication network including, but not limited to, direct PC-to-PC connections, local area networks (LANs), wide area networks (WANs), modem-to-modem connections, the Internet, combinations of the above, or now known or later developed Within the field of networking, any other communication network that allows two or more computers to communicate with each other.

In one embodiment, server 102 is configured to store input utterances and/or input text. Additionally, the server may access data stored in the data storage device 106 via a storage area network (SAN), LAN, data bus, or the like.

The data storage device 106 may include hard disks (including hard disks arranged in a redundant array of independent disks (RAID)), tape memory drives including magnetic tape data storage devices, optical storage devices, or the like. In one embodiment, data storage device 106 may store sentences in English or other languages. Data may be arranged in a database and accessed through Structured Query Language (SQL) queries, or other database query languages or operations.

FIG. 2 illustrates one embodiment of a data management system 200 configured to store input utterances and/or input text. In one embodiment, data management system 200 may include server 102 . Server 102 may be coupled to data bus 202 . In one embodiment, the data management system 200 may also include a first data storage device 204 , a second data storage device 206 and/or a third data storage device 208 . In other embodiments, data management system 200 may include additional data storage devices (not shown). In one embodiment, a corpus of learning text such as the NUS Corpus of Learners English (NUCLE) may be stored in the first data storage device 204 . The second data storage device 206 may store eg a corpus of non-learning text. Examples of non-study texts may include parallel corpora, news or journal texts, and other publicly available texts. In some embodiments, non-learned text is selected from sources believed to contain relatively few errors. The third data storage device 208 may contain calculated data, input text and or input utterance data. In other embodiments, the data may be stored together in the consolidated data storage device 210 .

In one embodiment, the server 102 may submit a query to a selected data storage device 204, 206 to retrieve an input sentence. Server 102 may store the consolidated data set in consolidated data storage device 210 . In one such embodiment, the server 102 may refer back to the consolidated data storage device 210 for a set of data elements associated with the specified sentence. Alternatively, the server 101 may query each of the data storage devices 204, 206, 208 independently or in a distributed query to obtain a set of data elements associated with an input sentence. In another alternative embodiment, multiple databases may be stored on a single consolidated data storage device 210 .

Data management system 200 may also include files for entering and processing utterances. In various embodiments, server 102 may communicate with data storage devices 204 , 206 , 208 via data bus 202 . Data bus 202 may include a SAN, LAN, or the like. The communications infrastructure may include Ethernet, Fiber Channel Arbitrated Loop (FC-AL), Small Computer System Interface (SCSI), Serial Advanced Technology Attachment (SATA), Advanced Technology Attachment (ATA), and/or other data storage-related Similar data communication strategy associated with communication. For example, server 102 may communicate indirectly with data storage devices 204 , 206 , 208 , 210 ; server 102 first communicates with memory server or memory controller 104 .

Server 102 may host a software application configured to analyze utterances and/or input text. The software application may further include modules for interfacing with the data storage devices 204, 206, 208, 210, interfacing with the network 108, interfacing with a user through the user interface device 110, and the like. In other embodiments, the server 102 may host an engine, an application plug-in, or an application programming interface (API).

FIG. 3 illustrates a computer system 300 adapted in accordance with certain embodiments of server 102 and/or user interface device 110 . Central processing unit (“CPU”) 302 is coupled to system bus 304 . The CPU 302 may be a general-purpose CPU or microprocessor, a graphics processing unit ("GPU"), a microcontroller, or the like that may be specially programmed to perform the methods as described in the flowcharts below. The present embodiment is not limited to the architecture of the CPU 302, as long as the CPU 302 directly or indirectly supports the modules and operations as described herein. CPU 302 can execute various logic instructions according to this embodiment.

Computer system 300 may also include random access memory (RAM) 308, which may be SRAM, DRAM, SDRAM, or the like. Computer system 300 may use RAM 308 to store various data structures used by software applications with code to analyze utterances. Computer system 300 may also include read only memory (ROM) 306, which may be PROM, EPROM, EEPROM, optical memory, or the like. The ROM may store configuration information for starting the computer system 300 . RAM 308 and ROM 306 hold user and system data.

Computer system 300 may also include input/output (I/O) adapter 310 , communication adapter 314 , user interface adapter 316 , and display adapter 322 . In some embodiments, I/O adapter 310 and/or user interface adapter 316 may enable a user to interact with computer system 300 to enter speech or text. In further embodiments, display adapter 322 may display a graphical user interface associated with a software and web-based application or a mobile application for generating with inserted punctuation, grammar correction, and other related text and speech editing functions.

I/O adapter 310 may connect to computer system 300 one or more memory devices 312 such as one or more of a hard drive, a computer disk (CD) drive, a floppy disk drive, and a tape drive. Communications adapter 314 may be adapted to couple computer system 300 to network 108, which may be one or more of a LAN, WAN, and/or the Internet. User interface adapter 316 couples user input devices such as keyboard 320 and pointing device 318 to computer system 300 . Display adapter 322 may be driven by CPU 302 to control the display on display device 324.

The application of the present disclosure is not limited to the architecture of computer system 300 . Rather, computer system 300 is provided as an example of one type of computing device that may be adapted to execute server 102 and/or user interface device 110 . For example, any suitable processor-based device may be used, including but not limited to personal digital assistants (PDAs), desktop computers, smart phones, computer game consoles, and multi-processor servers. Additionally, the systems and methods of the present disclosure may be implemented on application specific integrated circuits (ASICs), very large scale integration (VLSI) circuits or other circuits. In fact, those skilled in the art may employ any number of suitable structures capable of performing logical operations in accordance with the described embodiments.

The schematic flow diagrams and related descriptions that follow are generally set forth as logical flow diagrams. As such, the depicted order and labeled steps are indicative of one embodiment of the presented method. Other steps and methods are contemplated that are equivalent in function, logic, or effect to one or more steps, or portions thereof, of the illustrated method. Additionally, the format and symbols used are provided to explain the logical steps of the method and are understood not to limit the scope of the method. Although various arrow types and line types may be used in the flowcharts, they are understood not to limit the scope of the corresponding method. In fact, some arrows or other connectors can be used to simply indicate the logical flow of the method. For example, an arrow may indicate a waiting or monitoring period of unspecified duration between listed steps of the depicted method. Additionally, the order in which a particular method occurs may or may not strictly adhere to the order of the corresponding steps shown.

Punctuation prediction

According to one embodiment, punctuation marks can be predicted from a standard text processing perspective, where only phonetic text is available, without relying on additional prosodic features such as pitch and break duration. For example, the task of punctuation prediction can be performed on transcribed dialogue speech text or utterances. Unlike many other corpora such as the broadcast news corpus, a conversational speech corpus may include dialogue in which informal and short sentences occur frequently. In addition, due to the nature of dialogue, it can also include more interrogative sentences than other corpora.

A natural way to relax the assumption of strong correlations encoded by latent event languages is to employ undirected graphical models, where arbitrarily overlapping features can be exploited. Conditional Random Fields (CRFs) have been widely used in various sequence labeling and segmentation tasks. A CRF can be a discriminative model of the conditional distribution of the complete label sequence given an observation. For example, the first-level linear chain CRF adopting the first-level Markov property can be defined by the following equation:

${\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span>}\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; t</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\underset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>$

where x is the observation and y is the sequence of markers. A feature function fk as a function of time step t can be defined over the entire observation x and two adjacent latent labels. Z(x) is a normalization factor to ensure a well formed probability distribution.

Figure 4 is a block diagram showing a graphical representation for a linear chain CRF. A series of first nodes 402a, 402b, 402c, ..., 402n is coupled to a series of second nodes 404a, 404b, 404c, ..., 404n. The second node may be an event associated with a corresponding node of the first node 402, such as a word level label. The punctuation prediction task can be modeled as the process of assigning labels to each word. A set of possible tags may include none (NONE), comma (,), period (.), question mark (?), and exclamation point (!). According to one embodiment, each word may be associated with an event. The event identifies which punctuation (possibly NONE) should be inserted after the word.

The training data for the model may include a set of utterances where punctuation marks are numbered as labels assigned to individual words. The tag NONE means that no punctuation is inserted after the current word. Any other tags identify where to insert the corresponding punctuation marks. The most probable sequence of labels is predicted and a punctuated text can then be constructed from the output of this class. An example of dialog punctuation can be shown in FIG. 5 .

Fig. 5 is an example punctuation of a training sentence for a linear chain conditional random field (CRF). Sentence 502 may be divided into words and a word-level label 504 assigned to each word. Word level labels 504 may indicate punctuation marks that follow words in the output sentence. For example, the word "no" is punctuated with "comma" to indicate that a comma should follow the word "no". Also, some words such as "please" are marked with "no" to indicate that there is no sign mark following the word "please".

According to one embodiment, the characteristics of the conditional random field can be factorized into the binary function that assigns a set of cliques (clique) at the current time step (in this case, the edge) and the characteristic function defined separately on the observation sequence product. The n-gram occurrences around the current word together with the position information are used as binary feature functions for n=1;2;3. Words that occur within 5 words from the current word are considered when building features. Special start and end symbols are used beyond utterance boundaries. For example, for the word shown in Figure 5, example features include the unary feature "do" at relative position 0, "please" at relative position -1, and the binary feature " Do you want", and the ternary feature "Don't please" at relative positions -2 to 0.

The linear chain CRF model in this embodiment may be able to utilize arbitrary overlapping features to model the correlation between words and punctuation marks. Therefore, strong correlation assumptions in latent event language models can be avoided. Further improvements to the model are provided by including analysis of long-range dependencies at the sentence level. For example, in the same utterance shown in Figure 5, the long-range correlation between the closing question mark and the far-distant occurrence of the indicative word "do you want" may not be captured.

Factorial-CRF (F-CRF), an example of a dynamic conditional random field, can be used as a framework for providing the ability to label multiple layers of labels simultaneously for a given sequence. F-CRF learns the joint conditional distribution of labels given observations. A dynamic conditional random field can be defined as the conditional probability of a sequence y of marker vectors given an observation x:

${\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span>}\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; t</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\underset{\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; C</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\underset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; c</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>$

where cliques are indexed at each time step, C is the set of clique indices, and y(c;t) is the set of changes in the unrolled version of the clique with index c at time t.

Figure 6 is a block diagram showing a graphical representation of a two-level factorial CRF. According to one embodiment, the F-CRF may have two layers of nodes as labels, where a clique includes two intra-chain edges (e.g., z2-z3 and y2-y3) and one inter-chain edge ( For example, z3-y3). A series of first nodes 602a, 602b, 602c, ..., 602n is coupled to a series of second nodes 604a, 604b, 604c, ..., 604n. The series of third nodes 606a, 606b, 606c, ..., 606n are coupled to the series of second nodes and the series of first nodes. The nodes of the series of second nodes are coupled to each other to provide long-range dependencies between nodes.

According to one embodiment, the second node is a word level node and the third node is a sentence level node. Each sentence-level node can be coupled with a corresponding word-level node. Both the sentence level node and the word level node may be coupled to the first node. Sentence-level nodes can capture long-range correlations between word-level nodes.

In F-CRF, two sets of labels can be assigned to words in an utterance: word-level labels and sentence-level labels. Word-level tags can include none, commas, periods, question marks, and/or exclamation points. Sentence level tags may include start of statement, inside of statement, start of question, inside of question, start of exclamation, and/or inside of exclamation. Word-level tags can be responsible for inserting punctuation marks (including none) after each word, while sentence-level tags can be used to mark sentence boundaries and identify sentence types (statement, question, or exclamation).

According to one embodiment, the labels from the word layer may be the same as those from the linear chain CRF. Sentence-level tags can be designed for three types of sentences: DEBEG and DEIN indicate the beginning and interior of declarative sentences, respectively, similar to QNBEG and QNIN (interrogative sentences) and EXBEG and EXIN (exclamatory sentences). The same example utterances we looked at in the previous section can be labeled with two layers of labels, as shown in Figure 7.

Figure 7 is an example token of a training sentence for a factorial conditional random field (CRF). Sentence 702 may be divided into words and each word labeled with a word-level label 704 and a sentence-level label 706 . For example, the word "not" can be tagged with a comma word-level tag and a statement-start sentence-level tag.

The analog eigenfactorization and n-ary eigenfunctions used in linear chain CRF can be used in F-CRF. When learning sentence-level labels in conjunction with word-level labels, the F-CRF model is able to leverage useful cues learned from the sentence level about sentence types (e.g., interrogative sentences, annotated with QNBEG, QNIN, QNIN, or declarative sentences, annotated with DEBEG, DEIN, DEIN), which can be used to guide the prediction of punctuation marks at each word, thus improving the performance at the word level.

For example, consider the utterances shown in Figure 7 for joint labeling. When the evidence shows that the utterance consists of two sentences, with the declarative followed by the interrogative, the model tends to label the second part of the utterance with the sequence of sentence labels: QNBEG, QNIN. These sentence-level labels help predict word-level labels at the end of an utterance as QMARKs, given the correlation between the two layers that exists at each time step. According to one embodiment, during learning, the two label layers may be learned jointly. Thus, word-level labels can affect sentence-level labels, and vice versa. The GRMM package can be used to construct both linear chain CRFs (LCRFs) and factorial CRFs (F-CRFs). Tree-based reparameterization (TRP) scheduling for belief propagation is used for approximate inference.

The techniques described above may allow the use of conditional random fields (CRFs) to perform in-utterance predictions without relying on prosodic cues. Thus, the method described may have post-processing for transcribing the utterances of the dialogue. Additionally, long-range correlations can be established between words in an utterance to improve prediction of punctuation in an utterance.

Experiments on a portion of the corpus of the IWSLT09 review campaign in which both Chinese and English dialogue speech texts were used were performed in different ways. Consider two multilingual datasets, the BTEC (Basic Tourist Expression Corpus) dataset and the CT (Challenge Task) dataset. The former includes travel-related sentences, while the latter includes human intervention in cross-language dialogues within the travel domain. The official IWSLT09 BTEC training set includes 19972 Chinese-English utterance pairs, and the CT training set includes 10061 such pairs. Each of the two datasets can be randomly divided into two parts, where 90% of the utterances are used to train the punctuation model and the remaining 10% are used to evaluate the predictive performance. For all experiments, the default segmentation of Chinese can be used as provided, while the English text can be preprocessed using the Penn tree graph database tokenizer. Table 1 provides the statistics of the two datasets after processing.

List the proportions of sentence types in the two datasets. Most sentences are declarative sentences. However, interrogative sentences appear more frequently in the BTEC dataset than in the CT dataset. For all datasets, exclamatory sentences contribute less than 1% and are not listed. In addition, utterances from CT datasets are longer (with more words per utterance), and thus multiple CT utterances often include multiple sentences.

Table 1: Statistics of BTEC and CT datasets

Additional experiments can be divided into two categories: copy the ending punctuation to the beginning of the sentence before training, or not copy the ending punctuation to the beginning of the sentence before training. This setting can be used to evaluate the impact of punctuation marks and proximity between indicated words on prediction tasks. Under each class, test two possible methods. One-pass methods perform prediction in a single step, where all punctuation marks are predicted sequentially from left to right. In the cascaded approach, the training sentences are formatted by first replacing special sentence boundary symbols with all sentence-ending punctuation marks. A model for sentence boundary prediction can be learned based on such training data. According to one embodiment, this step may be followed by prediction of punctuation marks.

Both trigram and 5-gram language models were tried for all combinations of the above settings. This provides a total of eight possible combinations based on the latent event language model. When training all language models, a modified Kneser-Ney smoothing for n-tuples can be used. To evaluate the performance of the punctuation prediction task, the calculations for precision (prec), recall (rec) and F1-measure (F1) are defined by the following equations:

${\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; prec</span>}<span\; class="notranslate">\; .</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; rec</span>}<span\; class="notranslate">\; .</span>}$

The performance of punctuation prediction on Chinese (CN) and English (EN) text in the correct recognition output of BTEC and CT databases is shown in Table 2 and Table 3, respectively. The performance of a hidden event language model depends heavily on whether a replication method is used and whether the actual language is considered. Specifically, for English, copying ending punctuation to the beginning of sentences before training is useful for improving overall predictive performance. In comparison, applying the same technique to Chinese destroys performance.

One explanation is that English interrogative sentences distinguish interrogative sentences from declarative sentences by beginning with a demonstrative word such as "do you" or "where". Therefore, copying the closing punctuation to the beginning of the sentence so that it is close to these indicator words helps to improve prediction accuracy. However, for interrogative sentences, Chinese exhibits very difficult syntactic structures.

First, in many cases, Chinese tends to use syntactically ambiguous particles at the end of sentences to indicate questions. Particles of this type include "do" and "do". Therefore, preserving the positions of ending punctuation marks before training yields better performance. Another finding is that, unlike English, those words that indicate interrogative sentences in Chinese can appear in almost any position in a Chinese sentence. Examples include where… (where…),… what (what…) or… how many/much…. This poses difficulties for simple latent event language models, which encode only simple correlations on surrounding words through n-gram language modeling.

Table 2: Punctuation prediction performance on Chinese (CN) and English (EN) text in the correct recognition output of the BETC dataset. Reports the percentage scores for Precision (Prec.), Recall (Rec.) and F1 measure (F1).

Table 3: Punctuation prediction performance on Chinese (CN) and English (EN) text in the correct recognition output of the CT database. Reports percentage scores for Precision (Prec.), Recall (Rec.) and F1 measure (F1)

By employing discriminative models that implement non-independent, overlapping features, LCRF models often outperform latent event language models. The F-CRF model further improves performance over the L-CRF model by introducing additional labeling layers that perform sentence segmentation and molecular type prediction. Perform statistical significance tests using bootstrap weight sampling. The improvement of F-CRF over L-CRF on Chinese and English texts in the CT database, and on English texts in the BTEC database was statistically significant (p<0.01). The improvement of F-CRF over L-CRF on Chinese text is smaller, probably because L-CRF already performs well on Chinese. F1 measures on the CT database were lower than those on BTEC, mainly because the CT database included longer utterances and fewer interrogative sentences. Overall, the proposed F-CRF model is robust and consistently works well regardless of the languages and databases it is tested on. This shows that the method is general and relies on minimal language assumptions, and thus can be easily used on other languages and databases.

Models can also be evaluated using text generated by the ASR system. For evaluation, the 1-best ASR output of impromptu speech from the official IWSLT08 BTEC evaluation database, published as part of the IWSLT09 corpus, can be used. The database includes 504 utterances in Chinese and 498 utterances in English. Unlike the correctly recognized text as described in Section 6.1, the ASR output contained substantial recognition errors (recognition accuracy was 86% for Chinese and 80% for English). In the database published by the organizers of IWSLT 2009, the correct punctuation is not marked in the ASR output. To perform experimental evaluations, the correct punctuation marks on the ASR output can be manually annotated. Evaluation results for each model are shown in Table 4. The results show that F-CRF still gives higher performance than L-CRF and latent event language model, and the improvement is statistically significant (p<0.01).

Table 4: Punctuation prediction performance on Chinese (CN) and English (EN) text in the ASR output of the IWSLT08 BTEC evaluation dataset. The report reports the percentage scores for Precision (Prec.), Recall (Rec.) and F1 measure (F1)

In another evaluation of the model, an indirect approach can be employed to automatically evaluate the performance of punctuation prediction on ASR output text and evaluate the resulting translation performance by feeding the evaluated ASR text into a prior art machine translation system. Translation performance is then measured by automated evaluation metrics that correlate well with human judgment. The existing phrase-based statistical machine translation toolkit Moses was used as the translation engine together with the entire IWSLT09BTEC training set used to train the translation system.

The Berkeley calibrator is used to align the training bilingual text with the lexicalized reranking model enabled. This is because lexicalized re-ranking gives better performance than simple distance-based re-ranking. In particular, the default lexicalized reordering model (msd-bidirectional-fe) is used. To tune the parameters of Moses, we use the official IWSLT05 evaluation set, where correct punctuation is present. Evaluation is performed on the ASR output of the IWSLT08 BTEC evaluation dataset, while punctuation marks are interpolated by each punctuation prediction method. The conditioning set and evaluation set consist of 7 reference translations. Following the convention in statistical machine translation, we report the BLEU-4 score, which was shown to correlate well with human judgment, and the nearest reference length as the valid reference length. The Minimum Error Rate Training (MERT) procedure is used to tune the model parameters of the translation system.

Due to the unstable nature of MERT, 10 runs are performed for each translation task with a different random initialization of the parameters in each run, and the BLEU-4 score averaged over the 10 runs is reported. The results are shown in Table 5. By applying F-CRF as a punctuation prediction model for ASR text, the best translation performance for both translation directions can be achieved. Additionally, we also evaluate translation performance when human-annotated punctuation marks are used for translation. The average BLEU scores for the two translation tasks are 31.58 (Chinese to English) and 24.16 (English to Chinese), showing that our punctuation prediction model gives competitive performance for spoken translation.

Table 5: Translation performance (average percentile score of BLEU) using Moses' punctuated ASR output

In accordance with the foregoing embodiments, one exemplary method for predicting punctuation of transcribed dialogue utterance text is described. The proposed method builds on the Dynamic Conditional Random Field (DCRF) framework, which performs punctuation prediction along with sentence boundary and sentence type prediction on speech utterances. Text processing according to DCRF can be done without relying on prosodic cues. Based on a latent event language model, exemplary embodiments outperform widely used conventional methods. The disclosed embodiments have been shown to be language-neutral and work well for both Chinese and English, and both correctly recognize and automatically recognize text. The disclosed embodiments also lead to better translation accuracy when punctuated automatically recognized text is used in subsequent translations.

Figure 8 is a flowchart illustrating one embodiment of a method for inserting punctuation into a sentence. In one embodiment, method 800 begins at block 802 by identifying words of an input utterance. At block 804, words are placed in a plurality of first nodes. At block 806, a word-level label is assigned to each first node of the plurality of first nodes based at least in part on neighboring nodes of the plurality of first nodes. According to one embodiment, sentence-level labels and/or word-level labels may also be assigned to the first node based in part on boundaries of the input utterance. At block 808, an output sentence is generated by combining words from the plurality of first nodes with punctuation marks selected in part on the word-level labels assigned to each of the first nodes.

Grammatical error correction

There is a difference between training on annotated learned text and training on non-learned text, namely whether the observed words can be used as features. When training on non-learned text, observed words cannot be used as features. The author's word choice will be "cancelled" from the text and serve as the correct class. Classifiers are trained to re-predict words given the surrounding context. The confusion set of possible classes is usually predefined. This selection task formulation is convenient because training examples can be created "for free" from arbitrary text that is assumed to be free of grammatical errors. A more realistic correction task is defined as follows: Given a specific word and its context, suggest an appropriate correction. The suggested correction can be the same as the observed word, ie no correction is necessary. The main difference is that the author's word choice can be encoded as part of the feature.

Article errors are a frequent type of error made by EFL beginners. For article errors, the classes are three articles, a, the, and the zero article. This covers article insertion, deletion and substitution errors. During training, each noun phrase (NP) in the training data is a training example. When training on the learned text, the correct classes are articles provided by human annotators. When training on non-learned text, the correct class is the article of observation. The context is encoded via a set of feature functions. During testing, each NP in the test set is a test case. When testing on the learned text, the correct class is the article provided by the human annotator, while when testing on the non-learned text, the correct class is the observed article.

Preposition mistakes are another frequent type of mistake made by EFL beginners. Mistakes on prepositions are done in a similar way to mistakes on articles, but typically focus on preposition substitution mistakes. In this work, the classes are 36 frequent English prepositions (about, along, among, around, as, at, beside, besides, between, by, down, during, except, for, from, in, inside, into, of, off, on, onto, outside, over, through, to, toward, towards, under, underneath, until, up, upon, with, within, without). Each prepositional phrase (PP) that depends on one of the 36 prepositions is a training or testing example. In this example, PPs governed by other prepositions are ignored.

Figure 9 illustrates one embodiment of a method 900 for correcting syntax errors. In one embodiment, method 900 may include receiving 902 natural language text input, where the input text includes grammatical errors, where a portion of the input text includes classes from a set of classes. The method 900 may also include generating 904 a plurality of selection tasks from the corpus of non-learned text assumed to be free of grammatical errors, wherein for each selection task, the classifier re-predicts the class used in the non-learned text. Further, the method 900 may include generating 906 a plurality of correction tasks from the corpus of learning text, wherein for each correction task, the classifier suggests a class to use in the learning text. Additionally, the method 900 may include training 908 a grammar correction model using a set of binary classification problems including multiple selection tasks and multiple correction tasks. This embodiment may also include using 910 the trained grammar correction model to predict the class of the text input from a set of possible classes.

According to one embodiment, grammatical error correction (GEC) is formulated as a classification problem and a linear classifier is used to solve the classification problem.

Classifiers are used to approximately learn the relationship between articles, prepositions and their contexts in text, as well as their efficient corrections. Articles or prepositions and their contexts are represented as feature vectors Calibration is class

In one embodiment, a binary linear classifier of the form uTX is used, where u is a weight vector. If the score is positive, the result is considered +1, and if the score is negative, the result is considered -1. A popular method for finding u is empirical risk minimization with least squares regularization. Given a training set {X _i ,Y _i } _i=1,...,n , the goal is to find the weight vector that minimizes the empirical loss on the training data.

where L is the loss function. In one embodiment, a revision of Huber's robust loss function is used. According to one embodiment, the regularization parameter λ can reach 10-4. A multi-class classification problem with m classes can be transformed into an m-ary classification problem in the one-vs-many setting. The classifier's prediction is the one with the highest score classifier.

Six feature extraction methods are implemented, three for articles and three for prepositions. The methods require different linguistic preprocessing: chunking, CCG analysis, and constituency analysis.

Examples of feature extraction for article errors include "DeFelice", "Han", and "Lee". DeFelice - A system for article errors uses a CCG analyzer to extract a rich set of syntactic and semantic features, including part-of-speech (POS) tags, hypernyms from wordnets, and named entities. The Han-system relies on shallow syntactic and lexical features derived from chunkers including words before, during and after NP, first words and POS tags. The Lee-system uses a compositional analyzer. Features include POS tags, surrounding words, first words, and hypernyms from WordNet.

Examples of feature extraction for preposition errors include "DeFelice", "TetreaultChunk", and "TetreaultParse". DeFelice – The system for prepositional errors uses a rich set of syntactic and semantic features similar to the system for article errors. In the reimplementation, no subclass partition dictionary is used. TetreaultChunk - system uses chunking to extract features from two word windows surrounding prepositions, including vocabulary and POS n-grams, and first words from adjacent constituents. TetreaultParse - System that extends TetreaultChunk by adding additional features derived from compositional and correlation parse trees.

For each of the above feature sets, observed articles or prepositions are added as additional features when training on the learned text.

According to one embodiment, alternate structure optimization (ASO) of a multi-task learning algorithm using a common structure of multiple related problems can be used for grammatical error correction. Suppose there are m binary classification problems. Each classifier ui is a weight vector of dimension p. Let θ be an orthogonal h × p matrix that captures the common structure of the m weight vectors. It is assumed that each weight vector can be decomposed into two parts: one part models the specific i-th classification problem and one part models the common structure.

u _i ＝w _i +Θ ^T v _i

The parameters [{w _i ,v _i },Θ] are learned by joint empirical risk minimization, i.e., by minimizing the joint empirical loss for m problems on the training data.

$\underset{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}<span\; class="notranslate">\; (</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; no</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; (</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&Theta;</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; Y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; .</span>$

In ASO, the problem used to find θ does not have to be the same as the target problem to be solved. Instead, auxiliary problems can be created automatically for the purpose of learning a separate objective of better θ.

Assuming that there are k-objective problems and m auxiliary problems, the approximate solution to the above problems can be obtained by the following algorithm:

1. Independently learn m linear classifiers u _i .

2. Let U=[u ₁ , u ₂ . . . u _m ] be a matrix p×m formed from m weight vectors.

3. At U: Perform singular value decomposition (SVD) on . The first h column vectors of _V1 are stored as columns of θ.

4. Learn w _j and v _j for each target problem by minimizing empirical risk:

$\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; no</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; (</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&Theta;</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; Y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; .</span>$

5. The weight vector for the jth target problem is:

u _j ＝w _j +Θ ^T v _j .

Beneficially, the selection task for non-learned text is a highly informative auxiliary problem for the correction task for learned text. For example, a classifier that can predict the presence or absence of the preposition on can be beneficial in correcting erroneous uses of on in the learned text, e.g., if the classifier's confidence in on is low but the author uses the preposition on, the author may have made a mistake. wrong. Because auxiliary questions can be created automatically, the power of large corpora of non-learned texts can be compromised.

In one embodiment, assume a syntax error correction task with m classes. For each class, a binary auxiliary problem is defined. The feature space for the auxiliary problem is to restrict the original feature space χ to all features except the observed words: The weight vectors for the auxiliary problem form the matrix U in step 2 of the ASO algorithm, from which θ is obtained by SVD. Given θ, the vectors wj and vj, j=1,...,k can be obtained from the annotated learning text using the full feature space χ.

This can be seen as an instance of transfer learning, since the auxiliary problem is trained on data from a different domain (not the learned text) and has a slightly different feature space The method is general and can be applied to any classification problem in GEC.

Evaluation metrics are defined for two experiments on non-learned text and learned text. For experiments on non-learned text, accuracy, defined as the number of correct predictions divided by the total number of test instances, is used as the evaluation metric. For experiments on learning text, the F1-measure is used as the evaluation metric. The F1-measure is defined as:

where precision is the number of suggested corrections agreed with human annotators divided by the total number of corrections suggested by the system, and recall is suggested corrections agreed with human annotators divided by the total number of corrections annotated by human annotators number of errors.

A set of experiments is designed to test the correction task on the NUCLE test data. A second set of experiments investigates the primary goal of this work: automatically correcting grammatical errors in the learned text. Test instances are extracted from NUCLE. Compared to previous selection tasks, the authors' observed word choices can be different from the correct class and observed words are available during testing. Investigate two different baseline and ASO methods.

The first baseline is a classifier trained on Gigaword in the same manner as described in the experiments on the choice task. A simple thresholding strategy is used to use observed words during testing. The system flags an error only if the difference between the classifier's confidence for its first choice and the observed word is above a threshold t. For each feature set, the threshold parameter t is tuned on the NUCLE development data. In the experiments, the value of t was between 0.7 and 1.2.

The second baseline is a classifier trained on NUCLE. The classifier was trained in the same manner as the Gigaword model, except that the author's observed word choices were included as features. The correct class during training is the correction provided by human annotators. Since the observed words are part of the features, the model does not require an additional thresholding step. In fact, thresholding is harmful in this situation. During training, the instances that do not contain errors will greatly outnumber the instances that do contain errors. To reduce this imbalance, all instances that contain errors are kept and a random sample of q percent of instances that do not contain errors is kept. For each dataset, the undersampling q is tuned on the NUCLE development data. In experiments, the value of q was between 20% and 40%.

The ASO method is trained in the following manner. Create binary auxiliary questions for articles or prepositions, i.e. there are 3 auxiliary questions for articles and 36 auxiliary questions for prepositions. The classifier for the auxiliary problem is trained on all 10 million instances from Gigaword in the same manner as the choice task experiments. The weight vectors for the auxiliary questions form the matrix U. Singular value decomposition (SVD) is performed to obtain U=V1DV2T. All columns of V1 are maintained to form θ. The target problem is again a binary classifier problem for each article or preposition, but this time trained on NUCLE. Include the author's observed word choices as features for the target question. Instances that do not contain errors are undersampled and the parameter q is tuned on the NUCLE development data. The value of q is between 20% and 40%. No thresholding is applied.

The learning curves for the correction task experiments on the NUCLE test data are shown in FIGS. 11 and 12 . Each subplot shows the curves for the three models described in the last section: ASO trained on NUCLE and Gigaword, a baseline classifier trained on NUCLE, and a baseline classifier trained on Gigaword. For ASO, the x-axis shows the number of training instances for the target question. We observe that training on annotated learning text can significantly improve performance. In three experiments, the NUCLE model outperformed the Gigaword model trained on 10 million instances. In the end, the ASO model showed the best results. In experiments where the NUCLE model already performed better than the Gigaword benchmark, ASO gave relatively or slightly better results. In those experiments where neither of the two baselines (TetreaultChunk, TetreaultParse) showed good performance, ASO achieved larger improvements over either baseline.

Semantic collocation error correction

In one embodiment, the frequency of collocation errors is due to the author's native or first language (L-1). These types of errors are known as "L1-transition errors". L1-transition errors were used to estimate how many errors in EFL writing could potentially be corrected using information about the author's L1-language. For example, L1-translation errors may be the result of imprecise translations between words in the author's L-1 language and English. In such an example, a word in Chinese that has multiple meanings may not translate precisely to, for example, English.

In one embodiment, the analysis is based on the NUS Corpus of English for Beginners (NUCLE). The corpus consists of around 1400 papers written by EFL University students on a wide range of topics like environmental pollution or healthcare. Most of the students are native speakers of Chinese. The corpus includes approximately one million words fully annotated with error labels and corrections. Annotations are stored in a balanced fashion. Each error label includes the start and end displacement of the annotation, the type of error, and the appropriate golden correction considered by the annotator. If the selected word or phrase is to be replaced by a correction, the annotator is asked to provide a correction that will result in a grammatical sentence.

In one embodiment, errors that have been flagged as mislabeled miscollocations/idioms/prepositions are analyzed. A fixed list of frequent English prepositions is used to automatically filter out all instances representing simple substitutions of prepositions. In a similar manner, the small number of article errors flagged as collocations will be filtered out. Ultimately, instances where the annotated phrase or suggested correction is longer than 3 words are filtered out because they contain highly context-specific corrections and are unlikely to generalize well (e.g., “for the simple reasons that these can help "→"simply to").

After filtering, 2747 collocation errors and their respective corrections were generated, these accounted for approximately 6% of all errors in NUCLE. This makes collocation errors the seventh largest error category after article errors, redundancy, prepositions, sentence word count, verb tense, and semantics. Not counting duplications, there are 2412 different collocation errors and corrections. Although there are other error types that are more frequent, collocation errors represent a particular challenge because the possible corrections are not limited to a closed set of choices, and they relate directly to semantics rather than syntax. Collocation errors were analyzed and it was found that they could be attributed to the following sources of confusion:

Spelling: Mistakes can be caused by similar orthography if the edit distance between the wrong phrase and its correction is less than a certain threshold.

Homophones: If the wrong word and its correction have the same pronunciation, errors can be caused by similar pronunciations. Monophonic dictionaries are used to map words to their phonetic representations.

Synonyms: If the wrong word and its correction are synonyms in WordNet, synonyms can cause errors. Use WordNet 3.0.

L1-transition: If the wrong phrase and its correction share a common translation in the Chinese-English phrase list, errors can be caused by L1-transition. The details of phrase table construction are described here. Although in this particular example the method is used on Chinese-English translation, the method can be applied to any language pair where a parallel corpus is available.

Since monophonic dictionaries and WordNet are defined for individual words, the matching process is extended to phrases in the following way: two phrases A and B have the same length and the i-th word in phrase A is the corresponding i-th word in phrase B homonyms/synonyms of words, then the two phrases A and B are considered homophones/synonyms.

Table 6: Analysis of collocation errors. The threshold for misspellings is 1 for phrases up to 6 letters and 2 for the remaining phrases.

Suspected error source mark type spell 154 131 homophone 2 2 synonyms 74 60 L1-transition 1016 782 L1 - Conversion w/o spelling 954 727

L1-transition w/o homonyms 1015 781 L1-transition w/o synonyms 958 737 L1 - Convert w/o spelling, homonyms, synonyms 906 692

Table 7: Examples of collocation errors with different sources of confusion. Corrections are shown in parentheses. For the L1-conversion, the shared Chinese translation is also shown. The L1-transition examples shown here do not belong to any other category.

The results of the analysis are shown in Table 6. Flag indicates that the run included duplicate error phrase correction pairs and type indicates a different error phrase-correction pair. Since collocation errors can be part of more than one category, the rows in the table do not add up to the total number of errors. The number of errors traceable to L1-transitions greatly exceeds that of all other categories. The table also shows the number of collocation errors that could be traced to L1-transitions but not other sources. 906 collocations with 692 different collocation types could be attributed to L1-transitions rather than spelling, homophones, or synonyms. Table 7 shows some examples of collocation errors for each category from our corpus. There are also types of collocation errors that cannot be traced to any of the aforementioned sources.

A method 1300 for correcting collocation errors in EFL composition is disclosed. One embodiment of this method 1300 includes automatically identifying 1302 one or more translation candidates in response to a corpus analysis of the parallel language text performed in the processing device. Additionally, the method 1300 may include using the processing device to determine 1304 features associated with each translation candidate. The method 1300 may also include generating 1306 a set of one or more weight values from a corpus of learning text stored in a data storage device. The method 1300 may further include calculating 1308, using the processing means, a score for each translation candidate in response to the features associated with the one or more translation candidates and the set of one or more weight values.

In one embodiment, the method is based on L1 -induced paraphrasing. L1-Elicited Paraphrasing with Parallel Corpus is used to automatically find collocation candidates from sentence-aligned L1-English parallel corpus. Since most papers in the corpus are written by native Chinese speakers, use the FBIS Chinese-English corpus, which consists of approximately 230,000 Chinese sentences (8.5 million words) from news articles, each with a single English translation . The English portion of the corpus is tokenized and lowercased. The Chinese part of the corpus is segmented using a maximum entropy segmenter. Subsequently, the text is automatically aligned at the word level using the Berkeley aligner. Extract English-L1 and L1-English phrases of up to three words from the aligned text using a phrase extraction heuristic. Given the English phrase e2, the paraphrase probability of the English phrase e1 is defined as:

$\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; f</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>$

where f denotes a foreign phrase in the L1 language. Phrase translation probabilities p(e ₁ |f) and p(f|e ₂ ) are estimated by maximum likelihood estimation and smoothed using Good-Turing smoothing. Ultimately, only paraphrases with probabilities above a certain threshold (set to 0.001 in this work) were retained.

In another embodiment, the method of collocation correction can be implemented in the framework of phrase-based statistical machine translation (SMT). Phrase-based SMT tries to find the highest-scoring translation e given an input sentence f. The decoding process to find the highest scoring translation is guided by a log-linear model that scores translation candidates using a set of feature functions hi,=1,...,n.

$\mathrm{<span\; class="notranslate">\; score</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; .</span>$

Typical features include phrase translation probability p(e|f), reverse phrase translation probability p(f|e), language model score p(e), and fixed phrase penalties. The optimization of the feature weights λ _i , i=1,...,n can be done by using minimum error rate training (MERT) on the development set of input sentences and parametric translations.

The phrase table of the phrase-based SMT decoder MOSES is modified to include collocation corrections with features derived from spelling, homonyms, synonyms, and L1-induced paraphrases.

Spelling: For each English word, the phrase table contains an entry consisting of the word itself and each word that lies within some edit distance from the original word. Each entry has a fixed characteristic of 1.0.

Homonyms: For each English word, the phrase table contains an entry consisting of the word itself and each word's homonyms. Homophones were determined using the CuVPlus dictionary. Each entry has a fixed characteristic of 1.0.

Synonyms: For each English word, the phrase table contains an entry consisting of the word itself and each of its synonyms in WordNet. If a word has more than one meaning, all its meanings are considered. Each entry has a fixed characteristic of 1.0.

L1-paraphrases: For each English phrase, the phrase table contains an entry consisting of each of the phrase and its L1-derived paraphrases. Each entry has two real-valued features: paraphrase probability and reverse paraphrase probability.

Baseline: Phrase tables built for spelling, homonyms, and synonyms are combined, where the combined phrase table contains three binary features for spelling, homonyms, and synonyms, respectively.

All: Phrase tables from spelling, homonyms, synonyms, and L1-paraphrases are combined, where the combined phrase table contains five features: three binary features for spelling, homonyms, and synonyms and one for L1- Two Realized Features of Interpretation Probability and Inverse L1-Interpretation Probability.

Additionally, each phrase table contains standard fixed phrase penalty features. The first four tables only contain collocation candidates for individual words. It is left to the decoder to construct corrections for longer phrases during decoding if necessary.

A set of experiments is performed to test the method for semantic collocation error correction. The dataset used for experiments is a development set of 770 sentences randomly sampled from the corpus and a test set of 856 sentences. Each sentence contains exactly one collocation error. Sampling is performed in such a way that sentences from the same document cannot end up in both the development and test sets. In order to keep the conditions as realistic as possible, the test set is not filtered in any way.

Evaluation metrics are also defined for experiments to evaluate pairing error correction. Perform automated and human assessments. The main evaluation metric is the Mean Reciprocal Rank (MRR), which is the arithmetic mean of the inverse ranks of the first correct answers returned by the system.

$\mathrm{<span\; class="notranslate">\; MRR</span>}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; N</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; rank</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>}$

where N is the size of the test set. If the system does not return the correct answer for the test instance, then Set to zero.

In human evaluation, the precision at rank k, k = 1, 2, 3 is additionally reported, where the precision is calculated as follows:

$\mathrm{<span\; class="notranslate">\; P</span>}\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; A</span>}}\mathrm{<span\; class="notranslate">\; score</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span>}{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; A</span>}<span\; class="notranslate">\; |</span>}$

where A is the set of returned answers of rank k or less and score(·) is a real-valued score function between zero and one.

In the collocation error experiments, the automatic correction of collocation errors can theoretically be divided into two steps: i) identifying the wrong collocations in the input, and ii) correcting the identified collocations. It is assumed that a mismatch has already been identified.

In the experiments, the start and end offsets of collocation errors provided by human annotators are used to identify the location of collocation errors. The translation of the remainder of the sentence is fixed in its identity. Phrase table entries where the phrase and the candidate correction are the same are removed, which effectively forces the system to change the recognized phrase. The distortion limit of the decoder is set to zero for monotonic decoding. For the language model, a 5-gram language model was used, trained with modified Kneser-Ney smoothing on the English Gigaword corpus. All experiments use the same language model to allow fair comparisons.

MERT training with the popular BLEU metric is performed on a development set of erroneous sentences and their corrections. Since the search space is limited to changing a single phrase per sentence, training converges relatively quickly after two or three iterations. After convergence, the model can be used to automatically correct new collocation errors.

The performance of the proposed method is evaluated on a test set of 85 sentences, each with one collocation error. Both automated and human assessments are performed. In the automated evaluation, the performance of the system is measured by computing the rank of the golden answers provided by human annotators in the n-best column of the system. The size of the n-best column is limited to the top 100 outputs. If no golden answer is found in the top 100 outputs, the rank is considered infinite, or in other words, the inverse rank is zero. Reports the number of test instances for which the golden answer is ranked among the top k answers, k=1,2,3,10,100. The results of the automated evaluation are shown in Table 8.

Table 8: Results of automated evaluation. Columns 2 to 6 show the number of golden answers ranked within the top k answers. The last column shows the mean rank reciprocal as a percentage. The higher the value, the better.

Model rank=1 Rank ≤ 2 Rank ≤ 3 Rank≤10 Rank≤100 MRR spell 35 41 42 44 44 4.51 homophone 1 1 1 1 1 0.11 synonyms 32 47 52 60 61 4.98 baseline 49 68 80 93 96 7.61 L1-interpretation 93 133 154 216 243 15.43 all 112 150 166 216 241 17.21

Table 9: Inter-annotator agreement P(E) = 0.5.

P(A) 0.8076

Kappa 0.6152

For collocation errors, there is usually more than one possible corrective answer. Thus, by considering only a single golden answer as correct and all others as false, automated evaluation underestimates the actual performance of the system. Perform manual assessment against system baseline and all. Two English speakers were recruited to judge a subset of 500 test sentences. For each sentence, the judge is shown the original sentence and the 3 best candidates for each of the two systems. Human evaluation is limited to the 3 best candidates, since answers at ranks greater than 3 will be less useful in practical applications. Candidates are displayed together in alphabetical order without any information about their rank or which system produced them or the golden answer provided by the annotator. Differences between candidate and original sentences are highlighted. For each candidate, a judger is asked to make a binary judgment as to whether the proposed candidate is a valid correction of the original. Valid corrections are represented by a score of 1.0, while invalid corrections are represented by a score of 0.0. Inter-annotator agreement is reported in Table 9. Probability of agreement P(A) is the percentage of times the annotators agree and P(E) is the expected agreement by chance, which is 0.5 in our case. The Kappa coefficient is defined as

$\mathrm{<span\; class="notranslate">\; Kappa</span>}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; A</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; )</span>}$

A Kappa coefficient of 0.6152 was obtained from the experiment, where a Kappa coefficient between 0.6 and 0.8 was considered to show substantial agreement. To calculate the precision at rank k, the judgments are averaged. Thus, for each returned answer, the system may receive a score of 0.0 (two judgments are negative), 0.5 (the judges disagree), or 1.0 (two judgments are positive).

All of the methods disclosed and claimed herein can be made and performed without undue experimentation in light of the present disclosure. Although the apparatus and methods of the present invention have been described in terms of preferred embodiments, it will be apparent to those skilled in the art that variations may be applied to the methods and in the steps or in the sequence of steps of the methods described herein without departing from concept, spirit and scope of the present invention. Furthermore, modifications may be made to the disclosed apparatus and components may be eliminated or substituted for those described herein, wherein the same or similar results are achieved. All such similar substitutes and modifications apparent to those skilled in the art are deemed to be within the spirit, scope and concept of the invention as defined by the appended claims.

Claims (26)

1., for correcting a method for grammar mistake, the method comprises:

Reception natural language text inputs, and described Text Input comprises grammar mistake, and wherein a part for input text comprises the class coming from one group of class;

Do not have the corpus of the non-learning text of grammar mistake to generate multiple selection task from hypothesis, wherein for each selection task, the class used in non-learning text predicted again by sorter;

Multiple correction tasks is generated from the corpus of learning text, wherein for each correction tasks, the class that sorter suggestion uses in learning text;

Use one group of binary class problem to train syntactic correction model, this group of binary class problem comprises multiple selection task and multiple correction tasks; And

The syntactic correction model of use training carrys out the class from one group of possible class prediction Text Input.

2. method according to claim 1, comprises further and exports suggestion, if so that the class of prediction is different from the class in Text Input, then the class of Text Input is changed over the class of prediction.

3. method according to claim 1, wherein said learning text is annotated to suppose correct class by teacher.

4. method according to claim 1, wherein said class is the article associated with the noun phrase in input text.

5. method according to claim 4, comprises further and extracts fundamental function for sorter from the noun phrase non-learning text and learning text.

6. method according to claim 1, wherein said class is the preposition associated with the prepositional phrase in input text.

7. method according to claim 6, comprises further and extracts fundamental function for sorter from the prepositional phrase of non-learning text and learning text.

8. method according to claim 1, wherein said non-learning text and learning text take on a different character space, and the feature space of learning text comprises the word used by author.

9. method according to claim 1, wherein trains syntactic correction model to comprise the loss function minimized on training data.

10. method according to claim 1, trains syntactic correction model to comprise further and identifies multiple linear classifier by analyzing non-learning text.

11. methods according to claim 10, wherein said linear classifier comprises weight factor further, and this weight factor is included in the matrix of weight factor.

12. methods according to claim 11, the matrix of wherein training described syntactic correction model to be included in weight factor further performs svd (SVD).

13. methods according to claim 12, wherein train syntactic correction model also can comprise recognition combination weighted value, the representative of this combined weights weight values is by the first weighted value element of analyzing non-learning text and identifying and the second weighted value element identified by minimizing empirical risk function and carrying out analytic learning text.

14. 1 kinds of equipment, comprising:

At least one processor and the storage arrangement being coupled to this at least one processor, at least one processor wherein said is configured to:

Reception natural language text inputs, and described Text Input comprises grammar mistake, and wherein a part for input text comprises the class coming from one group of class;

Do not have the corpus of the non-learning text of grammar mistake to generate multiple selection task from hypothesis, wherein for each selection task, the class used in non-learning text predicted again by sorter;

Multiple correction tasks is generated from the corpus of learning text, wherein for each correction tasks, the class that sorter suggestion uses in learning text;

Use one group of binary class problem to train syntactic correction model, this group of binary class problem comprises multiple selection task and multiple correction tasks; And

The syntactic correction model of use training carrys out the class from one group of possible class prediction Text Input.

15. equipment according to claim 14, comprise further and export suggestion, if so that the class of prediction is different from the class in Text Input, then the class of Text Input is changed over the class of prediction.

16. equipment according to claim 14, wherein said learning text is annotated to suppose correct class by teacher.

17. equipment according to claim 14, wherein said class is the article associated with the noun phrase in described input text.

18. equipment according to claim 17, comprise further and extract fundamental function for sorter from the noun phrase non-learning text and learning text.

19. equipment according to claim 14, wherein said class is the preposition associated with the prepositional phrase in input text.

20. equipment according to claim 19, comprise further and extract fundamental function for sorter from the prepositional phrase of non-learning text and learning text.

21. equipment according to claim 14, wherein said non-learning text and learning text take on a different character space, and the feature space of learning text comprises the word used by author.

22. equipment according to claim 14, wherein train syntactic correction model to comprise the loss function minimized on training data.

23. equipment according to claim 14, wherein train described syntactic correction model to comprise further and identify multiple linear classifier by analyzing non-learning text.

24. equipment according to claim 23, wherein said linear classifier comprises weight factor further, and this weight factor is included in the matrix of weight factor.

25. equipment according to claim 24, the matrix of wherein training described syntactic correction model to be included in weight factor further performs svd (SVD).

26. equipment according to claim 25, wherein train syntactic correction model also can comprise recognition combination weighted value, the representative of this combined weights weight values is by the first weighted value element of analyzing non-learning text and identifying and the second weighted value element identified by minimizing empirical risk function and carrying out analytic learning text.