JPH04360291A - Method for learning neural net - Google Patents

Abstract

PURPOSE:To improve the accuracy of high output selected by a candidate selecting means out of plural outputs by multiplying an error calculated from the output of a neural net and similarity by a term to be increased/decreased in accordance with the size of the similarity and minimizing the multiplied value as an error function. CONSTITUTION:In this learning method, a feature vector is inputted and similarity to plural categories is outputted. An error calculated from the output of the neural net and the similarity is multiplied by the term to be increased/ decreased in accordance with the size of the similarity and the multiplied result is used as an error function and minimized. Namely in the flow chart of a neural net learning procedure, an error adjusting step 14 is added to a convensional learning method. In the step 14, an error calculated by error calculation between the output value of a step 13 and a teaching signal is multiplied by an error function to adjust the error due to the size of similarity. Thus the highly accurate collating result of pattern information can be obtained.

Description

[Detailed description of the invention]

0001

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a neural network learning method, and more particularly to a neural network learning method for matching pattern information expressed by feature vectors having a finite number of elements.

0002

2. Description of the Related Art FIG. 4 shows a diagram for explaining the configuration of a pattern recognition device. This pattern recognition device is for recognizing pattern information typified by sounds, images, characters, and the like. The pattern recognition device includes a cutting means 41, a normalizing means 42, a feature extracting means 43, and a pattern matching means 44. When the pattern information a is input, the cutting means 4
1, a partial pattern of the pattern information a to be recognized is cut out, and then, normalization means 42 performs normalization such as positioning and scaling. Further, the feature extraction means 43 converts the normalized input into a feature vector b having a finite number of elements, and the pattern matching means 44
The optimal category is selected as result c.

[0003] In the selection of the optimal category performed by the pattern matching means 44, external information other than the matching result c of the feature vector b may also be used as a selection criterion. For example, in a character recognition device, a word including a character to be recognized can be estimated from the recognition results of characters before and after the character to be recognized. The category of the character to be recognized determined from the estimated word is external information independent of the feature vector b generated from the character. This external information is also used when performing pattern matching. When performing pattern matching in this manner, a pattern matching device is used as the pattern matching means 44.

FIG. 5 is a diagram for explaining the configuration of a pattern matching device. The pattern matcher includes similarity generation means 51 and candidate selection means 52. Among them, the similarity generation means 51 generates the similarity between the input feature vector b and the dictionary vector representing each category, and generates information on the category name and similarity for several categories with the highest similarity. d is output to the candidate selection means 52. The candidate selection means 52 weights the degree of similarity based on external information e obtained separately from the degree of similarity information d, selects one final category name, and outputs the selected category name as a result f. Depending on the external information e, the category that shows the highest degree of similarity in the similarity generation means 51 is not necessarily selected, but a category that ranks second or lower may be selected.

[0005] Here, a neural network may be used as the similarity generation means 51. Similarity generation means 5
1 needs to output similarities for multiple categories, so the neural network also needs to output analog values instead of binary values of "0" or "1." FIG. 6 shows a diagram for explaining a three-layer neural network. This neural network is used as similarity generation means 51. This neural network receives a feature vector with six elements as input and outputs similarity for each of three categories. The neural network is composed of an input layer unit 61, a middle layer unit 62, and an output layer unit 63. Each unit has a state variable “Number 1”
, "Math 2" and "Math 3", respectively, where "Math 1" is the state variable of the input layer, "Math 2" is the state variable of the middle layer, and "Math 3" is the state variable of the input layer.
” indicates the state variable of the output device.

[Math 1]

[Math 2]

[Math 3]

[0006] Here, the subscripts I, H, and O mean input layer, intermediate layer, and output layer, respectively, and i, j, and k respectively mean i.
, j, means the k-th unit. All the input layer units, all the hidden layer units, all the hidden layer units, and all the output layer units are each combined weight "Equation 4".
” and “number 5”.

[Math 4]

[Equation 5] First, the input feature vector is sent to the input layer unit 6 for each element.
1 and the state variable “Math 1” of the input layer unit 61
each takes the value of each element of the feature vector. Next, each input layer unit 61 sends an output value of "Math. 6" to the intermediate layer unit 62.

[Equation 6] Here, the function on the right side above is a linearly differentiable monotonically increasing function. Each intermediate layer unit 62 corresponds to each input layer unit 61.
The sum of the output value "Equation 7" multiplied by the connection load "Equation 4" is set as the state variable "Equation 8".

[Math 7]

[Math. 8] Similarly, the state variables of the output layer unit 63 are:

[Number 9
], and the final output from the output layer unit is

[Number 10
] becomes.

[0007] Neural network learning is based on the similarity between the input feature vector and the category used as the teacher signal (usually 0).
(more than 1.0 or less) (hereinafter referred to as learning patterns) are prepared, and the connection weight between the units is set by the formula 4.
This is done by changing the equation 5. That is, consider the following function as a learning evaluation function. E is the learning evaluation function.

[Formula 11] Here, y'k is the similarity to the k-th category in the learning pattern. To minimize the learning evaluation function E,

[Equation 12] The connection load is changed by [Equation 12]. Here, ε is a positive constant. This learning method is called steepest descent.

FIG. 7 shows a neural network learning procedure according to a conventional method. Step 71; The input feature vector is read into the input layer unit. Step 72; Proceed the calculation to the intermediate layer unit and the output layer unit to obtain the output value of the neural network. Step 73; Calculate the error between the output value of the neural network and the teacher signal. Step 74; Determine the amount of change in connection load. Step 75; By the amount of change obtained in step 74,
Change bond loads. Learning is performed by performing the above operations for each learning pattern.

[0009]

[Problem to be solved by the invention] However, in order to generate similarities for each category from feature vectors using a neural network, it is necessary to prepare learning patterns that include similarities of various sizes and have the neural network learn them. . This is because if you prepare only feature vectors that are close to dictionary vectors that represent a certain category, that is, have a high degree of similarity, and perform learning, the output unit corresponding to that category will output a high degree of similarity no matter what kind of input it receives. This is because there is a risk that the child will learn to do the same thing. Therefore, there is a problem in that learning patterns that give low similarity to feature vectors that are far from dictionary vectors must also be prepared.

FIG. 8 is a schematic diagram showing the output of a neural network when learning is performed using a conventional learning method. The figure schematically shows the relationship between the degree of similarity in learning patterns in the neural network where learning has converged and the output from the actual output unit. The horizontal axis is a scale of similarity in learning patterns, and the vertical axis shows the output value of the neural network. As a result of the neural network learning the similarity with the value on the horizontal axis (located on the thick line upward to the right), the output of the neural network for that feature vector tends to be distributed in the range between the two dotted lines. . In this way, in the conventional neural network learning method, learning is performed so that the output uniformly approaches the degree of similarity in the learning pattern, regardless of the degree of similarity.

[0011] Here, the candidate selection means 52 shown in FIG. 5 selects one candidate from among several categories showing a relatively high degree of similarity. For this reason, whether a candidate is selected or not by the candidate selection means 52 is generally a problem in areas with a high degree of similarity. In other words, in the similarity generating means 51, high output values must be accurate, but even if low output values are somewhat inaccurate, the result will not be affected. However, in the conventional neural network learning method, learning is performed uniformly regardless of the degree of similarity.

[0012] In general, neural networks have a limited number of intermediate layer units, so they cannot completely represent input and output mappings for a large number of learning patterns, and the evaluation function E will not become smaller than a certain value. Learning converges there. When a neural network is trained using a conventional method, the entire learning may be stopped due to an attempt to express an image with a low degree of similarity in the learning pattern. When a neural network trained in this manner is used in the similarity generating means 51, there is a problem that the precision of the high similarities selected by the candidate selecting means 52 deteriorates.

[0013] The present invention has been made in view of the above points.
When using a neural network as a similarity generation means for a pattern matcher, the pattern matcher can be The purpose of this study is to provide a neural network learning method that can obtain highly accurate matching results for pattern information.

[0014]

[Means for solving the problem] In a neural network learning method that takes feature vectors as input and outputs similarities for multiple categories, a term that increases or decreases depending on the magnitude of the similarity is added to the error calculated from the output and the similarity. Multiply, use this as an error function, and minimize the error function. Furthermore, in the neural network learning method described above, similarity is used as a term that increases or decreases depending on the magnitude of similarity. Furthermore, the square of the similarity is used as a term that increases or decreases depending on the magnitude of the similarity.

[0015]

[Operation] In the learning of the neural network used as the similarity generation means of the pattern matcher, the error in the similarity between the output of the output layer unit and the learning pattern, which is the teacher signal, is multiplied by a monotonically increasing function as the learning evaluation function. By adjusting the connection weight so as to minimize this amount, it is possible to improve the accuracy of output at values where the degree of similarity is relatively high.

[0016]

Embodiment The neural network learning method of the present invention uses learning patterns similar to those applied to conventional methods. In the following example, a learning method of a neural network when a learning pattern is given will be described.

[0017] Here, it is assumed that the degree of similarity between a dictionary vector of a certain category and an input feature vector is expressed as a real number between 0 and 1.0. If the similarity is 0, the input feature vector is far from the dictionary vector, and the similarity is 1.
If it is 0, the input feature vector is completely equal to (close to) the dictionary vector. The following evaluation function is used to train the neural network.

[Formula 13] Here, g(y'k) is 0≦y'k≦1.0, and y
It is a function that increases monotonically with respect to 'k, and "Equation 14" is the output from the final output layer unit. y'k is the similarity to the k-th category in the learning pattern.

##EQU00001## Here, "Equation 15" is used to find the square error of the similarity between the output of the output layer unit and the learning pattern that is the teacher signal. By multiplying this by g(y'k) and taking the sum of all output units, the evaluation function E' shown in equation (6) can be obtained.

[Math. 15]

Next, g(y'k) is a term that increases or decreases depending on the degree of similarity. Two examples of g(y'k) are shown below. First, when the error function g(y'k)=y'k, the evaluation function E'1 is

[Math. 16]

Furthermore, the error function g(y'k)=y'k
2, the evaluation function E'2 is

[Formula 17] Equations (7) and (8) above are obtained by multiplying the error between the output value of the neural network and the similarity in the learning pattern by the similarity y'k and y'k 2 . When a neural network is trained to minimize these evaluation functions E'1 and E'2, the amount of change in connection weights is weighted depending on the degree of similarity. In other words, when the degree of similarity is high, the error between the output and the degree of similarity is made relatively large, and when the degree of similarity is low, it is made relatively small. The amount of change becomes relatively large, and the amount of change becomes relatively small for the connection weights that were involved in producing a low degree of similarity. When a neural network with the same number of units as the conventional method is trained using this learning method, when the learning converges, it can output a similarity value that is higher than that of a neural network trained using the conventional method and is faithful to the learned pattern. We can expect it to become like this.

FIG. 1 shows a flowchart of a neural network learning procedure according to an embodiment of the present invention. This figure shows a process in which error adjustment in step 14 is added to the conventional learning method shown in FIG. In step 14, the error calculated by calculating the error between the output value and the teacher signal in step 13 is multiplied by the error function g(y'k) to obtain the similarity y'k.
Adjust the error due to the size of .

FIG. 2 is a schematic diagram showing the output of a neural network when learning is performed using a learning method according to an embodiment of the present invention. The horizontal and vertical axes of this figure are the same as those of the conventional schematic diagram of FIG. In FIG. 2, there is a tendency for the output of the neural network to be distributed in a range close to the learning pattern with a relatively high degree of similarity in the learning pattern compared to the learning results obtained by the conventional method. As a result of learning the similarity in the learning patterns on the horizontal axis located on the thick line in FIG. 2, the output of the neural network is distributed within the range between the dotted lines. Therefore, it can be seen that the error is small in areas with high similarity.

Furthermore, FIG. 3 is a diagram showing the relationship between similarity in learning patterns and output error when learning is performed using the learning method according to the present invention, and FIG. A diagram showing the relationship between similarity in learning patterns and output error. Figures 3 and 9 show the results of simulations that apply actual data to the concept of the schematic diagrams in Figures 2 and 8 shown above, and the learning results in the conventional learning method shown in Figure 9 are approximate. It is distributed evenly, and in the learning method of the present invention shown in FIG. 3, it is distributed at the lower right in the horizontal axis direction. This results in
The higher the similarity, the smaller the error. If the similarity is low, even if there is some error, it will not affect the results.

[0023] By using a neural network created by such a learning method as a similarity generation means, it is possible to construct a pattern matcher with higher accuracy than conventional methods.

[0024]

[Effects of the Invention] As described above, according to the present invention, a trained neural network is used as a similarity generation means.
Since the pattern matcher can generate relatively high degrees of similarity selected by the candidate selection means in the similarity generation means, it can generate more accurate pattern information with a higher degree of similarity than when using a neural network trained by a conventional method. It is possible to obtain the following matching results.

[Brief explanation of drawings]

FIG. 1 is a flowchart of a learning procedure of a neural network according to an embodiment of the present invention.

FIG. 2 is a schematic diagram showing the output of a neural network when learning is performed using a learning method according to an embodiment of the present invention.

FIG. 3 is a diagram showing the relationship between similarity in learning patterns learned by the learning method according to the present invention and output error.

FIG. 4 is a diagram for explaining the configuration of a pattern recognition device.

FIG. 5 is a diagram for explaining the configuration of a pattern matcher.

FIG. 6 is a diagram for explaining a three-layer neural network.

FIG. 7 is a flowchart of a procedure for learning a neural network according to a conventional method.

FIG. 8 is a schematic diagram showing the output of a neural network when learning is performed using a conventional learning method.

FIG. 9 is a diagram showing the relationship between similarity in learning patterns and output errors when learning is performed using a conventional learning method.

[Explanation of symbols]

41 Cutting means 42 Normalization means 43 Feature extraction means 44 Pattern matching means 51 Similarity generation means 52 Candidate selection means 61 Input layer unit 62 Middle layer unit 63 Output layer unit

Claims (3)

[Claims] Claim 1: A neural network learning method that takes feature vectors as input and outputs similarities for multiple categories, in which the error calculated from the output of the neural network and the similarities is multiplied by a term that increases or decreases depending on the magnitude of the similarities. ,
A neural network learning method characterized by using this as an error function and minimizing the error function. 2. The neural network learning method according to claim 1, wherein a degree of similarity is used as a term that increases or decreases depending on the magnitude of the degree of similarity. 3. The neural network learning method according to claim 1, wherein the square of the similarity is used as a term that increases or decreases depending on the magnitude of the similarity.