Feature Selection of SVM-RFE Combined with a TD Reinforcement Learning

Ⅰ. Introduction

Many discrimination applications, such as the architecture of massively multi-player online games or gene regulatory networks for differential-expressed genes (DEGs) of RNA-seq infer knowledge from a huge amount of data, Big-Data[1]. There have been main existing issues such as providing guarantees consistency on simultaneous participants of the MMOGs or on the intermediate causality on the GRNs. This knowledge is used to obtain a deeply untestable and relevant meaningful concept that generated the huge data. For example, in terms of Big-Data of DEGs of GRNs, they are consisting of measurements also known as to the featured genes, which is characterizing a basis for the mechanisms through inferring causality between biologically relevant genes[2]-[4].

Also, Interaction information brought into the distributed servers between the huge numbers of participants of MMOGs might be considerable and mainly responsible for encompassing a Big-Data world. They require new criteria for load balancing because the huge amount of history information might be enhancing the most relevant and less redundant form. Based on the measurements associated with a label specifying the category might be estimated dependencies between the examples and their labels by machine learning models[5]-[8].

However, the RNA-seq let researchers gather data sets of ever increasing sizes and the servers of MMOGs learn each other by interactions and bring the huge world, which is not always significant [2]-[4]. This issues might compromise the machine learning models, which is extremely influenced by data qualification, such as redundancy, noise or undermined unreliability. The purpose of feature selection is to try to remove irrelevancies and minimize redundancy patterns to improve the generalizations of the machine learning models[5]-[8]. In terms of RNA-seq, the maximum gene relevancy and minimum gene redundancy is very significant and essential[2]-[4].

Therefore, we try to consider the data feature representation for taking advantages of the server platform of the MMOGs. Thanks to the reinforcement learning [5][6], the self-teaching models are designed for Big-Data on the above-mentioned issues in different kinds of applications, such as MMOGs or GRNs. Before adopting the software on the servers of MMOGs, the feature selection on the huge amount of data might be helpful for reducing server loads on Big-Data issues. Moreover, the feature selection might be appropriate for the prediction of resource demands in other similar researches. Moreover, we think our proposed methods based on reinforcement learning is beyond the qualification of the server platform of MMOGs because enhanced selection of the data makes the enriched meaningful features.

Therefore, we propose a novel algorithm of exploiting the improved effectiveness of criteria derived from “support vector machines-recursive feature elimination” (“SVM-RFE”) [2] combined with temporal-difference (TD) reinforcement learning[5][6] for feature selection in application to differential-expressed genes (DEGs) of RNA-seq. In the application of MMOGs, our proposed model can learn the patterns (features) of past accumulated server loads. By learning the overhead features, the server can take actions promptly to achieve better supports in advance. We employ a TD reinforcement learning to learn how to control the weight vectors of the SVM-RFE. Our experiment results show that the criteria of the weight vector improved by the back-propagation of TD errors accomplishes better results and performs well over the Big-Data sets.

Ⅱ. Materials and Methods

2.2 Methods

The TD, (as known as SARSA) [5][6] starts with a simple policy, samples state spaces with and improves the policy. SARSA means: “S(tate) → A(ction) → R(eward) → S(tate) → A(ction)”

TD prediction is for a control i.e action-selection, generalization of the iteration of the policy and balances between exploration and exploitation. The Q function is that “Q(s^t,a^t)←(s^t,a^t)+α(r^t+1+γQ(s^t+1,a^t+1)– Q(s^t,a^t))”. After per transition from a non-terminal state s^t., this Q function is updated. If s^t+1 is a terminal, then the zero is defined in the “Q(s^t+1,a^t+1)”. This update rule is defined in every element of “(s^t, a^t, r^t+1, s^t+1, a^t+1)” [5][6].

The TD reinforcement learning algorithm attempt to estimate or enhance the policy, which is defined to make better action-selection, known as the on-policy. The on-policy often uses soft action choice, i.e. π(s; a) > 0 ∀a, performs for the exploration and tries to find out the best policy that develops. It might be trapped into local minima. The on-policy TD reinforcement learning means that while learning the best optimal policy per iteration, it uses the current estimate of the best optimal policy to engender the behavior for the exploration and converges to an optimal policy, while all “S(tate) → A(ction)” pairs have been visited a limited times and the best policy converges into the limit with the greedy policy (“ε =1/t”)[5][6].

Fig. 1.

On-policy TD control algorithm

Ⅲ. Proposed Algorithm

We propose “SVM-RFE” with the TD reinforcement learning for feature selection in some applications. There is the proposed algorithm in Fig 2. Because the choice of criteria is depending on the parameters of “SVM-RFE”, it could be necessary to train the TD reinforcement learning with different sets of feature patterns. The TD reinforcement learning tries to learn how to control the criteria based on the weight vectors of the “SVM-RFE”. For the criterion of Leray et al. [9], they use a back-propagation (gradient descent) by the derivatives of ||w||² with respect to scaling vector associated to variables[7].

Fig. 2.

Proposed algorithm

The SVM–RFE and gradient of ||w||² are as same as they have the identical ranking criterion [9]. For previous developed feature selection algorithms, choice criteria are subject to the human-interaction(heuristic individual) feature evaluation functions such as hyper-parameters. Several methods use individual evaluation of the features for ranking them and do not take into considerations for their dependencies or their correlations[2][4].

This may be problematic for selecting discriminate relevant sets of feature patterns. In our proposed “SVM-RFE” with TD reinforcement learning, SVM has been trained in each iteration, depending on different sets of action reinforcement. The reinforcement has been improving for selecting more differential-expressed genes by back propagation of gradient descent. In our proposed algorithm, we try to eliminate that flaws of the heuristic technique and give a good change to over-fitting problem by learning TD.

Ⅳ. Evaluation of the proposed algorithm

For feature selection on the gene expression micro-array data, it is informative to select a minimal relevant gene subset. The topmost-ranked feature genes might not be the most relevant featured genes.

Although the “SVM-RFE” is known for the correlation-based, some of the previous results are unstable. Currently, that is known that there is smaller is the best featured genes with gene samples. However, because of small size of samples and high correlation among genes, finding the smallest relevant gene subset might not justify the superiority of the gene selection algorithm comparing with others. That is because of “Curse of Dimension”[4].

The smaller relevant subsets might not be extracted in a controllable computation limits. In the controllable limits, we might select the featured subsets that describe the complicated cancerous reaction and regulation in “AML/ALL” leukemia data set[3]. Some results might cover the cancer mechanisms or some might not. The proposed “SVM-RFE” with the TD reinforcement learning is comparing with the previous developed “SVM-RFE” based on the results on Golub. et. al.[3].

The “AML/ALL” leukemia consists of 72 samples (47 for ALL and 25 for AML). In the “AML/ALL”, there are 7,129 feature genes. We try to describe how many featured genes are almost the same.

Fig. 3 shows that the featured genes of “SVM-RFE” with the TD reinforcement learning, “SVM-RFE” [2] and the ranked list of Golub. et. al.[3]. The all genes selected in “AML/ALL” leukemia [3] are not the same with our result. However gene, M27891 (Cystatin), Y00787(IL-8 precursor), M63138(cathepsin D), M92287(cyclin D3), X17042(Proteoglycan), M28130 (IL-8), X95735(Zyxin), M69043(MAD-3), and M96326 (Azurocidin) are in the relevant topmost 25. This result indicates that the proposed algorithm can improve the performance by using TD reinforcement learning prediction in terms of generalization of the iteration method and balances between exploration and exploitation. It also was proved that the smaller is the best featured genes with gene samples in controllable computational limits with the accuracy in smaller size of samples and high correlation among genes. Fig. 3 shows that the feature genes of “SVM-RFE” with TD reinforcement learning is considerably comparative with Golub. et. al.[3].

Fig. 3.

Comparison of the proposed algorithm with SVM-RFE based on Golub

Ⅴ. Conclusion

We propose the algorithm to select some featured genes in a controllable computation and to ensure the training model before evaluation for the relevancy of featured genes. This makes to consider that the training model are explored and enhanced by featured genes. We exploit the efficiency of criteria derived from “support vector machines-recursive feature elimination” (“SVM-RFE”) [2] combined with the TD reinforcement learning [5][6] for feature selection in application to differential-expressed genes of RNA-seq. We employ the TD reinforcement learning to learn how to control the criteria based on the weight vectors of the “SVM-RFE”. We exploit a gradient descent by the derivatives of the weight vectors of “SVM-RFE”.

The ranking criterion based on the ϵ-greedy has been slightly effected in an iterative way of TD reinforcement learning. The SVM has been trained in each iteration, depending on different sets of reinforcement leaning policy. The reinforcement leaning policy has been improving for selecting more differential-expressed genes by back propagation of gradient descent. We try to eliminate that flaws of the heuristic technique and give a good change to over-fitting problem by learning the learning policy of TD reinforcement learning. The result show that the reinforced criteria on the weight vector improved by the back-propagation of TD errors acquires better results.

References

• D. Silver, A. Huang, C. J. Maddison, A. Guez, L.t Sifre, G. V. D. Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, "Mastering the game of go with deep neural networks and tree search", Nature, 529(7587), p484-489, Jan), (2016. [https://doi.org/10.1038/nature16961]
• I. Guyon, J. Weston, S. Barnhill, and V. Vapnik, "Gene selection for cancer classification using support vector machine", Mach. Learn, 46, p389-422, Jan), (2002.
• T. R. Golub, D. K. Slonim, P. Tamayo, C. Huard, M. Gaasenbeek, J. P. Mesirov, H. Coller, M. L. Loh, J. R. Downing, M. A. Caligiuri, C. D. Bloomfield, and E. S. Lander, "Molecular classification of cancer: class discovery and class prediction by gene expression monitoring", Science, 286(5439), p531-537, Oct), (1999. [https://doi.org/10.1126/science.286.5439.531]
• Y. Tang, Y. Q. Zhang, and Z. Huang, "Development of Two-Stage SVM-RFE Gene Selection Strategy for Microarray Expression Data Analysis", IEEE ACM Transactions on Computational Biology and Bioinformatics, 4(3), p365-381, Aug), (2007. [https://doi.org/10.1109/tcbb.2007.70224]
• R. S. Sutton, "Learning to predict by the methods of temporal differences", Machine learning, 3(1), p9-44, Aug), (1988. [https://doi.org/10.1007/bf00115009]
• R. S. Sutton, and A. G. Barto, "Reinforcement learning", An introduction, MIT press Cambridge, second edition, ISBN: 9780262039246 (2018).
• T. Tieleman, and G. Hinton, "Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude", COURSERA: Neural Networks for Machine Learning, 4(2), p26-31, https://www.youtube.com/watch?v=LGA-gRkLEsI (2012).
• V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, "Playing atari with deep reinforcement learning", arXiv preprint arXiv:1312.5602, NIPS 2013, (2013).
• P. Leray, and P. Gallinari, "Feature selection with neural networks", Behaviormetrika, 26, p6-16, (Jan), (1999).