Due to the mismatch between training and test conditions, speaker verification in real environments, continues to be a challenging problem. An effective way of improving such a system is taking advantage of multiple speaker verification systems. In this paper, we propose a novel sample specific speaker verification system. Using this system, the best classifiers are selected as the ensemble set and the optimal weights are obtained for each test sample. In this process, more reliable scores are forced to have higher weights, while less reliable scores are forced to have lower weights. We achieved an improvement of 0.81% in equal error rate (EER), 0.76% in minimum decision cost function (minDCF) and 3.62% in minimum log-likelihood ratio cost (minCLLR) on the NIST 2004 Speaker Recognition Evaluation dataset.