COMPARING THE LIKELIHOOD FUNCTIONS OF PHYLOGENETIC TREES

A. BAR-HEN^{1}* AND H. KISHINO^{2}

1The Institute of Statistical Mathematics, 4-6-7 Minami-Azabu, Minato-ku, Tokyo 106-8569, Japan 2Department of Social and International Studies, University of Tokyo, 3-8-1 Komaba, Meguro-ku, Tokyo 153-8902, Japan

(Received March 29, 1996; revised August 10, 1998)

Abstract. DNA sequence data provide a good source of information on the evolutionary history of organisms. Among the proposed methods, the maximum likelihood methods require an explicit probabilistic model of nucleotide substitution that makes the assumption clear. However, procedures for testing hypotheses on topologies have not been well developed. We propose a revised version of the maximum likelihood estimator of a tree and derive some of its properties. Then we present tests to compare given trees and to derive the most likely candidates for the true topology, applying to maximum likelihoods the notion of contrast, as defined in the framework of the analysis of variance, and the procedures used in multiple comparison. Finally, an example is presented.

Key words and phrases: Maximum likelihood, multiple comparisons, phylogeny.