Hybrid speciation

Hybrid speciation is a form of speciation where hybridization between two different species leads to a new species, reproductively isolated from the parent species. Previously, reproductive isolation between two species and their parents was thought to be particularly difficult to achieve, and thus hybrid species were thought to be very rare. With DNA analysis becoming more accessible in the 1990s, hybrid speciation has been shown to be a somewhat common phenomenon, particularly in plants. In botanical nomenclature, a hybrid species is also called a nothospecies. Hybrid species are by their nature polyphyletic.

Ecology
A hybrid may occasionally be better fitted to the local environment than the parental lineage, and as such, natural selection may favor these individuals. If reproductive isolation is subsequently achieved, a separate species may arise. Reproductive isolation may be genetic, ecological, behavioral, spatial, or a combination of these.

If reproductive isolation fails to establish, the hybrid population may merge with either or both parent species. This will lead to an influx of foreign genes into the parent population, a situation called an introgression. Introgression is a source of genetic variation, and can in itself facilitate speciation. There is evidence that introgression is a ubiquitous phenomenon in plants and animals, even in humans, where genetic material from Neanderthals and Denisovans is responsible for much of the immune genes in non-African populations.

Ecological constraints
For a hybrid form to persist, it must be able to exploit the available resources better than either parent species, which, in most cases, it will have to compete with. For example: while grizzly bears and polar bears may be able to mate and produce offspring, a grizzly–polar bear hybrid is apparently less- suited in either of the parents' ecological niches than the original parent species themselves. So: although the hybrid is fertile (i.e. capable of reproduction and thus theoretically could propagate), this poor adaptation would be unlikely to support the establishment of a permanent population.

Likewise, lions and tigers have historically overlapped in a portion of their range and can theoretically produce wild hybrids: ligers, which are a cross between a male lion and female tiger, and tigons, which are a cross between a male tiger and a female lion; however, tigers and lions have thus far only hybridized in captivity. In both ligers and tigons, the females are fertile and the males are sterile. One of these hybrids (the tigon) carries growth-inhibitor genes from both parents and thus is smaller than either parent species and might in the wild come into competition with smaller carnivores, e.g. the leopard. The other hybrid, the liger, ends up larger than either of its parents: about a thousand pounds (450 kilograms) fully grown. No tiger-lion hybrids are known from the wild, and the ranges of the two species no longer overlap (tigers are not found in Africa, and while there was formerly overlap in the distribution of the two species in Asia, both have been extirpated from much of their respective historic ranges, and the Asiatic lion is now restricted to the Gir Forest National Park, where tigers are mostly absent).

Some situations may favor hybrid population. One example is rapid turnover of available environment types, like the historical fluctuation of water level in Lake Malawi, a situation that generally favors speciation. A similar situation can be found where closely related species occupy a chain of islands. This will allow any present hybrid population to move into new, unoccupied habitats, avoiding direct competition with parent species and giving a hybrid population time and space to establish. Genetics, too, can occasionally favor hybrids. In the Amboseli National Park in Kenya, yellow baboons and anubis baboons regularly interbreed. The hybrid males reach maturity earlier than their pure-bred cousins, setting up a situation where the hybrid population may over time replace one or both of the parent species in the area.

Genetics of hybridization
Genetics are more variable and malleable in plants than in animals, probably reflecting the higher activity level in animals. Hybrids' genetics will necessarily be less stable than those of species evolving through isolation, which explains why hybrid species appear more common in plants than in animals. Many agricultural crops are hybrids with double or even triple chromosome sets. Having multiple sets of chromosomes is called polyploidy. Polyploidy is usually fatal in animals where extra chromosome sets upset fetal development, but is often found in plants. A form of hybrid speciation that is relatively common in plants occurs when an infertile hybrid becomes fertile after doubling of the chromosome number.

Hybridization without change in chromosome number is called homoploid hybrid speciation. This is the situation found in most animal hybrids. For a hybrid to be viable, the chromosomes of the two organisms will have to be very similar, i.e., the parent species must be closely related, or else the difference in chromosome arrangement will make mitosis problematic. With polyploid hybridization, this constraint is less acute.

Super-numerary chromosome numbers can be unstable, which can lead to instability in the genetics of the hybrid. The European edible frog appears to be a species, but is actually a triploid semi-permanent hybrid between pool frogs and marsh frogs. In most populations, the edible frog population is dependent on the presence of at least one of the parent species to be maintained, as each individual need two gene sets from one parent species and one from the other. Also, the male sex determination gene in the hybrids is only found in the genome of the pool frog, further undermining stability. Such instability can also lead to rapid reduction of chromosome numbers, creating reproductive barriers and thus allowing speciation.

Homoploid hybrid speciation
Hybrid speciation in animals is primarily homoploid. While thought not to be very common, a few animal species are the result of hybridization, mostly insects such as tephritid fruitflies that inhabit Lonicera plants and Heliconius butterflies,  as well as some fish, one marine mammal, the clymene dolphin, a few birds. and certain Bufotes toads.

One bird is an unnamed form of Darwin's finch from the Galapagos island of Daphne Major, described in 2017 and likely founded in the early 1980s by a male Española cactus finch from Española Island and a female medium ground finch from Daphne Major. Another is the great skua, which has a surprising genetic similarity to the physically very different pomarine skua; most ornithologists now assume it to be a hybrid between the pomarine skua and one of the southern skuas. The golden-crowned manakin was formed 180,000 years ago by hybridization between snow-capped and opal-crowned manakins.

A 2021 DNA study determined that the Columbian mammoth of North America was a hybrid species between woolly mammoths and another lineage, discovered in Krestovka, descended from steppe mammoths. The two populations had diverged from the ancestral steppe mammoth earlier in the Pleistocene. Analysis of genetic material recovered from their remains showed that half of the ancestry of the Columbian mammoths originated from the Krestovka lineage and the other half from woolly mammoths, with the hybridization happening more than 420,000 years ago, during the Middle Pleistocene. This is the first evidence of hybrid speciation obtained from prehistoric DNA.

Multiple hybrids during rapid divergence
Rapidly diverging species can sometimes form multiple hybrid species, giving rise to a species complex, like several physically divergent but closely related genera of cichlid fishes in Lake Malawi. The duck genus Anas (mallards and teals) has a very recent divergence history, many of the species are inter-fertile, and quite a few of them are thought to be hybrids. While hybrid species generally appear rare in mammals, the American red wolf appears to be a hybrid species of the Canis species complex, between gray wolf and coyote. Hybridization may have led to the species-rich Heliconius butterflies, though this conclusion has been criticized.

Hybrid speciation in plants
Hybrid speciation occurs when two divergent lineages (e.g., species) with independent evolutionary histories come into contact and interbreed. Hybridization can result in speciation when hybrid populations become isolated from the parental lineages, leading to divergence from the parent populations.

Polyploid hybrid speciation
In cases where the first-generation hybrids are viable but infertile, fertility can be restored by whole genome duplication (polyploidy), resulting in reproductive isolation and polyploid speciation. Polyploid speciation is commonly observed in plants because their nature allows them to support genome duplications. Polyploids are considered a new species because the occurrence of a whole genome duplication imposes post-zygotic barriers, which enable reproductive isolation between parent populations and hybrid offspring. Polyploids can arise through single step mutations or through triploid bridges. In single step mutations, allopolyploids are the result of unreduced gametes in crosses between divergent lineages. The F1 hybrids produced from these mutations are infertile due to failure of bivalent pairing of chromosomes and segregation into gametes which leads to the production of unreduced gametes by single division meiosis, which results in unreduced, diploid (2N) gametes. Triploid bridges occur in low frequencies in populations and are produced when unreduced gametes combine with haploid (1N) gametes to produce a triploid offspring that can function as a bridge to the formation of tetraploids. In both paths, the polyploid hybrids are reproductively isolated from the parents due to the difference in ploidy. Polyploids manage to remain in populations because they generally experience less inbreeding depression and have higher self-fertility.

Homoploid hybrid speciation
Homoploid (diploid) speciation is another result of hybridization, but unlike polyploid speciation, it is observed less commonly because the hybrids are not characterized by a genome duplication and isolation must develop through other mechanisms. In homoploid speciation, the hybrids remain diploid. Studies on diploid hybrid populations of Louisiana irises show how these populations occur in Hybrid zones created by disturbances and ecotones (Anderson 1949). The existence of these novel niches allows for the persistence of hybrid lineages. For example, established sunflower (Helianthus) hybrid species represent transgressive phenotypes and display genomic divergence separating them from the parent species.