Improving reproductive performance remains a major goal in dairy cattle. Service sire has been recently recognized as an important factor affecting herd fertility. Most genome-wide mapping studies for bull fertility have focused on genetic markers only on autosomal chromosomes. Little is known, however, about the genetic role of sex chromosomes on dairy bull fertility. As such, the main objective of this study was to dissect the genetic contribution of chromosomes X and Y (BTAX and BTAY) on sire conception rate (SCR) in US Holstein bulls. The analysis included a total of 5.1k Holstein bulls with SCR records and genotypes for roughly 3.3k SNPs located on the pseudoautosomal region (PAR), 49.7k BTAX-specific SNPs, and 1.2k BTAY-specific SNPs. The effect of each SNP on SCR was evaluated using an efficient 2-step mixed model-based approach. Three regions located on BTAX (−log₁₀P = 6.7) and one region located on PAR (−log₁₀P = 4.9) showed significant associations with service sire fertility. Interestingly, these regions harbor candidate genes, such as TBL1X and PIH1D3, that are directly implicated in spermatogenesis and testis development. On the other hand, BTAY showed very low genetic variability, and none of the segregating markers was associated with SCR. This study provides evidence of the importance of both PAR and BTAX-specific regions on male fertility in dairy cattle. These findings may help to improve conception rates in dairy herds through accurate genome-guided decisions on bull fertility.