This study was conducted to investigate the correlation between milk production and rumen bacteria of dairy cows. A total of 334 Holstein lactating cows fed a same diet with a ratio of forage-to-concentrate at 40:60 were selected to record milk composition and milk yield. Rumen contents of all the cows were collected by an oral stomach tube for analysis of volatile fatty acids (VFA) and 16S rRNA gene sequencing of V3 and V4 regions using Illumina HiSeq. Spearman correlation was analyzed between VFA or milk performance and bacterial communities. Only bacterial genera that averaged >1% abundance in at least 60% of samples were included in the correlation analyses. An unclassified genus from family Succinivibrionaceae was positively correlated with propionate (R = 0.56, P < 0.01). An unclassified genus from family Rikenellaceae was negatively correlated with propionate (R = −0.54, P < 0.01) and valerate (R = −0.50, P < 0.01). Negative correlations also existed between a genus from Christensenellaceae and propionate (R = −0.55, P < 0.01) or valerate (R = −0.53, P < 0.01). Weak correlations (0.3 < |R| < 0.5, P < 0.01) included the positive correlations between Prevotella, Lachnospira and several VFA, and negative correlations between VFA and bacterial genera like Ruminococcaceae NK4A214 group, Ruminococcus, Lachnospiraceae NK3A20 group, Acetitomaculum, or Fibrobacter. Moderate correlations existed between milk yield, milk composition and bacterial communities. Milk yield was negatively correlated with bacteria from family Rikenellaceae (R = −0.22, P < 0.01) and Ruminococcaceae (R = −0.32, P < 0.01). Milk fat content was negatively correlated with genus from family Succinivibrionaceae (R = −0.24, P < 0.01) and genus Lachnospira (R = −0.22, P < 0.01), but positively correlated with Ruminococcaceae NK4A214 (R = 0.25, P < 0.01), Rikenellaceae RC9 (R = 0.22, P < 0.01), and Christensenellaceae R7 group (R = 0.27, P < 0.01). Milk protein content was only positively correlated with Prevotellaceae. In summary, our study suggests a connection between physiological parameters of dairy cows and several genera of their rumen communities, which may play a potential role in shaping dairy cows’ milk production and physiological parameters.