Consumers’ demand for “clean label” foods requires replacing commonly used additives with natural ingredient alternatives such as oat β-glucan, a soluble dietary fiber. Its consumption is associated with the reduction of coronary heart disease risk and its use improves the texture and mouthfeel of food products. However, adding β-glucan in dairy formulations often causes phase separation. This phenomenon is attributed to the thermodynamic incompatibility of micellar casein and β-glucan. The objective of this study was to investigate the casein - β-glucan interaction mechanism. Casein and β-glucan were mixed at 9 concentrations with ranges of 1.2 to 2% β-glucan and 4 to 6% casein and processed under 3 thermal conditions, namely pasteurization at 73°C/15 s, retort sterilization at 121°C/5 min and control (not treated). A phase behavior diagram was constructed to determine the critical biopolymer concentrations for phase separation. The stability of the samples was calculated as the percentage of the total volume occupied by the upper phase. Dynamic light scattering (DLS) was applied to investigate the biopolymer interactions through particle size distribution. The experimental design is a full factorial of 2 factors with multiple levels each. The phase behavior diagram showed that the heat treatment intensity increased significantly (P < 0.05) the critical total biopolymer (casein and β-glucan) concentration from 6.75% to 7.8%. Increasing the biopolymer concentration reduced the stability to 28%, while increasing the thermal treatment intensity increased the stability at levels higher than 90%. DLS results showed that increasing β-glucan concentration increased significantly the mean particle size from 300 nm to 1.8 μm (P < 0.05) revealing aggregate formation. The results indicate that segregative phase separation occurred due to limited biopolymer compatibility. Increasing the biopolymer concentration decreased the formulations’ stability, while increasing the thermal treatment intensity resulted to more stable samples. This was confirmed by DLS, which showed smaller particle size for high heat treatment and low concentrations.