Calfhood diseases during early stages of growth can detrimentally affect productive performance in dairy cattle, resulting in considerable economic losses to the dairy industry. Health issues can alter calf behavior, so that change on behavior pattern could be used as an earlier indicator to prevent diseases. However, in large dairy operations, the daily monitoring of calf behavior is laborious, and the large number of animals becomes a limiting factor for such evaluation. Thus, the objective of this study was to develop an automated computer vision system to individually monitor behavior of dairy calves housed in groups. The experiment was conducted at the Dairy Research Cattle Center of the University of Wisconsin-Madison. Five calves were housed in a group from the 4th to 8th weeks of age. All calves had ad libitum access to water and calf starter from the first week of life until the end of the trial. A Wi-Fi camera with night vision capability (Amcrest Outdoor Wi-Fi Camera, model: IP3M-956EW) was installed 4 m above the pen. Images were acquired every 5 s and sent to a data storage in the cloud platform. For each image, calves were spatially located and then labeled for their respective identification (ID, from Calf 1 to Calf 5) and classified for animal behavior (lying, drinking, eating, and standing). A total of 650 images were used for training purpose and 100 images were used for validation. A deep neural network approach called Mask RCNN was implemented to generate the predictions. The algorithm was trained by adopting the strategy of transfer learning, for which the weights from COCO data set were used. All analyzes were implemented in Python using the open source frameworks TensorFlow and Keras. The accuracy to recognize a specific calf were: 77% (Calf 1), 70% (Calf 2), 80% (Calf 3), 92% (Calf 4), 80% (Calf 5). The accuracy to predict the behavior activity of lying, drinking, eating, and standing were: 91, 86, 85, and 100%, respectively. These results indicate that a computer vision system can be a powerful tool to monitor behavior of dairy calves housed in groups.