The common currency in developing solutions to sustainable dairy production are often focusing on increased animal production efficiency through improved animal health. In particular, the use of data driven technologies for early disease detection has shown a lot of promise. In this study, we show how systematically recording fertility and disease events from herd management systems can be helpful in improving existing animal monitoring systems. Moreover, while previous research focused primarily on detecting single outcomes such as mastitis lameness and insemination outcomes, we propose a multiclass prediction model which forecasts a probability distribution over 12 possible life events. Traditional techniques such as Markov for discrimination models can use historical sequences of disease and fertility records to predict a cow’s future state. Additionally, we investigated whether more advanced recurrent neural network algorithms are better able to uncover the complex data patterns hidden in the event sequences. Finally, we examined if augmenting a cow’s history of events with pictures take can enhance the predictive performance even further by making use of convolutional neural network models. While most picture studies are conducted as experimental designs using complex video camera setups and expensive equipment such as thermal scanners, we worked with pictures taken by the farmers and personel their smartphones, which to our most recent knowledge, has not yet been applied in the context of animal monitoring systems. While the Markov for discrimination models their percentage correctly classified (PCC) and Top-3 PCC ranged from 67% to 68% and from 87% to 88% respectively, the neural network models achieved a PCC of 75% and a Top-3 PCC of 95%. Results show that an ensemble model incorporating sequential as well pictorial information performs best and that the model is able to accurately predict future states such as calving, mastitis, pregnancy and death with an accuracy of 97%. The framework presented in this research can be used to enhance current animal monitoring systems with better animal health and higher sustainability for the dairy industry as a result.