We conducted a systematic review of literature on the residual effects of grain legumes in cereal-based systems of sub-Saharan Africa (SSA) to quantify the magnitude and variability of rotational effects, to explore the importance of environmental and management factors in determining variability and to evaluate the evidence of the different mechanisms that explain rotational effects. We retrieved 44 unique publications providing 199 observations comparing continuous cereal performance with that of a grain legume-cereal rotation. The overall mean yield increase of 0.49 t grain ha−1, equal to an increase of 41% of the continuous cereal yield, is highly significant, but the variability in residual effects is large. Effects were more pronounced in southern Africa, the highlands of East Africa and the Guinea savannah, and less in the humid forest/derived savannah of West Africa and the Sudano-Sahelian zone. Maize showed stronger yield responses after a legume than millet and sorghum. Agro-ecological zone and cereal type were however confounded. All grain legume types significantly improved cereal yields, with stronger residual effects observed after soybean and groundnut than after cowpea. Fertiliser N application to cereals reduces the residual effects of legumes, but the response at 60–120 kg N ha−1 still equalled 0.32 t ha−1 or 59% of the response when no N is applied. The sustained benefits with large N applications indicate the importance of non-N effects. While mechanisms for improved soil P availability after grain legumes have been studied in some detail, it remains uncertain how important these are in farmers’ fields. Grain legumes are unlikely to have a major influence on the availability of nutrients other than N and P, or on soil pH. Beneficial impacts of grain legumes on soil organic matter content can occur if legumes contribute to a greater overall cropping productivity, but studies generally report no such impacts. Evidence of impacts of grain legumes on weeds is limited to striga. Studies on the impacts on nematode pressure in cereals are inconclusive, probably because legumes act as a host for some of the key nematode genera that harm maize. The impact on the pressure of other pests and diseases in cereals is probably important, but evidence on this from SSA is lacking. Future research on N2-fixation by grain legumes and residual N benefits should focus on explaining the wide variability observed among sites. There is a clear need for more detailed mechanistic studies to assess the occurrence and relevance of non-N effects of grain legumes, particularly in relation to common pests and diseases in cereals.