Grain legumes (cowpea, peanut, and soybean) play important roles in household food and income security in smallholder farming systems in the Guinea Savanna agro-ecological zones of Ghana. However, yields are low, rarely exceeding 600 kg ha−1, prompting the need to evaluate responses of grain legumes to P fertilizer applications for two seasons. Conducting P studies is critical to help farmers adopt economic-based recommendations. Treatments evaluated in 2015 for the three crops were (i) farmers’ practice (no input and planted by farmer); (ii) control (no input and planted by researcher), and (iii) triple super phosphate (TSP) fertilizer. However, for soybean, an additional two treatments (inoculant only and inoculant plus TSP fertilizer) were included. In 2016, the treatments were the same, except on-farm demonstrations were not conducted on cowpea. The demonstrations were laid out in a Randomized Complete Block Design with each demonstration representing a replicate within a region. On average, P-fertilizer application increased yields by 296; 527, and 390 kg ha−1 for cowpea, peanut, and soybean grains, respectively. On average over the two seasons, P-fertilizer increased yield by 9.85; 13.00, and 17.56 per kg ha−1 kg−1 P applied for cowpea, soybean, and peanut, respectively, and these applications were cost effective. Peanut showed little response to P in the Upper East Region compared with a greater response in the Northern and Upper West Regions, suggesting that benefits from P-fertilizer for peanut may be location-specific. On average, rhizobium inoculation increased grain yield by 157 kg ha−1 across the three regions and significantly positive effects of inoculation were observed in both seasons. Our results show that substantial increases in grain legume yield may be achieved by applying P fertilizers, but farmers cannot afford them because of their relatively high cost. Planting adapted and improved varieties and using rhizobium inoculants may provide the most economically viable and low risk options for increasing yields of grain legumes in the savanna agro-ecological zones of Ghana.