Mr Sanchez secured 180 votes out of 350, with 169 votes against the motion and one abstention.
Ciudadanos, which prior to the vote was leading in the national opinion polls, was the only major party that supported Mr Rajoy.
Rajoy was prime minister for seven years.
Spanish Prime Minister-elect Pedro Sanchez has begun the task of harnessing the unlikely forces that brought him to power.
Mr Rajoy admitted defeat ahead of a no-confidence vote today saying it had been "an honour" to serve the country. "It has been an honor to leave Spain better than I found it", Rajoy said in his final comments as prime minister. MPs from the anti-establishment Podemos party shouted: "Yes you can!" as he was named prime minister.
Sanchez said he meant to call elections before the end of this parliamentary term in 2020, but he didn't say when, and he probably will want to make his mark first with some headline policies before going to the polls. The Socialists only have 84 seats - just under a quarter of the total.
The motion of no confidence was tabled after the PP was implicated in a huge corruption scandal. Sanchez backed Rajoy on his firm line in defense of the Spanish constitution, which doesn't permit regions to break away.
ABC Sets Meetings to Hear 'Roseanne' Spinoff Ideas
She's not doing well and doesn't want to travel, and she's gone radio silent on me, so I'm just going to step away. As Radar has reported, the disgraced comedian's former co-stars are busy discussing saving the show.
Google reportedly won’t renew its controversial military AI contract
Academics and researchers also lashed out at Google for abandoning the company's original motto: " Don't be evil ". Thousands of staff signed the petition , which stated: "Google should not be in the business of war".
After Months Of Talks, Italy Has A New Coalition Government
That sparked a breakdown of the coalition's first attempt to form a government and a furious reaction from the coalition leaders. But the two parties reached a new coalition deal Thursday reviving their plans to take power with Giuseppe Conte at the helm.
After the vote result was announced, members of Podemos smiled and hugged each other before bursting into a chant of "Yes, we can" - the slogan of their party, a product of the widespread anger at spending cuts imposed by Rajoy's first government at the height of the euro zone crisis.
He faces a tough time, however, catering to demands from small nationalist parties whose votes he captured in the no-confidence motion. The public reaction emboldened Sanchez to file the motion against Rajoy, and compelled his new allies to unite behind him. Despite these failures, he was re-elected as party leader in 2017.
He also warned of risky alliances with political groups that openly challenge the Spanish Constitution and question territorial unity.
As such, even if he has pledged to govern long enough to restore "institutional stability" before calling early elections, Sanchez's new government will likely be highly unstable.
"The PP has had corrupt people, I acknowledge it but the PP is not a corrupt party", he said, before accusing Sanchez of "opportunism at the service of personal ambition".
The end of Mr Rajoy's six-year reign as PM was the first removal of a serving leader by the parliament in Madrid in four decades of democracy.