He is the first Spanish prime minister to lose such a vote since Spain transitioned to democracy in 1977.
There were 180 votes in favour, 169 against and one abstention. Spain's King Felipe VI still has to swear him in.
Sanchez, 46, is now the prime minister in waiting.
Mr Sánchez tabled the motion of no confidence after the PP was implicated in a huge corruption scandal.
Socialist leader Pedro Sanchez in the parliament after the motion of no confidence vote.
The new PM has promised to call early elections, but has not set a date. Rajoy has already warned of the "political instability" a Sanchez-led government would suffer.
Aitor Esteban of the Basque PNV nationalist party, whose support proved decisive for the motion's success, on Thursday warned that such a minority government would be "weak and hard, complicated".
"This is going to be a constant bing, bang, boom".
The 5 Major Questions That Loom Over The 2018 NBA Finals
James would end the night with 51 points on just 32 shots, a line made fuller by his eight rebounds and eight assists. But instead of trying for a winning layup, he moved away from Durant at the hoop and the game went into overtime.
Fed says Deutsche, NYC’s most active CRE lender, is "troubled"
A number of regional offices have already been shut, including in Houston, Texas. S&P said the "deeper restructuring" entails a "non-negligible execution risks".
LeBron James left in disbelief after brilliant Game 1 performance
And, along the way, they got a reminder that this opponent can't so easily be counted out. LeBron was fantastic in last year's Finals - just not as good as Durant.
Following the vote on Friday morning, Rajoy.
The Basque Nationalist Party, whose five seats were key to Sanchez securing enough parliamentary backing, withdrew support from Rajoy after dozens of people linked to his center-right People's Party (PP) were sentenced to decades in jail in a corruption trial.
The former economics professor regained the Socialists' leadership a year ago.
It also ordered the party to pay back 245,000 euros ($290,000) received from the scheme to help finance election campaigns.
In this December 22, 2017 photo, Spain's Prime Minister Mariano Rajoy leaves after a news conference in Madrid, Spain.
Sanchez completed his spectacular turnaround after seizing the opportunity provided by last week's court verdict in a massive corruption case involving Rajoy's conservative Popular Party.
Sanchez had been Rajoy's most loyal backer in his takeover of Catalonia's regional government following its failed secession attempt a year ago.
"Are you Mother Teresa of Calcutta?"