Thick black smoke was seen billowing across the capital's skyline from the hotel in Knightsbridge.
The hotel says it is "aware of a situation".
The cause of the fire is also not yet known, and firefighters are likely going to be on the scene throughout the night, the London Fire Brigade said.
One witness, Jack, 19, told MailOnline: "I came out of Knightbridge Station when I saw a huge amount of black smoke". There were hundreds of fire engines, ambulances, police racing by and helicopters overhead.
"After Grenfell, I feel like people are a lot more on edge about these things, people just wanted to get out".
Cavaliers' Rodney Hood says playing in Cleveland 'has been tough'
It's a part of my growth. "Just try and be physical, I think that's the biggest thing T-Lue's been stressing to me once I got there".
Garbine Muguruza sends Maria Sharapova to worst Slam loss in 6 years
Serena Williams has provided a positive update on the chest injury that caused her to pull out of the 2018 French Open on Monday. An anti-climactic match ended predictably, with a Sharapova backhand nestling in the net - her 27th unforced error.
Only a few of Guatemala volcano's dead have been identified
Work was disrupted on Tuesday when a new eruption sent hot gas and molten rock streaming down its south side. A baby buried under ash after a volcanic eruption has miraculously survived.
Transport for London said the blaze was also affecting entry and exit to Knightsbridge tube station.
He said: "I obviously was quite shocked when I saw all that smoke".
The 47-year-old financial services worker, visiting from Las Vegas, said: "I was just coming up from the Tube and smelled the fire. I can't see any smoke, there are firefighters and the whole building is dripping wet right now", he added.
Roads were closed off at Scotch Corner and Hyde Park Corner, busy junctions near the hotel - an Edwardian-style building that opened its doors in 1902 and was bought by Mandarin Oriental in 1996.
The hotel overlooks Hyde Park and is located less than a mile away from the Harrods and Harvey Nichols department stores, and Buckingham Palace.
The 12-storey Mandarin Oriental has previously been used by the Royal Family.