One of the dead was the shooter, the source said, adding there was no apparent link to terrorism.
A white tarp covered what was believed to be a body in front of the entrance to the office building but police declined to confirm if it was the shooter.
A police source said the assailant, who worked nearby, was believed to be a disgruntled employee who fired at a person against whom he had a grievance. The nature of the grievance was not known.
"I heard the gunshots. It was like pop, pop, pop. It was definitely in a bunch," said Dahlia Anister, 33, who works at an office near the 102-story Empire State Building.
The shooting started shortly after 9 a.m. on the busy sidewalk on Fifth Avenue outside the Midtown Manhattan building.
It came at the height of the tourist season outside one of New York City's most popular attractions. Police cordoned off the area around the building, one of the most recognizable in the world.
The Empire State Building is two blocks from Pennsylvania Station and eight blocks from Grand Central Terminal, two of New York City's main transportation hubs.
Mail courier James Bolden, 31, said he saw a "guy laying on the (sidewalk), bleeding from the neck and barely breathing."
"Everybody was crowded around him taking pictures and video, and security guys were yelling everybody to get back, and give him space. He was barely breathing," Bolden said.
One witness said she saw a woman who was shot in the foot and another woman being taken away in an ambulance.
"I was walking down 33rd (Street) and there's a dead guy. I just saw pools of blood. He was laying down and the was blood pooling (around him)," Justin Kellis, 35, who works nearby.
The United States has had two other mass shooting cases this summer. On July 20, a gunman opened fire at a midnight screening of the Batman film "The Dark Knight Rises" in Aurora, Colorado, killing 12 people and wounding 58.
On August 5, a gunman killed six people and critically wounded three at a Sikh temple outside Milwaukee before police shot him dead in an attack authorities treated as an act of domestic terrorism.
This was the second high-profile shooting incident in two weeks in New York's tourist-heavy midtown Manhattan. On August 12 New York City police shot and killed a knife-wielding suspect as he sought to evade them through Saturday afternoon traffic and pedestrians in Times Square.
The Empire State Building was the world's tallest building for 40 years from its completion in 1931 until construction of the World Trade Center. Following the September 11, 2001, attacks, it was again the tallest building in the city, though was recently surpassed by a new tower under construction at the World Trade Center.