After Robert, fellow British actor and Avengers Assemble star Tom Hiddleston is second, followed by Pirates of the Caribbean heartthrob Johnny Depp.
Prometheus star Michael Fassbender is fourth and Sherlock actor Benedict Cumberbatch came in fifth place, up from 33 last year.
The top 10 is completed by Robert Downey Jr, Pattinson's Twilight co-star Taylor Lautner, The Vampire Diaries' Paul Wesley, Scottish star James McAvoy and Immortals actor Henry Cavill.
One Direction's Harry Styles, meanwhile, may be a favourite with teenagers but he failed to even make the top 20 in the Glamour list.
The former X Factor was at 21 followed by his bandmates Zayn Malik (27), Niall Horan (34), Louis Tomlinson (48) and Liam Payne (77).
The top 50 also includes David Tennant (39), Colin Firth (41), Doctor Who star Matt Smith (42), David Beckham (46), and Justin Bieber, who slips two places from last year to 49.
Over 40,000 people voted in Glamour magazine's 50 sexiest men of 2012 poll.