(Jan. 30) -- Jake Gyllenhaal is taking the death of Heath Ledger "harder than most people," as Ledger's "Brokeback Mountain" co-star and the godfather of his daughter tries to make sense of the sudden death.

PEOPLE, citing a source on the set of Gyllenhaal's latest movie, "Brothers," says that the actor is very distraught over the news, and that he left the set immediately when he learned of Ledger's passing.

"This has had a strong personal affect" on Gyllenhaal, the source says.

After leaving the set, Gyllenhaal then returned, but the source says it was obvious he was only there physically.

"He was there, but he wasn't with us. It was obviously a major trauma," the source says, adding that Gyllenhaal was "sitting in the director's chair staring off into space."

Ledger and Gyllenhaal shared the screen in "Brokeback Mountain," in which they played two cowboys who become romantically involved and are forced to hide their relationship. Ledger met Michelle Williams on the same set, with whom he had a daughter, Matilda, that Gyllenhall became the godfather of.