The four Muslims identify themselves as female and have been repeatedly arrested for wearing women's clothes and hair accessories.
They appealed to a secular court to overturn the Islamic Sharia law making it illegal for men to pose as women.
People convicted under this law can be jailed or fined.
But the high court in Seremban, Negri Sembilan state, denied the petitioners' case to declare the Sharia law - used by Muslims in religious courts - as unconstitutional.
The transsexuals' lawyer, Aston Paiva, told the Associated Press news agency that they had asked the court to uphold freedom of expression and forbid gender discrimination.
He added that the four, who are in their 20s and work as makeup artists, were "disappointed and distraught" after the verdict. They are still deciding whether to appeal against the verdict.
The BBC's Jennifer Pak in Kuala Lumpur says support for the case in Muslim-majority Malaysia is hard to come by.
