How long do I have to stay out from work?

I saw a doctor on Sunday morning and he put me on azithromycin for bronchitis. He told me that I should not return to work until Wednesday. This seems like too long. I thought you were only contagious for 24 hours after starting antibiotics. Should I listen to the doctor or am I safe to return to school/work tomorrow?