If it is on the trunk then it is not so bad, it is when the outer branches get covered and choke out the leaves. Many times on trunks people mistake lichen for moss.
Smaller trees have even more problem I talked to a lady when I was in Savanah GA and she was complaining that the spanish moss was choking out the crepe myrtel
BTW lichen is a symbiote of fungus and algea. it lives in many moist dark surfaces, like roofs, trunks, stone... it is benign on mature bark.