Would you give me a close up of a few of the leaves closest to the fruit and also a close up of the fruit itself off the tree,so I can see the brown areas better?
Also a shot of the base of the tree and area around that.
Are you using any chemicals on this tree?
This can be many things.I need those shots to better determine what the problem is.
There is also A good book by L.C.Knorr "Citrus Diseases and Disorders" you might try and find.