Did you wait a few min for the heat to build up in the pipe before you opened the door or did you open the door right away after opening the damper? If you wait 3 or 4 min after you open the damper, more heat will go up the pipe and create a better draft, once that draft gets started it may help keep smoke from coming out of the door. Remember the smoke and heat will take the least path of resistance.BTW, in the past, opening a door has done nothing to improve the billowing smoke.
Drilling holes in the baffle is an interesting idea. I may do that. And maybe try to un-warp it a bit while I'm at it.