Is it possible that one of your rear brakes are binding / is your handbrake going off fully?
I sometime get a clunk when pulling off if I've had my handbrake on and its been left for a couple of weeks. I suspect on mine its condensation and the rear pads just sticking to the disc's and there is some residue on the disc's as you can feel the brakes catching (not binding) untill they are warmed up.
I don't put the handbrake on when its parked in my garage and it does not happen.
Maybe its a hint for me to give the rears a good looking at / service and grease the linkage and pins.