Federer feels worst is behind him but not rushing return