Please read these links found here: About the T Replacement Category - #2 by KSman
HPTA restart
That clomid dose is to high. LH can then be too high and T–>E2 inside the testes can raise E2 quite high. When you stop clomid, you are swimming in E2 that shuts you down. See the link for suggested approaches. Note that some guys feel like shit on clomid, they can do well using nolvadex that does not cause that problem in any guys.
Doctors do not know about these fine details. You have to understand and advocate for yourself.