Longitudinal follicle tracking is needed in clinical practice for diagnosis and management in assisted reproduction. Follicles are tracked over the in-vitro fertilization (IVF) cycle, and this analysis is usually performed manually by a medical practitioner. It is a challenging manual analysis and is prone to error as it is largely operator dependent. In this paper we propose a two-stage framework to address the clinical need for follicular growth tracking. The first stage comprises of an unsupervised deep learning network SFR-Net to automate registration of each and every follicle across the IVF cycle. SFR-Net is composed of the standard 3DUNet [1] and Multi-Scale Residual Blocks (MSRB) [2] in order to register follicles of varying sizes. In the second stage we use the registration result to track individual follicles across the IVF cycle. The 3D Transvaginal Ultrasound (3D TVUS) volumes were acquired from 26 subjects every 2-3 days, resulting in a total of 96 volume pairs for the registration and tracking task. On the test dataset we have achieved an average DICE score of 85.84% for the follicle registration task, and we are successfully able to track follicles above 4 mm. Ours is the novel attempt towards automated tracking of follicular growth [3].Clinical Relevance- Accurate tracking of follicle count and growth is of paramount importance to increase the effectiveness of IVF procedure. Correct predictions can help doctors provide better counselling to the patients and individualize treatment for ovarian stimulation. Favorable outcome of this assisted reproductive technique depends on the estimates of the quality and quantity of the follicular pool. Therefore, automated longitudinal tracking of follicular growth is highly demanded in Assisted Reproduction clinical practice. [4].