Atlases have a tremendous impact on the study of anatomy and function, such as in neuroimaging, or cardiac analysis. They provide a means to compare corresponding measurements across populations, or model the variability in a population. Current approaches to construct atlases rely on examples that show the same anatomical structure (e.g., the brain). If we study large heterogeneous clinical populations to capture subtle characteristics of diseases, we cannot assume consistent image acquisition any more. Instead we have to build atlases from imaging data that show only parts of the overall anatomical structure. In this paper we propose a method for the automatic contruction of an un-biased whole body atlas from so-called fragments. Experimental results indicate that the fragment based atlas improves the representation accuracy of the atlas over an initial whole body template initialization.