Structure-based fragment growing is one of the key techniques in fragment-based drug design. Fragment growing is commonly practiced based on structural and biophysical data. Computational workflows are employed to predict which fragment elaborations could lead to high-affinity binders. Several such workflows exist but many are designed to be long running noninteractive systems. Shape-based descriptors have been proven to be fast and perform well at virtual-screening tasks. They could, therefore, be applied to the fragment-growing problem to enable an interactive fragment-growing workflow. In this work, we describe and analyze the use of specific shape-based directional descriptors for the task of fragment growing. The performance of these descriptors that we call ray volume matrices (RVMs) is evaluated on two data sets containing protein-ligand complexes. While the first set focuses on self-growing, the second measures practical performance in a cross-growing scenario. The runtime of screenings using RVMs as well as their robustness to three dimensional perturbations is also investigated. Overall, it can be shown that RVMs are useful to prefilter fragment candidates. For up to 84% of the 3299 generated self-growing cases and for up to 66% of the 326 generated cross-growing cases, RVMs could create poses with less than 2 Å root-mean-square deviation to the crystal structure with average query speeds of around 30,000 conformations per second. This opens the door for fast explorative screenings of fragment libraries.