Epidermal-type Transglutaminase 3 (TGase 3) is a Ca(2+)-dependent enzyme involved in the cross-linking of structural proteins required in the assembly of the cell envelope. We have recently shown that calcium-activated TGase 3, like TGase 2, can bind, hydrolyze, and is inhibited by GTP despite lacking structural homology with other GTP-binding proteins. Here we report the crystal structure determined at 2.0 A resolution of TGase 3 in complex with GMP to elucidate the structural features required for nucleotide recognition. Binding affinities for various nucleotides were found by fluorescence displacement to be as follows: guanosine 5'-3-O-(thio)triphosphate (GTPgammaS) (0.4 microm), GTP (0.6 microm), GDP (1.0 microm), GMP (0.4 microm), and ATP (28.0 microm). Furthermore, we found that GMP binds as a reversible, noncompetitive inhibitor of TGase 3 transamidation activity, similar to GTPgammaS and GDP. A genetic algorithm similarity program (GASP) approach (virtual ligand screening) identified three compounds from the Lead Quest trade mark data base (Tripos Inc.) based on superimposition of GTPgammaS, GDP, and GMP guanine nucleotides from our crystal structures to generate the minimum align flexible fragment. These three were nucleotide analogs without a phosphate group containing the minimal binding motif for TGase 3 that includes a nucleoside recognition groove. Binding affinities were measured as follows: TP349915 (K(d) = 4.1 microm), TP395289 (K(d) = 38.5 microm), TP394305 (K(d) = 1.0 mm). Remarkably, these compounds do not inhibit but instead activate TGase 3 transamidation by about 10-fold. These results suggest that the nucleotide binding pocket in TGase 3 may be exploited to either enhance or inhibit the enzymatic activity as required for different therapeutic approaches.