The intensity of backscattered ultrasound signal from heart muscle is known to be related to the angle between cardiac fibers and the insonification direction. In this work, a GPU-based method of simulating three-dimensional (3D) echocardiographic images from an empirically derived angle-to-backscatter relationship is developed and validated. Images of a rotating fiber phantom are simulated, and it is validated that the angle-to-backscatter relationship is accurately reflected by the simulated envelope data. In a second experiment, echocardiography images are simulated from a diffusion tensor magnetic resonance imaging (DT-MRI) volume of a canine heart to demonstrate that the method produces view-dependent speckle. 3D volumes of a parametrically generated ideal left ventricle phantom are also simulated and processed into fiber orientation maps using the underlying quantitative parameters. Images are simulated based on the characteristics of a 35-by-32 two-dimensional (2D) matrix array probe and a clinical one-dimensional (1D) phased array probe. The processed fiber volumes exhibit good agreement with the virtual phantom’s ground truth, having an average acute angle error (AAE) of less than 10 degrees for both probes. The simulation method is fast and opens a new approach for ultrasound fiber imaging.