Breast tomosynthesis alleviates the obscuring effects of overlapping breast tissue overlap in the diagnosis of breast cancer in screening mammography. In breast tomosynthesis, multiple projection images are acquired when the x-ray tube travels within a limited angular range, typically < 50°. The projection images are reconstructed into tomographic slices that are parallel to the detector surface. Factors affecting the image quality of breast tomosynthesis include detector performance, acquisition geometry and reconstruction method. To investigate the impact of each factor, we built a three dimensional linear system model for breast tomsynthesis, validated it using experimental measurements of linear system parameters including noise power spectrum (NPS) and presampling modulation transfer function (MTF). Experiments were performed on a prototype breast tomosynthesis system equipped with an amorphous selenium (a-Se) digital mammography detector. Filtered backprojection (FBP) method was used for image reconstruction. Due to the limited angular range, the reconstructed 3D volume (512x512x40) had an anisotropic voxel size (0.085x0.085x1mm3). Scatter free uniform images were acquired for NPS analysis and an edge phantom was imaged for MTF measurements. It was found that the model agreed well with measurement, hence can be used to predict the imaging performance with different parameters and lead to optimization of breast tomosynthesis.