NOBUGS 2016 Proceedings Realistic simulations of experiments at large scale photon facilities, such as optical laser laboratories, synchrotrons, and Free Electron Lasers, are of vital importance for the successful preparation, execution, and analysis of these experiments investigating ever more complex physical systems, e.g. biomolecules, complex materials, and ultrashort lived states of highly excited matter. Traditional photon science modeling takes into account only isolated aspects of an experiment, such as the beam propagation, the photon-matter interaction, or the scattering process, making idealized assumptions about the remaining parts, e.g. the source spectrum, temporal structure and coherence properties of the photon beam, or the detector response. In SIMEX, we have implemented a platform for complete start-to-end simulations, following the radiation from the source, through the beam transport optics to the sample or target under investigation, its interaction with and scattering from the sample and its registration in a photon detector, including a realistic model of the detector response to the radiation. Data analysis tools can be hooked up to the modeling pipeline easily. This allows researchers and facility operators to simulate their experiments and instruments in real life scenarios, identify promising and unattainable regions of the parameter space and ultimately make better use of expensive beamtime. Our software consists of a generic backbone defining the user and data interfaces to Calculators which are responsible for the simulation of the segments in the virtual beamline. A number of specific Calculators for the photon source, photon propagation, photon-matter interaction, photon scattering, photon detection, and photon data analysis are pre-installed. Further contributed Calculators can easily be integrated by inheriting from the abstract interfaces and providing only a few, well defined interface methods. A common data format description facilitates the data exchange among simulation codes. In this paper, we describe the general structure and implementation of the SIMEX software and discuss a number of applications: Modeling of single particle imaging at the European X-Ray Free Electron Laser (XFEL), a pump-and probe experiment with ultrashort pulsed optical laser excitation of a metal foil and subsequent probing by small angle scattering with coherent XFEL radiation, and a long pulse optical laser shock compression experiment probed by synchrotron radiation.