Warning
This document describes an early prototype of the LSST Alert Generation system. As of mid-2019, much of the material presented here is obsolete. Please refer to the Science Pipelines documentation for current information.
1 Overview¶
Ultimately, LSST Alert Production (AP) will need to perform all steps of the Level 1 pipeline automatically on each new raw image from the telescope. Details are in the LSST Data Management Science Pipelines Design (LDM-151 [2]) and the LSST Data Products Definition Document (DPDD [1]). Briefly summarized below and in Figure 1, the first portion of these steps are:
- Instrument Signature Removal (ISR), which includes crosstalk and linearity corrections, bad pixel masking, bias subtraction, and flat field division,
- Point Source Function (PSF) and background characterization, which includes source detection and measurement,
- Photometric and World Coordinate System (WCS) calibration, and
- Image Differencing with an appropriate template (which may need constructing) to create Difference Image Analysis (DIA) source catalogs.
A number of packages in the LSST Software Stack have been developed to perform portions of one or more of these four tasks, but they are not yet linked together in an automated way. This Prototype AP Pipeline is the first step toward the goal of an automatic Level 1 Pipeline that can process a set of raw images through each of these steps and verify that the output is what we expect.
To achieve this, there are two main python modules under development:
ap_pipe and ap_verify.
The former, described here, is responsible for running the Prototype Pipeline. The latter
uses ap_pipe
to verify the output.
2 Dataset¶
We use images from the High Cadence Transient Survey (HiTS) [3], a three-year survey from 2013–2015 which used the Dark Energy Camera (DECam) on the 4 m Blanco telescope at Cerro Tololo Interamerican Observatory (CTIO). The primary science goal of this survey is to detect and follow up optical transients that have characteristic timescales from hours to days, with a particular interest in catching early supernovae. The distribution of the HiTS survey fields in the sky is shown in Figure 2.
We select HiTS fields Blind15A_26
, Blind15A_40
, and Blind15A_42
, which
are repeat observations of the same region of sky as Blind14A_04
, Blind14A_10
, and Blind14A_09
fields, respectively. Each field has 34–36 individual visits over about a month, and each DECam image
consists of 62 CCDs (however, two of these are non-functional).
Most of the visits are in the g-band, some are in the r-band, and a few are in i.
All of these data are in the git-lfs repo ap_verify_hits2015
in raw format, along with the corresponding DECam Master Calibration files and camera defect files.
The data also exist on the lsst-dev
server at /datasets/decam/_internal/raw/hits
with the
calibration files at /dataset/decam/_internal/calib/cpHits
and the camera defects at
/dataset/decam/_internal/calib/bpmDes
. The raw images and calibration files were originally obtained
from the NOAO Science Archive by searching for PI Förster.
3 Tutorial¶
This tutorial walks a user through using ap_pipe
to run four main processing steps
with the LSST Stack on a portion of the Dataset described above:
ingestImagesDecam.py
, from theobs_decam
package but drawing heavily onpipe_tasks
,ingestCalibs.py
, from thepipe_tasks
package,processCcd.py
, from thepipe_tasks
package, andimageDifference.py
, also from thepipe_tasks
package.
The prerequisites for running ap_pipe
are:
- The LSST Science Pipelines
stack with the
lsst_apps
packages (a subset oflsst_distrib
) - The
obs_decam
andap_pipe
packages - A clone of the
ap_verify_hits2015
dataset
You may set all of this up as follows:
setup lsst_apps
# If using the lsst-dev server, lsst_apps may be setup using a centrally
# available weekly build, but the following packages are not included
git clone https://github.com/lsst/obs_decam.git
git clone https://github.com/lsst-dm/ap_pipe.git
setup -k -r obs_decam
setup -k -r ap_pipe
git clone https://github.com/lsst/ap_verify_hits2015.git
Once you are ready, run ap_pipe
from the command line. You must point to the dataset
with the -d
flag, a desired output location on disk with the -o
flag, and provide
a valid visit and ccdnum dataId string with the -i
flag.
python ap_pipe/bin.src/ap_pipe.py -d ap_verify_hits2015/ -o output_dir -i "visit=410985 ccdnum=25"
Note
At present (DM-11390), the template used for
difference imaging is hard-wired to visit 410929 and ccdnum 25.
This is a single CCD only of one of the Blind15A_40
visits. If you would like to use
a different template, you must manually set this in the source code
(ap_pipe/python/lsst/ap/pipe/ap_pipe.py
, in the function runPipelineAlone
).
This functionality will be improved when we switch from using a visit as a template to using coadds
by default (see DM-11422).
This will run all four steps of the pipeline in order. Each step will create a new subdirectory
in output_dir
: one for ingested images, one for ingested calibration products, one for
processed images, and one for difference imaging.
4 Results¶
The difference image and DIA Source catalog for a test image are available
on the lsst-dev
server at /project/mrawls/prototype_ap/zomg7/diffim/deepDiff/v410985/
.
A screenshot of the calibrated exposure (calexp) and difference image is shown
in Figure 3.
Future work will use coadded 2014 visits of the three HiTS fields described in Dataset
as templates and the corresponding 2015 visits as science.
This pipeline is a core component of the
AP Minimum Viable System
(note this link is private to LSST team members only).
The goal is to run and verify each component of LSST image processing as we build a
fully functional ap_verify
system.
The ap_pipe
pipeline will also be available to users who wish to process any DECam dataset independently.
5 References¶
[1] | [LSE-163]. Mario Jurić and others. LSST Data Products Definition Document. 2017. URL, https://ls.st/LSE-163. |
[2] | (1, 2) [LDM-151]. John D. Swinbank and others. Data Management Science Pipelines Design. 2017. URL, https://ls.st/LDM-151. |
[3] | (1, 2) F. Förster, J. C. Maureira, J. San Mart’ın, M. Hamuy, J. Mart’ınez, P. Huijse, G. Cabrera, L. Galbany, T. de Jaeger, S. González\ndash Gaitán, J. P. Anderson, H. Kunkarayakti, G. Pignata, F. Bufano, J. Litt’ın, F. Olivares, G. Medina, R. C. Smith, A. K. Vivas, P. A. Estévez, R. Muñoz, and E. Vera. The High Cadence Transient Survey (HITS). I. Survey Design and Supernova Shock Breakout Constraints. \apj , 832:155, December 2016. arXiv:1609.03567, doi:10.3847/0004-637X/832/2/155. |