.. _getting_started: Getting Started =============== .. contents:: :depth: 2 :local: Installation ------------ To install PICOcode, clone the git repository: :: git clone https://github.com/picoexperiment/PICOcode.git This downloads the repository as PICOcode under your current working directory and configures the copy for Git tracking. This clone can be made on your own computer or in your $HOME directory on a shared system like ComputeCanada (or both). Directions for each case follow. Creating a virtual environment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ It is common practice to create a virtual environment for python and install packages within it. This has several advantages, including keeping separate sets of installed packages in different environments, and saving you a huge headache later if something goes wrong with your python installation, since you can just delete the folder which contains the environment instead of having to purge and reinstall python. To create a virtual environment, first download the :code:`virtualenv` package by calling .. code:: none pip install virtualenv or :code:`pip3 install virtualenv` if you have python 2 and 3 installed, e.g. on some versions of Ubuntu. Then, create a virtual evironment by calling .. code:: none python -m virtualenv where :code:`` is the name of a new folder in which the environment will be created. Then, activate it by calling :code:`source /bin/activate` on Unix/Mac OS, or :code:`\Scripts\activate.bat` on Windows from the command prompt. On your own computer ^^^^^^^^^^^^^^^^^^^^ When the :code:`git clone` download is complete you can install PICOcode and its dependencies with: :: pip install -e PICOcode --user The optional :code:`--user` flag installs the packages in :code:`~/.local/lib/`; you may skip this if using a virtual environment. If you'd prefer a system-wide installation, you can run :code:`pip` with root/admin privileges (e.g. prepend :code:`sudo`). The optional :code:`-e` flag installs the package in editable mode so that edits to the local code are automatically linked to the user's python environment, and is recommended if you plan on developing. On ComputeCanada ^^^^^^^^^^^^^^^^ If you don't yet have a ComputeCanada (CC) account linked to PICO, see `docdb:4287 `__ for instructions, including creating a python virtual environment. When you're able to :code:`ssh` to graham.computecanada.ca you'll find the PICO-60 and PICO-40L raw data grouped by runID under the PICO project directory: :code:`~/projects/rrg-kenclark/pico/30l-16-data/` and :code:`~/projects/rrg-kenclark/pico/40l-19-data/`. Detailed instructions for setting up your Python environment on CC can be found at `the CC Python wiki page `__, but it should be sufficient to run the following: :: module load python/3.7.7 scipy-stack/2020a This loads the `CC-provided Lmod modules `__ for Python 3.7.7 and most of the packages required to use PICOcode into your current shell. This can be added to your :code:`~/.bashrc` to have it loaded automatically on each login. Keep in mind that non-interactive bash shells don't source this file, so you'll likely need to add this line to any CC bash scripts you write. Finally, install PICOcode by running the following in the directory where you cloned the GitHub repository: :: pip install -e PICOcode At this point all PICOcode requirements should be satisfied, and the package should be ready to use. Initial Setup ^^^^^^^^^^^^^ There are two files which may require some setup after installing PICOcode. PICOcode.conf ############# The first time that PICOcode is imported, a config file is created in the user's home directory depending on the system: +--------------------+-----------------------------------------------------+ | Linux/Mac OS | :code:`$HOME/.PICOcode.conf` | +--------------------+-----------------------------------------------------+ | Windows | :code:`C:\Users\\Documents\.PICOcode.conf` | +--------------------+-----------------------------------------------------+ This config file is structured as a `JSON `__ file containing two entries: - `ARCHIVE_DIR`: the directory which contains raw PICO data, either in directories or as archives, in subfolders denoted by a *data_series*, i.e. raw data exists as :code:`//` or :code:`//.tar` - `SCRATCH_DIR`, which contains extracted data if the runs are archived. By default these point to the relevant locations on the Graham node of Compute Canada, which PICO uses for data processing. If you are using PICOcode on your own computer, you should edit this file to point to convenient directories. librefprop ########## The `NIST REFPROP `__ program is used to compute various fluid properties, which is mainly used to calculate the Seitz threshold. The use of :mod:`~PICOcode.REFPROP` (and its python wrapper `ctREFPROP `__) depend on a file called librefprop, the format of which depends on the system in use. Several versions have been compiled and are provided in :code:`PICOcode/REFPROP/lib`. When first loaded, PICOcode will try each of these pre-compiled librefprop files until one works; if none of them do, it will throw an error. If this occurs, you may have to compile your own version (see instructions `here `__). Please contact Colin if you are having issues with this. Accessing PICO Data with PICOcode --------------------------------- PICO data comes in two forms: raw and reconstructed (a.k.a. recon). Raw data are generally time series of transducer data for a given run or event (e.g. pressure, temperature, acoustics, pictures), while recon data consists of processed raw data which is used to categorize events (e.g. pressure or Seitz threshold, AP to discriminate neutrons/alphas, Dytran to discriminate multiples). Raw and reconstructed data are accessed using the `Event `_ and `ReconFile `_ classes. The use of these are described below. The Event class ^^^^^^^^^^^^^^^ Note that the Event class is loaded to PICOcode's namespace upon importing PICOcode, so it may be called with: >>> PICOcode.Event(...). This class is used to access raw data for individual PICO events. PICO data is stored with one directory per run, and one subdirectory per event (starting at 0, with up to 100 events per run), with some files related to run conditions stored in the run folder: .. code:: none ├── .txt ├── DAQ30l_Setup.xml ├── DAQversion.txt ├── RunParameters.txt ├── 0 │ ├── Event.txt │ ├── fastDAQ_0.bin │ ├── fastDAQ_0_cal.txt │ ├── PLClog.txt │ ├── slowDAQ_0.txt │ ├── temperature.txt │ └── Images │ ├── cam0_image30.png │ ├── cam0_image31.png │ ├── ... │ ├── cam0_image70.png │ ├── cam1_image30.png │ ├── ... │ └── cam3_image70.png ├── 1 │ └── ... └── ... PICO-40L, PICO-60, and some PICO-2L data are curently stored as archives on Compute Canada (CC) at: +--------------------+----------------+-----------------------------------+ | Experiment Name | Data Series | Location on CC | +====================+================+===================================+ | PICO-40L | 40l-19-data | /project/6007972/pico/40l-19-data | +--------------------+----------------+-----------------------------------+ | PICO-60 | 30l-16-data | /project/6007972/pico/30l-16-data | +--------------------+----------------+-----------------------------------+ | PICO-2L | 2l-16-data | /project/6007972/pico/2l-16-data | +--------------------+----------------+-----------------------------------+ The Event class is called by passing the full path to the run as the first argument, followed by the event number, followed by the name of the data to load as strings (either as a list, or as comma-separated values passed to \*args). The requested data may include any or all of the following: =============== ============================================== ======================================================= Load option File location Description of data =============== ============================================== ======================================================= event <*run*>/<*event*>/Event.txt When the event began, trigger info, pressure setpoint, etc. fastDAQ <*run*>/<*event*>/fastDAQ_0.bin Data collected by the fastDAQ system. <*run*>/<*event*>/fastDAQ_1.bin Includes signals from piezos, Dytrans. <*run*>/<*event*>/fastDAQ_0_cal.txt <*run*>/<*event*>/fastDAQ_1_cal.txt slowDAQ <*run*>/<*event*>/slowDAQ_0.txt Data collected by the pressure cart PLC. Includes signals from pressure transducers, position transducers, valve states, DIO, etc. Similar data to *PLC*, but lower level and higher timing resolution. temperature <*run*>/<*event*>/temperature.txt Data collected by the temperature PLC. Includes all RTDs, chiller setpoints, chiller loop flow data, humidity sensors. PLC <*run*>/<*event*>/PLClog.txt Data collected over modbus from pressure cart. Lower timing resolution than *slowDAQ*, and contains some other info, e.g. PID values, sensor rms, etc. DAQsettings <*run*>/DAQ30l_Setup.xml XML file containing run info from the DAQ VI. includes info such as setpoint, time spent compressed between events, pressure scan info (if used), etc. rundata <*run*>/<*run*>.txt Text file containing all info from each events' "Event.txt" file. Same info as the *event* option, but for all events in the run. camdata -- Currently Unimplemented. =============== ============================================== ======================================================= Once the Event class has been loaded with one or more options, they are accessed via Event.<*option*>.<*parameter*>. For example, to access the data for PT4: :: >>> event = PICOcode.Event("20200928_0", 24, 'slowDAQ') >>> PT4 = event.slowDAQ.PT4 If working on a Graham node (a Compute Canada resource), or if you have altered your `PICOcode.conf` file appropriately, the path to the run directory may be substituted by the run ID only, and the `Event` class will search the locations contained in `PICOcode.conf`. Example usage of the Event class ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ First, import PICOcode. >>> import PICOcode In this example, load fastDAQ, slowDAQ, and PLC data for the PICO-40L run 20200721_0 event 22. >>> event = PICOcode.Event("20200721_0", 22, "fastDAQ", "slowDAQ", "PLC") Loading fastDAQ Loading slowDAQ Loading PLC Load options may alternatively be passed as a list (for compatibility with old code). Use verbose=False to suppress info about loading data: >>> event = PICOcode.Event("20200721_0", 22, ["fastDAQ", "slowDAQ", "PLC"], verbose=False) If we wanted to plot the pressure of the freon for this event versus time, we can use `matplotlib`: >>> import matplotlib.pyplot as plt >>> plt.plot( event.slowDAQ.elapsed_time, event.slowDAQ.PT4 ) >>> plt.show() Which would produce the following plot (with some added elements): .. image:: images/Example_Plotting_PT4.png :width: 300px :align: center The ReconFile class ^^^^^^^^^^^^^^^^^^^ Note that the ReconFile class is loaded to PICOcode's namespace upon importing PICOcode, so it may be called with: >>> PICOcode.ReconFile(...). This class is used to access reconstructed PICO data, i.e. data that has been processed from its "raw" form. These data are stored in "recon files" located at :code:`/project/6007972/pico/recon///output/`, where :code:`` is either :code:`current` or :code:`devel`, and :code:`` specifies the run series, e.g. :code:`40l-19`. Within the recon output directory, each processed run has an associated recon folder containing its processed data. In addition, concatenated files exist in the output directory with data from all runs. The current list of processed files are: ================== =============== ========================================= File Prefix Recon type Data description ================== =============== ========================================= merged_all 2 Merged file containing data from all files. merged_lvl1_xyz 2 Merged file with all data except XYZLookup abub3hs 8 Pixel coordinates for bubbles in each event. Dytran 1 Dytran info, t0, and fit parameters. FastDAQ 1 Piezo info, t0, power, and AP parameters. History 1 Pressure and temperature data at trigger, Seitz threshold, alarm conditions. xyz_L1 2 Level one 3D reconstruction. XYZLookup 2 Level two 3D reconstruction. ================== =============== ========================================= If the recon file being loaded is for a run (and it is located in `output//`), then the file prefix is appended with `_.txt`. If the file is the concatenated file (located in `output/`), then the prefix is instead appended with `_all.txt`. All data from recon files are loaded into :mod:`numpy` arrays. This is useful for making cuts; see the example below for how to do so. Example usage of the ReconFile class ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ First, import PICOcode >>> import PICOcode An example of loading the `merged_all` file from the output directory in the `current` branch, then printing the freon pressure at the time of the trigger: >>> recon = PICOcode.ReconFile("/project/6007972/pico/recon/current/40l-19/output/merged_all_all.txt") >>> print(recon.PT4) [45.95491 45.01076 52.85783 ... 56.51385 56.51385 45.03973] >>> print(type(recon.PT4)) Making a cut to scan for events at 35 psi, and which triggered within 1 PSI of the intended setpoint, then printing them one per line: >>> cut = (recon.pset==35) & ( abs(recon.pset - recon.PT4) <= 1 ) >>> for run, ev in zip(recon.run[cut], recon.ev[cut]): print(run, ev) 20200107_3 0 20200107_3 1 20200107_3 2 20200107_3 3 20200107_3 4 20200107_3 5 20200109_0 0 ... Update the cut to only include data after run 20200713_7 (this is the first run in which four cameras were recording data). Load the level two reconstruction, and plot X versus Y for events from the previous cut in which the bubble nucleated at least 5 mm from the wall (note this is ignoring the jar dome): >>> cut = cut & (recon.run>"20200713_7") >>> import matplotlib.pyplot as plt >>> xyz = PICOcode.ReconFile("/project/6007972/pico/recon/current/40l-19/output/XYZLookup_all.txt") >>> R = 145 # Outer jar radius, in mm >>> for run, ev in zip(recon.run[cut], recon.ev[cut]): ... xyzcut = (xyz.run==run) & (xyz.ev==ev) ... if xyz.cR2[cut]**0.5 < R-5: plt.plot( xyz.cX[cut], xyz.cY[cut], 'ob' ) The above example would produce the following plot (with some added decoration): .. image:: images/Example_ReconFile.png :width: 300px :align: center Note that `XYZLookup_all.txt` should be merged into `merged_all_all.txt`, but this is not currently possible. In the future (when it is merged), the above example will not be useful.