Using omniplate to analysis data from plate readers¶

The software corrects fluorescence for autofluorescence, calculates the fluorescence per cell, and estimates the growth rate and other growth characteristics.

We recommend running ipython via a jupyter notebook or a terminal and then importing omniplate.

Getting started¶

Loading the software
Getting help

How the data is stored¶

The dataframes
Working with a subset of the data
Restricting the range of time to be analysed

Checking the data is as expected¶

Checking the data
Checking the contents of the wells
Ignoring wells

Analysing OD data¶

Analysing OD data: correcting for non-linearities and for the media
Analysing OD data: plotting
Analysing OD data: estimating growth rates
Speeding up and trouble shooting estimating growth rates
Finding the local maximum growth rate

Analysing fluorescence data¶

Correcting autofluorescence: GFP
Correcting autofluorescence: mCherry
Checking which corrections have been performed
Estimating the time-derivative of the fluorescence

Specialising to mid-log, or exponential, phase¶

Quantifying the behaviour during mid-log growth

Making life easier¶

Saving figures
Extracting numerical values from a column
Getting a smaller dataframe for plotting directly

Exporting and importing the dataframes¶

Exporting and importing the dataframes

An example of processing the data to create a new column¶

Adding new processed data

Analysing multiple data sets simultaneously¶

Loading and processing more than one data set
Renaming conditions and strains
Averaging over experiments
Importing processed data for more than one experiment

We first import some standard packages, although these packages are only necessary if you wish to develop additional analysis.

In [1]:
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
%matplotlib inline
import seaborn as sns
# makes figures look better in Jupyter
sns.set_context('talk')

Loading the software¶

In [2]:
%run import_local # ignore this command
import omniplate as om

There are two ways of starting the software.

You can start without choosing a data set. In the following, only the working directory is specifed and the contents of that directory are displayed:

In [3]:
p= om.platereader(wdir= 'data')

Working directory is /Users/pswain/Dropbox/scripts/ompkg/data.
Files available are: 
---
{0: 'ExampleData.tsv',
 1: 'ExampleData.xlsx',
 2: 'ExampleDataContents.xlsx',
 3: 'ExampleData_r.tsv',
 4: 'ExampleData_s.tsv',
 5: 'ExampleData_sc.tsv',
 6: 'Glu.xlsx',
 7: 'GluContents.xlsx',
 8: 'GluGal.xlsx',
 9: 'GluGalContents.xlsx',
 10: 'GluGal_r.tsv',
 11: 'GluGal_s.tsv',
 12: 'GluGal_sc.tsv',
 13: 'Glu_r.tsv',
 14: 'Glu_s.tsv',
 15: 'Glu_sc.tsv',
 16: 'HxtA.xlsx',
 17: 'HxtAContents.xlsx',
 18: 'HxtB.xlsx',
 19: 'HxtBContents.xlsx',
 20: 'Sunrise.xlsx',
 21: 'SunriseContents.xlsx'}

A data set generated by the plate reader can then be loaded.

Two files are needed: the first file has the raw data, either produced by the plate reader - at the moment a data file from a Tecan is expected, but we are happy to add other plate readers, or you can use data already parsed into columns in a tsv or csv file; and the second file describes the contents of the wells of the plate.

In [4]:
p.load('ExampleData.xlsx', 'ExampleDataContents.xlsx')

Loading ExampleData.xlsx

Experiment: ExampleData 
---
Conditions:
	 0.25% Mal
	 0.5% Mal
	 1% Mal
	 1.5% Mal
	 2% Mal
	 2% Raf
	 3% Mal
	 4% Mal
Strains:
	 Mal12:GFP
	 Mal12:mCherry
	 Mal12:mCherry,Gal10:GFP
	 Null
	 WT
Data types:
	 OD
	 GFP
	 AutoFL
	 mCherry
Ignored wells:
	 None


Warning: wells with no strains have been changed to "Null"
to avoid confusion with pandas.

Alternatively, you can load data from a tsv or csv file, with the format

time    well    OD      GFP     AutoFL  mCherry
0       0.0     A1      0.25     46.0    18.0    19.0
1       0.23    A1      0.27     45.0    17.0    17.0

etc., and where time is in hours. To do so, change the platereadertype:

In [5]:
p= om.platereader("ExampleData.tsv", "ExampleDataContents.xlsx", 
                  platereadertype="tidy", wdir= "data")

Loading ExampleData.tsv
Columns must be labelled 'time', 'well', 'OD', etc., and time must be in units of hours.

Experiment: ExampleData 
---
Conditions:
	 0.25% Mal
	 0.5% Mal
	 1% Mal
	 1.5% Mal
	 2% Mal
	 2% Raf
	 3% Mal
	 4% Mal
Strains:
	 Mal12:GFP
	 Mal12:mCherry
	 Mal12:mCherry,Gal10:GFP
	 Null
	 WT
Data types:
	 Unnamed: 0
	 OD
	 GFP
	 AutoFL
	 mCherry
Ignored wells:
	 None


Warning: wells with no strains have been changed to "Null"
to avoid confusion with pandas.

You can also change the datasheet for the data,

p= om.platereader("ExampleData.xlsx", "ExampleContents.xlsx", dsheetnumber= 1)

and the working directory,

p.changewdir("newdata")

Getting help¶

To open this webpage, use

p.webhelp()

Typing

help(p)

gives information on all the methods available in platereader.

For example,

help(p.correctauto)

gives information on a particular method.

You can use

p.info

to see the current status of the data processing;

p.log

to see a log of all previous processing steps and p.savelog() to save this log to a file;

p.attributes

to see all the attributes of p; and

p.corrections()

to see the corrections that have been applied.

The dataframes¶

The data are stored in three Pandas dataframes:

  • p.r: contains the data stored by well
  • p.s: contains time-series of processed data, which are created from all the relevant wells
  • p.sc: contains summary statistics, such as the maximum growth rate

You can see what columns are in these dataframes using, for example:

p.r.columns

Use .head() to show only the beginning of the dataframe:

p.r.head()

There is also one further dataframe that stores the contents of the wells: p.wellsdf.

In [6]:
p.r.head()
Out[6]:
Unnamed: 0 time well OD GFP AutoFL mCherry experiment condition strain
0 0 0.000000 A1 0.2555 46.0 18.0 19.0 ExampleData 2% Raf Mal12:GFP
1 1 0.232306 A1 0.2725 45.0 17.0 17.0 ExampleData 2% Raf Mal12:GFP
2 2 0.464583 A1 0.2767 46.0 19.0 18.0 ExampleData 2% Raf Mal12:GFP
3 3 0.696840 A1 0.2870 45.0 19.0 20.0 ExampleData 2% Raf Mal12:GFP
4 4 0.929111 A1 0.2939 47.0 20.0 20.0 ExampleData 2% Raf Mal12:GFP
In [7]:
p.s.head()
Out[7]:
experiment condition strain time Unnamed: 0_mean OD_mean GFP_mean AutoFL_mean mCherry_mean Unnamed: 0_err OD_err GFP_err AutoFL_err mCherry_err
0 ExampleData 0.25% Mal Mal12:GFP 0.000000 945.0 0.260733 45.000000 17.333333 18.666667 840.0 0.012772 1.00000 0.57735 2.081666
1 ExampleData 0.25% Mal Mal12:GFP 0.232306 946.0 0.264400 43.666667 17.000000 17.666667 840.0 0.011888 0.57735 0.00000 0.577350
2 ExampleData 0.25% Mal Mal12:GFP 0.464583 947.0 0.268533 45.000000 19.000000 18.666667 840.0 0.008025 1.00000 0.00000 1.154701
3 ExampleData 0.25% Mal Mal12:GFP 0.696840 948.0 0.275833 44.333333 19.000000 18.666667 840.0 0.009900 0.57735 0.00000 1.154701
4 ExampleData 0.25% Mal Mal12:GFP 0.929111 949.0 0.279233 45.000000 19.333333 19.666667 840.0 0.009209 0.00000 0.57735 0.577350
In [8]:
p.sc.head()
Out[8]:
experiment strain condition OD_corrected Unnamed: 0_corrected_for_media OD_corrected_for_media GFP_corrected_for_media AutoFL_corrected_for_media mCherry_corrected_for_media Unnamed: 0_corrected_for_autofluorescence GFP_corrected_for_autofluorescence mCherry_corrected_for_autofluorescence
0 ExampleData Mal12:GFP 0.25% Mal False False False False False False False False False
1 ExampleData Mal12:GFP 0.5% Mal False False False False False False False False False
2 ExampleData Mal12:GFP 1% Mal False False False False False False False False False
3 ExampleData Mal12:GFP 1.5% Mal False False False False False False False False False
4 ExampleData Mal12:GFP 2% Mal False False False False False False False False False

Working with a subset of the data¶

For almost all of platereader's methods you can work with a subset of data.

For example,

  • you can include particular conditions by specifying
conditions = ['2% Glu', '1% Glu']
  • you can include all conditions containing the word Glu, by specifying
conditionincludes= 'Glu'
  • you can exclude all conditions containing either the words Glu or Gal, by specifying
conditionexcludes= ['Glu', 'Gal']

You can at the same time set experiments, experimentincludes, and experimentexcludes and strains, strainincludes, and strainexcludes.

Restricting the range of time to be analysed¶

Data at the beginning or at the end of the experiment may be corrupted. You can ignore such data using, for example

p.restricttime(tmin= 2, tmax= 15)

or

p.restricttime(tmin= 2)

Although this command will remove data from the p.s dataframe, it will not remove data from p.r and so could potentially be reversed.

Checking the data¶

You can display the data by the wells in the plate. In this example, columns 11 and 12 contain only media and no cells.

In [9]:
p.plot(y= 'OD', plate= True)
p.plot(y= 'GFP', plate= True)

A dataframe is created too for the contents of the wells:

Checking the contents of wells¶

You can search the p.wellsdf dataframe for experiments, conditions, and strains and for wells:

In [10]:
p.showwells(strains= 'Mal12:GFP', conditions= '1% Mal')

 experiment condition    strain well
ExampleData    1% Mal Mal12:GFP   D1
ExampleData    1% Mal Mal12:GFP   D2
ExampleData    1% Mal Mal12:GFP   D3
In [11]:
p.contentsofwells(['A1', 'D1'])

A1
--
 experiment condition    strain
ExampleData    2% Raf Mal12:GFP

D1
--
 experiment condition    strain
ExampleData    1% Mal Mal12:GFP

With showwells, you can more easily see which strains are in which conditions for each experiment by setting concise =True:

In [12]:
p.showwells(concise= True, strainincludes= 'GFP', 
            sortby= 'strain')

 experiment condition                  strain
ExampleData    2% Raf               Mal12:GFP
ExampleData    3% Mal               Mal12:GFP
ExampleData    2% Mal               Mal12:GFP
ExampleData  1.5% Mal               Mal12:GFP
ExampleData    1% Mal               Mal12:GFP
ExampleData  0.5% Mal               Mal12:GFP
ExampleData 0.25% Mal               Mal12:GFP
ExampleData    4% Mal               Mal12:GFP
ExampleData    2% Mal Mal12:mCherry,Gal10:GFP
ExampleData    3% Mal Mal12:mCherry,Gal10:GFP
ExampleData    4% Mal Mal12:mCherry,Gal10:GFP
ExampleData    2% Raf Mal12:mCherry,Gal10:GFP
ExampleData  0.5% Mal Mal12:mCherry,Gal10:GFP
ExampleData    1% Mal Mal12:mCherry,Gal10:GFP
ExampleData  1.5% Mal Mal12:mCherry,Gal10:GFP
ExampleData 0.25% Mal Mal12:mCherry,Gal10:GFP

Ignoring wells¶

Wells that have substantial measurement error can be ignored. For example:

In [13]:
p.ignorewells(['B2', 'B3'])
p.plot(y= 'OD', plate= True)

You can reinstate wells, but this process will remove any corrections you have made to the raw data.

In [14]:
p.ignorewells(clearall= True)
p.plot(y= 'OD', plate= True)

Warning: all corrections and analysis to raw data have been lost. No wells have been ignored.

Analysing OD data: correcting for non-linearities and for the media¶

Correcting for the media¶

There is a signal from the media in which the cells are growing.

Use p.correctmedia() to correct for media effects.

In [15]:
p.correctmedia('OD')

ExampleData: Correcting OD for media in 0.25% Mal.
ExampleData: Correcting OD for media in 0.5% Mal.
ExampleData: Correcting OD for media in 1% Mal.
ExampleData: Correcting OD for media in 1.5% Mal.
ExampleData: Correcting OD for media in 2% Mal.
ExampleData: Correcting OD for media in 2% Raf.
ExampleData: Correcting OD for media in 3% Mal.
ExampleData: Correcting OD for media in 4% Mal.

Typically, there is a well for each type of media, but you can also use one medium to correct all the strains regardless of the media that they are actually growing in:

p.correctmedia('OD', commonmedia= '2% Glu')

or to correct only a subset of wells

p.correctmedia('OD', conditionincludes= 'Glu', conditionexcludes= ['NaCl', 'HP'], 
               commonmedia= '2% Glu')

which uses 2% Glu to correct all wells whose condition contains the word Glu but not the words NaCl and HP.

Correcting OD¶

Typically plate readers have a non-linear relationship between OD and cell number for sufficiently large ODs.

You can use your own calibration data or omniplate's default calibration data to correct for this non-linearity.

In [16]:
p.correctOD()

Fitting dilution data for OD correction for non-linearities.
Using default data.

The function used for this correction is plotted unless figs= False. Note that the corrected values (the relative cell density) are in arbitrary units.

You can change the calibration data used:

p.correctOD(ODfname= 'ODcorrection_Raffinose_Haploid.txt')

Analysing OD data: plotting¶

All plotting uses Seaborn's relplot function, and you can set most of the variables that Seaborn allows.

In [17]:
p.plot(y= 'OD', wells= True, strainincludes= 'Gal10:GFP', 
       conditionincludes= '5')

If you do not plot individual wells, errors are shown by shading using the standard deviation over all relevant wells.

In [18]:
p.plot(y= 'OD', conditionincludes= '0.25')
p.plot(y= 'OD', strainincludes= 'Gal10:GFP', 
       hue= 'condition', style= 'strain')

You can also use seaborn directly:

In [19]:
fg= sns.relplot(x= "time", y= "OD", data= p.r, kind= "line", 
                hue= "condition", col= "strain", errorbar="sd")

Analysing OD data: estimating growth rates¶

You can estimate (specific) growth rates using all replicates for each strain in each condition using a Gaussian process-based algorithm.

If the maximum growth rate is a local maximum, it is marked with a yellow circle.

Local maxima are identified by the prominence of the local peaks. If the peak is falsely identified, you can custom how the local maxima are detected by specifying the degree of prominence with the peakprominence parameter, which has a default value of 0.05.

If you would prefer not to have local maximum growth rates shown on the plots, set plotlocalmax= False.

In [20]:
p.getstats(strains="Mal12:GFP", conditionincludes= "0.25")

Fitting log_OD for ExampleData: Mal12:GFP in 0.25% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 7.352026e+02
hparam[0]= 5.539094e+02 [1.000000e-05, 1.000000e+05]
hparam[1]= 4.746352e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 4.001934e-04 [1.000000e-05, 1.000000e+02]

Three hyperparameters are estimated during the fitting, and you can change the lower and upper bounds for these hyperparameters. The bounds are specifed in log10 space.

For example, to change the bounds on the measurement noise (parameter 2) to $10^{-2}$ and $10^{0}$ use:

In [21]:
p.getstats(strains="Mal12:GFP", conditionincludes= "0.25", 
           bd= {2: (-2,0)})

Fitting log_OD for ExampleData: Mal12:GFP in 0.25% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 4.013305e+02
hparam[0]= 1.033631e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 1.311187e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

You can specify the bounds on multiple parameters, such as parameters 0 and 1, using

p.getstats(strainincludes="Mal12:GFP", conditionincludes= "0.25", 
    bd= {0: (-2, 2), 1: (-4, -1)})

Running

p.getstats()

will fit all strains in all conditions.

Growth statistics are saved for the strains processed in p.sc.

For example, the maximum growth rate and its error are given by:

In [22]:
p.sc[["experiment", "condition", "strain", "max_gr", "max_gr_err"]].head()
Out[22]:
experiment condition strain max_gr max_gr_err
0 ExampleData 0.25% Mal Mal12:GFP 0.193041 0.006055
1 ExampleData 0.5% Mal Mal12:GFP NaN NaN
2 ExampleData 1% Mal Mal12:GFP NaN NaN
3 ExampleData 1.5% Mal Mal12:GFP NaN NaN
4 ExampleData 2% Mal Mal12:GFP NaN NaN

We can get the growth rate for all strains too:

In [23]:
p.getstats(bd= {2: (-2,0)}, figs=False)

Fitting log_OD for ExampleData: Mal12:GFP in 0.25% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 4.013271e+02
hparam[0]= 1.096367e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 1.321018e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:mCherry in 0.25% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 2.631610e+02
hparam[0]= 1.473057e+01 [1.000000e-05, 1.000000e+05]
hparam[1]= 2.241809e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:mCherry,Gal10:GFP in 0.25% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 3.962679e+02
hparam[0]= 3.173850e+04 [1.000000e-05, 1.000000e+05]
hparam[1]= 1.220296e+02 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: WT in 0.25% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 2.592092e+02
hparam[0]= 1.409977e+01 [1.000000e-05, 1.000000e+05]
hparam[1]= 2.171745e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:GFP in 0.5% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 3.994562e+02
hparam[0]= 8.870165e-01 [1.000000e-05, 1.000000e+05]
hparam[1]= 1.072733e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:mCherry in 0.5% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 2.633272e+02
hparam[0]= 7.085256e-01 [1.000000e-05, 1.000000e+05]
hparam[1]= 1.137790e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:mCherry,Gal10:GFP in 0.5% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 4.001977e+02
hparam[0]= 2.864966e+02 [1.000000e-05, 1.000000e+05]
hparam[1]= 4.376949e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: WT in 0.5% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 2.607085e+02
hparam[0]= 5.117231e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 1.842995e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:GFP in 1% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 4.013800e+02
hparam[0]= 9.535585e-01 [1.000000e-05, 1.000000e+05]
hparam[1]= 9.802573e+00 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:mCherry in 1% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 2.633676e+02
hparam[0]= 2.469157e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 1.479137e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:mCherry,Gal10:GFP in 1% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 4.062888e+02
hparam[0]= 1.146346e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 1.250512e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: WT in 1% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 2.626936e+02
hparam[0]= 6.598670e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 2.140143e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:GFP in 1.5% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 4.022905e+02
hparam[0]= 3.998852e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 1.651418e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:mCherry in 1.5% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 2.592242e+02
hparam[0]= 1.996183e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 1.605456e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:mCherry,Gal10:GFP in 1.5% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 4.065042e+02
hparam[0]= 1.350210e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 1.477886e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: WT in 1.5% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 2.624978e+02
hparam[0]= 6.738966e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 2.175107e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:GFP in 2% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 3.926601e+02
hparam[0]= 3.492261e+03 [1.000000e-05, 1.000000e+05]
hparam[1]= 8.173146e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:mCherry in 2% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 2.651517e+02
hparam[0]= 2.020937e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 1.667629e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:mCherry,Gal10:GFP in 2% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 4.058202e+02
hparam[0]= 1.066708e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 1.421462e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: WT in 2% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 2.631537e+02
hparam[0]= 3.807227e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 2.094081e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:GFP in 2% Raf
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 4.004294e+02
hparam[0]= 3.600991e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 2.373880e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:mCherry in 2% Raf
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 2.629441e+02
hparam[0]= 9.017364e+02 [1.000000e-05, 1.000000e+05]
hparam[1]= 9.996412e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:mCherry,Gal10:GFP in 2% Raf
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 4.066874e+02
hparam[0]= 4.265188e+02 [1.000000e-05, 1.000000e+05]
hparam[1]= 9.029565e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: WT in 2% Raf
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 2.645926e+02
hparam[0]= 1.436593e+01 [1.000000e-05, 1.000000e+05]
hparam[1]= 3.090361e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:GFP in 3% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 4.050808e+02
hparam[0]= 9.597429e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 2.459292e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:mCherry in 3% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 2.662962e+02
hparam[0]= 3.697249e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 2.233224e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:mCherry,Gal10:GFP in 3% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 4.083727e+02
hparam[0]= 2.928347e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 2.047192e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: WT in 3% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 2.637569e+02
hparam[0]= 1.174970e+01 [1.000000e-05, 1.000000e+05]
hparam[1]= 2.972114e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:GFP in 4% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 4.050363e+02
hparam[0]= 3.071393e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 1.944139e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:mCherry in 4% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 2.665344e+02
hparam[0]= 4.265246e+00 [1.000000e-05, 1.000000e+05]
hparam[1]= 2.393259e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: Mal12:mCherry,Gal10:GFP in 4% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 4.019600e+02
hparam[0]= 7.030613e+03 [1.000000e-05, 1.000000e+05]
hparam[1]= 1.456838e+02 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

Fitting log_OD for ExampleData: WT in 4% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 2.530088e+02
hparam[0]= 1.399786e+04 [1.000000e-05, 1.000000e+05]
hparam[1]= 1.257120e+02 [1.000000e-04, 1.000000e+04]
hparam[2]= 1.000000e-02 [1.000000e-02, 1.000000e+00]
Warning: hyperparameter 2 is at a lower bound.

In [24]:
p.plot(y= "gr", strains="Mal12:GFP", conditionincludes= "1", 
       hue= "condition", style="strain")

Speeding up and trouble shooting for estimating growth rates¶

The optimisation routine that fits the hyperparameters of the Gaussian process to OD data can be customised by changing options to getstats.

As well as changing the bounds on the hyperparameters with bd, useful options are:

  • cvfn : Determines the covariance function of the Gaussian process – either matern (default), sqexp (squared exponential), or nn (neural network).
  • noruns : Determines the number of fitting attempts with each having a randomly chosen initial guess for the hyperparameters (default is 5).
  • exitearly : If True (default), stops fitting once a fit is found rather than perform all noruns fits and take the best.
  • noinits : Determines the number of attempts to find a good initial guess for the hyperparameters before running the fitting (default is 100).
  • nosamples : The number of samples used to determine the statistics once a fit has been found (default is 100).
  • iskip : To speed up the fitting for large datasets by ignoring data points and so reducing the size of the dataset. For example, iskip= 2 means that only every second data point is used.
  • findareas : If False (default), the areas under the OD versus time and the growth rate versus time will not be calculated. Estimating the errors in these quantities is slow.

You can see the quality of the fit using

p.sc[['experiment', 'condition', 'strain', 'OD logmaxlike']]

where larger logmaxlike is better.

The bounds on hyperparameter 2 is a measure of how noisy you think your measurements are. Occasionally with the default bounds, we find that the growth rate fluctuates in small waves over time or has two peaks separated by a shallow trough. This behaviour is often because the software is fitting the noise in the data. Changing the bounds on hyperparameter 2, particularly increasing the lower bound so that the minimal level of noise is higher, usually stops these fluctuations so that the growth rate varies smoothly.

Finding the local maximum growth rate¶

Scipy's find_peaks is used to find the local maximum growth rate, which is useful if the highest growth rate is at t= 0.

You can specify properties that a peak must satisfy to be considered a maximum using the options of find_peaks.

First, try

p.getstats(showpeakproperties= True, width= 0, 
           prominence= 0)

which will display the width and prominence of the local peaks to get an idea of the values to set. These values are given in the number of x and y points and not in real units.

Once you have a minimum value of one of these parameters that a true local maximum should satisfy then use, for example,

p.getstats(width= 15)

so that all local maxima will have a width of at least 15 x units.

Note that these arguments for find_peaks must go at the end of your call to getstats.

Correcting autofluorescence: GFP¶

GFP is usually corrected for autofluorescence using the method of Lichten et al., which requires measurement of fluorescence of a reference strain both at the GFP wavelength and at a higher wavelength (called AutoFL).

First, we should correct fluorescence measurements for the fluorescence of the media.

In [25]:
p.correctmedia(["GFP", "AutoFL"])

ExampleData: Correcting GFP for media in 0.25% Mal.
ExampleData: Correcting GFP for media in 0.5% Mal.
ExampleData: Correcting GFP for media in 1% Mal.
ExampleData: Correcting GFP for media in 1.5% Mal.
ExampleData: Correcting GFP for media in 2% Mal.
ExampleData: Correcting GFP for media in 2% Raf.
ExampleData: Correcting GFP for media in 3% Mal.
ExampleData: Correcting GFP for media in 4% Mal.
ExampleData: Correcting AutoFL for media in 0.25% Mal.
ExampleData: Correcting AutoFL for media in 0.5% Mal.
ExampleData: Correcting AutoFL for media in 1% Mal.
ExampleData: Correcting AutoFL for media in 1.5% Mal.
ExampleData: Correcting AutoFL for media in 2% Mal.
ExampleData: Correcting AutoFL for media in 2% Raf.
ExampleData: Correcting AutoFL for media in 3% Mal.
ExampleData: Correcting AutoFL for media in 4% Mal.

Note that correctmedia can give negative values because of measurement errors. These negative values will be ignored by correctauto when called with two fluorescence measurements.

Alternatively, not correcting for media and running correctauto with only one fluorescence value (which automatically corrects for media) is another choice.

Second, we can run correctauto to correct for autofluorescence.

By setting useGPs to True or False, you can specify whether you would like to use Gaussian processes to estimate errors in the corrected fluorescence, which is the default method, or whether you would prefer to use the standard deviation over replicate wells, which will be faster but requires sufficent wells.

In [26]:
p.correctauto(["GFP", "AutoFL"], strains="Mal12:GFP", conditionincludes="1", 
              refstrain= "WT", figs=False)

Using WT as the reference.
Using two fluorescence wavelengths.
Correcting autofluorescence using GFP and AutoFL.
ExampleData: Processing reference strain WT for GFP in 1% Mal.

Fitting log_GFP for ExampleData: Mal12:GFP in 1% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
 Warning: optimisation of hyperparameters failed at run 1.
 Warning: optimisation of hyperparameters failed at run 2.
 Warning: optimisation of hyperparameters failed at run 3.
 Warning: optimisation of hyperparameters failed at run 4.
 Warning: optimisation of hyperparameters failed at run 5.
log(max likelihood)= 3.046955e+02
hparam[0]= 3.263413e+01 [1.000000e-05, 1.000000e+05]
hparam[1]= 5.397113e+00 [1.000000e-04, 1.000000e+04]
hparam[2]= 4.537550e-03 [1.000000e-05, 1.000000e+02]
ExampleData: Processing reference strain WT for GFP in 1.5% Mal.

Fitting log_GFP for ExampleData: Mal12:GFP in 1.5% Mal
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
 Warning: optimisation of hyperparameters failed at run 1.
 Warning: optimisation of hyperparameters failed at run 2.
 Warning: optimisation of hyperparameters failed at run 3.
 Warning: optimisation of hyperparameters failed at run 4.
log(max likelihood)= 2.864686e+02
hparam[0]= 2.692870e+01 [1.000000e-05, 1.000000e+05]
hparam[1]= 5.467345e+00 [1.000000e-04, 1.000000e+04]
hparam[2]= 5.350065e-03 [1.000000e-05, 1.000000e+02]

The results are stored in the p.s dataframe and can be plot by specifying c-GFPperODd. The corrected fluorescence for the wild-type strain, which should be zero, gives an indication of the size of the errors.

In [27]:
p.plot(y= "c-GFPperOD", strains="Mal12:GFP", conditionincludes="1",
       hue= "condition", style="strain")

Correcting autofluorescence: mCherry¶

mCherry is corrected for autofluorescence by using the fluorescence of a reference strain at the same OD as the strain of interest.

The code for correcting autofluorescence in this case also accounts for the fluorescence of the media, and the results should be the same whether or not the media is corrected.

You can again specify whether or not omniplate should use Gaussian processes.

In [28]:
p.correctauto("mCherry", conditionincludes= ["1"],  strainincludes= "mCherry",
              refstrain= "WT", useGPs=False)

Using WT as the reference.
Using one fluorescence wavelength.
Correcting autofluorescence using mCherry.
ExampleData: Processing reference strain WT for mCherry in 1% Mal.

ExampleData: Processing reference strain WT for mCherry in 1.5% Mal.
In [29]:
p.plot(y= "c-mCherryperOD", conditionincludes=["1"], strainincludes= "mCherry",
       hue= "strain", style= "condition", nonull= True)

Checking which corrections have been performed¶

You can find which corrections have been performed for a strain in a condition in an experiment (or for all strains in all conditions in all experiments):

In [30]:
p.corrections(conditionincludes= '1%', strainincludes= 'mCherry')
Out[30]:
10 18
experiment ExampleData ExampleData
strain Mal12:mCherry Mal12:mCherry,Gal10:GFP
condition 1% Mal 1% Mal
OD_corrected True True
Unnamed: 0_corrected_for_media False False
OD_corrected_for_media True True
GFP_corrected_for_media True True
AutoFL_corrected_for_media True True
mCherry_corrected_for_media False False
Unnamed: 0_corrected_for_autofluorescence False False
GFP_corrected_for_autofluorescence False False
mCherry_corrected_for_autofluorescence True True

Estimating the time-derivative of the fluorescence¶

You can also estimate the derivative of, for example, the fluorescence per cell. Here logs= False because we want the derivative of the fluorescence and not the derivative of the logarithm of the fluorescence:

In [31]:
p.getstats('c-mCherryperOD', conditions= ['1% Mal'], 
           strains= ['Mal12:mCherry'], logs= False,
          noruns= 5, exitearly= False)

Fitting c-mCherryperOD for ExampleData: Mal12:mCherry in 1% Mal
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= -9.895582e+02
hparam[0]= 3.696450e+03 [1.000000e-05, 1.000000e+05]
hparam[1]= 1.064858e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 2.647310e+01 [1.000000e-05, 1.000000e+02]

The results are stored in the p.s and p.sc dataframes:

In [32]:
p.s.query('condition == "1% Mal" and strain == "Mal12:mCherry"')[['time', 
                                                                  'd/dt_c-mCherryperOD']].head()
Out[32]:
time d/dt_c-mCherryperOD
1155 0.000000 5.212760
1156 0.232306 5.407086
1157 0.464583 5.595943
1158 0.696840 5.760226
1159 0.929111 5.895222

Quantifying the behaviour during mid-log growth¶

Using nunchaku (Huo & Swain), omniplate will automatically identify the segment of the time series where growth is exponential and calculate statistics of all variables in the s dataframe in this mid-log segment.

Black squares mark the segment of growth identified as mid-log.

In [33]:
p.getmidlog(conditions="1% Mal")

Finding mid-log growth for ExampleData : Mal12:GFP in 1% Mal

WARNING:root:Nunchaku: the best number of segments equals the largest candidate.

Finding mid-log growth for ExampleData : Mal12:mCherry in 1% Mal

getting evidence matrix:   0%|          | 0/103 [00:00<?, ?it/s]
getting model evidence:   0%|          | 0/4 [00:00<?, ?it/s]
WARNING:root:Nunchaku: the best number of segments equals the largest candidate.

getting internal boundaries:   0%|          | 0/3 [00:00<?, ?it/s]
Finding mid-log growth for ExampleData : Mal12:mCherry,Gal10:GFP in 1% Mal

WARNING:root:Nunchaku: the best number of segments equals the largest candidate.

Finding mid-log growth for ExampleData : WT in 1% Mal

getting evidence matrix:   0%|          | 0/103 [00:00<?, ?it/s]
getting model evidence:   0%|          | 0/4 [00:00<?, ?it/s]
WARNING:root:Nunchaku: the best number of segments equals the largest candidate.

getting internal boundaries:   0%|          | 0/3 [00:00<?, ?it/s]

All mid-log statistics are stored in the sc dataframe.

In [34]:
p.sc[p.sc.condition=="1% Mal"][["strain", "condition"] + [col for col in p.sc.columns if "midlog" in col]]
Out[34]:
strain condition mean_midlog_time mean_midlog_Unnamed: 0_mean mean_midlog_OD_mean mean_midlog_GFP_mean mean_midlog_AutoFL_mean mean_midlog_mCherry_mean mean_midlog_Unnamed: 0_err mean_midlog_OD_err ... max_midlog_c-mCherry max_midlog_c-mCherry_err max_midlog_c-mCherryperOD max_midlog_c-mCherryperOD_err max_midlog_smoothed_c-mCherryperOD max_midlog_smoothed_c-mCherryperOD_err max_midlog_d/dt_c-mCherryperOD max_midlog_d/dt_c-mCherryperOD_err max_midlog_d/dt_d/dt_c-mCherryperOD max_midlog_d/dt_d/dt_c-mCherryperOD_err
2 Mal12:GFP 1% Mal 3.948622 1172.000000 0.468057 177.024832 19.729921 16.789474 840.000000 0.009736 ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
10 Mal12:mCherry 1% Mal 4.064756 3272.500000 0.449530 18.764979 8.603785 42.900000 593.969696 0.005064 ... 42.396682 3.996731 64.302352 8.813568 58.268406 5.234895 6.137184 0.850426 2.393542 1.553519
18 Mal12:mCherry,Gal10:GFP 1% Mal 4.180891 5373.000000 0.446530 13.944644 6.584433 41.736842 840.000000 0.001900 ... 36.820418 5.714479 61.438330 10.961472 NaN NaN NaN NaN NaN NaN
26 Null 1% Mal NaN NaN NaN NaN NaN NaN NaN NaN ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
34 WT 1% Mal 4.916409 7476.166667 0.141013 13.257507 5.066549 17.750000 593.969696 0.001544 ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN

5 rows × 126 columns

You can display the results with plot.

In [35]:
p.plot(x="mean_midlog_GFP_mean", y="strain")

Saving figures¶

You can at any time save all figures that are currently displayed (but not those displayed inline in Jupyter):

p.savefigs()

which saves each figure as a separate page in a single PDF file.

Extracting numerical values from a column¶

It may be useful to extract the numerical values given in, for example, the name of an experiment or condition. Calling

p.addnumericcolumn('new column name', 'original column')

extracts any numbers from each entry in the original column, makes a new column in all the dataframes called 'new column name', and places these numbers in the appropriate entry in this new column.

For example:

In [36]:
p.addnumericcolumn('concentration', 'condition')
p.sc[['condition', 'strain', 'concentration']].head()
Out[36]:
condition strain concentration
0 0.25% Mal Mal12:GFP 0.25
1 0.5% Mal Mal12:GFP 0.50
2 1% Mal Mal12:GFP 1.00
3 1.5% Mal Mal12:GFP 1.50
4 2% Mal Mal12:GFP 2.00

It is also possible to specify which number in the column entry you would like using picknumber and you can find numbers next to a substring of interest using leftsplitstring and rightsplitstring.

Getting a smaller dataframe for plotting directly¶

You can get a subset of the data as a dataframe, from either the r (the raw time-series), s (the processed time-series), or sc (the summary statistics) dataframes, using, for example:

In [37]:
subdf= p.getdataframe('s', conditionincludes= 'Raf', strainincludes= 'Gal')
subdf.head()
Out[37]:
experiment condition strain time Unnamed: 0_mean OD_mean GFP_mean AutoFL_mean mCherry_mean Unnamed: 0_err ... c-mCherry_err c-mCherryperOD c-mCherryperOD_err smoothed_c-mCherryperOD smoothed_c-mCherryperOD_err d/dt_c-mCherryperOD d/dt_c-mCherryperOD_err d/dt_d/dt_c-mCherryperOD d/dt_d/dt_c-mCherryperOD_err concentration
2835 ExampleData 2% Raf Mal12:mCherry,Gal10:GFP 0.000000 5040.0 0.176736 6.561484 2.288807 25.333333 840.0 ... NaN NaN NaN NaN NaN NaN NaN NaN NaN 2.0
2836 ExampleData 2% Raf Mal12:mCherry,Gal10:GFP 0.232306 5041.0 0.183252 5.219432 2.265758 24.666667 840.0 ... NaN NaN NaN NaN NaN NaN NaN NaN NaN 2.0
2837 ExampleData 2% Raf Mal12:mCherry,Gal10:GFP 0.464583 5042.0 0.195068 7.876621 3.576935 24.000000 840.0 ... NaN NaN NaN NaN NaN NaN NaN NaN NaN 2.0
2838 ExampleData 2% Raf Mal12:mCherry,Gal10:GFP 0.696840 5043.0 0.204218 6.532866 2.888802 24.333333 840.0 ... NaN NaN NaN NaN NaN NaN NaN NaN NaN 2.0
2839 ExampleData 2% Raf Mal12:mCherry,Gal10:GFP 0.929111 5044.0 0.211668 8.188081 3.201234 26.333333 840.0 ... NaN NaN NaN NaN NaN NaN NaN NaN NaN 2.0

5 rows × 35 columns

Exporting and importing the dataframes¶

The three dataframes

p.r
p.s
p.sc

can all be saved as tsv (or json or csv) files. A logfile of the commands that you've used will also be saved.

In [38]:
p.exportdf()

Exported successfully.

The files can be imported similarly:

In [39]:
q= om.platereader(wdir= 'data', ls= False)
q.importdf('ExampleData')

Imported data/ExampleData_r.tsv
Imported data/ExampleData_s.tsv
Imported data/ExampleData_sc.tsv


Experiment: ExampleData 
---
Conditions:
	 0.25% Mal
	 0.5% Mal
	 1% Mal
	 1.5% Mal
	 2% Mal
	 2% Raf
	 3% Mal
	 4% Mal
Strains:
	 Mal12:GFP
	 Mal12:mCherry
	 Mal12:mCherry,Gal10:GFP
	 Null
	 WT
Data types:
	 Unnamed: 0
	 OD
	 GFP
	 AutoFL
	 mCherry
Ignored wells:
	 None

Adding new processed data¶

Here is a more complex example where we wish to plot the fluorescence at the time when the growth rate is maximum.

We first create a new field, GFP mean at max growth rate, in the dataframe p.sc.

In [40]:
s= 'Mal12:GFP'
# store results as an array of dictionaries to eventually convert into a dataframe
results= []
for e in p.allexperiments:
    for c in p.allconditions[e]:
        # find the time of maximum growth rate for the condition
        tm= p.sc.query('experiment == @e and condition == @c and strain == @s')['time_of_max_gr'].values[0]
        # take the relevant sub-dataframe for the condition
        df= p.s.query('experiment == @e and condition == @c and strain == @s')
        # find GFP mean at time tm
        i= np.argmin(np.abs(df['time'].values - tm))
        results.append({'GFP_mean_at_max_gr' : df['GFP_mean'][df.index[i]],
                     'experiment' : e, 'condition' : c, 'strain' : s})
# convert to dataframe
rdf= pd.DataFrame(results)
# add to existing dataframe by experiment, condition, and strain
p.sc= pd.merge(p.sc, rdf, how= 'outer')
In [41]:
# check results
p.sc[['experiment', 'condition', 'strain', 'local_max_gr', 'GFP_mean_at_max_gr']].head()
Out[41]:
experiment condition strain local_max_gr GFP_mean_at_max_gr
0 ExampleData 0.25% Mal Mal12:GFP 0.194106 64.626427
1 ExampleData 0.5% Mal Mal12:GFP 0.270241 103.901319
2 ExampleData 1% Mal Mal12:GFP 0.316688 151.594702
3 ExampleData 1.5% Mal Mal12:GFP 0.306435 178.638601
4 ExampleData 2% Mal Mal12:GFP 0.291507 182.950773
In [42]:
# plot results
p.plot(x= 'max_gr', y= 'GFP_mean_at_max_gr', hue= 'condition', 
       style= 'experiment', conditionincludes= 'Mal')

Loading and processing more than one data set¶

It is also possible to load more than one data set and simultaneously process the data.

In [43]:
p= om.platereader(['HxtA.xlsx', 'HxtB.xlsx'], ['HxtAContents.xlsx', 'HxtBContents.xlsx'], wdir= 'data')  

Loading HxtA.xlsx
Loading HxtB.xlsx

Experiment: HxtA 
---
Conditions:
	 0.05% Gal
	 1% Gal
	 2% Gal
	 2% Glu
Strains:
	 BY4741
	 Hxt1:GFP
	 Hxt2:GFP
	 Hxt3:GFP
	 Hxt4:GFP
	 Null
Data types:
	 OD
	 GFP_80Gain
	 AutoFL_80Gain
	 GFP_65Gain
	 AutoFL_65Gain
Ignored wells:
	 None

Experiment: HxtB 
---
Conditions:
	 0.05% Gal
	 1% Gal
	 2% Gal
	 2% Glu
Strains:
	 BY4741
	 Hxt5:GFP
	 Hxt6:GFP
	 Hxt7:GFP-L
	 Hxt7:GFP-N
	 Null
Data types:
	 OD
	 GFP_80Gain
	 AutoFL_80Gain
	 GFP_65Gain
	 AutoFL_65Gain
Ignored wells:
	 None


Warning: wells with no strains have been changed to "Null"
to avoid confusion with pandas.

By default, correctOD corrects all experiments for OD, but you can specify the experiment variable to specialise:

In [44]:
p.correctOD(figs= False)
p.correctmedia('OD')

Fitting dilution data for OD correction for non-linearities.
Using default data.
HxtA: Correcting OD for media in 0.05% Gal.
HxtA: Correcting OD for media in 1% Gal.
HxtA: Correcting OD for media in 2% Gal.
HxtA: Correcting OD for media in 2% Glu.
HxtB: Correcting OD for media in 0.05% Gal.
HxtB: Correcting OD for media in 1% Gal.
HxtB: Correcting OD for media in 2% Gal.
HxtB: Correcting OD for media in 2% Glu.
In [45]:
p.plot(y= 'OD', hue= 'condition', strains= 'BY4741', conditionincludes= 'Gal', 
       style= 'strain', size= 'experiment')

You can also run routines by experiment:

In [46]:
p.getstats(strainincludes= 'Hxt6', conditionincludes= 'Glu', 
           experiments= 'HxtB')  

Fitting log_OD for HxtB: Hxt6:GFP in 2% Glu
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
log(max likelihood)= 7.576059e+02
hparam[0]= 1.878684e+04 [1.000000e-05, 1.000000e+05]
hparam[1]= 7.658942e+01 [1.000000e-04, 1.000000e+04]
hparam[2]= 8.700447e-04 [1.000000e-05, 1.000000e+02]
In [47]:
p.plot(x= 'OD_mean', y= 'gr', strainincludes= 'Hxt6', conditionincludes= 'Glu', 
       experiments= 'HxtB')
In [48]:
p.correctauto('GFP_65Gain', strains= ['Hxt6:GFP'], refstrain= 'BY4741',  
              conditions= '2% Glu', experiments= 'HxtB', figs= False)    

Using BY4741 as the reference.
Using one fluorescence wavelength.
Correcting autofluorescence using GFP_65Gain.
HxtB: Processing reference strain BY4741 for GFP_65Gain in 2% Glu.

Fitting log_['GFP_65Gain'] for HxtB: Hxt6:GFP in 2% Glu
Taking natural logarithm of the data.
GP Warning: input data is not sorted. Sorting...
Using a (twice differentiable) Matern covariance function.
hparam[0] determines the amplitude of variation
hparam[1] determines the stiffness
hparam[2] determines the variance of the measurement error
 Warning: optimisation of hyperparameters failed at run 1.
 Warning: optimisation of hyperparameters failed at run 2.
 Warning: optimisation of hyperparameters failed at run 3.
 Warning: optimisation of hyperparameters failed at run 4.
log(max likelihood)= -8.398895e+01
hparam[0]= 2.222727e+01 [1.000000e-05, 1.000000e+05]
hparam[1]= 6.022066e+00 [1.000000e-04, 1.000000e+04]
hparam[2]= 7.282952e-02 [1.000000e-05, 1.000000e+02]
In [49]:
p.plot(y= 'c-GFP_65GainperOD', strains= 'Hxt6:GFP', conditions= '2% Glu',
       experiments= 'HxtB', ylim= [-10, None], style= 'condition')

Renaming conditions and strains¶

When combining multiple experiments, you may wish for a more consistent or convenient convention for naming strains and conditions.

You can replace names with alternatives using for example

p.rename({'77.WT' : 'WT', '409.Hxt4' : 'Hxt4'})

to simplify strains 77.WT to WT and 409.Hxt4 to Hxt4, and where the dictionary comprises pairs of (old name, new name).

Averaging over experiments¶

The plate reader often does not perfectly increment time between measurements and different experients can have slightly different time points despite the plate reader having the same settings. These unique times prevent the plotting module seaborn from taking averages.

If experiments have measurements that start at the same time point and have the same interval between measurements, then setting a commontime for all experiments will allow seaborn to perform averaging.

The commontime array runs from tmin to tmax with an interval dt between time points. These parameters are automatically calculated, but may also be specified.

Each entry in the time column in the dataframes is assigned a commontime value - the closest commontime point to that time point.

In [50]:
p.addcommonvar('time')

Finding common time.
r dataframe
 HxtA      time_min = 0.000000e+00 ; dtime = 2.135500e-01
 HxtB      time_min = 0.000000e+00 ; dtime = 2.135500e-01
 Using dtime= 2.14e-01
 Using time_min= 0.00e+00


s dataframe
 HxtA      time_min = 0.000000e+00 ; dtime = 2.135500e-01
 HxtB      time_min = 0.000000e+00 ; dtime = 2.135500e-01
 Using dtime= 2.14e-01
 Using time_min= 0.00e+00

With commontime defined, you can average over experiments by plotting with x= commontime. For example, the strain BY4741 is in both experiments:

In [51]:
# no averaging over experiments
p.plot(x= 'time', y= 'OD', strains= 'BY4741', hue= 'condition', style= 'experiment', 
       title= 'no averaging over experiments')
# average over experiments
p.plot(x= 'commontime', y= 'OD', strains= 'BY4741', hue= 'condition', style= 'strain',
       title= 'averaging over experiments')

Importing processed data for more than one experiment¶

Exported data for multiple experiments can be simultaneously imported:

In [52]:
p= om.platereader(wdir= 'data')        

Working directory is /Users/pswain/Dropbox/scripts/ompkg/data.
Files available are: 
---
{0: 'ExampleData.tsv',
 1: 'ExampleData.xlsx',
 2: 'ExampleDataContents.xlsx',
 3: 'ExampleData_r.tsv',
 4: 'ExampleData_s.tsv',
 5: 'ExampleData_sc.tsv',
 6: 'Glu.xlsx',
 7: 'GluContents.xlsx',
 8: 'GluGal.xlsx',
 9: 'GluGalContents.xlsx',
 10: 'GluGal_r.tsv',
 11: 'GluGal_s.tsv',
 12: 'GluGal_sc.tsv',
 13: 'Glu_r.tsv',
 14: 'Glu_s.tsv',
 15: 'Glu_sc.tsv',
 16: 'HxtA.xlsx',
 17: 'HxtAContents.xlsx',
 18: 'HxtB.xlsx',
 19: 'HxtBContents.xlsx',
 20: 'Sunrise.xlsx',
 21: 'SunriseContents.xlsx'}
In [53]:
p.importdf(['Glu', 'GluGal'])  

Imported data/Glu_r.tsv
Imported data/Glu_s.tsv
Imported data/Glu_sc.tsv

Imported data/GluGal_r.tsv
Imported data/GluGal_s.tsv
Imported data/GluGal_sc.tsv


Experiment: Glu 
---
Conditions:
	 0% Glu
	 0.0625% Glu
	 0.125% Glu
	 0.25% Glu
	 0.5% Glu
	 1% Glu
	 media
Strains:
	 BY4741
	 Null
Data types:
	 OD
Ignored wells:
	 None

Experiment: GluGal 
---
Conditions:
	 0% Gal
	 0% Glu
	 0.0625% Gal
	 0.0625% Glu
	 0.125% Gal
	 0.125% Glu
	 0.2% Gal
	 0.2% Glu
	 0.25% Gal
	 0.25% Glu
	 0.3% Gal
	 0.3% Glu
	 0.35% Gal
	 0.35% Glu
	 0.5% Gal
	 0.5% Glu
	 1% Gal
	 1% Gal-NoSC
	 1% Glu
	 1% Glu-NoSC
	 2% Gal
	 2% Glu
	 media
Strains:
	 BY4741
	 Null
Data types:
	 OD
Ignored wells:
	 None

The dataframe from each experiment is combined into one creating an amalgamated p.s and p.sc (as well as p.r). Data from these dataframes can be plotted as usual:

In [54]:
p.plot(x= 'OD_mean', y= 'gr', conditions= '1% Glu', hue= 'experiment', title= '1% glucose')
In [55]:
p.plot(x= 'max_gr', y= 'condition', hue= 'experiment', style= 'strain', 
       conditionincludes= 'Glu')
In [56]:
p.plot(x= 'max_gr', y= 'log2_OD_ratio', hue= 'experiment', style= 'strain', 
       conditionincludes= 'Glu')
In [57]:
p.plot(x= 'max_gr', y= 'log2_OD_ratio', col= 'experiment', style= 'strain', 
       conditionincludes= 'Glu', aspect= 0.7)
In [ ]: