Quickstart

import impact as impt
import os

C:\Users\navee\Anaconda3\lib\site-packages\IPython\html.py:14: ShimWarning: The `IPython.html` package has been deprecated since IPython 4.0. You should import from `notebook` instead. `IPython.html.widgets` has moved to `ipywidgets`.
  "`IPython.html.widgets` has moved to `ipywidgets`.", ShimWarning)

The impact framework is designed to help scientists parse, interpret, explore and visualize data to understand and engineer micorbial physiology. The core framework is open-source and written entirely in python.

Data is parsed into an object-oriented data structure, built on top of a relational mapping to most sql databases. This allows for efficient saving and querying to ease data exploration.

Here we provide the basics to get started analyzing data with the core framework. Before getting started, it is worthwhile to understand the basic data schema:

Model	Function
TrialIdentifier	Describes a trial (time, analyte, strain, media, etc.)
AnalyteData	(time, data) points and vectors for quantified data (g/L product, OD, etc.)
SingleTrial	All analytes for a given unit (e.g. a tube, well on plate, bioreactor, etc.)
ReplicateTrial	Contains a set of SingleTrials with replicates grouped to calculate statistics
Experiment	All of the trials performed on a given date

On import, data will automatically be parsed into this format. In addition, data will most commonly be queried by metadata in the TrialIdentifier which is composed of three main identifiers:

Model	Function
Strain	Describes the organism being characterized (e.g. strain, knockouts, plasmids, etc.)
Media	Described the medium used to characterize the organism (e.g. M9 + 0.02% glc_D)
Environment	The conditions and labware used (e.g. 96-well plate, 250RPM, 37C)

Importing data

Data is imported using the parse_raw_data function from the .parsing module. This function returns an Experiment, which is the result of organizing all of your data.

To parse data, the data is usually provided in an xlsx file in one of the desired formats. If your data doesn’t conform to one of the built-in formats, you can use the provided parsers as a cookbook to build your own. Generally, minor edits are required to conform to new data.

Here we use the sample test data, which is a typical format for data from HPLC. Each row is a specific trial and time points, and the columns represent the different analytes, and their types. You can see this data in sample_data/Fermentation_1_impact.xlsx

from impact.parsers import Parser
from pprint import pprint
expt = Parser.parse_raw_data('default_titers',
                             file_name = os.path.join('sample_data','Fermentation_1_impact.xlsx'),
                             id_type='traverse')
expt.calculate()

Importing data from sample_data\Fermentation_1_impact.xlsx...0.0s
Parsed 600 timeCourseObjects in 0.292s...Number of lines skipped:  0
Parsing time point list...Parsed 600 time points in 0.4s
Parsing analyte list...Parsed 6 analytes in 79.8ms
Parsing single trial list...Parsed 2 replicates in 0.0s
Analyzing data...No blanks were indicated. Blank subtraction will not be done.Ran analysis in 0.2s

c:\users\navee\dropbox\naveen\impact\impact\impact\core\features\ProductYield.py:22: RuntimeWarning: invalid value encountered in true_divide
  self.substrate_consumed

The data is now imported and organized, we can quickly get an overview of what we’ve imported.

print(expt)

strain                        media                   environment    analytes
----------------------------  ----------------------  -------------  ------------------------------------------------------------------------------------
BD2 Δ(adhE,ldhA,pflB) + pBD1  mod M9 + 1.0 a.u. IPTG  None           ['ac', 'etoh', 'acald', 'suc', 'rs23bdo', '13bdo', 'm23bdo', 'od', 'acet', 'glc__D']
BD4 Δ(adhE,ldhA,pflB) + pBD3  mod M9 + 1.0 a.u. IPTG  None           ['ac', 'etoh', 'acald', 'suc', 'rs23bdo', '13bdo', 'm23bdo', 'od', 'acet', 'glc__D']

Before we dive into data analysis, it is worth having a basic understanding of the schema to know where to look for data.

Firstly, all data is funneled into a ReplicateTrial, even if you only have one replicate. As such, it is convenient to always look for data in this object. This object contains both an avg and std attribute where you can find the respective statistics. avg and std attributes are instances of SingleTrial, so we can access the statistical data similarly to the raw data itself.

Querying and filtering for data

After import, data is all sorted into python objects, associated to an sql database using an object-relational mapper, SQLalchemy. Usually, we’re interested in comparing a set of features and a set of conditions (strain, media, environment) and the queryable database allows us to search for the data we are interested in.

Although it is usually simple to use the ORM to access the database directly, basic querying can also be done using python list comprehensions. The major limitation is that you will only query experiments loaded in memory, e.g. experiments that were parsed into this notebook.

reps = [rep for rep in expt.replicate_trials]
for rep in reps:
    print(rep.trial_identifier)

strain: BD2 Δ(adhE,ldhA,pflB) + pBD1,   media: mod M9 + 1.0 a.u. IPTG,  env: None
strain: BD4 Δ(adhE,ldhA,pflB) + pBD3,   media: mod M9 + 1.0 a.u. IPTG,  env: None

reps = [rep for rep in expt.replicate_trials
       if rep.trial_identifier.strain.name == 'BD1']
for rep in reps:
    print(rep.trial_identifier)

To use the database, we must query data through a session object. The session is open for the entire application.

from impact.database import session, create_db
create_db()
session

<sqlalchemy.orm.session.Session at 0x17721fffba8>

Now that we have a session, we can use the standard SQLalchemy ORM language to query - it is described in detail here http://docs.sqlalchemy.org/en/latest/orm/tutorial.html#querying

session.add(expt)

reps = session.query(impt.ReplicateTrial)\
                .join(impt.ReplicateTrialIdentifier)\
                .join(impt.Strain)\
                .filter(impt.Strain.name == 'strain1').all()

for rep in reps:
    print(rep.trial_identifier)

Visualization

Several packages already exist for visualization in python. The most popular one in matplotlib, it has very simple syntax which should feel familiar for matlab users; however, matplotlib generates static plots. The Impact visualization module is built around plotly, which generated dynamic javascript plots, and as such it is worthwhile understanding the basic syntax of plotly charts.

import impact.plotting as implot
import numpy as np

# Charts are made up in a hierarchical structure, but can be quickly generated as follows:
x = np.linspace(0,10,10)
y = np.linspace(0,10,10)**2
implot.plot([implot.go.Scatter(x=x,y=y),
             implot.go.Scatter(x=x,y=y*2)])

# For more control over these plots, they can be built form the ground up
# Traces are defined for each feature
traces = [implot.go.Scatter(x=x,y=y),
             implot.go.Scatter(x=x,y=y*2)]

layout = implot.go.Layout(height=400,width=400)

# Traces are joined to a figure
fig = implot.go.Figure(data=traces, layout=layout)

# And a figure is printed using plot
implot.plot(fig)

It should be noted that the implot package offers a direct wrapper to useful plotly functions, which could also be accessed with plotly directly. The Impact visualization module offers functions to help extract useful data and generate traces.

from impact.plotting import time_profile_traces

implot.plot(time_profile_traces(replicate_trials=expt.replicate_trials,
                               analyte='etoh',
                               label=lambda rep: str(rep.trial_identifier.strain)))

for rep in expt.replicate_trials:
    layout = implot.go.Layout(title=str(rep.trial_identifier.strain))
    data = [implot.go.Scatter(x=trial.analyte_dict['etoh'].time_vector,
                              y=trial.analyte_dict['etoh'].data_vector,
                              name=trial.trial_identifier.replicate_id)
            for trial in rep.single_trials]
    data.append(implot.go.Scatter(x=rep.avg.analyte_dict['etoh'].time_vector,
                              y=rep.avg.analyte_dict['etoh'].data_vector,
                               error_y = {'array':rep.std.analyte_dict['etoh'].data_vector},
                              name='statistic'))
    data = sorted(data,key=lambda data: str(data['name']))
    implot.plot(implot.go.Figure(data=data,layout=layout))

import impact.plotting
import plotly.graph_objs as go
from plotly.offline import iplot

iplot(
    [go.Scatter(
            x=rep.avg.analyte_dict['etoh'].time_vector,
            y=rep.avg.analyte_dict['etoh'].data_vector,
            name=str(rep.trial_identifier),
            error_y=dict(type='data', array=rep.std.analyte_dict['etoh'].data_vector)
               )
     for rep in expt.replicate_trials]
)

Exploring features

With a standard schema for the data, we can now begin to explore some of the features which have been generated. Features include things like:

• rate (\(g\ \ h^{-1}\))
• yield (\(g_{product}\ \ g_{substrate}^{-1}\))
• specific productivity (\(g\ \ gdw^{-1}\ \ h^{-1}\))
• normalized data (e.g. \(a.u. fluorescence\ \ OD_{600}^{-1}\))

iplot(
    [go.Scatter(
            x=rep.avg.analyte_dict['etoh'].time_vector,
            y=rep.avg.analyte_dict['etoh'].specific_productivity,
            name=str(rep.trial_identifier),
            error_y=dict(type='data', array=rep.std.analyte_dict['etoh'].specific_productivity)
               )
     for rep in expt.replicate_trials]
)

We can also choose to plot only the end points, making it easier to interpret a large number of strains:

iplot(
    [go.Bar(
            x=[str(rep.trial_identifier.strain)],
            y=[rep.avg.analyte_dict['etoh'].data_vector[-1]],
            name=str(rep.trial_identifier.strain),
            error_y=dict(type='data', array=[rep.std.analyte_dict['etoh'].data_vector[-1]])
               )
    for rep in expt.replicate_trials]
)

Using only the end point, it becomes easier to visualize all analytes:

unique_analytes = list(set([analyte for rep in expt.replicate_trials for analyte in rep.avg.analyte_dict]))
len(unique_analytes)

10

fig = implot.tools.make_subplots(rows=2,cols=5,subplot_titles=unique_analytes)
colors = implot.cl.scales['3']['qual']['Set1']

row, col, flag = 1, 0, True
for analyte in unique_analytes:
    col += 1
    if col == 6:
        row += 1
        col = 1

    for rep, color in zip(expt.replicate_trials,colors):
        trace = go.Bar(
            x=[str(rep.trial_identifier.strain)],
            y=[rep.avg.analyte_dict[analyte].data_vector[-1]],
            name=str(rep.trial_identifier.strain),
            error_y={'array':[rep.std.analyte_dict[analyte].data_vector[-1]]},
            marker={'color':color},
            showlegend=flag,
            legendgroup=str(rep.trial_identifier.strain)
        )
        fig.append_trace(trace,row,col)
    flag = False

# Finally we can set some aesthetics
for x in range(1,11):
    fig['layout']['xaxis'+str(x)]['showticklabels'] = False

implot.plot(fig)

This is the format of your plot grid:
[ (1,1) x1,y1 ]    [ (1,2) x2,y2 ]    [ (1,3) x3,y3 ]    [ (1,4) x4,y4 ]    [ (1,5) x5,y5 ]
[ (2,1) x6,y6 ]    [ (2,2) x7,y7 ]    [ (2,3) x8,y8 ]    [ (2,4) x9,y9 ]    [ (2,5) x10,y10 ]