TEM Architecture
TEM Architecture

TEM Architecture

Welcome to the documentation for the Allen Institute next generation TEM data acquisition system. This system is used at the Allen Institute for Brain Science to do large-scale high-throughput TEM imaging of brain tissue. The system is primarily comprised of a number of Docker containers communicating via a message broker. Using this architecture, the system is modular, high performance, and stable.

An overview of the architecture is as follows, the pyTEM service is a hierarchical state machine that coordinates the entire system. This service includes the routines for auto-focusing the microscope, centering the beam, etc, all encoded as state machines. Communication is handled by a message broker and our Pigeon client library. Under the hood this uses the STOMP protocol for communication. There are also services which handle communication with a single piece of hardware, such as the scope, stage, and camera. Once an image (tile) is collected from the camera it is saved to disk and a message is sent to the image processing pipeline. This pipeline performs some simple image processing such as flatfield correction and contrast enhancement, calculates some statistics, such as min, max, and mean image values, a histogram, and a focus quality metric, and finally checks that there is overlap between tiles. These metrics are received by two services, the first of which is the QC which monitors the tile statistics to ensure there are no errors in the acquisition. If any errors are detected, a message will be sent to pyTEM instructing it to either immediately stop imaging, or stop imaging at the end of the montage. The second service is the buffer which collects the processed images, and the statistics. After a montage is completed successfully, this service sends the tiles and metadata to aloha via HTTP. This service

This is an event driven system with various components running in docker containers, along with other services running on other machines. Below, is a diagram showing a general overview of this system.

%3 cluster_AWS AWS cluster_Docker Compose Docker Compose cluster_Platform9 Platform9 cf8eed15f78c4c868f6f3f701f316c5d Database 4e577b850a9f4ec1866e4f2a427b5021 TEM DB 4e577b850a9f4ec1866e4f2a427b5021->cf8eed15f78c4c868f6f3f701f316c5d ee3e999eee544cba97f11aaec8b7fca3 pyTEM 4e577b850a9f4ec1866e4f2a427b5021->ee3e999eee544cba97f11aaec8b7fca3 744d6806964d477692fbe32abf81fce4 AC/QC 4e577b850a9f4ec1866e4f2a427b5021->744d6806964d477692fbe32abf81fce4 d45f885b282e454ca2d4e13f9c4478d9 Aloha d45f885b282e454ca2d4e13f9c4478d9->4e577b850a9f4ec1866e4f2a427b5021 02b735888158471484d341761d99509d Storage Bucket 02b735888158471484d341761d99509d->d45f885b282e454ca2d4e13f9c4478d9 02b735888158471484d341761d99509d->744d6806964d477692fbe32abf81fce4 8d4ec7e462da4e5196ec1866056b0684 AC/QC user 8d4ec7e462da4e5196ec1866056b0684->744d6806964d477692fbe32abf81fce4 b01a7c16cda2494eafade7a2b30c12f7 Microscope Operator b01a7c16cda2494eafade7a2b30c12f7->ee3e999eee544cba97f11aaec8b7fca3 b15f0c892f6c46eaa1f20f2542f04187 ActiveMQ fe87b5d8589f4e569d7c18127c253144 Microscope Service ee3e999eee544cba97f11aaec8b7fca3->fe87b5d8589f4e569d7c18127c253144 b258bb1e503a4a85bc5161a5b700d6d8 Camera Service ee3e999eee544cba97f11aaec8b7fca3->b258bb1e503a4a85bc5161a5b700d6d8 a7c81b8a00b94b7586bc2e06c3fd027a Stage Service ee3e999eee544cba97f11aaec8b7fca3->a7c81b8a00b94b7586bc2e06c3fd027a e3d67ddac39945ad9eab6a77da0328cf Image Processing Pipeline ee3e999eee544cba97f11aaec8b7fca3->e3d67ddac39945ad9eab6a77da0328cf e3d67ddac39945ad9eab6a77da0328cf->b01a7c16cda2494eafade7a2b30c12f7 a66a3d4dcdfb47b1bade232bce773652 Buffer Service e3d67ddac39945ad9eab6a77da0328cf->a66a3d4dcdfb47b1bade232bce773652 da8e0ed710954df585d3b782e3f76884 QC Service e3d67ddac39945ad9eab6a77da0328cf->da8e0ed710954df585d3b782e3f76884 5181f71f26ff49f7a50fa2dd45d6c0a8 Montage Fit e3d67ddac39945ad9eab6a77da0328cf->5181f71f26ff49f7a50fa2dd45d6c0a8 a66a3d4dcdfb47b1bade232bce773652->d45f885b282e454ca2d4e13f9c4478d9 da8e0ed710954df585d3b782e3f76884->ee3e999eee544cba97f11aaec8b7fca3 5181f71f26ff49f7a50fa2dd45d6c0a8->a66a3d4dcdfb47b1bade232bce773652 5181f71f26ff49f7a50fa2dd45d6c0a8->da8e0ed710954df585d3b782e3f76884 bd88b2138e6e48339e73e1b036a4cf0a Microscope bd88b2138e6e48339e73e1b036a4cf0a->fe87b5d8589f4e569d7c18127c253144 a339ea250ca54486a018b1fdfd21aa36 Stage a339ea250ca54486a018b1fdfd21aa36->a7c81b8a00b94b7586bc2e06c3fd027a a09015cfaf784a4f8a05d6546f685785 Camera a09015cfaf784a4f8a05d6546f685785->b258bb1e503a4a85bc5161a5b700d6d8 177f7414527345b9b09bdc2f8139d96e Log Server 49c20d1aa8c342739a55469af1d05108 Container Registry 6e21132d730f4ed982979d5ef5308c6e GitHub Actions 6e21132d730f4ed982979d5ef5308c6e->49c20d1aa8c342739a55469af1d05108

Most of the nodes in this diagram are hyperlinks to further documentation of this system.

This system primarily sends messages between services using the STOMP protocul with the Pigeon library. The configuration for this system as a whole is stored on GitHub at AllenInstitute/TEM_config. This repository includes both a Docker compose file for starting all the services that are part of the system, and the configuration files for the services.