Tiny Zoo

Remote Control Microscopy

Australia has many excellent zoos of varying size, but this is (probably) the smallest, at approximately one billionth the size of Werribee Open Range Zoo. Parking would be kind of a hassle, but you can visit online SOON at tiny.zoic.org.

This article discusses the history and aims of the project. For more technical specifics of the software, see the Tiny Zoo repo.

Species

The microscope is powerful enough to see various microfauna which can grow to about grow up to 500µm long.

I also found one enormous larva (coleoptera?) which came zipping past like an express train.

The aim is to observe living creatures and their behaviours in their environment. Eventually I hope to have OpenCV detect, record and follow interesting species.

FARlabs allows students to perform remote experiments with radioisotopes, to measure the differences between different types of radiation.
Open Flexure is a servoed microscope which uses microscope lenses and “flexures”, bendy bits of 3d print, instead of all the metal parts which Tiny Zoo uses. It has a 12 × 12 × 4mm range of motion and is more aimed at higher magnification viewing of prepared slides.

This project may also owe just a little something to reading James Blish’s Surface Tension at an impressionable age.

Camera & Optics

Currently a 12Mpix USB webcam capable of capturing JPEGs at about 20fps, see Using the Waveshare IMX577 Camera from Python

Lenses

I’ve got the setup working with two lenses:

A “100x” industrial microscope C-mount lens. This is really about 1x in microscope terms I think.
A “4x” microscope objective with a 0.5x C-mount converter

I did try out a “10x” microscope objective but getting it to focus proved very difficult. It might be practical with autofocus and maybe an improved Z axis. Or maybe this lens is just no good with this setup.

I’d like to look at upgrading to an infinity corrected lens setup as well. These lenses are more expensive but may offer better sharpness and a more convenient geometry.

I’m also considering mounting two cameras and lenses side by side and allowing zoom to switch between them, to cover a wider range of zoom than the digital zoom can allow.

Autofocus

The depth of field of microscope lenses is very shallow: a matter of microns! Jogging the Z axis back and forth a micron at a time is pretty tedious. Some form of autofocus will be very helpful to the user.

There are various methods of measuring focus which can be combined with small movements of the Z axis to automatically fine-tune focus. I’ve tried out variance-of-laplacian and it seems to give a pretty clear peak.

I’ve also looked at Deep learning for single-shot autofocus microscopy which seems like a really interesting way to do autofocus more efficiently. I’ve noticed when manually focussing that it’s often possible to estimate if the focal plane is too high or too low just by looking at the shape of the blur, so this technique seems promising.

Lighting

Lighting is definitely a big issue here, and I’ve been experimenting with lighting from above, below, on the sides, etc. I think it might end up needing to have several lights so that lighting can be adjusted depending on the situation.

So far the best setup has been a ring of six LEDs with the lens peeking through the center. These are too bright sometimes though, overloading the sensor, so PWM control of the lights would make sense.

I’m also quite taken with the appearance of dark field microscopy for this and stumbled upon the Arlow-Abbe Condenser which looks like an interesting concept as it would make it possible to switch lighting options under software control. The condenser would have to move with the lens though, which would require significant changes to the stage setup.

Motion Platform & Controller

The stage is approx 100x100x100 mm using 3 stepper axes with about 1.25µm resolution:

Motors: 1.8⁰ step angle so 200 steps per rev
Screws: Tr6x2(P1) threads move 2mm per rev so 10µm per step.
Stepper Drivers: 8 microsteps per motor step so about 1.25µm per microstep.

See Three-axis motion with GRBL.

More details TBC and 3D models for the parts connecting the stages will be included under models/.

Improved Z axis

The useful range of the Z axis for focussing is only 2-3mm which makes using a 100mm stage seem a bit silly. I’m thinking of replacing it with a shorter, lighter, more precise stage.

With a microscope lens on board and the long tube (~180mm) required to adapt it to the camera, there’s a noticeable vibration in the image when moving unless acceleration is severely limited. A more rigid stage and/or more sophisticated kinematics might help.

GRBL Controller

A 3-axis GRBL controller board drives the three axes and has microstepping drivers so the steppers can be run at 1, 2, 4, 8 or 16 microsteps per step.

This board can potentially control the lighting as well, there are several interfaces for laser and/or spindle control which I could repurpose to drive LED lights.

Server Software

A really simple aiohttp based server is under server/. Static files including HTML, CSS and Javscript are under server/static/.

This software is simple enough that it could run easily on an RPi or similar SBC. It currently doesn’t do any image processing, just picks the frames up and sends them.

Autofocus and video clip recording will need more processing power and OpenCV for image processing so the plan is to run it on a Jetson Nano I happen to have laying around, which has good GPU capabilities plus a MIPI-CSI2 interface for the camera.

Streaming Images

Images are captured at 1024x768 pixels and served up as JPEG frames in a multipart/x-mixed-replace Motion JPEG (MJPG) stream. Clients only recieve frames when they’re ready for them, as a primitive form of throttling.

Capturing Video

Movement is a really important part of identifying small, semi-transparent creatures, so as well as screenshots users can capture small videos.

To capture video, a /vid handler starts a task to enqueue frames, and spawns a process which uses OpenCV) to encode frames into an AVI file using an XVID (MPEG4) codec. The output video is written to a temporary FIFO which is streamed to the client as it becomes available.

At the moment, videos are 200 frames (about 10 seconds) long, less than 2MB in size.

User Interface

Simple Javascript UI should mostly just display the image received, but also should allow control through many different mechanisms:

Keyboard controls
Touch screen ‘joystick’/’sliders’ in bottom corners.
Touching image to recenter
Pinch/stretch image to zoom
Gamepad API

A “map” of the whole area in reduced resolution would be nice too, and a scale indicator in mm.

Control of lighting might be useful as well.

Communicating with the GRBL controller

The server communicates with the server using G-code over serial over USB. The serial port on this particular controller appears as /dev/ttyACM0 or similar under Linux with identifiers 0483:5740 STMicroelectronics Virtual COM Port.

The server sends G0 commands to move the stages and also polls their position using the ? realtime command. A separate task listens to the serial data received from the controller for position updates and relays these to each connected client.

Eventually some more sophisticated movement may be implemented but this will do for now.

Reflector

Eventually it might be a good idea to implement an intelligent proxy which can receive a single stream from the (each!) microscope and multiplex them out to multiple destinations. I did get as far as setting up a linode to host this and configuring letsencrypt.

Other Thoughts

I need some way to map ‘zoom’ onto ‘pixels per mm’ for both movement and for displaying a scale.
Motion detection and tracking would be lovely.
Autofocus using variance of Laplacian.
Video recording / downloading.
Focus stacking would be nice.

... and another thing ...