Network API
There are two different ways that you can interact with the Pupil Core Network API:
- Pupil Remote : Simple text-based API to remote control Pupil Core software.
- IPC Backbone : msgpack based API with access to realtime data.
In the sections below, we outline how the basic communication works and how you can access the different stages of the Network API.
If you want to run the Python examples below you will have to install the following dependencies:
Pupil Remote
Pupil Remote provides a simple, text-based API to remote control the Pupil Core software, as well as to access the second Network API stage (IPC Backbone). It uses ZeroMQ's REQ-REP pattern for reliable one-to-one communication.
This is how you connect to Pupil Remote using pyzmq :
TIP
Pupil Remote accepts requests via a
REP
socket, by default on port
50020
. Alternatively, you can provide a custom port via the
--port
application argument.
After opening the
REQ
socket, you can send simple text messages to control Pupil Capture and Pupil Service functions:
WARNING
For every message that you send to Pupil Remote, you need to receive the response. If you do not call
recv()
, Pupil Capture might become unresponsive!
This is an overview over all available Pupil Remote commands:
Delayed Execution
Pupil Remote commands can be subject to transmission delay (e.g. from network latency). This is especially important to keep in mind for the
T
,
t
, and
R
commands.
We do not recommend using
R
for time synchronization. In addition to transmission delay, not all recording processes are guaranteed to start simultaneously when Pupil Capture receives the command. Please read our
Best Practices
for more appropriate methods of time synchronization.
TIP
Pupil Service does not support the creation of recordings, i.e. the
R
and
r
commands do not work with Pupil Service.
See this Python script for a full example interaction with Pupil Remote.
Pupil Groups
The Pupil Groups plugin uses the ZRE protocol to implement real-time local network discovery and many-to-many communication. Common workflows like starting and stopping a recording are already implemented by Pupil Capture to use and respond to the Pupil Groups interface, if available.
If you want to integrate Pupil Groups in your own app or device, have a look at the
ZRE protocol specification
. From Python we use the
pyre
library to communicate between groups, but any ZRE implementation can be used. The Pupil Groups plugin joins a user-defined group, by default
pupil-groups
. Make sure that all devices are in the same group. You can broadcast notifications to all members in the group, by adding the key-value-pair
"remote_notify": "all"
. All messages, whose topic stars with
remote_notify
are also broadcasted. The following notifications are broadcasted by default:
-
recording.should_start
-
recording.should_stop
IPC Backbone
The IPC Backbone grants you realtime access to nearly all data generated by Pupil Capture and Pupil Service. It uses ZeroMQ's PUB-SUB pattern for one-to-many communication.
If you want to tap into the IPC Backbone you will need both the IP address and the session's unique port. You can request them from
Pupil Remote
:
Reading from the IPC Backbone
To start reading from the IPC Backbone, you need to subscribe to the
topic
of your desired data. Once the subscription was successful, you will start receiving data.
See the data conventions and message format sections for details on the data format.
IPC Backbone Message Format
All messages on the IPC Backbone are multipart messages containing (at least) two message frames:
-
Frame 1
contains the topic string, e.g.pupil.0
,logging.info
,notify.recording.has_started
-
Frame 2
contains amsgpack
encoded key-value mapping. This is the actual message . We choosemsgpack
as the serializer due to its efficient format (45% smaller thanjson
, 200% faster thanujson
) and because encoders exist for almost every language.
Message Topics
Messages can have any topic chosen by the user. See topics below for a list of message types used by Pupil apps.
Pupil and Gaze Messages
Pupil data is sent from the eye0 and eye1 process with the topic format
pupil.<EYE_ID>.<PUPIL_DETECTOR_IDENTIFIER>
, where
EYE_ID
is
0
or
1
for eye0 and eye1 respectively, and
PUPIL_DETECTOR_IDENTIFIER
is a string that uniquely identifies the pupil detector plugin that produced the message. In the case of the built-in pupil detectors, the identifier corresponds to
2d
and
3d
respectively. Therefore, the built-in detectors publish the following four topics:
pupil.0.2d
,
pupil.1.2d
,
pupil.0.3d
,
pupil.1.3d
.
Gaze mappers receive this data and publish messages with topic
gaze
. See the
Timing & Data Conventions
section for example messages for the
pupil
and
gaze
topics.
Notification Message
Pupil uses special messages called
notifications
to coordinate all activities. Notifications are key-value mappings with the required field
subject
. Subjects are grouped by categories
category.command_or_statement
. Example:
recording.should_stop
.
The message topic construction:
You should use the
notify
topic for coordination with the app. All notifications on the IPC Backbone are automatically made available to all plugins in their
on_notify
callback
and used in all Pupil apps.
In stark contrast to gaze and pupil, the notify topic should not be used at high volume. If you find that you need to write more than 10 messages a second, it is probably not a notification but another kind of data. Use a custom topic instead.
The script above requires you to implement a custom Plugin to process the incoming messages. Alternatively, you can use remote annotations.
Fixation Messages
The Online Fixation Detector in Pupil Capture publishes the following notification:
When
method
is set to
3d gaze
, it will also contain the gaze point position:
The Offline Fixation Detector in Pupil Player additionally includes the following keys:
Blink Messages
The online Blink Detector in Pupil Capture publishes the following notification:
Remote Annotations
You can also create annotation events programmatically and send them using the IPC, or by sending messages to the Pupil Remote interface. Here is an example annotation.
TIP
You can add custom fields to your annotation which will be included in the csv export .
TIP
This script demonstrates how to send remote annotations. Use this script as a starting point for your integrations.
Log Messages
The topic is
logging.log_level_name
(debug, info, warning, error,...). The message is a key-value mapping that contains all attributes of the python
logging.record
instance.
Pupil Detector Plugin Notifications
The following are notifications handled by pupil detector plugins, that can be sent to the IPC Backbone:
In response to
pupil_detector.broadcast_properties
, zero or more pupil detector plugins will respond with the following messages:
For an example script that showcases pupil detector plugins' network API, please consult this helper script .
Writing to the IPC Backbone
You can send notifications to the IPC Backbone for everybody to read as well. Pupil Remote acts as an intermediary for reliable transport:
We say reliable transport because Pupil Remote will confirm every notification we send with 'Notification received'. When we get this message we have a guarantee that the notification was received by the Pupil Core software.
If we listen to the Backbone using our subscriber from above, we will see the message again because we have subscribed to all notifications.
Writing to the Backbone directly
If you want to write messages other than notifications onto the IPC Backbone, you can publish to the bus directly. Because this uses a PUB socket, you should read up on Delivery Guarantees PUB-SUB below.
Communicating with Pupil Service
This code shows how to use notifications to start the eye windows, set a calibration method and close Pupil Service:
The code demonstrates how you can listen to all notifications from Pupil Service. It requires a little helper script called zmq_tools.py .
IPC Backbone Implementation
This section provides detailed inside information about the IPC Backbone implementation. Please see specifically the subsection about delivery guarantees .
Pupil Core software uses a PUB-SUB Proxy as their messaging bus. We call it the
IPC Backbone
. The IPC Backbone runs as a thread in the main process. It is basically a big message relay station. Actors can push messages into it and subscribe to other actors' messages. Therefore, it is the Backbone of all communication to/from and within the Pupil Core software.
TIP
Note - The main process does not do any CPU heavy work. It only runs the proxy, launches other processes and does a few other light tasks.
IPC Backbone used by Pupil Capture and Service
The IPC Backbone has a
SUB
and a
PUB
address. Both are bound to a random port on app launch and are known to all components of the app. All processes and threads within the app use the IPC Backbone to communicate.
-
Using a
ZMQ PUB
socket, other actors in the app connect to thepub_port
of the Backbone and publish messages to the IPC Backbone. (For important low volume msgs a PUSH socket is also supported.) -
Using a
ZMQ SUB
socket, other actors connect to thesub_port
of the Backbone to subscribe to parts of the message stream.
Example: The eye process sends pupil data onto the IPC Backbone. The gaze mappers in the world process receive this data, generate gaze data and publish it on the IPC Backbone. World, Launcher, and Eye exchange control messages on the bus for coordination.
Delivery guarantees ZMQ
ZMQ is a great abstraction for us. It is super fast, has a multitude of language bindings and solves a lot of the nitty-gritty networking problems we don't want to deal with. As our short description of ZMQ does not do ZMQ any justice, we recommend reading the ZMQ guide if you have the time. Below are some insights from the guide that are relevant for our use cases.
- Messages are guaranteed to be delivered whole or not at all.
- Unlike bare TCP it is ok to connect before binding.
- ZMQ will try to repair broken connections in the background for us.
- It will deal with a lot of low level tcp handling so we don't have to.
Delivery Guarantees PUB-SUB
ZMQ PUB SUB will make no guarantees for delivery. Reasons for not receiving messages are:
-
Async Connect
/The Late joiner
: PUB sockets drop messages before a connection has been established and topics subscribed. ZMQ connects asynchronously in the background. -
The Snail
: If SUB sockets do not consume delivered messages fast enough they start dropping them. -
Fast close
: A PUB socket may loose packages if you close it right after sending.
For more information see ZMQ Guide Chapter 5 - Advanced Pub-Sub Patterns .
TIP
In order to avoid accidentally dropping notifications in Pupil, we use a
PUSH
instead of an
PUB
socket. It acts as an intermediary for notifications and guarantees that any notification sent to the IPC Backbone, is processed and published by it.
Delivery Guarantees REQ-REP
When writing to the Backbone via REQ-REP we will get confirmations/replies for every message sent. Since REPREQ requires lockstep communication that is always initiated from the actor connecting to Pupil Capture/Service. It does not suffer the above issues.
Delivery Guarantees in general
We use TCP in ZMQ, it is generally a reliable transport. The app communicates to the IPC Backbone via localhost loopback, this is very reliable. We have not been able to produce a dropped message for network reasons on localhost.
However, unreliable, congested networks (e.g. wifi with many actors) can cause problems when talking and listening to Pupil Capture/Service from a different machine. If using a unreliable network we will need to design our scripts and apps so that interfaces are able to deal with dropped messages.
Latency
Latency is bound by the latency of the network. On the same machine we can use the loopback interface (localhost) and do a quick test to understand delay and jitter of Pupil Remote requests...
... and when talking directly to the IPC Backbone and waiting for the same message to appear to the subscriber: