添加链接
link管理
链接快照平台
  • 输入网页链接,自动生成快照
  • 标签化管理网页链接
Skip to content
Core

Network API

There are two different ways that you can interact with the Pupil Core Network API:

  1. Pupil Remote : Simple text-based API to remote control Pupil Core software.
  2. IPC Backbone : msgpack based API with access to realtime data.

In the sections below, we outline how the basic communication works and how you can access the different stages of the Network API.

If you want to run the Python examples below you will have to install the following dependencies:

Pupil Remote

Pupil Remote provides a simple, text-based API to remote control the Pupil Core software, as well as to access the second Network API stage (IPC Backbone). It uses ZeroMQ's REQ-REP pattern for reliable one-to-one communication.

This is how you connect to Pupil Remote using pyzmq :

TIP

Pupil Remote accepts requests via a REP socket, by default on port 50020 . Alternatively, you can provide a custom port via the --port application argument.

After opening the REQ socket, you can send simple text messages to control Pupil Capture and Pupil Service functions:

WARNING

For every message that you send to Pupil Remote, you need to receive the response. If you do not call recv() , Pupil Capture might become unresponsive!

This is an overview over all available Pupil Remote commands:

Delayed Execution

Pupil Remote commands can be subject to transmission delay (e.g. from network latency). This is especially important to keep in mind for the T , t , and R commands.

We do not recommend using R for time synchronization. In addition to transmission delay, not all recording processes are guaranteed to start simultaneously when Pupil Capture receives the command. Please read our Best Practices for more appropriate methods of time synchronization.

TIP

Pupil Service does not support the creation of recordings, i.e. the R and r commands do not work with Pupil Service.

See this Python script for a full example interaction with Pupil Remote.

Pupil Groups

The Pupil Groups plugin uses the ZRE protocol to implement real-time local network discovery and many-to-many communication. Common workflows like starting and stopping a recording are already implemented by Pupil Capture to use and respond to the Pupil Groups interface, if available.

If you want to integrate Pupil Groups in your own app or device, have a look at the ZRE protocol specification . From Python we use the pyre library to communicate between groups, but any ZRE implementation can be used. The Pupil Groups plugin joins a user-defined group, by default pupil-groups . Make sure that all devices are in the same group. You can broadcast notifications to all members in the group, by adding the key-value-pair "remote_notify": "all" . All messages, whose topic stars with remote_notify are also broadcasted. The following notifications are broadcasted by default:

  • recording.should_start
  • recording.should_stop

IPC Backbone

The IPC Backbone grants you realtime access to nearly all data generated by Pupil Capture and Pupil Service. It uses ZeroMQ's PUB-SUB pattern for one-to-many communication.

If you want to tap into the IPC Backbone you will need both the IP address and the session's unique port. You can request them from Pupil Remote :

Reading from the IPC Backbone

To start reading from the IPC Backbone, you need to subscribe to the topic of your desired data. Once the subscription was successful, you will start receiving data.

See the data conventions and message format sections for details on the data format.

IPC Backbone Message Format

All messages on the IPC Backbone are multipart messages containing (at least) two message frames:

  • Frame 1 contains the topic string, e.g. pupil.0 , logging.info , notify.recording.has_started

  • Frame 2 contains a msgpack encoded key-value mapping. This is the actual message . We choose msgpack as the serializer due to its efficient format (45% smaller than json , 200% faster than ujson ) and because encoders exist for almost every language.

Message Topics

Messages can have any topic chosen by the user. See topics below for a list of message types used by Pupil apps.

Pupil and Gaze Messages

Pupil data is sent from the eye0 and eye1 process with the topic format pupil.<EYE_ID>.<PUPIL_DETECTOR_IDENTIFIER> , where EYE_ID is 0 or 1 for eye0 and eye1 respectively, and PUPIL_DETECTOR_IDENTIFIER is a string that uniquely identifies the pupil detector plugin that produced the message. In the case of the built-in pupil detectors, the identifier corresponds to 2d and 3d respectively. Therefore, the built-in detectors publish the following four topics: pupil.0.2d , pupil.1.2d , pupil.0.3d , pupil.1.3d .

Gaze mappers receive this data and publish messages with topic gaze . See the Timing & Data Conventions section for example messages for the pupil and gaze topics.

Notification Message

Pupil uses special messages called notifications to coordinate all activities. Notifications are key-value mappings with the required field subject . Subjects are grouped by categories category.command_or_statement . Example: recording.should_stop .

The message topic construction:

You should use the notify topic for coordination with the app. All notifications on the IPC Backbone are automatically made available to all plugins in their on_notify callback and used in all Pupil apps.

In stark contrast to gaze and pupil, the notify topic should not be used at high volume. If you find that you need to write more than 10 messages a second, it is probably not a notification but another kind of data. Use a custom topic instead.

The script above requires you to implement a custom Plugin to process the incoming messages. Alternatively, you can use remote annotations.

Fixation Messages

The Online Fixation Detector in Pupil Capture publishes the following notification:

When method is set to 3d gaze , it will also contain the gaze point position:

The Offline Fixation Detector in Pupil Player additionally includes the following keys:

The online Blink Detector in Pupil Capture publishes the following notification:

Remote Annotations

You can also create annotation events programmatically and send them using the IPC, or by sending messages to the Pupil Remote interface. Here is an example annotation.

TIP

You can add custom fields to your annotation which will be included in the csv export .

TIP

This script demonstrates how to send remote annotations. Use this script as a starting point for your integrations.

Log Messages

The topic is logging.log_level_name (debug, info, warning, error,...). The message is a key-value mapping that contains all attributes of the python logging.record instance.

Pupil Detector Plugin Notifications

The following are notifications handled by pupil detector plugins, that can be sent to the IPC Backbone:

In response to pupil_detector.broadcast_properties , zero or more pupil detector plugins will respond with the following messages:

For an example script that showcases pupil detector plugins' network API, please consult this helper script .

Writing to the IPC Backbone

You can send notifications to the IPC Backbone for everybody to read as well. Pupil Remote acts as an intermediary for reliable transport:

We say reliable transport because Pupil Remote will confirm every notification we send with 'Notification received'. When we get this message we have a guarantee that the notification was received by the Pupil Core software.

If we listen to the Backbone using our subscriber from above, we will see the message again because we have subscribed to all notifications.

Writing to the Backbone directly

If you want to write messages other than notifications onto the IPC Backbone, you can publish to the bus directly. Because this uses a PUB socket, you should read up on Delivery Guarantees PUB-SUB below.

Communicating with Pupil Service

This code shows how to use notifications to start the eye windows, set a calibration method and close Pupil Service:

The code demonstrates how you can listen to all notifications from Pupil Service. It requires a little helper script called zmq_tools.py .

IPC Backbone Implementation

This section provides detailed inside information about the IPC Backbone implementation. Please see specifically the subsection about delivery guarantees .

Pupil Core software uses a PUB-SUB Proxy as their messaging bus. We call it the IPC Backbone . The IPC Backbone runs as a thread in the main process. It is basically a big message relay station. Actors can push messages into it and subscribe to other actors' messages. Therefore, it is the Backbone of all communication to/from and within the Pupil Core software.

TIP

Note - The main process does not do any CPU heavy work. It only runs the proxy, launches other processes and does a few other light tasks.

IPC Backbone used by Pupil Capture and Service

The IPC Backbone has a SUB and a PUB address. Both are bound to a random port on app launch and are known to all components of the app. All processes and threads within the app use the IPC Backbone to communicate.

  • Using a ZMQ PUB socket, other actors in the app connect to the pub_port of the Backbone and publish messages to the IPC Backbone. (For important low volume msgs a PUSH socket is also supported.)
  • Using a ZMQ SUB socket, other actors connect to the sub_port of the Backbone to subscribe to parts of the message stream.

Example: The eye process sends pupil data onto the IPC Backbone. The gaze mappers in the world process receive this data, generate gaze data and publish it on the IPC Backbone. World, Launcher, and Eye exchange control messages on the bus for coordination.

Delivery guarantees ZMQ

ZMQ is a great abstraction for us. It is super fast, has a multitude of language bindings and solves a lot of the nitty-gritty networking problems we don't want to deal with. As our short description of ZMQ does not do ZMQ any justice, we recommend reading the ZMQ guide if you have the time. Below are some insights from the guide that are relevant for our use cases.

  • Messages are guaranteed to be delivered whole or not at all.
  • Unlike bare TCP it is ok to connect before binding.
  • ZMQ will try to repair broken connections in the background for us.
  • It will deal with a lot of low level tcp handling so we don't have to.

Delivery Guarantees PUB-SUB

ZMQ PUB SUB will make no guarantees for delivery. Reasons for not receiving messages are:

  • Async Connect / The Late joiner : PUB sockets drop messages before a connection has been established and topics subscribed. ZMQ connects asynchronously in the background.
  • The Snail : If SUB sockets do not consume delivered messages fast enough they start dropping them.
  • Fast close : A PUB socket may loose packages if you close it right after sending.

For more information see ZMQ Guide Chapter 5 - Advanced Pub-Sub Patterns .

TIP

In order to avoid accidentally dropping notifications in Pupil, we use a PUSH instead of an PUB socket. It acts as an intermediary for notifications and guarantees that any notification sent to the IPC Backbone, is processed and published by it.

Delivery Guarantees REQ-REP

When writing to the Backbone via REQ-REP we will get confirmations/replies for every message sent. Since REPREQ requires lockstep communication that is always initiated from the actor connecting to Pupil Capture/Service. It does not suffer the above issues.

Delivery Guarantees in general

We use TCP in ZMQ, it is generally a reliable transport. The app communicates to the IPC Backbone via localhost loopback, this is very reliable. We have not been able to produce a dropped message for network reasons on localhost.

However, unreliable, congested networks (e.g. wifi with many actors) can cause problems when talking and listening to Pupil Capture/Service from a different machine. If using a unreliable network we will need to design our scripts and apps so that interfaces are able to deal with dropped messages.

Latency

Latency is bound by the latency of the network. On the same machine we can use the loopback interface (localhost) and do a quick test to understand delay and jitter of Pupil Remote requests...

... and when talking directly to the IPC Backbone and waiting for the same message to appear to the subscriber: