Since the day we launched, ProPublica has encouraged people to republish our stories for free. We even license our stories under Creative Commons (CC). However, in the past we've had trouble knowing precisely which stories had been republished where, and we had no way of knowing how many people were reading our stories on sites that republished them under our CC license.
Shortly after the redesign of our site, we started working on a system that would help us solve this problem. When we found out that Jeremy Ashkenas, a developer at DocumentCloud, was working on a similar problem, we joined forces, and finished work on a lightweight stats tracker, which we are open sourcing today.
World, meet
Pixel Ping
.
Pixel Ping is a minimalist pixel tracker -- written in
CoffeeScript
and compiled into JavaScript -- that runs on top of a tremendously fast server-side JavaScript framework called
node.js
. Pixel Ping tracks each request for a small 1x1 image and periodically flushes aggregated totals of those requests to an endpoint (in our case, a MySQL database).
Pixel Ping is not a particularly new idea, but other solutions (Google Analytics, Omniture, etc) are designed for a different environment and purpose, and can require a fair amount of resources up front. Also, one of the requirements in our use case is that our pixel tracker count only what’s necessary, and not create any privacy worries on the part of sites that republish our material. So our internal implementation of Pixel Ping just tracks page impressions. It doesn’t know anything about individual users, and it doesn’t set or read any cookies.
Pixel Ping is damn fast: an ApacheBench
test
averaged 3ms per request for 1,000 requests (yadda-yadda, local requests, grain of salt, etc.), and it also only uses around 8 megabytes of memory while it is running.
Along with Pixel Ping we're open sourcing a simple backend for Pixel Ping, written in Ruby on Rails, called
Pixel Pong
, which we use at ProPublica to provide a reporting interface on the collected data.
The Setup
Pixel Ping Only
Before you install Pixel Ping you'll need
node.js
, and
npm
. Once you've installed both, you can run:
npm install pixel-ping
and then run:
pixel-ping path/to/config.json
where config.json looks something like this:
"host": "127.0.0.1",
"port": "9187",
"interval": 600,
"endpoint": "http://www.example.com/save_hits"
You can test that it works by going to
http://127.0.0.1:9187/pixel.gif?key=test
in your browser.
Pixel Ping with Pixel Pong
The easiest way to get up and running with Pixel Ping is to install Pixel Pong with
git clone http://github.com/propublica/pixel-pong.git
then install DocumentCloud's jammit with:
cd pixel-pong; rake gems:install
You'll also need a database.yml in the config directory then run
rake db:create; rake db:migrate
Then install Pixel Ping with:
rake pixel:install
and then add a file in the config directory called config/pixel-ping.development.json with the following:
"port": "9187",
"host": "127.0.0.1",
"interval": 5,
"endpoint": "http://127.0.0.1:3000/stats.json",
"secret": "one very good secret"
Once you're all set you can start the Rails app with:
script/server
then in a new shell go to the Pixel Pong directory and start Pixel Ping with:
rake pixel:start
Now if you go to the Pixel Ping tracking gif url:
http://127.0.0.1:9187/pixel.gif?key=test☃http://www.example.com
a couple of times you'll see hits logged every second in the
reporting interface
.
Pixel Pong accepts flushes from Pixel Ping that are delimited by a ☃, in this case the title would be "test" and the url would be "http://www.example.com".
If you point your web browser to:
http://127.0.0.1:9187/pixel.js
You'll see a small bit of javascript that you can include on your own pages. It will insert the pixel and form the request properly so you won't have to find the ☃ on your keyboard.
Both DocumentCloud and ProPublica have been running Pixel Ping instances in production for over a month without problems, but as with any open source project we expect there will be bugs. If you find any you can use the issue trackers (
Pixel Ping
and
Pixel Pong
) or send an email to
[email protected]
.
Thank you for your interest in republishing this story. You are are free to republish it so long as you do the following:
You have to credit
ProPublica and any co-reporting partners
. In the byline, we prefer “Author Name, Publication(s).” At the top of the text of your story, include a line that reads: “This story was originally published by ProPublica.” You must link the word “ProPublica” to the original URL of the story.
If you’re republishing online, you must link to the URL of this story on propublica.org, include all of the links from our story, including our newsletter sign up language and link, and use our
PixelPing tag
.
If you use canonical metadata, please use the ProPublica URL. For more information about canonical metadata,
refer to this Google SEO link
.
You can’t edit our material, except to reflect relative changes in time, location and editorial style. (For example, “yesterday” can be changed to “last week,” and “Portland, Ore.” to “Portland” or “here.”)
You cannot republish our photographs or illustrations without specific permission. Please contact
[email protected]
.
It’s okay to put our stories on pages with ads, but not ads specifically sold against our stories. You can’t state or imply that donations to your organization support ProPublica’s work.
You can’t sell our material separately or syndicate it. This includes publishing or syndicating our work on platforms or apps such as Apple News, Google News, etc.
You can’t republish our material wholesale, or automatically; you need to select stories to be republished individually. (To inquire about syndication or licensing opportunities, contact
[email protected]
.)
You can’t use our work to populate a website designed to improve rankings on search engines or solely to gain revenue from network-based advertisements.
We do not generally permit translation of our stories into another language.
Any website our stories appear on must include a prominent and effective way to contact you.
If you share republished stories on social media, we’d appreciate being tagged in your posts. We have official accounts for ProPublica on
Twitter
,
Facebook
and
Instagram
.