There’s something I love about bad robots. To this day, I can’t watch this KetchupBot video without smiling.
Part of what makes this robot so great, is that it hits the perfect level of crappiness. I think if it were any better or worse at applying ketchup, it would be less funny. There’s also something about putting a lot of careful work into a machine that’s both crappy and useless.
Along these lines, Declan, Max, Phil and I built a Slackbot to imitate our friends. Every hour or so, it simulates a conversation among the users of our Slack workspace and posts it to a special channel in real time. It does a pretty good job of matching each person’s style, vocabulary, subject matter, punctuation, and spelling. It doesn’t do a good job of generating sentences that make sense. I think it kind of landed in KetchupBot territory.
The basis of the bot is a statistical model called a Markov chain trained on our Slack history.
What is a Markov Chain
A Markov chain models a system as a set of discrete states and transition probabilities between these state. I think a diagram is the best way to explain.
Let’s model the weather in Somerville, MA with a Markov chain model. The state will be the weather during one hour of the day.
In this model, the set possible states is
rain. This diagram says if it is sunny right now, there is a 90% chance that it will also be sunny in the next hour, and a 10% chance it will be raining. If it is raining right now, there’s an 80% chance it will be raining in the next hour and a 20% chance it will be sunny in the next hour. The arrows leading out of each node sum to one, because it will always be doing something in the next hour.
This whole structure conforms to what’s known as the Markov property, where the only factor that determines the probability of future states is the present state. To put it another way, the model is memory-less. To predict the state in an hour with a Markov chain, all we need to know is the current state.
How can I use it to make fun of my friends?
To connect this to the problem of generating the sentence in a particular style, we need to change what a state means and what a transition represents. What would it look like if we view a sentence as a sequence of states. The current state is the last word typed, and the next state is the next word. The transition probabilities represent the probabilities of saying each word next. We’d be treating a sentence as a kind of semi-random walk through word space.
A Markov chain for sentences
Imagine you heard me say
I have a doctor's appointment on _____. You’ve got a pretty good idea of what I’m about to say. Almost certainly the name of a day, and probably a weekday. Maybe I’ll say
the on my way to
the twenty-fifth of July. There’s an outside chance I’ll surprise you by saying
a boat or
Aquidneck Island. Still, you can narrow it down a lot.
Problem is, the Markov model only gets to look at the current state, which we said was just the last word. In this example that would be
on _____. With this restriction, you’re probably much less sure of what I’m about to say, but there’s still some information here. The next word is probably not another proposition like
at, for example.
Well, we can play kind of a funny trick and decide to call the last two words the state. Doesn’t this break the Markov property? No, the transition probabilities still only depend on the current state, we’ve just changed the definition of a state. Now the first state is
I have and the second state is
have a. The final state is
appointment on. This state actually has a lot of information about what the next word will be, almost as much as the full sentence.
There’s a tradeoff to opening up the state like this, and it has to do with the transition probabilities. There are way more of them now! As the number of words in the state increases, the number of possible states grows exponentially and the number of possible transitions does too.
Estimating the probabilities
Having a large number of states is a problem, because we’re going to have a finite amount of data to estimate these probabilities from. Let’s see what that looks like. A simple way to find these probabilities is to look at a corpus of text, find all the number of times each state ocurred, and count the number of times each possible transition actually happened. It’s a little silly to build a Markov chain from a single sentence, but let’s use that as the corpus, and take a look at the graph with a two word state and the above strategy for calculating transition probabilities.
This graph is pretty boring. Each state occurs exactly once, so the graph is linear. We only have one example of a transition from
I have, and it’s to
have a, so the estimated probability of that transition is one.
What if we use this corpus of four sentences instead:
My favorite color is red
My favorite color is blue
My favorite color is green
Your favorite color is blue
Now the graph would look like:
This Markov chain is a bit more interesting and illustrates the probability calculation. Since we have two instances of
color is being followed by
is blue, it is twice as probable in the Markov chain. There are also two states that lead to the
favorite color state, each with probability one.
This model doesn’t reflect the fact that three of the four sentences start with
My. We can solve that by adding special start and end tokens to the beginning and endings of the sentence. Effectively, just pretend that each sentence looks like this:
START START My favorite color is red END END
We need two of each because our states are two words.
Generating a new sentence from a Markov chain is simple. Start at the
START START state and pick the next state according to the listed probabilities. This Markov chain could generate a sentence that doesn’t appear in the input list. It has 12.5% chance of generating this sentence, for example:
Your favorite color is red
Markov Chain for Conversations
So far we’ve been generating sentences, but our Slackbot generates conversations between multiple people instead. We considered different ways of doing this, but settled on a method that I think is kind of clever. We replace each word with a tuple representing the word and the speaker. We also added an extra token indicating that a single message has ended. A conversation like this:
Bob: Hey there
Is now represented with these tokens:
Hey is now a completely different token from Bob saying
Hey. Instead of generating sentences by generating sequences of words, we can generate conversations as sequences of these tuple tokens. Here’s what a model for this one sentence would look like:
This graph looks little uglier, but it can encode a bunch of information we care about. Most importantly, it has the power to bridge the gap between individual messages so that they can be related to each other. It also captures when the pattern people choose to break up their messages. For example,
I don’t know
is very different from
SEND tokens allow us to encode that.
The downside of the tuple token approach is, of course, that it increases the number of states and possible transitions. We tried to limit this effect by ignoring case and punctuation on the output side of nodes. For example, these conversations:
Bob: Hey there
Bob: Hey there
would lead to this Markov chain:
In this case, Alice retains the ability to say both
HEY, but Bob doesn’t care about that distinction.
Want to try it out? Checkout our GitHub repo.»
Declan, Max, Phil and I have been working on this project for about a year. I’m pretty proud of it.
We built Reality as a Service, a free hosted robotics platform. From building robots like ZIPY, we know that reality can be very different from simulation, even for a simple system. RaaS allows everyone to test out control theory or reinforcement learning code on real hardware with features you’d find in the real world like latency, friction, and imperfect measurement. Until now, the only way to do this was to build your own robot. And who wants to do that?
What is RaaS?
Our system is based on OpenAI Gym. Gym is a really cool project. It provides simulations of some simple interactive physical systems and exposes a common interface for controlling inputs and measuring the state. Our first robots are designed to match Gym’s
Pendulum-v0 environment, which allows users to control the torque on the pivot of a simple pendulum with the goal of balancing it in the inverted position.
RaaS is really easy to use, especially if you’re already familiar with Gym.
import gym import gym_raas # import our extra RaaS Pendulum environment env = gym.make('RaasPendulum-v0') # initialize the RaaS Pendulum environment env.reset() for _ in range(1000): # run for 1000 steps env.render() action = my_controller.get_action(observation) # <-- 🚨 your controller goes here 🚨 observation, reward, done, info = env.step(action) # execute the chosen action env.close()
gym_raas adds a special new environment called
RaasPendulum-v0 to the stock
gym module. This environment’s interface is the same as
Pendulum-v0. When code like this is run on our hardware, it controls a DC motor mounted to the base of the pendulum and reads back position and angular velocity information from a rotary encoder. When run on other computers, the code controls a simulation tuned to approximately match our hardware. Users can develop their code locally using the simulation and then test it on RaaS.
How do I try it?
You can try it by going to raas.perciplex.com and submitting a job. All you have to do is provide the URL to a Github project with a controller defined in a function called
run.py in the main branch. We wrote an example controller that’s easy to try right away:
https://github.com/perciplex/raas-example. If you want to write your own code, you can start by forking our starter repository.
After you submit a job, it will be added to the queue. Once your job is complete, you’ll be able to see a graph of the results, an animation showing you what happened, read the standard out, and download the data.
How does it work?
The center of the RaaS universe is a Flask server running on AWS Elastic Beanstalk. It maintains a list of queued, running, and completed jobs. Jobs can be added the the queue and viewed using a web app built with React.
The robots are Raspberry Pi 4s connected to motor with built in rotary encoders. Each robot continually polls the server for jobs. If one is waiting in the queue, that job is popped and assigned to the requesting robot. It spins up a Docker container, downloads the user’s code from Github, and attempts to run the code. When it completes or crashes, the data is saved to AWS S3, and metadata about the run is returned to the server, which marks the job as complete. The block diagram below outlines the system.
Go try it out!
Seriously, we’re begging you. Go try it out. 🙏»
I keep hearing about small delays in the vaccine development process that I don’t understand given how urgent the need is. I read a perfect example today. A New York Times article sharing good news about the effectiveness of the Pfizer vaccine mentioned that the company would be applying for emergency use authorization “within days”. A similar article cited a period of weeks for the Moderna vaccine. This progress is great news. It made me wonder, though, why not today? What prevented these companies from being ready to file with the FDA right away?
By all accounts, the rapid progress of COVID-19 vaccine candidates has been miraculous. Still, I’m confused by the apparent lack of urgency in certain parts of the process. Most of the vaccine development work is probably difficult to speed up. For example, it isn’t possible to speed up the rate at which trial participants get exposed to the virus so that effectiveness can be proven1. But what prevents Pfizer and Moderna from pre-writing their application to the FDA while the trial is running and netting a few days or weeks? Even just a few days has serious value. Of course, the data is still coming in so there would be some uncertainty about what should be written in the application. But they must have a pretty good sense of how the trial is going. They could even pre-write two or three versions to cover the most likely scenarios. And of all possible trial outcomes, why would these companies both be unprepared for best-case scenarios where their vaccines are proving to be highly effective?
Getting a bit more speculative, why aren’t scientists at Pfizer and Moderna already in conversation with decision makers at the FDA so that the evaluation process can be started based on preliminary data? Then they could be possibly be ready to issue emergency use authorization immediately once the trial concludes. If there is an unexpected change in the latest data regulators should of course be prepared to reassess. To be clear, I am not suggesting that the approval process be less rigorous. I just want it happening in parallel with the trial as much as it can. Given how ridiculously destructive this pandemic is, I sure hope COVID vaccines are the FDA’s top priority.
This possibility reminds me of the mechanics of my thesis defense. My adviser, the main decision maker, was already very familiar with my work. Little information was actually transmitted to him that day. He already knew what he needed to know to make his decision. In fact, we wouldn’t have even gone forward with the defense if he wasn’t ready to pass me. He didn’t take a month after the defense to closely read my thesis, consider all the arguments, check the math, and then deliberate. That process occurred while the data was coming in. Maybe some parallelization of the approval process like this is happening for the COVID vaccines, but I’ve never read anything that gives the sense that it is.
Possible answers to my own questions:
Why weren’t the EUA applications ready?
Maybe a document like this is way more complicated than I’m imagining and the drug companies are writing them as fast as possible. This sort of suggests that writing the application is a similar challenge to developing the vaccine. I hope that’s not right.
Maybe the drug companies are already working with the FDA, and the timing of the application isn’t important. As I mentioned above, I’ve never seen anything that suggests this.
Maybe these companies aren’t incentivized to actually get the vaccine to market as fast as they can. Off the bat this didn’t seem right to me, but could the advance purchase agreements that Moderna and Pfizer signed with governments eliminate their urgency?
Why aren’t the drug companies working with the FDA?
Maybe this just isn’t the way the FDA usually works and they weren’t able to adapt to this extraordinary situation.
Obviously scientific integrity is important here. Maybe the scientists are intentionally kept separate from other parts of Pfizer and from the FDA to prevent some kind of academic dishonesty. I wonder about the relative dangers involved, but I could imagine an explanation like this.
Answers that other people have suggested:
Commenters on Hacker News and Reddit have suggested that the things I want to happen are already happening. I’ve looked into this a little bit, and I’m unsure. Both Moderna and Pfizer have been granted the FDA’s Fast Track status. This sounds great. It apparently means these companies have the opportunity for closer collaboration with the FDA, at least on clinical trial design. Some have suggested that the companies would also already be sharing advance results from their studies, but I haven’t found any reference saying that.
Fast Track also gives them the opportunity to participate in Rolling Review, which allows them to submit sections of their applications as they become ready. The references I’ve found discussing this are mostly press releases by the companies or closely based on them. They discuss participation in Rolling Review in Europe and elsewhere, but I haven’t seen that they are doing this in the US.»
I've become interested in hammock camping lately. I was inspired by Shug, a YouTuber who hikes and camps with a hammock and has mastered the art of enjoying himself in the woods.
There are a lot of ways to hang a hammock. The crazy array of options is pretty confusing at first, but I think that's actually part of the appeal. Picking your system and dialing it all in is exciting and makes the experience more personal.
One of the most baffling items, and one that comes up a lot, is something called a whoopie sling. In videos and diagrams it just looks like a funky rope. It's not clear what it even does. I eventually figued it out. It's really just an adjustable length rope with a loop on each end. You use it to connect some kind of tree strap to one end of your hammock. You can make one by tying a few knots in a hollow-cored rope. People generally use 7/64" Amsteel rope. The adjustable length is achieved by looping one end of the line and running it through itself, creating what's called a bury. When you pull on either end, the rope around the buried section contracts down and binds. It's the same mechanism by which a Chinese finger trap grabs a finger.
When I was making my own, I kept looking for a good diagram. The construction is really simple when it's in your hands but a bit confusing on screen. I couldn't find a good diagram, so I made one. I hope it's helpful!
I also made one for a continuous loop, which people use to bundle up the fabric of their hammock and create a loop to connect to.
Zach wrote the music, Matt made the art, I wrote the code, and we all designed the game. I'm very proud of how it came out.
The theme of the jam was "Keep it alive". We built an abstract survival game called Good Shapes, Bad Shapes, where you play a glowing circle trying to avoid a set of evil polygons. The game is dark, mostly illuminated by light emitted by the player. The player's energy fades slowly over time, so to stay alive they must collect the glowing orbs which dart erratically around the level. While scrambling after these orbs in the dark, it's really easy to run into one of these assholes:
|The triangle rotates to track the player and quickly shoots at them when locked on.|
|The square relentlessly pursues the player by moving side to side or up and down.|
|The pentagon teleports repeatedly and unpredictably to locations in a ring around surrounding player.|
|The hexagon bounces around the map and suddenly expands when a player gets near.|
All of these shapes resemble the walls of the map itself, making it hard to tell what's safe and what isn't. I think the game works well, hard as it is. I've never gotten to 60 seconds.»