Introduction

The current generation, namely the 2nd generation of artificial neural networks, have made a tremendous breakthrough in numerous fields. Although a wide range of problems is being solved by artificial neural networks, more precisely by deep neural networks, they are biologically inaccurate and do not exhibit the actual mechanisms of our brain’s neurons.

Spiking neural networks are considered as the 3rd generation of neural networks. SNNs are fundamentally different than ANNs. Neurons send out short pulses/spikes of electrical energy as signals if they have received enough of these themselves. These spikes are discrete events that take place at points in time, rather than continuous values. Essentially, once a stimulated neuron reaches a certain potential, it spikes and the potential of that neuron is reset.

Biological Background [1]

Note: Mostly adapted from Duane Nykamp's videos, slides, and tutorials. Hand drawn figures are recreated for visual understanding.

A neuron is a particular type of cell in our body and like all cells, it is surrounded by a thin membrane. This membrane does a lot of things. It controls what goes in and out of the cell. There's a lot of water both in and outside the cell and in this water are all kinds of different substances. For our neuron model, we're going to concern ourselves with two of these. One of these is sodium ion (Na+). That’s in the water, both in and outside the cell. It turns out that the concentration of sodium ions is higher outside the cell than inside the cell. The cell membrane is impermeable to sodium ions. So, usually the sodium ions can't get through but there are specific channels in the neuron membrane that allow sodium ions to go through. These channels are specific to sodium they only allow sodium ions to go through. They can open and close. What will happen if we have a sodium channel that is open, is that sodium can flow in and out of the cell. Although it can flow in both directions because the concentration of sodium is higher outside the cell it will tend to flow into the cell. Now, because there's a difference in ion concentrations between the inside and outside of the cell, there's a voltage difference between the inside and outside. It turns out that it's more negative inside the cell. What happens to the voltage inside the cell when sodium ions rush into the cell, that will make the voltage in the cell go up. So, if the sodium ion channels open the voltage inside the cell goes up.

Another important ion that is both inside and outside the cell is potassium. Potassium which we denote by K+ is also positively charged. Potassium plays a different role than sodium, because the concentration of potassium is higher inside the cell than outside the cell. The cell membrane also has channels that are specific to potassium. Let's draw the potassium channels in blue and sodium channels green. The potassium channels also open and close which will illustrate by putting doors on them. Since the potassium concentration is higher inside the cell, opening the potassium channels will have a different effect than the sodium channels. When a potassium channel is open, potassium will tend to flow out of the cell. Since positive charges flow out in this case, the voltage inside the cell will decrease.

There are obviously a lot more technical details behind this, but for the purposes of our simple model, this will be enough.

• Na+ channels open → the voltage inside increases

• K+ channels open → the voltage inside decreases

Note: The important things to remember is that there's more sodium outside the cell so when sodium channels open the voltage inside the cell increases and there's more potassium inside the cell so that when potassium channels open the voltage inside the cell decreases.

Alan Lloyd Hodgkin and Andrew Huxley, two physiologists described in the 1950s some important features of how the sodium and potassium channels operate.

First of all when a cell is at rest both the sodium and the potassium channels are mainly closed but the sodium channels do something strange. We've already established that when these channels open it tends to increase the voltage but it turns out that the reverse is also true. Increasing the voltage inside the cell also tends to open sodium channels. This leads to some interesting behavior. So let's imagine that we increase the voltage (Neurons have other channels that might lead the voltage to go up, mainly if they receive a signal from another neuron.) That might lead to more sodium channels opening and what happens if more sodium channels open? Well, that will tend to increase the voltage which will then, in turn, tend to make more sodium channels open. What's going on here, that we have a positive feedback loop.

As you can imagine such positive feedback loops could lead to things getting out of control. The voltage inside the cell would keep getting higher and higher as more and more channels open leading to more sodium rushing inside the cell.

It turns out that, although the sodium channels can only increase the voltage. So far the voltage inside the cell is normally negative and eventually, sodium channels can make the voltage even reach positive values, but not a whole lot more than that. The way these sodium channels work is that when a cell is at rest the sodium channels don't open enough to get this positive feedback loop going. There's a threshold voltage which must be reached in order to trigger the positive feedback loop. If the voltage is below the threshold, the voltage goes back down to resting levels. But if somehow we manage to kick the voltage above the threshold then we trigger the positive feedback loop.

The sodium channels rapidly open and the voltage rapidly shoots upward and if it weren't for the potassium channels the voltage would stay up at this high level indefinitely. But before we get into the effect of the potassium channels it's worthwhile to first study the effect of the sodium channels by themselves.

What we'll do next is model the effect of the sodium channels on the voltage. We'll see if we can model this threshold effect of voltage using a single autonomous differential equation.

Considering sodium channels only

We have learned three important properties of the sodium channels. First, since the concentration of sodium ions is higher outside the cell, sodium rushes into the cell when the sodium channels open, increasing the voltage inside the cell. Second, since sodium channels tend to open when the voltage increases, they create a positive feedback loop. Increasing the voltage leads to more sodium channels opening, which leads to an increased voltage, etc. This positive feedback loop is fast, which can lead to the voltage shooting up rapidly. However, there is a threshold voltage which the voltage must cross before the positive feedback loop is triggered. If the voltage starts below the threshold, it will stay low or even decrease. Only if the voltage is somehow kicked above a threshold, say due to input from other neurons, is the positive feedback loop engaged, and the voltage shoots up fast.

Using the three properties of the sodium channels, let's create a dynamical system model of a neuron in the form an autonomous differential equation. For simplicity, we won't model the sodium channels directly, but just model the resulting dynamics of the neuron's voltage.

For now, our dynamical system model will have one state variable v(t), which will be the voltage of the neuron at time t. Well, let's not make it exactly the real voltage of the neuron, but a fake voltage to make the equation easier. Here's why we'll use a fake voltage.

When a neuron is at rest, its voltage is near a resting potential, which could be something like -70mV. The negative value means that the voltage inside the neuron is lower than the voltage outside. But, to make the model simpler, let's forget about this fact, and just pretend the resting voltage is zero.

Also, if the sodium channels were all the way open, the voltage would go to some high value (something like positive 40 mV). But, let's pretend that this voltage is 1. That way, our voltage will live between 0 and 1. It isn't the real voltage but just a rescaled voltage that tells us where we are between the minimum and maximum voltage.

There's one more voltage we need to add to our model: the threshold voltage. It is somewhere between the resting potential and the maximum voltage. (In real life, it's much closer to the resting potential). Let's leave it as an unspecified parameter and call the threshold voltage a.

To model the influence of the sodium channels on the voltage, here's what we want our model to do. If the voltage is below a, then the voltage should decrease back down to the resting potential 0. However, if the voltage somehow gets kicked above v=a, , then the voltage should increase to 1.

If we write the differential equation model in the form

$$\frac{dv}{dt} = f(v)$$

what should the function f(v) look like to have these properties? For starters, what should the phase line look like?

We definitely want stable equilibria at v=0 and v=1. The threshold voltage v=a should be an unstable equilibrium between them. We want the vector field, or direction arrows pointing away from a and toward both 0 and 1. In this way, the voltage will move upward if it is above a and downward otherwise (assuming we stay between 0 and 1).

To get this behavior in the phase line, what should the function f look like? f should be positive in the interval where v is between a and 1 and should be negative elsewhere. The simplest form of such an f is a cubic $f(v) = -v(v-a)(v-1)$.

With this function f , the differential equation is

$$\frac{dv}{dt} = -v(v-a)(v-1)$$.

For this model, can you verify that the equilibria are 0, a, and 1, that the equilibria 0 and 1 are stable and the equilibrium v=a is unstable?

The neuron model has the required behavior. Let's imagine that the neuron starts at rest. If it then receives an input that bumps its voltage upward, but the voltage stays below a, what happens? The voltage decays back down to zero. (Since we aren't modeling the inputs, just imagine that you change the voltage by hand and start it at a new value.)

If the neuron receives a larger input so that its voltage is kicked above a, what happens? The voltage increases toward 1. In this way, the neuron responds with a large deflection of its voltage only if it receives a big enough input.

According to this model, the neuron's voltage will stay at a high value forever, approaching the value of 1 as time increases, since v=1 is a stable equilibrium.

In reality, though, the voltage should just go up briefly and then come back down. We want it to go back down so that it can respond to future inputs.

Unfortunately, we cannot get this behavior with our one-dimensional autonomous differential equation model. In such a model, the state variable can never turn around, as it can only move in one direction at each point in the phase line.

The secret to bringing the voltage back down lies with the potassium channels, which we have ignored so far. When we add the potassium channels to our model, we will add a second variable and a second differential equation. With such a two-dimensional state space, we will be able to make the voltage turn around and go back down, and maybe even have other behaviors.

Considering both potassium and sodium channels

So far, we've built a one-dimensional autonomous differential equation model that just incorporates the effects of sodium. In the model,

$$\frac{dv}{dt} = -v(v-a)(v-1)$$

the voltage lives between zero and one. If it starts below the threshold v=a, the voltage decays to zero. If it somehow gets kicked above the threshold, such as via receiving enough input from other neurons, then the voltage rises to one and stays there.

We want to bring the voltage back down so that the neuron can be ready to listen to more input. To allow the voltage to turn around, we must add another state variable so that we have a two-dimensional differential equation. We'll add a variable that represents the state of potassium channels.

Since the concentration of potassium is higher inside the cell than outside, when the potassium channels open, potassium rushes out of the cell and the voltage inside goes down. Hodgkin and Huxley determined that the potassium channels share an important property with the sodium channels: potassium channels tend to open when the voltage inside the cell increases. In this case, since potassium channels bring the voltage down, the potassium channels act as negative feedback. If the voltage increases, potassium channels will open, which will counteract the voltage change, lowering the voltage.

There's another way in which potassium channels differ from sodium channels. Sodium channels are fast, opening quickly in response to increased voltage. Potassium channels are a bit more sluggish. When the voltage increases, potassium channels take their time in the opening, allowing the fast positive feedback of the sodium channels to cause the voltage to shoot up quickly. But then, after a delay -- those lazy potassium channels take a whole millisecond or so to wake up -- the potassium channels get their act together and open up. The result is that they can finally overpower the sodium channels and bring the voltage back down. (The potassium channels actually get some help from the sodium channels tiring out and closing back up, but we aren't modeling that.)

Let's see if we can add this effect of potassium channels to our model so that, by the combination of sodium and potassium, we can get this transient upward and downward excursion of voltage, that is a spike, in response to an input.

First, we need another state variable to represent the state of the potassium channels. We'll use the variable w(t) to indicate how many potassium channels are open at time t. Let's not be precise about what the actual values of w are; we could say it's proportional to the number of open potassium channels.

Next, we need to come up with a reasonable form for the change in w , i.e., $\frac{dw}{dt}$. $\frac{dw}{dt}$ should depend on voltage so that w will increase when v gets large. For simplicity, let's make the equation for w be a linear differential equation that depends on both w and v. As a first stab, let's try an equation of the form

$$\frac{dw}{dt} = v-\gamma w$$

where $\gamma$ is a positive parameter.

If we pretend for a moment that v is a constant number, we can analyze this equation. It has one equilibrium: $w = v/\gamma$. Do you know how to show that this equilibrium is stable? (Remember $\gamma$ is positive, so the derivative of the right-hand side with respect to w will always be $-\gamma$, a negative number.)

What the equation $\frac{dw}{dt} = v-\gamma w$ does is make w(t) evolve toward the number $v(t)/\gamma$. You can think of $v/\gamma$ as a target for w that w is chasing. Since v is not really a fixed number but instead evolves with time, this target will be a moving target for w(t).

Another property of the potassium channels is that they are slow. To make w(t) slow, we need $\frac{dw}{dt}$ to be small. How do we make it small? We can simply multiply the expression for $\frac{dw}{dt}$ by a small number. If we call that small number ε, then we can write the equation for w as

$$\frac{dw}{dt} = \varepsilon (v-\gamma w)$$

With this addition of ε, w(t) still chases $v(t)/\gamma$, but it just chases it more slowly.

We've introduced two positive parameters. ε controls the speed of w(t), and $\gamma$ controls how strongly the potassium channels depend on voltage.

The model now captures how potassium channels respond to an increased voltage by slowly opening. To complete the negative feedback effect of the potassium channels, we have to include in our equations the influence of potassium on voltage. Remember, when the potassium channels open, they cause the voltage to go down. In other words, they have a negative effect on the change in voltage of $\frac{dv}{dt}$. Consistent with our simple-minded manner in deriving these equations, let's just subtract off w from the $\frac{dv}{dt}$ equation. The equation for v becomes

$$\frac{dv}{dt} = -v(v-a)(v-1) - w$$

Combining the $\frac{dv}{dt}$ equation with the $\frac{dw}{dt}$ equation, we now have a coupled system of two autonomous differential equations for our state variables v and w. They are coupled because the equation for v depends on w and the equation for w depend on v. These equations for the activity of a neuron are called the Fitzhugh-Nagumo equations, which are a simplified version of the Hodgkin-Huxley equations for a spiking neuron.

Our remaining task is to analyze this two-dimensional system of equations and see what kind of behavior we'll get from it. We can show how, for example, you can get a single spike of voltage from an input, where the voltage shoots up and then goes back down. But, we'll stop here for now.

References

[1] Nykamp DQ, “A simple spiking neuron model.” From Math Insight. http://mathinsight.org/simple_spiking_neuron_model