Skip to content

promote: refresh 1 pool lectures whose canonical source moved #26

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
github-actions wants to merge 1 commit into
base: main
Choose a base branch
from
Open

promote: refresh 1 pool lectures whose canonical source moved #26

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 70 additions & 61 deletions lectures/markov_jump_lq.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ $$
The problem is

$$
-x_0' P x_0 - \rho = \min_{\{u_t\}_{t=0}^\infty} E \sum_{t=0}^{\infty} \beta^t r(x_t, u_t)
x_0' P x_0 + \rho = \min_{\{u_t\}_{t=0}^\infty} E \sum_{t=0}^{\infty} \beta^t r(x_t, u_t)
$$

subject to the transition law for the state.
Expand All @@ -112,7 +112,7 @@ where $P$ solves the algebraic matrix Riccati equation

$$
P = R+ \beta A' P A
-(\beta B' P A + W)' (Q + \beta B P B )^{-1} (\beta B P A + W)
-(\beta B' P A + W)' (Q + \beta B' P B )^{-1} (\beta B' P A + W)
$$

and the constant $\rho$ satisfies
Expand Down Expand Up @@ -197,27 +197,31 @@ $$
-x_t' P_i x_t - \rho_i
$$

The optimal value functions $- x' P_i x - \rho_i$ for
$i = 1, \ldots, n$ satisfy the $N$ interrelated Bellman equations
The losses $x' P_i x + \rho_i$ for
$i = 1, \ldots, N$ satisfy the $N$ interrelated Bellman equations

$$
\begin{aligned}
-x' P_i x - \rho_i & = \max_u -
x'R_i x & + u' Q_i u + 2 u' W_i x +
\beta \sum_j \Pi_{ij}E ((A_i x + B_i u + C_i w)' P_j
(A_i x + B_i u + C_i w) x + \rho_j)
\end{aligned}
x' P_i x + \rho_i = \min_u \Big\{ x'R_i x + u' Q_i u + 2 u' W_i x +
\beta \sum_j \Pi_{ij} \, \mathbb{E} \big[ (A_i x + B_i u + C_i w)' P_j
(A_i x + B_i u + C_i w) + \rho_j \big] \Big\}
$$

The decision maker chooses $u_t$ before it learns $s_{t+1}$.

Next period's value matrices enter through their conditional expectation

$$
\bar P_i = \sum_j \Pi_{ij} P_j .
$$

The matrices $P_{s_t} = P_i$ and the scalars
$\rho_{s_t} = \rho_i, i = 1, \ldots$, n satisfy the following stacked system of
$\rho_{s_t} = \rho_i, i = 1, \ldots, N$ satisfy the following stacked system of
**algebraic matrix Riccati** equations:

$$
P_i = R_i + \beta \sum_j A_i' P_j A_i
\Pi_{ij}
-\sum_j \Pi_{ij}[ (\beta B_i' P_j A_i + W_i)' (Q + \beta B_i' P_j B_i)^{-1}
(\beta B_i' P_j A_i + W_i)]
P_i = R_i + \beta A_i' \bar P_i A_i
- (\beta B_i' \bar P_i A_i + W_i)' (Q_i + \beta B_i' \bar P_i B_i)^{-1}
(\beta B_i' \bar P_i A_i + W_i)
$$

$$
Expand All @@ -228,15 +232,17 @@ $$
and the $F_i$ matrices in the optimal decision rules are

$$
F_i = (Q_i + \beta \sum_j \Pi_{ij} B_i' P_j B_i)^{-1}
(\beta \sum_j \Pi_{ij}(B_i' P_j A_i )+ W_i)
F_i = (Q_i + \beta B_i' \bar P_i B_i)^{-1}
(\beta B_i' \bar P_i A_i + W_i)
$$

## Applications
The conditional expectation $\bar P_i$ appears inside the inverse.

Averaging the quadratic forms $(\beta B_i' P_j A_i + W_i)' (Q_i + \beta B_i' P_j B_i)^{-1} (\beta B_i' P_j A_i + W_i)$ over $j$ instead gives a different and incorrect recursion, except when next period's Markov state is known with certainty or when $Q_i + \beta B_i' P_j B_i$ and $B_i' P_j A_i$ do not vary with $j$.

We now describe Python code and some examples.
## Applications

To begin, we import these Python modules
We use the following imports.

```{code-cell} ipython
import numpy as np
Expand Down Expand Up @@ -440,15 +446,15 @@ u1_star = - ex1_a.Fs[0, 0, 1] - ex1_a.Fs[0, 0, 0] * k_grid
u2_star = - ex1_a.Fs[1, 0, 1] - ex1_a.Fs[1, 0, 0] * k_grid

fig, ax = plt.subplots()
ax.plot(k_grid, k_grid + u1_star, label=r"$\overline{s}_1$ (high)")
ax.plot(k_grid, k_grid + u2_star, label=r"$\overline{s}_2$ (low)")
ax.plot(k_grid, k_grid + u1_star, lw=2, label=r"$\overline{s}_1$ (high)")
ax.plot(k_grid, k_grid + u2_star, lw=2, label=r"$\overline{s}_2$ (low)")

# The optimal k*
ax.scatter([0.5, 0.5], [0.5, 0.5], marker="*")
ax.plot([k_star[0], k_star[0]], [0., 1.0], '--')
ax.plot([k_star[0], k_star[0]], [0., 1.0], '--', lw=2)

# 45 degree line
ax.plot([0., 1.], [0., 1.], '--', label="45 degree line")
ax.plot([0., 1.], [0., 1.], '--', lw=2, label="45 degree line")

ax.set_xlabel("$k_t$")
ax.set_ylabel("$k_{t+1}$")
Expand All @@ -462,7 +468,7 @@ for both Markov states $s_t$.

It also plots the 45 degree line.

Notice that the two $s_t$-dependent *closed loop* functions that
The two $s_t$-dependent *closed loop* functions that
determine $k_{t+1}$ as functions of $k_t$ share the same
rest point (also called a fixed point) at $k_t = 0.5$.

Expand All @@ -471,21 +477,26 @@ in which the adjustment cost is lower, makes $k_{t+1}$ a flatter
function of $k_t$ in Markov state $2$.

This happens because when $k_t$ is not at its fixed point,
$| u_{t,2}| > | u_{t,2} |$, so
$| u_{t,2}| > | u_{t,1} |$, so
that the decision-maker adjusts toward the fixed point faster when
the Markov state $s_t$ takes a value that makes it cheaper.

```{code-cell} python3
---
mystnb:
figure:
caption: Optimal path of capital
name: fig-mjlq-k-path
---
# Compute time series
T = 20
x0 = np.array([[0., 1.]]).T
x_path = ex1_a.compute_sequence(x0, ts_length=T)[0]

fig, ax = plt.subplots()
ax.plot(range(T), x_path[0, :-1])
ax.plot(range(T), x_path[0, :-1], lw=2)
ax.set_xlabel("$t$")
ax.set_ylabel("$k_t$")
ax.set_title("Optimal path of $k_t$")
plt.show()
```

Expand Down Expand Up @@ -540,21 +551,18 @@ for i, λ in enumerate(λ_vals):
```

```{code-cell} python3
for i, state_var in enumerate(state_vec1):
fig, ax = plt.subplots()
ax.plot(λ_vals, F1[:, i], label=r"$\overline{s}_1$", color="b")
ax.plot(λ_vals, F2[:, i], label=r"$\overline{s}_2$", color="r")

fig, axes = plt.subplots(1, len(state_vec1), figsize=(12, 4))
for i, (state_var, ax) in enumerate(zip(state_vec1, axes)):
ax.plot(λ_vals, F1[:, i], lw=2, label=r"$\overline{s}_1$", color="b")
ax.plot(λ_vals, F2[:, i], lw=2, label=r"$\overline{s}_2$", color="r")
ax.set_xlabel(r"$\lambda$")
ax.set_ylabel("$F_{s_t}$")
ax.set_title(f"Coefficient on {state_var}")
ax.set_title(f"coefficient on {state_var}")
ax.legend()
plt.show()
plt.tight_layout()
plt.show()
```

Notice how the decision rules’ constants and slopes behave as functions
of $\lambda$.

Evidently, as the Markov chain becomes *more nearly periodic* (i.e., as
$\lambda \rightarrow 1$), the dynamic program adjusts capital
faster in the low adjustment cost Markov state to take advantage of what
Expand Down Expand Up @@ -606,9 +614,9 @@ for i, λ in enumerate(λ_vals):
```

```{code-cell} python3
for i, state_var in enumerate(state_vec1):
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
fig, axes = plt.subplots(1, len(state_vec1), figsize=(12, 5),
subplot_kw={'projection': '3d'})
for i, (state_var, ax) in enumerate(zip(state_vec1, axes)):
# high adjustment cost, blue surface
ax.plot_surface(λ_grid, δ_grid, F1_grid[:, :, i], color="b")
# low adjustment cost, red surface
Expand All @@ -617,7 +625,8 @@ for i, state_var in enumerate(state_vec1):
ax.set_ylabel(r"$\delta$")
ax.set_zlabel("$F_{s_t}$")
ax.set_title(f"coefficient on {state_var}")
plt.show()
plt.tight_layout()
plt.show()
```

The following code defines a wrapper function that computes optimal
Expand All @@ -633,8 +642,7 @@ def run(construct_func, vals_dict, state_vec):
Qs, Rs, Ns, As, Bs, Cs, k_star = construct_func(**vals_dict)

# Symmetric Π
# Notice that pure periodic transition is a special case
# when λ=1
# pure periodic transition is the special case λ=1
print("symmetric Π case:\n")
λ_vals = np.linspace(0., 1., 10)
F1 = np.empty((λ_vals.size, len(state_vec)))
Expand All @@ -649,17 +657,16 @@ def run(construct_func, vals_dict, state_vec):
F1[i, :] = mplq.Fs[0, 0, :]
F2[i, :] = mplq.Fs[1, 0, :]

for i, state_var in enumerate(state_vec):
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(λ_vals, F1[:, i], label=r"$\overline{s}_1$", color="b")
ax.plot(λ_vals, F2[:, i], label=r"$\overline{s}_2$", color="r")

fig, axes = plt.subplots(1, len(state_vec), figsize=(4 * len(state_vec), 4))
for i, (state_var, ax) in enumerate(zip(state_vec, axes)):
ax.plot(λ_vals, F1[:, i], lw=2, label=r"$\overline{s}_1$", color="b")
ax.plot(λ_vals, F2[:, i], lw=2, label=r"$\overline{s}_2$", color="r")
ax.set_xlabel(r"$\lambda$")
ax.set_ylabel(r"$F(\overline{s}_t)$")
ax.set_title(f"coefficient on {state_var}")
ax.legend()
plt.show()
plt.tight_layout()
plt.show()

# Plot optimal k*_{s_t} and k that optimal policies are targeting
# only for example 1
Expand All @@ -669,17 +676,16 @@ def run(construct_func, vals_dict, state_vec):
for i in range(2):
F = [F1, F2][i]
c = ["b", "r"][i]
ax.plot([0, 1], [k_star[i], k_star[i]], "--",
ax.plot([0, 1], [k_star[i], k_star[i]], "--", lw=2,
color=c, label=r"$k^*(\overline{s}_"+str(i+1)+")$")
ax.plot(λ_vals, - F[:, 1] / F[:, 0], color=c,
ax.plot(λ_vals, - F[:, 1] / F[:, 0], color=c, lw=2,
label=r"$k^{target}(\overline{s}_"+str(i+1)+")$")

# Plot a vertical line at λ=0.5
ax.plot([0.5, 0.5], [min(k_star), max(k_star)], "-.")
ax.plot([0.5, 0.5], [min(k_star), max(k_star)], "-.", lw=2)

ax.set_xlabel(r"$\lambda$")
ax.set_ylabel("$k$")
ax.set_title("Optimal k levels and k targets")
ax.text(0.5, min(k_star)+(max(k_star)-min(k_star))/20, r"$\lambda=0.5$")
ax.legend(bbox_to_anchor=(1., 1.))
plt.show()
Expand All @@ -705,16 +711,17 @@ def run(construct_func, vals_dict, state_vec):
F1_grid[i, j, :] = mplq.Fs[0, 0, :]
F2_grid[i, j, :] = mplq.Fs[1, 0, :]

for i, state_var in enumerate(state_vec):
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
fig, axes = plt.subplots(1, len(state_vec), figsize=(4 * len(state_vec), 5),
subplot_kw={'projection': '3d'})
for i, (state_var, ax) in enumerate(zip(state_vec, axes)):
ax.plot_surface(λ_grid, δ_grid, F1_grid[:, :, i], color="b")
ax.plot_surface(λ_grid, δ_grid, F2_grid[:, :, i], color="r")
ax.set_xlabel(r"$\lambda$")
ax.set_ylabel(r"$\delta$")
ax.set_zlabel(r"$F(\overline{s}_t)$")
ax.set_title(f"coefficient on {state_var}")
plt.show()
plt.tight_layout()
plt.show()
```

To illustrate the code with another example, we shall set
Expand Down Expand Up @@ -778,7 +785,7 @@ run(construct_arrays1, {"f2_vals":[0.5, 1.]}, state_vec1)

## Example 2

We now add to the example 1 setup another state variable $w_t$
Example 2 adds to the example 1 setup another state variable $w_t$
that follows the evolution law

$$
Expand All @@ -795,7 +802,7 @@ $$
r(s_{t},k_{t},w_{t})=f_{1,s_{t}} k_{t}-f_{2, s_{t}} k_{t}^{2}-d_{s_{t}} (k_{t+1}-k_{t})^{2}-w_{t}k_{t},
$$

We now let the continuous part of the state at time $t$ be
The continuous part of the state at time $t$ is now
$x_{t}=\begin{bmatrix} k_{t}\\ 1\\ w_{t} \end{bmatrix}$
and continue to set the control $u_{t}=k_{t+1}-k_{t}$.

Expand Down Expand Up @@ -934,3 +941,5 @@ of optimal tax-smoothing and government debt in several interesting directions
1. {doc}`How to Pay for a War: Part 2 <tax_smoothing_2>`
1. {doc}`How to Pay for a War: Part 3 <tax_smoothing_3>`

{doc}`policy_regimes` uses the method to compute equilibria of a market in which a tax rule switches between regimes according to a Markov chain.

Loading