Execute and cache your pages¶
Jupyter Book can automatically run and cache any notebook pages. Notebooks can either be run each time the documentation is built, or cached locally so that notebooks will only be re-run when the code cells in a notebook have changed.
Caching behaviour is controlled with the execute: section in your _config.yml file. See
the sections below for each configuration option and its effect.
:::{tip}
If you’d like to execute code that is in your Markdown files,
you can use the {code-cell} directive in MyST Markdown.
See Notebooks written entirely in Markdown for more information.
:::
Trigger notebook execution¶
By default, Jupyter Book will execute any content files that have a notebook structure and that are missing at least one output. This is equivalent to the following configuration in _config.yml`:
execute:
execute_notebooks: auto
This will only execute notebooks that are missing at least one output. If the notebook has all of its outputs populated, then it will not be executed.
To force the execution of all notebooks, regardless of their outputs, change the above configuration value to:
execute_notebooks: force
To cache execution outputs with jupyter-cache, change the above configuration value to:
execute:
execute_notebooks: cache
See Caching the notebook execution for more information.
To turn off notebook execution, change the above configuration value to:
execute:
execute_notebooks: 'off'
Exclude files from execution¶
To exclude certain file patterns from execution, use the following configuration:
execute:
exclude_patterns:
- 'pattern1'
- 'pattern2'
- '*pattern3withwildcard'
Any file that matches one of the items in exclude_patterns will not be executed.
:::{tip} To auto-exclude all files outside of your table of contents, see Disable building files that aren’t specified in the TOC :::
Caching the notebook execution¶
You may also cache the results of executing a notebook page using jupyter-cache.
In this case, when a page is executed, its outputs will be stored in a local database.
This allows you to be sure that the outputs in your documentation are up-to-date,
while saving time avoiding unnecessary re-execution.
It also allows you to store your .ipynb files in your git repository without their outputs,
but still leverage a cache to save time when building your site.
When you re-build your site, the following will happen:
Notebooks that have not seen changes to their code cells since the last build will not be re-executed. Instead, their outputs will be pulled from the cache and inserted into your site.
Notebooks that have had any change to their code cells will be re-executed and the cache will be updated with the new outputs.
To enable caching of notebook outputs, use the following configuration:
execute:
execute_notebooks: cache
By default, the cache will be placed in the parent of your build folder.
Generally, this is in _build/.jupyter_cache.
You may also specify a path to the location of a jupyter cache you’d like to use:
execute:
cache: path/to/mycache
The path should point to an empty folder, or a folder where a jupyter cache already exists.
Execution configuration¶
You can control notebook execution and how output content is handled at a project level using your _config.yml but, in some cases, also at a notebook and code cell level.
Below we explore a number of ways to achieve this.
:::{seealso} How should I add cell tags and metadata to my notebooks? and Formatting code outputs. :::
The execution working directory¶
:::{important}
The default behaviour of cache is now to run in the local directory.
This is a change from v0.7.
:::
By default, the command working directory (cwd) in which a notebook runs will be the directory in which it is located (for both auto and cache). This means that notebooks requiring access to assets in relative paths will work.
Alternatively, if you wish for your notebooks to isolate your notebook execution in a temporary folder,
you can use the following _config.yml setting:
execute:
run_in_temp: true
Setting execution timeout¶
Execution timeout defines the maximum time (in seconds) each notebook cell is allowed to run for. If the execution takes longer an exception will be raised. The default is 30 seconds, so in cases of long-running cells you may want to specify a higher value. The timeout option can also be set to -1, to remove any restriction on execution time.
You can set the timeout for all notebook executions in your _config.yml:
execute:
timeout: 100
This global value can also be overridden per notebook by adding this to your notebook metadata:
{
"metadata": {
"execution": {
"timeout": 30
}
}
Dealing with code that raises errors¶
In some cases, you may want to intentionally show code that doesn’t work (e.g., to show the error message).
You can allow errors for all notebooks in your _config.yml:
execute:
allow_errors: true
This global value can also be overridden per notebook by adding this to your notebook metadata:
{
"metadata": {
"execution": {
"allow_errors": false
}
}
Lastly, you can allow errors at a cell level, by adding a raises-exception tag to your code cell.
This can be done via a Jupyter interface, or via the {code-cell} directive like so:
```{code-cell}
---
tags: [raises-exception]
---
print(thisvariabledoesntexist)
```
Which produces:
print(thisvariabledoesntexist)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-1-125c53ec1b82> in <module>
----> 1 print(thisvariabledoesntexist)
NameError: name 'thisvariabledoesntexist' is not defined
Dealing with code that produces stderr¶
You may also wish to control how stderr outputs are dealt with.
Alternatively, you can configure how stdout is dealt with at a global configuration level, using the nb_output_stderr configuration value.
You can configure the default behaviour for all notebooks in your _config.yml:
execute:
stderr_output: show
Where the value is one of:
"show"(default): show all stderr (unless aremove-stderrtag is present)"remove": remove all stderr"remove-warn": remove all stderr, but log a warning if any found"warn","error"or"severe": log the stderr at a certain level, if any found.
You can also remove stderr at a cell level, using the remove-stderr cell tag, like so:
```{code-cell} ipython3
:tags: [remove-stderr]
import sys
print("this is some stdout")
print("this is some stderr", file=sys.stderr)
```
which produces
import sys
print("this is some stdout")
print("this is some stderr", file=sys.stderr)
this is some stdout
Dealing with code that produces stdout¶
Similar to stderr, you can remove stdout at a cell level with the remove-stdout tag, by which
```{code-cell} ipython3
:tags: [remove-stdout]
import sys
print("this is some stdout")
print("this is some stderr", file=sys.stderr)
```
produces the following:
import sys
print("this is some stdout")
print("this is some stderr", file=sys.stderr)
this is some stderr
Execution statistics¶
As notebooks are executed, certain statistics are stored on the build environment by MyST-NB.
The simplest way to access and visualise this data is using the {nb-exec-table} directive.
:::{seealso} The MyST-NB documentation, for creating your own directives to manipulate this data. :::
The simple directive
```{nb-exec-table}
```
produces:
Document |
Modified |
Method |
Run Time (s) |
Status |
|---|---|---|---|---|
TimeSeriesData/1_Data_Analysis_Cycle |
2021-01-17 22:06 |
auto |
0.96 |
✅ |
TimeSeriesData/3_Algorithms_ML_TS_Linear |
2021-01-17 22:06 |
auto |
8.64 |
❌ |
TimeSeriesData/4_Algorithms_TS_NonLinear |
2021-01-17 22:06 |
auto |
2.39 |
❌ |
TimeSeriesData/Practice/3_Setting_Analysis |
2021-01-17 22:06 |
auto |
6.33 |
❌ |
TimeSeriesData/Practice/4_FE_Analysis |
2021-01-17 22:06 |
auto |
9.31 |
❌ |
TimeSeriesData/Practice/5_Agile_Analysis |
2021-01-17 22:06 |
auto |
16.06 |
❌ |
TimeSeriesData/Practice/6_TimeSeries_Analysis |
2021-01-17 22:07 |
auto |
13.16 |
❌ |
TimeSeriesData/Practice/Basic/1_Tutorial_Pandas |
2021-01-17 22:07 |
auto |
1.66 |
❌ |
TimeSeriesData/Practice/Basic/2_Tutorial_Numpy |
2021-01-17 22:07 |
auto |
1.36 |
❌ |
content/citations |
2021-01-17 22:07 |
auto |
0.9 |
✅ |
content/code-outputs |
2021-01-17 22:07 |
auto |
4.56 |
✅ |
content/execute |
2021-01-17 22:07 |
auto |
1.11 |
✅ |
content/layout |
2021-01-17 22:07 |
auto |
1.8 |
✅ |
content/math |
2021-01-17 22:07 |
auto |
0.94 |
✅ |
file-types/myst-notebooks |
2021-01-17 22:07 |
auto |
0.9 |
✅ |
file-types/notebooks |
2021-01-17 22:07 |
auto |
2.29 |
❌ |
interactive/hiding |
2021-01-17 22:07 |
auto |
1.89 |
✅ |
interactive/interactive |
2021-01-17 22:07 |
auto |
2.45 |
❌ |
interactive/launchbuttons |
2021-01-17 22:07 |
auto |
1.53 |
✅ |
reference/cheatsheet |
2021-01-17 22:07 |
auto |
2.97 |
✅ |
start/overview |
2021-01-17 22:07 |
auto |
2.34 |
✅ |