@educational-technology-collective/etc_jupyterlab_telemetry_coursera

Introduction

This README will discuss the installation and development of the combined extensions that comprise the ETC Coursera Telemetry Framework. Additional instruction may be found in the README files of each of the individual extensions.

The ETC Coursera Telemetry Framework currently consists of 4 extensions.

1. This extension, ETC JupyterLab Telemetry Coursera (etc_jupyterlab_telemetry_coursera), which consumes events and logs them to an S3 bucket.
2. ETC JupyterLab Telemetry Library (etc-jupyterlab-telemetry-library), which emits telemetry related events.
3. ETC JupyterLab Feedback (etc_jupyterlab_feedback), which emits Feedback related events.
4. ETC JupyterLab Notebook State Provider (etc-jupyterlab-notebook-state-provider), which keeps track of Notebook diffs.

The modular design allows for components to be included or excluded from deployments as needed.

A Summary of Each of the Components

This Extension, ETC JupyterLab Telemetry Coursera (etc_jupyterlab_telemetry_coursera)

The purpose of this extension is to consume events from other extensions and log those events to an AWS S3 Bucket.

ETC JupyterLab Telemetry Library (etc-jupyterlab-telemetry-library)

The purpose of the ETC JupyterLab Telemetry Library is to emit telemetry events that can be consumed by this extension.

ETC JupyterLab Feedback (etc_jupyterlab_feedback)

The purpose of the ETC JupyterLab Feedback extension is to emit Feedback related events that can be consumed by this extension.

ETC JupyterLab Notebook State Provider (etc-jupyterlab-notebook-state-provider)

The purpose of the ETC JupyterLab Notebook State Provider extension is to keep track of diffs of Notebooks. The purpose of diffing the Notebooks is to reduce storage space.

Configuration

ETC JupyterLab Telemetry Coursera, ETC JupyterLab Telemetry Library, and ETC JupyterLab Feedback each have their own configuration files. For extensions that have configurable settings, the configuration file must be present for the respective extension to start. ETC JupyterLab Notebook State Provider doesn't have configurable settings.

The configuration file for each extension must be placed in one of the config directories given by jupyter --paths e.g., /etc/jupyter.

Configuration File Examples

ETC JupyterLab Telemetry Coursera

The configuration file must be named jupyter_etc_jupyterlab_telemetry_coursera_config.py.

# This file should be saved into one of the config directories provided by `jupyter lab --path`.
# The following configuration settings can be used in order to construct the `bucket_url` variable,
# which the extension will use for logging telemetry data.

# The value of the `url` property will be the first component of the bucket_url.
c.ETCJupyterLabTelemetryCourseraApp.url = "https://telemetry.mentoracademy.org"

# The value of the `bucket` property will be appended to the `bucket_url` after the url.
c.ETCJupyterLabTelemetryCourseraApp.bucket = "telemetry-edtech-labs-si-umich-edu"

# The value of the `path` property will be appended to the `bucket_url` after the `bucket`.
c.ETCJupyterLabTelemetryCourseraApp.path = "dev/test-telemetry"

# The values of the`env_path_segment_names` property will be appended to the `bucket_url` after the `path`.
c.ETCJupyterLabTelemetryCourseraApp.env_path_segment_names = ['COURSE_ID']

# The value of the `telemetry_file_path` property is the path to the telemetry touch file.
c.ETCJupyterLabTelemetryCourseraApp.telemetry_file_path = ".telemetry"

# If a telemetry touch file is present at `telemetry_file_path` its contents will
# be appended to the `bucket_url` after the `env_path_segment_names`.  The telemetry touch
# file should contain a relative path e.g., "segment1/segment2".
# The presense of a telemetry touch file will enable telemetry unless it is explicitly disabled.

# Telemetry can be explicitly turned on or off by setting the `telemetry` property; this setting overrides all other settings.
c.ETCJupyterLabTelemetryCourseraApp.telemetry = True

# Adding events to this list will result in the complete Notebook being logged at the respective event as opposed to a diff.
c.ETCJupyterLabTelemetryCourseraApp.capture_notebook_events = ['save_notebook']

# The Notebook will autosave at the given value of the `save_interval` property.
c.ETCJupyterLabTelemetryCourseraApp.save_interval = 1

ETC JupyterLab Telemetry Library

Setting the value of the property will enable or disable the respective event. The configuration file must be named jupyter_etc_jupyterlab_telemetry_library_config.py.

c.ETCJupyterLabTelemetryLibraryApp.notebook_clipboard_event = True
c.ETCJupyterLabTelemetryLibraryApp.notebook_visibility_event = True
c.ETCJupyterLabTelemetryLibraryApp.notebook_save_event = True
c.ETCJupyterLabTelemetryLibraryApp.notebook_close_event = True
c.ETCJupyterLabTelemetryLibraryApp.notebook_open_event = True
c.ETCJupyterLabTelemetryLibraryApp.notebook_cell_remove_event = True
c.ETCJupyterLabTelemetryLibraryApp.notebook_cell_add_event = True
c.ETCJupyterLabTelemetryLibraryApp.notebook_cell_execution_event = True
c.ETCJupyterLabTelemetryLibraryApp.notebook_scroll_event = True
c.ETCJupyterLabTelemetryLibraryApp.notebook_active_cell_change_event = True
c.ETCJupyterLabTelemetryLibraryApp.notebook_cell_error_event = True

ETC JupyterLab Feedback

The configuration file must be named jupyter_etc_jupyterlab_feedback_config.py.

c.ETCJupyterLabFeedbackApp.nbgrader_validate_enabled = True

# The number of seconds the extension will wait for a response from the nbgrader application before timing out.
c.ETCJupyterLabFeedbackApp.nbgrader_validate_timeout = 60

c.ETCJupyterLabFeedbackApp.feedback_enabled = True

# The number of seconds the extension will wait for a response from the feedback API before timing out.
c.ETCJupyterLabFeedbackApp.feedback_timeout = 60

# The URL for the Feedback API endpoint.
c.ETCJupyterLabFeedbackApp.feedback_url = "https://umichproseaitutor.com/CodeRepairs/Post" 

# The path to the SSL certificate.
c.ETCJupyterLabFeedbackApp.cert_path = "/etc/ssl/certs/umichproseaitutorpubliconly.pem"

Installation

Installing the etc-jupyterlab-telemetry-coursera extension will install all of the dependencies listed in setup.py; this includes the other 3 extension that comprise the ETC Coursera Telemetry Framework. See the install_requires property of the setup_args dictionary in setup.py.

Install

To install the extension:

pip install etc-jupyterlab-telemetry-coursera

Uninstall

To remove the extension, execute:

pip uninstall etc-jupyterlab-telemetry-coursera

Troubleshoot

If you are seeing the frontend extension, but it is not working, check that the server extension is enabled:

jupyter server extension list

If the server extension is installed and enabled, but you are not seeing the frontend extension, check the frontend extension is installed:

jupyter labextension list

Development

Each of the 4 extensions are composed of a server extension (Python) and a client extension (TypeScript). The code for the server extension is in the directory with the same name as the respective extension; for example, for this extension, the server code is in a directory of this repository named etc_jupyterlab_telemetry_coursera. The code for the client extension is in the directory named src of each of the respective repositories.

The 4 extensions that comprise the ETC Coursera Telemetry Framework can be developed and debugged concurrently. This is accomplished by creating an environment and doing a "development installation" of each of the extensions into that environment. Once the development environment is set up, when you make changes to any of the client extensions those changes will be reflected in the running application after refreshing the application in the browser. Changes to any of the server extensions will be reflected when JupyterLab is restarted.

The following instructions explain how to set up an environment for development.

Set up the development environment.

The development environment will consist of the 4 extensions running concurrently.

Create an environment.

Create a conda environment that contains the resources needed to build and publish the extension.

conda create -n etc_jupyterlab_telemetry_coursera --override-channels --strict-channel-priority -c conda-forge -c nodefaults jupyterlab=3 nodejs jupyter-packaging git

conda activate etc_jupyterlab_telemetry_coursera

Once the environment is activated install the build package using pip.

pip install build

Clone each of the extensions.

Change directory to the directory that will contain your repositories and clone each of the extensions.

git clone https://github.com/educational-technology-collective/etc_jupyterlab_telemetry_coursera.git
git clone https://github.com/educational-technology-collective/etc_jupyterlab_notebook_state_provider.git
git clone https://github.com/educational-technology-collective/etc_jupyterlab_telemetry_library.git
git clone https://github.com/educational-technology-collective/etc_jupyterlab_feedback.git

Do a "development installation" of the ETC Coursera Telemetry Framework.

Development install the etc_jupyterlab_telemetry_coursera extension.

From the directory the contains your repositories:

cd etc_jupyterlab_telemetry_coursera

# Install package in development mode.
pip install -e .

# Link your development version of the extension with JupyterLab.
jupyter labextension develop . --overwrite

# Server extension must be manually enabled in develop mode.
jupyter server extension enable etc_jupyterlab_telemetry_coursera

# Automatically build the extension when a change is made.
jlpm run watch

Development install the etc_jupyterlab_notebook_state_provider extension.

From the directory the contains your repositories:

cd etc_jupyterlab_notebook_state_provider

# Install package in development mode.
pip install -e .

# Link your development version of the extension with JupyterLab.
jupyter labextension develop . --overwrite

# Server extension must be manually enabled in develop mode.
jupyter server extension enable etc_jupyterlab_notebook_state_provider

# Automatically build the extension when a change is made.
jlpm run watch

Development install the etc_jupyterlab_telemetry_library extension.

From the directory the contains your repositories:

cd etc_jupyterlab_telemetry_library.

# Install package in development mode.
pip install -e .

# Link your development version of the extension with JupyterLab.
jupyter labextension develop . --overwrite

# Server extension must be manually enabled in develop mode.
jupyter server extension enable etc_jupyterlab_telemetry_library.

# Automatically build the extension when a change is made.
jlpm run watch

Development install the etc_jupyterlab_feedback extension.

From the directory the contains your repositories:

cd etc_jupyterlab_feedback

# Install package in development mode.
pip install -e .

# Link your development version of the extension with JupyterLab.
jupyter labextension develop . --overwrite

# Server extension must be manually enabled in develop mode.
jupyter server extension enable etc_jupyterlab_feedback

# Automatically build the extension when a change is made.
jlpm run watch

Start JupyterLab

Once the development environment has been set up you can start JupyterLab. The running JupyterLab instance will reflect the current state of each of the extension respositories.

jupyter lab

Development Workflow

The development workflow generally involves making changes to one of the 4 client extensions or one of the 4 server extensions and then testing the changes in the running JupyterLab application.

Client Development Workflow

• Make changes to the TypeScript client extension.
• Refresh the browser.
• Observe the changes in the running application.

Server Development Workflow

• Make changes to the Python server extension.
• Stop the Jupyter server.
• Start the Jupyter server.
• Observe the changes in the running application.

Publish Changes

Once you have made changes to an extension, in order for those changes to be present in an installation you will need to publish the changes to PyPI.

Change directory into the extension directory where you want to publish the changes.

# Patch the version.
jlpm version --patch

# Build the extension with the changes that you have made and the new version number.
jlpm run build

# Publish the changes to NPM.
# This step is necessary in for these changes to be available to subsequent development installations.
npm publish --access public

# Remove any prior builds from the `dist` directory.
rm -rf dist

# Build the extension
python -m build .

# Publish the extension to PyPI.
twine upload dist/*

Resources

• Extension Tutorial
• JupyterLab Extensions by Examples
• Gitter