Mozilla Kuma software Documentation

Kuma Documentation
Release latest
Mozilla
January 08, 2015
Contents
1
Development
2
Getting Started
2.1 Installation . . . . . . . . . . .
2.2 Vendor Library . . . . . . . . .
2.3 Kuma via Vagrant . . . . . . .
2.4 Celery and Rabbit . . . . . . .
2.5 Running Kuma with mod_wsgi
2.6 Email from Kuma . . . . . . .
2.7 Search . . . . . . . . . . . . .
2.8 Development . . . . . . . . . .
2.9 Localization . . . . . . . . . .
2.10 The Kuma Test Suite . . . . . .
3
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
5
5
8
10
13
15
18
19
20
22
27
i
ii
Kuma Documentation, Release latest
Kuma is the platform that powers MDN (developer.mozilla.org)
Contents
1
Kuma Documentation, Release latest
2
Contents
CHAPTER 1
Development
Code https://github.com/mozilla/kuma
Issues http://mzl.la/mdn_backlog (Product)
https://github.com/mozilla/kuma/issues?state=open (Engineering)
https://prs.paas.allizom.org/mozilla:kuma,kuma-lib,kumascript,mozhacks (PR Queue)
Dev Docs https://kuma.readthedocs.org/en/latest/installation-vagrant.html
CI Server https://travis-ci.org/mozilla/kuma
Mailing list https://lists.mozilla.org/listinfo/dev-mdn
IRC irc://irc.mozilla.org/mdndev
http://logs.glob.uno/?c=mozilla%23mdndev (logs)
Servers http://mzl.la/deployed-mdn (What’s Deployed)
https://developer-dev.allizom.org/ (dev)
https://developer.allizom.org/ (stage)
https://developer.mozilla.org/ (prod)
3
Kuma Documentation, Release latest
4
Chapter 1. Development
CHAPTER 2
Getting Started
Want to help make MDN great? Here are some options:
1. Contribute to our selenium test suite
2. Contribute to the platform itself. Our contribution guide lists some good first projects and offers direction on
submitting code.
Contents:
2.1 Installation
This page describes the manual installation procedure. If you can, you should set up the vagrant-managed virtual
machine instead.
2.1.1 Requirements
To run everything and make all the tests pass locally, you’ll need the following things (in addition to Git, of course).
• Python 2.6.
• setuptools or pip.
• MySQL Server and client headers.
• Memcached Server and libmemcached.
• Elasticsearch 0.90.9.
• RabbitMQ.
• libxml and headers.
• libxslt and headers.
• libjpeg and headers.
• zlib and headers.
• libmagic and headers.
• libtidy and headers
• Several Python packages. See Installing the Packages.
Installation for these is very system dependent. Using a package manager, like yum, aptitude, or brew, is encouraged.
5
Kuma Documentation, Release latest
Additional Requirements
If you want to use Apache, instead of the dev server (not strictly required but it’s more like our production environment)
you’ll also need:
• Apache HTTPD Server.
• mod_wsgi
See the documentation on WSGI for more information and instructions.
2.1.2 Getting the Source
First, to follow the instructions from Webdev Bootcamp, fork the project into your own account. Then get the source
using:
mkdir mdn # you probably want to do this, since you’ll have to create
cd mdn
# product_details_json/ as a sibling of kuma/ later.
git clone [email protected]:<your_username>/kuma.git
cd kuma
git submodule update --init --recursive
2.1.3 Installing the Packages
Compiled Packages
There are a small number of compiled packages, including the MySQL Python client. You can install these using pip
(if you don’t have pip, you can get it with easy_install pip) or via a package manager. To use pip, you only
need to do this:
sudo pip install -r requirements/compiled.txt
Python Packages
All of the pure-Python requirements are available in a git repository, known as a vendor library. This allows them to
be available on the Python path without needing to be installed in the system, allowing multiple versions for multiple
projects simultaneously.
2.1.4 Configuration
Start by creating a file named settings_local.py, and putting this line in it:
from settings import *
Now you can copy and modify any settings from settings.py into settings_local.py and the value will
override the default.
Database
At a minimum, you will need to define a database connection. An example configuration is:
6
Chapter 2. Getting Started
Kuma Documentation, Release latest
DATABASES = {
’default’: {
’NAME’: ’kuma’,
’ENGINE’: ’django.db.backends.mysql’,
’HOST’: ’localhost’,
’USER’: ’kuma’,
’PASSWORD’: ’’,
’OPTIONS’: {’init_command’: ’SET storage_engine=InnoDB’},
’TEST_CHARSET’: ’utf8’,
’TEST_COLLATION’: ’utf8_unicode_ci’,
},
}
Note the two settings TEST_CHARSET and TEST_COLLATION. Without these, the test suite will use MySQL’s
(moronic) defaults when creating the test database (see below) and lots of tests will fail. Hundreds.
Once you’ve set up the database, you can generate the schema with Django’s syncdb command:
mkdir ../product_details_json
./manage.py syncdb
./manage.py migrate
This will generate an empty database, which will get you started!
Initializing Mozilla Product Details
One of the packages Kuma uses, Django Mozilla Product Details, needs to fetch JSON files containing historical
Firefox version data and write them within its package directory. To set this up, just run:
./manage.py update_product_details
...to do the initial fetch.
Media
If you want to see images and have the pages formatted with CSS you need to set your settings_local.py with
the following:
DEBUG = True
TEMPLATE_DEBUG = DEBUG
SERVE_MEDIA = True
Setting DEBUG = False will put the installation in production mode and ask for minified assets. In that case, you
will need to generate CSS from stylus and compress resource:
./scripts/compile-stylesheets
./manage.py compress_assets
Configure Persona
Add the following to settings_local.py so that Persona works with the development instance:
SITE_URL = ’http://localhost:8000’
PROTOCOL = ’http://’
DOMAIN = ’localhost’
PORT = 8000
2.1. Installation
7
Kuma Documentation, Release latest
SESSION_COOKIE_SECURE = False # needed if the server is running on http://
SESSION_EXPIRE_AT_BROWSER_CLOSE = False
The SESSION_EXPIRE_AT_BROWSER_CLOSE setting is not strictly necessary, but it’s convenient for development.
Secure Cookies
To prevent error messages like Forbidden (CSRF cookie not set.):, you need to set your
settings_local.py with the following:
CSRF_COOKIE_SECURE = False
2.1.5 Testing it Out
To start the dev server, run ./manage.py runserver, then open up http://localhost:8000. If everything’s working, you should see the MDN home page!
You might need to first set LC_CTYPE if you’re on Mac OS X until bug 754728 is fixed:
export LC_CTYPE=en_US
2.1.6 What’s next?
See development for further instructions.
Some site funcationaly require waffle flags. Waffle flags include:
• kumaediting: Allows creation, editing, and translating of documents
• page_move: Allows moving of documents
• revision-dashboard-newusers: Allows searching of new users through the revision dashboard
• events_map: Allows display of map on the events page
• elasticsearch: Enables elastic search for site search
To create or modify waffle flags, visit “/admin/” and click the “Waffle” link.
2.1.7 Last Steps
Setting Up Search
See the search documentation for steps to get Elasticsearch search working.
2.2 Vendor Library
The vendor directory contains the source code for Kuma libraries.
8
Chapter 2. Getting Started
Kuma Documentation, Release latest
2.2.1 Getting the Vendor Libraries
To get the vendor libraries in your Kuma clone,
$ git submodule update --init
Git will fetch all the vendor submodules.
2.2.2 Updating the Vendor Library
Danger: We are moving all libraries from vendor to pip. Any pull requests with vendor updates will be rejected
unless it is an emergency situation.
From time to time we need to update libraries, either for new versions of libraries or to add a new library. There are
two ways to do that.
Updating a Library with Git Submodules
If the library is in vendor/src, it was pulled directly from version control, and if that version control was git, update
the submodule like so:
$
$
$
$
$
$
cd vendor/src/$LIBRARY
git fetch origin
git checkout <REFSPEC>
cd ../..
git add src/$LIBRARY
git ci -m "Updating $LIBRARY"
Just like updating any other git submodule.
Adding a New Library with Git Submodules
Technically this can be done with pip install --no-install but we do this:
$
$
$
$
$
$
$
cd vendor/src
git clone git://<repo>
cd ../..
./addsubmodules.sh
vim kuma.pth # Add the new library’s path
git add kuma.pth
git ci -m "Adding $LIBRARY"
Using PyPI
Follow the playdoh instructions for non-git based repos, replacing vendor-local with vendor.
2.2.3 Requirements Files
We are in the process of moving all our libraries to pip requirements.
2.2. Vendor Library
9
Kuma Documentation, Release latest
We still maintain requirements files in requirements/. Sometimes people will use these to install the requirements
in a virtual environment. When you update the vendor repo, you should make sure to update version numbers (if
necessary) in the requirements files.
2.3 Kuma via Vagrant
Core developers run Kuma in a Vagrant-managed virtual machine so we can run the entire MDN stack. (Django,
KumaScript, Search, Celery, etc.) If you’re on Mac OS X or Linux and looking for a quick way to get started, you
should try these instructions.
At the end, you’ll earn the badge:
Note: If you have problems using vagrant, check Troubleshooting and Getting More Help below.
2.3.1 Install and run everything
1. Install VirtualBox 4.x from http://www.virtualbox.org/
Note:
(Windows) After installing
Files\Oracle\VirtualBox\VBoxManage.exe;
VirtualBox
you
need
to
set
PATH=C:\Program
2. Install vagrant >= 1.6 using the installer from vagrantup.com
3. Install the vagrant-vbguest plugin.
4. Fork the project. (See GitHub and Webdev Bootcamp)
5. Clone your fork of Kuma and update submodules:
git clone [email protected]:<your_username>/kuma.git
cd kuma
git submodule update --init --recursive
6. Copy a vagrantconfig_local.yaml file for your VM:
cp vagrantconfig_local.yaml-dist vagrantconfig_local.yaml
7. Start the VM and install everything. (approx. 30 min on a fast net connection).:
vagrant up
Note: VirtualBox creates VMs in your system drive. Kuma’s VM is 3GB. If it won’t fit on your system drive,
you will need to change that directory to another drive.
At the end, you should see:
=> default: notice: Finished catalog run in .... seconds
If the above process fails with an error, check Troubleshooting.
8. Add the hostnames to the end of your hosts file with this shell command:
echo ’192.168.10.55 developer-local.allizom.org mdn-local.mozillademos.org’ | sudo tee -a /etc/h
9. Log into the VM with ssh:
10
Chapter 2. Getting Started
Kuma Documentation, Release latest
vagrant ssh
10. Use foreman inside the VM to start all site services:
foreman start
You should see output like:
20:32:59
20:32:59
20:32:59
20:32:59
...
web.1
celery.1
kumascript.1
stylus.1
|
|
|
|
started
started
started
started
with
with
with
with
pid
pid
pid
pid
2244
2245
2246
2247
11. Visit https://mdn-local.mozillademos.org and add an exception for the security certificate if prompted
12. Visit the homepage at https://developer-local.allizom.org
13. You’ve installed Kuma! If you want the badge, email a screenshot of your browser to mdn-dev at mozilla dot
com.
2.3.2 Create an admin user
You will want to make yourself an admin user to enable important site features.
1. Sign up/in with Persona
2. After you sign in, SSH into the vm and make yourself an admin:
vagrant ssh
mysql -uroot kuma
UPDATE auth_user set is_staff = 1, is_active=1, is_superuser = 1 WHERE username = ’YOUR_USERNAME
You should see:
Query OK, 1 row affected (0.01 sec)
Rows matched: 1 Changed: 1 Warnings: 0
2.3.3 Enable Important Site Features
Some site features are controlled using django-waffle. You control these features in the waffle admin.
Some site features are controlled using constance. You control these features in the constance config admin panel.
GitHub Auth
To enable GitHub authentication ...
Register your own OAuth application on GitHub:
• Application name: MDN (<username>)
• Homepage url: https://developer-local.allizom.org/docs/MDN/Contribute/Howto/Create_an_MDN_account
• Application description: My own GitHub app for MDN!
• Authorization callback URL: https://developer-local.allizom.org/users/github/login/callback/
Add a django-allauth social app for GitHub:
2.3. Kuma via Vagrant
11
Kuma Documentation, Release latest
• Provider: GitHub
• Name: developer-local.allizom.org
• Client id: <your GitHub App Client ID>
• Secret key: <your GitHub App Client Secret>
• Sites: example.com -> Chosen sites
GitHub auth is also (temporarily) behind a waffle flag. So, add a waffle flag called github_login and set “Everyone” to “Yes”.
Now you can sign in with GitHub at https://developer-local.allizom.org
Wiki Editing
The central feature of MDN is wiki editing. We use a waffle flag called kumaediting to control edits to the wiki.
So we can effectively put the site into “read-only” and/or “write-by-staff-only” modes.
To enable wiki editing on your MDN vm, add a waffle flag called kumaediting and set “Everyone” to “Yes”.
KumaScript
To enable KumaScript (Kuma’s template system):
1. Sign in
2. Visit the constance config admin panel
3. Change KUMASCRIPT_TIMEOUT to 600
4. Click “Save” at the bottom
2.3.4 Create pages
You can visit https://developer-local.allizom.org/docs/new to create new wiki pages as needed.
Many core MDN contributors create a personal User:<username> page as a testing sandbox.
2.3.5 Developing with Vagrant
Edit files as usual on your host machine; the current directory is mounted via NFS at /home/vagrant/src within
the VM. Updates should be reflected without any action on your part.
• See development for tips not specific to vagrant.
• Useful vagrant sub-commands:
vagrant
vagrant
vagrant
vagrant
vagrant
12
ssh
suspend
halt
up
destroy
#
#
#
#
#
Connect to the VM via ssh
Sleep the VM, saving state
Shutdown the VM
Boot up the VM
Destroy the VM
Chapter 2. Getting Started
Kuma Documentation, Release latest
2.3.6 Troubleshooting
Errors during vagrant up
vagrant up starts the virtual machine. The first time you run vagrant up it also provisions the vm - i.e., it
automatically installs and configures Kuma software on the vm. We provision the vm with puppet manifests in the
puppet/manifests directory.
Sometimes we put puppet declarations in the wrong order. Which means some errors can be fixed by simply provisioning the vm again:
vagrant provision
In some rare occasions you might need to run this multiple times. If you see the same error over and over, please ask
for more help.
On Ubuntu, vagrant up might fail after being unable to mount NFS shared folders. First, make sure you have
the nfs-common and nfs-server packages installed and also note that you can’t export anything via NFS inside an
encrypted volume or home dir.
If that doesn’t help you can disable nfs by setting the nfs flag in the vagrantconfig_local.yaml file you just created.
nfs: false
Note: If you decide to run nfs: false, the system will be a lot slower. There is also the potential of running into
weird issues with puppet, since the current puppet configurations do not currently support nfs: false.
If you have other problems during vagrant up, please ask for more help.
Errors after switching branches
• You should occasionally re-run the Puppet setup, especially after updating code with major changes. This
will ensure that the VM environment stays up to date with configuration changes and installation of additional
services.
– On the Host:
vagrant provision
– Inside the VM:
sudo puppet apply /home/vagrant/src/puppet/manifests/dev-vagrant.pp
Getting more help
If you have more problems using vagrant, please:
1. Paste errors to pastebin
2. email [email protected].
3. After you email dev-mdn, you can also ask in IRC
2.4 Celery and Rabbit
Kuma uses Celery to enable offline task processing for long-running jobs like sending email notifications and rerendering the Knowledge Base.
2.4. Celery and Rabbit
13
Kuma Documentation, Release latest
Though Celery supports multiple message backends, we use, and recommend that you use, RabbitMQ. RabbitMQ is
an AMQP message broker written in Erlang.
2.4.1 When is Celery Appropriate
You can use Celery to do any processing that doesn’t need to happen in the current request-response cycle. Examples
are generating thumbnails, sending out notification emails, updating content that isn’t about to be displayed to the user,
and others.
Ask yourself the question: “Is the user going to need this data on the page I’m about to send them?” If not, using a
Celery task may be a good choice.
2.4.2 RabbitMQ
Installing
RabbitMQ should be installed via your favorite package manager. It can be installed from source but has a number of
Erlang dependencies.
Configuring
RabbitMQ takes very little configuration.
# Start the server.
sudo rabbitmq-server -detached
# Set up the permissions.
rabbitmqctl add_user kuma kuma
rabbitmqctl add_vhost kuma
rabbitmqctl set_permissions -p kuma kuma ".*" ".*" ".*"
That should do it. You may need to use sudo for rabbitmqctl. It depends on the OS and how Rabbit was installed.
2.4.3 Celery
Installing
Celery (and Django-Celery) is part of our vendor library. You shouldn’t need to do any manual installation.
Configuring and Running
We set some reasonable defaults for Celery in settings.py.
These can be overriden either in
settings_local.py or via the command line when running manage.py celeryd.
In settings_local.py you should set at least this, if you want to use Celery:
CELERY_ALWAYS_EAGER = False
This defaults to True, which causes all task processing to be done online. This lets you run Kuma even if you don’t
have Rabbit or want to deal with running workers all the time.
You can also configure the log level or concurrency. Here are the defaults:
14
Chapter 2. Getting Started
Kuma Documentation, Release latest
CELERYD_LOG_LEVEL = logging.INFO
CELERYD_CONCURRENCY = 4
Then to start the Celery workers, you just need to run:
./manage.py celeryd
This will start Celery with the default number of worker threads and the default logging level. You can change those
with:
./manage.py celeryd --log-level=DEBUG -c 10
This would start Celery with 10 worker threads and a log level of DEBUG.
2.5 Running Kuma with mod_wsgi
2.5.1 Requirements
• See the installation docs.
• Apache HTTP server
• mod_rewrite
• mod_headers
• mod_expires
• mod_wsgi
• Not mod_python! It is incompatible with mod_wsgi.
2.5.2 Overview
Setting up Kuma to run as a WSGI application is fairly straightforward. You will need to install the requirements and
clone the vendor repo as described in the installation docs.
There are 3 steps once Kuma is installed:
• Set the document root.
• Set up aliases.
• Some file permissions.
• Set up WSGI itself.
Apache Modules
Most of the Apache modules are part of a default Apache install, but may need to be activated. If they aren’t installed,
all of them, including mod_wsgi should be installable via your favorite package manager.
2.5. Running Kuma with mod_wsgi
15
Kuma Documentation, Release latest
WSGI Configuration
In the Apache config (or <VirtualHost>) you will need the following:
Note that values may be slightly different.
DocumentRoot /path/to/kuma/webroot/
<Directory "/path/to/kuma/webroot/">
Options +FollowSymLinks
</Directory>
Alias /media/ "/path/to/kuma/media/"
Alias /admin-media/ \
"/path/to/kuma/vendor/src/django/django/contrib/admin/media/"
WSGISocketPrefix /var/run/wsgi
WSGIDaemonProcess kuma processes=8 threads=1 \
maximum-requests=4000
WSGIProcessGroup kuma
WSGIScriptAlias /k "/path/to/kuma/wsgi/kuma.wsgi"
WSGISocketPrefix: May or may not be necessary. It was for me.
WSGIDaemonProcess: processes should be set to the number of cores. threads should probably be left at
1. maximum-requests is good at between 4000 and 10000.
WSGIScriptAlias: Will make Kuma accessible from http://domain/k, and we use rewrites in
webroot/.htaccess to hide the /k. This will change soon, and the .htaccess file won’t be necessary.
The Alias directives let Kuma access its CSS, JS, and images through Apache, reducing the load on Django.
Configuration
Most of our settings.py is under version control, but can be overridden in a file called settings_local.py
in the base of the app (the same place as settings.py).
You can see example settings in
docs/settings/settings_local.prod.py:
from settings import *
DEBUG = False
TEMPLATE_DEBUG = False
# The default database should point to the master.
DATABASES = {
’default’: {
’NAME’: ’kitsune’,
’ENGINE’: ’django.db.backends.mysql’,
’HOST’: ’’,
’USER’: ’’,
’PASSWORD’: ’’,
’OPTIONS’: {’init_command’: ’SET storage_engine=InnoDB’},
},
}
# This is used to hash some things in Django.
16
Chapter 2. Getting Started
Kuma Documentation, Release latest
SECRET_KEY = ’replace me with something long and random’
DEBUG_PROPAGATE_EXCEPTIONS = DEBUG
2.5.3 File Permissions
To upload files, the webserver needs write access to media/uploads and all its subdirectories. The directories we
currently use are:
media/uploads
media/uploads/avatars
media/uploads/images
media/uploads/images/thumbnails
media/uploads/gallery/images
media/uploads/gallery/images/thumbnails
media/uploads/gallery/videos
media/uploads/gallery/videos/thumbnails
media/uploads and its subdirectories should never be added to version control, as they are installation-/contentspecific.
Product Details JSON
Some people have issues with django-mozilla-product-details and file permissions. The management
command manage.py update_product_details writes a number of JSON files to disk, and the webserver
then needs to read them.
If you get file system errors from product_details, make sure the files are readable by the webserver (should be
by default) and the directory is readable and executable.
By default, product_details stores the JSON files in:
vendor/src/django-mozilla-product-details/product_details/json
This is configurable. If you have multiple web servers, they should share this data. You can set the
PROD_DETAILS_DIR variable in settings_local.py to a different path, for example on NFS.
2.5.4 Debugging
Debugging via WSGI is a little more interesting than via the dev server. One key difference is that you cannot use
pdb. Writing to stdout is not allowed within the WSGI process, and will result in a Internal Server Error.
There are three relevant cases for debugging via WSGI (by which I mean, where to find stack traces):
Apache Error Page
So you’ve got a really bad error and you aren’t even seeing the Kuma error page! This is usually caused by an
uncaught exception during the WSGI application start-up. Our WSGI script tries to run all the initial validation that
the dev server runs, to catch these errors early.
So where is the stack trace? You’ll need to look in your Apache error logs. Where these are is OS-dependent, but
a good place to look is /var/log/httpd. If you are using SSL, also check the SSL VirtualHost�s logs, for
example /var/log/httpd/ssl_error_log.
2.5. Running Kuma with mod_wsgi
17
Kuma Documentation, Release latest
With DEBUG=True
With DEBUG = True in your settings_local.py, you will see a stack trace in the browser on error. Problem
solved!
With DEBUG=False
With DEBUG = False in your settings_local.py, you’ll see our Server Error message. You can still get
stack traces, though, by setting the ADMINS variable in settings_local.py:
ADMINS = (
(’me’, ’[email protected]’),
)
Django will email you the stack trace. Provided you’ve set up email.
2.5.5 Reloading WSGI
WSGI keeps Python and Kuma running in an isolated process. That means code changes aren’t automatically reflected
on the server. In most default configurations of mod_wsgi, you can simply do this:
touch wsgi/kuma.wsgi
That will cause the WSGI process to reload.
2.6 Email from Kuma
The default settings for Kuma do not send email. If you want to get email, you should double check one thing first:
are there any rows in the tidings_watch table? If there are, you may be sending email to real users. The script in
scripts/anonymize.sql will truncate this table. Simply run it against your Kuma database:
mysql -ukuma -p kuma < scripts/anonymize.sql
2.6.1 Sending Email
So now you know you aren’t emailing real users, but you’d still like to email yourself and test email in general. There
are a few settings you’ll need to use.
First, set the EMAIL_BACKEND. This document assumes you’re using the SMTP mail backend.
EMAIL_BACKEND = ’django.core.mail.backends.smtp.EmailBackend’
If you have sendmail installed and working, that should do it. However, you might get caught in spam filters. An
easy workaround for spam filters or not having sendmail working is to send email via a Gmail account.
EMAIL_USE_TLS = True
EMAIL_PORT = 587
EMAIL_HOST = ’smtp.gmail.com’
EMAIL_HOST_USER = ’<your gmail address>@gmail.com’
EMAIL_HOST_PASSWORD = ’<your gmail password>’
Yeah, you need to put your Gmail password in a plain text file on your computer. It’s not for everyone. Be very careful
copying and pasting settings from settings_local.py if you go this route.
18
Chapter 2. Getting Started
Kuma Documentation, Release latest
2.7 Search
Kuma uses Elasticsearch to power its on-site search facility.
Elasticsearch search gives us a number of advantages over MySQL’s full-text search or Google’s site search.
• Much faster than MySQL. * And reduces load on MySQL.
• We have total control over what results look like.
• We can adjust searches with non-visible content.
• We don’t rely on Google reindexing the site.
• We can fine-tune the algorithm ourselves.
2.7.1 Installing Elasticsearch Search
We currently require Elasticsearch 0.90.9. You may be able to install this from a package manager like yum, aptitude,
or brew.
If not, you can easily download the source and compile it. Generally all you’ll need to do is:
$ cd elasticsearch-0.90.9
$ bin/elasticsearch -f
Then run the Kuma search tests:
$ ./manage.py test -s --noinput --logging-clear-handlers search
If the tests pass, everything is set up correctly!
2.7.2 Using Elasticsearch
Having Elasticsearch installed will allow the search tests to run, which may be enough. But you want to work on or
test the search app, you will probably need to actually see search results!
The Easy, Sort of Wrong Way
The easiest way to start Elasticsearch for testing is:
$ cd path/to/elasticsearch-0.90.9
$ bin/elasticsearch -f
Then from the Kuma source code path:
$ ./manage.py reindex
If you need to update the search indexes:
$ ./manage.py reindex
While this method is very easy, you will need to reindex after any time you run the search tests, as they will overwrite
the data files Elasticsearch uses.
2.7. Search
19
Kuma Documentation, Release latest
The Ellaborate, Kinda Proper Way
Assuming you’re running the full stack with foreman start (or any other way that makes sure the Celery workers
run) there is a better way:
• Open the Django admin UI under http://127.0.0.1:8000/admin/search/index/ or https://developerlocal.allizom.org/admin/search/index/ if you’re using Vagrant.
• Add a search index by clicking on the “Add index” button in the top right corner, safe it by clicking on the same
button in the lower right corner to safe it to the database.
• On the search index list view again, select the just created index (the top most) and select “Populate search index
with Celery task” from the actions dropdown below.
• Once the population is ready the “populated” field with show up as a green checkbox image. You’ll also get an
email (probably via the console if you’re developing kuma locally) notifying you of the completion.
• To actually enable that newly created search index you have to promote it now. On the search index list view
again, select the just created index (the top most) and select “Promote search index” from the actions dropwdown
below.
• Once the search index is promoted the “promoted” and the “is current index” field will show up as a green
checkbox image. The index is now live.
Similarly you can also demote a search index and it will automatically fall back to the previously created index (by
created date). That helps to figure out issues in production and should allow for a smooth deployment of search index
changes. It’s recommmended to keep a few search indexes around just in case.
If no index is created in the admin UI the fallback “main_index” index will be used instead.
Warning: If you delete any of the search indexes in the admin interface they will be deleted on Elasticsearch as
well.
2.8 Development
We strongly suggest using a Vagrant-managed VM if you can. Or, you can use the manual installation steps.
2.8.1 Running Kuma in Vagrant VM
If you are using a Vagrant-managed VM, you can start all Kuma servers and services with:
vagrant ssh
foreman start
2.8.2 Running Kuma manually
If you are using manual installation, you can run the django server with:
./manage.py runserver
and the kumascript service with:
node kumascript/run.js
Note: Before running kumascript, you need to install the node.js fibers module by running npm install
fibers.
20
Chapter 2. Getting Started
Kuma Documentation, Release latest
2.8.3 Log in
You can log into MDN using Persona or GitHub. For GitHub, you must first enable GitHub Auth as described in the
installation instructions.
2.8.4 Set up permissions
Some features are only available to privileged users. To manage permissions use the Auth -> Users section of the
django admin interface.
2.8.5 Compiling Stylus Files
If you’re updating the Stylus CSS files, you’ll need to compile them before you can see your updates within the
browser. To compile stylus files, run the following from the command line:
./scripts/compile-stylesheets
The relevant CSS files will be generated and placed within the media/redesign/css directory. You can add a -w flag to
that call to compile stylesheets upon save.
2.8.6 Hacking on bleeding edge features
To hack on the features not yet ready for production you have to enable them first.
Enable Kumascript
Kuma uses a separate nodejs-based service to process templates in wiki pages. Its use is disabled by default, to
enable: open the django admin interface and in the Constance section change the value of KUMASCRIPT_TIMEOUT
parameter to a positive value (such as 2.0 seconds).
2.8.7 Running the Tests
A great way to check that everything really is working is to run the test suite.
Django tests
If you’re not using the vagrant VM, you’ll need to add an extra grant in MySQL for your database user:
GRANT ALL ON test_NAME.* TO USER@localhost;
Where NAME and USER are the same as the values in your database configuration.
The test suite will create and use this database, to keep any data in your development database safe from tests.
Running the test suite is easy:
./manage.py test -s --noinput --logging-clear-handlers
Note that this will try (and fail) to run tests that depend on apps disabled via INSTALLED_APPS. You should run a
subset of tests:
2.8. Development
21
Kuma Documentation, Release latest
./manage.py test actioncounters authkeys contentflagging devmo landing kpi kuma
For more information, see the test documentation.
Kumascript tests
If you’re changing Kumascript, be sure to run its tests too. See https://github.com/mozilla/kumascript
2.8.8 Coding Conventions
Tests
• Avoid naming test files test_utils.py, since we use a library with the same name.
test__utils.py instead.
Use
• If you’re expecting reverse to return locales in the URL, use LocalizingClient instead of the default
client for the TestCase class.
2.9 Localization
Kuma is localized with gettext. User-facing strings in the code or templates need to be marked for gettext localization.
We use Verbatim to provide an easy interface to localizing these files. Localizers are also free to download the PO files
and use whatever tool they are comfortable with.
2.9.1 Making Strings Localizable
Making strings in templates localizable is exceptionally easy. Making strings in Python localizable is a little more
complicated. The short answer, though, is just wrap the string in _().
Interpolation
A string is often a combination of a fixed string and something changing, for example, Welcome, James is a
combination of the fixed part Welcome,, and the changing part James. The naive solution is to localize the first part
and the follow it with the name:
_(’Welcome, ’) + username
This is wrong!
In some locales, the word order may be different. Use Python string formatting to interpolate the changing part into
the string:
_(’Welcome, {name}’).format(name=username)
Python gives you a lot of ways to interpolate strings. The best way is to use Py3k formatting and kwargs. That’s the
clearest for localizers.
The worst way is to use %(label)s, as localizers seem to have all manner of trouble with it. Options like %s and
{0} are somewhere in the middle, and generally OK if it’s clear from context what they will be.
22
Chapter 2. Getting Started
Kuma Documentation, Release latest
Localization Comments
Sometimes, it can help localizers to describe where a string comes from, particularly if it can be difficult to find in the
interface, or is not very self-descriptive (e.g. very short strings). If you immediately precede the string with a comment
that starts L10n:, the comment will be added to the PO file, and visible to localizers.
Adding Context with msgctxt
Strings may be the same in English, but different in other languages. English, for example, has no grammatical gender,
and sometimes the noun and verb forms of a word are identical.
To make it possible to localize these correctly, we can add “context” (known in gettext as “msgctxt”) to differentiate
two otherwise identical strings.
For example, the string “Search” may be a noun or a verb in English. In a heading, it may be considered a noun, but
on a button, it may be a verb. It’s appropriate to add a context (like “button”) to one of them.
Generally, we should only add context if we are sure the strings aren’t used in the same way, or if localizers ask us to.
Plurals
“You have 1 new messages” grates on discerning ears. Fortunately, gettext gives us a way to fix that in English and
other locales, the ngettext function:
ngettext(’singular’, ’plural’, count)
A more realistic example might be:
ngettext(’Found {count} result.’,
’Found {count} results’,
len(results)).format(count=len(results))
This method takes three arguments because English only needs three, i.e., zero is considered “plural” for English.
Other locales may have different plural rules, and require different phrases for, say 0, 1, 2-3, 4-10, >10. That’s
absolutely fine, and gettext makes it possible.
Strings in Templates
When putting new text into a template, all you need to do is wrap it in a _() call:
<h1>{{ _(’Heading’) }}</h1>
Adding context is easy, too:
<h1>{{ _(’Heading’, ’context’) }}</h1>
L10n comments need to be Jinja2 comments:
{# L10n: Describes this heading #}
<h1>{{ _(’Heading’) }}</h1>
Note that Jinja2 escapes all content output through {{ }} by default. To put HTML in a string, you’ll need to add
the |safe filter:
<h1>{{ _(’Firefox <span>Help</span>’)|safe }}</h1>
2.9. Localization
23
Kuma Documentation, Release latest
To interpolate, you should use one of two Jinja2 filters: |f() or, in some cases, |fe(). |f() has exactly the same
arguments as u’’.format():
{{ _(’Welcome, {name}!’)|f(name=request.user.username) }}
The |fe() is exactly like the |f() filter, but escapes its arguments before interpolating, then returns a “safe” object.
Use it when the localized string contains HTML:
{{ _(’Found <strong>{0}</strong> results.’)|fe(num_results) }}
Note that you do not need to use |safe with |fe(). Also note that while it may look similar, the following is not
safe:
{{ _(’Found <strong>{0}</strong> results.’)|f(num_results)|safe }}
The ngettext function is also available:
{{ ngettext(’Found {0} result.’,
’Found {0} results.’,
num_results)|f(num_results) }}
Using {% trans %} Blocks for Long Strings
When a string is very long, i.e. long enough to make Github scroll sideways, it should be line-broken and put in a {%
trans %} block. {% trans %} blocks work like other block-level tags in Jinja2, except they cannot have other
tags, except strings, inside them.
The only thing that should be inside a {% trans %} block is printing a string with {{ string }}. These are
defined in the opening {% trans %} tag:
{% trans user=request.user.username %}
Thanks for registering, {{ user }}! We’re so...
hope that you’ll...
{% trans %}
Strings in Python
NB: Whenever you are adding a string in Python, ask yourself if it really needs to be there, or if it should be in the
template. Keep logic and presentation separate!
Strings in Python are more complex for two reasons:
1. We need to make sure we’re always using Unicode strings and the Unicode-friendly versions of the functions.
2. If you use the ugettext function in the wrong place, the string may end up in the wrong locale!
Here’s how you might localize a string in a view:
from tower import ugettext as _
def my_view(request):
if request.user.is_superuser:
msg = _(u’Oh hi, staff!’)
else:
msg = _(u’You are not staff!’)
Interpolation is done through normal Python string formatting:
24
Chapter 2. Getting Started
Kuma Documentation, Release latest
msg = _(u’Oh, hi, {user}’).format(user=request.user.username)
ugettext supports context, too:
msg = _(’Search’, ’context’)
L10n comments are normal one-line Python comments:
# L10n: A message to users.
msg = _(u’Oh, hi there!’)
If you need to use plurals, import the function ungettext from Tower:
from tower import ungettext, ugettext as _
n = len(results)
msg = ungettext(’Found {0} result’, ’Found {0} results’, n).format(n)
Lazily Translated Strings
You can use ugettext or ungettext only in views or functions called from views. If the function will be evaluated
when the module is loaded, then the string may end up in English or the locale of the last request! (We’re tracking
down that issue.)
Examples include strings in module-level code, arguments to functions in class definitions, strings in functions called
from outside the context of a view. To localize these strings, you need to use the _lazy versions of the above
methods, ugettext_lazy and ungettext_lazy. The result doesn’t get translated until it is evaluated as a
string, for example by being output or passed to unicode():
from tower import ugettext_lazy as _lazy
PAGE_TITLE = _lazy(u’Page Title’)
ugettext_lazy also supports context.
It is very important to pass Unicode objects to the _lazy versions of these functions. Failure to do so results in
significant issues when they are evaluated as strings.
If you need to work with a lazily-translated string, you’ll first need to convert it to a unicode object:
from tower import ugettext_lazy as _lazy
WELCOME = _lazy(u’Welcome, %s’)
def my_view(request):
# Fails:
WELCOME % request.user.username
# Works:
unicode(WELCOME) % request.user.username
2.9.2 Getting the Localizations
Localizations are not stored in this repository, but are in Mozilla’s SVN:
http://svn.mozilla.org/projects/mdn/trunk/locale
2.9. Localization
25
Kuma Documentation, Release latest
You don’t need the localization files for general development. However, if you need them for something, they’re pretty
easy to get:
$ cd kuma
$ svn checkout https://svn.mozilla.org/projects/mdn/trunk/locale
(Alternatively, you can do yourself a favor and use:
$ git svn clone -r HEAD https://svn.mozilla.org/projects/mdn/trunk/locale
if you’re a git fan.)
2.9.3 Updating the Localizations
Updating strings is easy. But when we add or update strings, we need to update Verbatim templates and PO files for
localizers. If you commit changes to SVN without updating Verbatim, localizers will have merge head-aches.
1. Check out the localizations (See get-localizations)
2. Run the following in the virtual machine (see installation-vagrant):
$ python manage.py extract
3. Commit the POT file.
If you used svn checkout above:
$ cd locale
$ svn up
$ svn ci -m "MDN string update YYYY-MM-DD"
If you used git svn clone above:
$
$
$
$
$
cd locale
git svn fetch
git add -A
git commit -m "MDN string update YYYY-MM-DD"
git svn dcommit
Note: You need verbatim permissions for the following. If you don’t have permissions, email groovecoder or mathjazz
to do the following ...
4. Go to the MDN templates on Verbatim
5. Click �Update all from VCS’
6. ssh to sm-verbatim01 (See L10n:Verbtim on wiki.mozilla.org)
7. Update all locales against templates:
sudo su verbatim
cd /data/www/localize.mozilla.org/verbatim/pootle_env/Pootle
POOTLE_SETTINGS=localsettings.py python2.6 manage.py
update_against_templates --project=mdn -v 2
2.9.4 Adding a new Locale
Adding a new locale is also easy.
26
Chapter 2. Getting Started
Kuma Documentation, Release latest
1. Check out the localizations (See get-localizations)
2. Follow the “Add locale” instructions on wiki.mozilla.org.
3. Update your locale repo to get the new locale:
$ cd locale
$ svn up
4. Add the locale to MDN_LANGUAGES in settings.py
5. Verify django loads new locale without errors by visiting the locale’s home page. E.g., https://developerlocal.allizom.org/ml/
6. BONUS: Use podebug to test a fake translation of the locale:
$ cd locale
$ podebug --rewrite=bracket templates/LC_MESSAGES/messages.pot
ml/LC_MESSAGES/messages.po
$ ./compile-mo.sh .
Restart the django server and re-visit the new locale to verify it shows “translated” strings in the locale.
2.10 The Kuma Test Suite
Kuma has a fairly comprehensive Python test suite. Changes should not break tests–only change a test if there is a
good reason to change the expected behavior–and new code should come with tests.
2.10.1 Running the Test Suite
If you followed the steps in the installation docs, then all you should need to do to run the test suite is:
./manage.py test
However, that doesn’t provide the most sensible defaults. Here is a good command to alias to something short:
./manage.py test -s --noinput --logging-clear-handlers
The -s flag is important if you want to be able to drop into PDB from within tests.
Some tests will fail. See Running a Subset below for running the subset that is expected to pass.
Some other helpful flags are:
-x: Fast fail. Exit immediately on failure. No need to run the whole test suite if you already know something is
broken.
--pdb: Drop into PDB on an uncaught exception. (These show up as E or errors in the test results, not F or failures.)
--pdb-fail: Drop into PDB on a test failure. This usually drops you right at the assertion.
Running a Subset
You can run part of the test suite by specifying the apps you want to run, like:
./manage.py test kuma
You can also exclude tests that match a regular expression with -e:
2.10. The Kuma Test Suite
27
Kuma Documentation, Release latest
./manage.py test -e "search"
To run the subset of tests that should pass:
./manage.py test actioncounters contentflagging devmo landing kuma
See the output of ./manage.py test --help for more arguments.
The Test Database
The test suite will create a new database named test_%s where %s is whatever value you have for
settings.DATABASES[’default’][’NAME’]. Make sure the user has ALL on the test database as well.
When the schema changes, you may need to drop the test database. You can also run the test suite with FORCE_DB
once to cause Django to drop and recreate it:
FORCE_DB=1 ./manage.py test -s --noinput --logging-clear-handlers
2.10.2 Adding Tests
Code should be written so it can be tested, and then there should be tests for it.
When adding code to an app, tests should be added in that app that cover the new functionality. All apps have a tests
module where tests should go. They will be discovered automatically by the test runner as long as the look like a test.
2.10.3 Changing Tests
Unless the current behavior, and thus the test that verifies that behavior is correct, is demonstrably wrong, don’t change
tests. Tests may be refactored as long as its clear that the result is the same.
2.10.4 Removing Tests
On those rare, wonderful occasions when we get to remove code, we should remove the tests for it, as well.
If we liberate some functionality into a new package, the tests for that functionality should move to that package, too.
28
Chapter 2. Getting Started
">