Zulip Documentation

Zulip Documentation
Zulip Documentation
Release 1.4.0
The Zulip Team
Aug 22, 2017
Overview
1
2
3
4
Zulip overview
1.1 Community . . . . . . . . . . . . . . . . . . .
1.2 Installing the Zulip Development environment
1.3 Running Zulip in production . . . . . . . . . .
1.4 Ways to contribute . . . . . . . . . . . . . . .
1.5 Google Summer of Code . . . . . . . . . . . .
1.6 How to get involved with contributing to Zulip
1.7 License . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3
3
4
4
4
5
5
6
Zulip architectural overview
2.1 Key Codebases . . . . . . . . . .
2.2 Usage assumptions and concepts .
2.3 Components . . . . . . . . . . .
2.4 Glossary . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
7
7
7
8
11
Directory structure
3.1 Core Python files . . . . . . . .
3.2 HTML Templates . . . . . . .
3.3 JavaScript and other static assets
3.4 Tests . . . . . . . . . . . . . .
3.5 Management commands . . . .
3.6 Scripts . . . . . . . . . . . . .
3.7 API and Bots . . . . . . . . . .
3.8 Production puppet configuration
3.9 Additional Django apps . . . .
3.10 Jinja2 Compatibility Files . . .
3.11 Translation files . . . . . . . .
3.12 Documentation . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
13
13
13
14
14
14
14
15
15
15
15
16
16
Zulip Roadmap
4.1 Introduction . . . . . . . . .
4.2 Major projects . . . . . . . .
4.3 Core User Experience . . . .
4.4 Social features . . . . . . . .
4.5 Real-time sync . . . . . . . .
4.6 Onboarding issues . . . . . .
4.7 Production installation issues
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
17
17
17
18
18
18
18
19
.
.
.
.
.
.
.
i
4.8
4.9
4.10
4.11
4.12
4.13
4.14
4.15
4.16
4.17
4.18
4.19
4.20
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
19
19
19
20
20
20
20
20
21
21
21
21
21
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
23
23
23
26
26
26
29
29
29
29
32
33
33
34
34
34
35
6
Requirements
6.1 Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2 Credentials needed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
37
37
38
7
Production Installation
7.1 Step 0: Subscribe . . . . . . . . . . . . . . .
7.2 Step 1: Install SSL Certificates . . . . . . . .
7.3 Step 2: Download and install latest release .
7.4 Step 3: Configure Zulip . . . . . . . . . . .
7.5 Step 4: Initialize Zulip database . . . . . . .
7.6 Step 5: Create a Zulip organization and login
7.7 Troubleshooting . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
39
39
39
39
40
40
41
41
8
Troubleshooting
8.1 Using supervisorctl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.2 Troubleshooting services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
43
43
45
9
Customize Zulip
9.1 Mobile and desktop apps . . . . . .
9.2 Terms of service and Privacy policy
9.3 Miscellaneous server settings . . .
9.4 Zulip announcement list . . . . . .
9.5 Enjoy your Zulip installation! . . .
47
47
47
48
48
48
5
ii
Administration and management .
Scalability and performance . . .
Technology improvements . . . .
Technical Debt . . . . . . . . . .
Security . . . . . . . . . . . . . .
Testing . . . . . . . . . . . . . .
Documentation . . . . . . . . . .
Integrations and bots . . . . . . .
Android app . . . . . . . . . . .
iOS app . . . . . . . . . . . . . .
Server/webapp support for mobile
Desktop apps . . . . . . . . . . .
Community . . . . . . . . . . . .
Version History
5.1 Unreleased . . . . .
5.2 1.6.0 – 2017-06-06 .
5.3 1.5.2 – 2017-06-01 .
5.4 1.5.1 – 2017-02-07 .
5.5 1.5.0 – 2017-02-06 .
5.6 1.4.3 - 2017-01-29 .
5.7 1.4.2 - 2016-09-27 .
5.8 1.4.1 - 2016-09-03 .
5.9 1.4.0 - 2016-08-25 .
5.10 1.3.13 - 2016-06-21
5.11 1.3.12 - 2016-05-10
5.12 1.3.11 - 2016-05-02
5.13 1.3.10 - 2016-01-21
5.14 1.3.9 - 2015-11-16 .
5.15 1.3.8 - 2015-11-15 .
5.16 1.3.7 - 2015-10-19 .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
10 Mobile push notification service
10.1 How to sign up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10.2 Why this is necessary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10.3 Security and privacy implications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
49
49
49
50
11 Secure, maintain, and upgrade
11.1 Upgrading . . . . . . . . . . .
11.2 Upgrading from a git repository
11.3 Backups . . . . . . . . . . . .
11.4 Monitoring . . . . . . . . . . .
11.5 Scalability . . . . . . . . . . .
11.6 Securing your Zulip server . . .
11.7 Management commands . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
51
51
53
53
55
55
56
56
12 Security Model
12.1 Secure your Zulip server like your email server
12.2 Encryption and Authentication . . . . . . . . .
12.3 Messages and History . . . . . . . . . . . . .
12.4 Users and Bots . . . . . . . . . . . . . . . . .
12.5 User-uploaded content . . . . . . . . . . . . .
12.6 Final notes and security response . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
59
59
59
60
61
61
62
13 Authentication methods
13.1 Adding additional methods using python-social-auth . . . . . . . . . . . . . . . . . . . . . . . . . .
13.2 Remote User SSO Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
63
63
64
14 Postgres database details
14.1 Remote Postgres database . . . . . .
14.2 Debugging postgres database issues .
14.3 Stopping the Zulip postgres database
14.4 Debugging issues starting postgres . .
14.5 Postgres Vacuuming alerts . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
67
67
68
68
68
69
15 Development environment installation
15.1 Requirements . . . . . . . . . . .
15.2 Recommended setup (Vagrant) . .
15.3 Advanced setup (non-Vagrant) . .
15.4 Slow internet connections . . . .
15.5 Installing remotely . . . . . . . .
15.6 Next steps . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
71
71
71
71
72
72
72
16 Vagrant environment setup tutorial
16.1 Requirements . . . . . . . . . . . . . . . .
16.2 Step 1: Install Prerequisites . . . . . . . .
16.3 Step 2: Get Zulip Code . . . . . . . . . . .
16.4 Step 3: Start the development environment
16.5 Step 4: Developing . . . . . . . . . . . . .
16.6 Next Steps . . . . . . . . . . . . . . . . .
16.7 Troubleshooting & Common Errors . . . .
16.8 Specifying a proxy . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
73
74
74
77
77
81
83
83
88
17 Zulip development environment setup without Vagrant
17.1 Installing directly on Ubuntu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.2 Installing manually on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17.3 Using Docker (experimental) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
91
91
91
97
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
iii
18 Using the Development Environment
99
19 Developing on a remote machine
19.1 Connecting to the remote environment . . . . . . . . . . .
19.2 Setting up the development environment . . . . . . . . . .
19.3 Running the development server . . . . . . . . . . . . . . .
19.4 Making changes to code on your remote development server
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
101
101
101
102
103
20 Writing a new integration
20.1 Types of integrations . . . . . . . . .
20.2 General advice . . . . . . . . . . . .
20.3 Webhook integrations . . . . . . . .
20.4 Python script and plugin integrations
20.5 Documenting your integration . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
107
107
108
108
109
110
21 Webhook walkthrough
21.1 Step 0: Create fixtures . . . . . . . . . . . . . .
21.2 Step 1: Initialize your webhook python package
21.3 Step 2: Create main webhook code . . . . . . .
21.4 Step 3: Create an api endpoint for the webhook .
21.5 Step 4: Create tests . . . . . . . . . . . . . . . .
21.6 Step 5: Create documentation . . . . . . . . . .
21.7 Step 5: Preparing a pull request to zulip/zulip . .
21.8 Advanced topics . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
111
111
112
112
113
114
116
117
117
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
22 Writing a new application feature
121
22.1 General Process in brief . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
22.2 Example Feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
23 Writing views in Zulip
23.1 What this covers . . . . . . . . . . . . .
23.2 What is a view? . . . . . . . . . . . . . .
23.3 Modifying urls.py . . . . . . . . . . . .
23.4 Writing human-readable views . . . . . .
23.5 Writing API REST endpoints . . . . . .
23.6 Legacy endpoints used by the web client
23.7 Webhook integration endpoints . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
129
129
129
129
130
130
134
134
24 Life of a Request
24.1 A request is sent to the server, and handled by Nginx . . . . . . . . . . . . .
24.2 Nginx secures traffic with SSL . . . . . . . . . . . . . . . . . . . . . . . . .
24.3 Static files are served directly by Nginx . . . . . . . . . . . . . . . . . . . .
24.4 Nginx routes other requests between django and tornado . . . . . . . . . . .
24.5 Django routes the request to a view in urls.py files . . . . . . . . . . . . . .
24.6 Views serving HTML are internationalized by server path . . . . . . . . . .
24.7 API endpoints use REST . . . . . . . . . . . . . . . . . . . . . . . . . . . .
24.8 Django calls rest_dispatch for REST endpoints, and authenticates . . . . . .
24.9 The view will authorize the user, extract request variables, and validate them
24.10 Results are given as JSON . . . . . . . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
135
135
135
135
136
136
136
136
137
138
138
25 Reading list
25.1 General programming/IT
25.2 Python . . . . . . . . .
25.3 Java/Android . . . . . .
25.4 JavaScript/ECMAScript
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
139
139
140
140
140
iv
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
25.5
25.6
25.7
25.8
25.9
Git/Version Control Systems (VCS) . . . . . . . .
Computer Science/Algorithms . . . . . . . . . . .
Community experience . . . . . . . . . . . . . . .
Competitions/Camps . . . . . . . . . . . . . . . .
Massive Open Online Courses (MOOC) Platforms
26 Git & GitHub Guide
26.1 Quick start: How Zulip uses Git and GitHub
26.2 Set up Git . . . . . . . . . . . . . . . . . . .
26.3 How Git is different . . . . . . . . . . . . .
26.4 Important Git terms . . . . . . . . . . . . .
26.5 Get Zulip code . . . . . . . . . . . . . . . .
26.6 Using Git as you work . . . . . . . . . . . .
26.7 Create a pull request . . . . . . . . . . . . .
26.8 Update a pull request . . . . . . . . . . . . .
26.9 Collaborate . . . . . . . . . . . . . . . . . .
26.10 Review changes . . . . . . . . . . . . . . .
26.11 Get and stay out of trouble . . . . . . . . . .
26.12 Zulip-specific tools . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
140
140
141
141
141
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
143
143
144
144
145
147
148
155
157
157
158
158
162
27 Version control
165
27.1 Commit Discipline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
27.2 Commit Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
28 Code style and conventions
28.1 Be consistent! . . . . . . .
28.2 Lint tools . . . . . . . . . .
28.3 Secrets . . . . . . . . . . .
28.4 Dangerous constructs . . . .
28.5 JS array/object manipulation
28.6 More arbitrary style things .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
169
169
169
170
170
172
173
29 Python static type checker (mypy)
29.1 type_debug.py . . . . . . . . . . . . . . . . . . . . . .
29.2 Zulip goals . . . . . . . . . . . . . . . . . . . . . . . . . .
29.3 Installing mypy . . . . . . . . . . . . . . . . . . . . . . . .
29.4 Running mypy on Zulip’s code locally . . . . . . . . . . .
29.5 Excluded files . . . . . . . . . . . . . . . . . . . . . . . .
29.6 Mypy is there to find bugs in Zulip before they impact users
29.7 Annotating strings . . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
177
177
178
178
178
179
179
179
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
30 Reviewing Zulip server code
181
30.1 Things to look for . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
30.2 Tooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
30.3 Additional Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
31 The chat.zulip.org community
31.1 This is a bleeding edge development server
31.2 Community conventions . . . . . . . . . .
31.3 Streams . . . . . . . . . . . . . . . . . . .
31.4 Chat meetings . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
183
183
183
184
184
32 Using zulipbot
187
32.1 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
v
33 Accessibility
33.1 Guidelines . . . . . .
33.2 Tools . . . . . . . . .
33.3 GitHub Issues . . . .
33.4 Additional Resources .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
189
189
189
190
190
34 Testing and writing tests
34.1 Overview . . . . . . . . . . . . . . . . . . . . .
34.2 Running tests . . . . . . . . . . . . . . . . . . .
34.3 Schema and initial data changes . . . . . . . . .
34.4 Wiping the test databases . . . . . . . . . . . .
34.5 Local browser testing (local app + web browser)
34.6 Python 3 Compatibility . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
191
191
191
192
192
193
193
35 Linters
35.1 Overview . . . . . . .
35.2 Running the linters . .
35.3 General considerations
35.4 Lint checks . . . . . .
35.5 lint . . . . . . . . . .
35.6 Philosophy . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
195
195
195
196
196
197
198
36 Backend Django tests
36.1 Overview . . . . . . . .
36.2 Running tests . . . . . .
36.3 How to write tests. . . .
36.4 Zulip Testing Philosophy
36.5 Testing considerations .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
201
201
201
202
202
205
37 JavaScript unit tests
37.1 HTML output . . . . . . . . . . . .
37.2 Coverage reports . . . . . . . . . .
37.3 Handling dependencies in unit tests
37.4 Creating new test modules . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
207
207
208
208
209
38 Web frontend black-box casperjs tests
211
38.1 Debugging Casper.JS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
38.2 Writing Casper tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
39 Travis CI
39.1 Goals . . . . . . . . . . . . . .
39.2 Configuration . . . . . . . . . .
39.3 Useful debugging tips and tools
39.4 Performance optimizations . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
215
215
215
216
216
40 Manual testing
219
40.1 Basic Stuff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
41 Settings system
231
41.1 Server settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
41.2 Realm settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
42 Real-time Push and Events
235
42.1 Generation system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
42.2 Delivery system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
vi
42.3 The initial data fetch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
43 Queue processors
239
43.1 Adding a new queue processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
43.2 Publishing events into a queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
43.3 Clearing a RabbitMQ queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
44 Zulip bot system
44.1 The bots system . .
44.2 How to run a bot . .
44.3 How to develop a bot
44.4 Bot API . . . . . . .
44.5 Common problems .
44.6 Future direction . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
241
241
242
243
244
246
247
45 Custom Apps
45.1 Definition . . . . . . . . . . . . .
45.2 Problem statement . . . . . . . .
45.3 A quick note on bots/integrations
45.4 Categories of custom apps . . . .
45.5 World Reader . . . . . . . . . . .
45.6 Zulip Reader . . . . . . . . . . .
45.7 Deployment issues . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
249
249
249
249
250
251
253
253
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
46 Unread counts and the pointer
257
46.1 Pointer logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
46.2 Unread count logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
46.3 Testing and development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
47 Markdown implementation
47.1 Testing . . . . . . . . . . . . . . . . .
47.2 Changing Zulip’s markdown processor
47.3 Zulip’s Markdown philosophy . . . . .
47.4 Zulip’s Changes to Markdown . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
261
261
262
262
263
48 Realms in Zulip
265
48.1 Creating Realms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
48.2 Subdomains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
49 Static asset pipeline
49.1 Primary build process . . . . . . . . . . . . .
49.2 Adding static files . . . . . . . . . . . . . . .
49.3 How it works in production . . . . . . . . . .
49.4 Webpack/CommonJS/ES6/Typescript modules
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
50 Schema Migrations
51 HTML and CSS
51.1 Zulip CSS organization
51.2 Editing Zulip CSS . . .
51.3 CSS Style guidelines . .
51.4 Validating CSS . . . . .
267
267
267
268
268
269
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
271
271
271
272
272
52 URL hashes and deep linking
273
52.1 Hashchange . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
52.2 Server-initiated reloads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
vii
52.3 All reloads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
53 Emoji
275
53.1 Emoji codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
53.2 Tooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
54 Full-text search
277
54.1 The default full-text search implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
54.2 An optional full-text search implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
55 Analytics
55.1 Analytics backend overview . . . . .
55.2 The *Count database tables . . . . .
55.3 CountStats . . . . . . . . . . . . . .
55.4 The FillState table . . . . . . . . . .
55.5 Performance strategy . . . . . . . . .
55.6 Backend Testing . . . . . . . . . . .
55.7 LoggingCountStats . . . . . . . . . .
55.8 Analytics UI development and testing
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
279
279
280
280
280
280
281
281
282
56 Translating Zulip
56.1 Translation style guides . . . . . . .
56.2 Translation process . . . . . . . . . .
56.3 Translators’ workflow . . . . . . . .
56.4 Testing translations . . . . . . . . . .
56.5 Setting the default language in Zulip .
56.6 Translation resource files . . . . . . .
56.7 Backend translations . . . . . . . . .
56.8 Frontend translations . . . . . . . . .
56.9 Transifex config . . . . . . . . . . .
56.10 Transifex CLI setup . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
283
283
284
284
285
286
286
286
287
288
288
57 Clients in Zulip
289
57.1 Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
57.2 Integrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
58 Logging and Performance Debugging
59 Zulip server release checklist
59.1 A week before the release
59.2 Final release preparation .
59.3 Executing the release . . .
59.4 Post-release . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
291
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
293
293
293
294
294
60 Zulip PyPI package release checklist
295
61 Documentation
61.1 Developer and sysadmin documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
61.2 Core website documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
61.3 General user documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
297
297
298
298
62 General user guide documentation
62.1 Editing and testing . . . . . .
62.2 Writing documentation . . . .
62.3 Features . . . . . . . . . . . .
62.4 Documentation template . . .
299
299
299
301
308
viii
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Zulip Documentation, Release 1.4.0
Zulip is a powerful, open source group chat application. Written in Python and using the Django framework, Zulip
supports both private messaging and group chats via conversation streams.
Zulip also supports fast search, drag-and-drop file uploads, image previews, group private messages, audible notifications, missed-message emails, desktop apps, and much more.
Further information on the Zulip project and its features can be found at https://www.zulip.org and in these docs. Our
code is available at our GitHub repository.
This set of documents covers installation and contribution instructions.
Contents:
• Overview
• Zulip in production
• Development environment
• Developer tutorials
• Code contribution guide
• Code testing
• Subsystem documentation
Zulip overview | Community | Installing for dev | Installing for production | Ways to contribute | How to get involved
| License
Overview
1
Zulip Documentation, Release 1.4.0
2
Overview
CHAPTER
1
Zulip overview
Zulip is a powerful, open source group chat application. Written in Python and using the Django framework, Zulip
supports both private messaging and group chats via conversation streams.
Zulip also supports fast search, drag-and-drop file uploads, image previews, group private messages, audible notifications, missed-message emails, desktop apps, and much more.
Further information on the Zulip project and its features can be found at https://www.zulip.org.
Community
There are several places online where folks discuss Zulip.
• The primary place is the Zulip development community Zulip server.
• For Google Summer of Code students and applicants, we have a mailing list for help, questions, and announcements. But it’s often simpler to visit chat.zulip.org instead.
• We have a public mailing list that is currently pretty low traffic because most discussions happen in our public
Zulip instance. We use it to announce Zulip developer community gatherings and ask for feedback on major
technical or design decisions. It has several hundred subscribers, so you can use it to ask questions about features
or possible bugs, but please don’t use it ask for generic help getting started as a contributor (e.g. because you
want to do Google Summer of Code). The rest of this page covers how to get involved in the Zulip project in
detail.
• Zulip also has a blog and twitter account.
• Last but not least, we use GitHub to track Zulip-related issues (and store our code, of course). Anybody with a
GitHub account should be able to create Issues there pertaining to bugs or enhancement requests. We also use
Pull Requests as our primary mechanism to receive code contributions.
The Zulip community has a Code of Conduct.
3
Zulip Documentation, Release 1.4.0
Installing the Zulip Development environment
The Zulip development environment is the recommended option for folks interested in trying out Zulip. This is
documented in the developer installation guide.
Running Zulip in production
Zulip in production supports Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial. Work is ongoing on adding support for
additional platforms. The installation process is documented at https://zulip.org/server.html and in more detail in the
documentation.
Ways to contribute
Zulip welcomes all forms of contributions! This page documents the Zulip development process.
• Pull requests. Before a pull request can be merged, you need to sign the Dropbox Contributor License Agreement. Also, please skim our commit message style guidelines. We encourage early pull requests for work in
progress. Prefix the title of your pull request with [WIP] and reference it when asking for community feedback.
When you are ready for final review, remove the [WIP].
• Testing. The Zulip automated tests all run automatically when you submit a pull request, but you can also run
them all in your development environment following the instructions in the testing docs. You can also try out
our new desktop client, which is in alpha; we’d appreciate testing and feedback.
• Developer Documentation. Zulip has a growing collection of developer documentation on Read The Docs.
Recommended reading for new contributors includes the directory structure and new feature tutorial. You can
also improve Zulip.org.
• Mailing lists and bug tracker. Zulip has a development discussion mailing list and uses GitHub issues . There
are also lists for the Android and iOS apps. Feel free to send any questions or suggestions of areas where
you’d love to see more documentation to the relevant list! Please report any security issues you discover to
zulip-security@googlegroups.com.
• App codebases. This repository is for the Zulip server and web app (including most integrations); the React
Native Mobile iOS app, Android app, new Electron desktop app, and legacy QT-based desktop app are all
separate repositories.
• Glue code. We maintain a Hubot adapter and several integrations (Phabricator, Jenkins, Puppet, Redmine,
and Trello), plus node.js API bindings, an isomorphic JavaScript library, and a full-text search PostgreSQL
extension, as separate repos.
• Translations. Zulip is in the process of being translated into 10+ languages, and we love contributions to our
translations. See our translating documentation if you’re interested in contributing!
• Code Reviews. Zulip is all about community and helping each other out. Check out #code review on
chat.zulip.org to help review PRs and give comments on other people’s work. Everyone is welcome to participate, even those new to Zulip! Even just checking out the code, manually testing it, and posting on whether
or not it worked is valuable.
4
Chapter 1. Zulip overview
Zulip Documentation, Release 1.4.0
Google Summer of Code
We participated in GSoC in 2016 (with great results) and are participating in 2017 as well. For guidance, please read
our GSoC instructions and ideas page and feel free to email our GSoC mailing list.
Note: For GSoC 2017, we will be focused on making our React Native app better rather than developing the Java
Android app and React Native app in parallel. You can review our detailed plan for further details on the motivation
and logistics.
How to get involved with contributing to Zulip
First, subscribe to the Zulip development discussion mailing list.
The Zulip project uses a system of labels in our issue tracker to make it easy to find a project if you don’t have your
own project idea in mind or want to get some experience with working on Zulip before embarking on a larger project
you have in mind:
• Integrations. Integrate Zulip with another piece of software and contribute it back to the community! Writing
an integration can be a great first contribution. There’s detailed documentation on how to write integrations in
the Zulip integration writing guide.
• Bite Size: Smaller projects that might be a great first contribution.
• Documentation: The Zulip project loves contributions of new documentation.
• Help Wanted: A broader list of projects that nobody is currently working on.
• Platform support: These are open issues about making it possible to install Zulip on a wider range of platforms.
• Bugs: Open bugs.
• Feature requests: Browsing this list can be a great way to find feature ideas to implement that other Zulip users
are excited about.
• 2016 roadmap milestone: The projects that are priorities for the Zulip project. These are great projects if you’re
looking to make an impact.
Another way to find issues in Zulip is to take advantage of our “area:” convention in separating out issues. We partition
all of our issues into areas like admin, compose, emoji, hotkeys, i18n, onboarding, search, etc. You can see this here:
https://github.com/zulip/zulip/labels
Click on any of the “area:” labels and you will see all the tickets related to your area of interest.
If you’re excited about helping with an open issue, make sure to claim the issue by commenting the following in the
comment section: “@zulipbot claim”. @zulipbot will assign you to the issue and label the issue as in progress. For
more details, check out @zulipbot.
You’re encouraged to ask questions on how to best implement or debug your changes – the Zulip maintainers are
excited to answer questions to help you stay unblocked and working efficiently. It’s great to ask questions in comments
on GitHub issues and pull requests, or on chat.zulip.org. We’ll direct longer discussions to Zulip chat, but please post
a summary of what you learned from the chat, or link to the conversation, in a comment on the GitHub issue.
We also welcome suggestions of features that you feel would be valuable or changes that you feel would make Zulip a
better open source project, and are happy to support you in adding new features or other user experience improvements
to Zulip.
If you have a new feature you’d like to add, we recommend you start by opening a GitHub issue about the feature
idea explaining the problem that you’re hoping to solve and that you’re excited to work on it. A Zulip maintainer will
usually reply within a day with feedback on the idea, notes on any important issues or concerns, and and often tips on
1.5. Google Summer of Code
5
Zulip Documentation, Release 1.4.0
how to implement or test it. Please feel free to ping the thread if you don’t hear a response from the maintainers – we
try to be very responsive so this usually means we missed your message.
For significant changes to the visual design, user experience, data model, or architecture, we highly recommend
posting a mockup, screenshot, or description of what you have in mind to zulip-devel@ to get broad feedback before
you spend too much time on implementation details.
Finally, before implementing a larger feature, we highly recommend looking at the new feature tutorial and coding
style guidelines on ReadTheDocs.
Feedback on how to make this development process more efficient, fun, and friendly to new contributors is very
welcome! Just send an email to the Zulip Developers list with your thoughts.
When you feel like you have completed your work on an issue, post your PR to the #code review stream on
chat.zulip.org. This is our lightweight process that gives other developers the opportunity to give you comments
and suggestions on your work.
License
Copyright 2011-2017 Dropbox, Inc., Kandra Labs, Inc., and contributors
Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an
“AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under the License.
The software includes some works released by third parties under other free and open source licenses. Those works
are redistributed under the license terms under which the works were received. For more details, see the docs/
THIRDPARTY file included with this distribution.
6
Chapter 1. Zulip overview
CHAPTER
2
Zulip architectural overview
Key Codebases
The core Zulip application is at https://github.com/zulip/zulip and is a web application written in Python 2.7 (soon to
also support Python 3) and using the Django framework. That codebase includes server-side code and the web client,
as well as Python API bindings and most of our integrations with other services and applications (see the directory
structure guide).
We maintain several separate repositories for integrations and other glue code: a Hubot adapter; integrations with
Phabricator, Jenkins, Puppet, Redmine, and Trello; node.js API bindings; and our full-text search PostgreSQL extension.
Our mobile clients are separate code repositories: Android and React Native iOS app. Our legacy desktop application
(implemented in QT/WebKit) and our new, alpha cross-platform desktop app (implemented in Electron) are also
separate repositories.
We use Transifex to do translations.
In this overview, we’ll mainly discuss the core Zulip server and web application.
Usage assumptions and concepts
Zulip is a real-time web-based chat application meant for companies and similar groups ranging in size from a small
team to more than a thousand users. It features real-time notifications, message persistence and search, public group
conversations (streams), invite-only streams, private one-on-one and group conversations, inline image previews, team
presence/buddy lists, a rich API, Markdown message support, and numerous integrations with other services. The
maintainer team aims to support users who connect to Zulip using dedicated iOS, Android, Linux, Windows, and
macOS clients, as well as people using modern web browsers or dedicated Zulip API clients.
A server can host multiple Zulip realms (organizations) at the same domain, each of which is a private chamber with
its own users, streams, customizations, and so on. This means that one person might be a user of multiple Zulip
realms. The administrators of a realm can choose whether to allow anyone to register an account and join, or only
allow people who have been invited, or restrict registrations to members of particular groups (using email domain
7
Zulip Documentation, Release 1.4.0
names or corporate single-sign-on login for verification). For more on security considerations, see the security model
section.
The default Zulip home screen is like a chronologically ordered inbox; it displays messages, starting at the oldest
message that the user hasn’t viewed yet (for more on that logic, see the guide to the pointer and unread counts). The
home screen displays the most recent messages in all the streams a user has joined (except for the streams they’ve
muted), as well as private messages from other users, in strict chronological order. A user can narrow to view only the
messages in a single stream, and can further narrow to focus on a topic (thread) within that stream. Each narrow has
its own URL. The user can quickly see what conversation they’re in – the stream and topic, or the names of the user(s)
they’re private messaging with – using the recipient bar displayed atop each conversation.
Zulip’s philosophy is to provide sensible defaults but give the user fine-grained control over their incoming information
flow; a user can mute topics and streams, and can make fine-grained choices to reduce real-time notifications they find
irrelevant.
Components
Django and Tornado
Zulip is primarily implemented in the Django Python web framework. We also make use of Tornado for the real-time
push system.
Django is the main web application server; Tornado runs the server-to-client real-time push system. The app servers
are configured by the Supervisor configuration (which explains how to start the server processes; see “Supervisor”
below) and the nginx configuration (which explains which HTTP requests get sent to which app server).
8
Chapter 2. Zulip architectural overview
Zulip Documentation, Release 1.4.0
Tornado is an asynchronous server and is meant specifically to hold open tens of thousands of long-lived (long-polling
or websocket) connections – that is to say, routes that maintain a persistent connection from every running client. For
this reason, it’s responsible for event (message) delivery, but not much else. We try to avoid any blocking calls in
Tornado because we don’t want to delay delivery to thousands of other connections (as this would make Zulip very
much not real-time). For instance, we avoid doing cache or database queries inside the Tornado code paths, since those
blocking requests carry a very high performance penalty for a single-threaded, asynchronous server.
The parts that are activated relatively rarely (e.g. when people type or click on something) are processed by the Django
application server. One exception to this is that Zulip uses websockets through Tornado to minimize latency on the
code path for sending messages.
There is detailed documentation on the real-time push and event queue system; most of the code is in zerver/
tornado.
nginx
nginx is the front-end web server to all Zulip traffic; it serves static assets and proxies to Django and Tornado. It
handles HTTP requests according to the rules laid down in the many config files found in zulip/puppet/zulip/
files/nginx/.
zulip/puppet/zulip/files/nginx/zulip-include-frontend/app is the most important of these
files. It explains what happens when requests come in from outside.
• In production, all requests to URLs beginning with /static/ are served from the corresponding files in /
home/zulip/prod-static/, and the production build process (tools/build-release-tarball)
compiles, minifies, and installs the static assets into the prod-static/ tree form. In development, files are
served directly from /static/ in the git repository.
• Requests to /json/events, /api/v1/events, and /sockjs are sent to the Tornado server. These are
requests to the real-time push system, because the user’s web browser sets up a long-lived TCP connection
with Tornado to serve as a channel for push notifications. nginx gets the hostname for the Tornado server via
puppet/zulip/files/nginx/zulip-include-frontend/upstreams.
• Requests to all other paths are sent to the Django app via the UNIX socket unix:/home/
zulip/deployments/uwsgi-socket
(defined
in
puppet/zulip/files/nginx/
zulip-include-frontend/upstreams). We use zproject/wsgi.py to implement uWSGI
here (see django.core.wsgi).
Supervisor
We use supervisord to start server processes, restart them automatically if they crash, and direct logging.
The config file is zulip/puppet/zulip/templates/supervisor/zulip.conf.template.erb. This
is where Tornado and Django are set up, as well as a number of background processes that process event queues. We
use event queues for the kinds of tasks that are best run in the background because they are expensive (in terms of
performance) and don’t have to be synchronous — e.g., sending emails or updating analytics. Also see the queuing
guide.
memcached
memcached is used to cache database model objects.
zerver/lib/cache.py and zerver/lib/
cache_helpers.py manage putting things into memcached, and invalidating the cache when values change. The
memcached configuration is in puppet/zulip/files/memcached.conf.
2.3. Components
9
Zulip Documentation, Release 1.4.0
Redis
Redis is used for a few very short-term data stores, such as in the basis of zerver/lib/rate_limiter.py, a
per-user rate limiting scheme example), and the email-to-Zulip integration.
Redis is configured in zulip/puppet/zulip/files/redis and it’s a pretty standard configuration except for
the last line, which turns off persistence:
# Zulip-specific configuration: disable saving to disk.
save ""
memcached was used first and then we added Redis specifically to implement rate limiting. We’re discussing switching
everything over to Redis.
RabbitMQ
RabbitMQ is a queueing system. Its config files live in zulip/puppet/zulip/files/rabbitmq. Initial
configuration happens in zulip/scripts/setup/configure-rabbitmq.
We use RabbitMQ for queuing expensive work (e.g. sending emails triggered by a message, push notifications, some
analytics, etc.) that require reliable delivery but which we don’t want to do on the main thread. It’s also used for
communication between the application server and the Tornado push system.
Two simple wrappers around pika (the Python RabbitMQ client) are in zulip/zerver/lib/queue.py.
There’s an asynchronous client for use in Tornado and a more general client for use elsewhere. Most of the processes started by Supervisor are queue processors that continually pull things out of a RabbitMQ queue and handle
them; they are defined in zerver/worker/queue_processors.py.
Also see the queuing guide.
PostgreSQL
PostgreSQL (also known as Postgres) is the database that stores all persistent data, that is, data that’s expected to live
beyond a user’s current session.
In production, Postgres is installed with a default configuration. The directory that would contain configuration files
(puppet/zulip/files/postgresql) has only a utility script and a custom list of stopwords used by a Postgresql extension.
In a development environment, configuration of that postgresql extension is handled by tools/
postgres-init-dev-db (invoked by tools/provision). That file also manages setting up the development
postgresql user.
tools/provision also invokes tools/do-destroy-rebuild-database to create the actual database
with its schema.
Nagios
Nagios is an optional component used for notifications to the system administrator, e.g., in case of outages.
zulip/puppet/zulip/manifests/nagios.pp installs Nagios plugins from puppet/zulip/files/
nagios_plugins/.
This component is intended to install Nagios plugins intended to be run on a Nagios server; most of the Zulip Nagios
plugins are intended to be run on the Zulip servers themselves, and are included with the relevant component of
10
Chapter 2. Zulip architectural overview
Zulip Documentation, Release 1.4.0
the Zulip server (e.g. puppet/zulip/manifests/postgres_common.pp installs a few under /usr/lib/
nagios/plugins/zulip_postgres_common).
Glossary
This section gives names for some of the elements in the Zulip UI used in Zulip development conversations. Contributions to extend this list are welcome!
• chevron: A small downward-facing arrow next to a message’s timestamp, offering contextual options, e.g.,
“Reply”, “Mute [this topic]”, or “Link to this conversation”. To avoid visual clutter, the chevron only appears in
the web UI upon hover.
• huddle: What the codebase calls a “group private message”.
• message editing: If the realm admin allows it, then after a user posts a message, the user has a few minutes to
click “Edit” and change the content of their message. If they do, Zulip adds a marker such as “(EDITED)” at
the top of the message, visible to anyone who can see the message.
• realm: What the codebase calls an “organization” in the UI.
• recipient bar: A visual indication of the context of a message or group of messages, displaying the stream and
topic or private message recipient list, at the top of a group of messages. A typical 1-line message to a new
recipient shows to the user as three lines of content: first the recipient bar, second the sender’s name and avatar
alongside the timestamp (and, on hover, the star and the chevron), and third the message content. The recipient
bar is or contains hyperlinks to help the user narrow.
• star: Zulip allows a user to mark any message they can see, public or private, as “starred”. A user can easily access messages they’ve starred through the “Starred messages” link in the menu near “Home”, or use “is:starred”
as a narrow or a search constraint. Whether a user has or has not starred a particular message is private; other
users and realm admins don’t know whether a message has been starred, or by whom.
• subject: What the codebase calls a “topic” in many places.
• bankruptcy: When a user has been off Zulip for several days and has hundreds of unread messages, they are
prompted for whether they want to mark all their unread messages as read. This is called “declaring bankruptcy”
(in reference to the concept in finance).
2.4. Glossary
11
Zulip Documentation, Release 1.4.0
12
Chapter 2. Zulip architectural overview
CHAPTER
3
Directory structure
This page documents the Zulip directory structure, where to find things, and how to decide where to put a file.
You may also find the new application feature tutorial helpful for understanding the flow through these files.
Core Python files
Zulip uses the Django web framework, so a lot of these paths will be familiar to Django developers.
• zproject/urls.py Main Django routes file. Defines which URLs are handled by which view functions or
templates.
• zerver/models.py Main Django models file. Defines Zulip’s database tables.
• zerver/lib/actions.py Most code doing writes to user-facing database tables.
• zerver/views/*.py Most Django views.
• zerver/webhooks/ Webhook views and tests for Zulip webhook integrations.
• zerver/tornado/views.py Tornado views.
• zerver/worker/queue_processors.py Queue workers.
• zerver/lib/*.py Most library code.
• zerver/lib/bugdown/ Backend Markdown processor.
• zproject/backends.py Authentication backends.
HTML Templates
See our translating docs for details on Zulip’s templating systems.
13
Zulip Documentation, Release 1.4.0
• templates/zerver/ For Jinja2 templates for the backend (for zerver app).
• static/templates/ Handlebars templates for the frontend.
JavaScript and other static assets
• static/js/ Zulip’s own JavaScript.
• static/styles/ Zulip’s own CSS.
• static/images/ Zulip’s images.
• static/third/ Third-party JavaScript and CSS that has been vendored.
• node_modules/ Third-party JavaScript installed via npm.
• static/assets/ For assets not to be served to the web (e.g. the system to generate our favicons).
Tests
• zerver/tests/ Backend tests.
• frontend_tests/node_tests/ Node Frontend unit tests.
• frontend_tests/casper_tests/ Casper frontend tests.
• tools/test-* Developer-facing test runner scripts.
Management commands
These are distinguished from scripts, below, by needing to run a Django context (i.e. with database access).
• zerver/management/commands/ Management commands one might run at a production deployment site
(e.g. scripts to change a value or deactivate a user properly).
Scripts
• scripts/ Scripts that production deployments might run manually (e.g., restart-server).
• scripts/lib/ Scripts that are needed on production deployments but humans should never run directly.
• scripts/setup/ Scripts that production deployments will only run once, during installation.
• tools/ Scripts used only in a Zulip development environment. These are not included in production release
tarballs for Zulip, so that we can include scripts here one wouldn’t want someone to run in production accidentally (e.g. things that delete the Zulip database without prompting).
14
Chapter 3. Directory structure
Zulip Documentation, Release 1.4.0
• tools/setup/ Subdirectory of tools/ for things only used during the development environment setup
process.
• tools/travis/ Subdirectory of tools/ for things only used to setup and run our tests in Travis CI. Actual
test suites should go in tools/.
API and Bots
• api/ Zulip’s Python API bindings (released separately).
• api/examples/ API examples.
• api/integrations/ Integrations distributed as part of the Zulip API bundle.
• api/bots/ Bots distributed as part of the Zulip API bundle.
• api/bots_api/ Framework for running and testing bots in api/bots/.
Production puppet configuration
This is used to deploy essentially all configuration in production.
• puppet/zulip/ For configuration for production deployments.
• puppet/zulip/manifests/voyager.pp Main manifest for Zulip standalone deployments.
Additional Django apps
• confirmation Email confirmation system.
• analytics Analytics for the Zulip server administrator (needs work to be useful to normal Zulip sites).
• corporate The old Zulip.com website. Not included in production distribution.
• zilencer Primarily used to hold management commands that aren’t used in production. Not included in
production distribution.
Jinja2 Compatibility Files
• zproject/jinja2/__init__.py Jinja2 environment.
• zproject/jinja2/backends.py Jinja2 backend.
• zproject/jinja2/compressors.py Jinja2 compatible functions of Django-Pipeline.
3.7. API and Bots
15
Zulip Documentation, Release 1.4.0
Translation files
• locale/ Backend (Django) translations data files.
• static/locale/ Frontend translations data files.
Documentation
• docs/ Source for this documentation.
You can consult the repository’s .gitattributes file to see exactly which components are excluded from production releases (release tarballs are generated using tools/build-release-tarball).
16
Chapter 3. Directory structure
CHAPTER
4
Zulip Roadmap
Introduction
Zulip has received a great deal of interest and attention since it was released as free and open source software by
Dropbox. That attention has come with a lot of active development work from members of the Zulip community.
From when Zulip was released as open source in late September 2015 through today (early November, 2016), more
than 150 people have contributed over 1000 pull requests to the various Zulip repositories, the vast majority of which
were submitted by Zulip’s users around the world (as opposed to the small core team that reviews and merges the pull
requests).
In any project, there can be a lot of value in periodically putting together a roadmap detailing the major areas where the
project is hoping to improve. This can be especially important in an open source project like Zulip, where development
is distributed across many people around the world. This roadmap is intended to organize a list of the most important
improvements that should be made to Zulip in the relatively near future. Our aim is to complete most of these
improvements by February 2017 and then prepare a new roadmap then.
This document is not meant to constrain in any way what contributions to Zulip will be accepted; instead, it will
be used by the Zulip core team to prioritize our efforts, measure progress on improving the Zulip product and hold
ourselves accountable for making Zulip improve rapidly.
This roadmap is the best place for contributors to look for substantial projects that will definitely be of value to the
community (if you’re looking for a starter project, see the guide to getting involved with Zulip).
We periodically update this roadmap by adding strikethrough to issues that have been resolved, but the linked GitHub
issues are the most up-to-date source for that information.
Update: As of May 2017, we are approaching the point where we need to update the roadmap due to much of it being
completed.
Without further ado, below is the current Zulip roadmap.
Major projects
There are 2 huge projects that Zulip is working on right now that are too big to have a coherent GitHub issue:
17
Zulip Documentation, Release 1.4.0
• We are working with a world-class designer on a major visual redesign of the Zulip webapp. Already complete
is completely redesining the streams and settings UIs, logged-out pages, and various other major components.
• We are writing a new React Native iOS app for Zulip to replace the old iOS app. The new app is progressing
rapidly, but is not yet feature complete. We expect it to be in the app store in May 2017.
Core User Experience
• Provide shorter UI/Keyboard sequence to edit the last message
• Better drafts management
• Make clicking on desktop notifications renarrow properly
• Add pretty bubbles for recipients in the compose box
• Make right sidebar buddy list UI scale well to large teams
• Display stream descriptions more prominently
• Add support for managing uploaded files
Social features
• Add support for showing “user is typing” notifications, at least for private messages
• Support lightweight emoji “reactions”
• Open graph previews of generic websites
• Add a “join Zulip chat” badge for projects that use Zulip to document that nicely
Real-time sync
The overall goal is to eliminate the few known issues where Zulip does not provide a seamless real-time sync experience.
• Notification bot advertisements for new streams don’t handle stream renames
• Avatar/name changes don’t propagate to already-sent messages
• Advance the pointer / where we load the user to based on unread counts in home view
• Fix the known bug where messages could be incorrectly marked as read
Onboarding issues
This category focuses on issues users experience when installing a new Zulip server, setting up a new Zulip realm, or
starting to use Zulip.
• Move Zulip’s prompt for permission to display notifications to be manually triggered
• Add a mechanism for deleting early test messages (e.g., administrators can hard-delete messages)
• Allow customizing emails when inviting new users
18
Chapter 4. Zulip Roadmap
Zulip Documentation, Release 1.4.0
Production installation issues
• Document or better script solution to rabbitmq startup issues
• Merge a supported way to use Zulip in Docker in production implementation.
Administration and management
• Make list of allowed domains web-configurable
• Statistics display for realm and server administrators
• Keep track of which users added which realm emoji
• Add setting to enable any user to add new realm emoji
• Make realm filters web-configurable
• Improve administrative controls for managing streams
• Enhance the LDAP integration and make it web-configurable
• Add a SAML integration for Zulip
Scalability and performance
Scalability and performance are not currently major problems for Zulip; it already scales well to thousands of users
and is significantly faster than proprietary alternatives. So, this is not a major focus area for the project.
• Make the Zulip Tornado service support horizontal scaling
• Make presence system scale well to 10000 users in a realm.
• Support running queue workers multithreaded in production to decrease minimum memory footprint
• Improve @-mentioning syntax based on stronger unique identifiers
Technology improvements
• Add support for Zulip running purely on Python 3
• Automatic thumbnailing of uploaded images’ previews to save bandwidth
• Upgrade Zulip to use Django 1.10. The patches needed to run Zulip were merged into mainline Django in
Django 1.10, so this will mean we don’t need to use a fork of Django anymore.
• Upgrade and remove from codebase all unnecessarily vendored JS libraries
• Add support for changing users’ email addresses
• Migrate from jslint to eslint
• Replace the slow closure-compiler based static asset toolchain
• Use a modern JavaScript bundler like webpack
• Add support for building frontend features in something like React
4.7. Production installation issues
19
Zulip Documentation, Release 1.4.0
Technical Debt
While the Zulip server has a great codebase compared to most projects of its size, it takes work to keep it that way.
• Migrate most web routes to REST API
• Split Tornado subsystem into a separate Django app
• Refactor zulip.css to be broken into components
Security
• Add support for 2-factor authentication on all platforms
• Add support for stronger security controls for uploaded files (The LOCAL_UPLOADS_DIR file uploads backend only supports world-readable uploads)
• Fix requirement to set a password when creating account via Google
• Add a retention policy feature that automatically deletes old messages
• Add UI for viewing and cancelling open Zulip invitations
Testing
• Extend Zulip’s automated test coverage to include all API endpoints
• Build automated tests for the client API bindings
• Add automated tests for the production upgrade process
Documentation
• Add an in-app mechanism for updating users about new Zulip features
• Significantly expand documentation of the Zulip API and integrating with Zulip.
• Write a visual design / frontend style guide for Zulip
• Update all screenshots to show the current Zulip UI
Nice to have
• Expand library of documentation on Zulip’s feature set. Currently most documentation is for either developers
or system administrators.
Integrations and bots
Integrations are essential to Zulip. While we currently have a reasonably good framework for writing new webhook
integrations for getting notifications into Zulip, it’d be great to streamline that process and make bots that receive
messages just as easy to build.
20
Chapter 4. Zulip Roadmap
Zulip Documentation, Release 1.4.0
• Add an outgoing webhook integration system
• Make setting up a new integration a smooth flow
• Default new incoming webhooks to permissions-limited incoming webhook bots
• Change how Zulip displays bot names to distinguish them from human users
Android app
• Add support for narrowing to @-mentions
• Support having multiple Zulip realms open simultaneously
iOS app
For the new React Native iOS app, the major goal for it is to be released into the app store. Since it is moving quickly,
we’re tracking its roadmap via GitHub milestones.
Server/webapp support for mobile
To support a great mobile experiences, we need to make some improvements in the Zulip server.
• Push notifications bouncer service for GCM and APNS
• A slick process for doing mobile login without typing your password on your phone
• @here mention support (that doesn’t spam people not currently online, i.e. no email/push notifications)
• Fix sending messages from mobile web
Desktop apps
The new cross-platform desktop app is implemented in Electron, and primarily needs work on installer tooling to finish
replacing the old app.
• Finish releasing the Electron app to replace the old desktop app
• Support having multiple Zulip realms open simultaneously
Community
These don’t get GitHub issues since they’re not technical projects, but they are important goals for the project.
• Expand the number of core developers able to do code reviews
• Have a successful season with Zulip’s Outreachy participants
• Have a successful season with Google Code In.
4.16. Android app
21
Zulip Documentation, Release 1.4.0
22
Chapter 4. Zulip Roadmap
CHAPTER
5
Version History
All notable changes to the Zulip server are documented in this file.
Unreleased
1.6.0 – 2017-06-06
Highlights:
• A complete visual redesign of the logged-out pages, including login, registration, integrations, etc.
• New visual designs for numerous UI elements, including the emoji picker, user profile popovers, sidebars,
compose, and many more.
• A complete redesign of the Zulip settings interfaces to look a lot nicer and be easier to navigate.
• Organization admins can now configure the login and registration pages to show visitors a nice organization
profile with custom text and images, written in Markdown.
• Massively improved performance for presence and settings pages, especially for very large organizations (1000+
users).
• A dozen useful new keyboard shortcuts, from editing messages to emoji reactions to drafts and managing
streams.
• Typing notifications for private message threads.
• Users can now change their own email address.
• New saved-drafts feature.
• The server can now run on a machine with as little as 2GB of RAM.
• The new Electron desktop app and new React Native mobile app for iOS are now the recommended Zulip apps.
• Mobile web now works much better, especially on iOS.
23
Zulip Documentation, Release 1.4.0
• Support for sending mobile push notifications via a new forwarding service
• Complete translations for Spanish, German, and Czech (and expanded partial translations for Japanese, Chinese,
French, Hungarian, Polish, Dutch, Russian, Bulgarian, Portuguese, Serbian, Malayalam, Korean, and Italian).
Full feature Changelog:
• Added Basecamp, Gogs, Greenhouse, Home Assistant, Slack, Splunk, and WordPress webhook integrations.
• Added LaTeX support to the markdown processor.
• Added support for filtering branches to all Git integrations.
• Added read-only access to organization-level settings for all users.
• Added UI for managing muted topics and uploaded files.
• Added UI for displaying message edit history.
• Added support for various features needed by new mobile app.
• Added deep links for settings/subscriptions interfaces.
• Added an animation when messages are edited.
• Added support for registration with GitHub auth (not just login).
• Added tracking of uploaded file quotas.
• Added option to display emoji as their alt codes.
• Added new audit log table, to eventually support an auditing UI.
• Added several new permissions-related organization settings.
• Added new endpoint for fetching presence data, useful in employee directories.
• Added typeahead for language for syntax highlighting in code blocks.
• Added support for basic markdown in stream descriptions.
• Added email notifications on new Zulip logins.
• Added security hardening before serving uploaded files.
• Added new PRIVACY_POLICY setting to provide a Markdown privacy policy.
• Added an icon to distinguish bot users as message senders.
• Added a command-line Slack importer tool using the API.
• Added new announcement notifications on stream creation.
• Added support for some newer unicode emoji code points.
• Added support for users deleting realm emoji they themselves uploaded.
• Added support for organization administrators deleting messages.
• Extended data available to mobile apps to cover the entire API.
• Redesigned bots UI. Now can change owners and reactivate bots.
• Redesigned the visuals of code blocks to be prettier.
• Changed right sidebar presence UI to only show recently active users in large organizations. This has a huge
performance benefit.
• Changed color for private messages to look better.
• Converted realm emoji to be uploaded, not links, for better robustness.
24
Chapter 5. Version History
Zulip Documentation, Release 1.4.0
• Switched the default password hasher for new passwords to Argon2.
• Increased the paragraph spacing, making multi-paragraph.
• Improved formatting of all Git integrations.
• Improved the UI of the /stats analytics pages.
• Improved search typeahead to support group private messages.
• Improved logic for when the compose box should open/close.
• Improved lightbox to support scrolling through images.
• Improved markdown support for bulleted lists.
• Improved copy-to-clipboard support in various places.
• Improved subject lines of missed message emails.
• Improved handling of users trying to login with Oauth without an account.
• Improved UI of off-the-Internet errors to not be hidden in narrow windows.
• Improved rate-limiting errors to be more easily machine-readable.
• Parallelized the backend test suite; now runs 1600 tests in <30s.
• Fixed numerous bugs and performance issues with stream management.
• Fixed an issue with the fake emails assigned to bot users.
• Fixed a major performance issue in stream creation.
• Fixed numerous minor accessibility issues.
• Fixed a subtle interaction between click-to-reply and copy-paste.
• Fixed various formatting issues with /me messages.
• Fixed numerous real-time sync issues involving users changing their name, avatar, or email address and streams
being renamed.
• Fixed numerous performance issues across the project.
• Fixed various left sidebar ordering and live-updated bugs.
• Fixed numerous bugs with the message editing widget.
• Fixed missing logging / rate limiting on browser endpoints.
• Fixed regressions in Zulip’s browser state preservation on reload logic.
• Fixed support for unicode characters in the email mirror system.
• Fixed load spikes when email mirror is receiving a lot of traffic.
• Fixed the ugly grey flicker when scrolling fast on Macs.
• Fixed previews of GitHub image URLs.
• Fixed narrowing via clicking on desktop notifications.
• Fixed Subscribed/Unsubscribed bookends appearing incorrectly.
• Eliminated the idea of a realm having a canonical domain; now there’s simply the list of allowed domains for
new users.
• Migrated avatars to a user-id-based storage setup (not email-based).
• Trailing whitespace is now stripped in code blocks, avoiding unnecessary scrollbars.
5.2. 1.6.0 – 2017-06-06
25
Zulip Documentation, Release 1.4.0
• Most API payloads now refer to users primarily by user ID, with email available for backwards-compatibility.
In the future, we may remove email support.
• Cleaned up Zulip’s supervisord configuration. A side effect is the names of the log files have changed for all the
queue workers.
• Refactored various endpoints to use a single code path for security hardening.
• Removed support for the MANDRILL_CLIENT setting. It hadn’t been used in years.
• Changed NOREPLY_EMAIL_ADDRESS setting to Name <user@example.com> format.
• Disabled the web tutorial on mobile.
• Backend test coverage is now 93%, with 100% in views code.
1.5.2 – 2017-06-01
• CVE-2017-0896: Restricting inviting new users to admins was broken.
• CVE-2015-8861: Insecure old version of handlebars templating engine.
1.5.1 – 2017-02-07
• Fix exception trying to copy node_modules during upgrade process.
• Improved styling of /stats page to remove useless login/register links.
1.5.0 – 2017-02-06
Highlights:
• Completely redesigned the Manage streams interface.
• Added support for emoji reactions to messages.
• Added a lightbox for viewing images and videos.
• Added an extensive user documentation site at /help/.
• Added admin setting to auto-linkify certain strings (useful for issue numbers and Git commit IDs).
• Upgraded how the main application runs from FastCGI on Django 1.8 to uwsgi and Django 1.10.
• Added preliminary support for open graph previews of links (the setting, INLINE_URL_EMBED_PREVIEW,
is disabled by default in this release).
Full feature Changelog:
• Added an emoji picker/browser to the compose box.
• Added markdown preview support to the compose box.
• Added a new analytics system to track interesting usage statistics.
• Added a /stats page with graphs of the analytics data.
• Added display of subscriber counts in Manage streams.
• Added support for filtering streams in Manage streams.
26
Chapter 5. Version History
Zulip Documentation, Release 1.4.0
• Added support for setting a stream description on creation.
• Added support for copying subscribers from existing streams on creation.
• Added several new search/filtering UI elements.
• Added UI for deactivating your own Zulip account.
• Added support for viewing the raw markdown content of a message.
• Added support for deploying Zulip with subdomains for each realm. This entailed numerous changes to ensure
a consistent experience.
• Added support for (optionally) using PGRoonga to support full-text search in all languages (not just English).
• Added AppFollow, GitLab, Google Calendar, GoSquared, HelloSign, Heroku, Librato, MailChimp, Mention,
Papertrail, Sentry, Solano Labs, Stripe and Zapier integrations.
• Added a webhook integration for GitHub, replacing the deprecated github-services hook.
• Normalized the message formatting for all the Zulip Git integrations.
• Added support for VMWare Fusion Vagrant provider for faster OSX development.
• Added a shields.io style badge for joining a Zulip server.
• Added admin setting for which email domains can join a realm.
• Added admin setting for controlling who can create streams.
• Added admin setting to limit stream creation to older users.
• Added a notification when you muted a topic.
• Added a new hotkey for muting/unmuting topics.
• Added support for testing websockets to the Nagios plugins.
• Added a configuration option to disable websockets.
• Added support for removing one’s own Zulip account.
• Added support for realm admins which auth backends are supported.
• Added new organization type concept. This will be used to control whether Zulip is optimized around protecting
user privacy vs. administrative control.
• Added #streamName syntax for linking to a stream.
• Added support for viewing markdown source of messages.
• Added setting to always send push notifications.
• Added setting to hide private message content in desktop notifications.
• Added buttons to download .zuliprc files.
• Added italics and strikethrough support in markdown implementation.
• Added errors for common installations mistakes (e.g. too little RAM).
• Added a new /authors page showing the contributors to the current Zulip version.
• Upgraded all Python dependencies to modern versions, including Django 1.10 (all of Zulip’s patches have been
merged into mainline).
• Increased backend test coverage of Python codebase to 90%.
• Increased mypy static type coverage of Python code to 100%.
5.5. 1.5.0 – 2017-02-06
27
Zulip Documentation, Release 1.4.0
• Added several new linters (eslint, pep8) and cleaned the codebase.
• Optimized the speed of the Zulip upgrade process, especially with Git.
• Have peer_add events send user_id, not email.
• Fixed problems with rabbitmq when installing Zulip.
• Fixed JavaScript not being gzip-compressed properly.
• Fixed a major performance bug in the Tornado service.
• Fixed a frontend performance bug creating streams in very large realms.
• Fixed numerous bugs where strings were not properly tagged for translation.
• Fixed several real-time sync bugs, and removed several AJAX calls. Zulip should be more performant than ever
before.
• Fixed Zulip Tornado service not working with http_proxy set in environment.
• Fixed text overflow in stream subscriptions.
• Fixed CSS issues with message topic editing.
• Fixed several transactionality bugs (e.g. in Huddle creation).
• Fixed missed-message email configuration error handling.
• Fixed annoying @-mentions in Jira integration.
• Fixed various mismatches between frontend and backend markdown implementations.
• Fixed various popover-related UI bugs.
• Fixed duplicate notifications with multiple open Zulip tabs.
• Fixed support for emailing the server administrator about backend exceptions.
• Cleaned up the “edit message” form.
• Eliminated most of the legacy API endpoints.
• Improved typeahead and autocomplete across the application. Highlights include much better handling of many
users with similar names.
• Improved the color scheme for code blocks.
• Improved the message editing UI in several ways.
• Improved how dates are displayed in the UI.
• Improved default settings for zxcvbn password strength checker.
• Upgraded jQuery to the latest 1.12 release.
• Made numerous improvements to the development tooling.
• Made extensive improvements to code organization.
• Restyled all the registration pages to look nicer and be responsive.
• Extensively refactored views to use common functions for fetching stream and message objects.
• Suppressed @-all mentions being treated as mentions on muted streams.
• Documented preliminary design for interactive bot system.
28
Chapter 5. Version History
Zulip Documentation, Release 1.4.0
1.4.3 - 2017-01-29
• CVE-2017-0881: Users could subscribe to invite-only streams.
1.4.2 - 2016-09-27
• Upgraded Django to version 1.8.15 (with the Zulip patches applied), fixing a CSRF vulnerability in Django (see
https://www.djangoproject.com/weblog/2016/sep/26/security-releases/), and a number of other Django bugs
from past Django stable releases that largely affects parts of Django that are not used by Zulip.
• Fixed buggy logrotate configuration.
1.4.1 - 2016-09-03
• Fixed settings bug upgrading from pre-1.4.0 releases to 1.4.0.
• Fixed local file uploads integration being broken for new 1.4.0 installations.
1.4.0 - 2016-08-25
• Migrated Zulip’s python dependencies to be installed via a virtualenv, instead of the via apt. This is a major
change to how Zulip is installed that we expect will simplify upgrades in the future.
• Fixed unnecessary loading of zxcvbn password strength checker. This saves a huge fraction of the uncached
network transfer for loading Zulip.
• Added support for using Ubuntu Xenial in production.
• Added a powerful and complete realm import/export tool.
• Added nice UI for selecting a default language to display settings.
• Added UI for searching streams in left sidebar with hotkeys.
• Added Semaphore, Bitbucket, and HelloWorld (example) integrations.
• Added new webhook-based integration for Trello.
• Added management command for creating realms through web UI.
• Added management command to send password reset emails.
• Added endpoint for mobile apps to query available auth backends.
• Added LetsEncrypt documentation for getting SSL certificates.
• Added nice rendering of unicode emoji.
• Added support for pinning streams to the top of the left sidebar.
• Added search box for filtering user list when creating a new stream.
• Added realm setting to disable message editing.
• Added realm setting to time-limit message editing. Default is 10m.
• Added realm setting for default language.
5.6. 1.4.3 - 2017-01-29
29
Zulip Documentation, Release 1.4.0
• Added year to timestamps in message interstitials for old messages.
• Added GitHub authentication (and integrated python-social-auth, so it’s easy to add additional social authentication methods).
• Added TERMS_OF_SERVICE setting using markdown formatting to configure the terms of service for a Zulip
server.
• Added numerous hooks to puppet modules to enable more configurations.
• Moved several useful puppet components into the main puppet manifests (setting a redis password, etc.).
• Added automatic configuration of postgres/memcached settings based on the server’s available RAM.
• Added scripts/upgrade-zulip-from-git for upgrading Zulip from a Git repo.
• Added preliminary support for Python 3. All of Zulip’s test suites now pass using Python 3.4.
• Added support for Name <email@example.com> format when inviting users.
• Added numerous special-purpose settings options.
• Added a hex input field in color picker.
• Documented new Electron beta app and mobile apps in the /apps/ page.
• Enabled Android Google authentication support.
• Enhanced logic for tracking origin of user uploads.
• Improved error messages for various empty narrows.
• Improved missed message emails to better support directly replying.
• Increased backend test coverage of Python code to 85.5%.
• Increased mypy static type coverage of Python code to 95%. Also fixed many string annotations to properly
handle unicode.
• Fixed major i18n-related frontend performance regression on /#subscriptions page. Saves several seconds of
load time with 1k streams.
• Fixed Jinja2 migration bug when trying to register an email that already has an account.
• Fixed narrowing to a stream from other pages.
• Fixed various frontend strings that weren’t marked for translation.
• Fixed several bugs around editing status (/me) messages.
• Fixed queue workers not restarting after changes in development.
• Fixed Casper tests hanging while development server is running.
• Fixed browser autocomplete issue when adding new stream members.
• Fixed broken create_stream and rename_stream management commands.
• Fixed zulip-puppet-apply exit code when puppet throws errors.
• Fixed EPMD restart being attempted on every puppet apply.
• Fixed message cache filling; should improve perf after server restart.
• Fixed caching race condition when changing user objects.
• Fixed buggy puppet configuration for supervisord restarts.
• Fixed some error handling race conditions when editing messages.
30
Chapter 5. Version History
Zulip Documentation, Release 1.4.0
• Fixed fastcgi_params to protect against the httpoxy attack.
• Fixed bug preventing users with mit.edu emails from registering accounts.
• Fixed incorrect settings docs for the email mirror.
• Fixed APNS push notification support (had been broken by Apple changing the APNS API).
• Fixed some logic bugs in how attachments are tracked.
• Fixed unnecessarily resource-intensive rabbitmq cron checks.
• Fixed old deployment directories leaking indefinitely.
• Fixed need to manually add localhost in ALLOWED_HOSTS.
• Fixed display positioning for the color picker on subscriptions page.
• Fixed escaping of Zulip extensions to markdown.
• Fixed requiring a reload to see newly uploaded avatars.
• Fixed @all warning firing even for @all.
• Restyled password reset form to look nice.
• Improved formatting in reset password links.
• Improved alert words UI to match style of other settings.
• Improved error experience when sending to nonexistent users.
• Portions of integrations documentation are now automatically generated.
• Restructured the URLs files to be more readable.
• Upgraded almost all Python dependencies to current versions.
• Substantially expanded and reorganized developer documentation.
• Reorganized production documentation and moved to ReadTheDocs.
• Reorganized .gitignore type files to be written under var/
• Refactored substantial portions of templates to support subdomains.
• Renamed local_settings.py symlink to prod_settings.py for clarity.
• Renamed email-mirror management command to email_mirror.
• Changed HTTP verb for create_user_backend to PUT.
• Eliminated all remaining settings hardcoded for zulip.com.
• Eliminated essentially all remaining hardcoding of mit.edu.
• Optimized the performance of all the test suites.
• Optimized Django memcached configuration.
• Removed old prototype data export tool.
• Disabled insecure RC4 cipher in nginx configuration.
• Enabled shared SSL session cache in nginx configuration.
• Updated header for Zulip static assets to reflect Zulip being open source.
5.9. 1.4.0 - 2016-08-25
31
Zulip Documentation, Release 1.4.0
1.3.13 - 2016-06-21
• Added nearly complete internationalization of the Zulip UI.
• Added warning when using @all/@everyone.
• Added button offering to subscribe at bottom of narrows to streams the user is not subscribed to.
• Added integrations with Airbrake, CircleCI, Crashlytics, IFTTT, Transifex, and Updown.io.
• Added menu option to mark all messages in a stream or topic as read.
• Added new Attachment model to keep track of uploaded files.
• Added caching of virtualenvs in development.
• Added mypy static type annotations to about 85% of the Zulip Python codebase.
• Added automated test of backend templates to test for regressions.
• Added lots of detailed documentation on the Zulip development environment.
• Added setting allowing only administrators to create new streams.
• Added button to exit the Zulip tutorial early.
• Added web UI for configuring default streams.
• Added new OPEN_REALM_CREATION setting (default off), providing a UI for creating additional realms on
a Zulip server.
• Fixed email_gateway_password secret not working properly.
• Fixed missing helper scripts for RabbitMQ Nagios plugins.
• Fixed skipping forward to latest messages (“More messages below” button).
• Fixed netcat issue causing Zulip installation to hang on Scaleway machines.
• Fixed rendering of /me status messages after message editing.
• Fixed case sensitivity of right sidebar fading when compose is open.
• Fixed error messages when composing to invalid PM recipients.
• Fixed LDAP auth backend not working with Zulip mobile apps.
• Fixed erroneous WWW-Authenticate headers with expired sessions.
• Changed “coworkers” to “users” in the Zulip UI.
• Changed add_default_stream REST API to correctly use PUT rather than PATCH.
• Updated the Zulip emoji set (the Android Emoji) to a modern version.
• Made numerous small improvements to the Zulip development experience.
• Migrated backend templates to the faster Jinja2 templating system.
• Migrated development environment setup scripts to tools/setup/.
• Expanded test coverage for several areas of the product.
• Simplified the API for writing new webhook integrations.
• Removed most of the remaining JavaScript global variables.
32
Chapter 5. Version History
Zulip Documentation, Release 1.4.0
1.3.12 - 2016-05-10
• CVE-2016-4426: Bot API keys were accessible to other users in the same realm.
• CVE-2016-4427: Deactivated users could access messages if SSO was enabled.
• Fixed a RabbitMQ configuration bug that resulted in reordered messages.
• Added expansive test suite for authentication backends and decorators.
• Added an option to logout_all_users to delete only sessions for deactivated users.
1.3.11 - 2016-05-02
• Moved email digest support into the default Zulip production configuration.
• Added options for configuring Postgres, RabbitMQ, Redis, and memcached in settings.py.
• Added documentation on using Hubot to integrate with useful services not yet integrated with Zulip directly
(e.g. Google Hangouts).
• Added new management command to test sending email from Zulip.
• Added Codeship, Pingdom, Taiga, Teamcity, and Yo integrations.
• Added Nagios plugins to the main distribution.
• Added ability for realm administrators to manage custom emoji.
• Added guide to writing new integrations.
• Enabled camo image proxy to fix mixed-content warnings for http images.
• Refactored the Zulip puppet modules to be more modular.
• Refactored the Tornado event system, fixing old memory leaks.
• Removed many old-style /json API endpoints
• Implemented running queue processors multithreaded in development, decreasing RAM requirements for a
Zulip development environment from ~1GB to ~300MB.
• Fixed rerendering the complete buddy list whenever a user came back from idle, which was a significant performance issue in larger realms.
• Fixed the disabling of desktop notifications from 1.3.7 for new users.
• Fixed the (admin) create_user API enforcing restricted_to_domain, even if that setting was disabled for the
realm.
• Fixed bugs changing certain settings in administration pages.
• Fixed collapsing messages in narrowed views.
• Fixed 500 errors when uploading a non-image file as an avatar.
• Fixed Jira integration incorrectly not @-mentioning assignee.
5.11. 1.3.12 - 2016-05-10
33
Zulip Documentation, Release 1.4.0
1.3.10 - 2016-01-21
• Added new integration for Travis CI.
• Added settings option to control maximum file upload size.
• Added support for running Zulip development environment in Docker.
• Added easy configuration support for a remote postgres database.
• Added extensive documentation on scalability, backups, and security.
• Recent private message threads are now displayed expanded similar to the pre-existing recent topics feature.
• Made it possible to set LDAP and EMAIL_HOST passwords in /etc/zulip/secrets.conf.
• Improved the styling for the Administration page and added tabs.
• Substantially improved loading performance on slow networks by enabling GZIP compression on more assets.
• Changed the page title in narrowed views to include the current narrow.
• Fixed several backend performance issues affecting very large realms.
• Fixed bugs where draft compose content might be lost when reloading site.
• Fixed support for disabling the “zulip” notifications stream.
• Fixed missing step in postfix_localmail installation instructions.
• Fixed several bugs/inconveniences in the production upgrade process.
• Fixed realm restrictions for servers with a unique, open realm.
• Substantially cleaned up console logging from run-dev.py.
1.3.9 - 2015-11-16
• Fixed buggy #! lines in upgrade scripts.
1.3.8 - 2015-11-15
• Added options to the Python api for working with untrusted server certificates.
• Added a lot of documentation on the development environment and testing.
• Added partial support for translating the Zulip UI.
• Migrated installing Node dependencies to use npm.
• Fixed LDAP integration breaking autocomplete of @-mentions.
• Fixed admin panel reactivation/deactivation of bots.
• Fixed inaccurate documentation for downloading the desktop apps.
• Fixed various minor bugs in production installation process.
• Fixed security issue where recent history on private streams might be visible to new users (to the Zulip team) who
were invited with that private stream as one of their initial streams (https://github.com/zulip/zulip/issues/230).
• Major preliminary progress towards supporting Python 3.
34
Chapter 5. Version History
Zulip Documentation, Release 1.4.0
1.3.7 - 2015-10-19
• Turn off desktop and audible notifications for streams by default.
• Added support for the LDAP authentication integration creating new users.
• Added new endpoint to support Google auth on mobile.
• Fixed desktop notifications in modern Firefox.
• Fixed several installation issues for both production and development environments.
• Improved documentation for outgoing SMTP and the email mirror integration.
5.16. 1.3.7 - 2015-10-19
35
Zulip Documentation, Release 1.4.0
36
Chapter 5. Version History
CHAPTER
6
Requirements
Note that if you just want to play around with Zulip and see what it looks like, it is easier to install it in a development
environment following these instructions, since then you don’t need to worry about setting up SSL certificates and an
authentication mechanism. Or, you can check out the Zulip development community server.
Server
Hardware Specifications
• CPU and Memory: For installations with 100+ users you’ll need a minimum of 2 CPUs and 4GB RAM. For
installations with fewer users, 1 CPU and 2GB RAM is sufficient. We strongly recommend against installing
with less than 2GB of RAM, as you will likely experience out of memory issues installing dependencies. We
recommend against using highly CPU-limited servers like the AWS t2 style instances for organizations with a
hundreds of users (active or no).
See our documentation on scalability for advice on hardware requirements for larger organizations.
• Disk space: You’ll need at least 10GB of free disk space. If you intend to store uploaded files locally rather than
on S3 you will likely need more.
Network and Security Specifications
• Incoming HTTPS access (usually port 443, though this is configurable) from the networks where your users
are (usually, the public Internet). If you also open port 80, Zulip will redirect users to HTTPS rather than not
working when users type e.g. http://zulip.example.com in their browser. If you are using Zulip’s
incoming email integration you may also need incoming port 25 open.
• Outgoing HTTP(S) access (ports 80 and 443) to the public Internet so that Zulip can properly manage inline
image previews. You’ll also need outgoing SMTP access to your SMTP server (the standard port for this is 587)
so that Zulip can send email.
37
Zulip Documentation, Release 1.4.0
Operating System
Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial are supported for running Zulip in production. 64-bit is recommended.
Domain name
You should already have a domain name available for your Zulip production instance. In order to generate valid SSL
certificates with Let’s Encrypt, and to enable other services such as Google Authentication, you’ll need to update the
domains A record to point to your production server.
Credentials needed
SSL Certificate
• SSL Certificate for the host you’re putting this on (e.g. zulip.example.com). The installation instructions contain
documentation for how to get an SSL certificate for free using LetsEncrypt.
Outgoing email
• Outgoing email (SMTP) credentials that Zulip can use to send outgoing emails to users (e.g. email address
confirmation emails during the signup process, missed message notifications, password reset, etc.). If you don’t
have an existing outgoing SMTP solution, read about free outgoing SMTP options and options for prototyping.
Once you have met these requirements, see full instructions for installing Zulip in production.
38
Chapter 6. Requirements
CHAPTER
7
Production Installation
Make sure you want to install a Zulip production server; if you’d instead like to test or develop a new feature, we
recommend the Zulip server development environment instead.
You will need an Ubuntu system that satisfies the installation requirements. In short, you need:
• A Ubuntu 14.04 Trusty or Ubuntu 16.04 Xenial 64-bit server
• At least 2GB RAM and 10 GB disk space (4GB and 2 CPUs recommended for 100+ users).
• A DNS name, an SSL certificate, and credentials for sending email.
Step 0: Subscribe
Please subscribe to low-traffic the Zulip announcements Google Group to get announcements about new releases,
security issues, etc.
Step 1: Install SSL Certificates
Zulip runs over https only and requires ssl certificates in order to work. It looks for the certificates in /etc/
ssl/private/zulip.key and /etc/ssl/certs/zulip.combined-chain.crt. Note that Zulip uses
nginx as its webserver and thus expects a chained certificate bundle
If you need an SSL certificate, see our SSL certificate documentation. If you already have an SSL certificate, just
install (or symlink) them into place at the above paths, and move on to the next step.
Step 2: Download and install latest release
If you haven’t already, download and unpack the latest built server tarball with the following commands:
39
Zulip Documentation, Release 1.4.0
sudo -i # If not already root
cd /root
wget https://www.zulip.org/dist/releases/zulip-server-latest.tar.gz
rm -rf /root/zulip && mkdir /root/zulip
tar -xf zulip-server-latest.tar.gz --directory=/root/zulip --strip-components=1
Then, run the Zulip install script:
/root/zulip/scripts/setup/install
This may take a while to run, since it will install a large number of dependencies. It also creates zulip user, which
will be used to run the various Zulip servers.
The Zulip install script is designed to be idempotent, so if it fails, you can just rerun it after correcting the issue that
caused it to fail. Also note that it automatically logs a transcript to /var/log/zulip/install.log; please
include a copy of that file in any bug reports.
Step 3: Configure Zulip
Configure the Zulip server instance by editing /etc/zulip/settings.py and providing values for the mandatory settings, which are all found under the heading ### MANDATORY SETTINGS.
These settings include:
• EXTERNAL_HOST: the user-accessible Zulip domain name for your Zulip installation (aka what users will
type in their web browser). This should of course match the DNS name you configured to point to your
server and for which you configured SSL certificates. If you plan to use multiple domains, add the others
to ALLOWED_HOSTS.
• ZULIP_ADMINISTRATOR: the email address of the person or team maintaining this installation and who will
get support and error emails.
• EMAIL_*, DEFAULT_FROM_EMAIL, and NOREPLY_EMAIL_ADDRESS: credentials for an outgoing
SMTP server so Zulip can send emails when needed (don’t forget to set email_password in the
zulip-secrets.conf file!). We highly recommend reading our production email docs and following the
test procedure discussed there to make sure you’ve setup outgoing email correctly, since outgoing email is the
most common configuration problem.
• AUTHENTICATION_BACKENDS: a list of enabled authentication mechanisms. You’ll need to enable at least
one authentication mechanism by uncommenting its corresponding line, and then also do any additional configuration required for that backend as documented in the settings.py file (the email backend requires no
extra configuration). See the section on Authentication for more detail on the available authentication backends
and how to configure them.
Step 4: Initialize Zulip database
At this point, you are done doing things as root. The remaining commands are run as the zulip user using su
zulip. To initialize the Zulip database for your production install, run:
su zulip -c /home/zulip/deployments/current/scripts/setup/initialize-database
The initialize-database script will report an error if you did not fill in all the mandatory settings from /etc/
zulip/settings.py. It is safe to rerun it after correcting the problem if that happens.
40
Chapter 7. Production Installation
Zulip Documentation, Release 1.4.0
This completes the process of installing Zulip on your server. However, in order to use Zulip, you’ll need to create an
organization in your Zulip installation.
Step 5: Create a Zulip organization and login
• If you haven’t already, verify that your outgoing email configuration works. The organization creation process
will fail if outgoing email is not configured properly.
• Run the organization (realm) creation management command :
su zulip # If you weren't already the zulip user
/home/zulip/deployments/current/manage.py generate_realm_creation_link
This will print out a secure 1-time use link that allows creation of a new Zulip organization on
your server.
For most servers, you will only ever do this once, but you can run manage.py
generate_realm_creation_link again if you want to host another organization on your Zulip server.
• Open the link generated with your web browser. You’ll see the create organization page (screenshot here). Enter
your email address and click Create organization.
• Check your email to find the confirmation email and click the link. You’ll be prompted to finish setting up your
organization and initial administrator user (screenshot here). Complete this form and log in!
Congratulations! You are logged in as an organization administrator for your new Zulip organization. After getting
oriented, we recommend visiting the “Organization settings” UI (linked from the upper-right gear menu in the Zulip
webapp) to configure important policy settings like how users can join your new organization. By default, your
organization will be configured as follows depending on what type of organization you selected:
Community Organization:
• restricted_to_domain=False: No restriction on user email addresses.
• invite_required=True: A user must be invited to join.
Corporate Organization:
• restricted_to_domain=True: New users must have an email address in the same domain (e.g.
@acme.com) as yours.
• invite_required=False: No invitation is required to join.
Next, you’ll likely want to do one of the following:
• Customize your Zulip organization.
• Learn about managing a production Zulip server.
Troubleshooting
If you get an error after scripts/setup/install completes, check /var/log/zulip/errors.log for a
traceback, and consult the troubleshooting section for advice on how to debug. If that doesn’t help, please visit #production help in the Zulip developerment community server for realtime help or email zulip-help@googlegroups.com
with the full traceback and we’ll try to help you out!
7.6. Step 5: Create a Zulip organization and login
41
Zulip Documentation, Release 1.4.0
42
Chapter 7. Production Installation
CHAPTER
8
Troubleshooting
Zulip uses Supervisor to monitor and control its many Python services. Read the next section, Using supervisorctl, to
learn how to use the Supervisor client to monitor and manage services.
If you haven’t already, now might be a good time to read Zulip’s architectural overview, particularly the Components
section. This will help you understand the many services Zulip uses.
If you encounter issues while running Zulip, take a look at Zulip’s logs, which are located in /var/log/zulip/.
That directory contains one log file for each service, plus errors.log (has all errors), server.log (has logs
from the Django and Tornado servers), and workers.log (has combined logs from the queue workers).
The section troubleshooting services on this page includes details about how to fix common issues with Zulip services.
If you run into additional problems, please report them so that we can update this page! The Zulip installation scripts
logs its full output to /var/log/zulip/install.log, so please include the context for any tracebacks from
that log.
Using supervisorctl
To see what Zulip-related services are configured to use Supervisor, look at /etc/supervisor/conf.d/
zulip.conf and /etc/supervisor/conf.d/zulip-db.conf.
Use the supervisor client supervisorctl to list the status of, stop, start, and restart various services.
Checking status with supervisorctl status
You can check if the zulip application is running using:
supervisorctl status
When everything is running as expected, you will see something like this:
43
Zulip Documentation, Release 1.4.0
process-fts-updates
˓→uptime 1:13:11
zulip-django
˓→uptime 1:13:11
zulip-senders:zulip-events-message_sender-0
˓→uptime 1:13:11
zulip-senders:zulip-events-message_sender-1
˓→uptime 1:13:11
zulip-senders:zulip-events-message_sender-2
˓→uptime 1:13:11
zulip-senders:zulip-events-message_sender-3
˓→uptime 1:13:11
zulip-senders:zulip-events-message_sender-4
˓→uptime 1:13:11
zulip-tornado
˓→uptime 1:13:11
zulip-workers:zulip-deliver-enqueued-emails
zulip-workers:zulip-events-confirmation-emails
˓→uptime 1:13:11
zulip-workers:zulip-events-digest_emails
˓→uptime 1:13:11
zulip-workers:zulip-events-email_mirror
˓→uptime 1:13:11
zulip-workers:zulip-events-error_reports
˓→uptime 1:13:11
zulip-workers:zulip-events-feedback_messages
˓→uptime 1:13:11
zulip-workers:zulip-events-missedmessage_mobile_notifications
˓→uptime 1:13:11
zulip-workers:zulip-events-missedmessage_reminders
˓→uptime 1:13:11
zulip-workers:zulip-events-signups
˓→uptime 1:13:11
zulip-workers:zulip-events-slowqueries
˓→uptime 1:13:11
zulip-workers:zulip-events-user-activity
˓→uptime 1:13:11
zulip-workers:zulip-events-user-activity-interval
˓→uptime 1:13:11
zulip-workers:zulip-events-user-presence
˓→uptime 1:13:11
RUNNING
pid 2194,
RUNNING
pid 2192,
RUNNING
pid 2209,
RUNNING
pid 2210,
RUNNING
pid 2211,
RUNNING
pid 2212,
RUNNING
pid 2208,
RUNNING
pid 2193,
STARTING
RUNNING
pid 2199,
RUNNING
pid 2205,
RUNNING
pid 2203,
RUNNING
pid 2200,
RUNNING
pid 2207,
RUNNING
pid 2204,
RUNNING
pid 2206,
RUNNING
pid 2198,
RUNNING
pid 2202,
RUNNING
pid 2197,
RUNNING
pid 2196,
RUNNING
pid 2195,
Restarting services with supervisorctl restart all
After you change configuration in /etc/zulip/settings.py or fix a misconfiguration, you will often want to
restart the Zulip application. You can restart Zulip using:
supervisorctl restart all
Stopping services with supervisorctl stop all
Similarly, you can stop Zulip using:
44
Chapter 8. Troubleshooting
Zulip Documentation, Release 1.4.0
supervisorctl stop all
Troubleshooting services
The Zulip application uses several major open source services to store and cache data, queue messages, and otherwise
support the Zulip application:
• postgresql
• rabbitmq-server
• nginx
• redis
• memcached
If one of these services is not installed or functioning correctly, Zulip will not work. Below we detail some common
configuration problems and how to resolve them:
• An AMQPConnectionError traceback or error running rabbitmqctl usually means that RabbitMQ is not running;
to fix this, try:
service rabbitmq-server restart
If RabbitMQ fails to start, the problem is often that you are using a virtual machine with broken DNS configuration; you can often correct this by configuring /etc/hosts properly.
• If your browser reports no webserver is running, that is likely because nginx is not configured properly and thus
failed to start. nginx will fail to start if you configured SSL incorrectly or did not provide SSL certificates. To
fix this, configure them properly and then run:
service nginx restart
• If your host is being port scanned by unauthorized users, you may see messages in /var/log/zulip/
server.log like
2017-02-22 14:11:33,537 ERROR Invalid HTTP_HOST header: '10.2.3.4'. You may need
˓→to add u'10.2.3.4' to ALLOWED_HOSTS.
Django uses the hostnames configured in ALLOWED_HOSTS to identify legitimate requests and block others.
When an incoming request does not have the correct HTTP Host header, Django rejects it and logs the attempt.
For more on this issue, see the Django release notes on Host header poisoning
Next: Making your Zulip instance awesome.
8.2. Troubleshooting services
45
Zulip Documentation, Release 1.4.0
46
Chapter 8. Troubleshooting
CHAPTER
9
Customize Zulip
Once you’ve got Zulip setup, you’ll likely want to configure it the way you like. Most configuration can be done by a
realm administrator on the web (see the documentation for realm administrators; this page discusses those additional
configuration items that can be done by a system administrator.
Mobile and desktop apps
The Zulip apps expect to be talking to to servers with a properly signed SSL certificate, in most cases and will not
accept a self-signed certificate. You should get a proper SSL certificate before testing the apps.
Because of how Google and Apple have architected the security model of their push notification protocols, the Zulip
mobile apps for iOS and Android can only receive push notifications from a single Zulip server. We have configured
that server to be push.zulipchat.com, and offer a push notification forwarding service that forwards push notifications through our servers to mobile devices. Read the linked documentation for instructions on how to register for
and configure this service.
By the end of summer 2017, all of the Zulip apps will have full support for multiple accounts, potentially on different
Zulip servers, with a convenient UI for switching between them.
Terms of service and Privacy policy
Zulip allows you to configure your server’s Terms of Service and Privacy Policy pages (/terms and /privacy,
respectively). You can use the TERMS_OF_SERVICE and PRIVACY_POLICY settings to configure the path to your
server’s policies. The syntax is Markdown (with support for included HTML). A good approach is to use paths like
/etc/zulip/terms.md, so that it’s easy to back up your policy configuration along with your other Zulip server
configuration.
47
Zulip Documentation, Release 1.4.0
Miscellaneous server settings
Zulip has dozens of settings documented in the comments in /etc/zulip/settings.py; you can review the
latest version of the settings.py template if you’ve deleted the comments or want to check if new settings have been
added in more recent versions of Zulip.
Since Zulip’s settings file is a Python script, there are a number of other things that one can configure that are not
documented; ask on chat.zulip.org if there’s something you’d like to do but can’t figure out how to.
Zulip announcement list
If you haven’t already, subscribe to the zulip-announce list so that you can receive important announces like new Zulip
releases or major changes to the app ecosystem..
Enjoy your Zulip installation!
If you discover things that you wish had been documented, please contribute documentation suggestions either via a
GitHub issue or pull request; we love even small contributions, and we’d love to make the Zulip documentation cover
everything anyone might want to know about running Zulip in production.
Next: Maintaining and upgrading Zulip in production.
48
Chapter 9. Customize Zulip
CHAPTER
10
Mobile push notification service
Zulip’s iOS and Android mobile apps support receiving push notifications from Zulip servers to let users know when
new messages have arrived. This is an important feature to having a great experience using the Zulip mobile apps.
For technical reasons (explained below), in order to deliver mobile push notifications in the app store versions of our
mobile apps, you will need to register your Zulip server with the Zulip mobile push notification service. This service
will forward push notifications generated by your server to the Zulip mobile app automatically.
How to sign up
Starting with Zulip 1.6, Zulip servers support forwarding push notifications to a central push notification forwarding
service. You can enable this for your Zulip server as follows:
1. First, contact support@zulipchat.com for instructions on registering your server (we hope to have a nice web
flow available for this soon).
2. Then, uncomment the PUSH_NOTIFICATION_BOUNCER_URL = https://push.zulipchat.com
line in your /etc/zulip/settings.py file, and restart your Zulip server.
That should be all you need to do!
To test whether you’ve configured it properly, just configure mobile push notifications to always be sent, start the Zulip
Android app, in the background, and then have another user send you private messages. They should appear in the
Android notification area.
Why this is necessary
Both Google’s and Apple’s push notification services have a security model that does not support mutually untrusted
self-hosted servers sending push notifications to the same app. In particular, when an app is published to their respective app stores, one must compile into the app a secret corresponding to the server that will be able to publish push
notifications for the app. This means that it is impossible for a single app in their stores to receive push notifications
from multiple, mutually untrusted, servers.
49
Zulip Documentation, Release 1.4.0
Zulip’s solution to this problem is to provide a central push notification forwarding service, which allows registered
Zulip servers to send push notifications to the Zulip app indirectly (through the forwarding service).
Security and privacy implications
We’ve designed this push notification bouncer service with security in mind:
• All of the network requests (both from Zulip servers to the Push Notification Service and from the Push Notification Service to the relevant Google and Apple services) are encrypted over the wire with SSL/TLS.
• The code for the push notification forwarding service is 100% open source and available as part of the Zulip
server project on GitHub. The service’s logging is designed to protect the privacy of users of Zulip servers that
are using the forwarding service.
• The push notification forwarding servers are professionally managed by a small team.
If you have any questions about the security model, contact support@zulipchat.com.
50
Chapter 10. Mobile push notification service
CHAPTER
11
Secure, maintain, and upgrade
This page covers topics that will help you maintain a healthy, up-to-date, and secure Zulip installation, including:
• Upgrading
• Upgrading from a git repository
• Backups
• Monitoring
• Scalability
• Management commands
You may also want to read this related content:
• Security Model
Upgrading
We recommend reading this entire section before doing your first upgrade.
To upgrade to a new version of the zulip server, download the appropriate release tarball from https://www.zulip.org/
dist/releases/.
You also have the option of creating your own release tarballs from a copy of the zulip.git repository using tools/
build-release-tarball. And, starting with Zulip version 1.4, you can upgrade Zulip to a version in a Git
repository directly.
Next, run as root:
/home/zulip/deployments/current/scripts/upgrade-zulip zulip-server-VERSION.tar.gz
The upgrade process will shut down the Zulip service and then run apt-get upgrade, a puppet apply, any database
migrations, and then bring the Zulip service back up. Upgrading will result in some brief downtime for the service,
51
Zulip Documentation, Release 1.4.0
which should be under 30 seconds unless there is an expensive transition involved. Unless you have tested the upgrade
in advance, we recommend doing upgrades at off hours.
Note that upgrading an existing Zulip production server from Ubuntu 14.04 Trusty to Ubuntu 16.04 Xenial will require
significant manual intervention on your part to migrate the data in the database from Postgres 9.3 to Postgres 9.5.
Contributions on testing and documenting this process are welcome!
Preserving local changes to configuration files
Warning: If you have modified configuration files installed by Zulip (e.g. the nginx configuration), the Zulip upgrade
process will overwrite your configuration when it does the puppet apply.
You can test whether this will happen assuming no upstream changes to the configuration using scripts/
zulip-puppet-apply (without the -f option), which will do a test puppet run and output and changes it would
make. Using this list, you can save a copy of any files that you’ve modified, do the upgrade, and then restore your
configuration.
If you need to do this, please report the issue so that we can make the Zulip puppet configuration flexible enough to
handle your setup.
Troubleshooting with the upgrade log
The Zulip upgrade script automatically logs output to /var/log/zulip/upgrade.log. Please use those logs to
include output that shows all errors in any bug reports.
After the upgrade, we recommend checking /var/log/zulip/errors.log to confirm that your users are not
experiencing errors after the upgrade.
Rolling back to a prior version
The Zulip upgrade process works by creating a new deployment under /home/zulip/deployments/ containing a complete copy of the Zulip server code, and then moving the symlinks at /home/zulip/deployments/
current and /root/zulip as part of the upgrade process.
This means that if the new version isn’t working, you can quickly downgrade to the old version by using /home/
zulip/deployments/<date>/scripts/restart-server to return to a previous version that you’ve deployed (the version is specified via the path to the copy of restart-server you call).
Updating settings
If required, you can update your settings by editing /etc/zulip/settings.py and then run /home/zulip/
deployments/current/scripts/restart-server to restart the server.
Applying Ubuntu system updates
While the Zulip upgrade script runs apt-get upgrade, you are responsible for running this on your system on a
regular basis between Zulip upgrades to ensure that it is up to date with the latest security patches.
52
Chapter 11. Secure, maintain, and upgrade
Zulip Documentation, Release 1.4.0
API and your Zulip URL
To use the Zulip API with your Zulip server, you will need to use the API endpoint of e.g. https://
zulip.example.com/api. Our Python API example scripts support this via the --site=https://zulip.
example.com argument. The API bindings support it via putting site=https://zulip.example.com in
your .zuliprc.
Every Zulip integration supports this sort of argument (or e.g. a ZULIP_SITE variable in a zuliprc file or the environment), but this is not yet documented for some of the integrations (the included integration documentation on
/integrations will properly document how to do this for most integrations). We welcome pull requests for
integrations that don’t discuss this!
Similarly, you will need to instruct your users to specify the URL for your Zulip server when using the Zulip desktop
and mobile apps.
Memory leak mitigation
As a measure to mitigate the impact of potential memory leaks in one of the Zulip daemons, the service automatically
restarts itself every Sunday early morning. See /etc/cron.d/restart-zulip for the precise configuration.
Upgrading from a git repository
Starting with version 1.4, the Zulip server supports doing deployments from a Git repository. To configure
this, you will need to add zulip::static_asset_compiler to your /etc/zulip/zulip.conf file’s
puppet_classes entry, like this:
puppet_classes = zulip::voyager, zulip::static_asset_compiler
Then, run scripts/zulip-puppet-apply to install the dependencies for building Zulip’s static assets. You can
configure the git repository that you’d like to use by adding a section like this to /etc/zulip/zulip.conf; by
default it uses the main zulip repository (shown below).
[deployment]
git_repo_url = https://github.com/zulip/zulip.git
Once that is done (and assuming the currently installed version of Zulip is new enough that this script exists), you can
do deployments by running as root:
/home/zulip/deployments/current/scripts/upgrade-zulip-from-git <branch>
and Zulip will automatically fetch the relevant branch from the specified repository, build the static assets, and deploy
that version. Currently, the upgrade process is slow, but it doesn’t need to be; there is ongoing work on optimizing it.
Backups
There are several pieces of data that you might want to back up:
• The postgres database. That you can back up like any postgres database; we have some example tooling for doing that incrementally into S3 using wal-e in puppet/zulip_internal/manifests/
postgres_common.pp (that’s what we use for zulip.com’s database backups). Note that this module isn’t
part of the Zulip server releases since it’s part of the zulip.com configuration (see https://github.com/zulip/zulip/
issues/293 for a ticket about fixing this to make life easier for running backups).
11.2. Upgrading from a git repository
53
Zulip Documentation, Release 1.4.0
• Any user-uploaded files. If you’re using S3 as storage for file uploads, this is backed up in S3, but if you
have instead set LOCAL_UPLOADS_DIR, any files uploaded by users (including avatars) will be stored in that
directory and you’ll want to back it up.
• Your Zulip configuration including secrets from /etc/zulip/. E.g. if you lose the value of secret_key,
all users will need to login again when you setup a replacement server since you won’t be able to verify their
cookies; if you lose avatar_salt, any user-uploaded avatars will need to be re-uploaded (since avatar filenames are computed using a hash of avatar_salt and user’s email), etc.
• The logs under /var/log/zulip can be handy to have backed up, but they do get large on a busy server, and
it’s definitely lower-priority.
If you are interested in backups because you are moving from one Zulip server to another server and can’t transfer a
full postgres dump (which is definitely the simplest approach), our draft conversion and export design document may
help. The tool is well designed and was tested carefully with dozens of realms as of mid-2016 but is not integrated
into Zulip’s regular testing process, and thus it is worth asking on the Zulip developers mailing list whether it needs
any minor updates to do things like export newly added tables.
Restore from backups
To restore from backups, the process is basically the reverse of the above:
• Install new server as normal by downloading a Zulip release tarball and then using scripts/setup/
install, you don’t need to run the initialize-database second stage which puts default data into
the database.
• Unpack to /etc/zulip the settings.py and zulip-secrets.conf files from your backups.
• Restore your database from the backup using wal-e; if you ran initialize-database anyway above,
you’ll want to first scripts/setup/postgres-init-db to drop the initial database first.
• Reconfigure rabbitmq to use the password from secrets.conf by running, as root, scripts/setup/
configure-rabbitmq.
• If you’re using local file uploads, restore those files to the path specified by settings.
LOCAL_UPLOADS_DIR and (if appropriate) any logs.
• Start the server using scripts/restart-server.
This restoration process can also be used to migrate a Zulip installation from one server to another.
We recommend running a disaster recovery after you setup backups to confirm that your backups are working; you
may also want to monitor that they are up to date using the Nagios plugin at: puppet/zulip_internal/files/
nagios_plugins/check_postgres_backup.
Contributions to more fully automate this process or make this section of the guide much more explicit and detailed
are very welcome!
Postgres streaming replication
Zulip has database configuration for using Postgres streaming replication; you can see the configuration in these files:
• puppet/zulip_internal/manifests/postgres_slave.pp
• puppet/zulip_internal/manifests/postgres_master.pp
• puppet/zulip_internal/files/postgresql/*
Contribution of a step-by-step guide for setting this up (and moving this configuration to be available in the main
puppet/zulip/ tree) would be very welcome!
54
Chapter 11. Secure, maintain, and upgrade
Zulip Documentation, Release 1.4.0
Monitoring
The complete Nagios configuration (sans secret keys) used to monitor zulip.com is available under puppet/
zulip_internal in the Zulip Git repository (those files are not installed in the release tarballs).
The Nagios plugins used by that configuration are installed automatically by the Zulip installation process in subdirectories under /usr/lib/nagios/plugins/. The following is a summary of the various Nagios plugins included
with Zulip and what they check:
Application server and queue worker monitoring:
• check_send_receive_time (sends a test message through the system between two bot users to check that
end-to-end message sending works)
• check_rabbitmq_consumers and check_rabbitmq_queues (checks for rabbitmq being down or
the queue workers being behind)
• check_queue_worker_errors (checks for errors reported by the queue workers)
• check_worker_memory (monitors for memory leaks in queue workers)
• check_email_deliverer_backlog and check_email_deliverer_process (monitors for
whether outgoing emails are being sent)
Database monitoring:
• check_postgres_replication_lag (checks streaming replication is up to date).
• check_postgres (checks the health of the postgres database)
• check_postgres_backup (checks backups are up to date; see above)
• check_fts_update_log (monitors for whether full-text search updates are being processed)
Standard server monitoring:
• check_website_response.sh (standard HTTP check)
• check_debian_packages (checks apt repository is up to date)
If you’re using these plugins, bug reports and pull requests to make it easier to monitor Zulip and maintain it in
production are encouraged!
Scalability
This section attempts to address the considerations involved with running Zulip with larger teams (especially >1000
users).
• For an organization with 100+ users, it’s important to have more than 4GB of RAM on the system. Zulip
will install on a system with 2GB of RAM, but with less than 3.5GB of RAM, it will run its queue processors
multithreaded to conserve memory; this creates a significant performance bottleneck.
• chat.zulip.org, with thousands of user accounts and thousands of messages sent every week, has 8GB of RAM,
4 cores, and 80GB of disk. The CPUs are essentially always idle, but the 8GB of RAM is important.
• We recommend using a remote postgres database for isolation, though it is not required. In the following,
we discuss a relatively simple configuration with two types of servers: application servers (running Django,
Tornado, RabbitMQ, Redis, Memcached, etc.) and database servers.
• You can scale to a pretty large installation (O(~1000) concurrently active users using it to chat all day) with
just a single reasonably large application server (e.g. AWS c3.2xlarge with 8 cores and 16GB of RAM) sitting
mostly idle (<10% CPU used and only 4GB of the 16GB RAM actively in use). You can probably get away
11.4. Monitoring
55
Zulip Documentation, Release 1.4.0
with half that (e.g. c3.xlarge), but ~8GB of RAM is highly recommended at scale. Beyond a 1000 active users,
you will eventually want to increase the memory cap in memcached.conf from the default 512MB to avoid
high rates of memcached misses.
• For the database server, we highly recommend SSD disks, and RAM is the primary resource limitation. We
have not aggressively tested for the minimum resources required, but 8 cores with 30GB of RAM (e.g. AWS’s
m3.2xlarge) should suffice; you may be able to get away with less especially on the CPU side. The database
load per user is pretty optimized as long as memcached is working correctly. This has not been tested, but
from extrapolating the load profile, it should be possible to scale a Zulip installation to 10,000s of active users
using a single large database server without doing anything complicated like sharding the database.
• For reasonably high availability, it’s easy to run a hot spare application server and a hot spare database (using
Postgres streaming replication; see the section on configuring this). Be sure to check out the section on backups
if you’re hoping to run a spare application server; in particular you probably want to use the S3 backend for
storing user-uploaded files and avatars and will want to make sure secrets are available on the hot spare.
• Zulip does not support dividing traffic for a given Zulip realm between multiple application servers. There
are two issues: you need to share the memcached/Redis/RabbitMQ instance (these should can be moved to a
network service shared by multiple servers with a bit of configuration) and the Tornado event system for pushing
to browsers currently has no mechanism for multiple frontend servers (or event processes) talking to each other.
One can probably get a factor of 10 in a single server’s scalability by supporting multiple tornado processes on
a single server, which is also likely the first part of any project to support exchanging events amongst multiple
servers.
Questions, concerns, and bug reports about this area of Zulip are very welcome! This is an area we are hoping to
improve.
Securing your Zulip server
Zulip’s security model is discussed in a separate document.
Management commands
Zulip has a large library of Django management commands. To use them, you will want to be logged in as the
zulip user and for the purposes of this documentation, we assume the current working directory is /home/zulip/
deployments/current.
Below, we show several useful examples, but there are more than 100 in total. We recommend skimming the usage
docs (or if there are none, the code) of a management command before using it, since they are generally less polished
and more designed for expert use than the rest of the Zulip system.
manage.py shell
You can get an iPython shell with full access to code within the Zulip project using manage.py shell, e.g., you
can do the following to change an email address:
$ /home/zulip/deployments/current/manage.py shell
In [1]: user_profile = get_user_profile_by_email("email@example.com")
In [2]: do_change_user_email(user_profile, "new_email@example.com")
56
Chapter 11. Secure, maintain, and upgrade
Zulip Documentation, Release 1.4.0
manage.py dbshell
This will start a postgres shell connected to the Zulip database.
Grant administrator access
You can make any user a realm administrator on the command line with the knight management command:
./manage.py knight username@example.com -f
Creating API super users with manage.py
If you need to manage the IRC, Jabber, or Zephyr mirrors, you will need to create API super users. To do this, use
./manage.py knight with the --permission=api_super_user argument. See api/integrations/
irc-mirror.py and api/integrations/jabber_mirror.py for further detail on these.
Exporting users and realms with manage.py export
If you need to do an export of a single user or of an entire realm, we have tools in management/ that essentially
export Zulip data to the file system.
export_single_user.py exports the message history and realm-public metadata for a single Zulip user (including that user’s received messages as well as their sent messages).
A good overview of the process for exporting a single realm when moving a realm to a new server (without moving
a full database dump) is in management/export.py. We recommend you read the comment there for words of wisdom
on speed, what is and is not exported, what will break upon a move to a new server, and suggested procedure.
Other useful manage.py commands
There are a large number of useful management commands under zerver/management/commands/; you can
also see them listed using ./manage.py with no arguments.
11.7. Management commands
57
Zulip Documentation, Release 1.4.0
58
Chapter 11. Secure, maintain, and upgrade
CHAPTER
12
Security Model
This section attempts to document the Zulip security model. Since this is new documentation, it likely does not cover
every issue; if there are details you’re curious about, please feel free to ask questions on the Zulip development mailing
list (or if you think you’ve found a security bug, please report it to zulip-security@googlegroups.com so we can do a
responsible security announcement).
Secure your Zulip server like your email server
• It’s reasonable to think about security for a Zulip server like you do security for a team email server – only
trusted administrators within an organization should have shell access to the server.
In particular, anyone with root access to a Zulip application server or Zulip database server, or with access to
the zulip user on a Zulip application server, has complete control over the Zulip installation and all of its data
(so they can read messages, modify history, etc.). It would be difficult or impossible to avoid this, because the
server needs access to the data to support features expected of a group chat system like the ability to search the
entire message history, and thus someone with control over the server has access to that data as well.
Encryption and Authentication
• Traffic between clients (web, desktop and mobile) and the Zulip is encrypted using HTTPS. By default, all Zulip
services talk to each other either via a localhost connection or using an encrypted SSL connection.
• Zulip requires CSRF tokens in all interactions with the web API to prevent CSRF attacks.
• The preferred way to login to Zulip is using an SSO solution like Google Auth, LDAP, or similar, but Zulip also
supports password authentication. See the authentication methods documentation for details on Zulip’s available
authentication methods.
59
Zulip Documentation, Release 1.4.0
Passwords
Zulip stores user passwords using the standard PBKDF2 algorithm.
Password strength is checked and
weak passwords are visually discouraged using the popular zxcvbn library. The minimum password strength
allowed is controlled by two settings in /etc/zulip/settings.py; PASSWORD_MIN_LENGTH and
PASSWORD_MIN_ZXCVBN_QUALITY. The former is self-explanatory; we will explain the latter.
Password strength estimation is a complicated topic that we can’t go into great detail on here; we recommend reading
the zxvcbn website for details if you are not familiar with password strength analysis.
In Zulip’s configuration, a password has quality X if zxcvbn estimates that it would take e^(X * 22) seconds to
crack the password with a specific attack scenario. The scenario Zulip uses is one where an the attacker breaks into
the Zulip server and steals the hashed passwords; in that case, with a slow hash, the attacker would be able to make
roughly 10,000 attempts per second. E.g. a password with quality 0.5 (the default), it would take an attacker about 16
hours to crack such a password in this sort of offline attack.
Another important attack scenario is the online attacks (i.e. an attacker sending tons of login requests guessing different
passwords to a Zulip server over the web). Those attacks are much slower (more like 10/second without rate limiting),
and you should estimate the time to guess a password as correspondingly longer.
As a server administrators, you must balance the security risks associated with attackers guessing weak passwords
against the usability challenges associated with requiring strong passwords in your organization.
Messages and History
• Zulip message content is rendered using a specialized Markdown parser which escapes content to protect against
cross-site scripting attacks.
• Zulip supports both public streams and private (“invite-only”) streams. Any Zulip user can join any public
stream in the realm, and can view the complete message history of any public stream without joining the stream.
• A private (“invite-only”) stream is hidden from users who are not subscribed to the stream. Users who are
not members of a private stream cannot read messages on the stream, send messages to the stream, or join the
stream, even if they are a Zulip realm administrator. Users can join private streams only when they are invited.
However, any member of a private stream can invite other users to the stream. When a new user joins a private
stream, they can see future messages sent to the stream, but they do not receive access to the stream’s message
history.
• Zulip supports editing the content and topics of messages that have already been sent. As a general philosophy,
our policies provide hard limits on the ways in which message content can be changed or undone. In contrast,
our policies around message topics favor usefulness (e.g. for conversational organization) over faithfulness to
the original.
The message editing policy can be configured on the /#organization page. There are three configurations provided out of the box: (i) users cannot edit messages at all, (ii) users can edit any message they have sent, and
(iii) users can edit the content of any message they have sent in the last N minutes, and the topic of any message
they have sent. In (ii) and (iii), topic edits can also be propagated to other messages with the same original topic,
even if those messages were sent by other users. The default setting is (iii), with N = 10.
In addition, and regardless of the configuration above, messages with no topic can always be edited to have
a topic, by anyone in the organization, and the topic of any message can also always be edited by a realm
administrator.
Also note that while edited messages are synced immediately to open browser windows, editing messages is
not a safe way to redact secret content (e.g. a password) shared unintentionally. Other users may have seen
and saved the content of the original message, or have an integration (e.g. push notifications) forwarding all
60
Chapter 12. Security Model
Zulip Documentation, Release 1.4.0
messages they receive to another service. Zulip also stores and sends to clients the content of every historical
version of a message.
Users and Bots
• There are three types of users in a Zulip realm: Administrators, normal users, and bots. Administrators have
the ability to deactivate and reactivate other human and bot users, delete streams, add/remove administrator
privileges, as well as change configuration for the overall realm (e.g. whether an invitation is required to join the
realm). Being a Zulip administrator does not provide the ability to interact with other users’ private messages or
the messages sent to private streams to which the administrator is not subscribed. However, a Zulip administrator
subscribed to a stream can toggle whether that stream is public or private. Also, Zulip realm administrators have
administrative access to the API keys of all bots in the realm, so a Zulip administrator may be able to access
messages sent to private streams that have bots subscribed, by using the bot’s credentials.
In the future, Zulip’s security model may change to allow realm administrators to access private messages (e.g.
to support auditing functionality).
• Every Zulip user has an API key, available on the settings page. This API key can be used to do essentially
everything the user can do; for that reason, users should keep their API key safe. Users can rotate their own API
key if it is accidentally compromised.
• To properly remove a user’s access to a Zulip team, it does not suffice to change their password or deactivate
their account in the SSO system, since neither of those prevents authenticating with the user’s API key or those
of bots the user has created. Instead, you should deactivate the user’s account in the “Organization settings”
interface (/#organization); this will automatically also deactivate any bots the user had created.
• The Zulip mobile apps authenticate to the server by sending the user’s password and retrieving the user’s API
key; the apps then use the API key to authenticate all future interactions with the site. Thus, if a user’s phone is
lost, in addition to changing passwords, you should rotate the user’s Zulip API key.
• Zulip bots are used for integrations. A Zulip bot can do everything a normal user in the realm can do including
reading other, with a few exceptions (e.g. a bot cannot login to the web application or create other bots). In
particular, with the API key for a Zulip bot, one can read any message sent to a public stream in that bot’s realm.
A likely future feature for Zulip is limited bots that can only send messages.
• Certain Zulip bots can be marked as “API super users”; these special bots have the ability to send messages that
appear to have been sent by another user (an important feature for implementing integrations like the Jabber,
IRC, and Zephyr mirrors).
User-uploaded content
• Zulip supports user-uploaded files; ideally they should be hosted from a separate domain from the main Zulip
server to protect against various same-domain attacks (e.g. zulip-user-content.example.com) using the S3 integration.
The URLs of user-uploaded files are secret; if you are using the “local file upload” integration, anyone with the
URL of an uploaded file can access the file. This means the local uploads integration is vulnerable to a subtle
attack where if a user clicks on a link in a secret .PDF or .HTML file that had been uploaded to Zulip, access to
the file might be leaked to the other server via the Referrer header (see the “Uploads world readable” issue on
GitHub).
The Zulip S3 file upload integration is relatively safe against that attack, because the URLs of files presented to
users don’t host the content. Instead, the S3 integration checks the user has a valid Zulip session in the relevant
realm, and if so then redirects the browser to a one-time S3 URL that expires a short time later. Keeping the
URL secret is still important to avoid other users in the Zulip realm from being able to access the file.
12.4. Users and Bots
61
Zulip Documentation, Release 1.4.0
• Zulip supports using the Camo image proxy to proxy content like inline image previews that can be inserted into
the Zulip message feed by other users over HTTPS.
• By default, Zulip will provide image previews inline in the body of messages when a message contains a link to
an image. You can control this using the INLINE_IMAGE_PREVIEW setting.
Final notes and security response
If you find some aspect of Zulip that seems inconsistent with this security model, please report it to zulipsecurity@googlegroups.com so that we can investigate and coordinate an appropriate security release if needed.
Zulip security announcements will be sent to zulip-announce@googlegroups.com, so you should subscribe if you are
running Zulip in production.
62
Chapter 12. Security Model
CHAPTER
13
Authentication methods
Zulip supports several different authentications methods:
• EmailAuthBackend - Email/password authentication.
• ZulipLDAPAuthBackend - LDAP username/password authentication.
• GoogleMobileOauth2Backend - Google authentication.
• GitHubAuthBackend - GitHub authentication.
• ZulipRemoteUserBackend - Authentication using an existing Single-Sign-On (SSO) system that can set
REMOTE_USER in Apache.
• DevAuthBackend - Only for development, passwordless login as any user.
It’s easy to add more; see the docs on python-social-auth below.
The setup documentation for most of these is simple enough that we’ve included it inline in /etc/zulip/
settings.py, right above to the settings used to configure them. The remote user authentication backend is more
complex since it requires interfacing with a generic third-party authentication system, and so we’ve documented it in
detail below.
Adding additional methods using python-social-auth
The implementation for GitHubAuthBackend is a small wrapper around the popular python-social-auth library. So if
you’d like to integrate Zulip with another authentication provider (e.g. Facebook, Twitter, etc.), you can do this by
writing a class similar to GitHubAuthBackend in zproject/backends.py and adding a few settings. Pull
requests to add new backends are welcome; they should be tested using the framework in test_auth_backends.
py.
63
Zulip Documentation, Release 1.4.0
Remote User SSO Authentication
Zulip supports integrating with a Single-Sign-On solution. There are a few ways to do it, but this section documents
how to configure Zulip to use an SSO solution that best supports Apache and will set the REMOTE_USER variable:
(0) Check that /etc/zulip/settings.py has zproject.backends.ZulipRemoteUserBackend as
the only enabled value in the AUTHENTICATION_BACKENDS list, and that SSO_APPEND_DOMAIN is correct set
depending on whether your SSO system uses email addresses or just usernames in REMOTE_USER.
Make sure that you’ve restarted the Zulip server since making this configuration change.
(1) Edit /etc/zulip/zulip.conf and change the puppet_classes line to read:
puppet_classes = zulip::voyager, zulip::apache_sso
(2) As root, run /home/zulip/deployments/current/scripts/zulip-puppet-apply to install our
SSO integration.
(3) To configure our SSO integration, edit /etc/apache2/sites-available/zulip-sso.example and
fill in the configuration required for your SSO service to set REMOTE_USER and place your completed configuration
file at /etc/apache2/sites-available/zulip-sso.conf
zulip-sso.example is correct configuration for using an htpasswd file for REMOTE_USER authentication,
which is useful for testing quickly. You can set it up by doing the following:
/home/zulip/deployments/current/scripts/restart-server
cd /etc/apache2/sites-available/
cp zulip-sso.example zulip-sso.conf
htpasswd -c /home/zulip/zpasswd username@example.com # prompts for a password
and then continuing with the steps below.
(4) Run a2ensite zulip-sso to enable the Apache integration site.
(5) Run service apache2 reload to use your new configuration. If Apache isn’t already running, you may
need to run service apache2 start instead.
Now you should be able to visit https://zulip.example.com/ and login via the SSO solution.
Troubleshooting Remote User SSO
This system is a little finicky to networking setup (e.g. common issues have to do with /etc/hosts not mapping settings.EXTERNAL_HOST to the Apache listening on 127.0.0.1/localhost, for example). It can
often help while debugging to temporarily change the Apache config in /etc/apache2/sites-available/
zulip-sso to listen on all interfaces rather than just 127.0.0.1 as you debug this. It can also be helpful to
change /etc/nginx/zulip-include/app.d/external-sso.conf to proxy_pass to a more explicit
URL possibly not over HTTPS when debugging. The following log files can be helpful when debugging this setup:
• /var/log/zulip/{errors.log,server.log} (the usual places)
• /var/log/nginx/access.log (nginx access logs)
• /var/log/apache2/zulip_auth_access.log (you may want to change LogLevel to “debug” in
the Apache config file to make this more verbose)
Here’s a summary of how the remote user SSO system works assuming you’re using HTTP basic auth; this summary
should help with understanding what’s going on as you try to debug:
64
Chapter 13. Authentication methods
Zulip Documentation, Release 1.4.0
• Since you’ve configured /etc/zulip/settings.py to only define the zproject.backends.
ZulipRemoteUserBackend, zproject/settings.py configures /accounts/login/sso as
HOME_NOT_LOGGED_IN, which makes https://zulip.example.com/ (a.k.a. the homepage for the
main Zulip Django app running behind nginx) redirect to /accounts/login/sso if you’re not logged in.
• nginx proxies requests to /accounts/login/sso/ to an Apache instance listening on localhost:8888
apache via the config in /etc/nginx/zulip-include/app.d/external-sso.conf (using the upstream localhost:8888 defined in /etc/nginx/zulip-include/upstreams).
• The Apache zulip-sso site which you’ve enabled listens on localhost:8888 and presents the
htpasswd dialogue; you provide correct login information and the request reaches a second Zulip Django
app instance that is running behind Apache with with REMOTE_USER set. That request is served by zerver.
views.remote_user_sso, which just checks the REMOTE_USER variable and either logs in (sets a cookie)
or registers the new user (depending whether they have an account).
• After succeeding, that redirects the user back to / on port 443 (hosted by nginx); the main Zulip Django app
sees the cookie and proceeds to load the site homepage with them logged in (just as if they’d logged in normally
via username/password).
Again, most issues with this setup tend to be subtle issues with the hostname/DNS side of the configuration. Suggestions for how to improve this SSO setup documentation are very welcome!
13.2. Remote User SSO Authentication
65
Zulip Documentation, Release 1.4.0
66
Chapter 13. Authentication methods
CHAPTER
14
Postgres database details
Remote Postgres database
This is a bit annoying to setup, but you can configure Zulip to use a dedicated postgres server by setting the
REMOTE_POSTGRES_HOST variable in /etc/zulip/settings.py, and configuring Postgres certificate authentication (see
http://www.postgresql.org/docs/9.1/static/ssl-tcp.html and http://www.postgresql.org/docs/9.1/static/libpq-ssl.html for
documentation on how to set this up and deploy the certificates) to make the DATABASES configuration in
zproject/settings.py work (or override that configuration).
If you want to use a remote Postgresql database, you should configure the information about the connection with the
server. You need a user called “zulip” in your database server. You can configure these options in /etc/zulip/settings.py
(the below descriptions are from the Postgresql documentation):
• REMOTE_POSTGRES_HOST: Name or IP address of the remote host
• REMOTE_POSTGRES_SSLMODE: SSL Mode used to connect to the server, different options you can use
are:
– disable: I don’t care about security, and I don’t want to pay the overhead of encryption.
– allow: I don’t care about security, but I will pay the overhead of encryption if the server insists on it.
– prefer: I don’t care about encryption, but I wish to pay the overhead of encryption if the server supports it.
– require: I want my data to be encrypted, and I accept the overhead. I trust that the network will make sure
I always connect to the server I want.
– verify-ca: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server
that I trust.
– verify-full: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server
I trust, and that it’s the one I specify.
Then you should specify the password of the user zulip for the database in /etc/zulip/zulip-secrets.conf:
postgres_password = xxxx
67
Zulip Documentation, Release 1.4.0
Finally, you can stop your database on the Zulip server via:
sudo service postgresql stop
sudo update-rc.d postgresql disable
In future versions of this feature, we’d like to implement and document how to the remote postgres database server
itself automatically by using the Zulip install script with a different set of puppet manifests than the all-in-one feature;
if you’re interested in working on this, post to the Zulip development mailing list and we can give you some tips.
Debugging postgres database issues
When debugging postgres issues, in addition to the standard pg_top tool, often it can be useful to use this query:
SELECT procpid,waiting,query_start,current_query FROM pg_stat_activity ORDER BY
˓→procpid;
which shows the currently running backends and their activity. This is similar to the pg_top output, with the added
advantage of showing the complete query, which can be valuable in debugging.
To stop a runaway query, you can run SELECT pg_cancel_backend(pid int) or SELECT
pg_terminate_backend(pid int) as the ‘postgres’ user. The former cancels the backend’s current
query and the latter terminates the backend process. They are implemented by sending SIGINT and SIGTERM to
the processes, respectively. We recommend against sending a Postgres process SIGKILL. Doing so will cause the
database to kill all current connections, roll back any pending transactions, and enter recovery mode.
Stopping the Zulip postgres database
To start or stop postgres manually, use the pg_ctlcluster command:
pg_ctlcluster 9.1 [--force] main {start|stop|restart|reload}
By default, using stop uses “smart” mode, which waits for all clients to disconnect before shutting down the database.
This can take prohibitively long. If you use the –force option with stop, pg_ctlcluster will try to use the “fast” mode
for shutting down. “Fast” mode is described by the manpage thusly:
With the –force option the “fast” mode is used which rolls back all active transactions, disconnects clients immediately
and thus shuts down cleanly. If that does not work, shutdown is attempted again in “immediate” mode, which can leave
the cluster in an inconsistent state and thus will lead to a recovery run at the next start. If this still does not help, the
postmaster process is killed. Exits with 0 on success, with 2 if the server is not running, and with 1 on other failure
conditions. This mode should only be used when the machine is about to be shut down.
Many database parameters can be adjusted while the database is running.
/etc/postgresql/9.1/main/postgresql.conf and issue a reload. The logs will note the change.
Just
modify
Debugging issues starting postgres
pg_ctlcluster often doesn’t give you any information on why the database failed to start. It may tell you to check the
logs, but you won’t find any information there. pg_ctlcluster runs the following command underneath when it actually
goes to start Postgres:
/usr/lib/postgresql/9.1/bin/pg_ctl start -D /var/lib/postgresql/9.1/main -s -o \
'-c config_file="/etc/postgresql/9.1/main/postgresql.conf"'
68
Chapter 14. Postgres database details
Zulip Documentation, Release 1.4.0
Since pg_ctl doesn’t redirect stdout or stderr, running the above can give you better diagnostic information. However,
you might want to stop Postgres and restart it using pg_ctlcluster after you’ve debugged with this approach, since it
does bypass some of the work that pg_ctlcluster does.
Postgres Vacuuming alerts
The autovac_freeze postgres alert from check_postgres is particularly important. This alert indicates
that the age (in terms of number of transactions) of the oldest transaction id (XID) is getting close to the
autovacuum_freeze_max_age setting. When the oldest XID hits that age, Postgres will force a VACUUM
operation, which can often lead to sudden downtime until the operation finishes. If it did not do this and the age of the
oldest XID reached 2 billion, transaction id wraparound would occur and there would be data loss. To clear the nagios
alert, perform a VACUUM in each indicated database as a database superuser (postgres).
See http://www.postgresql.org/docs/9.1/static/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND for more
details on postgres vacuuming.
14.5. Postgres Vacuuming alerts
69
Zulip Documentation, Release 1.4.0
70
Chapter 14. Postgres database details
CHAPTER
15
Development environment installation
Requirements
The Zulip development environment can be installed on macOS, Windows, and Linux (Ubuntu recommended).
You’ll need at least 2GB of available RAM.
Installing the Zulip development environment requires downloading several hundred megabytes of dependencies, so
you will need an active, reasonably fast, internet connection throughout the entire installation processes. You
can configure a proxy if you need one.
Recommended setup (Vagrant)
For first-time contributors on macOS, Windows, and Ubuntu, we recommend using the Vagrant development
environment.
This method creates a virtual machine (for Windows and macOS) or a Linux container (for Ubuntu) inside which the
Zulip server and all related services will run. Vagrant adds a bit of overhead to using the Zulip development server,
but provides an isolated environment that is easy to install, update, and uninstall. It has been well-tested and performs
well.
Advanced setup (non-Vagrant)
For more experienced contributors, or for first-time contributors who don’t want to or can’t use Vagrant, Zulip supports
a wide range of ways to install the Zulip development environment on macOS and Linux (Ubuntu recommended):
• On Ubuntu 16.04 Xenial and 14.04 Trusty, you can easily install without using Vagrant.
• On other Linux distributions, you’ll need to follow slightly different instructions to install manually.
• On macOS and Linux (Ubuntu recommended), you can install using Docker, though support for this remains
experimental.
71
Zulip Documentation, Release 1.4.0
Unfortunately, the only supported method to install on Windows is the Vagrant method.
Slow internet connections
If you have a very slow network connection, however, you may want to avoid using Vagrant (which involves downloading an Ubuntu virtual machine or Linux Container) and either install directly on Ubuntu 16.04 Xenial or 14.04
Trust, or use the manual install process instead. These options only support Linux.
An alternative option if you have poor network connectivity is to rent a cloud server and install the Zulip development
environment for remote use. See next section for details.
Installing remotely
The Zulip development environment works well on remote virtual machines. This can be a good alternative for those
with poor network connectivity or who have limited storage/memory on their local machines.
We recommend giving the Zulip development environment its own virtual machine, running Ubuntu 14.04 or 16.04,
with at least 2GB of memory.
If the Zulip development environment will be the only thing running on the remote virtual machine, we recommend
installing directly. Otherwise, we recommend the Vagrant method so you can easily uninstall if you need to.
If you want to run a non-Ubuntu distribution, follow the generic Linux directions.
Next steps
Once you’ve installed the Zulip development environment, you’ll want to read these documents to learn how to use it:
• Using the Development Environment
• Testing
And if you’ve setup the Zulip development environment on a remote machine, take a look at our tips for developing
remotely.
72
Chapter 15. Development environment installation
CHAPTER
16
Vagrant environment setup tutorial
This section guides first-time contributors through installing the Zulip development environment on Windows, macOS,
and Ubuntu.
The recommended method for installing the Zulip development environment is to use Vagrant with VirtualBox on
Windows and macOS, and Vagrant with LXC on Ubuntu. This method creates a virtual machine (for Windows and
macOS) or a Linux container (for Ubuntu) inside which the Zulip server and all related services will run.
Contents:
• Requirements
• Step 1: Install Prerequisites
• Step 2: Get Zulip code
• Step 3: Start the development environment
• Step 4: Developing
• Troubleshooting & Common Errors
• Specifying a proxy
If you encounter errors installing the Zulip development environment, check Troubleshooting & Common Errors.
If that doesn’t help, please visit #provision help in the Zulip development community server for real-time help, send a
note to the Zulip-devel Google group or file an issue.
When reporting your issue, please include the following information:
• host operating system
• installation method (Vagrant or direct)
• whether or not you are using a proxy
• a copy of Zulip’s vagrant provisioning logs, available in /var/log/provision.log on your virtual
machine
73
Zulip Documentation, Release 1.4.0
Requirements
Installing the Zulip development environment requires downloading several hundred megabytes of dependencies. You
will need an active internet connection throughout the entire installation processes. (See Specifying a proxy if you
need a proxy to access the internet.)
• All: 2GB available RAM, Active broadband internet connection.
• macOS: macOS (10.11 El Capitan or 10.12 Sierra recommended), Git, VirtualBox, Vagrant.
• Ubuntu: 14.04 64-bit or 16.04 64-bit, Git, Vagrant, lxc.
– or Debian: 9.0 “stretch” 64-bit
• Windows: Windows 64-bit (Win 10 recommended), hardware virtualization enabled (VT-X or AMD-V), administrator access, Git for Windows (which installs Git BASH), VirtualBox, Vagrant.
Don’t see your system listed above? See Advanced setup for details about installing for other Linux and UNIX
platforms.
Step 1: Install Prerequisites
Jump to:
• macOS
• Ubuntu
• Debian
• Windows
macOS
1. Install Vagrant (1.8.4-1.8.6, do not use 1.8.7).
2. Install VirtualBox (>= 5.1.8).
(For a non-free option, but better performance, you can also use VMWare Fusion with the VMWare Fusion Vagrant
plugin.)
Now you are ready for Step 2: Get Zulip Code..
Ubuntu
The setup for Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial are the same.
If you’re in a hurry, you can copy and paste the following into your terminal after which you can jump to Step 2: Get
Zulip Code:
sudo apt-get -y purge vagrant && \
wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.deb && \
sudo dpkg -i vagrant*.deb && \
sudo apt-get -y install build-essential git ruby lxc lxc-templates cgroup-lite redir &
˓→& \
vagrant plugin install vagrant-lxc && \
vagrant lxc sudoers
74
Chapter 16. Vagrant environment setup tutorial
Zulip Documentation, Release 1.4.0
For a step-by-step explanation, read on.
1. Install Vagrant
For both 14.04 Trusty and 16.04 Xenial, you’ll need a more recent version of Vagrant than what’s available in the
official Ubuntu repositories.
First uninstall any vagrant package you may have installed from the Ubuntu repository:
christie@ubuntu-desktop:~
$ sudo apt-get purge vagrant
Now download and install the .deb package for Vagrant 1.8.6:
christie@ubuntu-desktop:~
$ wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.deb
christie@ubuntu-desktop:~
$ sudo dpkg -i vagrant*.deb
2. Install remaining dependencies
Now install git and lxc-related packages:
christie@ubuntu-desktop:~
$ sudo apt-get install build-essential git ruby lxc lxc-templates cgroup-lite redir
3. Install the vagrant lxc plugin:
christie@ubuntu-desktop:~
$ vagrant plugin install vagrant-lxc
Installing the 'vagrant-lxc' plugin. This can take a few minutes...
Installed the plugin 'vagrant-lxc (1.2.1)'!
If you encounter an error when trying to install the vagrant-lxc plugin, see this.
4. Configure sudo to be passwordless
Finally, configure sudo to be passwordless when using Vagrant LXC:
christie@ubuntu-desktop:~
$ vagrant lxc sudoers
[sudo] password for christie:
If you encounter an error running vagrant lxc sudoers, see this.
Now you are ready for Step 2: Get Zulip Code.
Debian
The setup for Debian 9.0 “stretch” is just like for Ubuntu 16.04, with one difference.
16.2. Step 1: Install Prerequisites
75
Zulip Documentation, Release 1.4.0
If you’re in a hurry, you can copy and paste the following into your terminal after which you can jump to Step 2: Get
Zulip Code:
sudo apt-get -y purge vagrant && \
wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.deb && \
sudo dpkg -i vagrant*.deb && \
sudo apt-get -y install build-essential git ruby lxc redir && \
vagrant plugin install vagrant-lxc && \
vagrant lxc sudoers
For a step-by-step explanation, follow the Ubuntu instructions above, with the following difference: in “2. Install
remaining dependencies”, the command is
sudo apt-get install build-essential git ruby lxc redir
Windows 10
1. Install Git for Windows, which installs Git BASH.
2. Install VirtualBox (version >= 5.1.6).
3. Install Vagrant (version 1.8.4-1.8.6, do not use 1.8.7).
(Note: While Git BASH is recommended, you may also use Cygwin. If you do, make sure to install default required
packages along with git, curl, openssh, and rsync binaries.)
After installing, you must run Git BASH as an administrator.
Also, you must have hardware virtualization enabled (VT-X or AMD-V) in your computer’s BIOS.
Enable native symlinks
The Zulip code includes symbolic links (symlinks). By default, native Windows symlinks are not enabled in either Git
BASH or Cygwin, so you need to do a bit of configuration. You must do this before you clone the Zulip code.
In Git for BASH:
Open Git BASH as an administrator and run:
$ git config --global core.symlinks true
Now confirm the setting:
$ git config core.symlinks
true
If you see true, you are ready for Step 2: Get Zulip Code.
Otherwise, if the above command prints false or nothing at all, then symlinks have not been enabled.
In Cygwin:
Open a Cygwin window as an administrator and do this:
christie@win10 ~
$ echo 'export "CYGWIN=$CYGWIN winsymlinks:native"' >> ~/.bash_profile
Next, close that Cygwin window and open another. If you echo $CYGWIN you should see:
76
Chapter 16. Vagrant environment setup tutorial
Zulip Documentation, Release 1.4.0
christie@win10 ~
$ echo $CYGWIN
winsymlinks:native
Now you are ready for Step 2: Get Zulip Code.
Step 2: Get Zulip Code
If you haven’t already created an ssh key and added it to your GitHub account, you should do that now by following
these instructions.
1. In your browser, visit https://github.com/zulip/zulip and click the fork button. You will need to be logged in
to GitHub to do this.
2. Open Terminal (macOS/Ubuntu) or Git BASH (Windows; must run as an Administrator).
3. In Terminal/Git BASH, clone your fork:
git clone git@github.com:YOURUSERNAME/zulip.git
This will create a ‘zulip’ directory and download the Zulip code into it.
Don’t forget to replace YOURUSERNAME with your git username. You will see something like:
christie@win10 ~
$ git clone git@github.com:YOURUSERNAME/zulip.git
Cloning into 'zulip'...
remote: Counting objects: 73571, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 73571 (delta 1), reused 0 (delta 0), pack-reused 73569
Receiving objects: 100% (73571/73571), 105.30 MiB | 6.46 MiB/s, done.
Resolving deltas: 100% (51448/51448), done.
Checking connectivity... done.
Checking out files: 100% (1912/1912), done.`
Now you are ready for Step 3: Start the development environment.
Step 3: Start the development environment
Change into the zulip directory and tell vagrant to start the Zulip development environment with vagrant up.
christie@win10 ~
$ cd zulip
christie@win10 ~/zulip
$ vagrant up
The first time you run this command it will take some time because vagrant does the following:
• downloads the base Ubuntu 14.04 virtual machine image (for macOS and Windows) or container (for Ubuntu)
• configures this virtual machine/container for use with Zulip,
• creates a shared directory mapping your clone of the Zulip code inside the virtual machine/container at ~/
zulip
16.3. Step 2: Get Zulip Code
77
Zulip Documentation, Release 1.4.0
• runs the tools/provision script inside the virtual machine/container, which downloads all required dependencies, sets up the python environment for the Zulip development server, and initializes a default test database.
We call this process “provisioning”.
You will need an active internet connection during the entire process. (See Specifying a proxy if you need a proxy to
access the internet.) vagrant up can fail while provisioning if your Internet connection is unreliable. To retry, you
can use vagrant provision (vagrant up will just boot the guest without provisioning after the first time).
Other common issues are documented in the Troubleshooting & Common Errors section. If that doesn’t help, please
visit #provision help in the Zulip development community server for real-time help.
On Windows, you will see The system cannot find the path specified. message several times. This
is expected behavior and is not an error.
Once vagrant up has completed, connect to the development environment with vagrant ssh:
christie@win10 ~/zulip
$ vagrant ssh
You should see something like this on Windows and macOS:
Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-85-generic x86_64)
* Documentation:
https://help.ubuntu.com/
System information as of Wed May
System load:
Usage of /:
Memory usage:
Swap usage:
0.61
3.5% of 39.34GB
7%
0%
4 21:45:43 UTC 2016
Processes:
88
Users logged in:
0
IP address for eth0: 10.0.2.15
Graph this data and manage this system at:
https://landscape.canonical.com/
Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud
0 packages can be updated.
0 updates are security updates.
Or something as brief as this in the case of Ubuntu:
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 4.4.0-21-generic x86_64)
* Documentation:
https://help.ubuntu.com/
Congrats, you’re now inside the Zulip development environment!
You can confirm this by looking at the command prompt, which starts with (zulip-venv)vagrant@. If it just
starts with vagrant@, your provisioning failed and you should look at the troubleshooting section.
Next, start the Zulip server:
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~
$ cd zulip
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~/zulip
$ ./tools/run-dev.py
You will see several lines of output starting with something like:
78
Chapter 16. Vagrant environment setup tutorial
Zulip Documentation, Release 1.4.0
2016-05-04 22:20:33,895 INFO: process_fts_updates starting
Recompiling templates
2016-05-04 18:20:34,804 INFO: Not in recovery; listening for FTS updates
done
Validating Django models.py...
System check identified no issues (0 silenced).
Django version 1.8
Tornado server is running at http://localhost:9993/
Quit the server with CTRL-C.
2016-05-04 18:20:40,716 INFO
Tornado loaded 0 event queues in 0.001s
2016-05-04 18:20:40,722 INFO
Tornado 95.5% busy over the past 0.0 seconds
Performing system checks...
And ending with something similar to:
http://localhost:9994/webpack-dev-server/
webpack result is served from http://localhost:9991/webpack/
content is served from /srv/zulip
webpack: bundle is now VALID.
2016-05-06 21:43:29,553 INFO
2016-05-06 21:43:35,007 INFO
Tornado
Tornado
31.6% busy over the past 10.6 seconds
23.9% busy over the past 16.0 seconds
Now the Zulip server should be running and accessible. Verify this by navigating to http://localhost:9991/ in the
browser on your main machine.
You should see something like this:
16.4. Step 3: Start the development environment
79
Zulip Documentation, Release 1.4.0
The Zulip server will continue to run and send output to the terminal window. When you navigate to Zulip in your
browser, check your terminal and you should see something like:
2016-05-04 18:21:57,547 INFO
127.0.0.1
GET
˓→(unauth via ?)
[04/May/2016 18:21:57]"GET / HTTP/1.0" 302 0
2016-05-04 18:21:57,568 INFO
127.0.0.1
GET
˓→?)
[04/May/2016 18:21:57]"GET /login HTTP/1.0" 301 0
2016-05-04 18:21:57,819 INFO
127.0.0.1
GET
˓→login/ (unauth via ?)
302 582ms (+start: 417ms) /
301
4ms /login (unauth via
200 209ms (db: 7ms/2q) /
Now you’re ready for Step 4: Developing.
80
Chapter 16. Vagrant environment setup tutorial
Zulip Documentation, Release 1.4.0
Step 4: Developing
Where to edit files
You’ll work by editing files on your host machine, in the directory where you cloned Zulip. Use your favorite editor
(Sublime, Atom, Vim, Emacs, Notepad++, etc.).
When you save changes they will be synced automatically to the Zulip development environment on the virtual machine/container.
Each component of the Zulip development server will automatically restart itself or reload data appropriately when
you make changes. So, to see your changes, all you usually have to do is reload your browser. More details on how
this works are available below.
Don’t forget to read through the code style guidelines for details about how to configure your editor for Zulip. For
example, indentation should be set to 4 spaces rather than tabs.
Understanding run-dev.py debugging output
It’s good to have the terminal running run-dev.py up as you work since error messages including tracebacks along
with every backend request will be printed there.
See Logging for further details on the run-dev.py console output.
Committing and pushing changes with git
When you’re ready to commit or push changes via git, you will do this by running git commands in Terminal (macOS/Ubuntu) or Git BASH (Windows) in the directory where you cloned Zulip on your main machine.
If you’re new to working with Git/GitHub, check out our Git & GitHub Guide.
Maintaining the development environment
If after rebasing onto a new version of the Zulip server, you receive new errors while starting the Zulip server or
running tests, this is probably not because Zulip’s master branch is broken. Instead, this is likely because we’ve
recently merged changes to the development environment provisioning process that you need to apply to your development environment. To update your environment, you’ll need to re-provision your vagrant machine using vagrant
provision (this just runs tools/provision from your Zulip checkout inside the Vagrant guest); this should
complete in about a minute.
After provisioning, you’ll want to (re)start the Zulip development server.
If you run into any trouble, the #provision help in the Zulip development community server for is a great place to ask
for help.
Rebuilding the development environment
If you ever want to recreate your development environment again from scratch (e.g. to test as change you’ve made to
the provisioning process, or because you think something is broken), you can do so using vagrant destroy and
then vagrant up. This will usually be much faster than the original vagrant up since the base image is already
cached on your machine (it takes about 5 minutes to run with a fast Internet connection).
16.5. Step 4: Developing
81
Zulip Documentation, Release 1.4.0
Any additional programs (e.g. Zsh, emacs, etc.) or configuration that you may have installed in the development environment will be lost when you recreate it. To address this, you can create a script called tools/
custom_provision in your Zulip Git checkout; and place any extra setup commands there. Vagrant will
run tools/custom_provision every time you run vagrant provision (or create a Vagrant guest via
vagrant up).
Shutting down the development environment for use later
To shut down but preserve the development environment so you can use it again later use vagrant halt or
vagrant suspend.
You can do this from the same Terminal/Git BASH window that is running run-dev.py by pressing ^C to halt the server
and then typing exit. Or you can halt vagrant from another Terminal/Git BASH window.
From the window where run-dev.py is running:
2016-05-04 18:33:13,330 INFO
127.0.0.1
GET
200
˓→via ?)
^C
KeyboardInterrupt
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~/zulip$ exit
logout
Connection to 127.0.0.1 closed.
christie@win10 ~/zulip
92ms /register/ (unauth
Now you can suspend the development environment:
christie@win10 ~/zulip
$ vagrant suspend
==> default: Saving VM state and suspending execution...
If vagrant suspend doesn’t work, try vagrant halt:
christie@win10 ~/zulip
$ vagrant halt
==> default: Attempting graceful shutdown of VM...
Check out the Vagrant documentation to learn more about suspend and halt.
Resuming the development environment
When you’re ready to work on Zulip again, run vagrant up. You will also need to connect to the virtual machine
with vagrant ssh and re-start the Zulip server:
christie@win10 ~/zulip
$ vagrant up
$ vagrant ssh
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~
$ cd zulip
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~/zulip
$ ./tools/run-dev.py
82
Chapter 16. Vagrant environment setup tutorial
Zulip Documentation, Release 1.4.0
Next Steps
Next, read the following to learn more about developing for Zulip:
• Git & GitHub Guide
• Using the Development Environment
• Testing
Troubleshooting & Common Errors
Below you’ll find a list of common errors and their solutions. Most issues are resolved by just provisioning again (via
tools/provision.py inside the Vagrant guest or equivalently vagrant provision from outside).
If these solutions aren’t working for you or you encounter an issue not documented below, there are a few ways to get
further help:
• Ask in #provision help in the Zulip development community server,
• send a note to the Zulip-devel Google group, or
• File an issue.
When reporting your issue, please include the following information:
• host operating system
• installation method (Vagrant or direct)
• whether or not you are using a proxy
• a copy of Zulip’s vagrant provisioning logs, available in /var/log/provision.log on your virtual
machine. If you choose to post just the error output, please include the beginning of the error output, not just
the last few lines.
The output of tools/diagnose run inside the Vagrant guest is also usually helpful.
Vagrant guest doesn’t show (zulip-venv) at start of prompt
This is caused by provisioning failing to complete successfully. You can see the errors in var/log/provision.
log; it should end with something like this:
ESC[94mZulip development environment setup succeeded!ESC[0m
The ESC stuff are the terminal color codes that make it show as a nice blue in the terminal, which unfortunately looks
ugly in the logs.
If you encounter an incomplete /var/log/provision.log file, you need to update your environment. Reprovision your vagrant machine; if the problem persists, please come chat with us (see instructions above) for help.
After you provision successfully, you’ll need to exit your vagrant ssh shell and run vagrant ssh again to get
the virtualenv setup properly.
16.6. Next Steps
83
Zulip Documentation, Release 1.4.0
The box ‘ubuntu/trusty64’ could not be found
If you see the following error when you run vagrant up:
The box 'ubuntu/trusty64' could not be found or
could not be accessed in the remote catalog. If this is a private
box on HashiCorp's Atlas, please verify you're logged in via
`vagrant login`. Also, please double-check the name. The expanded
URL and error message are shown below:
URL: ["https://atlas.hashicorp.com/ubuntu/trusty64"]
Then the version of curl that ships with Vagrant is not working on your machine. You are most likely to encounter
this error on Windows/Cygwin and macOS.
On macOS this error is most likely to occur with Vagrant version 1.8.7 and is a known issue.
The solution is to downgrade Vagrant to version 1.8.6 (available here), or to use your system’s version of curl instead
of the one that ships with Vagrant:
sudo ln -nsf /usr/bin/curl /opt/vagrant/embedded/bin/curl
On Windows/Cygwin, the fix is simple: replace it with the version from Cygwin.
First, determine the location of Cygwin’s curl with which curl:
christie@win10 ~/zulip
$ which curl
/usr/bin/curl
Now determine the location of Vagrant with which vagrant:
christie@win10 ~/zulip
$ which vagrant
/cygdrive/c/HashiCorp/Vagrant/bin/vagrant
The path up until /bin/vagrant is what you need to know. In the example above it’s /cygdrive/c/
HashiCorp/Vagrant.
Finally, copy Cygwin’s curl to Vagrant embedded/bin directory:
christie@win10 ~/zulip
$ cp /usr/bin/curl.exe /cygdrive/c/HashiCorp/Vagrant/embedded/bin/
Now re-run vagrant up and vagrant should be able to fetch the required box file.
Vagrant was unable to mount VirtualBox shared folders
For the following error:
Vagrant was unable to mount VirtualBox shared folders. This is usually
because the filesystem "vboxsf" is not available. This filesystem is
made available via the VirtualBox Guest Additions and kernel
module. Please verify that these guest additions are properly
installed in the guest. This is not a bug in Vagrant and is usually
caused by a faulty Vagrant box. For context, the command attempted
was:
mount -t vboxsf -o uid=1000,gid=1000 keys /keys
84
Chapter 16. Vagrant environment setup tutorial
Zulip Documentation, Release 1.4.0
If this error starts happening unexpectedly, then just run:
vagrant reload
This is equivalent of running a halt followed by an up (aka rebooting the guest). After this, you can do vagrant
provision and vagrant ssh.
os.symlink error
If you receive the following error while running vagrant up:
==> default: Traceback (most recent call last):
==> default: File "./emoji_dump.py", line 75, in <module>
==> default:
==> default: os.symlink('unicode/{}.png'.format(code_point), 'out/{}.png'.
˓→format(name))
==> default: OSError
==> default: :
==> default: [Errno 71] Protocol error
Then Vagrant was not able to create a symbolic link.
First, if you are using Windows, make sure you have run Git BASH (or Cygwin) as an administrator. By default,
only administrators can create symbolic links on Windows.
Second, VirtualBox does not enable symbolic links by default. Vagrant starting with version 1.6.0 enables symbolic
links for VirtualBox shared folder.
You can check to see that this is enabled for your virtual machine with vboxmanage command.
Get the name of your virtual machine by running vboxmanage list vms and then print out the custom settings
for this virtual machine with vboxmanage getextradata YOURVMNAME enumerate:
christie@win10 ~/zulip
$ vboxmanage list vms
"zulip_default_1462498139595_55484" {5a65199d-8afa-4265-b2f6-6b1f162f157d}
christie@win10 ~/zulip
$ vboxmanage getextradata zulip_default_1462498139595_55484 enumerate
Key: VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip, Value: 1
Key: supported, Value: false
If you see “command not found” when you try to run VBoxManage, you need to add the VirtualBox directory to your
path. On Windows this is mostly likely C:\Program Files\Oracle\VirtualBox\.
If vboxmanage enumerate prints nothing,
or shows a value of 0 for VBoxInternal2/SharedFoldersEnableSymlinksCreate/srv_zulip, then enable symbolic links by running this command in
Terminal/Git BASH/Cygwin:
vboxmanage setextradata YOURVMNAME VBoxInternal2/SharedFoldersEnableSymlinksCreate/
˓→srv_zulip 1
The virtual machine needs to be shut down when you run this command.
Connection timeout on vagrant up
If you see the following error after running vagrant up:
16.7. Troubleshooting & Common Errors
85
Zulip Documentation, Release 1.4.0
default:
default:
default:
default:
default:
default:
SSH address: 127.0.0.1:2222
SSH username: vagrant
SSH auth method: private key
Error: Connection timeout. Retrying...
Error: Connection timeout. Retrying...
Error: Connection timeout. Retrying...
A likely cause is that hardware virtualization is not enabled for your computer. This must be done via your computer’s
BIOS settings. Look for a setting called VT-x (Intel) or (AMD-V).
If this is already enabled in your BIOS, double-check that you are running a 64-bit operating system.
For further information about troubleshooting vagrant timeout errors see this post.
Vagrant up fails with subprocess.CalledProcessError
The vagrant up command basically does the following:
• Downloads an Ubuntu image and starts it using a Vagrant provider.
• Uses vagrant ssh to connect to that Ubuntu guest, and then runs tools/provision, which has a lot
of subcommands that are executed via Python’s subprocess module. These errors mean that one of those
subcommands failed.
To debug such errors, you can log in to the Vagrant guest machine by running vagrant ssh, which should
present you with a standard shell prompt. You can debug interactively by using e.g. cd zulip && ./tools/
provision, and then running the individual subcommands that failed. Once you’ve resolved the problem, you can
rerun tools/provision to proceed; the provisioning system is designed to recover well from failures.
The zulip provisioning system is generally highly reliable; the most common cause of issues here is a poor network
connection (or one where you need a proxy to access the Internet and haven’t configured the development environment
to use it.
Once you’ve provisioned successfully, you’ll get output like this:
Zulip development environment setup succeeded!
(zulip-venv) vagrant@vagrant-base-trusty-amd64:~/zulip$
If the (zulip-venv) part is missing, this is because your installation failed the first time before the Zulip virtualenv
was created. You can fix this by just closing the shell and running vagrant ssh again, or using source /srv/
zulip-venv/bin/activate.
Finally, if you encounter any issues that weren’t caused by your Internet connection, please report them! We try hard
to keep Zulip development environment provisioning free of bugs.
pip install fails during vagrant up on Ubuntu
Likely causes are:
1. Networking issues
2. Insufficient RAM. Check whether you’ve allotted at least two gigabytes of RAM, which is the minimum Zulip
requires. If not, go to your VM settings and increase the RAM, then restart the VM.
86
Chapter 16. Vagrant environment setup tutorial
Zulip Documentation, Release 1.4.0
npm install errors
The tools/provision script may encounter an error related to npm install that looks something like:
==> default: + npm install
==> default: Traceback (most recent call last):
==> default:
File "/srv/zulip/tools/provision", line 195, in <module>
==> default:
==> default: sys.exit(main())
==> default:
File "/srv/zulip/tools/provision", line 191, in main
==> default:
==> default: run(["npm", "install"])
==> default:
File "/srv/zulip/scripts/lib/zulip_tools.py", line 78, in run
==> default:
==> default: raise subprocess.CalledProcessError(rc, args)
==> default: subprocess
==> default: .
==> default: CalledProcessError
==> default: :
==> default: Command '['npm', 'install']' returned non-zero exit status 34
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.
Usually this error is not fatal. Try connecting to the development environment and re-trying the command from withing
the virtual machine:
christie@win10 ~/zulip
$ vagrant ssh
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~
$ cd zulip
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~/zulip
$ npm install
npm WARN optional Skipping failed optional dependency /chokidar/fsevents:
npm WARN notsup Not compatible with your operating system or architecture: fsevents@1.
˓→0.12
These are just warnings so it is okay to proceed and start the Zulip server.
vagrant-lxc errors
Permissions errors
When building the development environment using Vagrant and the LXC provider, if you encounter permissions errors,
you may need to chown -R 1000:$(whoami) /path/to/zulip on the host before running vagrant up
in order to ensure that the synced directory has the correct owner during provision. This issue will arise if you run
id username on the host where username is the user running Vagrant and the output is anything but 1000. This
seems to be caused by Vagrant behavior; for more information, see the vagrant-lxc FAQ entry about shared folder
permissions.
NoMethodError
If you see the following error when you try to install the vagrant-lxc plugin:
16.7. Troubleshooting & Common Errors
87
Zulip Documentation, Release 1.4.0
/usr/lib/ruby/2.3.0/rubygems/specification.rb:946:in `all=': undefined method `group_
˓→by' for nil:NilClass (NoMethodError)
from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:275:in `with_isolated_gem'
from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:231:in `internal_install'
from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:102:in `install'
from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:62:in `block in install_
˓→plugin'
from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:72:in `install_plugin'
from /usr/share/vagrant/plugins/commands/plugin/action/install_gem.rb:37:in `call'
from /usr/lib/ruby/vendor_ruby/vagrant/action/warden.rb:34:in `call'
from /usr/lib/ruby/vendor_ruby/vagrant/action/builder.rb:116:in `call'
from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `block in run'
from /usr/lib/ruby/vendor_ruby/vagrant/util/busy.rb:19:in `busy'
from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `run'
from /usr/share/vagrant/plugins/commands/plugin/command/base.rb:14:in `action'
from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:32:in `block in
˓→execute'
from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in `each'
from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in `execute'
from /usr/share/vagrant/plugins/commands/plugin/command/root.rb:56:in `execute'
from /usr/lib/ruby/vendor_ruby/vagrant/cli.rb:42:in `execute'
from /usr/lib/ruby/vendor_ruby/vagrant/environment.rb:268:in `cli'
from /usr/bin/vagrant:173:in `<main>'
And you have vagrant version 1.8.1, then you need to patch vagrant manually. See this post for an explanation of the
issue, which should be fixed when Vagrant 1.8.2 is released.
In the meantime, read this post for how to create and apply the patch.
It will look something like this:
christie@xenial:~
$ sudo patch --directory /usr/lib/ruby/vendor_ruby/vagrant < vagrant-plugin.patch
patching file bundler.rb
Permissions errors when running the test suite in LXC
See “Possible testing issues”.
Specifying a proxy
If you need to use a proxy server to access the Internet, you will need to specify the proxy settings before running
Vagrant up. First, install the Vagrant plugin vagrant-proxyconf:
vagrant plugin install vagrant-proxyconf.
Then create ~/.zulip-vagrant-config and add the following lines to it (with the appropriate values in it for
your proxy):
HTTP_PROXY http://proxy_host:port
HTTPS_PROXY http://proxy_host:port
NO_PROXY localhost,127.0.0.1,.example.com
Now run vagrant up in your terminal to install the development server. If you ran vagrant up before and failed,
you’ll need to run vagrant destroy first to clean up the failed installation.
88
Chapter 16. Vagrant environment setup tutorial
Zulip Documentation, Release 1.4.0
You can also change the port on the host machine that Vagrant uses by adding to your ~/.
zulip-vagrant-config file. E.g. if you set:
HOST_PORT 9971
(and halt and restart the Vagrant guest), then you would visit http://localhost:9971/ to connect to your development
server.
If you’d like to be able to connect to your development environment from other machines than the VM host, you can
manually set the host IP address in the ‘~/.zulip-vagrant-config’ file as well. For example, if you set:
HOST_IP_ADDR 0.0.0.0
(and restart the Vagrant guest), your host IP would be 0.0.0.0, a special value for the IP address that means any IP
address can connect to your development server.
16.8. Specifying a proxy
89
Zulip Documentation, Release 1.4.0
90
Chapter 16. Vagrant environment setup tutorial
CHAPTER
17
Zulip development environment setup without Vagrant
Contents:
• Installing directly on Ubuntu
• Installing manually on Linux
• Using Docker (experimental)
Installing directly on Ubuntu
Start by cloning this repository: git clone https://github.com/zulip/zulip.git
If you’d like to install a Zulip development environment on a computer that’s already running Ubuntu 14.04 Trusty or
Ubuntu 16.04 Xenial, you can do that by just running:
# From a clone of zulip.git
./tools/provision
source /srv/zulip-venv/bin/activate
./tools/run-dev.py # starts the development server
Note that there is no supported uninstallation process without Vagrant (with Vagrant, you can just do vagrant
destroy to clean up the development environment).
Once you’ve done the above setup, you can pick up the documentation on using the Zulip development environment,
ignoring the parts about vagrant (since you’re not using it).
Installing manually on Linux
• Debian or Ubuntu systems
• Fedora 22 (experimental)
• CentOS 7 Core (experimental)
91
Zulip Documentation, Release 1.4.0
• OpenBSD 5.8 (experimental)
• Fedora/CentOS common steps
• Steps for all systems
If you really want to install everything manually, the below instructions should work.
Install the following non-Python dependencies:
• libffi-dev — needed for some Python extensions
• postgresql 9.1 or later — our database (client, server, headers)
• nodejs 0.10 (and npm)
• memcached (and headers)
• rabbitmq-server
• libldap2-dev
• python-dev
• python3-dev
• python-virtualenv
• redis-server — rate limiting
• tsearch-extras — better text search
• libfreetype6-dev — needed before you pip install Pillow to properly generate emoji PNGs
On Debian or Ubuntu systems:
Using the official Ubuntu repositories, PGroonga PPA and tsearch-extras deb package:
Start by cloning this repository: git clone https://github.com/zulip/zulip.git
sudo apt-get install closure-compiler libfreetype6-dev libffi-dev \
memcached rabbitmq-server libldap2-dev redis-server \
postgresql-server-dev-all libmemcached-dev python-dev \
python3-dev python-virtualenv hunspell-en-us nodejs \
nodejs-legacy npm git yui-compressor puppet gettext postgresql
# If
sudo
sudo
# On
sudo
# On
sudo
using Ubuntu, install PGroonga from its PPA
add-apt-repository -ys ppa:groonga/ppa
apt-get update
14.04
apt-get install postgresql-9.3-pgroonga
16.04
apt-get install postgresql-9.5-pgroonga
# If using Debian, follow the instructions here: http://pgroonga.github.io/install/
˓→debian.html
# Next, install Zulip's tsearch-extras postgresql extension
# If on 14.04 or 16.04, you can use the Zulip PPA for tsearch-extras:
cd zulip
sudo apt-add-repository -ys ppa:tabbott/zulip
sudo apt-get update
# On 14.04
92
Chapter 17. Zulip development environment setup without Vagrant
Zulip Documentation, Release 1.4.0
sudo apt-get install postgresql-9.3-tsearch-extras
# On 16.04
sudo apt-get install postgresql-9.5-tsearch-extras
# Otherwise, you can download a .deb directly
# If on 12.04 or wheezy:
wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.1-tsearch˓→extras_0.1.2_amd64.deb
sudo dpkg -i postgresql-9.1-tsearch-extras_0.1.2_amd64.deb
# If on 14.04:
wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/postgresql-9.3˓→tsearch-extras_0.1.3_amd64.deb
sudo dpkg -i postgresql-9.3-tsearch-extras_0.1.3_amd64.deb
# If on 15.04 or jessie:
wget https://dl.dropboxusercontent.com/u/283158365/zuliposs/postgresql-9.4-tsearch˓→extras_0.1_amd64.deb
sudo dpkg -i postgresql-9.4-tsearch-extras_0.1_amd64.deb
# If on 16.04 or stretch
wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/postgresql-9.5˓→tsearch-extras_0.2_amd64.deb
sudo dpkg -i postgresql-9.5-tsearch-extras_0.2_amd64.deb
Alternatively, you can always build the package from tsearch-extras git.
Now continue with the All Systems instructions below.
Using the official Zulip PPA (for 14.04 Trusty or 16.04 Xenial):
Start by cloning this repository: git clone https://github.com/zulip/zulip.git
sudo add-apt-repository ppa:tabbott/zulip
sudo apt-get update
sudo apt-get install closure-compiler libfreetype6-dev libffi-dev \
memcached rabbitmq-server libldap2-dev redis-server \
postgresql-server-dev-all libmemcached-dev python-dev \
hunspell-en-us nodejs nodejs-legacy npm git yui-compressor \
puppet gettext tsearch-extras
Now continue with the All Systems instructions below.
On Fedora 22 (experimental):
These instructions are experimental and may have bugs; patches welcome!
Start by cloning this repository: git clone https://github.com/zulip/zulip.git
sudo dnf install libffi-devel memcached rabbitmq-server \
openldap-devel python-devel redis postgresql-server \
postgresql-devel postgresql libmemcached-devel freetype-devel \
nodejs npm yuicompressor closure-compiler gettext
Now continue with the Common to Fedora/CentOS instructions below.
17.2. Installing manually on Linux
93
Zulip Documentation, Release 1.4.0
On CentOS 7 Core (experimental):
These instructions are experimental and may have bugs; patches welcome!
Start by cloning this repository: git clone https://github.com/zulip/zulip.git
# Add user zulip to the system (not necessary if you configured zulip
# as the administrator user during the install process of CentOS 7).
useradd zulip
# Create a password for zulip user
passwd zulip
# Allow zulip to sudo
visudo
# Add this line after line `root
zulip
ALL=(ALL)
ALL
ALL=(ALL)
ALL`
# Switch to zulip user
su zulip
# Enable EPEL 7 repo so we can install rabbitmq-server, redis and
# other dependencies
sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.
˓→rpm
# Install dependencies
sudo yum install libffi-devel memcached rabbitmq-server openldap-devel \
python-devel redis postgresql-server postgresql-devel postgresql \
libmemcached-devel wget python-pip openssl-devel freetype-devel \
libjpeg-turbo-devel zlib-devel nodejs yuicompressor \
closure-compiler gettext
# We need these packages to compile tsearch-extras
sudo yum groupinstall "Development Tools"
# clone Zulip's git repo and cd into it
cd && git clone https://github.com/zulip/zulip && cd zulip/
## NEEDS TESTING: The next few DB setup items may not be required at all.
# Initialize the postgres db
sudo postgresql-setup initdb
# Edit the postgres settings:
sudo vi /var/lib/pgsql/data/pg_hba.conf
# Change these lines:
host
all
host
all
# to this:
host
all
host
all
all
all
127.0.0.1/32
::1/128
ident
ident
all
all
127.0.0.1/32
::1/128
md5
md5
Now continue with the Common to Fedora/CentOS instructions below.
On OpenBSD 5.8 (experimental):
These instructions are experimental and may have bugs; patches welcome!
94
Chapter 17. Zulip development environment setup without Vagrant
Zulip Documentation, Release 1.4.0
Start by cloning this repository: git clone https://github.com/zulip/zulip.git
doas pkg_add sudo bash gcc postgresql-server redis rabbitmq \
memcached node libmemcached py-Pillow py-cryptography py-cffi
# Get tsearch_extras and build it (using a modified version which
# aliases int4 on OpenBSD):
git clone https://github.com/blablacio/tsearch_extras
cd tsearch_extras
gmake && sudo gmake install
# Point environment to custom include locations and use newer GCC
# (needed for Node modules):
export CFLAGS="-I/usr/local/include -I/usr/local/include/sasl"
export CXX=eg++
# Create tsearch_data directory:
sudo mkdir /usr/local/share/postgresql/tsearch_data
# Hack around missing dictionary files -- need to fix this to get the
# proper dictionaries from what in debian is the hunspell-en-us
# package.
sudo touch /usr/local/share/postgresql/tsearch_data/english.stop
sudo touch /usr/local/share/postgresql/tsearch_data/en_us.dict
sudo touch /usr/local/share/postgresql/tsearch_data/en_us.affix
Finally continue with the All Systems instructions below.
Common to Fedora/CentOS instructions
Start by cloning this repository: git clone https://github.com/zulip/zulip.git
# Build and install postgres tsearch-extras module
wget https://launchpad.net/~tabbott/+archive/ubuntu/zulip/+files/tsearch-extras_0.1.3.
˓→tar.gz
tar xvzf tsearch-extras_0.1.3.tar.gz
cd ts2
make
sudo make install
# Hack around missing dictionary files -- need to fix this to get the
# proper dictionaries from what in debian is the hunspell-en-us
# package.
sudo touch /usr/share/pgsql/tsearch_data/english.stop
sudo touch /usr/share/pgsql/tsearch_data/en_us.dict
sudo touch /usr/share/pgsql/tsearch_data/en_us.affix
# Edit the postgres settings:
sudo vi /var/lib/pgsql/data/pg_hba.conf
# Add this line before the first uncommented line to enable password
# auth:
host
all
all
127.0.0.1/32
md5
# Start the services
sudo systemctl start redis memcached rabbitmq-server postgresql
17.2. Installing manually on Linux
95
Zulip Documentation, Release 1.4.0
# Enable automatic service startup after the system startup
sudo systemctl enable redis rabbitmq-server memcached postgresql
Finally continue with the All Systems instructions below.
All Systems:
Make sure you have followed the steps specific for your platform:
• Debian or Ubuntu systems
• Fedora 22 (experimental)
• CentOS 7 Core (experimental)
• OpenBSD 5.8 (experimental)
• Fedora/CentOS
For managing Zulip’s python dependencies, we recommend using virtualenvs.
You must create two virtualenvs. One for Python 2 and one for Python 3. You must also install appropriate python
packages in them.
You should either install the virtualenvs in /srv, or put symlinks to them in /srv. If you don’t do that, some scripts
might not work correctly.
You can run tools/setup/setup_venvs.py to do this. This script will create two virtualenvs - /srv/zulip-venv
and /srv/zulip-py3-venv.
If you want to do it manually, here are the steps:
sudo virtualenv /srv/zulip-venv -p python2 # Create a python2 virtualenv
sudo chown -R `whoami`:`whoami` /srv/zulip-venv
source /srv/zulip-venv/bin/activate # Activate python2 virtualenv
pip install --upgrade pip # upgrade pip itself because older versions have known
˓→issues
pip install --no-deps -r requirements/py2_dev.txt # install python packages required
˓→for development
sudo virtualenv /srv/zulip-py3-venv -p python3 # Create a python3 virtualenv
sudo chown -R `whoami`:`whoami` /srv/zulip-py3-venv
source /srv/zulip-py3-venv/bin/activate # Activate python3 virtualenv
pip install --upgrade pip # upgrade pip itself because older versions have known
˓→issues
pip install --no-deps -r requirements/py3_dev.txt # install python packages required
˓→for development
Now run these commands:
./tools/install-mypy
sudo mkdir /srv/zulip-emoji-cache
sudo chown -R `whoami`:`whoami` /srv/zulip-emoji-cache
./tools/setup/emoji/download-emoji-data
./tools/setup/emoji/build_emoji
./tools/setup/build_pygments_data.py
./scripts/setup/generate_secrets.py --development
if [ $(uname) = "OpenBSD" ]; then
sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /var/postgresql/
˓→tsearch_data/
96
Chapter 17. Zulip development environment setup without Vagrant
Zulip Documentation, Release 1.4.0
else
sudo cp ./puppet/zulip/files/postgresql/zulip_english.stop /usr/share/postgresql/
˓→9.*/tsearch_data/
fi
./scripts/setup/configure-rabbitmq
./tools/setup/postgres-init-dev-db
./tools/do-destroy-rebuild-database
./tools/setup/postgres-init-test-db
./tools/do-destroy-rebuild-test-database
./manage.py compilemessages
sudo ./scripts/lib/install-node
npm install
To start the development server:
./tools/run-dev.py
. . . and visit http://localhost:9991/.
Proxy setup for by-hand installation
If you are building the development environment on a network where a proxy is required to access the Internet, you
will need to set the proxy in the environment as follows:
• On Ubuntu, set the proxy environment variables using:
export https_proxy=http://proxy_host:port
export http_proxy=http://proxy_host:port
• And set the npm proxy and https-proxy using:
npm config set proxy http://proxy_host:port
npm config set https-proxy http://proxy_host:port
Using Docker (experimental)
Start by cloning this repository: git clone https://github.com/zulip/zulip.git
The docker instructions for development are experimental, so they may have bugs. If you try them and run into any
issues, please report them!
You can also use Docker to run a Zulip development environment. First, you need to install Docker in your development machine following the instructions. Some other interesting links for somebody new in Docker are:
• Get Started
• Understand the architecture
• Docker run reference
• Dockerfile reference
Then you should create the Docker image based on Ubuntu Linux, first go to the directory with the Zulip source code:
docker build -t user/zulipdev .
17.3. Using Docker (experimental)
97
Zulip Documentation, Release 1.4.0
Commit and tag the provisioned images. The below will install Zulip’s dependencies:
docker run -itv $(pwd):/srv/zulip -p 9991:9991 user/zulipdev /bin/bash
$ /bin/bash sudo chown -R zulip:zulip /srv/zulip
$ /bin/bash /srv/zulip/tools/provision --docker
docker ps -af ancestor=user/zulipdev
docker commit -m "Zulip installed" <container id> user/zulipdev:v2
Now you can run the docker server with:
docker run -itv $(pwd):/srv/zulip -p 9991:9991 user/zulipdev:v2 \
/srv/zulip/tools/start-dockers
You’ll want to read the guide for Zulip development to understand how to use the Zulip development. Note
that start-dockers automatically runs tools/run-dev.py inside the container; you can then visit
http://localhost:9991 to connect to your new Zulip Docker container.
To view the container’s run-dev.py console logs to get important debugging information (and e.g. outgoing emails)
printed by the Zulip development environment, you can use:
docker logs --follow <container id>
To restart the server use:
docker ps
docker restart <container id>
To stop the server use:
docker ps
docker kill <container id>
If you want to connect to the Docker instance to run commands (e.g. build a release tarball), you can use:
docker ps
docker exec -it <container id> /bin/bash
$ source /home/zulip/.bash_profile
$ <Your commands>
$ exit
If you want to run all the tests you need to start the servers first, you can do it with:
docker run -itv $(pwd):/srv/zulip user/zulipdev:v2 /bin/bash
$ tools/test-all-docker
You can modify the source code in your development machine and review the results in your browser.
Currently, the Docker workflow is substantially less convenient than the Vagrant workflow and less documented; please
contribute to this guide and the Docker tooling if you are using Docker to develop Zulip!
98
Chapter 17. Zulip development environment setup without Vagrant
CHAPTER
18
Using the Development Environment
Once the development environment is running, you can visit http://localhost:9991/ in your browser. By default, the
development server homepage just shows a list of the users that exist on the server and you can login as any of them by
just clicking on a user. This setup saves time for the common case where you want to test something other than the login
process; to test the login process you’ll want to change AUTHENTICATION_BACKENDS in the not-PRODUCTION
case of zproject/settings.py from zproject.backends.DevAuthBackend to use the auth method(s) you’d like
to test.
While developing, it’s helpful to watch the run-dev.py console output, which will show any errors your Zulip
development server encounters.
To manually query the Postgres database, run psql zulip for an interactive console.
When you make a change, here’s a guide for what you need to do in order to see your change take effect in Development:
• If you change JavaScript, CSS, or Jinja2 backend templates (under templates/), you’ll just need to reload
the browser window to see changes take effect. The Handlebars frontend HTML templates (static/
templates) are automatically recompiled by the tools/compile-handlebars-templates job,
which runs as part of tools/run-dev.py.
• If you change Python code used by the main Django/Tornado server processes, these services are run on top of
Django’s manage.py runserver which will automatically restart the Zulip Django and Tornado servers whenever
you save changes to Python code. You can watch this happen in the run-dev.py console to make sure the
backend has reloaded.
• The Python queue workers will also automatically restart when you save changes. However, you may need to
ctrl-C and then restart run-dev.py manually if a queue worker has crashed.
• If you change the database schema, you’ll need to use the standard Django migrations process to create and then
run your migrations; see the new feature tutorial for an example. Additionally you should check out the detailed
testing docs for how to run the tests properly after doing a migration.
(In production, everything runs under supervisord and thus will restart if it crashes, and upgrade-zulip will take
care of running migrations and then cleanly restaring the server for you).
99
Zulip Documentation, Release 1.4.0
100
Chapter 18. Using the Development Environment
CHAPTER
19
Developing on a remote machine
The Zulip developer environment works well on remote virtual machines. This can be a good alternative for those
with poor network connectivity or who have limited storage/memory on their local machines.
We recommend giving the Zulip development environment its own virtual machine, running Ubuntu 14.04 or 16.04,
with at least 2GB of memory. If the Zulip development environment will be the only thing running on the remote
virtual machine, we recommend installing directly. Otherwise, we recommend the Vagrant method so you can easily
uninstall if you need to.
Connecting to the remote environment
The best way to connect to your server is using the command line tool ssh.
• On macOS and Linux/UNIX, ssh is a part of Terminal.
• On Windows, ssh comes with Bash for Git.
Open Terminal or Bash for Git, and connect with the following:
$ ssh username@host
If you have poor internet connectivity, we recommend using Mosh as it is more reliable over slow or unreliable
networks.
Setting up the development environment
After you have connected to your remote server, you need to install the development environment.
If the Zulip development environment will be the only thing running on the remote virtual machine, we recommend
installing directly. Otherwise, we recommend the Vagrant method so you can easily uninstall if you need to.
101
Zulip Documentation, Release 1.4.0
Running the development server
Once you have set up the development environment, you can start up the development instance of Zulip with the
following command in the directory where you cloned Zulip:
./tools/run-dev.py --interface=''
This will start up the Zulip server on port 9991. You can then navigate to http://<REMOTE_IP>:9991 and you
should see something like this screenshot of the Zulip development environment:
You can port forward using ssh instead of running the development environment on an exposed interface.
For more information, see Using the development environment.
102
Chapter 19. Developing on a remote machine
Zulip Documentation, Release 1.4.0
Making changes to code on your remote development server
To see changes on your remote development server, you need to do one of the following:
• Edit locally: Clone Zulip code to your computer and then use your favorite editor to make changes. When you
want to see changes on your remote Zulip development instance, sync with Git.
• Edit remotely: Edit code directly on your remote Zulip development instance using a Web-based IDE (recommended for beginners) or a command line editor.
Editing locally
If you want to edit code locally install your favorite text editor. If you don’t have a favorite, here are some suggestions:
• atom
• emacs
• vim
• spacemacs
• sublime
Next, follow our Git and GitHub Guide to clone and configure your fork of zulip on your local computer.
Once you have cloned your code locally, you can get to work.
Syncing changes
The easiest way to see your changes on your remote development server is to push them to GitHub and them fetch
and merge them from the remote server.
For more detailed instructions about how to do this, see our Git & GitHub Guide. In brief, the steps are as follows.
On your local computer:
1. Open Terminal (macOS/Linux) or Git for BASH.
2. Change directory to where you cloned Zulip (e.g. cd zulip).
3. Use git add and git commit to stage and commit your changes (if you haven’t already).
4. Push your commits to GitHub with git push origin branchname.
Be sure to replace branchname with the name of your actual feature branch.
Once git push has completed successfully, you are ready to fetch the commits from your remote development
instance:
1. In Terminal or Git BASH, connect to your remote development instance with ssh user@host.
2. Change to the zulip directory (e.g., cd zulip).
3. Fetch new commits from GitHub with git fetch origin.
4. Change to the branch you want to work on with git checkout branchname.
5. Merge the new commits into your branch with git merge origin/branchname.
19.4. Making changes to code on your remote development server
103
Zulip Documentation, Release 1.4.0
Editing remotely
Web-based IDE
If you are relatively new to working on the command line, or just want to get started working quickly, we recommend
web-based IDE Codeanywhere.
To setup Codeanywhere for Zulip:
1. Create a Codeanywhere account and log in.
2. Create a new SFTP-SSH project. Use Public key for authentication.
3. Click GET YOUR PUBLIC KEY to get the new public key that Codeanywhere generates when you create a
new project. Add this public key to ~/.ssh/authorized_keys on your remote development instance.
4. Once you’ve added the new public key to your remote development instance, click CONNECT.
Now your workspace should look similar this:
Command line editors
Another way to edit directly on the remote development server is with a command line text editor on the remote
machine.
Two editors often available by default on Linux systems are:
• Nano: A very simple, beginner-friendly editor. However, it lacks a lot of features useful for programming, such
as syntax highlighting, so we only recommended it for quick edits to things like configuration files. Launch by
running command nano <filename>. Exit by pressing control-X.
104
Chapter 19. Developing on a remote machine
Zulip Documentation, Release 1.4.0
• Vim: A very powerful editor that can take a while to learn. Launch by running vim <filename>. Quit
Vim by pressing escape, typing :q, and then pressing return. Vim comes with a program to learn it called
vimtutor (just run that command to start it).
Other options include:
• emacs
• spacemacs
Next steps
Next, read the following to learn more about developing for Zulip:
• Git & GitHub Guide
• Using the Development Environment
• Testing
19.4. Making changes to code on your remote development server
105
Zulip Documentation, Release 1.4.0
106
Chapter 19. Developing on a remote machine
CHAPTER
20
Writing a new integration
Integrations are one of the most important parts of a group chat tool like Zulip, and we are committed to making
integrating with Zulip and getting you integration merged upstream so everyone else can benefit from it as easy as
possible while maintaining the high quality of the Zulip integrations library.
On this page you’ll find:
• An overview of the different types of integrations possible with Zulip.
• General advice for writing integrations.
• Details about writing webhook integrations.
• Details about writing Python script and plugin integrations.
• A guide to documenting your integration.
A detailed walkthrough of a simple “Hello World” integration can be found in the webhook walkthrough.
Contributions to this guide are very welcome, so if you run into any issues following these instructions or come up
with any tips or tools that help writing integration, please email zulip-devel@googlegroups.com, open an issue, or
submit a pull request to share your ideas!
Types of integrations
We have several different ways that we integrate with 3rd party products, ordered here by which types we prefer to
write:
1. Webhook integrations (examples: Freshdesk, GitHub), where the third-party service supports posting content
to a particular URI on our site with data about the event. For these, you usually just need to create a new python
package in the zerver/webhooks/ directory. You can easily find recent commits adding new integrations
to crib from via git log zerver/webhooks/.
2. Python script integrations (examples: SVN, Git), where we can get the service to call our integration (by
shelling out or otherwise), passing in the required data. Our preferred model for these is to ship these integrations
in our API release tarballs (by writing the integration in api/integrations).
107
Zulip Documentation, Release 1.4.0
3. Plugin integrations (examples: Jenkins, Hubot, Trac) where the user needs to install a plugin into their existing
software. These are often more work, but for some products are the only way to integrate with the product at all.
General advice
• Consider using our Zulip markup to make the output from your integration especially attractive or useful (e.g.
emoji, markdown emphasis, @-mentions, or !avatar(email)).
• Use topics effectively to ensure sequential messages about the same thing are threaded together; this makes for
much better consumption by users. E.g. for a bug tracker integration, put the bug number in the topic for all
messages; for an integration like Nagios, put the service in the topic.
• Integrations that don’t match a team’s workflow can often be uselessly spammy. Give careful thought to providing options for triggering Zulip messages only for certain message types, certain projects, or sending different
messages to different streams/topics, to make it easy for teams to configure the integration to support their
workflow.
• Consistently capitalize the name of the integration in the documentation and the Client name the way the vendor
does. It’s OK to use all-lower-case in the implementation.
• Sometimes it can be helpful to contact the vendor if it appears they don’t have an API or webhook we can use –
sometimes the right API is just not properly documented.
• A helpful tool for testing your integration is UltraHook, which allows you to receive webhook calls via your
local Zulip development environment. This enables you to do end-to-end testing with live data from the service
you’re integrating and can help you spot why something isn’t working or if the service is using custom HTTP
headers.
Webhook integrations
A webhook allows a third-party service to push data to you when something happens. It’s different from making a
REST API call, where you send a request to the service’s API and wait for a response. With a webhook, the thirdparty service sends you an HTTP POST when it has something for you. Your webhook integration defines the URI the
service uses to communicate with Zulip, and handles that incoming data.
New Zulip webhook integrations can take just a few hours to write, including tests and documentation, if you use the
right process. Here’s how we recommend doing it:
• First, use http://requestb.in/ or a similar site to capture an example webhook payload from the service you’re
integrating. You can use these captured payloads to create a set of test fixtures for your integration under
zerver/webhooks/mywebhook/fixtures/.
• Then write a draft webhook handler under zerver/webhooks/; there are a lot of examples in that directory.
We recommend templating off a short one (like stash or zendesk), since the longer ones usually just have
more complex parsing which can obscure what’s common to all webhook integrations. In addition to writing
the integration itself, you’ll need to create Integration object and add it to WEBHOOK_INTEGRATIONS in
zerver/lib/integrations.py; search for webhook in that file to find the existing ones (and please
add yours in the alphabetically correct place).
• Then write a test for your fixture in the tests.py file in the zerver/webhooks/mywebhook directory.
You can now iterate on debugging the tests and webhooks handler until they work, all without ever needing to
post directly from the service you’re integrating with to your Zulip development machine. You can run just the
tests for one integration like this:
108
Chapter 20. Writing a new integration
Zulip Documentation, Release 1.4.0
test-backend zerver/webhooks/pagerduty/
See this guide for more details on the Zulip test runner.
• Once you’ve gotten your webhook working and passing a test, capture payloads for the other common types
of posts the service’s webhook will make, and add tests for them; usually this part of the process is pretty fast.
Webhook integration tests should all use fixtures (as opposed to contacting the service), since otherwise the tests
can’t run without Internet access and some sort of credentials for the service.
• Finally, write documentation for the integration; there’s a detailed guide below.
See the “Hello World” example in the webhook walkthrough for a detailed look at how to write a simple webhook.
Files that need to be created
Select a name for your webhook and use it consistently. The examples below are for a webhook named ‘MyWebHook’.
• static/images/integrations/logos/mywebhook.png: An image to represent your integration
in the user interface. Generally this Should be the logo of the platform/server/product you are integrating. See
Documenting your integration for details.
• static/images/integrations/mywebbook/001.png: A screen capture of your integration for use
in the user interface. You can add as many images as needed to effectively document your webhook integration.
See Documenting your integration for details.
• zerver/webhooks/mywebhook/fixtures/messagetype.json: Sample json payload data used by
tests. Add one fixture file per type of message supported by your integration. See Testing and writing tests for
details.
• zerver/webhooks/mywebhook/__init__.py: Empty file that is obligatory part of every python package. Remember to git add it.
• zerver/webhooks/mywebhook/view.py: Includes the main webhook integration function including
any needed helper functions.
• zerver/webhooks/mywebhook/tests.py: Add tests for your webbook. See Testing and writing tests
for details.
• zerver/webhooks/mywebhook/doc.html: Add end-user documentation. See Documenting your integration for details.
Files that need to be updated
• zerver/lib/integrations.py: Add your integration to WEBHOOK_INTEGRATIONS to register it.
This will automatically register a url for the webhook of the form api/v1/external/mywebhook and
associate with the function called api_mywebhook_webhook in zerver/webhooks/mywebhook/
view.py.
Python script and plugin integrations
For plugin integrations, usually you will need to consult the documentation for the third party software in order to
learn how to write the integration. But we have a few notes on how to do these:
• You should always send messages by POSTing to URLs of the form https://zulip.example.com/v1/
messages/.
20.4. Python script and plugin integrations
109
Zulip Documentation, Release 1.4.0
• We usually build Python script integration with (at least) 2 files: zulip_foo_config.py containing the
configuration for the integration including the bots’ API keys, plus a script that reads from this configuration to
actually do the work (that way, it’s possible to update the script without breaking users’ configurations).
• Be sure to test your integration carefully and document how to install it (see notes on documentation below).
• You should specify a clear HTTP User-Agent for your integration. The user agent should at a minimum identify
the integration and version number, separated by a slash. If possible, you should collect platform information
and include that in ()s after the version number. Some examples of ideal UAs are:
ZulipDesktop/0.7.0 (Ubuntu; 14.04)
ZulipJenkins/0.1.0 (Windows; 7.2)
ZulipMobile/0.5.4 (Android; 4.2; maguro)
Documenting your integration
Every Zulip integration must be documented in zerver/webhooks/mywebhook/doc.html (or templates/
zerver/integrations/<integration_name>.html, for non-webhook integrations). Usually, this involves a few steps:
• Add text explaining all of the steps required to setup the integration, including what URLs to use, etc. If there
are any screens in the product involved, take a few screenshots with the input fields filled out with sample values
in order to make the instructions really easy to follow. For the screenshots, use a bot with a name like “GitHub
Bot”, an email address for the bot like github-bot@zulip.example.com, and an obviously fake API
key like abcdef123456790.
• Make sure you’ve added your integration to zerver/lib/integrations.py; this results in your integration appearing on the /integrations page.
• You’ll need to add a SVG graphic of your integration’s logo under the static/images/integrations/
logos/<name>.svg, where <name> is the name of the integration, all in lower case; you can usually find
them in the product branding or press page. Make sure to optimize the SVG graphic by running svgo -f
path-to-file.
If you cannot find a SVG graphic of the logo, please find and include a PNG image of the logo instead.
• Finally, generate a message sent by the integration and take a screenshot of the message to provide an example
message in the documentation. If your new integration is a webhook integration, you can generate such a
message from your test fixtures using send_webhook_fixture_message:
./manage.py send_webhook_fixture_message \
--fixture=zerver/webhooks/pingdom/fixtures/imap_down_to_up.json \
'--url=/api/v1/external/pingdom?stream=stream_name&api_key=api_key'
When generating the screenshot of a sample message, give your test bot a nice name like “GitHub Bot”, use the
project’s logo as the bot’s avatar, and take the screenshots showing the stream/topic bar for the message, not just
the message body.
When writing documentation for your integration, be sure to use the {{ external_api_uri }} template
variable, so that your integration documentation will provide the correct URL for whatever server it is deployed
on. If special configuration is required to set the SITE variable, you should document that too, inside an {% if
api_site_required %} check.
110
Chapter 20. Writing a new integration
CHAPTER
21
Webhook walkthrough
Below explains each part of a simple webhook integration, called Hello World. This webhook sends a “hello” message
to the test stream and includes a link to the Wikipedia article of the day, which it formats from json data it receives
in the http request.
Use this walkthrough to learn how to write your first webhook integration.
Step 0: Create fixtures
The first step in creating a webhook is to examine the data that the service you want to integrate will be sending to
Zulip.
You can use http://requestb.in or a similar tool to capture webhook payload(s) from the service you are integrating.
Examining this data allows you to do two things:
1. Determine how you will need to structure your webhook code, including what message types your integration
should support and how; and,
2. Create fixtures for your webhook tests.
A test fixture is a small file containing test data, one for each test. Fixtures enable the testing of webhook integration
code without the need to actually contact the service being integrated.
Because Hello World is a very simple webhook that does one thing, it requires only one fixture, zerver/
webhooks/helloworld/fixtures/hello.json:
{
"featured_title":"Marilyn Monroe",
"featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe",
}
When writing your own webhook integration, you’ll want to write a test function for each distinct message condition
your webhook supports. You’ll also need a corresponding fixture for each of these tests. Depending on the type of
data the 3rd party service sends, your fixture may contain JSON, URL encoded text, or some other kind of data. See
Step 4: Create tests or Testing for further details.
111
Zulip Documentation, Release 1.4.0
Step 1: Initialize your webhook python package
In the zerver/webhooks/ directory, create new subdirectory that will contain all of corresponding code. In our
example it will be helloworld. The new directory will be a python package, so you have to create an empty
__init__.py file in that directory via e.g. touch zerver/webhooks/helloworld/__init__.py.
Step 2: Create main webhook code
The majority of the code for your webhook integration will be in a single python file, zerver/webhooks/
mywebhook/view.py.
The Hello World integration is in zerver/webhooks/helloworld/view.py:
from
from
from
from
from
from
__future__ import absolute_import
django.utils.translation import ugettext as _
zerver.lib.actions import check_send_message
zerver.lib.response import json_success, json_error
zerver.decorator import REQ, has_request_variables, api_key_only_webhook_view
zerver.lib.validator import check_dict, check_string
from zerver.models import Client, UserProfile
from django.http import HttpRequest, HttpResponse
from typing import Dict, Any, Iterable, Optional, Text
@api_key_only_webhook_view('HelloWorld')
@has_request_variables
def api_helloworld_webhook(request, user_profile,
payload=REQ(argument_type='body'),
stream=REQ(default='test'),
topic=REQ(default='Hello World')):
# type: (HttpRequest, UserProfile, Dict[str, Iterable[Dict[str, Any]]], Text,
˓→Optional[Text]) -> HttpResponse
# construct the body of the message
body = 'Hello! I am happy to be here! :smile:'
# try to add the Wikipedia article of the day
# return appropriate error if not successful
try:
body_template = '\nThe Wikipedia featured article for today is **[{featured_
˓→title}]({featured_url})**'
body += body_template.format(**payload)
except KeyError as e:
return json_error(_("Missing key {} in JSON").format(str(e)))
# send the message
check_send_message(user_profile, request.client, 'stream',
[stream], topic, body)
# return json result
return json_success()
The above code imports the required functions and defines the main webhook function
api_helloworld_webhook,
decorating
it
with
api_key_only_webhook_view
and
112
Chapter 21. Webhook walkthrough
Zulip Documentation, Release 1.4.0
has_request_variables. The has_request_variables decorator allows you to access request
variables with REQ(). You can find more about REQ and request variables in Writing views.
You must pass the name of your webhook to the api_key_only_webhook_view decorator so your webhook can
access the user_profile and request.client (Zulip’s analogue of UserAgent) fields from the request. Here
we have used HelloWorld. To be consistent with Zulip code style, use the name of the product you are integrating
in camel case, spelled as the product spells its own name (except always first letter upper-case).
The api_key_only_webhook_view decorator indicates that the 3rd party service will send the authorization
as an API key in the query parameters. If your service uses HTTP Basic authentication, you would instead use the
authenticated_rest_api_view decorator.
You should name your webhook function as such api_webhookname_webhook where webhookname is the
name of your webhook and is always lower-case.
At minimum, the webhook function must accept request (Django HttpRequest object), and user_profile
(Zulip’s user object). You may also want to define additional parameters using the REQ object.
In the example above, we have defined payload which is populated from the body of the http request, stream with
a default of test (available by default in the Zulip development environment), and topic with a default of Hello
World. If your webhook uses a custom stream, it must exist before a message can be created in it. (See Step 4: Create
tests for how to handle this in tests.)
The line that begins # type is a mypy type annotation. See this page for details about how to properly annotate your
webhook functions.
In the body of the function we define the body of the message as Hello! I am happy to be here!
:smile:. The :smile: indicates an emoji. Then we append a link to the Wikipedia article of the day as provided by the json payload. If the json payload does not include data for featured_title and featured_url
we catch a KeyError and use json_error to return the appropriate information: a 400 http status code with
relevant details.
Then we send a public (stream) message with check_send_message which will validate the message and then
send it.
Finally, we return a 200 http status with a JSON format success message via json_success().
Step 3: Create an api endpoint for the webhook
In order for a webhook to be externally available, it must be mapped to a url. This is done in zerver/lib/
integrations.py.
Look for the lines beginning with:
WEBHOOK_INTEGRATIONS = [
And you’ll find the entry for Hello World:
WebhookIntegration('helloworld', display_name='Hello World'),
This tells the Zulip api to call the api_helloworld_webhook function in zerver/webhooks/
helloworld/view.py when it receives a request at /api/v1/external/helloworld.
This line also tells Zulip to generate an entry for Hello World on the Zulip integrations page using static/images/
integrations/logos/helloworld.png as its icon.
At this point, if you’re following along and/or writing your own Hello World webhook, you have written enough code
to test your integration.
21.4. Step 3: Create an api endpoint for the webhook
113
Zulip Documentation, Release 1.4.0
First, get an API key from the Your bots section of your Zulip user’s Settings page. If you haven’t created a bot already,
you can do that there. Then copy its API key and replace the placeholder “<api_key>” in the examples with your real
key. This is how Zulip knows the request is from an authorized user.
Now you can test using Zulip itself, or curl on the command line.
Using manage.py from within the Zulip development environment:
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$
./manage.py send_webhook_fixture_message \
> --fixture=zerver/webhooks/helloworld/fixtures/hello.json \
> '--url=http://localhost:9991/api/v1/external/helloworld?api_key=<api_key>'
After which you should see something similar to:
2016-07-07 15:06:59,187 INFO
127.0.0.1
POST
200 143ms (mem: 6ms/13) (md:
˓→43ms/1) (db: 20ms/9q) (+start: 147ms) /api/v1/external/helloworld (helloworld˓→bot@zulip.com via ZulipHelloWorldWebhook)
Using curl:
curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":"Marilyn
˓→Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }' http://
˓→localhost:9991/api/v1/external/helloworld\?api_key\=<api_key>
After which you should see:
{"msg":"","result":"success"}
Using either method will create a message in Zulip:
Step 4: Create tests
Every webhook integration should have a corresponding test file: zerver/webhooks/mywebhook/tests.py.
The Hello World integration’s tests are in zerver/webhooks/helloworld/tests.py
You should name the class <WebhookName>HookTests and have it inherit from the base class
WebhookTestCase. For our HelloWorld webhook, we name the test class HelloWorldHookTests:
class HelloWorldHookTests(WebhookTestCase):
STREAM_NAME = 'test'
URL_TEMPLATE = "/api/v1/external/helloworld?&api_key={api_key}"
FIXTURE_DIR_NAME = 'helloworld'
# Note: Include a test function per each distinct message condition your
integration supports
def test_hello_message(self):
# type: () -> None
expected_subject = u"Hello World";
expected_message = u"Hello! I am happy to be here! :smile: \nThe Wikipedia
˓→featured article for today is **[Marilyn Monroe](https://en.wikipedia.org/wiki/
˓→Marilyn_Monroe)**";
˓→
114
Chapter 21. Webhook walkthrough
Zulip Documentation, Release 1.4.0
# use fixture named helloworld_hello
self.send_and_test_stream_message('hello', expected_subject, expected_message,
content_type="application/x-www-form˓→urlencoded")
def get_body(self, fixture_name):
# type: (Text) -> Text
return self.fixture_data("helloworld", fixture_name, file_type="json")
In the above example, STREAM_NAME, URL_TEMPLATE, and FIXTURE_DIR_NAME refer to class
attributes from the base class, WebhookTestCase.
These are needed by the helper function
send_and_test_stream_message to determine how to execute your test. STREAM_NAME should be
set to your default stream. If it doesn’t exist, send_and_test_stream_message will create it while executing
your test.
If your test expects a stream name from a test fixture, the value in the fixture and the value you set for STREAM_NAME
must match. The test helpers use STREAM_NAME to create the destination stream, and then create the message to send
using the value from the fixture. If these don’t match, the test will fail.
URL_TEMPLATE defines how the test runner will call your webhook, in the same way you would provide a webhook
URL to the 3rd party service. api_key={api_key} says that an API key is expected.
In get_body, the first argument in the call to self.fixture_data specifies the prefix of your fixture file names,
and file_type their type. Common types are json and txt.
When writing tests for your webhook, you’ll want to include one test function (and corresponding fixture) per each
distinct message condition that your integration supports.
If, for example, we added support for sending a goodbye message to our Hello World webhook, we would add
another test function to HelloWorldHookTests class called something like test_goodbye_message:
def test_goodbye_message(self):
# type: () -> None
expected_subject = u"Hello World";
expected_message = u"Hello! I am happy to be here! :smile:\nThe Wikipedia
˓→featured article for today is **[Goodbye](https://en.wikipedia.org/wiki/Goodbye)**";
# use fixture named helloworld_goodbye
self.send_and_test_stream_message('goodbye', expected_subject, expected_
˓→message,
content_type="application/x-www-form˓→urlencoded")
As well as a new fixture goodbye.json in zerver/webhooks/helloworld/fixtures/:
{
"featured_title":"Goodbye",
"featured_url":"https://en.wikipedia.org/wiki/Goodbye",
}
Also consider if your integration should have negative tests, a test where the data from the test fixture should result in
an error. For details see Negative tests, below.
Once you have written some tests, you can run just these new tests from within the Zulip development environment
with this command:
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$
./tools/test-backend zerver/webhooks/helloworld
21.5. Step 4: Create tests
115
Zulip Documentation, Release 1.4.0
(Note: You must run the tests from the top level of your development directory. The standard location in a Vagrant
environment is /srv/zulip. If you are not using Vagrant, use the directory where you have your development
environment.)
You will see some script output and if all the tests have passed, you will see:
Running zerver.webhooks.helloworld.tests.HelloWorldHookTests.test_goodbye_message
Running zerver.webhooks.helloworld.tests.HelloWorldHookTests.test_hello_message
DONE!
Step 5: Create documentation
Next, we add end-user documentation for our webhook integration. You can see the existing examples at https:
//zulipchat.com/integrations or by accessing /integrations in your Zulip development environment.
There are two parts to the end-user documentation on this page.
The first is the lozenge in the grid of integrations, showing your integration logo and name, which links
to the full documentation. This is generated automatically once you’ve registered the integration in WEBHOOK_INTEGRATIONS in zerver/lib/integrations.py, and supports some customization via options
to the WebhookIntegration class.
Second, you need to write the actual documentation content in zerver/webhooks/mywebhook/doc.html.
<p>Learn how Zulip integrations work with this simple Hello World example!</p>
<p>
The Hello World webhook will use the <code>test</code> stream, which is
created by default in the Zulip development environment. If you are running
Zulip in production, you should make sure this stream exists.
</p>
<p>
Next, on your {{ settings_html|safe }}, create a Hello World bot. Construct the
URL for the Hello World bot using the API key and stream name:
<code>{{ external_api_uri_subdomain }}/v1/external/helloworld?api_key=abcdefgh&
˓→amp;stream=test</code>
</p>
<p>
To trigger a notication using this webhook, use `send_webhook_fixture_message`
from the Zulip command line:
</p>
<div class="codehilite">
<pre>(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:/srv/zulip$
./manage.py send_webhook_fixture_message \
> --fixture=zerver/webhooks/helloworld/fixtures/hello.json \
> '--url=http://localhost:9991/api/v1/external/helloworld?api_key=&lt;api_key&gt;'
</pre>
</div>
<p>Or, use curl:</p>
<div class="codehilite">
<pre>curl -X POST -H "Content-Type: application/json" -d '{ "featured_title":
˓→"Marilyn Monroe", "featured_url":"https://en.wikipedia.org/wiki/Marilyn_Monroe" }'
˓→http://localhost:9991/api/v1/external/helloworld\?api_key\=&lt;api_key&gt;</pre>
</div>
116
Chapter 21. Webhook walkthrough
Zulip Documentation, Release 1.4.0
<p><b>Congratulations! You're done!</b><br/> Your messages may look like:</p>
<img class="screenshot" src="/static/images/integrations/helloworld/001.png"/>
See Documenting your integration for further details, including how to easily create the message screenshot.
Step 5: Preparing a pull request to zulip/zulip
When you have finished your webhook integration and are ready for it to be available in the Zulip product, follow
these steps to prepare your pull request:
1. Run tests including linters and ensure you have addressed any issues they report. See Testing and Linters for
details.
2. Read through Code styles and conventions and take a look through your code to double-check that you’ve
followed Zulip’s guidelines.
3. Take a look at your git history to ensure your commits have been clear and logical (see Version Control for
tips). If not, consider revising them with git rebase --interactive. For most webhooks, you’ll want
to squash your changes into a single commit and include a good, clear commit message.
4. Push code to your fork.
5. Submit a pull request to zulip/zulip.
If you would like feedback on your integration as you go, feel free to post a message on the public Zulip instance. You
can also create a [WIP] pull request while you are still working on your integration. See the Git guide for more on
Zulip’s pull request process.
Advanced topics
More complex implementation or testing needs may require additional code, beyond what the standard helper functions
provide. This section discusses some of these situations.
Negative tests
A negative test is one that should result in an error, such as incorrect data. The helper functions may interpret this as
a test failure, when it should instead be a successful test of an error condition. To correctly test these cases, you must
explicitly code your test’s execution (using other helpers, as needed) rather than call the usual helper function.
Here is an example from the WordPress webhook:
def test_unknown_action_no_data(self):
# type: () -> None
#
#
#
#
Mimic send_and_test_stream_message() to manually execute a negative test.
Otherwise its call to send_json_payload() would assert on the non-success
we are testing. The value of result is the error message the webhook should
return if no params are sent. The fixture for this test is an empty file.
# subscribe to the target stream
self.subscribe_to_stream(self.TEST_USER_EMAIL, self.STREAM_NAME)
21.7. Step 5: Preparing a pull request to zulip/zulip
117
Zulip Documentation, Release 1.4.0
# post to the webhook url
post_params = {'stream_name': self.STREAM_NAME,
'content_type': 'application/x-www-form-urlencoded'}
result = self.client_post(self.url, 'unknown_action', **post_params)
# check that we got the expected error message
self.assert_json_error(result, "Unknown WordPress webhook action: WordPress Action
˓→")
In a normal test, send_and_test_stream_message would handle all the setup and then check that the webhook’s response matches the expected result. If the webhook returns an error, the test fails. Instead, explicitly do the
setup it would have done, and check the result yourself.
Here, subscribe_to_stream is a test helper that uses TEST_USER_EMAIL and STREAM_NAME (attributes
from the base class) to register the user to receive messages in the given stream. If the stream doesn’t exist, it creates
it.
client_post, another helper, performs the HTTP POST that calls the webhook. As long as self.url is correct,
you don’t need to construct the webhook URL yourself. (In most cases, it is.)
assert_json_error then checks if the result matches the expected error.
If you had used
send_and_test_stream_message, it would have called send_json_payload, which checks the result
with assert_json_success.
Custom query parameters
Custom arguments passed in URL query parameters work as expected in the webhook code, but require special handling in tests.
For example, here is the definition of a webhook function that gets both stream and topic from the query parameters:
def api_querytest_webhook(request, user_profile, client,
payload=REQ(argument_type='body'), stream=REQ(default='test
˓→'),
topic=REQ(default='Default Alert')):
In actual use, you might configure the 3rd party service to call your Zulip integration with a URL like this:
http://myhost/api/v1/external/querytest?api_key=abcdefgh&stream=alerts&topic=queries
It provides values for stream and topic, and the webhook can get those using REQ without any special handling.
How does this work in a test?
The new attribute TOPIC exists only in our class so far. In order to construct a URL with a query parameter for
topic, you can pass the attribute TOPIC as a keyword argument to build_webhook_url, like so:
class QuerytestHookTests(WebhookTestCase):
STREAM_NAME = 'querytest'
TOPIC = "Default Topic"
URL_TEMPLATE = "/api/v1/external/querytest?api_key={api_key}&stream={stream}"
FIXTURE_DIR_NAME = 'querytest'
def test_querytest_test_one(self):
# type: () -> None
118
Chapter 21. Webhook walkthrough
Zulip Documentation, Release 1.4.0
# construct the URL used for this test
self.TOPIC = u"Query Test"
self.url = self.build_webhook_url(topic=self.TOPIC)
# define the expected message contents
expected_subject = u"Query Test"
expected_message = u"This is a test of custom query parameters."
self.send_and_test_stream_message('test_one', expected_subject, expected_
message,
content_type="application/x-www-form˓→urlencoded")
˓→
def get_body(self, fixture_name):
# type: (Text) -> Text
return self.fixture_data("querytest", fixture_name, file_type="json")
You can also override get_body if your test data needs to be constructed in an unusual way. For more, see the
definition for the base class, WebhookTestCase in zerver/lib/test_classes.py.
21.8. Advanced topics
119
Zulip Documentation, Release 1.4.0
120
Chapter 21. Webhook walkthrough
CHAPTER
22
Writing a new application feature
The changes needed to add a new feature will vary, of course, but this document provides a general outline of what
you may need to do, as well as an example of the specific steps needed to add a new feature: adding a new option to
the application that is dynamically synced through the data system in real-time to all browsers the user may have open.
As you read this, you may find you need to learn about Zulip’s real-time push system; the real-time push and events
documentation has a detailed explanation of how everything works.
General Process in brief
Adding a field to the database
Update the model: The server accesses the underlying database in zerver/ models.py. Add a new field in the
appropriate class.
Create and run the migration: To create and apply a migration, run:
./manage.py makemigrations
./manage.py migrate
Test your changes: Once you’ve run the migration, restart memcached on your development server (/etc/init.
d/memcached restart) and then restart run-dev.py to avoid interacting with cached objects.
Backend changes
For most new features/settings, the property_types framework handles most of this automatically, but it’s valuable to understand the flow even if the property_types framework means you don’t have to write any code.
Database interaction: Add any necessary code for updating and interacting with the database in zerver/lib/
actions.py. It should update the database and send an event announcing the change.
Application state: Modify the fetch_initial_state_data and apply_event functions in zerver/lib/
events.py to update the state based on the event you just created.
121
Zulip Documentation, Release 1.4.0
Backend implementation: Make any other modifications to the backend required for your feature to do what it’s
supposed to do.
New views: Add any new application views to zerver/urls.py. This includes both views that serve HTML (new
pages on Zulip) as well as new API endpoints that serve JSON-formatted data.
Testing: At the very least, add a test of your event data flowing through the system in test_events.py and an
API test in (e.g. for a Realm setting, in test_realm.py).
Frontend changes
JavaScript: Zulip’s JavaScript is located in the directory static/js/. The exact files you may need to change
depend on your feature. If you’ve added a new event that is sent to clients, be sure to add a handler for it to static/
js/server_events_dispatch.js.
CSS: The primary CSS file is static/styles/zulip.css. If your new feature requires UI changes, you may
need to add additional CSS to this file.
Templates: The initial page structure is rendered via Jinja2 templates located in templates/zerver. For
JavaScript, Zulip uses Handlebars templates located in static/templates. Templates are precompiled as part of
the build/deploy process.
Zulip is fully internationalized, so when writing both HTML templates or JavaScript code that generates user-facing
strings, be sure to tag those strings for translation.
Testing: There are two types of frontend tests: node-based unit tests and blackbox end-to-end tests. The blackbox
tests are run in a headless browser using Casper.js and are located in frontend_tests/casper_tests/. The
unit tests use Node’s assert module are located in frontend_tests/node_tests/. For more information on
writing and running tests see the testing documentation.
Documentation changes
After implementing the new feature, you should document it and update any existing documentation that might be
relevant to the new feature. For more information on the kinds of documentation Zulip has, see Documentation.
Example Feature
This example describes the process of adding a new setting to Zulip: a flag that restricts inviting new users to admins
only (the default behavior is that any user can invite other users). This flag is an actual Zulip feature. You can review
the original commit in the Zulip repo. Note that the code involved in adding a realm feature has been refactored
significantly since this feature was created, and Zulip has since been upgraded from Django 1.6 to 1.10.
Update the model
First, update the database and model to store the new setting. Add a new boolean field, invite_by_admins_only,
to the Realm model in zerver/models.py.
--- a/zerver/models.py
+++ b/zerver/models.py
@@ -108,6 +108,7 @@ class Realm(ModelReprMixin, models.Model):
restricted_to_domain = models.BooleanField(default=True) # type: bool
invite_required = models.BooleanField(default=False) # type: bool
+
invite_by_admins_only = models.BooleanField(default=False) # type: bool
122
Chapter 22. Writing a new application feature
Zulip Documentation, Release 1.4.0
create_stream_by_admins_only = models.BooleanField(default=False) # type: bool
mandatory_topics = models.BooleanField(default=False) # type: bool
The Realm model also contains an attribute, property_types, which other functions use to handle most realm
settings without any custom code for the setting (more on this process below). The attribute is a dictionary, where
the key is the name of the realm field and the value is the field’s type. Add the new field to the property_types
dictionary.
# Define the types of the various automatically managed properties
property_types = dict(
# ...
invite_by_admins_only=bool,
# ...
Note: the majority of realm settings can be included in property_types. However, there are some properties that
need custom logic and thus cannot use the property_types framework. For example:
• The realm authentication_methods attribute is a bitfield and needs additional code for validation and
updating.
• The allow_message_editing and message_content_edit_limit_seconds fields depend on
one another, so they are also handled separately and not included in property_types.
When creating a realm property that is not a boolean, Text or integer field, or when adding a field that is dependent on
other fields, handle it separately and do not add the field to the property_types dictionary. The steps below will
point out where to write code for these cases.
Create the migration
Create the migration file: ./manage.py makemigrations. Make sure to commit the generated file to git:
git add zerver/migrations/NNNN_realm_invite_by_admins_only.py (NNNN is a number that
is equal to the number of migrations.)
If you run into problems, the Django migration documentation is helpful.
Test your migration changes
Apply the migration: ./manage.py migrate
Output:
shell $ ./manage.py migrate
Operations to perform:
Synchronize unmigrated apps: staticfiles, analytics, pipeline
Apply all migrations: zilencer, confirmation, sessions, guardian, zerver, sites,
˓→auth, contenttypes
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Installing custom SQL...
Running migrations:
Rendering model states... DONE
Applying zerver.0026_realm_invite_by_admins_only... OK
22.2. Example Feature
123
Zulip Documentation, Release 1.4.0
Handle database interactions
Next, we will move on to implementing the backend part of this feature. Like typical apps, we will need our backend
to update the database and send some response to the client that made the request.
Beyond that, we need to orchestrate notifications to other clients (or other users, if you will) that our setting has
changed. Clients find out about settings through two closely related code paths. When a client first contacts the server,
the server sends the client its initial state. Subsequently, clients subscribe to “events,” which can (among other things)
indicate that settings have changed. For the backend piece, we will need our action to make a call to send_event to
send the event to clients that are active. We will also need to modify fetch_initial_state_data so that the
new field is passed to clients. See our event system docs for all the gory details.
Anyway, getting back to implementation details...
In zerver/lib/actions.py, the function do_set_realm_property takes in the name of a realm property
to update and the value it should have. This function updates the database and triggers an event to notify clients about
the change. It uses the field’s type, specified in the Realm.property_types dictionary, to validate the type of the
value before updating the property; this is primarily an assertion to help catch coding mistakes, not to check for bad
user input.
After updating the given realm field, do_set_realm_property creates an ‘update’ event with the name of the
property and the new value. It then calls send_event, passing the event and the list of users whose browser sessions
should be notified as the second argument. The latter argument can be a single user (if the setting is a personal one,
like time display format), members in a particular stream only or all active users in a realm.
# zerver/lib/actions.py
def do_set_realm_property(realm, name, value):
# type: (Realm, str, Union[Text, bool, int]) -> None
"""Takes in a realm object, the name of an attribute to update, and the
value to update.
"""
property_type = Realm.property_types[name]
assert isinstance(value, property_type), (
'Cannot update %s: %s is not an instance of %s' % (
name, value, property_type,))
setattr(realm, name, value)
realm.save(update_fields=[name])
event = dict(
type='realm',
op='update',
property=name,
value=value,
)
send_event(event, active_user_ids(realm))
If the new realm property being added does not fit into the do_set_realm_property framework (such as the
authentication_methods field), you’ll need to create a new function to explicitly update this field and send an
event.
# zerver/lib/actions.py
def do_set_realm_authentication_methods(realm, authentication_methods):
# type: (Realm, Dict[str, bool]) -> None
for key, value in list(authentication_methods.items()):
index = getattr(realm.authentication_methods, key).number
realm.authentication_methods.set_bit(index, int(value))
124
Chapter 22. Writing a new application feature
Zulip Documentation, Release 1.4.0
realm.save(update_fields=['authentication_methods'])
event = dict(
type="realm",
op="update_dict",
property='default',
data=dict(authentication_methods=realm.authentication_methods_dict())
)
send_event(event, active_user_ids(realm))
Update application state
You then need to add code to ensure that your new setting is included in the data sent down to clients, both when a
new client is loaded, and when changes happen. The fetch_initial_state_data function is responsible for
the former (data added to the state here will be available both in page_params in the browser, as well as to API
clients like the mobile apps). The apply_event function in zerver/lib/events.py is important for making
sure the state is always correct, even in the event of rare race conditions.
# zerver/lib/events.py
def fetch_initial_state_data(user_profile, event_types, queue_id, include_
˓→subscribers=True):
# ...
if want('realm'):
for property_name in Realm.property_types:
state['realm_' + property_name] = getattr(user_profile.realm, property_name)
state['realm_authentication_methods'] = user_profile.realm.authentication_methods_
˓→dict()
state['realm_allow_message_editing'] = user_profile.realm.allow_message_editing
# ...
def apply_event(state, events, user_profile, include_subscribers):
for event in events:
# ...
elif event['type'] == 'realm':
field = 'realm_' + event['property']
state[field] = event['value']
# ...
If you are adding a realm property that fits the property_types framework, you don’t need to change
fetch_initial_state_data or apply_event because there is already code to get the initial data and handle the realm update event type. However, if you are adding a property that is handled separately, you will need
to explicitly add the property to the state dictionary in the fetch_initial_state_data function. E.g., for
authentication_methods:
def fetch_initial_state_data(user_profile, event_types, queue_id, include_
˓→subscribers=True):
# ...
if want('realm'):
# ...
state['realm_authentication_methods'] = user_profile.realm.authentication_
˓→methods_dict()
# ...
For this setting, one won’t need to change apply_event since its default code for realm event types handles this
case correctly, but for a totally new type of feature, a few lines in that function may be needed.
22.2. Example Feature
125
Zulip Documentation, Release 1.4.0
Add a new view
You will need to add a view for clients to access that will call the actions.py code to update the database. This
example feature adds a new parameter that will be sent to clients when the application loads and should be accessible
via JavaScript. There is already a view that does this for related flags: update_realm in zerver/views/
realm.py. So in this case, we can add our code to the existing view instead of creating a new one.
Since this feature adds a checkbox to the admin page and a new property to the Realm model that can be modified from
there, you need to add a parameter for the new field to the update_realm function in zerver/views/realm.
py.
def update_realm(request, user_profile, name=REQ(validator=check_string,
˓→default=None),
# ...,
invite_by_admins_only=REQ(validator=check_bool, default=None),
# ...):
# type: (HttpRequest, UserProfile, ..., Optional[bool], ...
# ...
If this feature fits the do_set_realm_property framework and does not require additional validation, this is the
only change to make to zerver/views/realm.py.
Text fields or other realm properties that need additional validation can be handled at the beginning of
update_realm.
# Additional validation/error checking beyond types go here, so
# the entire request can succeed or fail atomically.
if default_language is not None and default_language not in get_available_language_
˓→codes():
raise JsonableError(_("Invalid language '%s'" % (default_language,)))
if description is not None and len(description) > 100:
return json_error(_("Realm description cannot exceed 100 characters."))
# ...
Then, the code in update_realm loops through the property_types dictionary and calls
do_set_realm_property on any property to be updated from the request. However, if the new feature is not in property_types, you will need to write the code to specifically handle it. Ex, for
authentication_methods:
# zerver/views/realm.py
# ...
if authentication_methods is not None and realm.authentication_methods_dict() !=
˓→authentication_methods:
do_set_realm_authentication_methods(realm, authentication_methods)
data['authentication_methods'] = authentication_methods
# ...
This completes the backend implementation. A great next step is to write the backend tests. With the
property_types framework, one just needs to add a line in test_events.py and test_realm.py with
a list of values to switch between in the test.
Update the front end
Then make the required front end changes: in this case a checkbox needs to be added to the admin page (and its value
added to the data sent back to server when a realm is updated) and the change event needs to be handled on the client.
126
Chapter 22. Writing a new application feature
Zulip Documentation, Release 1.4.0
To add the checkbox to the admin page, modify the relevant template, static/templates/admin_tab.
handlebars (omitted here since it is relatively straightforward). Then add code to handle changes to the new
form control in static/js/admin.js.
var url = "/json/realm";
var new_invite_by_admins_only =
$("#id_realm_invite_by_admins_only").prop("checked");
data[invite_by_admins_only] = JSON.stringify(new_invite_by_admins_only);
channel.patch({
url: url,
data: data,
success: function (data) {
# ...
if (data.invite_by_admins_only) {
ui_report.success("New users must be invited by an admin!", invite_by_admins_
˓→only_status);
} else {
ui_report.success("Any user may now invite new users!", invite_by_admins_only_
˓→status);
}
# ...
}
});
Finally, update server_events_dispatch.js to handle related events coming from the server.
# static/js/server_events_dispatch.js
function dispatch_normal_event(event) {
switch (event.type) {
# ...
case 'realm':
if (event.op === 'update' && event.property === 'invite_by_admins_only') {
page_params.realm_invite_by_admins_only = event.value;
}
}
}
Any code needed to update the UI should be placed in dispatch_normal_event callback (rather than the
channel.patch) function. This ensures the appropriate code will run even if the changes are made in another
browser window. In this example most of the changes are on the backend, so no UI updates are required.
Update documentation
After you add a new view, you should document your feature. This feature adds new functionality that restricts inviting
new users to admins only. A recommended way to document this feature would be to update and/or augment Zulip’s
user documentation to reflect your changes and additions.
At the very least, this will involve adding (or modifying) a Markdown file documenting the feature to templates/
zerver/help/ in the main Zulip server repository, where the source for Zulip’s user documentation is stored. For
information on writing user documentation, see Zulip’s general user guide documentation.
For a more concrete example of writing documentation for a new feature, see the original commit in the Zulip repo
that documented this feature, the current source, and the final rendered documentation.
22.2. Example Feature
127
Zulip Documentation, Release 1.4.0
128
Chapter 22. Writing a new application feature
CHAPTER
23
Writing views in Zulip
What this covers
This page documents how views work in Zulip. You may want to read the new feature tutorial or the integration guide,
and treat this as a reference.
If you have experience with Django, much of this will be familiar, but you may want to read about how REST requests
are dispatched, and how request authentication works.
This document supplements the new feature tutorial and the testing documentation.
What is a view?
A view in Zulip is everything that helps implement a server endpoint. Every path that the Zulip server supports (doesn’t
show a 404 page for) is a view. The obvious ones are those you can visit in your browser, for example /integrations,
which shows the integration documentation. These paths show up in the address bar of the browser. There are other
views that are only seen by software, namely the API views. They are used to build the various clients that Zulip has,
namely the web client (which is also used by the desktop client) and the mobile clients.
Modifying urls.py
A view is anything with an entry in the appropriate urls.py, usually zproject/urls.py. Zulip views either serve
HTML (pages for browsers) or JSON (data for Zulip clients on all platforms, custom bots, and integrations).
The format of the URL patterns in Django is documented here, and the Zulip specific details for these are discussed in
detail in the life of a request doc.
We have two Zulip-specific conventions we use for internationalization and for our REST API, respectively.
129
Zulip Documentation, Release 1.4.0
Writing human-readable views
If you’re writing a new page for the website, make sure to add it to i18n_urls in zproject/urls.py
i18n_urls = [
...
+
url(r'^quote-of-the-day/$', TemplateView.as_view(template_name='zerver/qotd.html
˓→')),
+
url(r'^postcards/$', 'zerver.views.postcards'),
]
As an example, if a request comes in for Spanish, language code es, the server path will be something like: es/
features/.
Decorators used for webpage views
This section documents a few simple decorators that we use for webpage views, as an introduction to view decorators.
require_post:
@require_post
def accounts_register(request):
# type: (HttpRequest) -> HttpResponse
This decorator ensures that the requst was a POST–here, we’re checking that the registration submission page is
requested with a post, and inside the function, we’ll check the form data. If you request this page with GET, you’ll get
a HTTP 405 METHOD NOT ALLOWED error.
zulip_login_required:
This decorator verifies that the browser is logged in (i.e. has a valid session cookie) before providing the view for this
route, or redirects the browser to a login page. This is used in the root path (/) of the website for the web client. If a
request comes from a browser without a valid session cookie, they are redirected to a login page. It is a small fork of
Django’s login_required, adding a few extra checks specific to Zulip.
@zulip_login_required
def home(request):
# type: (HttpRequest) -> HttpResponse
Writing a template
Templates for the main website are found in templates/zerver.
Writing API REST endpoints
These are code-parseable views that take x-www-form-urlencoded or JSON request bodies, and return JSON-string
responses. Almost all Zulip view code is in the implementations of API REST endpoints.
The REST API does authentication of the user through rest_dispatch, which is documented in detail at
zerver/lib/rest.py. This method will authenticate the user either through a session token from a cookie on the browser,
or from a base64 encoded email:api-key string given via HTTP Basic Auth for API clients.
130
Chapter 23. Writing views in Zulip
Zulip Documentation, Release 1.4.0
>>> import requests
>>> r = requests.get('https://api.github.com/user', auth=('hello@example.com',
˓→'0123456789abcdeFGHIJKLmnopQRSTUV'))
>>> r.status_code
-> 200
Request variables
Most API views will have some arguments that are passed as part of the request to control the behavior of the view. In
any well-engineered view, you need to write code to parse and validate that the arguments exist and have the correct
form. For many applications, this leads to one of several bad outcomes:
• The code isn’t written, so arguments aren’t validated, leading to bugs and confusing error messages for users of
the API.
• Every function starts with a long list of semi-redundant validation code, usually with highly inconsistent error
messages.
• Every view function comes with another function that does the validation that has the problems from the last
bullet point.
In Zulip, we solve this problem with a the special decorator called has_request_variables which allows a
developer to declare the arguments a view function takes and validate their types all within the def line of the function.
We like this framework because we have found it makes the validation code compact, readable, and conveniently
located in the same place as the method it is validating arguments for.
Here’s an example:
from zerver.decorator import has_request_variables, REQ, JsonableError, \
require_realm_admin
@require_realm_admin
@has_request_variables
def create_user_backend(request, user_profile, email=REQ(), password=REQ(),
full_name=REQ(), short_name=REQ()):
# ... code here
You will notice the special REQ() in the keyword arguments to create_user_backend.
has_request_variables parses the declared keyword arguments of the decorated function, and for each
that has an instance of REQ as the default value, it extracts the HTTP parameter with that name from the request,
parses it as JSON, and passes it to the function. It will return an nicely JSON formatted HTTP 400 error in the event
that an argument is missing, doesn’t parse as JSON, or otherwise is invalid.
require_realm_admin is another decorator which checks the authorization of the given user_profile to
make sure it belongs to a realm administrator (and thus has permission to create a user); we show it here primarily to
show how has_request_variables should be the inner decorator.
The implementation of has_request_variables is documented in detail in zerver/lib/request.py)
REQ also helps us with request variable validation. For example:
• msg_ids = REQ(validator=check_list(check_int)) will check that the msg_ids HTTP parameter is a list of integers, marshalled as JSON, and pass it into the function as the msg_ids Python keyword
argument.
• streams_raw = REQ("subscriptions", validator=check_list(check_string)) will
check that the “subscriptions” HTTP parameter is a list of strings, marshalled as JSON, and pass it into the
function with the Python keyword argument streams_raw.
23.5. Writing API REST endpoints
131
Zulip Documentation, Release 1.4.0
• message_id=REQ(converter=to_non_negative_int) will check that the message_id HTTP
parameter is a string containing a non-negative integer (converter differs from validator in that it does
not automatically marshall the input from JSON).
See zerver/lib/validator.py for more validators and their documentation.
Deciding which HTTP verb to use
When writing a new API view, you should writing a view to do just one type of thing. Usually that’s either a read or
write operation.
If you’re reading data, GET is the best option. Other read-only verbs are HEAD, which should be used for testing if a
resource is available to be read with GET, without the expense of the full GET. OPTIONS is also read-only, and used
by clients to determine which HTTP verbs are available for a given path. This isn’t something you need to write, as it
happens automatically in the implementation of rest_dispatch–see zerver/lib/rest.py for more.
If you’re creating new data, try to figure out if the thing you are creating is uniquely identifiable. For example, if you’re
creating a user, there’s only one user per email. If you can find a unique ID, you should use PUT for the view. If you
want to create the data multiple times for multiple requests (for example, requesting the send_message view multiple
times with the same content should send multiple messages), you should use POST.
If you’re updating existing data, use PATCH.
If you’re removing data, use DELETE.
Idempotency
When writing a new API endpoint, with the exception of things like sending messages, requests should be safe to
repeat, without impacting the state of the server. This is idempotency.
You will often want to return an error if a request to change something would do nothing because the state is already
as desired, to make debugging Zulip clients easier. This means that the response for repeated requests may not be the
same, but the repeated requests won’t change the server more than once or cause unwanted side effects.
Making changes to the database
If the view does any modification to the database, that change is done in a helper function in zerver/lib/
actions.py. Those functions are responsible for doing a complete update to the state of the server, which often
entails both updating the database and sending any events to notify clients about the state change. When possible,
we prefer to design a clean boundary between the view function and the actions function is such that all user input
validation happens in the view code (i.e. all 400 type errors are thrown there), and the actions code is responsible for
atomically executing the change (this is usually signalled by having the actions function have a name starting with
do_. So in most cases, errors in an actions function will be the result of an operational problem (e.g. lost connection
to the database) and lead to a 500 error. If an actions function is responsible for validation as well, it should have a
name starting with check_.
For example, in zerver/views/realm.py:
@require_realm_admin
@has_request_variables
def update_realm(request, user_profile, name=REQ(validator=check_string,
˓→default=None), ...)):
# type: (HttpRequest, UserProfile, ...) -> HttpResponse
realm = user_profile.realm
data = {} # type: Dict[str, Any]
132
Chapter 23. Writing views in Zulip
Zulip Documentation, Release 1.4.0
if name is not None and realm.name != name:
do_set_realm_name(realm, name)
data['name'] = 'updated'
and in zerver/lib/actions.py:
def do_set_realm_name(realm, name):
# type: (Realm, Text) -> None
realm.name = name
realm.save(update_fields=['name'])
event = dict(
type="realm",
op="update",
property='name',
value=name,
)
send_event(event, active_user_ids(realm))
realm.save() actually saves the changes to the realm to the database, and send_event sends the event to active
clients belonging to the provided list of users (in this case, all altive users in the Zulip realm).
Calling from the web application
You should always use channel. to make an HTTP <method> call to the Zulip JSON API. As an example, in
static/js/admin.js
var url = "/json/realm";
var data = {
name: JSON.stringify(new_name),
}
channel.patch({
url: url,
data: data,
success: function (response_data) {
if (response_data.name !== undefined) {
ui_report.success(i18n.t("Name changed!"), name_status);
}
...
Calling from an API client
Here’s how you might manually make a call from python:
payload = {'name': new_name}
# email and API key
api_auth = ('hello@example.com', '0123456789abcdeFGHIJKLmnopQRSTUV')
r = requests.patch(SERVER_URL + 'api/v1/realm',
data=json.dumps(payload),
auth=api_auth,
)
This is simply an illustration; we recommend making use of the Zulip Python API bindings since they provide a nice
interface for accessing the API.
23.5. Writing API REST endpoints
133
Zulip Documentation, Release 1.4.0
Legacy endpoints used by the web client
New features should conform the REST API style. The legacy, web-only endpoints can’t effectively enforce usage of
a browser, so they aren’t preferable from a security perspective, and it is generally a good idea to make your feature
available to other clients, especially the mobile clients.
These endpoints make use of some older authentication decorators, authenticated_json_api_view,
authenticated_json_post_view, and authenticated_json_view, so you may see them in the code.
Webhook integration endpoints
Webhooks are called by other services, often to send a message as part of those services’ integrations. They are most
often POST requests, and often there is very little you can customize about them. Usually you can expect that the
webhook for a service will allow specification for the target server for the webhook, and an API key.
If the webhook does not have an option to provide a bot email, use the api_key_only_webhook_view decorator,
to fill in the user_profile and request.client fields of a request:
@api_key_only_webhook_view('PagerDuty')
@has_request_variables
def api_pagerduty_webhook(request, user_profile,
payload=REQ(argument_type='body'),
stream=REQ(default='pagerduty'),
topic=REQ(default=None)):
request.client will be the result of get_client("ZulipPagerDutyWebhook") in this example and it
will be passed to check_send_message. For more information, see Clients in Zulip.
134
Chapter 23. Writing views in Zulip
CHAPTER
24
Life of a Request
It can sometimes be confusing to figure out how to write a new feature, or debug an existing one. Let us try to follow
a request through the Zulip codebase, and dive deep into how each part works.
We will use as our example the creation of users through the API, but we will also highlight how alternative requests
are handled.
A request is sent to the server, and handled by Nginx
When Zulip is deployed in production, all requests go through nginx. For the most part we don’t need to know how
this works, except for when it isn’t working. Nginx does the first level of routing–deciding which application will
serve the request (or deciding to serve the request itself for static content).
In development, tools/run-dev.py fills the role of nginx. Static files are in your git checkout under static,
and are served unminified.
Nginx secures traffic with SSL
If you visit your Zulip server in your browser and discover that your traffic isn’t being properly encrypted, an nginx
misconfiguration is the likely culprit.
Static files are served directly by Nginx
Static files include JavaScript, css, static assets (like emoji, avatars), and user uploads (if stored locally and not on S3).
location /static/ {
alias /home/zulip/prod-static/;
error_page 404 /static/html/404.html;
}
135
Zulip Documentation, Release 1.4.0
Nginx routes other requests between django and tornado
All our connected clients hold open long-polling connections so that they can receive events (messages, presence
notifications, and so on) in real-time. Events are served by Zulip’s tornado application.
Nearly every other kind of request is served by the zerver Django application.
Here is the relevant nginx routing configuration.
Django routes the request to a view in urls.py files
There are various urls.py files throughout the server codebase, which are covered in more detail in the directory
structure doc.
The main Zulip Django app is zerver. The routes are found in
zproject/urls.py
zproject/legacy_urls.py
There are HTML-serving, REST API, legacy, and webhook url patterns. We will look at how each of these types of
requests are handled, and focus on how the REST API handles our user creation example.
Views serving HTML are internationalized by server path
If we look in zproject/urls.py, we can see something called i18n_urls. These urls show up in the address bar of the
browser, and serve HTML.
For example, the /hello page (preview here) gets translated in Chinese at zh-cn/hello/ (preview here).
Note the zh-cn prefix–that url pattern gets added by i18n_patterns.
API endpoints use REST
Our example is a REST API endpoint. It’s a PUT to /users.
With the exception of Webhooks (which we do not usually control the format of), legacy endpoints, and logged-out
endpoints, Zulip uses REST for its API. This means that we use:
• POST for creating something new where we don’t have a unique ID. Also used as a catch-all if no other verb is
appropriate.
• PUT for creating something for which we have a unique ID.
• DELETE for deleting something
• PATCH for updating or editing attributes of something.
• GET to get something (read-only)
• HEAD to check the existence of something to GET, without getting it; useful to check a link without downloading a potentially large link
• OPTIONS (handled automatically, see more below)
136
Chapter 24. Life of a Request
Zulip Documentation, Release 1.4.0
Of these, PUT, DELETE, HEAD, OPTIONS, and GET are idempotent, which means that we can send the request
multiple times and get the same state on the server. You might get a different response after the first request, as we like
to give our clients an error so they know that no new change was made by the extra requests.
POST is not idempotent–if I send a message multiple times, Zulip will show my message multiple times. PATCH is
special–it can be idempotent, and we like to write API endpoints in an idempotent fashion, as much as possible.
This cookbook and tutorial can be helpful if you are new to REST web applications.
PUT is only for creating new things
If you’re used to using PUT to update or modify resources, you might find our convention a little strange.
We use PUT to create resources with unique identifiers, POST to create resources without unique identifiers (like
sending a message with the same content multiple times), and PATCH to modify resources.
In our example, create_user_backend uses PUT, because there’s a unique identifier, the user’s email.
OPTIONS
The OPTIONS method will yield the allowed methods.
This request: OPTIONS https://chat.zulip.org/api/v1/users yields a response with this HTTP
header: Allow: PUT, GET
We can see this reflected in zproject/urls.py:
url(r'^users$', 'zerver.lib.rest.rest_dispatch',
{'GET': 'zerver.views.users.get_members_backend',
'PUT': 'zerver.views.users.create_user_backend'}),
In this way, the API is partially self-documenting.
Legacy endpoints are used by the web client
The endpoints from the legacy JSON API are written without REST in mind. They are used extensively by the web
client, and use POST.
You can see them in zproject/legacy_urls.py.
Webhook integrations may not be RESTful
Zulip endpoints that are called by other services for integrations have to conform to the service’s request format. They
are likely to use only POST.
Django calls rest_dispatch for REST endpoints, and authenticates
For requests that correspond to a REST url pattern, Zulip configures its url patterns (see zerver/lib/rest.py) so that
the action called is rest_dispatch. This method will authenticate the user, either through a session token from a
cookie, or from an email:api-key string given via HTTP Basic Auth for API clients.
It will then look up what HTTP verb was used (GET, POST, etc) to make the request, and then figure out which view
to show from that.
24.8. Django calls rest_dispatch for REST endpoints, and authenticates
137
Zulip Documentation, Release 1.4.0
In our example,
{'GET': 'zerver.views.users.get_members_backend',
'PUT': 'zerver.views.users.create_user_backend'}
is supplied as an argument to rest_dispatch, along with the HTTPRequest. The request has the HTTP
verb PUT, which rest_dispatch can use to find the correct view to show: zerver.views.users.
create_user_backend.
The view will authorize the user, extract request variables, and validate them
This is covered in good detail in the writing views doc.
Results are given as JSON
Our API works on JSON requests and responses. Every API endpoint should return json_error in the case of an
error, which gives a JSON string:
{'result':
'error', 'msg':
<some error message>}
in a HTTP Response with a content type of ‘application/json’.
To pass back data from the server to the calling client, in the event of a successfully handled
request, we use json_success(data=<some python object which can be converted to a
JSON string>.
This will result in a JSON string:
{'result': 'success', 'msg': '',
'var_name2': 'var_value2'...}
'data'='{'var_name1':
'var_value1',
with a HTTP 200 status and a content type of ‘application/json’.
That’s it!
138
Chapter 24. Life of a Request
CHAPTER
25
Reading list
Here you can find a list of relevant resources that you may find useful for learning new skills, or polishing the ones
you already have.
The topics cover a wide variety of topics, from basic Python coding to general developing guidelines.
Feel free to create a pull request in Zulip’s GitHub repository with any interesting books, articles or videos you would
like to see in this list.
Some titles have been shortened for organizational purposes.
General programming/IT
Book - Clean Code: A Handbook of Agile Software Craftsmanship (Not free!)
Books - Free programming books list
Blog - Free Code Camp blog
Blog - Idle Words talks transcripts
Tutorial - HTTP Can Do That?!, by Sumana Harihareswara (PyCon 2016)
Video - Minimum Viable Documentation, by Matthew Lyon (WriteTheDocs 2014)
Video - NoOps, by Kelsey Hightower (DepOpsDays 2016)
Video - The mind behind Linux (TED interview)
Tutorial - Learn code the hard way
Tutorial - What happens when...
Article - An Interview With Linus Torvalds
Article - Effective Learning Strategies for Programmers
Article - Readme Driven Development
139
Zulip Documentation, Release 1.4.0
Article - Systematic Debugging
Paper - Floating-Point Arithmetic
Python
Video - Intro to Python for beginners, by Jessica McKellar (PyCon 2013)
Video - Breaking the rules, by Jessica McKellar (PyCon Sweden)
Video - Build & break a Python sandbox, by Jessica McKellar (PyCon 2014)
Video - Cache me if you can, by Guillaume Ardaud (PyCon 2014)
Video - Loop like a native, by Ned Batchelder (PyCon 2013)
Video - Modern Dictionaries, by Raymond Hettinger (SF Python)
Video - Python Language, by Guido van Rossum (PyCon 2016)
Video - The Mighty Dictionary, by Brandon Rhodes (PyCon 2010)
Tutorial - Code Like a Pythonista: Idiomatic Python
Article - Static types in Python, oh my(py)!
Guide - The Hitchhiker’s Guide to Python!
Java/Android
Course - Android Development for Beginners
Blog - Java Tutorials for Beginners
JavaScript/ECMAScript
Tutorial - clean-code-javascript Software engineering principles
Course - React native and redux course (Not free!)
Slides - TypeScript vs. CoffeeScript vs. ES6
Git/Version Control Systems (VCS)
You may want to take a look first at our Git and GitHub guide.
Article - Git tips
Computer Science/Algorithms
Blog - GeeksforGeeks
Book Introduction to Algorithms (Not free!)
140
Chapter 25. Reading list
Zulip Documentation, Release 1.4.0
Blog - Setosa data visualization and visual explanations
Course - Algorithms, Part I
Course - Open Source Society University
Course - MIT CSAIL 6.828: Operative Systems Engineering
Community experience
Book - Producing Open Source Software
Article - Advice on Starting And Running A New Open Source Project
Article - How to ask good questions
Article - Notes for New FLOSS Contributors
Article - To be mentored
Article - To mentor
List of good projects for new contributors
Competitions/Camps
CodeForces
Free Code Camp
Massive Open Online Courses (MOOC) Platforms
Coursera
edX
MIT OpenCourseWare
Udacity
25.7. Community experience
141
Zulip Documentation, Release 1.4.0
142
Chapter 25. Reading list
CHAPTER
26
Git & GitHub Guide
Quick start: How Zulip uses Git and GitHub
This quick start provides a brief overview of how Zulip uses Git and GitHub.
Those who are familiar with Git and GitHub should be able to start contributing with these details in mind:
• We use GitHub for source control and code review. To contribute, fork zulip/zulip (or the appropriate repository, if you are working on something else besides Zulip server) to your own account and then create feature/issue branches. When you’re ready to get feedback, submit a work-in-progress (WIP) pull request. We
encourage you to submit WIP pull requests early and often.
• We use a rebase-oriented workflow. We do not use merge commits. This means you should use git fetch
followed by git rebase rather than git pull (or you can use git pull --rebase). Also, to prevent
pull requests from becoming out of date with the main line of development, you should rebase your feature
branch prior to submitting a pull request, and as needed thereafter. If you’re unfamiliar with how to rebase a
pull request, read this excellent guide.
We use this strategy in order to avoid the extra commits that appear when another branch is merged, that clutter
the commit history (it’s popular with other large projects such as Django). This makes Zulip’s commit history
more readable, but a side effect is that many pull requests we merge will be reported by GitHub’s UI as closed
instead of merged, since GitHub has poor support for rebase-oriented workflows.
• We have a code style guide, a commit message guide, and strive for each commit to be a minimal coherent
idea (see commit discipline for details).
• We provide many tools to help you submit quality code. These include linters, tests, continuous integration
with TravisCI, and mypy.
• We use zulipbot to manage our issues and pull requests to create a better GitHub workflow for contributors.
Finally, take a quick look at Zulip-specific Git scripts, install the Zulip developer environment, and then configure your
fork for use with TravisCI.
The following sections will help you be awesome with Zulip and Git/GitHub in a rebased-based workflow. Read
through it if you’re new to git, to a rebase-based git workflow, or if you’d like a git refresher.
143
Zulip Documentation, Release 1.4.0
Set up Git
If you’re already using Git, have a client you like, and a GitHub account, you can skip this section. Otherwise, read
on!
Install and configure Git, join GitHub
If you’re not already using Git, you might need to install and configure it.
You’ll also need a GitHub account, which you can sign up for here. We also recommend you create an ssh key if you
don’t already have one and add it to your GitHub account.
If you plan on using Git from the command line, the following tips can make your experience better:
• Install the command auto-completion and/or git-prompt plugins available for Bash and Zsh.
• Configure Git with your user details and aliases for commands you’ll use often.
Get a graphical client
Even if you’re comfortable using git on the command line, having a graphic client can be useful for viewing your
repository. This is especially when doing a complicated rebases and similar operations because you can check the
state of your repository after each command to see what changed. If something goes wrong, this helps you figure out
when and why.
If you don’t already have one installed, here are some suggestions:
• macOS: GitX-dev
• Ubuntu/Linux: git-cola, gitg, gitk
• Windows: SourceTree
If you like working on the command line, but want better visualization and navigation of your git repo, try Tig, a
cross-platform ncurses-based text-mode interface to Git.
And, if none of the above are to your liking, try one of these.
How Git is different
Whether you’re new to Git or have experience with another version control system (VCS), it’s a good idea to learn
a bit about how Git works. We recommend this excellent presentation Understanding Git from Nelson Elhage and
Anders Kaseorg and the Git Basics chapter from Pro Git by Scott Chacon and Ben Straub.
Here are the top things to know:
• Git works on snapshots: Unlike other version control systems (e.g., Subversion, Perforce, Bazaar), which track
files and changes to those files made over time, Git tracks snapshots of your project. Each time you commit or
otherwise make a change to your repository, Git takes a snapshot of your project and stores a reference to that
snapshot. If a file hasn’t changed, Git creates a link to the identical file rather than storing it again.
• Most Git operations are local: Git is a distributed version control system, so once you’ve cloned a repository,
you have a complete copy of that repository’s entire history. Staging, committing, branching, and browsing
history are all things you can do locally without network access and without immediately affecting any remote
repositories. To make or receive changes from remote repositories, you need to git fetch, git pull, or
git push.
144
Chapter 26. Git & GitHub Guide
Zulip Documentation, Release 1.4.0
• Nearly all Git actions add information to the Git database, rather than removing it. As such, it’s hard to
make Git perform actions that you can’t undo. However, Git can’t undo what it doesn’t know about, so it’s a
good practice to frequently commit your changes and frequently push your commits to your remote repository.
• Git is designed for lightweight branching and merging. Branches are simply references to snapshots. It’s
okay and expected to make a lot of branches, even throwaway and experimental ones.
• Git stores all data as objects, of which there are four types: blob (file), tree (directory), commit (revision),
and tag. Each of these objects is named by a unique hash, the SHA-1 has of its contents. Most of the time you’ll
refer to objects by their truncated hash or more human-readable reference like HEAD (the current branch). Blobs
and trees represent files and directories. Tags are named references to other objects. A commit object includes:
tree id, zero or more parents as commit ids, an author (name, email, date), a committer (name, email, date), and
a log message. A Git repository is a collection of mutable pointers to these objects called refs.
• Cloning a repository creates a working copy. Every working copy has a .git subdirectory, which contains
its own Git repository. The .git subdirectory also tracks the index, a staging area for changes that will become
part of the next commit. All files outside of .git is the working tree.
• Files tracked with Git have possible three states: committed, modified, and staged. Committed files are
those safely stored in your local .git repository/database. Staged files have changes and have been marked
for inclusion in the next commit; they are part of the index. Modified files have changes but have not yet been
marked for inclusion in the next commit; they have not been added to the index.
• Git commit workflow is as follows: Edit files in your working tree. Add to the index (that is stage) with git
add. Commit to the HEAD of the current branch with git commit.
Important Git terms
When you install Git, it adds a manual entry for gitglossary. You can view this glossary by running man
gitglossary. Below we’ve included the git terms you’ll encounter most often along with their definitions from
gitglossary.
branch
A “branch” is an active line of development. The most recent commit on a branch is referred to as the tip of that
branch. The tip of the branch is referenced by a branch head, which moves forward as additional development is done
on the branch. A single Git repository can track an arbitrary number of branches, but your working tree is associated
with just one of them (the “current” or “checked out” branch), and HEAD points to that branch.
cache
Obsolete for: index
checkout
The action of updating all or part of the working tree with a tree object or blob from the object database, and updating
the index and HEAD if the whole working tree has been pointed at a new branch.
26.4. Important Git terms
145
Zulip Documentation, Release 1.4.0
commit
As a noun: A single point in the Git history; the entire history of a project is represented as a set of interrelated
commits. The word “commit” is often used by Git in the same places other revision control systems use the words
“revision” or “version”. Also used as a short hand for commit object.
As a verb: The action of storing a new snapshot of the project’s state in the Git history, by creating a new commit
representing the current state of the index and advancing HEAD to point at the new
fast-forward
A fast-forward is a special type of merge where you have a revision and you are “merging” another branch’s changes
that happen to be a descendant of what you have. In such these cases, you do not make a new mergecommit but instead
just update to his revision. This will happen frequently on a remote-tracking branch of a remote repository.
fetch
Fetching a branch means to get the branch’s head ref from a remote repository, to find out which objects are missing
from the local object database, and to get them, too. See also git-fetch(1).
hash
In Git’s context, synonym for object name.
head
A named reference to the commit at the tip of a branch. Heads are stored in a file in $GIT_DIR/refs/heads/ directory,
except when using packed refs. (See git-pack-refs(1).)
HEAD
The current branch. In more detail: Your working tree is normally derived from the state of the tree referred to by
HEAD. HEAD is a reference to one of the heads in your repository, except when using a detached HEAD, in which
case it directly references an arbitrary commit.
index
A collection of files with stat information, whose contents are stored as objects. The index is a stored version of your
working tree. Truth be told, it can also contain a second, and even a third version of a working tree, which are used
when merging.
pull
Pulling a branch means to fetch it and merge it. See also git- pull(1).
146
Chapter 26. Git & GitHub Guide
Zulip Documentation, Release 1.4.0
push
Pushing a branch means to get the branch’s head ref from a remote repository, find out if it is a direct ancestor to the
branch’s local head ref, and in that case, putting all objects, which are reachable from the local head ref, and which are
missing from the remote repository, into the remote object database, and updating the remote head ref. If the remote
head is not an ancestor to the local head, the push fails.
rebase
To reapply a series of changes from a branch to a different base, and reset the head of that branch to the result.
Get Zulip code
Zulip uses a forked-repo and rebase-oriented workflow.. This means that all contributors create a fork of the Zulip
repository they want to contribute to and then submit pull requests to the upstream repository to have their contributions
reviewed and accepted. We also recommend you work on feature branches.
Step 1a: Create your fork
The following steps you’ll only need to do the first time you setup a machine for contributing to a given Zulip project.
You’ll need to repeat the steps for any additional Zulip projects (list) that you work on.
The first thing you’ll want to do to contribute to Zulip is fork (see how) the appropriate Zulip repository. For the main
server app, this is zulip/zulip.
Step 1b: Clone to your machine
Next, clone your fork to your local machine:
$ git clone git@github.com:christi3k/zulip.git
Cloning into 'zulip'
remote: Counting objects: 86768, done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 86768 (delta 5), reused 1 (delta 1), pack-reused 86752
Receiving objects: 100% (86768/86768), 112.96 MiB | 523.00 KiB/s, done.
Resolving deltas: 100% (61106/61106), done.
Checking connectivity... done.
Note: If you receive an error while cloning, you may not have added your ssh key to GitHub.
Step 1c: Connect your fork to Zulip upstream
Next you’ll want to configure an upstream remote repository for your fork of Zulip. This will allow you to sync
changes from the main project back into your fork.
First, show the currently configured remote repository:
$ git remote -v
origin git@github.com:YOUR_USERNAME/zulip.git (fetch)
origin git@github.com:YOUR_USERNAME/zulip.git (push)
26.5. Get Zulip code
147
Zulip Documentation, Release 1.4.0
Note: If you’ve cloned the repository using a graphical client, you may already have the upstream remote repository
configured. For example, when you clone zulip/zulip with the GitHub desktop client it configures the remote repository
zulip and you see the following output from git remote -v:
origin
origin
zulip
zulip
git@github.com:YOUR_USERNAME/zulip.git (fetch)
git@github.com:YOUR_USERNAME/zulip.git (push)
https://github.com/zulip/zulip.git (fetch)
https://github.com/zulip/zulip.git (push)
If your client hasn’t automatically configured a remote for zulip/zulip, you’ll need to with:
$ git remote add upstream https://github.com/zulip/zulip.git
Finally, confirm that the new remote repository, upstream, has been configured:
$ git remote -v
origin git@github.com:YOUR_USERNAME/zulip.git (fetch)
origin git@github.com:YOUR_USERNAME/zulip.git (push)
upstream https://github.com/zulip/zulip.git (fetch)
upstream https://github.com/zulip/zulip.git (push)
Step 2: Set up the Zulip development environment
If you haven’t already, now is a good time to install the Zulip development environment (overview). If you’re new
to working on Zulip or open source projects in general, we recommend following our detailed guide for first-time
contributors.
Step 3: Configure Travis CI (continuous integration)
This step is optional, but recommended.
Zulip Server is configured to use Travis CI to test and create builds upon each new commit and pull request. Travis CI
is free for open source projects and it’s easy to configure for your own fork of Zulip. After doing so, TravisCI will run
tests for new refs you push to GitHub and email you the outcome (you can also view the results in the web interface).
First, sign in to Travis CI with your GitHub account and authorize Travis CI to access your GitHub account and
repositories. Once you’ve done this, Travis CI will fetch your repository information and display it on your profile
page. From there you can enable integration with Zulip. (See screen cast.)
Using Git as you work
Know what branch you’re working on
When using Git, it’s important to know which branch you currently have checked out because most git commands
implicitly operate on the current branch. You can determine the currently checked out branch several ways.
One way is with git status:
$ git status
On branch issue-demo
nothing to commit, working directory clean
Another is with git branch which will display all local branches, with a star next to the current branch:
148
Chapter 26. Git & GitHub Guide
Zulip Documentation, Release 1.4.0
$ git branch
* issue-demo
master
To see even more information about your branches, including remote branches, use git branch -vva:
$ git branch -vva
517468b troubleshooting tip about provisioning
* issue-123
master
f0eaee6 [origin/master] bug: Fix traceback in get_missed_
˓→message_token_from_address().
remotes/origin/HEAD
-> origin/master
remotes/origin/issue-1234 4aeccb7 Another test commit, with longer message.
remotes/origin/master
f0eaee6 bug: Fix traceback in get_missed_message_token_
˓→from_address().
remotes/upstream/master
dbeab6a Optimize checks of test database state by moving
˓→into Python.
You can also configure Bash and Zsh to display the current branch in your prompt.
Keep your fork up to date
You’ll want to keep your fork up-to-date with changes from Zulip’s main repositories.
Note about git pull: You might be used to using git pull on other projects. With Zulip, because we don’t use
merge commits, you’ll want to avoid it. Rather that using git pull, which by default is a shortcut for git fetch
&& git merge FETCH_HEAD (docs), you should use git fetch and then git rebase.
First, fetch changes from Zulip’s upstream repository you configured in the step above:
$ git fetch upstream
Next, checkout your master branch and rebase it on top of upstream/master:
$ git checkout master
Switched to branch 'master'
$ git rebase upstream/master
This will rollback any changes you’ve made to master, update it from upstream/master, and then re-apply your
changes. Rebasing keeps the commit history clean and readable.
When you’re ready, push your changes to your remote fork. Make sure you’re in branch master and the run git
push:
$ git checkout master
$ git push origin master
You can keep any branch up to date using this method. If you’re working on a feature branch (see next section), which
we recommend, you would change the command slightly, using the name of your feature-branch rather than
master:
$ git checkout feature-branch
Switched to branch 'feature-branch'
$ git rebase upstream/master
$ git push origin feature-branch
26.6. Using Git as you work
149
Zulip Documentation, Release 1.4.0
Work on a feature branch
One way to keep your work organized is to create a branch for each issue or feature. Recall from how Git is different
that Git is designed for lightweight branching and merging. You can and should create as many branches as you’d
like.
First, make sure your master branch is up-to-date with Zulip upstream (see how).
Next, from your master branch, create a new tracking branch, providing a descriptive name for your feature branch:
$ git checkout master
Switched to branch 'master'
$ git checkout -b issue-1755-fail2ban
Switched to a new branch 'issue-1755-fail2ban'
Alternatively, you can create a new branch explicitly based off upstream/master:
$ git checkout -b issue-1755-fail2ban upstream/master
Switched to a new branch 'issue-1755-fail2ban'
Now you’re ready to work on the issue or feature.
Run linters and tests locally
In addition to having Travis run tests and linters each time you push a new commit, you can also run them locally. See
testing for details.
Stage changes
Recall that files tracked with Git have possible three states: committed, modified, and staged.
To prepare a commit, first add the files with changes that you want to include in your commit to your staging area.
You add both new files and existing ones. You can also remove files from staging when necessary.
Get status of working directory
To see what files in the working directory have changes that have not been staged, use git status.
If you have no changes in the working directory, you’ll see something like this:
$ git status
On branch issue-123
nothing to commit, working directory clean
If you have unstaged changes, you’ll see something like this:
On branch issue-123
Untracked files:
(use "git add <file>..." to include in what will be committed)
newfile.py
nothing added to commit but untracked files present (use "git add" to track)
150
Chapter 26. Git & GitHub Guide
Zulip Documentation, Release 1.4.0
Stage additions with git add
To add changes to your staging area, use git add <filename>. Because git add is all about staging the
changes you want to commit, you use it to add new files as well as files with changes to your staging area.
Continuing our example from above, after we run git add newfile.py, we’ll see the following from git
status:
On branch issue-123
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
new file:
newfile.py
You can view the changes in files you have staged with git diff --cached. To view changes to files you haven’t
yet staged, just use git diff.
If you want to add all changes in the working directory, use git add -A (documentation).
You can also stage changes using your graphical Git client.
If you stage a file, you can undo it with git reset HEAD <filename>. Here’s an example where we stage a
file test3.txt and then unstage it:
$ git add test3.txt
On branch issue-1234
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
new file:
test3.txt
$ git reset HEAD test3.txt
$ git status
On branch issue-1234
Untracked files:
(use "git add <file>..." to include in what will be committed)
test3.txt
nothing added to commit but untracked files present (use "git add" to track)
Stage deletions with git rm
To remove existing files from your repository, use git rm (documentation). This command can either stage the file
for removal from your repository AND delete it from your working directory or just stage the file for deletion and
leave it in your working directory.
To stage a file for deletion and remove it from your working directory, use git rm <filename>:
$ git rm test.txt
rm 'test.txt'
$ git status
On branch issue-1234
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
deleted:
test.txt
26.6. Using Git as you work
151
Zulip Documentation, Release 1.4.0
$ ls test.txt
ls: No such file or directory
To stage a file for deletion and keep it in your working directory, use git rm --cached <filename>:
$ git rm --cached test2.txt
rm 'test2.txt'
$ git status
On branch issue-1234
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
deleted:
test2.txt
$ ls test2.txt
test2.txt
If you stage a file for deletion with the --cached option, and haven’t yet run git commit, you can undo it with
git reset HEAD <filename>:
$ git reset HEAD test2.txt
Unfortunately, you can’t restore a file deleted with git rm if you didn’t use the --cache option. However, git
rm only deletes files it knows about. Files you have never added to git won’t be deleted.
Commit changes
When you’ve staged all your changes, you’re ready to commit. You can do this with git commit -m "My
commit message." to include a commit message.
Here’s an example of committing with the -m for a one-line commit message:
$ git commit -m "Add a test commit for docs."
[issue-123 173e17a] Add a test commit for docs.
1 file changed, 1 insertion(+)
create mode 100644 newfile.py
You can also use git commit without the -m option and your editor to open, allowing you to easily draft a multi-line
commit message.
How long your commit message should be depends on where you are in your work. Using short, one-line messages
for commits related to in-progress work makes sense. For a commit that you intend to be final or that encompasses a
significant amount or complex work, you should include a longer message.
Keep in mind that your commit should contain a ‘minimal coherent idea’ and have a quality commit message. See
Zulip docs Commit Discipline and Commit messages for details.
Here’s an example of a longer commit message that will be used for a pull request:
Integrate Fail2Ban.
Updates Zulip logging to put an unambiguous entry into the logs such
that fail2ban can be configured to look for these entries.
Tested on my local Ubuntu development server, but would appreciate
152
Chapter 26. Git & GitHub Guide
Zulip Documentation, Release 1.4.0
someone testing on a production install with more users.
Fixes #1755.
The first line is the summary. It’s a complete sentence, ending in a period. It uses a present-tense action verb,
“Integrate”, rather than “Integrates” or “Integrating”.
The following paragraphs are full prose and explain why and how the change was made. It explains what testing was
done and asks specifically for further testing in a more production-like environment.
The final paragraph indicates that this commit addresses and fixes issue #1755. When you submit your pull request,
GitHub will detect and link this reference to the appropriate issue. Once your commit is merged into zulip/master,
GitHub will automatically close the referenced issue. See Closing issues via commit messages for details.
Make as many commits as you need to to address the issue or implement your feature.
Push your commits to GitHub
As you’re working, it’s a good idea to frequently push your changes to GitHub. This ensures your work is backed up
should something happen to your local machine and allows others to follow your progress. It also allows you to work
from multiple computers without losing work.
Pushing to a feature branch is just like pushing to master:
$ git push origin <branch-name>
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (6/6), 658 bytes | 0 bytes/s, done.
Total 6 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3), completed with 1 local objects.
To git@github.com:christi3k/zulip.git
issue-demo -> issue-demo
* [new branch]
If you want to see what git will do without actually performing the push, add the -n (dry-run) option: git push
-n origin <branch-name>. If everything looks good, re-run the push command without -n.
If the feature branch does not already exist on GitHub, it will be created when you push and you’ll see * [new
branch] in the command output.
Examine and tidy your commit history
Examining your commit history prior to submitting your pull request is a good idea. Is it tidy such that each commit
represents a minimally coherent idea (see commit discipline)? Do your commit messages follow Zulip’s style? Will
the person reviewing your commit history be able to clearly understand your progression of work?
On the command line, you can use the git log command to display an easy to read list of your commits:
$ git log --all --graph --oneline --decorate
* 4f8d75d (HEAD -> 1754-docs-add-git-workflow) docs: Add details about configuring
˓→Travis CI.
* bfb2433 (origin/1754-docs-add-git-workflow) docs: Add section for keeping fork up˓→to-date to Git Guide.
* 4fe10f8 docs: Add sections for creating and configuring fork to Git Guide.
* 985116b docs: Add graphic client recs to Git Guide.
* 3c40103 docs: Add stubs for remaining Git Guide sections.
26.6. Using Git as you work
153
Zulip Documentation, Release 1.4.0
* fc2c01e docs: Add git guide quickstart.
| * f0eaee6 (upstream/master) bug: Fix traceback in get_missed_message_token_from_
˓→address().
Alternatively, use your graphical client to view the history for your feature branch.
If you need to update any of your commits, you can do so with an interactive rebase. Common reasons to use an
interactive rebase include:
• squashing several commits into fewer commits
• splitting a single commit into two or more
• rewriting one or more commit messages
There is ample documentation on how to rebase, so we won’t go into details here. We recommend starting with
GitHub’s help article on rebasing and then consulting Git’s documentation for git-rebase if you need more details.
If all you need to do is edit the commit message for your last commit, you can do that with git commit --amend.
See Git Basics - Undoing Things for details on this and other useful commands.
Force-push changes to GitHub after you’ve altered your history
Any time you alter history for commits you have already pushed to GitHub, you’ll need to prefix the name of your
branch with a +. Without this, your updates will be rejected with a message such as:
$ git push origin 1754-docs-add-git-workflow
To git@github.com:christi3k/zulip.git
! [rejected] 1754-docs-add-git-workflow -> 1754-docs-add-git-workflow (non-fast˓→forward)
error: failed to push some refs to 'git@github.com:christi3k/zulip.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
Re-running the command with +<branch> allows the push to continue by re-writing the history for the remote
repository:
$ git push origin +1754-docs-add-git-workflow
Counting objects: 12, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (12/12), done.
Writing objects: 100% (12/12), 3.71 KiB | 0 bytes/s, done.
Total 12 (delta 8), reused 0 (delta 0)
remote: Resolving deltas: 100% (8/8), completed with 2 local objects.
To git@github.com:christi3k/zulip.git
+ 2d49e2d...bfb2433 1754-docs-add-git-workflow -> 1754-docs-add-git-workflow (forced
˓→update)
This is perfectly okay to do on your own feature branches, especially if you’re the only one making changes to the
branch. If others are working along with you, they might run into complications when they retrieve your changes
because anyone who has based their changes off a branch you rebase will have to do a complicated rebase.
154
Chapter 26. Git & GitHub Guide
Zulip Documentation, Release 1.4.0
Create a pull request
When you’re ready for feedback, submit a pull request. At Zulip we recommend submitting work-in-progress pull
requests early and often. This allows you to get feedback and help with your bugfix or feature. Prefix work-in-progress
pull requests with [WIP].
Pull requests are a feature specific to GitHub. They provide a simpler, web-based way to submit your work (often
called “patches”) to a project. It’s called a pull request because you’re asking the project to pull changes from your
fork.
If you’re unfamiliar with how to create a pull request, checkout GitHub’s documentation on creating a pull request
from a fork. You might also find GitHub’s article about pull requests helpful.
Step 1: Update your branch with git rebase
The best way to update your branch is with git fetch and git rebase. Do not use git pull or git merge
as this will create merge commits. See keep your fork up to date for details.
Here’s an example (you would replace issue-123 with the name of your feature branch):
$ git checkout issue123
Switched to branch 'issue-123'
$ git fetch upstream
remote: Counting objects: 69, done.
remote: Compressing objects: 100% (23/23), done.
remote: Total 69 (delta 49), reused 39 (delta 39), pack-reused 7
Unpacking objects: 100% (69/69), done.
From https://github.com/zulip/zulip
69fa600..43e21f6 master
-> upstream/master
$ git rebase upstream/master
First, rewinding head to replay your work on top of it...
Applying: troubleshooting tip about provisioning
Step 2: Push your updated branch to your remote fork
Once you’ve updated your local feature branch, push the changes to GitHub:
$ git push origin issue-123
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (6/6), 658 bytes | 0 bytes/s, done.
Total 6 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3), completed with 1 local objects.
To git@github.com:christi3k/zulip.git
+ 2d49e2d...bfb2433 issue-123 -> issue-123
If your push is rejected with error failed to push some refs then you need to prefix the name of your branch with a +:
$ git push origin +issue-123
Counting objects: 6, done.
Delta compression using up to 4 threads.
26.7. Create a pull request
155
Zulip Documentation, Release 1.4.0
Compressing objects: 100% (4/4), done.
Writing objects: 100% (6/6), 658 bytes | 0 bytes/s, done.
Total 6 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3), completed with 1 local objects.
To git@github.com:christi3k/zulip.git
+ 2d49e2d...bfb2433 issue-123 -> issue-123 (forced update)
This is perfectly okay to do on your own feature branches, especially if you’re the only one making changes to the
branch. If others are working along with you, they might run into complications when they retrieve your changes
because anyone who has based their changes off a branch you rebase will have to do a complicated rebase.
Step 3: Open the pull request
If you’ve never created a pull request or need a refresher, take a look at GitHub’s article creating a pull request from a
fork. We’ll briefly review the process here.
The first step in creating a pull request is to use your web browser to navigate to your fork of Zulip. Sign in to GitHub
if you haven’t already.
Next, navigate to the branch you’ve been working on. Do this by clicking on the Branch button and selecting the
relevant branch. Finally, click the New pull request button.
Alternatively, if you’ve recently pushed to your fork, you will see a green Compare & pull request button.
You’ll see the Open a pull request page:
156
Chapter 26. Git & GitHub Guide
Zulip Documentation, Release 1.4.0
Provide a title and first comment for your pull request. When ready, click the green Create pull request to submit the
pull request.
Note: Pull request titles are different from commit messages. Commit messages can be edited with git commit
--amend, git rebase -i, etc., while the title of a pull request can only be edited via GitHub.
Update a pull request
As you get make progress on your feature or bugfix, your pull request, once submitted, will be updated each time you
push commits to your remote branch. This means you can keep your pull request open as long as you need, rather than
closing and opening new ones for the same feature or bugfix.
It’s a good idea to keep your pull request mergeable with Zulip upstream by frequently fetching, rebasing, and pushing
changes. See keep your fork up to date for details. You might also find this excellent article How to Rebase a Pull
Request helpful.
And, as you address review comments others have made, we recommend posting a follow-up comment in which you:
a) ask for any clarifications you need, b) explain to the reviewer how you solved any problems they mentioned, and c)
ask for another review.
Collaborate
Fetch another contributor’s branch
What happens when you would like to collaborate with another contributor and they have work-in-progress on their
own fork of Zulip? No problem! Just add their fork as a remote and pull their changes.
$ git remote add <username> https://github.com/<username>/zulip.git
$ git fetch <username>
Now you can checkout their branch just like you would any other. You can name the branch anything you want, but
using both the username and branch name will help you keep things organized.
$ git checkout -b <username>-<branchname>
Checkout a pull request locally
Just as you can checkout any user’s branch locally, you can also checkout any pull request locally. GitHub provides a
special syntax (details) for this since pull requests are specific to GitHub rather than Git.
First, fetch and create a branch for the pull request, replacing ID and BRANCHNAME with the ID of the pull request
and your desired branch name:
$ git fetch upstream pull/ID/head:BRANCHNAME
Now switch to the branch:
$ git checkout BRANCHNAME
Now you work on this branch as you would any other.
26.8. Update a pull request
157
Zulip Documentation, Release 1.4.0
Review changes
Changes on (local) working tree
Display changes between index and working tree (what is not yet staged for commit):
$ git diff
Display changes between index and last commit (what you have staged for commit):
$ git diff --cached
Display changes in working tree since last commit (changes that are staged as well as ones that are not):
$ git diff HEAD
Changes within branches
Use any git-ref to compare changes between two commits on the current branch.
Display changes between commit before last and last commit:
$ git diff HEAD^ HEAD
Display changes between two commits using their hashes:
$ git diff e2f404c 7977169
Changes between branches
Display changes between tip of topic branch and tip of master branch:
$ git diff topic master
Display changes that have occurred on master branch since topic branch was created:
$ git diff topic...master
Display changes you’ve committed so far since creating a branch from upstream/master:
$ git diff upstream/master...HEAD
Get and stay out of trouble
Git is a powerful yet complex version control system. Even for contributors experienced at using version control, it can
be confusing. The good news is that nearly all Git actions add information to the Git database, rather than removing
it. As such, it’s hard to make Git perform actions that you can’t undo. However, git can’t undo what it doesn’t know
about, so it’s a good practice to frequently commit your changes and frequently push your commits to your remote
repository.
158
Chapter 26. Git & GitHub Guide
Zulip Documentation, Release 1.4.0
Undo a merge commit
A merge commit is a special type of commit that has two parent commits. It’s created by Git when you merge
one branch into another and the last commit on your current branch is not a direct ancestor of the branch you are
trying to merge in. This happens quite often in a busy project like Zulip where there are many contributors because
upstream/zulip will have new commits while you’re working on a feature or bugfix. In order for Git to merge your
changes and the changes that have occurred on zulip/upstream since you first started your work, it must perform a
three-way merge and create a merge commit.
Merge commits aren’t bad, however, Zulip don’t use them. Instead Zulip uses a forked-repo, rebase-oriented workflow.
A merge commit is usually created when you’ve run git pull or git merge. You’ll know you’re creating a
merge commit if you’re prompted for a commit message and the default is something like this:
Merge branch 'master' of https://github.com/zulip/zulip
#
#
#
#
#
Please enter a commit message to explain why this merge is necessary,
especially if it merges an updated upstream into a topic branch.
Lines starting with '#' will be ignored, and an empty message aborts
the commit.
And the first entry for git log will show something like:
commit e5f8211a565a5a5448b93e98ed56415255546f94
Merge: 13bea0e e0c10ed
Author: Christie Koehler <ck@christi3k.net>
Date:
Mon Oct 10 13:25:51 2016 -0700
Merge branch 'master' of https://github.com/zulip/zulip
Some graphical Git clients may also create merge commits.
To undo a merge commit, first run git reflog to identify the commit you want to roll back to:
$ git reflog
e5f8211 HEAD@{0}: pull upstream master: Merge made by the 'recursive' strategy.
13bea0e HEAD@{1}: commit: test commit for docs.
Reflog output will be long. The most recent git refs will be listed at the top. In the example above e5f8211
HEAD@{0}: is the merge commit made automatically by git pull and 13bea0e HEAD@{1}: is the last commit I made before running git pull, the commit that I want to rollback to.
Once you’d identified the ref you want to revert to, you can do so with git reset:
$ git reset --hard 13bea0e
HEAD is now at 13bea0e test commit for docs.
Important: git reset --hard <commit> will discard all changes in your working directory and index since
the commit you’re resetting to with . This is the main way you can lose work in Git. If you need to keep any changes
that are in your working directory or that you have committed, use git reset --merge <commit> instead.
You can also use the relative reflog HEAD@{1} instead of the commit hash, just keep in mind this changes as you run
git commands.
Now when I look at the git reflog, I see the tip of my branch is pointing to my last commit 13bea0e before the merge:
26.11. Get and stay out of trouble
159
Zulip Documentation, Release 1.4.0
$ git reflog
13bea0e HEAD@{2}: reset: moving to HEAD@{1}
e5f8211 HEAD@{3}: pull upstream master: Merge made by the 'recursive' strategy.
13bea0e HEAD@{4}: commit: test commit for docs.
And the first entry git log shows is this:
commit 13bea0e40197b1670e927a9eb05aaf50df9e8277
Author: Christie Koehler <ck@christi3k.net>
Date:
Mon Oct 10 13:25:38 2016 -0700
test commit for docs.
Restore a lost commit
We’ve mentioned you can use git reset --hard to rollback to a previous commit. What if you run git reset
--hard and then realize you actually need one or more of the commits you just discarded? No problem, you can
restore them with git cherry-pick (docs).
For example, let’s say you just committed “some work” and your git log looks like this:
* 67aea58 (HEAD -> master) some work
* 13bea0e test commit for docs.
You then mistakenly run git reset --hard 13bea0e:
$ git reset --hard 13bea0e
HEAD is now at 13bea0e test commit for docs.
$ git log
* 13bea0e (HEAD -> master) test commit for docs.
And then realize you actually needed to keep commit 67aea58. First, use git reflog to confirm that commit you
want to restore and then run git cherry-pick <commit>:
$ git reflog
13bea0e HEAD@{0}: reset: moving to 13bea0e
67aea58 HEAD@{1}: commit: some work
$ git cherry-pick 67aea58
[master 67aea58] some work
Date: Thu Oct 13 11:51:19 2016 -0700
1 file changed, 1 insertion(+)
create mode 100644 test4.txt
Recover from a git rebase failure
One situation in which git rebase will fail and require you to intervene is when your change, which git will try to
re-apply on top of new commits from which ever branch you are rebasing on top of, is to code that has been changed
by those new commits.
For example, while I’m working on a file, another contributor makes a change to that file, submits a pull request and
has their code merged into master. Usually this is not a problem, but in this case the other contributor made a change
160
Chapter 26. Git & GitHub Guide
Zulip Documentation, Release 1.4.0
to a part of the file I also want to change. When I try to bring my branch up to date with git fetch and then git
rebase upstream/master, I see the following:
First, rewinding head to replay your work on top of it...
Applying: test change for docs
Using index info to reconstruct a base tree...
M
README.md
Falling back to patching base and 3-way merge...
Auto-merging README.md
CONFLICT (content): Merge conflict in README.md
error: Failed to merge in the changes.
Patch failed at 0001 test change for docs
The copy of the patch that failed is found in: .git/rebase-apply/patch
When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".
This message tells me that Git was not able to apply my changes to README.md after bringing in the new commits
from upstream/master.
Running git status also gives me some information:
rebase in progress; onto 5ae56e6
You are currently rebasing branch 'docs-test' on '5ae56e6'.
(fix conflicts and then run "git rebase --continue")
(use "git rebase --skip" to skip this patch)
(use "git rebase --abort" to check out the original branch)
Unmerged paths:
(use "git reset HEAD <file>..." to unstage)
(use "git add <file>..." to mark resolution)
both modified:
README.md
no changes added to commit (use "git add" and/or "git commit -a")
To fix, open all the files with conflicts in your editor and decide which edits should be applied. Git uses standard
conflict-resolution (<<<<<<<, =======, and >>>>>>>) markers to indicate where in files there are conflicts.
Once you’ve done that, save the file(s), stage them with git add and then continue the rebase with git rebase
--continue:
$ git add README.md
$ git rebase --continue
Applying: test change for docs
For help resolving merge conflicts, see basic merge conflicts, advanced merging, and/or GitHub’s help on how to
resolve a merge conflict.
Working from multiple computers
Working from multiple computers with Zulip and Git is fine, but you’ll need to pay attention and do a bit of work to
ensure all of your work is readily available.
Recall that most Git operations are local. When you commit your changes with git commit they are safely stored
in your local Git database only. That is, until you push the commits to GitHub, they are only available on the computer
26.11. Get and stay out of trouble
161
Zulip Documentation, Release 1.4.0
where you committed them.
So, before you stop working for the day, or before you switch computers, push all of your commits to GitHub with
git push:
$ git push origin <branchname>
When you first start working on a new computer, you’ll clone the Zulip repository and connect it to Zulip upstream.
A clone retrieves all current commits, including the ones you pushed to GitHub from your other computer.
But if you’re switching to another computer on which you have already cloned Zulip, you need to update your local
Git database with new refs from your GitHub fork. You do this with git fetch:
$ git fetch <usermame>
Ideally you should do this before you have made any commits on the same branch on the second computer. Then you
can git merge on whichever branch you need to update:
$ git checkout <my-branch>
Switched to branch '<my-branch>'
$ git merge origin/master
If you have already made commits on the second computer that you need to keep, you’ll need to use git log
FETCH_HEAD to identify that hashes of the commits you want to keep and then git cherry-pick <commit>
those commits into whichever branch you need to update.
Zulip-specific tools
This section will document the zulip-specific git tools contributors will find helpful.
Set up git repo script
In the tools directory of zulip/zulip you’ll find a bash script setup-git-repo. This script installs the Zulip
pre-commit hook. This hook will run each time you git commit to automatically run linters, etc. The hook passes
no matter the result of the linter, but you should still pay attention to any notices or warnings it displays.
It’s simple to use. Make sure you’re in the clone of zulip and run the following:
$ ./tools/setup-git-repo
The script doesn’t produce any output if successful. To check that the hook has been installed, print a directory listing
for .git/hooks and you should see something similar to:
$ ls -l .git/hooks
pre-commit -> ../../tools/pre-commit
Reset to pull request
tools/reset-to-pull-request is a short-cut for checking out a pull request locally. It works slightly differently from the method described above in that it does not create a branch for the pull request checkout.
This tool checks for uncommitted changes, but it will move the current branch using git reset --hard.
Use with caution.
162
Chapter 26. Git & GitHub Guide
Zulip Documentation, Release 1.4.0
First, make sure you are working in a branch you want to move (in this example, we’ll use the local master branch).
Then run the script with the ID number of the pull request as the first argument.
$ git checkout master
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
$ ./tools/reset-to-pull-request 1900
+ request_id=1900
+ git fetch upstream pull/1900/head
remote: Counting objects: 159, done.
remote: Compressing objects: 100% (17/17), done.
remote: Total 159 (delta 94), reused 91 (delta 91), pack-reused 51
Receiving objects: 100% (159/159), 55.57 KiB | 0 bytes/s, done.
Resolving deltas: 100% (113/113), completed with 54 local objects.
From https://github.com/zulip/zulip
refs/pull/1900/head -> FETCH_HEAD
* branch
+ git reset --hard FETCH_HEAD
HEAD is now at 2bcd1d8 troubleshooting tip about provisioning
Fetch a pull request and rebase
tools/fetch-rebase-pull-request is a short-cut for checking out a pull request locally in its own branch
and then updating it with any changes from upstream/master with git rebase.
First, make sure you are working in branch master. Then run the script with the ID number of the pull request as the
first argument.
$ tools/fetch-rebase-pull-request 1913
+ request_id=1913
+ git fetch upstream pull/1913/head
remote: Counting objects: 4, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (4/4), done.
From https://github.com/zulip/zulip
refs/pull/1913/head -> FETCH_HEAD
* branch
+ git checkout upstream/master -b review-1913
Branch review-1913 set up to track remote branch master from upstream.
Switched to a new branch 'review-1913'
+ git reset --hard FETCH_HEAD
HEAD is now at 99aa2bf Add provision.py fails issue in common erros
+ git pull --rebase
Current branch review-1913 is up to date.
26.12. Zulip-specific tools
163
Zulip Documentation, Release 1.4.0
164
Chapter 26. Git & GitHub Guide
CHAPTER
27
Version control
Commit Discipline
We follow the Git project’s own commit discipline practice of “Each commit is a minimal coherent idea”. This
discipline takes a bit of work, but it makes it much easier for code reviewers to spot bugs, and makes the commit
history a much more useful resource for developers trying to understand why the code works the way it does, which
also helps a lot in preventing bugs.
Coherency requirements for any commit:
• It should pass tests (so test updates needed by a change should be in the same commit as the original change,
not a separate “fix the tests that were broken by the last commit” commit).
• It should be safe to deploy individually, or comment in detail in the commit message as to why it isn’t (maybe
with a [manual] tag). So implementing a new API endpoint in one commit and then adding the security checks
in a future commit should be avoided – the security checks should be there from the beginning.
• Error handling should generally be included along with the code that might trigger the error.
• TODO comments should be in the commit that introduces the issue or functionality with further work required.
When you should be minimal:
• Significant refactorings should be done in a separate commit from functional changes.
• Moving code from one file to another should be done in a separate commits from functional changes or even
refactoring within a file.
• 2 different refactorings should be done in different commits.
• 2 different features should be done in different commits.
• If you find yourself writing a commit message that reads like a list of somewhat dissimilar things that you did,
you probably should have just done 2 commits.
When not to be overly minimal:
• For completely new features, you don’t necessarily need to split out new commits for each little subfeature of
the new feature. E.g. if you’re writing a new tool from scratch, it’s fine to have the initial tool have plenty of
165
Zulip Documentation, Release 1.4.0
options/features without doing separate commits for each one. That said, reviewing a 2000-line giant blob of
new code isn’t fun, so please be thoughtful about submitting things in reviewable units.
• Don’t bother to split back end commits from front end commits, even though the backend can often be coherent
on its own.
Other considerations:
• Overly fine commits are easily squashed, but not vice versa, so err toward small commits, and the code reviewer
can advise on squashing.
• If a commit you write doesn’t pass tests, you should usually fix that by amending the commit to fix the bug, not
writing a new “fix tests” commit on top of it.
Zulip expects you to structure the commits in your pull requests to form a clean history before we will merge them;
it’s best to write your commits following these guidelines in the first place, but if you don’t, you can always fix your
history using git rebase -i.
Never mix multiple changes together in a single commit, but it’s great to include several related changes, each in their
own commit, in a single pull request. If you notice an issue that is only somewhat related to what you were working
on, but you feel that it’s too minor to create a dedicated pull request, feel free to append it as an additional commit in
the pull request for your main project (that commit should have a clear explanation of the bug in its commit message).
This way, the bug gets fixed, but this independent change is highlighted for reviewers. Or just create a dedicated pull
request for it. Whatever you do, don’t squash unrelated changes together in a single commit; the reviewer will ask you
to split the changes out into their own commits.
It can take some practice to get used to writing your commits with a clean history so that you don’t spend much time
doing interactive rebases. For example, often you’ll start adding a feature, and discover you need to a refactoring partway through writing the feature. When that happens, we recommend stashing your partial feature, do the refactoring,
commit it, and then finish implementing your feature.
Commit Messages
First, check out these examples of commits with good commit messages. There are a few mistakes we see often that
make commit messages less useful than they could be.
• The first line of commit messages should be written in the imperative and be kept relatively short while concisely
explaining what the commit does. And it should be clear about what part of the code is affected. For example:
Good:
gather_subscriptions: Fix exception handling bad input.
zjsunit: Fix running stream_data and node tests individually.
Add GitLab integration.
Bad:
bugfix
gather_subscriptions was broken
fixing exception when given bad input
fix bug #234.
• Use present-tense action verbs in your commit messages.
Bad:
gather_subscriptions: Fixing exception when given bad input.
gather_subscriptions: Fixed exception when given bad input.
166
Chapter 27. Version control
Zulip Documentation, Release 1.4.0
• Please use a complete sentence in the summary, ending with a period.
• The rest of the commit message should be written in full prose and explain why and how the change was made. If
the commit makes performance improvements, you should generally include some rough benchmarks showing
that it actually improves the performance.
• When you fix a GitHub issue, mark that you’ve fixed the issue in your commit message so that the issue is
automatically closed when your code is merged. Zulip’s preferred style for this is to have the final paragraph of
the commit message read e.g. “Fixes: #123.”
• Any paragraph content in the commit message should be line-wrapped to less than 76 characters per line, so that
your commit message will be reasonably readable in git log in a normal terminal.
• In your commit message, you should describe any manual testing you did in addition to running the automated
tests, and any aspects of the commit that you think are questionable and you’d like special attention applied to.
27.2. Commit Messages
167
Zulip Documentation, Release 1.4.0
168
Chapter 27. Version control
CHAPTER
28
Code style and conventions
Be consistent!
Look at the surrounding code, or a similar part of the project, and try to do the same thing. If you think the other code
has actively bad style, fix it (in a separate commit).
When in doubt, send an email to zulip-devel@googlegroups.com with your question.
Lint tools
You can run them all at once with
./tools/lint
You can set this up as a local Git commit hook with
``tools/setup-git-repo``
The Vagrant setup process runs this for you.
lint runs many lint checks in parallel, including
• JavaScript (ESLint)
• Python (Pyflakes)
• templates
• Puppet configuration
• custom checks (e.g. trailing whitespace and spaces-not-tabs)
169
Zulip Documentation, Release 1.4.0
Secrets
Please don’t put any passwords, secret access keys, etc. inline in the code. Instead, use the get_secret function in
zproject/settings.py to read secrets from /etc/zulip/secrets.conf.
Dangerous constructs
Misuse of database queries
Look out for Django code like this:
[Foo.objects.get(id=bar.x.id)
for bar in Bar.objects.filter(...)
if bar.baz < 7]
This will make one database query for each Bar, which is slow in production (but not in local testing!). Instead of a
list comprehension, write a single query using Django’s QuerySet API.
If you can’t rewrite it as a single query, that’s a sign that something is wrong with the database schema. So don’t defer
this optimization when performing schema changes, or else you may later find that it’s impossible.
UserProfile.objects.get() / Client.objects.get / etc.
In our Django code, never do direct UserProfile.objects.get(email=foo) database queries. Instead
always use get_user_profile_by_{email,id}. There are 3 reasons for this:
1. It’s guaranteed to correctly do a case-inexact lookup
2. It fetches the user object from remote cache, which is faster
3. It always fetches a UserProfile object which has been queried using .select_related(), and thus will perform well
when one later accesses related models like the Realm.
Similarly we have get_client and get_stream functions to fetch those commonly accessed objects via remote
cache.
Using Django model objects as keys in sets/dicts
Don’t use Django model objects as keys in sets/dictionaries – you will get unexpected behavior when dealing with
objects obtained from different database queries:
For
example,
UserProfile.objects.only("id").get(id=17) in set([UserProfile.
objects.get(id=17)]) is False
You should work with the IDs instead.
user_profile.save()
You should always pass the update_fields keyword argument to .save() when modifying an existing Django model
object. By default, .save() will overwrite every value in the column, which results in lots of race conditions where
unrelated changes made by one thread can be accidentally overwritten by another thread that fetched its UserProfile
object before the first thread wrote out its change.
170
Chapter 28. Code style and conventions
Zulip Documentation, Release 1.4.0
Using raw saves to update important model objects
In most cases, we already have a function in zephyr/lib/actions.py with a name like do_activate_user that will correctly
handle lookups, caching, and notifying running browsers via the event system about your change. So please check
whether such a function exists before writing new code to modify a model object, since your new code has a good
chance of getting at least one of these things wrong.
Naive datetime objects
Python allows datetime objects to not have an associated timezone, which can cause time-related bugs that are hard to
catch with a test suite, or bugs that only show up during daylight savings time.
Good ways to make timezone-aware datetimes are below. We import timezone function as from django.
utils.timezone import now as timezone_now and from django.utils.timezone import
utc as timezone_utc. When Django is not available, timezone_utc should be replaced with pytz.utc
below.
• timezone_now() when Django is available, such as in zerver/.
• datetime.now(tz=pytz.utc) when Django is not available, such as for bots and scripts.
• datetime.fromtimestamp(timestamp, tz=timezone_utc) if creating a datetime from a timestamp. This is also available as zerver.lib.timestamp.timestamp_to_datetime.
• datetime.strptime(date_string, format).replace(tzinfo=timezone_utc) if creating
a datetime from a formatted string that is in UTC.
Idioms that result in timezone-naive datetimes, and should be avoided, are datetime.now() and datetime.
fromtimestamp(timestamp) without a tz parameter, datetime.utcnow() and datetime.
utcfromtimestamp(), and datetime.strptime(date_string, format) without replacing the
tzinfo at the end.
Additional notes:
• Especially in scripts and puppet configuration where Django is not available, using time.time() to get
timestamps can be cleaner than dealing with datetimes.
• All datetimes on the backend should be in UTC, unless there is a good reason to do otherwise.
x.attr('zid') vs. rows.id(x)
Our message row DOM elements have a custom attribute zid which contains the numerical message ID. Don’t access
this directly as x.attr('zid') ! The result will be a string and comparisons (e.g. with <=) will give the wrong
result, occasionally, just enough to make a bug that’s impossible to track down.
You should instead use the id function from the rows module, as in rows.id(x). This returns a number. Even
in cases where you do want a string, use the id function, as it will simplify future code changes. In most contexts in
JavaScript where a string is needed, you can pass a number without any explicit conversion.
JavaScript var
Always declare JavaScript variables using var:
var x = ...;
28.4. Dangerous constructs
171
Zulip Documentation, Release 1.4.0
In a function, var is necessary or else x will be a global variable. For variables declared at global scope, this has no
effect, but we do it for consistency.
JavaScript has function scope only, not block scope. This means that a var declaration inside a for or if acts the
same as a var declaration at the beginning of the surrounding function. To avoid confusion, declare all variables
at the top of a function.
JavaScript for (i in myArray)
Don’t use it: [1], [2], [3]
jQuery global state
Don’t mess with jQuery global state once the app has loaded. Code like this is very dangerous:
$.ajaxSetup({ async: false });
$.get(...);
$.ajaxSetup({ async: true });
jQuery and the browser are free to run other code while the request is pending, which could perform other Ajax
requests with the altered settings.
Instead, switch to the more general $.ajax_ function, which can take options like async.
Translation tags
Remember to tag all user-facing strings for translation, whether they are in HTML templates or JavaScript editing the
HTML (e.g. error messages).
State and logs files
Do not write state and logs files inside the current working directory in the production environment. This will not
do what you expect, because the current working directory for the app changes every time we do a deploy. Instead,
hardcode a path in settings.py – see SERVER_LOG_PATH in settings.py for an example.
JS array/object manipulation
For generic functions that operate on arrays or JavaScript objects, you should generally use Underscore. We used
to use jQuery’s utility functions, but the Underscore equivalents are more consistent, better-behaved and offer more
choices.
A quick conversion table:
$.each → _.each (parameters to the callback reversed)
$.inArray → _.indexOf (parameters reversed)
$.grep → _.filter
$.map → _.map
$.extend → _.extend
There’s a subtle difference in the case of _.extend; it will replace attributes with undefined, whereas jQuery won’t:
172
Chapter 28. Code style and conventions
Zulip Documentation, Release 1.4.0
$.extend({foo: 2}, {foo: undefined});
_.extend({foo: 2}, {foo: undefined});
// yields {foo: 2}, BUT...
// yields {foo: undefined}!
Also, _.each does not let you break out of the iteration early by returning false, the way jQuery’s version does. If
you’re doing this, you probably want _.find, _.every, or _.any, rather than ‘each’.
Some Underscore functions have multiple names. You should always use the canonical name (given in large print in
the Underscore documentation), with the exception of _.any, which we prefer over the less clear ‘some’.
More arbitrary style things
General
Indentation is four space characters for Python, JS, CSS, and shell scripts. Indentation is two space characters for
HTML templates.
We never use tabs anywhere in source code we write, but we have some third-party files which contain tabs.
Keep third-party static files under the directory zephyr/static/third/, with one subdirectory per third-party
project.
We don’t have an absolute hard limit on line length, but we should avoid extremely long lines. A general guideline is:
refactor stuff to get it under 85 characters, unless that makes the code a lot uglier, in which case it’s fine to go up to
120 or so.
Whitespace guidelines:
• Put one space (or more for alignment) around binary arithmetic and equality operators.
• Put one space around each part of the ternary operator.
• Put one space between keywords like if and while and their associated open paren.
• Put one space between the closing paren for if and while-like constructs and the opening curly brace. Put the
curly brace on the same line unless doing otherwise improves readability.
• Put no space before or after the open paren for function calls and no space before the close paren for function
calls.
• For the comma operator and colon operator in languages where it is used for inline dictionaries, put no space
before it and at least one space after. Only use more than one space for alignment.
JavaScript
Don’t use == and != because these operators perform type coercions, which can mask bugs. Always use === and
!==.
End every statement with a semicolon.
if statements with no braces are allowed, if the body is simple and its extent is abundantly clear from context and
formatting.
Anonymous functions should have spaces before and after the argument list:
var x = function (foo, bar) { // ...
When calling a function with an anonymous function as an argument, use this style:
28.6. More arbitrary style things
173
Zulip Documentation, Release 1.4.0
$.get('foo', function (data) {
var x = ...;
// ...
});
The inner function body is indented one level from the outer function call. The closing brace for the inner function
and the closing parenthesis for the outer call are together on the same line. This style isn’t necessarily appropriate for
calls with multiple anonymous functions or other arguments following them.
Use
$(function () { ...
rather than
$(document).ready(function () { ...
and combine adjacent on-ready functions, if they are logically related.
The best way to build complicated DOM elements is a Mustache template like zephyr/static/templates/
message.handlebars. For simpler things you can use jQuery DOM building APIs like so:
var new_tr = $('<tr />').attr('id', zephyr.id);
Passing a HTML string to jQuery is fine for simple hardcoded things:
foo.append('<p id="selected">foo</p>');
but avoid programmatically building complicated strings.
We used to favor attaching behaviors in templates like so:
<p onclick="select_zephyr({{id}})">
but there are some reasons to prefer attaching events using jQuery code:
• Potential huge performance gains by using delegated events where possible
• When calling a function from an onclick attribute, this is not bound to the element like you might think
• jQuery does event normalization
Either way, avoid complicated JavaScript code inside HTML attributes; call a helper function instead.
HTML / CSS
Don’t use the style= attribute. Instead, define logical classes and put your styles in external files such as zulip.
css.
Don’t use the tag name in a selector unless you have to. In other words, use .foo instead of span.foo. We shouldn’t
have to care if the tag type changes in the future.
Don’t use inline event handlers (onclick=, etc. attributes). Instead, attach a jQuery event handler ($('#foo').
on('click', function () {...})) when the DOM is ready (inside a $(function () {...}) block).
Use this format when you have the same block applying to multiple CSS styles (separate lines for each selector):
selector1,
selector2 {
};
174
Chapter 28. Code style and conventions
Zulip Documentation, Release 1.4.0
Python
• Scripts should start with #!/usr/bin/env python and not #/usr/bin/python (the right Python may
not be installed in /usr/bin) or #/usr/bin/env python2.7 (bad for Python 3 compatibility). Don’t
put a shebang line on a Python file unless it’s meaningful to run it as a script. (Some libraries can also be run as
scripts, e.g. to run a test suite.)
• Scripts should be executed directly (./script.py), so that the interpreter is implicitly found from the shebang
line, rather than explicitly overridden (python script.py).
• The first import in a file should be from __future__ import absolute_import, per PEP 328
• Put all imports together at the top of the file, absent a compelling reason to do otherwise.
• Unpacking sequences doesn’t require list brackets:
[x, y] = xs
x, y = xs
# unnecessary
# better
• For string formatting, use x % (y,) rather than x % y, to avoid ambiguity if y happens to be a tuple.
• When selecting by id, don’t use foo.pk when you mean foo.id. E.g.
recipient = Recipient(type_id=huddle.pk, type=Recipient.HUDDLE)
should be written as
recipient = Recipient(type_id=huddle.id, type=Recipient.HUDDLE)
in case we ever change the primary keys.
Tests
All significant new features should come with tests. See testing.
Third party code
When adding new third-party packages to our codebase, please include “[third]” at the beginning of the commit
message. You don’t necessarily need to do this when patching third-party code that’s already in tree.
28.6. More arbitrary style things
175
Zulip Documentation, Release 1.4.0
176
Chapter 28. Code style and conventions
CHAPTER
29
Python static type checker (mypy)
mypy is a compile-time static type checker for Python, allowing optional, gradual typing of Python code. Zulip is
using mypy’s Python 2 compatible syntax for type annotations, which means that type annotations are written inside
comments that start with # type:. Here’s a brief example of the mypy syntax we’re using in Zulip:
user_dict = {} # type: Dict[str, UserProfile]
def get_user_profile_by_email(email):
# type: (str) -> UserProfile
... # Actual code of the function here
You can learn more about it at:
• The mypy cheat sheet is the best resource for quickly understanding how to write the PEP 484 type annotations
used by mypy correctly.
• The Python 2 type annotation syntax spec in PEP 484
• Using mypy with Python 2 code
The mypy type checker is run automatically as part of Zulip’s Travis CI testing process in the ‘static-analysis’ build.
type_debug.py
zerver/lib/type_debug.py has a useful decorator print_types. It prints the types of the parameters of
the decorated function and the return type whenever that function is called. This can help find out what parameter
types a function is supposed to accept, or if parameters with the wrong types are being passed to a function.
Here is an example using the interactive console:
>>> from zerver.lib.type_debug import print_types
>>>
>>> @print_types
... def func(x, y):
...
return x + y
177
Zulip Documentation, Release 1.4.0
...
>>> func(1.0, 2)
func(float, int) -> float
3.0
>>> func('a', 'b')
func(str, str) -> str
'ab'
>>> func((1, 2), (3,))
func((int, int), (int,)) -> (int, int, int)
(1, 2, 3)
>>> func([1, 2, 3], [4, 5, 6, 7])
func([int, ...], [int, ...]) -> [int, ...]
[1, 2, 3, 4, 5, 6, 7]
print_all prints the type of the first item of lists. So [int, ...] represents a list whose first element’s type is
int. Types of all items are not printed because a list can have many elements, which would make the output too large.
Similarly in dicts, one key’s type and the corresponding value’s type are printed. So {1:
'c'} will be printed as {int: str, ...}.
'a', 2:
'b', 3:
Zulip goals
Zulip is hoping to reach 100% of the codebase annotated with mypy static types, and then enforce that it stays that
way. Our current coverage is shown in Coveralls.
Installing mypy
If you installed Zulip’s development environment correctly, mypy should already be installed inside the Python 3
virtualenv at zulip-py3-venv (mypy only supports Python 3). If it isn’t installed (e.g. because you haven’t
reprovisioned recently), you can run tools/install-mypy to install it.
Running mypy on Zulip’s code locally
To run mypy on Zulip’s python code, run the command:
tools/run-mypy
It will output errors in the same style as a compiler would. For example, if your code has a type error like this:
foo = 1
foo = '1'
you’ll get an error like this:
test.py: note: In function "test":
test.py:200: error: Incompatible types in assignment (expression has type "str",
˓→variable has type "int")
If you need help interpreting or debugging mypy errors, please feel free to mention @sharmaeklavya2 or @timabbott
on your pull request (or email zulip-devel@googlegroups.com) to get help; we’d love to both build a great troubleshooting guide in this doc and also help contribute improvements to error messages upstream.
178
Chapter 29. Python static type checker (mypy)
Zulip Documentation, Release 1.4.0
Since mypy is a new tool under rapid development and occasionally makes breaking changes, Zulip is using a pinned
version of mypy from its git repository rather than tracking the (older) latest mypy release on PyPI.
Excluded files
Since several Python files in Zulip’s code don’t pass mypy’s checks (even for unannotated code) right now, a list of
files to be excluded from the check for CI is present in tools/run-mypy.
To run mypy on all Python files, ignoring the exclude list, you can pass the --all option to tools/run-mypy.
tools/run-mypy --all
If you type annotate some of those files so that they pass without errors, please remove them from the exclude list.
Mypy is there to find bugs in Zulip before they impact users
For the purposes of Zulip development, you can treat mypy like a much more powerful linter that can catch a wide
range of bugs. If, after running tools/run-mypy on your Zulip branch, you get mypy errors, it’s important to get
to the bottom of the issue, not just do something quick to silence the warnings. Possible explanations include:
• A bug in any new type annotations you added.
• A bug in the existing type annotations.
• A bug in Zulip!
• Some Zulip code is correct but confusingly reuses variables with different types.
• A bug in mypy (though this is increasingly rare as mypy is now fairly mature as a project).
Each explanation has its own solution, but in every case the result should be solving the mypy warning in a way
that makes the Zulip codebase better. If you need help understanding an issue, please feel free to mention @sharmaeklavya2 or @timabbott on the relevant pull request or issue on GitHub.
If you think you have found a bug in Zulip or mypy, inform the zulip developers by opening an issue on Zulip’s GitHub
repository or posting on zulip-devel. If it’s indeed a mypy bug, we can help with reporting it upstream.
Annotating strings
In Python 3, strings can have non-ASCII characters without any problems. Such characters are required to support
languages which use non-latin scripts like Japanese and Hindi. They are also needed to support special characters like
mathematical symbols, musical symbols, etc. In Python 2, however, str generally doesn’t work well with non-ASCII
characters. That’s why unicode was introduced in Python 2.
But there are problems with the unicode and str system. Implicit conversions between str and unicode use the
ascii codec, which fails on strings containing non-ASCII characters. Such errors are hard to detect by people who
always write in English. To minimize such implicit conversions, we should have a strict separation between str and
unicode in Python 2. It might seem that using unicode everywhere will solve all problems, but unfortunately it
doesn’t. This is because some parts of the standard library and the Python language (like keyword argument unpacking)
insist that parameters passed to them are str.
To make our code work correctly in Python 2, we have to identify strings which contain data which could come from
non-ASCII sources like stream names, people’s names, domain names, content of messages, emails, etc. These strings
29.5. Excluded files
179
Zulip Documentation, Release 1.4.0
should be unicode. We also have to identify strings which should be str like Exception names, attribute names,
parameter names, etc.
Mypy can help with this. We just have to annotate each string as either str or unicode and mypy’s static type
checking will tell us if we are incorrectly mixing the two. However, unicode is not defined in Python 3. We
want our code to be Python 3 compatible in the future. This can be achieved using ‘typing.Text’, a Python 2 and 3
compatibility type.
typing.Text is defined as str in Python 3 and as unicode in Python 2. We’ll be using Text (instead of
unicode) and str to annotate strings in Zulip’s code. We follow the style of doing from typing import
Text and using Text for annotation instead of doing import typing and using typing.Text for annotation,
because Text is used so extensively for type annotations that we don’t need to be that verbose.
Sometimes you’ll find that you have to convert strings from one type to another. zerver/lib/str_utils.py
has utility functions to help with that. It also has documentation (in docstrings) which explains the right way to use
them.
180
Chapter 29. Python static type checker (mypy)
CHAPTER
30
Reviewing Zulip server code
This document is a brief discussion of what we look for when reviewing contributions to Zulip. It’s meant partially
for developers who want to get their code merged faster, and partially for developers who have made successful pull
requests already and would like to start participating in code review.
Things to look for
• The Travis CI build. The tests need to pass. One can investigate any failures and figure out what to fix by
clicking on a red X next to the commit hash or the Detail links on a pull request. (Example: in #1219, click the
red X next to f1f474e to see the build jobs for that commit, at least one of which has failed. Click on the link
for Travis continuous integrations details to see the tests Travis ran on that commit, at least one of which failed,
and go to one of the failing tests to see the error.) Since Coveralls’s data on changes isn’t always accurate, one
should look at the test coverage situation directly.
• Technical design. There are a lot of considerations here: security, migration paths/backwards compatibility,
cost of new dependencies, interactions with features, speed of performance, API changes. Security is especially
important and worth thinking about carefully with any changes to security-sensitive code like views.
• User interface and visual design. If frontend changes are involved, the reviewer will check out the code, play
with the new UI, and verify it for both quality and consistency with the rest of the Zulip UI. We highly encourage
posting screenshots to save reviewers time in getting a feel for what the feature looks like – you’ll get a quicker
response that way.
• Error handling. The code should always check for invalid user input. User-facing error messages should be
clear and when possible be actionable (it should be obvious to the user what they need to do in order to correct
the problem).
• Testing. The tests should validate that the feature works correctly, and specifically test for common error conditions, bad user input, and potential bugs that are likely for the type of change being made. Tests that exclude
whole classes of potential bugs are preferred when possible (e.g., the common test suite test_bugdown.py
between the frontend and backend Markdown processors or the GetEventsTest test for buggy race condition
handling).
181
Zulip Documentation, Release 1.4.0
Backend: we are trying to maintain ~100% test coverage on the backend, so backend changes should have
negative tests for the various error conditions. Frontend: If the feature involves frontend changes, there should
be frontend tests. See the test writing documentation for more details.
• mypy annotations. New functions should be annotated using mypy and existing annotations should be updated.
Use of Any, ignore, and unparameterized containser should be limited to cases where a more precise type
cannot be specified.
• Translation. Make sure that the strings are marked for translation.
• Clear function, argument, variable, and test names. Every new piece of Zulip code will be read many times
by other developers, and future developers will grep for relevant terms when researching a problem, so it’s
important that variable names communicate clearly the purpose of each piece of the codebase.
• Duplicated code. Code duplication is a huge source of bugs in large projects and makes the codebase difficult to
understand, so we avoid significant code duplication wherever possible. Sometimes avoiding code duplication
involves some refactoring of existing code; if so, that should usually be done as its own series of commits (not
squashed into other changes or left as a thing to do later). That series of commits can be in the same pull request
as the feature that they support, and we recommend ordering the history of commits so that the refactoring
comes before the feature. That way, it’s easy to merge the refactoring (and minimize risk of merge conflicts) if
there are still user experience issues under discussion for the feature itself.
• Completeness. For refactorings, verify that the changes are complete. Usually one can check that efficiently
using git grep, and it’s worth it, as we very frequently find issues by doing so.
• Documentation updates. If this changes how something works, does it update the documentation in a corresponding way? If it’s a new feature, is it documented, and documented in the right place?
• Good comments. It’s often worth thinking about whether explanation in a commit message or pull request
discussion should be included in a comment, /docs, or other documentation. But it’s better yet if verbose
explanation isn’t needed. We prefer writing code that is readable without explanation over a heavily commented
codebase using lots of clever tricks.
• Coding style. See the Zulip code-style documentation for details. Our goal is to have as much of this as possible
verified via the linters and tests, but there’s always going to be unusual forms of Python/JavaScript style that our
tools don’t check for.
• Clear commit messages. See the Zulip version control documentation for details on what we look for.
Tooling
To make it easier to review pull requests, use our git tool tools/fetch-rebase-pull-request to check out
a pull request locally and rebase it against master. If a pull request just needs a little fixing to make it mergeable, feel
free to do that in a new commit, then push your branch to GitHub and mention the branch in a comment on the pull
request. That’ll save the maintainer time and get the PR merged quicker.
Additional Resources
We also strongly recommend reviewers to go through the following resources.
• The Gentle Art of Patch Review article by Sarah Sharp
• Zulip & Good Code Review article by Sumana Harihareswara
• Code Review - A consolidation of advice and stuff from the sinternet article by James J. Porter
• Zulip Code of Conduct
182
Chapter 30. Reviewing Zulip server code
CHAPTER
31
The chat.zulip.org community
chat.zulip.org is the primary communication forum for the Zulip community.
You can go through the simple signup process at that link, and then you will soon be talking to core Zulip developers
and other users. To get help in real time, you will have the best luck finding core developers roughly between 17:00
UTC and 2:00 UTC or during office hours and sprints, but the sun never sleeps on the Zulip community. Most
questions get a reply within minutes to a few hours, depending on the time of day.
This is a bleeding edge development server
The chat.zulip.org server is frequently deployed off of master from the Zulip Git repository, so please point out
anything you notice that seems wrong! We catch many bugs that escape code review this way.
The chat.zulip.org server is a development and testing server, not a production service, so don’t use it for anything
mission-critical, secret/embarrassing, etc.
Community conventions
• Send any test messages to #test here or as a PM to yourself to avoid disrupting others.
• When asking for help, provide the details needed for others to help you. E.g. include the full traceback in a code
block (not a screenshot), a link to the code or a WIP PR you’re having trouble debugging, etc.
• Ask questions on streams rather than PMing core contributors. You’ll get answers faster since other people can
help, and it makes it possible for other developers to learn from reading the discussion.
• Use @-mentions sparingly. Unlike IRC or Slack, in Zulip, it’s usually easy to see which message you’re replying
to, so you don’t need to mention your conversation partner in every reply. Mentioning other users is great for
timely questions or making sure someone who is not online sees your message.
• Converse informally; there’s no need to use titles like “Sir” or “Madam”.
• Use gender-neutral language.
183
Zulip Documentation, Release 1.4.0
• Follow the community code of conduct.
• Participate! Zulip is a friendly and welcoming community, and we love meeting new people, hearing about what
brought them to Zulip, and getting their feedback. If you’re not sure where to start, introduce yourself and your
interests in #new members, using your name as the topic.
Streams
There are a few streams worth highlighting that are relevant for everyone, even non-developers:
• #announce is for announcements and discussions thereof; we try to keep traffic there to a minimum.
• #design is where we discuss the UI design and collect feedback on potential design changes. We love feedback,
so don’t hesitate to speak up!
• #documentation is where we discuss improving Zulip’s user, sysadmin, and developer documentation.
• #production help is for production environment related discussions.
• #test here is for sending test messages without inconveniencing other users :). We recommend muting this
stream when not using it.
• #translation is for discussing Zulip’s translations.
There are dozens of streams for development discussions in the Zulip community (e.g. one for each app, etc.); check
out the Streams page to see the descriptions for all of them. Relevant to almost everyone are these:
• #checkins is for progress updates on what you’re working on and its status; usually folks post with their name
as the topic. Everyone is welcome to participate!
• #development help is for asking for help with any Zulip server/webapp development work (use the app streams
for help working on one of the apps).
• #code review is for getting feedback on your work. We encourage all developers to comment on work posted
here, even if you’re new to the Zulip project; reviewing other PRs is a great way to develop experience, and even
just manually testing a proposed new feature and posting feedback is super helpful.
• #learning is for posting great learning resources one comes across.
Chat meetings
We have regular chat meetings on Zulip to coordinate work on various parts of the Zulip project. While most developer
discussions happen asynchonrously, these meetings are used mainly to coordinate work within a major area of Zulip.
These meetings are usually scheduled in Pacific time mornings, since that seems to be the best time for our global
contributor base (the part of the world where it’s the deep middle of the night is the Pacific Ocean).
Anyone is welcome to attend and contribute to the discussions in these meetings, and they’re a great time to stop by
and introduce yourself if you’d like to get involved (though really, any time is, so).
Here are the regular meetings that exist today along with their usual times (actual times are listed in the linked agenda
documents):
• Mobile team on #mobile, generally Wednesdays at 10AM Pacific time. Agendas.
• Backend/infrastructure team on #backend, generally Fridays at 10AM Pacific time. Agendas.
• Bots and integrations team on #integrations, generally Thursdays at 9AM Pacific time. Agendas.
184
Chapter 31. The chat.zulip.org community
Zulip Documentation, Release 1.4.0
Office hours and sprints
We also do project-wide ad-hoc “office hours” and remote sprints irregularly, about once a month.
Anyone can schedule one: announce it in #announce and on the zulip-devel mailing list a few days ahead of time, and
ideally, tell Sumana so she can put it on the public Zulip meetings calendar.
Office hours are simply times for us to informally discuss current global project priorities, find out what questions
people have, and so on. We set them up so people know there’ll be more people around at a particular time to chat.
You don’t need to RSVP and you don’t need to show up on time or stop conversations when the “hour” stops. They
start in #general and conversations move into other streams and topics as they come up.
Sprints are times when Zulip developers get together in chat, and sometimes in person, to work on related issues at the
same time.
31.4. Chat meetings
185
Zulip Documentation, Release 1.4.0
186
Chapter 31. The chat.zulip.org community
CHAPTER
32
Using zulipbot
Zulip uses @zulipbot, a GitHub workflow bot deployed on all Zulip repositories, to handle issues and pull requests in
our repositories in order to create a better workflow for Zulip contributors.
Its purpose is to work around various limitations in GitHub’s permissions and notifications systems to make it possible
to have a much more democractic workflow for our contributors. It allows anyone to self-assign or label an issue, not
just the core contributors trusted with full write access to the repository (which is the only model GitHub supports).
Usage
• Claim an issue — Comment @zulipbot claim on the issue you want to claim; @zulipbot will assign you
to the issue and label the issue as in progress.
– If you’re a new contributor, @zulipbot will give you read-only collaborator access to the repository and
leave a welcome message on the issue you claimed.
– You can also claim an issue that you’ve opened by including @zulipbot claim in the body of your
issue.
– If you accidentally claim an issue you didn’t want to claim, comment @zulipbot abandon to abandon
an issue.
• Label your issues — Add appropriate labels to issues that you opened by including @zulipbot label in
an issue comment or the body of your issue followed by the desired labels enclosed within double quotes ("").
– For example, to add the bug and help wanted labels to your issue, comment or include @zulipbot
label "bug" "help wanted" in the issue body.
– You’ll receive an error message if you try to add any labels to your issue that don’t exist in your repository.
– If you accidentally added the wrong labels, you can remove them by commenting @zulipbot remove
followed by the desired labels enclosed with double quotes ("").
• Find unclaimed issues - Use the GitHub search feature to find unclaimed issues by adding one of the following
filters to your search:
187
Zulip Documentation, Release 1.4.0
– -label:
"in progress" (excludes issues labeled with the in progress label)
– no:assignee (shows issues without assignees)
Issues labeled with the in progress label and/or assigned to other users have already been claimed.
• Collaborate in area label teams - Receive notifications on issues and pull requests within your fields of expertise on the Zulip server repository by joining the Zulip server area label teams. These teams correspond to
the repository’s area labels, although some teams are associated with multiple labels; for example, the area:
message-editing and area: message view labels are both related to the Server message view team. Feel free to
join as many area label teams as as you’d like!
After your request to join an area label team is approved, you’ll receive notifications for any issues labeled with
the team’s corresponding area label as well as any pull requests that reference issues labeled with your team’s
area label.
• Track inactive claimed issues — If a claimed issue has not been updated for a week, @zulipbot will post a
comment on the inactive issue to ask the assignee(s) if they are still working on the issue.
If you see this comment on an issue you claimed, you should post a comment on the issue to notify @zulipbot
that you’re still working on it.
If @zulipbot does not receive a response from the assignee within 3 days of an inactive issue prompt, @zulipbot
will automatically remove the issue’s current assignee(s) and the “in progress” label to allow others to work on
an inactive issue.
• Receive Travis build status notifications — If you would like to receive a notification whenever the build
status of your pull request is updated, label your pull request with the “travis updates” label using the command
@zulipbot label "travis updates", and @zulipbot will let you know the build status (e.g. passed,
failed, errored) of your pull request once all tests finish.
Contributing
If you wish to help develop and contribute to @zulipbot, check out the zulip/zulipbot repository on GitHub and read
the project’s contributing guidelines for more information.
188
Chapter 32. Using zulipbot
CHAPTER
33
Accessibility
Guidelines
In order to accommodate all users, Zulip strives to implement accessibility best practices in its user interface. There
are many aspects to accessibility; here are some of the more important ones to keep in mind.
• All images should have alternative text attributes for the benefit of users who cannot see them (this includes
users who are utilizing a voice interface to free up their eyes to look at something else instead).
• The entire application should be usable via a keyboard (many users are unable to use a mouse, and many
accessibility aids emulate a keyboard).
• Text should have good enough contrast against the background to enable even users with moderate visual impairment to be able to read it.
• ARIA (Accessible Rich Internet Application) attributes should be used appropriately to enable screen readers
and other alternative interfaces to navigate the application effectively.
There are many different standards for accessibility, but the most relevant one for Zulip is the W3C’s WCAG (Web
Content Accessibility Guidelines), currently at version 2.0. Whenever practical, we should strive for compliance with
the AA level of this specification. (The W3C itself recommends not trying to comply with the highest AAA level for
an entire web site or application, as it is not possible for some content.)
Tools
There are tools available to automatically audit a web page for compliance with many of the WCAG guidelines. Here
are some of the more useful ones:
• Accessibility Developer Tools This open source Chrome extension from Google adds an accessibility audit to
the “Audits” tab of the Chrome Developer Tools. The audit is performed against the page’s DOM via JavaScript,
allowing it to identify some issues that a static HTML inspector would miss.
• aXe An open source Chrome and Firefox extension which runs a somewhat different set of checks than Google’s
Chrome extension.
189
Zulip Documentation, Release 1.4.0
• Wave This web application takes a URL and loads it in a frame, reporting on all the issues it finds with links
to more information. Has the advantage of not requiring installation, but requires a URL which can be directly
accessed by an external site.
• Web Developer This browser extension has many useful features, including a convenient link for opening the
current URL in Wave to get an accessibility report.
Note that these tools cannot catch all possible accessibility problems, and sometimes report false positives as well.
They are a useful aid in quickly identifying potential problems and checking for regressions, but their recommendations
should not be blindly obeyed.
GitHub Issues
Problems with Zulip’s accessibility should be reported as GitHub issues with the “accessibility” label. This label can
be added by entering the following text in a separate comment on the issue:
@zulipbot label "accessibility"
If you want to help make Zulip more accessible, here is a list of the currently open accessibility issues.
Additional Resources
For more information about making Zulip accessible to as many users as possible, the following resources may be
useful.
• Web Content Accessibility Guidelines (WCAG) 2.0
• WAI-ARIA - Web Accessibility Initiative Accessible Rich Internet Application Suite
• WebAIM - Web Accessibility in Mind
• The MDN page on accessibility
• The Open edX Accessibility Guidelines for developers
190
Chapter 33. Accessibility
CHAPTER
34
Testing and writing tests
Overview
Zulip has a full test suite that includes many components. The most important components are documented in depth
in their own sections:
• Django: backend Python tests
• Casper: end-to-end UI tests
• Node: unit tests for JS front end code
• Linters: Our parallel linter suite
• Travis CI details: How all of these run in Travis CI
This document covers more general testing issues, such as how to run the entire test suite, how to troubleshoot database
issues, how to manually test the front end, and how to plan for the future upgrade to Python3.
We also document how to manually test the app.
Running tests
Zulip tests must be run inside a Zulip development environment; if you’re using Vagrant, you will need to enter the
Vagrant environment before running the tests:
vagrant ssh
cd /srv/zulip
Then, to run the full Zulip test suite, do this:
./tools/test-all
191
Zulip Documentation, Release 1.4.0
This runs the linter (tools/lint) plus all of our test suites; they can all be run separately (just read tools/
test-all to see them). You can also run individual tests which can save you a lot of time debugging a test failure,
e.g.:
./tools/lint # Runs all the linters in parallel
./tools/test-backend zerver.tests.test_bugdown.BugdownTest.test_inline_youtube
./tools/test-backend BugdownTest # Run `test-backend --help` for more options
./tools/test-js-with-casper 09-navigation.js
./tools/test-js-with-node utils.js
The above setup instructions include the first-time setup of test databases, but you may need to rebuild the test database
occasionally if you’re working on new database migrations. To do this, run:
./tools/do-destroy-rebuild-test-database
Possible testing issues
• When running the test suite, if you get an error like this:
sqlalchemy.exc.ProgrammingError: (ProgrammingError) function ts_match_locs_
array(unknown, text, tsquery) does not
exist
LINE 2: ...ECT message_id, flags, subject, rendered_content, ts_match_l...
^
˓→
. . . then you need to install tsearch-extras, described above. Afterwards, re-run the init*-db and the
do-destroy-rebuild*-database scripts.
• When building the development environment using Vagrant and the LXC provider, if you encounter permissions
errors, you may need to chown -R 1000:$(whoami) /path/to/zulip on the host before running
vagrant up in order to ensure that the synced directory has the correct owner during provision. This issue
will arise if you run id username on the host where username is the user running Vagrant and the output
is anything but 1000. This seems to be caused by Vagrant behavior; for more information, see the vagrant-lxc
FAQ entry about shared folder permissions.
Schema and initial data changes
If you change the database schema or change the initial test data, you have to regenerate the pristine test database by
running tools/do-destroy-rebuild-test-database.
Wiping the test databases
You should first try running: tools/do-destroy-rebuild-test-database
If that fails you should try to do:
sudo -u postgres psql
> DROP DATABASE zulip_test;
> DROP DATABASE zulip_test_template;
and then run tools/do-destroy-rebuild-test-database
192
Chapter 34. Testing and writing tests
Zulip Documentation, Release 1.4.0
Recreating the postgres cluster
warning
This is irreversible, so do it with care, and never do this anywhere in production.
If your postgres cluster (collection of databases) gets totally trashed permissions-wise, and you can’t otherwise repair
it, you can recreate it. On Ubuntu:
sudo pg_dropcluster --stop 9.1 main
sudo pg_createcluster --locale=en_US.utf8 --start 9.1 main
Local browser testing (local app + web browser)
This section is about troubleshooting your local development environment.
There is a separate manual testing doc that enumerates things you can test as part of manual QA.
Clearing the development database
You can use:
./tools/do-destroy-rebuild-database
to drop the database on your development environment and repopulate your it with the Shakespeare characters and
some test messages between them. This is run automatically as part of the development environment setup process,
but is occasionally useful when you want to return to a clean state for testing.
JavaScript manual testing
debug.js has some tools for profiling JavaScript code, including:
• ‘print_elapsed_time‘: Wrap a function with it to print the time that function takes to the JavaScript console.
• ‘IterationProfiler‘: Profile part of looping constructs (like a for loop or $.each). You mark sections of the iteration
body and the IterationProfiler will sum the costs of those sections over all iterations.
Chrome has a very good debugger and inspector in its developer tools. Firebug for Firefox is also pretty good. They
both have profilers, but Chrome’s is a sampling profiler while Firebug’s is an instrumenting profiler. Using them both
can be helpful because they provide different information.
Python 3 Compatibility
Zulip is working on supporting Python 3, and all new code in Zulip should be Python 2+3 compatible. We have
converted most of the codebase to be compatible with Python 3 using a suite of 2to3 conversion tools and some
manual work. In order to avoid regressions in that compatibility as we continue to develop new features in Zulip,
we have a special tool, tools/check-py3, which checks all code for Python 3 syntactic compatibility by running
a subset of the automated migration tools and checking if they trigger any changes. tools/check-py3 is run
automatically in Zulip’s Travis CI tests (in the ‘static-analysis’ build) to avoid any regressions, but is not included in
test-all since it is quite slow.
34.5. Local browser testing (local app + web browser)
193
Zulip Documentation, Release 1.4.0
To run tools/check-py3, you need to install the modernize and future Python packages (which are included
in requirements/py3k.txt, which itself is included in requirements/dev.txt, so you probably already
have these packages installed).
To run check-py3 on just the Python files in a particular directory, you can change the current working directory
(e.g. cd zerver/) and run check-py3 from there.
Also, if you’re using Vagrant or if you set up virtualenvs in the non-Vagrant setup, you should have two different
virtualenvs: one for Python 2 (which is the one used by default), and another one for Python 3. This is useful if you
want to do some manual testing using either version.
To switch between both virtualenvs, run:
• Use Python 2: source /srv/zulip-venv/bin/activate
• Use Python 3: source /srv/zulip-py3-venv/bin/activate
194
Chapter 34. Testing and writing tests
CHAPTER
35
Linters
Overview
Zulip does extensive linting of much of its source code, including Python/JavaScript files, HTML templates
(Django/handlebars), CSS files, JSON fixtures, Markdown documents, puppet manifests, and shell scripts.
For some files we simply check for small things like trailing whitespace, but for other files, we are quite thorough
about checking semantic correctness.
Obviously, a large reason for linting code is to enforce the Zulip coding standards. But we also use the linters to
prevent common coding errors.
We borrow some open source tools for much of our linting, and the links below will direct you to the official documentation for these projects.
• eslint
• mypy
• puppet (puppet provides its own mechanism for validating manifests)
• pyflakes
Zulip also uses some home-grown code to perform tasks like validating indentation in template files, enforcing coding
standards that are unique to Zulip, allowing certain errors from third party linters to pass through, and exempting
legacy files from lint checks.
Running the linters
If you run ./tools/test-all, it will automatically run the linters (with one small exception: it does not run mypy
against scripts).
You can also run them individually:
195
Zulip Documentation, Release 1.4.0
./tools/lint
./tools/run-mypy
./tools/run-mypy --scripts-only
Finally, you can rely on our Travis CI setup to run linters for you, but it is good practice to run lint checks locally.
Note: The linters only check files that git tracks. Remember to git add new files before running lint checks.
Our linting tools generally support the ability to lint files individually–with some caveats–and those options will be
described later in this document.
We may eventually bundle run-mypy into lint, but mypy is pretty resource intensive compared to the rest of the
linters, because it does static code analysis. So we keep mypy separate to allow folks to quickly run the other lint
checks.
General considerations
Once you have read the Zulip coding guidelines, you can be pretty confident that 99% of the code that you write will
pass through the linters fine, as long as you are thorough about keeping your code clean. And, of course, for minor
oversights, lint is your friend, not your foe.
Occasionally, our linters will complain about things that are more of an artifact of the linter limitations than any actual
problem with your code. There is usually a mechanism where you can bypass the linter in extreme cases, but often
it can be a simple matter of writing your code in a slightly different style to appease the linter. If you have problems
getting something to lint, you can submit an unfinished PR and ask the reviewer to help you work through the lint
problem, or you can find other people in the Zulip Community to help you.
Also, bear in mind that 100% of the lint code is open source, so if you find limitations in either the Zulip home-grown
stuff or our third party tools, feedback will be highly appreciated.
Finally, one way to clean up your code is to thoroughly exercise it with tests. The Zulip test documentation describes
our test system in detail.
Lint checks
Most of our lint checks get performed by ./tools/lint. These include the following checks:
• Check Python code with pyflakes.
• Check JavaScript code with eslint.
• Check Python code for custom Zulip rules.
• Check non-Python code for custom Zulip rules.
• Check puppet manifests with the puppet validator.
• Check HTML templates for matching tags and indentations.
• Check CSS for parsability.
• Check JavaScript code for addClass calls.
The remaining lint checks occur in ./tools/run-mypy. It is probably somewhat of an understatement to call
“mypy” a “linter,” as it performs static code analysis of Python type annotations throughout our Python codebase.
Our documentation on using mypy covers mypy in more detail.
The rest of this document pertains to the checks that occur in ./tools/lint.
196
Chapter 35. Linters
Zulip Documentation, Release 1.4.0
lint
Zulip has a script called lint that lives in our “tools” directory. It is the workhorse of our linting system, although in
some cases it dispatches the heavy lifting to other components such as pyflakes, eslint, and other home grown tools.
You can find the source code here.
In order for our entire lint suite to run in a timely fashion, the lint script performs several lint checks in parallel by
forking out subprocesses. This mechanism is still evolving, but you can look at the method run_parallel to get
the gist of how it works.
Note that our project does custom regex-based checks on the code, and we also customize how we call pyflakes and
pycodestyle (pep8). The code for these types of checks mostly lives here.
Special options
You can use the -h option for lint to see its usage. One particular flag to take note of is the --modified flag, which
enables you to only run lint checks against files that are modified in your git repo. Most of the “sub-linters” respect
this flag, but some will continue to process all the files. Generally, a good workflow is to run with --modified
when you are iterating on the code, and then run without that option right before commiting new code.
If you need to troubleshoot the linters, there is a --verbose option that can give you clues about which linters may
be running slow, for example.
Lint checks
The next part of this document describes the lint checks that we apply to various file types.
Generic source code checks
We check almost our entire codebase for trailing whitespace. Also, we disallow tab (\t) characters in all but two files.
We also have custom regex-based checks that apply to specific file types. For relatively minor files like Markdown
files and JSON fixtures, this is the extent of our checking.
Finally, we’re checking line length in Python code (and hope to extend this to other parts of the codebase soon). You
can use #ignorelinelength for special cases where a very long line makes sense (e.g. a link in a comment to an
extremely long URL).
Python code
The bulk of our Python linting gets outsourced to the “pyflakes” tool. We call “pyflakes” in a fairly vanilla fashion, and
then we post-process its output to exclude certain types of errors that Zulip is comfortable ignoring. (One notable class
of error that Zulip currently tolerates is unused imports–because of the way mypy type annotations work in Python 2,
it would be inconvenient to enforce this too strictly.)
Zulip also has custom regex-based rules that it applies to Python code. Look for python_rules in the source code
for lint. Note that we provide a mechanism to exclude certain lines of codes from these checks. Often, it is simply
the case that our regex approach is too crude to correctly exonerate certain valid constructs. In other cases, the code
that we exempt may be deemed not worthwhile to fix.
35.5. lint
197
Zulip Documentation, Release 1.4.0
JavaScript code
We check our JavaScript code in a few different ways:
• We run eslint.
• We perform custom Zulip regex checks on the code.
• We verify that all addClass calls, with a few exceptions, explicitly contain a CSS class.
The last check happens via a call to ./tools/find-add-class. This particular check is a work in progress, as
we are trying to evolve a more rigorous system for weeding out legacy CSS styles, and the ability to quickly introspect
our JS code for addClass calls is part of our vision.
Puppet manifests
We use Puppet as our tool to manage configuration files, using puppet “manifests.” To lint puppet manifests, we use
the “parser validate” option of puppet.
HTML Templates
Zulip uses two HTML templating systems:
• Django templates
• handlebars
Zulip has an internal tool that validates both types of templates for correct indentation and matching tags. You can
find the code here:
• driver: check-templates
• engine: lib/template_parser.py
We exempt some legacy files from indentation checks, but we are hoping to clean those files up eventually.
CSS
Zulip does not currently lint its CSS for any kind of semantic correctness, but that is definitely a goal moving forward.
We do ensure that our home-grown CSS parser can at least parse the CSS code. This is a slightly more strict check
than checking that the CSS is compliant to the official spec, as our parser will choke on unusual constructs that we
probably want to avoid in our code, anyway. (When the parser chokes, the lint check will fail.)
You can find the code here:
• driver: check-css
• engine: lib/css_parser.py
Markdown, shell scripts, JSON fixtures
We mostly validate miscellaneous source files like .sh, .json, and .md files for whitespace issues.
Philosophy
If you want to help improve Zulip’s system for linting, here are some considerations.
198
Chapter 35. Linters
Zulip Documentation, Release 1.4.0
Speed
We want our linters to be fast enough that most developers will feel comfortable running them in a pre-commit hook,
so we run our linters in parallel and support incremental checks.
Accuracy
We try to catch as many common mistakes as possible, either via a linter or an automated test.
Completeness
Our goal is to have most common style issues by caught by the linters, so new contributors to the codebase can
efficiently fix produce code with correct style without needing to go back-and-forth with a reviewer.
35.6. Philosophy
199
Zulip Documentation, Release 1.4.0
200
Chapter 35. Linters
CHAPTER
36
Backend Django tests
Overview
Zulip uses the Django framework for its Python back end. We use the testing framework from django.test to test our
code. We have over a thousand automated tests that verify that our backend works as expected.
All changes to the Zulip backend code should be supported by tests. We enforce our testing culture during code review,
and we also use coverage tools to measure how well we test our code. We mostly use tests to prevent regressions in
our code, but the tests can have ancillary benefits such as documenting interfaces and influencing the design of our
software.
If you have worked on other Django projects that use unit testing, you will probably find familiar patterns in Zulip’s
code. This document describes how to write tests for the Zulip back end, with a particular emphasis on areas where
we have either wrapped Django’s test framework or just done things that are kind of unique in Zulip.
Running tests
Our tests live in zerver/tests/. You can run them with ./tools/test-backend. The tests run in parallel
using multiple threads in your development environment, and can finish in under 30s on a fast machine. When you
are in iterative mode, you can run individual tests or individual modules, following the dotted.test.name convention
below:
cd /srv/zulip
./tools/test-backend zerver.tests.test_queue_worker.WorkerTest
There are many command line options for running Zulip tests, such as a --verbose option. The best way to learn
the options is to use the online help:
./tools/test-backend -h
We also have ways to instrument our tests for finding code coverage, URL coverage, and slow tests. Use the -h option
to discover these features. We also have a --profile option to facilitate profiling tests.
201
Zulip Documentation, Release 1.4.0
Another thing to note is that our tests generally “fail fast,” i.e. they stop at the first sign of trouble. This is generally
a good thing for iterative development, but you can override this behavior with the --nonfatal-errors option.
A useful option to combine with that is the --rerun option, which will rerun just the tests that failed in the last test
run.
How to write tests.
Before you write your first tests of Zulip, it is worthwhile to read the rest of this document, and you can also read some
of the existing tests in zerver/tests to get a feel for the patterns we use.
A good practice is to get a “failing test” before you start to implement your feature. First, it is a useful exercise to
understand what needs to happen in your tests before you write the code, as it can help drive out simple design or help
you make incremental progress on a large feature. Second, you want to avoid introducing tests that give false positives.
Ensuring that a test fails before you implement the feature ensures that if somebody accidentally regresses the feature
in the future, the test will catch the regression.
Another important files to skim are zerver/lib/test_helpers.py, which contains test helpers. zerver/lib/test_classes.py,
which contains our ZulipTestCase and WebhookTestCase classes.
Setting up data for tests
All tests start with the same fixture data. (The tests themselves update the database, but they do so inside a transaction
that gets rolled back after each of the tests complete. For more details on how the fixture data gets set up, refer to
tools/setup/generate-fixtures.)
The fixture data includes a few users that are named after Shakesepeare characters, and they are part of the “zulip.com”
realm.
Generally, you will also do some explicit data setup of your own. Here are a couple useful methods in ZulipTestCase:
• common_subscribe_to_streams
• send_message
• make_stream
• subscribe_to_stream
More typically, you will use methods directly from the backend code. (This ensures more end-to-end testing, and
avoids false positives from tests that might not consider ancillary parts of data setup that could influence tests results.)
Here are some example action methods that tests may use for data setup:
• check_send_message
• do_change_is_admin
• do_create_user
• do_make_stream_private
Zulip Testing Philosophy
If there is one word to describe Zulip’s philosophy for writing tests, it is probably “flexible.” (Hopefully “thorough”
goes without saying.)
202
Chapter 36. Backend Django tests
Zulip Documentation, Release 1.4.0
When in doubt, unless speed concerns are prohibitive, you usually want your tests to be somewhat end-to-end, particularly for testing endpoints.
These are some of the testing strategies that you will see in the Zulip test suite...
Endpoint tests
We strive to test all of our URL endpoints. The vast majority of Zulip endpoints support a JSON interface. Regardless
of the interface, an endpoint test generally follows this pattern:
• Set up the data.
• Login with self.login() or set up an API key.
• Use a Zulip test helper to hit the endpoint.
• Assert that the result was either a success or failure.
• Check the data that comes back from the endpoint.
Generally, if you are doing endpoint tests, you will want to create a test class that is a subclass of ZulipTestCase,
which will provide you helper methods like the following:
• api_auth
• assert_json_error
• assert_json_success
• client_get
• client_post
• get_api_key
• get_streams
• login
• send_message
Library tests
For certain Zulip library functions, especially the ones that are not intrinsically tied to Django, we use a classic unit
testing approach of calling the function and inspecting the results.
For these types of tests, you will often use methods like self.assertEqual(), self.assertTrue(), etc.,
which come with unittest via Django.
Fixture-driven tests
Particularly for testing Zulip’s integrations with third party systems, we strive to have a highly data-driven approach to
testing. To give a specific example, when we test our GitHub integration, the test code reads a bunch of sample inputs
from a JSON fixture file, feeds them to our GitHub integration code, and then verifies the output against expected
values from the same JSON fixture file.
Our fixtures live in zerver/fixtures.
36.4. Zulip Testing Philosophy
203
Zulip Documentation, Release 1.4.0
Mocks and stubs
We use mocks and stubs for all the typical reasons:
• to more precisely test the target code
• to stub out calls to third-party services
• to make it so that you can run your tests on the airplane without wifi
For mocking we generally use the “mock” library and use mock.patch as a context manager or decorator. We also
take advantage of some context managers from Django as well as our own custom helpers. Here is an example:
with self.settings(RATE_LIMITING=True):
with mock.patch('zerver.decorator.rate_limit_user') as rate_limit_mock:
api_result = my_webhook(request)
self.assertTrue(rate_limit_mock.called)
Follow this link for more information on the “settings” context manager.
Template tests
In zerver/tests/test_templates.py we have a test that renders all of our back end templates with a “dummy” context, to
make sure the templates don’t have obvious errors. (These tests won’t catch all types of errors; they are just a first line
of defense.)
SQL performance tests
A common class of bug with Django systems is to handle bulk data in an inefficient way, where the back end populates
objects for join tables with a series of individual queries that give O(N) latency. (The remedy is often just to call
select_related(), but sometimes it requires a more subtle restructuring of the code.)
We try to prevent these bugs in our tests by using a context manager called queries_captured() that captures
the SQL queries used by the back end during a particular operation. We make assertions about those queries, often
simply asserting that the number of queries is below some threshold.
Event-based tests
The Zulip back end has a mechanism where it will fetch initial data for a client from the database, and then
it will subsequently apply some queued up events to that data to the data structure before notifying the client.
The EventsRegisterTest.do_test() helper helps tests verify that the application of those events via apply_events() produces the same data structure as performing an action that generates said event.
This is a bit esoteric, but if you read the tests, you will see some of the patterns. You can also learn more about our
event system in the new feature tutorial.
Negative tests
It is important to verify error handling paths for endpoints, particularly situations where we need to ensure that we
don’t return results to clients with improper authentication or with limited authorization. A typical test will call
the endpoint with either a non-logged in client, an invalid API key, or missing input fields. Then the test will call
assert_json_error() to verify that the endpoint is properly failing.
204
Chapter 36. Backend Django tests
Zulip Documentation, Release 1.4.0
Testing considerations
Here are some things to consider when writing new tests:
• Duplication We try to avoid excessive duplication in tests. If you have several tests repeating the same type of
test setup, consider making a setUp() method or a test helper.
• Network independence Our tests should still work if you don’t have an internet connection. For third party
clients, you can simulate their behavior using fixture data. For third party servers, you can typically simulate
their behavior using mocks.
• Coverage We have 100% line coverage on several of our backend modules. You can use the --coverage
option to generate coverage reports, and new code should have 100% coverage, which generally requires testing
not only the “happy path” but also error handling code and edge cases. It will generate a nice HTML report that
you can view right from your browser (the tool prints the URL where the report is exposed in your development
environment).
Note that test-backend --coverage will assert that various specific files in the project have 100% test coverage and throw an error if their coverage has fallen. One of our project goals is to expand that checking to ever-larger
parts of the codebase.
36.5. Testing considerations
205
Zulip Documentation, Release 1.4.0
206
Chapter 36. Backend Django tests
CHAPTER
37
JavaScript unit tests
As an alternative to the black-box whole-app testing, you can unit test individual JavaScript files.
If you are writing JavaScript code that manipulates data (as opposed to coordinating UI changes), then you probably
modify existing unit test modules to ensure the quality of your code and prevent regressions.
The JS unit tests are written to work with node. You can find them in frontend_tests/node_tests. Here is
an example test from frontend_tests/node_tests/stream_data.js:
(function test_get_by_id() {
stream_data.clear_subscriptions();
var id = 42;
var sub = {
name: 'Denmark',
subscribed: true,
color: 'red',
stream_id: id
};
stream_data.add_sub('Denmark', sub);
sub = stream_data.get_sub('Denmark');
assert.equal(sub.color, 'red');
sub = stream_data.get_sub_by_id(id);
assert.equal(sub.color, 'red');
}());
The names of the node tests generally align with the names of the modules they test. If you modify a JS module in
static/js you should see if there are corresponding test in frontend_tests/node_tests. If there are, you
should strive to follow the patterns of the existing tests and add your own tests.
HTML output
The JavaScript unit tests can generate output to be viewed in the browser. The best examples of this are in
frontend_tests/node_tests/templates.js.
207
Zulip Documentation, Release 1.4.0
The main use case for this mechanism is to be able to unit test templates and see how they are rendered without the
complications of the surrounding app. (Obviously, you still need to test the app itself!) The HTML output can also
help to debug the unit tests.
Each test calls a method named write_handlebars_output after it renders a template with similar data. This
API is still evolving, but you should be able to look at existing code for patterns.
When you run tools/test-js-with-node, it will present you with a message like “To see more output, open
var/test-js-with-node/index.html.” Basically, you just need to open the file in the browser. (If you are running a VM,
this might require switching to another terminal window to launch the open command.)
Coverage reports
You can automatically generate coverage reports for the JavaScript unit tests like this:
tools/test-js-with-node --coverage
Then open coverage/lcov-report/js/index.html in your browser. Modules we don’t test at all aren’t
listed in the report, so this tends to overstate how good our overall coverage is, but it’s accurate for individual files.
You can also click a filename to see the specific statements and branches not tested. 100% branch coverage isn’t
necessarily possible, but getting to at least 80% branch coverage is a good goal.
Handling dependencies in unit tests
The following scheme helps avoid tests leaking globals between each other.
First, if you can avoid globals, do it, and the code that is directly under test can simply be handled like this:
var search = require('js/search_suggestion.js');
For deeper dependencies, you want to categorize each module as follows:
• Exercise the module’s real code for deeper, more realistic testing?
• Stub out the module’s interface for more control, speed, and isolation?
• Do some combination of the above?
For all the modules where you want to run actual code, add a statement like the following to the top of your test file:
add_dependencies({
_: 'node_modules/underscore/underscore.js',
util: 'js/util.js',
Dict: 'js/dict.js',
Handlebars: 'handlebars',
Filter: 'js/filter.js',
typeahead_helper: 'js/typeahead_helper.js',
stream_data: 'js/stream_data.js',
narrow: 'js/narrow.js'
});
For modules that you want to completely stub out, please use a pattern like this:
set_global('page_params', {
email: 'bob@zulip.com'
});
208
Chapter 37. JavaScript unit tests
Zulip Documentation, Release 1.4.0
// then maybe further down
global.page_params.email = 'alice@zulip.com';
Finally, there’s the hybrid situation, where you want to borrow some of a module’s real functionality but stub out other
pieces. Obviously, this is a pretty strong smell that the other module might be lacking in cohesion, but that code might
be outside your jurisdiction. The pattern here is this:
// Use real versions of parse/unparse
var narrow = require('js/narrow.js');
set_global('narrow', {
parse: narrow.parse,
unparse: narrow.unparse
});
// But later, I want to stub the stream without having to call super˓→expensive
// real code like narrow.activate().
global.narrow.stream = function () {
return 'office';
};
Creating new test modules
The nodes tests rely on JS files that use the module pattern. For example, to test the foobar.js file, you would first
add the following to the bottom of foobar.js:
if (typeof module !== 'undefined') {
module.exports = foobar;
}
This makes foobar.js follow the CommonJS module pattern, so it can be required in Node.js, which runs our tests.
Now create frontend_tests/node_tests/foobar.js. At the top, require the Node.js assert module, and
the module you’re testing, like so:
var assert = require('assert');
var foobar = require('js/foobar.js');
And of course, if the module you’re testing depends on other modules, or modifies global state, you may need to
review the section on handling dependencies above.
Define and call some tests using the assert module. Note that for “equal” asserts, the actual value comes first, the
expected value second.
(function test_somefeature() {
assert.strictEqual(foobar.somefeature('baz'), 'quux');
assert.throws(foobar.somefeature('Invalid Input'));
}());
The test runner (index.js) automatically runs all .js files in the frontend_tests/node directory.
37.4. Creating new test modules
209
Zulip Documentation, Release 1.4.0
210
Chapter 37. JavaScript unit tests
CHAPTER
38
Web frontend black-box casperjs tests
These live in frontend_tests/casper_tests/. This is a “black box” integration test; we load the frontend
in a real (headless) browser, from a real (development) server, and simulate UI interactions like sending messages,
narrowing, etc., by actually clicking around the UI and waiting for things to change before doing the next step. These
tasks are fantastic for ensuring the overall health of the project, but are also costly to maintain and keep free of
nondeterministic failures, so we usually prefer to write a Node test instead when possible.
Since the Casper tests interact with a real dev server, they can often catch backend bugs as well.
You can run the casper tests with ./tools/test-js-with-casper
test-js-with-casper 06-settings.js to run a single test file from
casper_tests/.
or as ./tools/
frontend_tests/
Debugging Casper.JS
Casper.js (via PhantomJS) has support for remote debugging. However, it is not perfect. Here are some steps for using
it and gotchas you might want to know; you’ll likely also want to read the section on writing tests (below) if you get
stuck, since the advice on how to write correct Casper selectors will likely be relevant.
The first thing to do when debugging Casper tests is to check the additional debug output that our framework provides:
• You can check the screenshots of what the UI looked like at the time of failures at var/casper/
casper-failure*.png.
• If it’s possible there’s a backend exception involved, var/casper/server.log will contain the server logs
from the casper run; it’s worth looking there for tracebacks if you get stuck.
Print debugging
If you need to use print debugging in casper, you can do using casper.log; see http://docs.casperjs.org/en/latest/
logging.html for details.
An additional debugging technique is to enable verbose mode in the Casper tests; you can do this by adding to the top
of the relevant test file the following:
211
Zulip Documentation, Release 1.4.0
var casper = require('casper').create({
verbose: true,
logLevel: "debug"
});
This can sometimes give insight into exactly what’s happening.
Remote debugging
This is a pain to setup with Vagrant because port 7777 and 9981 aren’t forwarded to the host by default, but can be
pretty useful in rare difficult cases.
To turn on remote debugging, pass --remote-debug to the ./frontend_tests/run-casper script. This
will run the tests with port 7777 open for remote debugging. You can now connect to localhost:7777 in a
Webkit browser. Somewhat recent versions of Chrome or Safari might be required.
• When connecting to the remote debugger, you will see a list of pages, probably 2. One page called
about:blank is the headless page in which the CasperJS test itself is actually running in. This is where
your test code is.
• The other page, probably localhost:9981, is the Zulip page that the test is testing—that is, the page running
our app that our test is exercising.
Since the tests are now running, you can open the about:blank page, switch to the Scripts tab, and open the
running 0x-foo.js test. If you set a breakpoint and it is hit, the inspector will pause and you can do your normal
JS debugging. You can also put breakpoints in the Zulip webpage itself if you wish to inspect the state of the Zulip
frontend.
Reproducing races only seen in Travis CI
We’ve sometimes found it useful for reproducing Casper race conditions in Casper tests that mostly only happen in
Travis CI with really cheap VPS servers (e.g. Scaleway’s 2GB x86). This works because an ultra slow machine is
more likely to have things happen in an order similar to what happens in Travis CI’s very slow containers.
Writing Casper tests
Probably the easiest way to learn how to write Casper tests is to study some of the existing test files. There are a few
tips that can be useful for writing Casper tests in addition to the debugging notes below:
• Run just the file containing your new tests as described above to have a fast debugging cycle.
• With frontend tests in general, it’s very important to write your code to wait for the right events. Before essentially every action you take on the page, you’ll want to use waitUntilVisible, waitWhileVisible,
or a similar function to make sure the page or elemant is ready before you interact with it. For instance, if you
want to click a button that you can select via #btn-submit, and then check that it causes success-elt to
appear, you’ll want to write something like:
casper.waitUntilVisible("#btn-submit", function () {
casper.click('#btn-submit')
casper.test.assertExists("#success-elt");
});
212
Chapter 38. Web frontend black-box casperjs tests
Zulip Documentation, Release 1.4.0
In many cases, you will actually need to wait for the UI to update clicking the button before doing asserts or the
next step. This will ensure that the UI has finished updating from the previous step before Casper attempts to next
step. The various wait functions supported in Casper are documented in the Casper here: http://docs.casperjs.
org/en/latest/modules/casper.html#waitforselector and the various assert statements available are documented
here: http://docs.casperjs.org/en/latest/modules/tester.html#the-tester-prototype
• The casper.wait style functions (waitWhileVisible, waitUntilVisible, etc.) cannot be chained
together in certain conditions without creating race conditions where the test may fail nondeterministically. For
example, don’t do this:
casper.waitUntilVisible('tag 1');
casper.click('button');
casper.waitUntilVisible('tag 2');
Instead, if you want to avoid race condition, wrap the second waitFor in a then function like this:
casper.then(function () {
casper.waitUntilVisible('tag 1', function () {
casper.click('#btn-submit');
});
});
casper.then(function () {
casper.waitUntilVisible('tag 2', function () {
casper.test.assertExists('#success-elt');
});
});
(You’ll also want to use selectors that are as explicit as possible, to avoid accidentally clicking multiple buttons
or the wrong button in your test, which can cause nondeterministic failures)
• Generally casper.waitUntilVisible is preferable to e.g. casper.waitForSelector, since the
former will confirm the thing is actually on screen. E.g. if you’re waiting to switch from one panel of the the
settings overlay to another by waiting for a particular widget to appear, casper.waitForSelector may
not actually wait (since the widget is probably in the DOM, just not visible), but casper.waitUntilVisible will
wait until it’s actually shown.
• The selectors (i.e. things you put inside casper.waitUntilVisible() and friends) appearing in Casper
tests are CSS3 selectors, which is a slightly different syntax from the jQuery selectors used in the rest of the
Zulip codebase; in particular, some expressions that work with jQuery (and thus normal Zulip JavaScript code)
won’t work with CSS3. It’s often helpful to debug selectors interactively, which you can do in the Chrome
JavaScript console. The way to do it is $$("#settings-dropdown"); that queries CSS3 selectors, so
you can debug your selector in the console and then paste it into your Casper test once it’s working. For other
browsers like Firefox, you can use querySelectorAll("#settings-dropdown"), syntax which is
only available in the browser’s JavaScript console.
You can learn more about these selectors and other JavaScript console tools here.
• The test suite uses a smaller set of default user accounts and other data initialized in the database than the
development environment; to see what differs check out the section related to options["test_suite"]
in zilencer/management/commands/populate_db.py.
• Casper effectively runs your test file in two phases – first it runs the code in the test file, which for most test files
will just collect a series of steps (each being a casper.then or casper.wait... call). Then, usually at the
end of the test file, you’ll have a casper.run call which actually runs that series of steps. This means that if
you write code in your test file outside a casper.then or casper.wait... method, it will actually run before all the Casper test steps that are declared in the file, which can lead to confusing failures where the new code
you write in between two casper.then blocks actually runs before either of them. See this for more details
about how Casper works: http://docs.casperjs.org/en/latest/faq.html#how-does-then-and-the-step-stack-work
38.2. Writing Casper tests
213
Zulip Documentation, Release 1.4.0
214
Chapter 38. Web frontend black-box casperjs tests
CHAPTER
39
Travis CI
The Zulip server uses Travis CI for its continuous integration. This page documents useful tools and tips to know
about when using Travis CI and debugging issues with it.
Goals
The overall goal of our Travis CI setup is to avoid regressions and minimize the total time spent debugging Zulip. We
do that by trying to catch as many possible future bugs as possible, while minimizing both latency and false positives,
both of which can waste a lot of developer time. There are a few implications of this overall goal:
• If a test is failing nondeterministically in Travis CI, we consider that to be an urgent problem.
• If the tests become a lot slower, that is also an urgent problem.
• Everything we do in CI should also have a way to run it quickly (under 1 minute, preferably under 3 seconds),
in order to iterate fast in development. Except when working on the Travis CI configuration itself, a developer
should never have to repeatedly wait 10 minutes for a full Travis run to iteratively debug something.
Configuration
The main Travis configuration file is .travis.yml. The specific test suites we have are listed in the matrix section,
which has a matrix of Python versions and test suites ($TEST_SUITE). We’ve configured it to use a few helper scripts
for each job:
• tools/travis/setup-$TEST_SUITE: The script that sets up the test environment for that suite (E.g.,
installing dependencies).
– For the backend and frontend suites, this is a thin wrapper around tools/provision, aka the development environment provision script.
– For the production suite, this is a more complicated process because of all the packages Travis installs.
See the comments in tools/travis/setup-production for details.
215
Zulip Documentation, Release 1.4.0
• tools/travis/$TEST_SUITE: The script that runs the actual test suite.
The main purpose of the distinction between the two is that if the setup-backend job fails, Travis CI will report
it as the suite having “Errored” (grey in their emails), whereas if the backend job fails, it’ll be reported as “Failed”
failure (red in their emails). Note that Travis CI’s web UI seems to make no visual distinction between these.
An important detail is that Travis CI will by default hide most phases other than the actual test; you can see this
easily by looking at the line numbers in the Travis CI output. There are actually a bunch of phases (e.g. the project’s
setup job, downloading caches near the beginning, uploading caches at the end, etc.), and if you’re debugging our
configuration, you’ll want to look at these closely.
Useful debugging tips and tools
• For performance issues, this statistics tool can give you test runtime history data that can help with determining
when a performance issue was introduced and whether it was fixed. Note you need to click the “Run” button for
it to do anything.
• You can sign up your personal repo for Travis CI so that every remote branch you push will be tested, which can
be helpful when debugging something complicated.
• You can use the ts tool to get timing for every line of your Travis scripts (which can help with performance
debugging). Note, however, that ts always return exit code 0, so with that patch, Travis CI will always report
success.
Performance optimizations
Caching
An important element of making Travis CI perform effectively is caching the provisioning of a Zulip development
environment. In particular, we cache the following across jobs:
• Python virtualenvs
• node_modules directories
• Built/downloaded emoji sprite sheets and data
This has a huge impact on the performance of running tests in Travis CI; without these caches, the average test time
would be several times longer.
We have designed these caches carefully (they are also used in production and the Zulip development environment)
to ensure that each is named by a hash of its dependencies, so Zulip should always be using the same version of
dependencies it would have used had the cache not existed. In practice, bugs are always possible, so be mindful of this
possibility.
A consequence of this caching is that test jobs for branches which modify package.json, requirements/, and
other key dependencies will be significantly slower than normal, because they won’t get to benefit from the cache.
Uninstalling packages
In the production suite, we run apt-get upgrade at some point (effectively, because the Zulip installer does).
This carries a huge performance cost in Travis CI, because (1) they don’t keep their test systems up to date and (2)
literally everything is installed in their build workers (e.g. several copies of Postgres, Java, MySQL, etc.).
216
Chapter 39. Travis CI
Zulip Documentation, Release 1.4.0
In order to make Zulip’s tests performance reasonably well, we uninstall (or mark with apt-mark hold) many of
these dependencies that are irrelevant to Zulip in tools/travis/setup-production.
39.4. Performance optimizations
217
Zulip Documentation, Release 1.4.0
218
Chapter 39. Travis CI
CHAPTER
40
Manual testing
As a general rule, we like to have automated tests for everything that can be practically tested. However, there are
certain types of bugs that are best caught with old fashioned manual testing (also called manual QA). Manual testing
not only catches bugs, but it also helps developers learn more about the system and think about the existing semantics
of a feature they’re working on.
This doc assumes you know how to set up a local development server and open the Zulip app in the browser. It also
assumes a basic knowledge of how to use Zulip.
Basic Stuff
When testing Zulip manually, here are things to focus on:
• The best bugs to catch are security/permissions bugs.
• Don’t rush manual testing. Look for small details like display glitches.
• Always test with multiple users (you can use incognito windows to facilitate this).
• Always keep the inspector console open and watch for warnings or errors.
• Be methodical about collecting information on bugs. (You will eventually want to create tickets, but you may
want to consolidate your own notes before filing tickets.)
You generally want to test with Cordelia as the primary user, and use Hamlet as her primary conversation partner. Use
Iago when you need to test administrative functions. Send messages to Othello or Prospero if you want to verify things
such as Cordelia not being able to receive messages not intended for her.
The rest of this document groups tasks into basic areas of functionality of the system. If you have multiple people
testing at once, you can divvy up QA tasks by these sections in the doc.
Message view
We mostly test the message view as part of testing everything else, but there are few things to specially test here.
219
Zulip Documentation, Release 1.4.0
Try using all the navigation hotkeys:
• Up/k
• Down/j
• PgUp/K
• PgDn/J/Spacebar
• End (or fn-right-arrow on OSX)
• also try scrolling aggressively with the mouse
Try narrowing from the message view:
• Hotkeys
– use Esc to go to home
– use s to narrow to a stream (select message first and verify in sidebar)
– use S to narrow to the topic (and verify in sidebar)
– use v to navigate to private messages
• Click on the recipient bar
– narrow to a stream
– narrow to a topic
– narrow to PMs with one user
– narrow to a group PM
• Click on the Zulip logo
– narrow to a topic
– click on the Zulip logo (and verify you’re in the home view)
Message editing
With message editing we mainly want to exercise topic changes.
Here are some tasks:
• Do lots of editing
– send a message to the topic “original”
– edit the message content
– send two messages to the “original” stream
– start to edit a message but then cancel
– change the topic for the first message to “change1” (just this message)
– narrow back to “original”
– send one more message to the stream
– change the topic for the last two messages to “change2”
– narrow back to “original”
– send two more messages to the stream
220
Chapter 40. Manual testing
Zulip Documentation, Release 1.4.0
– edit the 2nd message on topic and change all messages to “change3”
• Test UI entry points
– hit “i” then down arrow to edit with the popup
– use the popup using the mouse
– enter edit mode using the pencil icon
Narrowing
Zulip uses the term “narrowing” to refer to opening different views of your messages, whether by clicking on sidebar
options, recipient bars, or by using search. The main focus of these tasks should be watching unread counts. Of course,
you also want to see messages show up in the message pane. And, finally, you should make sure that no messages
outside the narrow show up in Cordelia’s view.
Important: Make sure that Cordelia is subscribed to Verona but not subscribed to Denmark; if not, you should use
different streams for your testing.
When testing narrows, you want to have Hamlet send the same message several times in a row, while cycling Cordelia
through various narrows.
Here are the main tasks for Hamlet (and each message gets sent several times):
• Send Cordelia/Othello a PM.
• Send Cordelia a PM.
• Send Othello a PM.
• Post to Verona/foo.
• Post to Verona/bar.
• Post to Denmark/foo.
• Post to Denmark/foo and mention Cordelia.
For each of the above types of messages, you will want to cycle through the following views for Cordelia (and have
Hamlet send new messages after each narrow):
• Go to Home view.
• Go to Private Messages view.
• Go to Private Messages w/Hamlet.
• Go to Private Messages w/Hamlet and Othello.
• Go to Verona view.
• Go to Verona/bar view.
• Go to Verona/foo view.
• Go to Denmark view.
• Go to Denmark/foo view.
There are 56 things to test here. If you can get into a rhythm where you can test each case in about 30 seconds, then
the whole exercise is about 30 minutes, assuming no bugs.
40.1. Basic Stuff
221
Zulip Documentation, Release 1.4.0
Composing messages
We have pretty good automated tests for our markdown processor, so manual testing is targeted more to other interactions. For composing a message, pay attention to details like what is automatically populated and where the focus is
placed.
• Hotkeys
– use r to reply to a stream message
– use r to reply to a PM
– use R to reply to the author of a PM
– use R to reply to the author of a PM stream
– use c to compose a stream message
– use C to compose a new PM
• Buttons
– Narrow to a stream and click on “New topic”
– Narrow “Private Messages” and click on “New topic”
– Narrow to a stream and click on “New private message”
– Narrow “Private Messages” and click on “New private message”
• Topics
– Compose/send a message to a stream with no topic.
– Compose/send a message to a stream with a new topic.
– Compose/send a message to a stream with autocomplete.
– Compose/send a message to a stream manually typing an existing topic.
• Formatting stuff
– Use the “A” icon to get markdown help.
– Use the eyeball icon to show a preview and send from preview mode.
– Toggle in and out of preview before sending a message.
– Use @-mention to mention Hamlet (and send him a message).
– Use #**devel** syntax and send to Hamlet, then follow the link.
– Create a bulleted list.
– Use the emoji icon to find an emoji in the picker.
• Attachments
– Send a message with an attachment using the paperclip icon.
– Send a message with multiple attachments.
– Copy an image from the clipboard.
– Use drag/drop from the desktop to upload an image.
• Drafts
– Start composing a message then click outside the compose box.
222
Chapter 40. Manual testing
Zulip Documentation, Release 1.4.0
– Use “restore drafts” to restore the draft.
– Start composing then use “Esc” to abort the message.
– Use “restore drafts” to restore the draft.
– Start composing a stream message and then abort using the little “x” icon in the compose box.
– Click on “New private message” and restore the draft. (You should now be sending to a stream.)
• Click to send
– Turn off enter-to-send.
* Send a two-paragraph message using tab and enter.
* Send a two-paragraph message using control-enter or command-enter.
– Turn on enter-to-send.
* Hit enter to send.
Popover menus
For this task you just want to go through all of our popover menus and exercise them. The main nuance here is that
you occasionally want to click somewhere on the UI outside of an existing popover to see if the popover menu is “too
sticky.” Also, occasionally actions will be somewhat jarring; for example, if you mute a message in the current view,
then the message will disappear from the view.
Here are the things to test:
• Stream sidebar menus
– Stream settings (just make sure it goes there)
– Narrow (and then have Hamlet send a message)
– Pin/unpin (do both)
– Compose (send a message to the stream)
– Mark as read (scroll back and then have Hamlet send you a message)
– Mute/unmute (do both)
– Unsubscribe (and then go to Stream settings in the gear menu to resubscribe)
– Choose custom color (play around with this)
• Topic sidebar menus
– Narrow (and then have Hamlet send a message)
– Mute/unmute (try both)
– Mark as read (scroll back and then have Hamlet send you a message)
• Left-message-pane menus (click on person’s name)
– Verify email
– Verify date message sent
– Send a PM (make sure compose box is filled out ok)
– Narrow to PMs with
– Narrow to PMs sent by
40.1. Basic Stuff
223
Zulip Documentation, Release 1.4.0
• Right-pane-pane menus (click on chevron when hovering)
– use “i” hotkey to open the menu
– Edit a message you sent (using the down-arrow key to navigate the popup)
– View Source for somebody else’s message (make sure it’s not editable)
– Reply (send a message)
– Collapse/uncollapse (try both)
– Mute/unmute (try both, watch left sidebar)
– Link to this conversation
• Buddy list chevron menus
– Narrow to PMs with
– Narrow to message sent by
– Compose a message to
Sidebar filtering
This is a fairly quick task where we test the search filters on the left sidebar and the buddy list. If Cordelia is not
subscribed to Denmark, subscribe her to that stream.
• Streams filtering
– Use “w” hotkey to open the search.
– Filter on “d”.
– Pin/unpin Denmark.
– Clear filter.
– Use “A” and “D” hotkeys to cycle through the streams.
– Filter again and then click somewhere else.
• Buddy list filtering
– Use “q” hotkey to open the search.
– Filter for Hamlet, Prospero, Othello, etc.
– Log on Hamlet and log off Hamlet while filtering for Hamlet.
– Log on/log off Hamlet while filtering for Othello.
– Log on/log off Hamlet while not filtering at all.
– Filter again and then click somewhere else.
Stream permissions
This is an important category to test, because we obviously do not want to have bugs where people can read messages
on streams they should not have access to.
The general flow here is for Hamlet to create the streams and verify that Cordelia has the correct visibility to them.
First, we start off with “positive” tests.
224
Chapter 40. Manual testing
Zulip Documentation, Release 1.4.0
• Positive tests
– Have Hamlet create a public stream w/Cordelia subscribed and have him post a message to the stream.
– Have Hamlet create a public stream without Cordelia and then...
* Have Hamlet post to the stream.
* Have Cordelia subscribe to the stream.
* Verify Cordelia can see the previous message.
* Have Cordelia post a message to the stream.
– Have Hamlet create an invite-only stream with Cordelia invited and test a two-way conversation between
the two users.
For negative tests, we want to dig a little deeper to find back doors for Cordelia to access the stream. Here are some
techniques to try:
• Try to have her compose a message to the stream by circumventing autocomplete.
• Try to have her narrow to the stream using stream:foo in search.
• Go to stream settings and see if the stream shows up.
For public streams, it’s ok for Cordelia to know the stream exists, and she can subsequently subscribe. For private
streams, she should not even know they exist (until she’s invited, of course).
• Negative tests
– Have Hamlet create a public stream without inviting Cordelia.
* Verify Cordelia can see the stream in her settings.
* Verify Cordelia can’t compose a message to the stream.
* Verify that Cordelia sees nothing when Hamlet posts to the stream.
– Have Hamlet create a public stream with Cordelia, but then have Iago revoke her subscription using the
admin page.
* Verify that the stream appears in Cordelia’s left sidebar and then goes away.
* Try to have Cordelia view the stream using a sneaky search along the lines of stream:foo.
– Have Hamlet create a private stream without inviting Cordelia.
* Verify Cordelia can’t compose a message to the stream.
Search
The main task for testing search is to play around with search suggestions (autocomplete). Once you select an option,
verify the message view is consistent with the search and that the left sidebar reflects the current narrow. If a search
comes up legitimately empty, have Hamlet send a message that matches the search.
Here are searches you should be able to do with autocomplete:
• stream:design
• stream:Verona topic:Verona1
• stream:Verona keyword
• sent by me
• @-mentions
40.1. Basic Stuff
225
Zulip Documentation, Release 1.4.0
• starred messages
• messages sent by Hamlet
• PMs with Hamlet
• PMs with Hamlet matching keyword “foo”
There are some things you can try that don’t come up in autocomplete:
• -stream:Verona (exclude Verona)
• stream:Verona stream:devel (should return no results)
Miscellaneous:
• Use the “/” hotkey to start a search.
• Use the “x” icon to clear a search.
• Use the “Esc” hotkey to clear a search.
Stream settings
Test various UI entry points into stream settings:
• Use small gear menu in left sidebar, then filter to “devel”.
• Use popover menu in left sidebar next to “devel”.
• Use gear menu above buddy list and filter to “devel”.
• Use gear menu and click on “devel.”
• Use gear menu and then click on chevron menu next to “devel.” (I’m not sure why we still have the chevron at
this writing.)
Create new public stream “public1” and add Hamlet:
• Type “public1” in the text box and then click “Create new stream.”
• Select “People must be invited” and then verify you can’t select “Announce stream”.
• Select “Anyone can join” again to make it be public.
• Check the checkbox for Hamlet.
• Hit the “Create” button.
Test subscribe/unsubscribe:
• Log in as Hamlet and go to his stream settings.
• As Cordelia, unsubscribe from “public1” using the checkmark in the streams settings page.
• Verify that Hamlet sees that Cordelia has unsubscribed (and the subscriber count should decrement).
• As Cordelia, resubscribe to “public1.”
• Verify Hamlet sees that change.
As Cordelia, exercise different options in Create Stream dialog by creating streams s1, s2, s3, etc.:
• s1: anyone can join, announce it, and add Hamlet using filter feature
• s2: people must be invited
• s3: anyone can join, don’t announce
226
Chapter 40. Manual testing
Zulip Documentation, Release 1.4.0
• s4: check all, then uncheck all, then invite only Hamlet
• s5: invite everybody but Hamlet
• s6:
– create the stream as public, but don’t subscribe anybody initially
– then click on stream options to add Hamlet using “Add” button
Test per-stream options:
• Use “devel” stream and send a message to it
• Do mute and unmute, have Hamlet send messages
• Test notifications on/off, have Hamlet send messages
• Test pin and unpin, view left sidebar
• Change stream color, and then view the left sidebar and the Home message view
• Verify stream subscriber counts in the main stream view
User Settings
You can modify per-user settings by choosing “Settings” in the gear menu. Do these tasks as Cordelia.
• Your account
– Change full name (Hamlet should see the name change)
– Customize avatar
– Deactivate account (and then log in as Iago to re-activate Cordelia)
• Display settings
– Right now, these unfortunately require reloads to take effect.
– Default language (change to Spanish)
– Display user list in narrow windows (verify by making window thinner)
– Display time in 24-hour format (and then test going back to AM/PM)
• Notifications
– Stream Message
* turn off notifications at user level
· create a new stream
· have Hamlet send a message
* turn on notifications at user level
· create a new stream
· have Hamlet send a message
· then turn off notifications for that stream
· have Hamlet send another message
– Private Messages and @-mentions
* Test Desktop/Audible options
40.1. Basic Stuff
227
Zulip Documentation, Release 1.4.0
* You can ignore other stuff for now
• Bots/API key
– Create a bot with a generic avatar and send it a PM
– Create a bot with a custom avatar and send it a PM
– Change your API key
• Alert words
– Create an alert word
– Have Hamlet send you a message that includes the alert word
• Zulip labs
– Turn on auto-scroll to new messages (and have Hamlet send you one)
– Turn on/off “Enable desktop notifications for new streams” and test. (We may eliminate this option soon.)
Keyboard Shorcuts
We mostly test keyboard shortcuts as part of other tasks.
Here are the tasks for this section:
• Use the ”?” hotkey to open the keyboard help
• Proofread the dialog for typos.
• Close the dialog.
• Re-open the keyboard help using the gear menu.
• Find a hotkey that you don’t frequently use and experiment with its usage.
Miscellaneous menu options
Make sure that these options launch appropriate help screens:
• Proofread and try a couple random options:
– Message formatting
– Search operators
• Make sure help launches in a separate browser tab:
– Desktop and mobile apps
– Integrations
– API documentation
Inviting users/tutorial
Here are the tasks:
• Invite ignore@zulip.com using the link beneath the buddy list but then don’t take further action.
• Fully invite foo@zulip.com using the gear menu.
228
Chapter 40. Manual testing
Zulip Documentation, Release 1.4.0
• Go to the development console to get the login link for foo@zulip.com.
• Go through the signup flow.
• Follow the tutorial.
• Use the gear menu to log out.
• Log back in as Cordelia (admittedly, this step doesn’t really QA much of our production code, since the login
flow is customized for the development environment).
To be continued...
This document does not cover settings/admin options yet. The main things to do when testing the settings system are:
• Verify that changes are synced to other users.
• Verify error messages appear if you do something wrong and look right.
• For organization settings, verify that they look right in read-only mode (i.e. when not logged into an administrator account).
40.1. Basic Stuff
229
Zulip Documentation, Release 1.4.0
230
Chapter 40. Manual testing
CHAPTER
41
Settings system
The page documents the Zulip settings system, and hopefully should help you decide how to correctly implement
new settings you’re adding to Zulip. We have two types of administrative settings in Zulip: server settings (which
are set via configuration files are apply to the whole Zulip installation), and realm settings (which are usually set via
the /#organization page in the Zulip web application) and apply to a single Zulip realm/organization (which for most
Zulip servers is the only realm on the server).
Philosophically, the goals of the settings system are to make it convenient for:
• Zulip server administrators to configure Zulip’s featureset for their server without needing to patch Zulip
• Realm administrators to configure settings for their organization independently without needing to talk with the
server administrator.
• Secrets (passwords, API keys, etc.) to be stored in a separate place from shareable configuration.
Server settings
Zulip uses the Django settings system, which means that the settings files are Python programs that set a lot of variables
with all-capital names like EMAIL_GATEWAY_PATTERN. You can access these anywhere in the Zulip Django code
using e.g.:
from django.conf import settings
print(settings.EMAIL_GATEWAY_PATTERN)
Additionally, if you need to access a Django setting in a shell script (or just on the command line for debugging), you
can use e.g.:
$ ./scripts/get-django-setting EMAIL_GATEWAY_PATTERN
%s@localhost:9991
Zulip has separated those settings that we expect a system administrator to change (with nice documentation) from the
~1000 lines of settings needed by the Zulip Django app. As a result, there are a few files involved in the Zulip settings
for server administrators. In a production environment, we have:
231
Zulip Documentation, Release 1.4.0
• /etc/zulip/settings.py
(the
template
is
in
the
Zulip
repo
at
zproject/
prod_settings_template.py) is the main system administrator-facing settings file for Zulip. It
contains all the server-specific settings, such as how to send outgoing email, the hostname of the Postgres
database, etc., but does not contain any secrets (e.g. passwords, secret API keys, cryptographic keys, etc.). The
way we generally do settings that can be controlled with shell access to a Zulip server is to put a default in
zproject/settings.py, and then override it here.
• /etc/zulip/zulip-secrets.conf (generated by scripts/setup/generate_secrets.py as
part of installation) contains secrets used by the Zulip installation. These are read using the standard
Python ConfigParser, and accessed in zproject/settings.py by the get_secret function. All
secrets/API keys/etc. used by the Zulip Django application should be stored here, and read using the
get_secret function in zproject/settings.py.
• zproject/settings.py is the main Django settings file for Zulip. It contains all the settings that are
constant for all Zulip installations (e.g. configuration for logging, static assets, middleware, etc.), as well
as default values for the settings the user would set in /etc/zulip/settings.py (you can look at the
DEFAULT_SETTINGS dictionary to easily review the settings available). zproject/settings.py has a
line from prod_settings import *, which has the effect of importing /etc/zulip/settings.
py in a prod environment (via a symlink).
In a development environment, we have zproject/settings.py, and additionally:
• zproject/dev_settings.py has the settings for the Zulip development environment; it mostly just imports prod_settings_template.py.
• zproject/dev-secrets.conf replaces /etc/zulip/zulip-secrets.conf.
• zproject/test_settings.py has the (default) settings used for the Zulip tests (both backend and
Casper), which are applied on top of the development environment settings.
When adding a new server setting to Zulip, you will typically add it in two or three places:
• In DEFAULT_SETTINGS in zproject/settings.py, with a default value for production environments.
If the settings has a secret key, you’ll add a get_secret call in zproject/settings.py (and the user
will add the value when they configure the feature).
• In an appropriate section of zproject/prod_settings_template.py, with documentation in the comments explaining the settings’s purpose and effect.
• Possibly also zproject/dev_settings.py, if the desired value of the setting for Zulip development
environments is different from the default for production (and similarly for zproject/test_settings.
py).
Most settings should be enabled in the development environment, to maximize convenience of testing all of Zulip’s
features; they should be enabled by default in production if we expect most Zulip sites to want those settings.
Testing non-default settings
You can write tests for settings using e.g. with self.settings(GOOGLE_CLIENT_ID=None). However,
this only works for settings which are checked at runtime, not settings which are only accessed in initialization of
Django (or Zulip) internals (e.g. DATABASES). See the Django docs on overriding settings in tests for more details.
Realm settings
Realm settings are preferred for any configuration that is a matter of organizational policy (as opposed to technical
capabilities of the server). As a result, configuration options for user-facing functionality is almost always added as a
232
Chapter 41. Settings system
Zulip Documentation, Release 1.4.0
new realm setting, not a server setting. The new feature tutorial documents the process for adding a new realm setting
to Zulip.
So for example, the following server settings will eventually be replaced with realm settings:
• NAME_CHANGES_DISABLED
• INLINE_IMAGE_PREVIEW
• ENABLE_GRAVATAR
• Which authentication methods are allowed should probably appear in both places; in server settings indicating
the capabilities of the server, and in the realm settings indicating which methods the realm administrator wants
to allow users to login with.
41.2. Realm settings
233
Zulip Documentation, Release 1.4.0
234
Chapter 41. Settings system
CHAPTER
42
Real-time Push and Events
Zulip’s “events system” is the server-to-client push system that powers our real-time sync. This document explains
how it works; to read an example of how a complete feature using this system works, check out the new application
feature tutorial.
Any single-page web application like Zulip needs a story for how changes made by one client are synced to other
clients, though having a good architecture for this is particularly important for a chat tool like Zulip, since the state is
constantly changing. When we talk about clients, think a browser tab, mobile app, or API bot that needs to receive
updates to the Zulip data. The simplest example is a new message being sent by one client; other clients must be
notified in order to display the message. But a complete application like Zulip has dozens of different types of data
that need to be synced to other clients, whether it be new streams, changes in a user’s name or avatar, settings changes,
etc. In Zulip, we call these updates that need to be sent to other clients events.
An important thing to understand when designing such a system is that events need to be synced to every client that
has a copy of the old data if one wants to avoid clients displaying inaccurate data to users. So if a user has two browser
windows open and sends a message, every client controlled by that user as well as any recipients of the message,
including both of those two browser windows, will receive that event. (Technically, we don’t need to send events to
the client that triggered the change, but this approach saves a bunch of unnecessary duplicate UI update code, since
the client making the change can just use the same code as every other client, maybe plus a little notification that the
operation succeeded).
Architecturally, there are a few things needed to make a successful real-time sync system work:
• Generation. Generating events when changes happen to data, and determining which users should receive each
event.
• Delivery. Efficiently delivering those events to interested clients, ideally in an exactly-once fashion.
• UI updates. Updating the UI in the client once it has received events from the server.
Reactive JavaScript libraries like React and Vue can help simplify the last piece, but there aren’t good standard systems
for doing generation and delivery, so we have to build them ourselves.
This document discusses how Zulip solves the generation and delivery problems in a scalable, correct, and predictable
way.
235
Zulip Documentation, Release 1.4.0
Generation system
Zulip’s generation system is built around a Python function, send_event(event, users). It accepts an event
data structure (just a Python dictionary with some keys and value; type is always one of the keys but the rest depends
on the specific event) and a list of user IDs for the users whose clients should receive the event. In special cases such as
message delivery, the list of users will instead be a list of dicts mapping user IDs to user-specific data like whether that
user was mentioned in that message. The data passed to send_event are simply marshalled as JSON and placed in
the notify_tornado RabbitMQ queue to be consumed by the delivery system.
Usually, this list of users is one of 3 things:
• A single user (e.g. for user-level settings changes).
• Everyone in the realm (e.g. for organization-level settings changes, like new realm emoji).
• Everyone who would receive a given message (for messages, emoji reactions, message editing, etc.); i.e. the
subscribers to a stream or the people on a private message thread.
It is the responsibility of the caller of send_event to choose the list of user IDs correctly. There can be security
problems if e.g. an event containing private message content is sent to the entire organization. However, if an event
isn’t sent to enough clients, there will likely be user-visible real-time sync bugs.
Most of the hard work in event generation is about defining consistent event dictionaries that are clear, readable, will
be useful to the wide range of possible clients, and make it easy for developers.
Delivery system
Zulip’s event delivery (real-time push) system is based on Tornado, which is ideal for handling a large number of open
requests. Details on Tornado are available in the architecture overview, but in short it is good at holding open a large
number of connections for a long time. The complete system is about 1500 lines of code in zerver/tornado/,
primarily zerver/tornado/event_queue.py.
Zulip’s event delivery system is based on “long-polling”; basically clients make GET /json/events calls to the
server, and the server doesn’t respond to the request until it has an event to deliver to the client. This approach is
reasonably efficient and works everywhere (unlike websockets, which have a decreasing but nonzero level of client
compatibility problems).
For each connected client, the Event Queue Server maintains an event queue, which contains any events that are to
be delivered to that client which have not yet been acknowledged by that client. Ignoring the subtle details around
error handling, the protocol is pretty simple; when a client does a GET /json/events call, the server checks if
there are any events in the queue. If there are, it returns the events immediately. If there aren’t, it records that queue as
having a waiting client (often called a handler in the code).
When it pulls an event off the notify_tornado RabbitMQ queue, it simply delivers the event to each queue
associated with one of the target users. If the queue has a waiting client, it breaks the long-poll connection by returning
an HTTP response to the waiting client request. If there is no waiting client, it simply pushes the event onto the queue.
When starting up, each client makes a POST /json/register to the server, which creates a new event queue for
that client and returns the queue_id as well as an initial last_event_id to the client (it can also, optionally,
fetch the initial data to save an RTT and avoid races; see the below section on initial data fetches for details on
why this is useful). Once the event queue is registered, the client can just do an infinite loop calling GET /json/
events with those parameters, updating last_event_id each time to acknowledge any events it has received
(see call_on_each_event in the Zulip Python API bindings for a complete example implementation). When
handling each GET /json/events request, the queue server can safely delete any events have have an event ID
less than or equal to the client’s last_event_id (event IDs are just a counter for the events a given queue has
received.)
236
Chapter 42. Real-time Push and Events
Zulip Documentation, Release 1.4.0
If network failures were impossible, the last_event_id parameter in the protocol would not be required, but it is
important for enabling exactly-once delivery in the presence of potential failures. (Without it, the queue server would
have to delete events from the queue as soon as it attempted to send them to the client; if that specific HTTP response
didn’t reach the client due to a network TCP failure, then those events could be lost).
The queue servers are a very high-traffic system, processing at a minimum a request for every message delivered to
every Zulip client. Additionally, as a workaround for low-quality NAT servers that kill HTTP connections that are
open without activity for more than 60s, the queue servers also send a heartbeat event to each queue at least once every
45s or so (if no other events have arrived in the meantime).
To avoid a large memory and other resource leak, the queues are garbage collected after (by default) 10 minutes of
inactivity from a client, under the theory that the client has likely gone off the Internet (or no longer exists) access;
this happens constantly. If the client returns, it will receive a “queue not found” error when requesting events; it’s
handler for this case should just restart the client / reload the browser so that it refetches initial data the same way it
would on startup. Since clients have to implement their startup process anyway, this approach adds minimal technical
complexity to clients. A nice side effect is that if the Event Queue Server server (which stores queues in memory)
were to crash and lose its data, clients would recover, just as if they had lost Internet access briefly (there is some DoS
risk to manage, though).
(The Event Queue Server is designed to save any event queues to disk and reload them when the server is restarted, and
catches exceptions carefully, so such incidents are very rare, but it’s nice to have a design that handles them without
leaving broken out-of-date clients anyway).
The initial data fetch
When a client starts up, it usually wants to get 2 things from the server:
• The “current state” of various pieces of data, e.g. the current settings, set of users in the organization (for
typeahead), stream, messages, etc. (aka the “initial state”).
• A subscription to receive updates to those data when they are changed by a client (aka an event queue).
Ideally, one would get those two things atomically, i.e. if some other user changes their name, either the name change
happens before the fetch (and thus the old name is in the initial state and there will be an event in the queue for the
name change) or after (the new name is in the initial state, and there is no event for that name change in the queue).
Achieving this atomicity goals means we save a huge amount of work that the N clients for Zulip don’t need to worry
about a wide range of potential rare and hard to reproduce race conditions; we just have to implement things correctly
once in the Zulip server.
This is quite challenging to do technically, because fetching the initial state for a complex web application like Zulip
might involve dozens of queries to the database, caches, etc. over the course of 100ms or more, and it is thus nearly
impossible to do all of those things together atomically. So instead, we use a more complicated algorithm that can
produce the atomic result from non-atomic subroutines. Here’s how it works when you make a register API
request; the logic is in zerver/views/events_register.py and zerver/lib/events.py. The request
is directly handled by Django:
• Django makes an HTTP request to Tornado, requesting that a new event queue be created, and records its queue
ID.
• Django does all the various database/cache/etc. queries to fetch the data, non-atomically, from the various data
sources (see the fetch_initial_state_data function).
• Django makes a second HTTP request to Tornado, requesting any events that had been added to the Tornado
event queue since it was created.
42.3. The initial data fetch
237
Zulip Documentation, Release 1.4.0
• Finally, Django “applies” the events (see the apply_events function) to the initial state that it fetched. E.g.
for a name change event, it finds the user data in the realm_user data struture, and updates it to have the new
name.
This achieves everything we desire, at the cost that we need to write the apply_events function. This is a difficult
function to implement correctly, because the situations that it tests for almost never happen (being race conditions). So
we have a special test class, EventsRegisterTest, that is specifically designed to test this function by ensuring
the possible race condition always happens. In particular, it does the following:
• Call fetch_initial_state_data to get the current state.
• Call a state change function that issues an event, e.g. do_change_full_name, and capture any events that
are generated.
• Call
apply_events(state, events),
fetch_initial_state_data again now.
and
compare
the
result
to
calling
The apply_events code is correct if those two results are identical.
The final detail we need to ensure that apply_events always works correctly is to make sure that we have
EventsRegisterTest tests for every event type that can be generated by Zulip. This can be tested manually using
test-backend --coverage EventsRegisterTest and then checking that all the calls to send_event
are covered. Someday we’ll add automation that verifies this directly by inspecting the coverage data.
In the Zulip webapp, the data returned by the register API is available via the page_params parameter.
Messages
One exception to the protocol described in the last section is the actual messages. Because Zulip clients usually fetch
them in a separate AJAX call after the rest of the site is loaded, we don’t need them to be included in the initial state
data. To handle those correctly, clients are responsible for discarding events related to messages that the client has not
yet fetched.
238
Chapter 42. Real-time Push and Events
CHAPTER
43
Queue processors
Zulip uses RabbitMQ to manage a system of internal queues. These are used for a variety of purposes:
• Asynchronously doing expensive operations like sending email notifications which can take seconds per email
and thus would otherwise timeout when 100s are triggered at once (E.g. inviting a lot of new users to a realm).
• Asynchronously doing non-time-critical somewhat expensive operations like updating analytics tables (e.g.
UserActivityInternal) which don’t have any immediate runtime effect.
• Communicating events to push to clients (browsers, etc.) from the main Zulip Django application process to the
Tornado-based events system. Example events might be that a new message was sent, a user has changed their
subscriptions, etc.
• Processing mobile push notifications and email mirroring system messages.
• Processing various errors, frontend tracebacks, and slow database queries in a batched fashion.
• Doing markdown rendering for messages delivered to the Tornado via websockets.
Needless to say, the RabbitMQ-based queuing system is an important part of the overall Zulip architecture, since it’s in
critical code paths for everything from signing up for account, to rendering messages, to delivering updates to clients.
We use the pika library to interface with RabbitMQ, using a simple custom integration defined in zerver/lib/
queue.py.
Adding a new queue processor
To add a new queue processor:
• Define the processor in zerver/worker/queue_processors.py using the @assign_queue decorator; it’s pretty easy to get the template for an existing similar queue processor. This suffices to test your queue
worker in the Zulip development environment (tools/run-dev.py will automatically restart the queue processors and start running your new queue processor code). You can also run a single queue processor manually
using e.g. ./manage.py process_queue --queue=user_activity.
239
Zulip Documentation, Release 1.4.0
• So that supervisord will known to run the queue processor in production, you will need to add to to
normal_queues in puppet/zulip/manifests/base.pp; the list there is used to generate /etc/
supervisor/conf.d/zulip.conf via a puppet template in app_frontend.pp.
The queue will automatically be added to the list of queues tracked by scripts/nagios/
check-rabbitmq-consumers, so Nagios can properly check whether a queue processor is running for
your queue. You still need to update the sample Nagios configuration in puppet/zulip_ops manually.
Publishing events into a queue
You can publish events to a RabbitMQ queue using the queue_json_publish function defined in zerver/
lib/queue.py.
Clearing a RabbitMQ queue
If you need to clear a queue (delete all the events in it), run ./manage.py purge_queue <queue_name>, for
example:
./manage.py purge_queue user_activity
You can also use the amqp tools directly. Install amqp-tools from apt and then run:
amqp-delete-queue --username=zulip --password='...' --server=localhost \
--queue=user_presence
with the RabbitMQ password from /etc/zulip/zulip-secrets.conf.
240
Chapter 43. Queue processors
CHAPTER
44
Zulip bot system
Zulip’s features can be extended by the means of bots and integrations.
• Integrations are used to connect Zulip with different chat, scheduling and workflow software. If this is what
you are looking for, please check out the integrations guide.
• Bots, as a more general concept, intercept and react to messages. If this is what you are looking for, read on!
The purpose of this documentation is to provide you with information about Zulip’s bot system.
On this page you’ll find:
• A step-by-step tutorial on how to run a bot.
• A step-by-step tutorial on how to develop a bot.
• A documentation of the bot API.
• Common problems when developing/running bots and their solutions.
Contributions to this guide are very welcome, so if you run into any issues following these instructions or come up
with any tips or tools that help with writing bots, please visit #bots on the Zulip development community server, open
an issue, or submit a pull request to share your ideas!
The bots system
Zulip’s bot system resides in the api directory.
The structure of the bots ecosystem in the api directory looks like the following:
api
--bots
--bot1
--bot2
|
--readme.md
--bot2.py
241
Zulip Documentation, Release 1.4.0
|
|
|
|
-
--bot2.config
--libraries
|
|
|
--lib1.py
--assets
|
--pic.png
bots_api
- bot_lib.py
- bots_test_lib.py
- run.py
- test_bots
integrations
Each subdirectory in bots contains a bot. When developing bots, try to use the structure outlined above as an
orientation.
How to run a bot
This guide will show you how to run a bot on a running Zulip server. It assumes you want to use one of the existing
api/bots bots in your Zulip organization. If you want to write a new one, you just need to write the <my-bot>.py
script and put it into /api/bots/<my-bot> directory.
You need:
• An account in an organization on a Zulip server (e.g. chat.zulip.org or yourSubdomain.zulipchat.com, or your
own development server). Within that Zulip organization, users will be able to interact with your bot.
• A computer where you’re running the bot from, with a clone of the Zulip repository, which contains the bot
library code in its api/bots_api/<my-bot> subdirectory. This is required to run your bot. The following
instructions assume this repository to be located in ~/zulip/.
Note: Please be considerate when testing experimental bots on public servers such as chat.zulip.org.
1. Register a new bot user on the Zulip server’s web interface.
• Log in to the Zulip server.
• Navigate to Settings -> Your bots -> Add a new bot, fill out the form and click on Create bot.
• A new bot user should appear in the Your bots panel.
2. Download the bot’s zuliprc configuration file to your computer.
• In the Your bots panel, click on the green icon to download its configuration file zuliprc (the structure of
this file is explained here.)
• Copy the file to a destination of your choice, e.g. to ~/.zuliprc or ~/zuliprc-test. Note that the
destination should be accessible from your Zulip dev environment (e.g. Vagrant or Digital Ocean).
3. Subscribe the bot to the streams that the bot needs to interact with.
• To subscribe your bot to streams, navigate to Manage Streams. Select a stream and add your bot by its
email address (the address you assigned in step 1).
• Now, the bot can do its job on the streams you subscribed it to.
• (In future versions of the API, this step may not be required).
4. Run the bot.
242
Chapter 44. Zulip bot system
Zulip Documentation, Release 1.4.0
• In your Zulip repository, navigate to ~/zulip/api/bots_api/
• Run
python run.py ../bots/<my-bot>/<my-bot>.py --config-file ~/.zuliprc`
(using the path to the .zuliprc file from step 2).
• Check the output of the command. It should start with the text the usage function returns, followed by
logging output similar to this:
INFO:root:starting message handling...
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection
˓→(1): localhost
• Congrats! Now, your bot should be ready to test on the streams you’ve subscribed it to.
Testing the helloworld bot
• The helloworld bot is a simple bot that responds with a ‘beep boop’ when queried. It can be used as a
template to build more complex bots.
• Go to a stream your bot is subscribed to. Talk to the bot by typing @<your bot name> followed by some
commands. If the bot is the helloworld bot, you should expect the bot to respond with “beep boop”.
How to develop a bot
The tutorial below explains the structure of a bot <my-bot>.py, which is the only file you need to create to develop
a new bot. You can use this as boilerplate code for developing your own bot.
Every bot is built upon this structure:
class MyBotHandler(object):
'''
A docstring documenting this bot.
'''
def usage(self):
return '''Your description of the bot'''
def handle_message(self, message, client, state_handler):
# add your code here
handler_class = MyBotHandler
• The class name (in this case MyBotHandler) can be defined by you and should match the name of your bot. To
register your bot’s class, adjust the last line handler_class = MyBotHandler to match your class name.
• Every bot needs to implement the functions
– usage(self)
– handle_message(self, message, client)
• These functions are documented in the next section.
44.3. How to develop a bot
243
Zulip Documentation, Release 1.4.0
Bot API
This section documents functions available to the bot and the structure of the bot’s config file.
With this API, you can
• intercept, view, and process messages sent by users on Zulip.
• send out new messages as replies to the processed messages.
With this API, you cannot
• modify an intercepted message (you have to send a new message).
• send messages on behalf of or impersonate other users.
• intercept private messages (except for PMs with the bot as an explicit recipient).
usage
usage(self)
is called to retrieve information about the bot.
Arguments
• self - the instance the method is called on.
Return values
• A string describing the bot’s functionality
Example implementation
def usage(self):
return '''
This plugin will allow users to flag messages
as being follow-up items. Users should preface
messages with "@followup".
Before running this, make sure to create a stream
called "followup" that your API user can send to.
'''
handle_message
handle_message(self, message, client)
handles user message.
244
Chapter 44. Zulip bot system
Zulip Documentation, Release 1.4.0
Arguments
• self - the instance the method is called on.
• message - a dictionary describing a Zulip message
• client - used to interact with the server, e.g. to send a message
• state_handler - used to save states/information of the bot beta
– use state_handler.set_state(state) to set a state (any object)
– use state_handler.get_state() to retrieve the state set; returns a NoneType object if no state
is set
Return values
None.
Example implementation
def handle_message(self, message, client, state_handler):
original_content = message['content']
original_sender = message['sender_email']
new_content = original_content.replace('@followup',
'from %s:' % (original_sender,))
client.send_message(dict(
type='stream',
to='followup',
subject=message['sender_email'],
content=new_content,
))
client.send_message
client.send_message(message)
will send a message as the bot user. Generally, this is less convenient than send_reply, but it offers additional flexibility
about where the message is sent to.
Arguments
• message - a dictionary describing the message to be sent by the bot
Example implementation
client.send_message(dict(
type='stream', # can be 'stream' or 'private'
to=stream_name, # either the stream name or user's email
subject=subject, # message subject
content=message, # content of the sent message
))
44.4. Bot API
245
Zulip Documentation, Release 1.4.0
client.send_reply
client.send_reply(message, response)
will reply to the triggering message to the same place the original message was sent to, with the content of the reply
being response.
Arguments
• message - Dictionary containing information on message to respond to (provided by handle_message).
• response - Response message from the bot (string).
client.update_message
client.update_message(message)
will edit the content of a previously sent message.
Arguments
• message - dictionary defining what message to edit and the new content
Example
From /zulip/api/bots/incrementor/incrementor.py:
client.update_message(dict(
message_id=self.message_id, # id of message to be updated
content=str(self.number), # string with which to update message with
))
Configuration file
[api]
key=<api-key>
email=<email>
site=<dev-url>
• key - the API key you created for the bot; this is how Zulip knows the request is from an authorized user.
• email - the email address of the bot, e.g. some-bot@zulip.com
• site - your development environment URL; if you are working on a development environment hosted on your
computer, use localhost:9991
Common problems
• I modified my bot’s code, yet the changes don’t seem to have an effect.
– Ensure that you restarted the run.py script.
246
Chapter 44. Zulip bot system
Zulip Documentation, Release 1.4.0
• My bot won’t start
– Ensure that your API config file is correct (download the config file from the server).
– Ensure that you bot script is located in zulip/api/bots/<my-bot>/
– Are you using your own Zulip development server? Ensure that you run your bot outside the Vagrant
environment.
– Some bots require Python 3. Try switching to a Python 3 environment before running your bot:
source /srv/zulip-py3-venv/bin/activate
Note that you can switch back to a Python 2 environment as follows:
source /srv/zulip-venv/bin/activate
• My bot works only on some streams.
– Subscribe your bot to other streams, as described here.
Future direction
The long-term plan for this bot system is to allow the same BotHandler code to eventually be usable in several
contexts:
• Run directly using the Zulip call_on_each_message API, which is how the implementation above works.
This is great for quick development with minimal setup.
• Run in a simple Python webserver server, processing messages received from Zulip’s outgoing webhooks integration.
• For bots merged into the mainline Zulip codebase, enabled via a button in the Zulip web UI, with no code
deployment effort required.
44.6. Future direction
247
Zulip Documentation, Release 1.4.0
248
Chapter 44. Zulip bot system
CHAPTER
45
Custom Apps
Definition
Zulip defines a “custom app” to be a piece of code that runs in the Zulip ecosystem, but which is not part of the
core Zulip codebase. Custom apps are mostly synonymous with “bots” and “integrations” in the Zulip ecosystem. We
currently do not support any kind of browser plugin model.
Problem statement
Zulip wants to enable people in the world to author custom apps with the following goals in mind:
• Simple custom apps should be simple to write and deploy.
• Custom app authors should be able to easily distribute their work.
• Zulip should provide deployment support for mature, general-purpose bots, ideally either within organizations (a
Zulip admin can vet her own custom apps and easily deploy them across upgrade cycles) or across organizations
(custom apps get distributed with the Zulip tarball).
This document describes Zulip’s current infrastructure, as well as laying out a roadmap for some future features.
A quick note on bots/integrations
As noted earlier, a custom app is just a generic term for what we often call bots or integrations. We recognize that bots
and integrations can have different connotations. A bot typically spends most of its time responding to Zulip messages.
An integration usually represents an app that interacts with some large third party system like an issue tracker. We
will use both terms in this document in an informal sense, but from an architecture standpoint, we treat bots and
integrations as essentially two shades of the same color. Many integrations are implemented as “bots.” Likewise, any
bot that does stuff outside of Zulip acts as an “integration.”
249
Zulip Documentation, Release 1.4.0
Since the line between what a “bot” is and what an “integration” is can get very blurry, we try to be informal about
“bots/integrations” and more formal about how “custom apps” actually function within the system.
Categories of custom apps
Stimulus/response and read/write
At the end of the day, most useful apps respond to some stimulus and produce a response. In the Zulip universe, Zulip
can be the source of the stimulus, or the target of the response, or both. Along those lines, we divide custom apps into
these three types:
• A Zulip Reader uses activity on Zulip to stimulate an external response. An example here would be a follow-up
bot that sees messages with the alert word “@todo” on a stream and then adds a task to a third party todo-list
tool.
• A Zulip Writer reacts to external stimuli and generates Zulip responses. An example here might be a build bot
that gets triggered by an automated code build finishing and then writes “build finished” to a Zulip stream.
• A Zulip Read/Writer reacts to a stimulus from Zulip by responding to Zulip. An example here would be a
math bot that sees a message saying “compute 2+2” and responds with “2+2=4” on the same stream or back to
the user in a PM.
The above three classifications represent kind of a Zulip-centric view of the universe, but we should put ourselves in
the shoes of somebody “out in the world.”
• A World Reader is an app that gets some stimulus from the outside world and produces a response in Zulip.
(So, a world reader is a Zulip writer.)
• A World Writer is an app that gets some stimulus from Zulip and produces a response in the outside world.
(So, a world writer is a Zulip reader.)
Some things are a little outside of the scope of this document. We could plausibly extend Zulip some day to host World
Reader/Writer apps that don’t even write Zulip messages but simply use Zulip as a kind of middleware platform.
More in the short term, we will have custom apps that may read/write from multiple sources. For example, a meeting
bot may take input from both a cron job and a Zulip stream, and it may write to both a Zulip stream and a third party
calendar tool. For the scope of this document, we won’t spend a lot of time talking about how to build these types of
apps, but we are aware that any solution needs to accommodate multiple sources and targets.
World Reader/Zulip Reader
Finally, we set the stage for how we talk about custom apps in terms of these two broad categories:
• A World Reader responds to stimuli from the outside world (and typically produces a response in Zulip).
• A Zulip Reader responds to stimuli from Zulip conversations (and typically produces a response in the outside
world).
Again, we recognize that there can be overlap between those two categories for complex custom apps, but we mostly
leave it as an exercise for the reader how to implement those apps.
Other classifications
We discussed one dimension for classifying custom apps, which is whether they are world-readers or Zulip-readers.
Here we cover a few other classification schemes briefly:
250
Chapter 45. Custom Apps
Zulip Documentation, Release 1.4.0
• Generality Does the custom app have a specific use case or a general one? The spectrum here could run from a
bot that Alice runs to update a text file on her laptop (specific) to a Twitter Bot that is optionally deployed on all
Zulip realms (general).
• Authorship Who wrote the custom app? Was it written by contributors to the Zulip project?
• Maturity How well tested is the custom app? Is it just a prototype? Has it been sanctioned by an open source
community? Has it been vetted by Zulip developers?
• Deployment Where does the custom app run? Does it run on Alice’s laptop? Does it run on a Zulip server?
Does it run as a plugin on third party infrastructure?
• Authorization What streams are the custom app allowed to read and write from? Which users can the custom
app interact with?
• Identity How does the custom app identify itself on Zulip? How does it identify itself to the outside world?
• Third party We call the non-Zulip target or source of a custom app the “world.” The “world” could be almost
anything, ranging from an electronic device or text file to a large third-party system like Twitter or GitHub.
A lot of the classification schemes are interrelated. Here are some examples:
• For specific-purpose custom apps, authors may be happy to just deploy them on their own hardware. For generaluse custom apps, authors may want to have them deployed on the Zulip server with super-user capabilities.
• As a custom app becomes more well-tested and well-vetted, the author will likely upgrade its deployment over
time. At first the author may run the custom app on their laptop, then they may find dedicated hardware, and
then finally they contribute the app to the Zulip project so that Zulip admins can deploy the app on Zulip servers.
• The nature of the third party will influence the deployment strategy. If I have a little home-grown gadget that can
turn off the lights in my kitchen, I may run a custom app on my laptop that reads my PMs for “turn-off-the-light”
messages. If I write a generic custom app that needs to update a third party corporate system based on Zulip
events, I may want to deploy code to a public webserver or try to get my code to be part of the Zulip project
itself.
World Reader
A World Reader custom app is an app that responds to stimuli from the world outside of Zulip. It typically functions
as a Zulip Writer and posts some kind of message to a Zulip stream or user to alert people of world events. Here are
some example stimuli:
• A Travis build finishes.
• Somebody tweets on Twitter.
• A hardware sensor notices a temperature increase.
• A pull request is submitted to GitHub.
• A cron job gets started on your laptop to send a reminder.
• Nagios detects a system anomaly.
Setting aside issues of how a custom app is constructed or deployed, you basically have to solve these problems:
• Detect events.
• Translate events into Zulip messages.
• Post the messages to Zulip.
45.5. World Reader
251
Zulip Documentation, Release 1.4.0
Zulip integrations
Zulip actually supports a bunch of integrations out-of-the-box that perform as World Readers.
The three different integration models basically differ in where they perform the main functions of a World Reader.
Webhook integrations
In a webhook integration, the deployment model is usually this::
3rd party hardware:
• detect event
• send data to Zulip webhook
Zulip:
• support webhook endpoint
• translate event to messages
• internally post messages
One current limitation of our system is that we don’t have a great way to deploy prototypes of webhook-based custom
apps before Zulip has vetted the translation and added an official endpoint. Maybe we could set up some kind of
webserver that can run translation code outside of Zulip and externally post the messages, and we could think about
how to structure the code so that it is easy to eventually turn it into a Zulip-hosted integration.
Python scripts
In script integrations, the deployment model is usually this:
Custom app author’s hardware:
• detect event by polling a third party system
• translate event in the script
• externally post messages
These type of integrations are typically easy to prototype, but they can be harder to deploy in production settings, since
we rely on the authors to run their own scripts.
In some cases authors might want to at least move the translation/posting code to live on Zulip, by contributing that
code to Zulip as a server-side integration. Then, there would still be the challenge of detecting events in the third party
system, where maybe the user submits a patch to the third party as well.
Plugin integrations
In plugin integrations, the deployment model is usually this:
Third party system (driver):
• detect event
Third party system (plugin):
• further detect/triage event
• translate event
252
Chapter 45. Custom Apps
Zulip Documentation, Release 1.4.0
• externally post to Zulip
For third parties that have a plugin model, there are often other issues at play, like the plugins may need to be written
in a non-Python language like Ruby. There are probably still some scenarios, however, where a lot of the logic for
translation could be moved to a Zulip-side integration, and then we supply very thin client code for the plugin.
Zulip Reader
A Zulip Reader custom app gets stimuli from Zerver itself. Most Zulip Reader apps are packaged/advertised more
as what people commonly call “bots” than as “integrations.” (But sometimes what is currently a “bot” should really
be deployed more like an “integration” in an ideal Zulip universe.)
Example custom Zulip Reader apps can be serious or whimsical.
Serious
• A user tags a message with an alert word like @followup or @ticket.
• A user needs help computing something, like a simple math expression or a timezone conversion.
• A World Reader custom app posts something to a Zulip stream that we want to cross-post to another external
system.
• A user wants the custom app to query the outside world, like look up the weather or search Wikipedia.
• A bot collects RSVPs for an event.
• A bot conducts a user survey.
Whimsical
• A user wants to see a random quote of the day or a random cat fact.
• A user wants to tell the office telepresence robot to “turn left.”
Setting aside whether a custom app is performing a serious or whimsical function, there are a few different types of
Zulip Readers:
• Some readers will do simple local computations and post right back to Zulip.
• Some readers will do more expensive/web-related computations like searching Wikipedia, but then post right
back to Zulip.
• Some readers will mutate the outside world in some way, like posting messages to third party APIs or controlling
hardware.
• Some readers will do some combination of the prior bullets.
Deployment issues
Zulip currently provides only minimal deployment support for Zulip Reader custom apps:
• It ships with a few native server-side bots like the welcome bot and the notifications bot. (These are nice to have,
but they are so tightly integrated into the Zulip core that they don’t act as great examples for future app authors,
and they are not easy to extend/customize.)
• Zulip does ship an API client that can conveniently read a .zuliprc file, poll for incoming messages/events,
and post new messages to the Zulip server.
45.6. Zulip Reader
253
Zulip Documentation, Release 1.4.0
Local deployment
If you download the API client and write a bot that reads from Zulip, you face the following challenges if you deploy
your code on your own devices:
• It can be difficult to keep the app running 24/7.
• You may have latency issues connecting to the server.
• If you want super-user permissions, you have to secure the API key.
• Without integration to the Zulip server, the app may spin needlessly during upgrades.
• If you’ve written a personal-use bot, it can be difficult to distribute code to your friends and have them be able
to deploy it.
• If you’ve written a general-use bot, it may be difficult to persuade your admin to give you a superuser account.
We want to make it easier to deploy Zulip Readers on Zulip hardware. The following document talks about how we
want to enable this from a code structuring standpoint:
Writing contrib bots
This document, on the other hand, is more about designing the Zulip backend system to support eventual deployment
of reader apps on the Zulip server.
Before we talk about server-side apps, we should consider an intermediate solution.
Non-Zulip dedicated hardware
There are some scenarios, mostly with general-purpose “serious” custom apps, where an app author might use the
following development process:
• Create a prototype and deploy it locally.
• Publicize the app and deploy it on non-Zulip hardware.
• Contribute the app to the Zulip distribution, so that admins can run it Zulip-side.
To give a concrete example, let’s say that I work for a company that is building an issue tracker, and we want to offer
Zulip support. I would start by writing a Zulip Reader that scans for the alert word @ticket on certain public Zulip
streams, and part of that app would have logic to post to my company’s issue-tracking API.
Once I’m confident in my prototype, I will probably run it on dedicated company hardware that might already have
tight physical security, 24/7 IT monitoring, etc.
But what if I don’t have this kind of infrastructure available to me? Typically what I will do instead is rent time on
some kind of hosting service. Some hosting platforms are basically just remote Unix systems, but others are more
oriented toward hosting web apps.
Zulip’s current roadmap assumes that authors will likely gravitate toward web-based solutions (even if it’s just running
a web server on their own Unix host in the cloud).
Zulip intends to offer support for “outgoing webhooks.” The term “outgoing webhook” can be confusing, depending
on your perspective, but it simply means that an HTTP request is outgoing from Zulip, so that it will hit a web endpoint
that runs a third-party custom app.
Zulip will allow the custom app author, probably with the help of a Zulip admin, to configure Zulip to send a subset of
Zulip messages to the author’s web endpoint, and then the protocol for the custom app will to read the HTTP request
and send some kind of HTTP response that optionally results in a message being written to Zulip. Meanwhile, the
custom app can mutate the “world” as it sees fit.
254
Chapter 45. Custom Apps
Zulip Documentation, Release 1.4.0
Zulip-side support for reader apps
Even for app authors that have access to dedicated hardware, there would be several advantages to running Zulip
Readers under the same umbrella as the core Zulip system.
• Your app will automatically inherit the uptime of the Zulip server itself (in terms of hardware availability).
• There will be no network latency between the app and the server.
• Securing apps to have superuser permissions will be less problematic.
• Keeping your app in sync with Zulip upgrades could become more automatic.
• Allowing multiple users in your realm to run their own copies of personal-use bots would be easier to administer.
The only problem with the above bullets is that we haven’t built out any of that infrastructure yet.
We do have pending PR #1393, which addresses some of the issues that might come up.
In order to run apps inside the Zulip server, we basically need to solve the problems below. (One assumption is that
we don’t run apps truly in-process.)
• Contributions: We need a process for users to contribute code.
• Configuration/Discovery: We need Zulip to be able to find which apps are allowed to run for a particular
deployment. (The admin may choose to run only a subset of contributed apps.)
• Queuing: We need to queue up events for readers, with some possible optimizations to scan for alert words
during the in-process part of the call.
• Drivers: We need a generic driver that can pull events off of a queue and hand them off to our specific reader
objects.
• Nannying: We need to launch readers with some kind of supervisord-like nannying.
• Pausing: We probably need a way to pause/stop readers without stopping the Zulip main processes. (At first
this may just be part of solving the nanny problem.)
• Identity: We need to identify reader instances as specific Zulip users (non-owned bot, human-owned bot, or
human).
• Superusers: We may need some readers to have users with special privileges like being auto-subscribed to all
public streams.
• Read-only: We may need some readers at the other end of the spectrum to be highly locked down, e.g. enforce
that they truly only have read access to Zulip messages.
• UI: We will want to provide some UI features that give admins and/or regular users visibility into which serverside apps are running.
45.7. Deployment issues
255
Zulip Documentation, Release 1.4.0
256
Chapter 45. Custom Apps
CHAPTER
46
Unread counts and the pointer
When you’re using Zulip and you reload, or narrow to a stream, how does Zulip decide where to place you?
Conceptually, Zulip takes you to the place where you left off (e.g. the first unread message), not the most recent
messages, to facilitate reviewing all the discussions that happened while you were away from your computer. The
scroll position is then set to keep that message in view and away from both the top and bottom of the visible section
of messages.
But there a lot of details around doing this right, and around counting unread messages. Here’s how Zulip currently
decides which message to select, along with some notes on improvements we’d like to make to the model.
First a bit of terminology:
• “Narrowing” is the process of filtering to a particular subset of the messages the user has access to.
• The blue cursor box (the “pointer”) is around is called the “selected” message. Zulip ensures that the currently
selected message is always in-view.
Pointer logic
Recipient bar: message you clicked
If you enter a narrow by clicking on a message group’s recipient bar (stream/topic or private message recipient list at
the top of a group of messages), Zulip will select the message you clicked on. This provides a nice user experience
where you get to see the stuff near what you clicked on, and in fact the message you clicked on stays at exactly the
same scroll position in the window after the narrowing as it was at before.
Search or sidebar click: unread/recent matching narrow
If you instead narrow by clicking on something in the left sidebar or typing some terms into the search box, Zulip
will instead select the first unread message matching that narrow, or if there are none, the most recent messages
matching that narrow. This provides the nice user experience of taking you to the start of the new stuff (with enough
messages you’ev seen before still in view at the top to provide you with context), which is usually what you want.
257
Zulip Documentation, Release 1.4.0
(When finding the “first unread message”, Zulip ignores unread messages in muted streams or in muted topics within
non-muted streams.)
Unnarrow: previous sequence
When you unnarrow using e.g. the escape key, you will automatically be taken to the same message that was selected
in the home view before you narrowed, unless in the narrow you read new messages, in which case you will be jumped
forward to the first unread and non-muted message in the home view (or the bottom of the feed if there is none). This
makes for a nice experience reading threads via the home view in sequence.
New home view: “high watermark”
When you open a new browser window or tab to the home view (a.k.a. the interleaved view you get if you visit /),
Zulip will select the furthest down that your cursor has ever reached in the home view. Because of the logic around
unnarrowing in the last bullet, this is usually just before the first unread message in the home view, but if you never go
to the home view, or you leave messages unread on some streams in your home view, this can lag.
We plan to change this to automatically advance the pointer in a way similar to the unnarrow logic.
Narrow in a new tab: closest to pointer
When you load a new browser tab or window to a narrowed view, Zulip will select the message closest to your pointer,
which is what you would have got had you loaded the browser window to your home view and then clicked on the
nearest message matching your narrow (which might have been offscreen).
We plan to change this to match the Search/sidebar behavior.
Forced reload: state preservation
When the server forces a reload of a browser that’s otherwise caught up (which happens within 30 minutes when a
new version of the server is deployed, usually at a type when the user isn’t looking at the browser), Zulip will preserve
the state – what (if any) narrow the user was in, the selected message, and even exact scroll position!
For more on the user experience philosophy guiding these decisions, see the architectural overview.
Unread count logic
How does Zulip decide whether a message has been read by the user? The algorithm needs to correctly handle a range
of ways people might use the product. The algorithm is as follows:
• Any message which is selected or above a message which is selected is marked as read. So messages are marked
as read as you scroll down the keyboard when the pointer passes over them.
• If the whitespace at the very bottom of the feed is in view, all messages in view are marked as read.
These two simple rules, combined with the pointer logic above, end up matching user expectations well for whether
the product should treat them as having read a set of messages (or not).
258
Chapter 46. Unread counts and the pointer
Zulip Documentation, Release 1.4.0
Testing and development
In a Zulip development environment, you can use manage.py mark_all_messages_unread to set every
user’s pointer to 0 and all messages as unread, for convenience in testing unread count related logic.
It can be useful to combine this with manage.py populate_db -n 3000 (which rebuilds the database with
3000 initial messages) to ensure a large number of messages are present.
46.3. Testing and development
259
Zulip Documentation, Release 1.4.0
260
Chapter 46. Unread counts and the pointer
CHAPTER
47
Markdown implementation
Zulip has a special flavor of Markdown, currently called ‘bugdown’ after Zulip’s original name of “humbug”. End
users are using Bugdown within the client, not original Markdown.
Zulip has two implementations of Bugdown. The backend implementation at zerver/lib/bugdown/ is based on
Python-Markdown and is used to authoritatively render messages to HTML (and implements slow/expensive/complex
features like querying the Twitter API to render tweets nicely). The frontend implementation is in JavaScript, based
on marked.js (static/js/echo.js), and is used to preview and locally echo messages the moment the sender
hits enter, without waiting for round trip from the server. Those frontend renderings are only shown to the sender of a
message, and they are (ideally) identical to the backend rendering.
The JavaScript markdown implementation has a function, markdown.contains_bugdown, that is used to check
whether a message contains any syntax that needs to be rendered to HTML on the backend. If markdown.
contains_bugdown returns true, the frontend simply won’t echo the message for the sender until it receives the
rendered HTML from the backend. If there is a bug where markdown.contains_bugdown returns false incorrectly, the frontend will discover this when the backend returns the newly sent message, and will update the HTML
based on the authoritative backend rendering (which would cause a change in the rendering that is visible only to the
sender shortly after a message is sent). As a result, we try to make sure that markdown.contains_bugdown is
always correct.
Testing
The Python-Markdown implementation is tested by zerver/tests/test_bugdown.py, and the marked.js
implementation and markdown.contains_bugdown are tested by frontend_tests/node_tests/
markdown.js.
A shared set of fixed test data (“test fixtures”) is present in zerver/fixtures/
bugdown-data.json, and is automatically used by both test suites; as a result, it the preferred place to add new
tests for Zulip’s markdown system.
If you’re going to manually test some changes in the frontend Markdown implementation, the easiest way to do this is
as follows:
1. Login to your development server.
2. Stop your Zulip server with ctrl-C, leaving the browser open.
261
Zulip Documentation, Release 1.4.0
3. Compose and send the messages you’d like to test. They will be locally echoed using the frontend rendering.
This procedure prevents any server-side rendering. If you don’t do this, backend will likely render the Markdown
you’re testing and swap it in before you can see the frontend’s rendering.
Changing Zulip’s markdown processor
When changing Zulip’s markdown syntax, you need to update several places:
• The backend markdown processor (zerver/lib/bugdown/__init__.py).
• The frontend markdown processor (static/js/markdown.js and sometimes static/third/
marked/lib/marked.js), or markdown.contains_bugdown if your changes won’t be supported in
the frontend processor.
• If desired, the typeahead logic in static/js/composebox_typeahead.js.
• The test suite, probably via adding entries to zerver/fixtures/bugdown-data.json.
• The in-app markdown documentation (templates/zerver/markdown_help.html).
• The list of changes to markdown at the end of this document.
Important considerations for any changes are:
• Security: A bug in the markdown processor can lead to XSS issues. For example, we should not insert unsanitized HTML from a third-party web application into a Zulip message.
• Uniqueness: We want to avoid users having a bad experience due to accidentally triggering markdown syntax
or typeahead that isn’t related to what they are trying to express.
• Performance: Zulip can render a lot of messages very quickly, and we’d like to keep it that way. New regular
expressions similar to the ones already present are unlikely to be a problem, but we need to be thoughtful about
expensive computations or third-party API requests.
• Database: The backend markdown processor runs inside a Python thread (as part of how we implement timeouts
for third-party API queries), and for that reason we currently should avoid making database queries inside the
markdown processor. This is a technical implementation detail that could be changed with a few days of work,
but is important detail to know about until we do that work.
• Testing: Every new feature should have both positive and negative tests; they’re easy to write and give us the
flexibility to refactor frequently.
Zulip’s Markdown philosophy
Note that this discussion is based on a comparison with the original Markdown, not newer Markdown variants like
CommonMark.
Markdown is great for group chat for the same reason it’s been successful in products ranging from blogs to wikis to
bug trackers: it’s close enough to how people try to express themselves when writing plain text (e.g. emails) that it
helps more than getting in the way.
The main issue for using Markdown in instant messaging is that the Markdown standard syntax used in a lot of
wikis/blogs has nontrivial error rates, where the author needs to go back and edit the post to fix the formatting after
typing it the first time. While that’s basically fine when writing a blog, it gets annoying very fast in a chat product; even
though you can edit messages to fix formatting mistakes, you don’t want to be doing that often. There are basically 2
types of error rates that are important for a product like Zulip:
262
Chapter 47. Markdown implementation
Zulip Documentation, Release 1.4.0
• What fraction of the time, if you pasted a short technical email that you wrote to your team and passed it through
your Markdown implementation, would you need to change the text of your email for it to render in a reasonable
way? This is the “accidental Markdown syntax” problem, common with Markdown syntax like the italics syntax
interacting with talking about char *s.
• What fraction of the time do users attempting to use a particular Markdown syntax actually succeed at doing
so correctly? Syntax like required a blank line between text and the start of a bulleted list raise this figure
substantially.
Both of these are minor issues for most products using Markdown, but they are major problems in the instant messaging
context, because one can’t edit a message that has already been sent and users are generally writing quickly. Zulip’s
Markdown strategy is based on the principles of giving users the power they need to express complicated ideas in a
chat context while minimizing those two error rates.
Zulip’s Changes to Markdown
Below, we document the changes that Zulip has against stock Python-Markdown; some of the features we modify /
disable may already be non-standard.
Basic syntax
• Enable nl2br extension: this means one newline creates a line break (not paragraph break).
• Allow only * syntax for italics, not _. This resolves an issue where people were using _ and hitting it by mistake
too often. Asterisks surrounded by spaces won’t trigger italics, either (e.g. with stock Markdown You should
use char * instead of void * there would produce undesired results).
• Allow only ** syntax for bold, not __ (easy to hit by mistake if discussing Python __init__ or something).
• Add ~~ syntax for strikethrough.
• Disable special use of \ to escape other syntax. Rendering \\ as \ was hugely controversial, but having no
escape syntax is also controversial. We may revisit this. For now you can always put things in code blocks.
Lists
• Allow tacking a bulleted list or block quote onto the end of a paragraph, i.e. without a blank line before it.
• Allow only * for bulleted lists, not + or - (previously created confusion with diff-style text sloppily not included
in a code block).
• Disable ordered list syntax: stock Markdown automatically renumbers, which can be really confusing when
sending a numbered list across multiple messages.
Links
• Enable auto-linkification, both for http://... and guessing at things like t.co/foo.
• Force links to be absolute. [foo](google.com) will go to http://google.com, and not http://
zulip.com/google.com which is the default behavior.
• Set target="_blank" and title=(the url) on every link tag so clicking always opens a new window.
• Disable link-by-reference syntax, [foo][bar] ... [bar]:
http://google.com.
• Enable linking to other streams using #**streamName**.
47.4. Zulip’s Changes to Markdown
263
Zulip Documentation, Release 1.4.0
Code
• Enable fenced code block extension, with syntax highlighting.
• Disable line-numbering within fenced code blocks – the <table> output confused our web client code.
Other
• Disable headings, both # foo and == foo == syntax: they don’t make much sense for chat messages.
• Disabled images with ![]() (images from links are shown as an inline preview).
• Allow embedding any avatar as a tiny (list bullet size) image. This is used primarily by version control integrations.
• We added the ~~~ quote block quote syntax.
264
Chapter 47. Markdown implementation
CHAPTER
48
Realms in Zulip
Zulip allows multiple realms to be hosted on a single instance. Realms are the Zulip codebases’s internal name for
what we refer to in user documentation as an organization (the name “realm” comes from Kerberos).
Creating Realms
There are two main methods for creating realms.
• Using unique link generator
• Enabling open realm creation
Using Unique Link Generator
./manage.py generate_realm_creation_link
The above command will output a URL which can be used for creating a new realm and an administrator user for that
realm. The link expires after the creation of the realm. The link also expires if not used within 7 days. The expiration
period can be changed by modifying REALM_CREATION_LINK_VALIDITY_DAYS in settings.py.
Enabling Open Realm Creation
If you want anyone to be able to create new realms on your server, you can enable Open Realm Creation. This will
add a Create new organization link to your Zulip homepage footer, and anyone can create a new realm by visiting
this link (/create_realm). This feature is disabled by default in production instances, and can be enabled by setting
OPEN_REALM_CREATION = True in settings.py.
265
Zulip Documentation, Release 1.4.0
Subdomains
A reasonable way to deploy a multi-realm Zulip server in production is to give each realm a unique subdomain. For example if the Zulip instance is hosted at zulip.example.com and the subdomain of your organization
is acme you can use acme.zulip.example.com for accessing the organization. This subdomain feature is not enabled by default, since it requires additional DNS configuration. You can enable subdomains by setting the value
of REALMS_HAVE_SUBDOMAINS to True in settings.py. For subdomains to work properly, you also have to change
your DNS records so that the subdomains point to your Zulip installation IP. An A record with host name value *
pointing to your IP should do the job.
Converting a production Zulip server from not using subdomains to using subdomains requires some setup work;
contact the Zulip development community for help with this.
Working With Subdomains In Development Environment
By default, Linux does not provide a convenient way to use subdomains in your local development environment. To
solve this problem, we use the zulipdev.com domain, which has a wildcard A record pointing to 127.0.0.1. When
REALMS_HAVE_SUBDOMAINS = True in zproject/dev_settings.py, you should use zulipdev.com to
connect to your Zulip development server instead of localhost. The default realm with the Shakespeare users has the
subdomain zulip and can be accessed by visiting zulip.zulipdev.com.
If you are behind a proxy server, this method won’t work. When you make a request to load zulipdev.com in your
browser, the proxy server will try to get the page on your behalf. Since zulipdev.com points to 127.0.0.1 the proxy
server is likely to give you a 503 error. The workaround is to disable your proxy for *.zulipdev.com. The DNS
lookup should still work even if you disable proxy for *.zulipdev.com. If it doesn’t you can add zulipdev.com records
in /etc/hosts file. The file should look something like this.
127.0.0.1
localhost
127.0.0.1
zulipdev.com
127.0.0.1
zulip.zulipdev.com
127.0.0.1
testsubdomain.zulipdev.com
These records are also useful if you want to e.g. run the casper tests when you are not connected to the Internet.
266
Chapter 48. Realms in Zulip
CHAPTER
49
Static asset pipeline
This page documents additional information that may be useful when developing new features for Zulip that require
front-end changes, especially those that involve adding new files. For a more general overview, see the new feature
tutorial.
Primary build process
Most of the existing JS in Zulip is written in IIFE-wrapped modules, one per file in the static/js directory. We
will over time migrate this to Typescript modules. In development mode files are loaded using webpack eval with
sourcemaps. In production mode (and when creating a release tarball using tools/build-release-tarball),
JavaScript files are concatenated and minified. We use the django pipeline extension to manage our static assets,
webpack, and typescript.
Adding static files
To add a static file to the app (JavaScript, CSS, images, etc), first add it to the appropriate place under static/.
• Third-party files that we haven’t patched should be installed via npm, so that it’s easy to upgrade them and
third-party code doesn’t bloat the Zulip repository. You can then access them in webpack.assets.json
via their paths under node_modules. You’ll want to add these to the package.json in the root of the
repository, and then provision (to have npm download them) before continuing. Your commit should also
update PROVISION_VERSION in version.py. When adding modules to package.json, please pin
specific versions of them (don’t using carets ^, tildes ~, etc). We prefer fixed versions so that when the upstream providers release new versions with incompatible APIs, it can’t break Zulip. We update those versions
periodically to ensure we’re running a recent version of third-party libraries.
• Third-party files that we have patched should all go in static/third/. Tag the commit with “[third]” when
adding or modifying a third-party package. Our goal is to the extent possible to eliminate patched third-party
code from the project.
267
Zulip Documentation, Release 1.4.0
– Our own JavaScript lives under static/js; Typescript files live under static/ts; CSS lives under
static/styles. Portico JavaScript (“portico” means for logged-out pages) lives under static/
js/portico.
After you add a new JavaScript file, it needs to be specified in the entries dictionary defined in tools/webpack.
assets.json to be included in the concatenated file; this will magically ensure it is available both in development
and production. CSS should be added to the STYLESHEETS section of PIPELINE in zproject/settings.py.
A few notes on doing this:
• If you plan to only use the JS/CSS within the app proper, and not on the login page or other standalone pages,
put it in the app bundle.
• If you plan to use it in both, put it in the common bundle.
• If it’s just used on a single standalone page (e.g. /stats), give it its own bundle. To load a bundle in the
relevant Jinja2 template for that page, use render_bundle and stylesheet for JS and CSS, respectively.
If you want to test minified files in development, look for the PIPELINE_ENABLED = line in zproject/
settings.py and set it to True – or just set DEBUG = False.
Note that static/html/{400,5xx}.html will only render properly if minification is enabled, since they, by
nature, hardcode the path static/min/portico.css.
How it works in production
You can learn a lot from reading about django-pipeline, but a few useful notes are:
• Zulip installs static assets in production in /home/zulip/prod-static. When a new version is deployed,
before the server is restarted, files are copied into that directory.
• We use the VFL (Versioned File Layout) strategy, where each file in the codebase (e.g. favicon.ico) gets
a new name (e.g. favicon.c55d45ae8c58.ico) that contains a hash in it. Each deployment, has a
manifest file (e.g. /home/zulip/deployments/current/staticfiles.json) that maps codebase
filenames to serving filenames for that deployment. The benefit of this VFL approach is that all the static files
for past deployments can coexist, which in turn eliminates most classes of race condition bugs where browser
windows opened just before a deployment can’t find their static assets. It also is necessary for any incremental
rollout strategy where different clients get different versions of the site.
• Some paths for files (e.g. emoji) are stored in the rendered_content of past messages, and thus cannot be
removed without breaking the rendering of old messages (or doing a mass-rerender of old messages).
Webpack/CommonJS/ES6/Typescript modules
New JS written for Zulip can be written as Typescript or if a more incremental migration is required, CommonJS
modules (bundled using webpack, though this will be taken care of automatically whenever run-dev.py is running).
(CommonJS is the same module format that Node uses, so see the Node documentation for more information on the
syntax.)
All JavaScript we provide will eventually be migrated to Typescript, which will make refactoring the frontend code
easier and allow static analyzers to reason about our code more easily.
Declare entry points in webpack.assets.json. Any modules you add will need to be required or imported from this file
(or one of its dependencies) in order to be included in the script bundle.
268
Chapter 49. Static asset pipeline
CHAPTER
50
Schema Migrations
Zulip uses the standard Django system for doing schema migrations. There is some example usage in the new feature
tutorial.
This page documents some important issues related to writing schema migrations.
• Large tables: For large tables like Message and UserMessage, you want to take precautions when adding
columns to the table, performing data backfills, or building indexes. We have a zerver/lib/migrate.py
library to help with adding columns and backfilling data. For building indexes on these tables, we should do this
using SQL with postgres’s CONCURRENTLY keyword.
• Numbering conflicts across branches: If you’ve done your schema change in a branch, and meanwhile another
schema change has taken place, Django will now have two migrations with the same number. To fix this,
you can either run ./tools/renumber-migrations which renumbers your migration(s) and fixes up
the “dependencies” entries in your migration(s), and then rewrite your git history as needed, or you can do it
manually. There is a tutorial here that walks you though that process.
• Atomicity. By default, each Django migration is run atomically inside a transaction. This can be problematic
if one wants to do something in a migration that touches a lot of data and would best be done in batches of
e.g. 1000 objects (e.g. a Message or UserMessage table change). There is a new Django feature added
in Django 1.10 that makes it possible to add atomic=False at the top of a Migration class and thus
not have the entire migration in a transaction. This should make it possible to use the batch update tools in
zerver/lib/migrate.py (originally written to work with South) for doing larger database migrations.
• Accessing code and models in RunPython migrations. When writing a migration that includes custom python
code (aka RunPython), you almost never want to import code from zerver or anywhere else in the codebase.
If you imagine the process of upgrading a Zulip server, it goes as follows: first a server admin checks out a recent
version of the code, and then runs any migrations that were added between the last time they upgraded and the
current check out. Note that for each migration, this means the migration is run using the code in the server
admin’s check out, and not the code that was there at the time the migration was written. This can be a difference
of thousands of commits for installations that are only upgraded occasionally. It is hard to reason about the effect
of a code change on a migration that imported it so long ago, so we recommend just copying any code you’re
tempted to import into the migration file directly, and have a linter rule enforcing this.
There is one special case where this doesn’t work: you can’t copy the definition of a model (like Realm) into a
migration, and you can’t import it from zerver.models for the reasons above. In this situation you should
269
Zulip Documentation, Release 1.4.0
use Django’s apps.get_model to get access to a model as it is at the time of a migration. Note that this
will work for doing something like Realm.objects.filter(..), but shouldn’t be used for accessing
Realm.subdomain or anything not related to the Django ORM.
• Making large migrations work. Major migrations should have a few properties:
– Unit tests. You’ll want to carefully test these, so you might as well write some unit tests to verify the
migration works correctly, rather than doing everything by hand. This often saves a lot of time in retesting the migration process as we make adjustments to the plan.
– Run in batches. Updating more than 1K-10K rows (depending on type) in a single transaction can lock
up a database. It’s best to do lots of small batches, potentially with a brief sleep in between, so that we
don’t block other operations from finishing.
– Rerunnability/idempotency. Good migrations are ones where if operational concerns (e.g. it taking down
the Zulip server for users) interfere with it finishing, it’s easy to restart the migration without doing a bunch
of hand investigation. Ideally, the migration can even continue where it left off, without needing to redo
work.
– Multi-step migrations. For really big migrations, one wants to split the transition into into several commits that are each individually correct, and can each be deployed independently:
1. First, do a migration to add the new column to the Message table and start writing to that column (but
don’t use it for anything)
2. Second, do a migration to copy values from the old column to the new column, to ensure that the two
data stores agree.
3. Third, a commit that stops writing to the old field.
4. Any cleanup work, e.g. if the old field were a column, we’d do a migration to remove it entirely here.
This multi-step process is how most migrations on large database tables are done in large-scale systems,
since it ensures that the system can continue running happily during the migration.
270
Chapter 50. Schema Migrations
CHAPTER
51
HTML and CSS
Zulip CSS organization
The Zulip application’s CSS can be found in the static/styles/ directory. Zulip uses Bootstrap as its main
third-party CSS library.
Zulip currently does not use any CSS preprocessors, and is organized into several files. For most pages, the CSS is
combined into a single CSS file by the static asset pipeline, controlled by the PIPELINE_CSS code in zproject/
settings.py.
The CSS files are:
• portico.css - Main CSS for logged-out pages
• pygments.css - CSS for Python syntax highlighting
• activity.css - CSS for the activity app
• fonts.css - Fonts for text in the Zulip app
• static/third/thirdparty-fonts.css - Font Awesome (used for icons)
The CSS for the Zulip web application UI is primarily here:
• settings.css - CSS for the Zulip settings (including organization settings) pages
• zulip.css - CSS for the rest of the Zulip logged-in app
• media.css - CSS for media queries (particularly related to screen width)
We are in the process of splitting zulip.css into several more files; help with that project is very welcome!
Editing Zulip CSS
If you aren’t experienced with doing web development and want to make CSS changes, we recommend reading the
excellent Chrome web inspector guide on editing HTML/CSS, especially the section on CSS to learn about all the great
271
Zulip Documentation, Release 1.4.0
tools that you can use to modify and test changes to CSS interactively in-browser (without even having the reload the
page!).
CSS Style guidelines
Avoid duplicated code
Without care, it’s easy for a web application to end up with thousands of lines of duplicated CSS code, which can
make it very difficult to understand the current styling or modify it. We would very much like to avoid such a fate. So
please make an effort to reuse existing styling, clean up now-unused CSS, etc., to keep things maintainable.
Be consistent with existing similar UI
Ideally, do this by reusing existing CSS declarations, so that any improvements we make to the styling can improve
all similar UI elements.
Use clear, unique names for classes and object IDs
This makes it much easier to read the code and use git grep to find where a particular class is used.
Validating CSS
When changing any part of the Zulip CSS, it’s important to check that the new CSS looks good at a wide range of
screen widths, from very wide screen (e.g. 1920px) all the way down to narrow phone screens (e.g. 480px).
For complex changes, it’s definitely worth testing in a few different browsers to make sure things look the same.
272
Chapter 51. HTML and CSS
CHAPTER
52
URL hashes and deep linking
Hashchange
The Zulip web application has a nice system of hash (#) URLs that can be used to deep-link into the application and
allow the browser’s “back” functionality to let the user navigate between parts of the UI. Some examples are:
• /#settings/your-bots: Bots section of the settings overlay.
• /#streams: Streams overlay, where the user manages streams (subscription etc.)
• /#streams/11/announce: Streams overlay with stream ID 11 (called “announce”) selected.
• /#narrow/stream/android/subject/fun: Message feed showing stream “android” and topic “fun”.
The main module in the frontend that manages this all is static/js/hashchange.js (plus hash_util.js
for all the parsing code), which is unfortunately one of our thorniest modules. Part of the reason that it’s thorny is that
it needs to support a lot of different flows:
• The user clicking on an in-app link, which in turn opens an overlay. For example the streams overlay opens
when the user clicks the small cog symbol on the left sidebar, which is in fact a link to /#streams. This
makes it easy to have simple links around the app without custom click handlers for each one.
• The user uses the “back” button in their browser (basically equivalent to the previous one, as a link out of the
browser history will be visited).
• The user clicking some in-app click handler (e.g. “Stream settings” for an invidual stream), that potentially does
several UI-manipulating things including e.g. loading the streams overlay, and needs to update the hash without
re-triggering the open animation (etc.).
• Within an overlay like the streams overlay, the user clicks to another part of the overlay, which should update
the hash but not re-trigger loading the overlay (which would result in a confusing animation experience).
• The user is in a part of the webapp, and reloads their browser window. Ideally the reloaded browser window
should return them to their original state.
• A server-initiated browser reload (done after a new version is deployed, or when a user comes back after being
idle for a while, see notes below), where we try to preserve extra state (e.g. content of compose box, scroll
position within a narrow) using the /#reload hash prefix.
273
Zulip Documentation, Release 1.4.0
When making changes to the hashchange system, it is essential to test all of these flows, since we don’t have great
automated tests for all of this (would be a good project to add them to the Casper suite) and there’s enough complexity
that it’s easy to accidentally break something.
Here’s some notes on how we handle these cases:
• hashchange.hashchanged is the function used to handle the hash, when it’s changed by the browser (e.g.
by clicking on a link to a hash or using the back button).
• hashchange.should_ignore is the function hashchange.hashchanged calls to make it possible
for clicking on links within a given overlay to just be managed by code within that overlay, without reloading
the overlay. It primarily checks whether the “main hash” (i.e. the first piece like settings for #settings/
your-account) is an overlay.
• hashchange.do_hashchange is what is called when the user reloads the browser. If the hash is nonempty,
it ensures the relevant overlay is opened or the user is narrowed as part of the page load process. It is also is
called by hashchange.hashchanged when the hash changes outside the should_ignore boundaries,
since the logic for that case is identical.
• reload.preserve_state is called when a server-initiated browser reload happens, and encodes a bunch
of data like the current scroll position into the hash.
• reload.initialize handles restoring the preserved state after a reload where the hash starts with /
#reload.
Server-initiated reloads
There are a few circumstances when the Zulip browser window needs to reload itself:
• If the browser has been offline for more than 10 minutes, the browser’s event queue will have been garbagecollected by the server, meaning the browser can no longer get real-time updates altogether. In this case, the
browser auto-reloads immediately in order to reconnect. We have coded an unsuspend trigger (based on some
clever time logic) that ensures we check immediately when a client unsuspends; grep for unsuspend to see
the code.
• If a new version of the server has been deployed, we want to reload the browser so that it will start running
the latest code. However, we don’t want server deploys to be disruptive. So, the backend preserves user-side
event queues (etc.) and just pushes a special restart event to all clients. That event causes the browser to
start looking for a good time to reload, based on when the user is idle (ideally, we’d reload when they’re not
looking and restore state so that the user never knew it happened!). The logic for doing this is in static/js/
reload.js; but regardless we’ll reload within 30 minutes unconditionally.
An important detail in server-initiated reloads is that we desynchronize when browsers start attempting them
randomly, in order to avoid a thundering herd situation bringing down the server.
All reloads
In addition to saving state as described above when reloading the browser, Zulip also does a few bookkeeping things
on page reload (like cleaning up its event queue, and saving any text in an open compose box as a draft).
274
Chapter 52. URL hashes and deep linking
CHAPTER
53
Emoji
Emoji seem like a simple idea, but there’s actually a ton of complexity that goes into an effective emoji implementation.
This document discusses a number of these issues.
Currently, Zulip uses the Noto (Android) emoji set. We are considering adding additional emoji sets as options.
Emoji codes
The Unicode standard has various ranges of characters set aside for emoji. So you can put emoji in your terminal using
actual unicode characters like and . If you paste those into Zulip, Zulip will render them as the corresponding emoji
image.
However, the Unicode committee did not standardize on a set of human-readable names for emoji. So, for example, when using the popular : based style for entering emoji from the keyboard, we have to decide whether to use
:angry: or :angry_face: to represent an angry face. Different products use different approaches, but for purposes like emoji pickers or autocomplete, you definitely want to pick exactly one of these names, since otherwise users
will always be seeing duplicates of a given emoji next to each other.
Picking which emoji name to use is surprisingly complicated! Zulip has a nice library, tools/setup/emoji/
emoji_setup_utils.py, which we use to make sense of these decisions with a relatively small list of hand-coded
exceptions.
Tooling
Zulip has a tool, tools/setup/emoji/build_emoji, that combines emoji-map.json (an open source
mapping of emoji names to short codes that is fairly liberal about including duplicates) and the Noto emoji font to
extract the emoji that we use in the product.
This tool generates a set of files under static/generated/emoji (or really, it generates the /srv/
zulip-emoji-cache/<sha1>/emoji tree, and static/generated/emoji is a symlink to that tree; we
do this in order to cache old versions to make provisioning and production deployments super fast in the common case
that we haven’t changed the emoji tooling).
275
Zulip Documentation, Release 1.4.0
The emoji tree generated by this process contains several import elements:
• emoji_codes.js: A set of mappings used by the Zulip frontend to understand what unicode emoji exist
and what their shortnames are, used for autocomplete, emoji pickers, etc. This has been deduplicated using the
logic in tools/setup/emoji/emoji_setup_utils.py to generally only have :angry: and not also
:angry_face:, since having both is ugly and pointless for purposes like autocomplete and emoji pickers.
• images/emoji/unicode/*.png: A farm of emoji
• images/emoji/*.png: A farm of symlinks from emoji names to the images/emoji/unicode/ tree.
This is used to serve individual emoji images, as well as for the backend markdown processor to know which
emoji names exist and what unicode emoji / images they map to. In this tree, we currently include all of the
emoji in emoji-map.json; this means that if you send :angry_face:, it won’t autocomplete, but will
still work (but not in previews).
• Some CSS and a PNG for an emoji spritesheet, used in Zulip for emoji pickers where we would otherwise need
to download over 1000 of individual emoji images (which would cause a browser performance problem). We
will likely eventually replace the images/emoji/unicode/ tree with using this spritesheet as well.
276
Chapter 53. Emoji
CHAPTER
54
Full-text search
Zulip supports full-text search, which can be combined arbitrarily with Zulip’s full suite of narrowing operators. By
default, it only supports English text, but there is an experimental PGroonga integration that provides full-text search
for all languages.
The user interface and feature set for Zulip’s full-text search is documented in the “Search operators” documentation
section in the Zulip app’s gear menu.
The default full-text search implementation
Zulip’s uses PostgreSQL’s built-in full-text search feature, with a custom set of English stop words to improve the
quality of the search results.
We use a small extension, tsearch_extras, for highlighting of the matching words. There is some discussion of removing this extension, at least as an option, so that Zulip can be used with database-as-a-service platforms.
In order to optimize the performance of delivering messages, the full-text search index is updated for newly sent
messages in the background, after the message has been delivered. This background updating is done by puppet/
zulip/files/postgresql/process_fts_updates, which is usually deployed on the database server, but
could be deployed on an application server instead.
An optional full-text search implementation
Zulip now supports using PGroonga for full-text search. PGroonga is a PostgreSQL extension that provides full-text
search feature. PostgreSQL’s built-in full-text search feature supports only one language at a time (in Zulip’s case,
English). PGroonga supports all languages simultaneously, including Japanese, Chinese and so on, all at once. We
expect to migrate Zulip’s full-text search to only support PGroonga once we have tested this new extension fully.
The following processes should be executed as the root user. Run:
sudo -i
277
Zulip Documentation, Release 1.4.0
How to enable full-text search across all languages
This section describes how to enable using PGroonga to back the full-text search feature.
To install PGroonga, add pgroonga = enabled in the [machine] section in /etc/zulip/zulip.conf:
[machine]
...
pgroonga = enabled
And then run as root:
/home/zulip/deployments/current/scripts/zulip-puppet-apply
Then, add USING_PGROONGA = true in /etc/zulip/settings.py:
USING_PGROONGA = True
And apply the PGroonga migrations:
cd /srv/zulip
./manage.py migrate pgroonga
Note that the migration may take a long time, and you can’t send new messages until the migration finishes.
Once the migrations are complete, restart Zulip:
su zulip -c /home/zulip/deployments/current/scripts/restart-server
Now, you can use full-text search across all languages.
How to disable full-text search across all languages
This section describes how to disable full-text search feature based on PGroonga.
If you want to fully remove PGroonga, first you need to remove the PGroonga column (as above, this will take a long
time and no messages can be sent while it is running). If you intend to re-enable PGroonga later, you can skip this step
(at the cost of your Message table being slightly larger than it would be otherwise).
/home/zulip/deployments/current/manage.py migrate pgroonga zero
Then, set USING_PGROONGA = False in /etc/zulip/settings.py:
USING_PGROONGA = False
And, restart Zulip:
su zulip -c /home/zulip/deployments/current/scripts/restart-server
Now, full-text search feature based on PGroonga is disabled. If you’d like, you can also remove the pgroonga =
enabled line in /etc/zulip/zulip.conf and uninstall the pgroonga packages.
278
Chapter 54. Full-text search
CHAPTER
55
Analytics
Zulip has a cool analytics system for tracking various useful statistics that currently power the /stats page, and
over time will power other features, like showing usage statistics for the various streams. It is designed around the
following goals:
• Minimal impact on scalability and service complexity.
• Well-tested so that we can count on the results being correct.
• Efficient to query so that we can display data in-app (e.g. on the streams page) with minimum impact on the
overall performance of those pages.
• Storage size smaller than the size of the main Message/UserMessage database tables, so that we can store the
data in the main postgres database rather than using a specialized database platform.
There are a few important things you need to understand in order to effectively modify the system.
Analytics backend overview
There are three main components:
• models: The UserCount, StreamCount, RealmCount, and InstallationCount tables (analytics/models.py) collect
and store time series data.
• stat definitions: The CountStat objects in the COUNT_STATS dictionary (analytics/lib/counts.py) define the set
of stats Zulip collects.
• accounting: The FillState table (analytics/models.py) keeps track of what has been collected for which
CountStats.
The next several sections will dive into the details of these components.
279
Zulip Documentation, Release 1.4.0
The *Count database tables
The Zulip analytics system is built around collecting time series data in a set of database tables. Each of these tables
has the following fields:
• property: A human readable string uniquely identifying a CountStat object.
tive_users:is_bot:hour” or “messages_sent:client:day”.
Example:
“ac-
• subgroup: Almost all CountStats are further sliced by subgroup. For “active_users:is_bot:day”, this column will
be False for measurements of humans, and True for measurements of bots. For “messages_sent:client:day”, this
column is the client_id of the client under consideration.
• end_time: A datetime indicating the end of a time interval. It will be on an hour (or UTC day) boundary for
stats collected at hourly (or daily) frequency. The time interval is determined by the CountStat.
• various “id” fields: Foreign keys into Realm, UserProfile, Stream, or nothing. E.g. the RealmCount table has a
foreign key into Realm.
• value: The integer counts. For “active_users:is_bot:hour” in the RealmCount table, this is the number of
active humans or bots (depending on subgroup) in a particular realm at a particular end_time. For “messages_sent:client:day” in the UserCount table, this is the number of messages sent by a particular user, from
a particular client, on the day ending at end_time.
• anomaly: Currently unused, but a key into the Anomaly table allowing someone to indicate a data irregularity.
There are four tables: UserCount, StreamCount, RealmCount, and InstallationCount. Every CountStat is initially
collected into UserCount, StreamCount, or RealmCount. Every stat in UserCount and StreamCount is aggregated
into RealmCount, and then all stats are aggregated from RealmCount into InstallationCount. So for example, “messages_sent:client:day” has rows in UserCount corresponding to (user, end_time, client) triples. These are summed
to rows in RealmCount corresponding to triples of (realm, end_time, client). And then these are summed to rows in
InstallationCount with totals for pairs of (end_time, client).
Note: In most cases, we do not store rows with value 0. See Performance Strategy below.
CountStats
CountStats declare what analytics data should be generated and stored. The CountStat class definition and instances
live in analytics/lib/counts.py. These declarations specify at a high level which tables should be populated
by the system and with what data.
The FillState table
The default Zulip production configuration runs a cron job once an hour that updates the *Count tables for each of
the CountStats in the COUNT_STATS dictionary. The FillState table simply keeps track of the last end_time that we
successfully updated each stat. It also enables the analytics system to recover from errors (by retrying) and to monitor
that the cron job is running and running to completion.
Performance strategy
An important consideration with any analytics system is performance, since it’s easy to end up processing a huge
amount of data inefficiently and needing a system like Hadoop to manage it. For the built-in analytics in Zulip, we’ve
280
Chapter 55. Analytics
Zulip Documentation, Release 1.4.0
designed something lightweight and fast that can be available on any Zulip server without any extra dependencies
through the carefully designed set of tables in Postgres.
This requires some care to avoid making the analytics tables larger than the rest of the Zulip database or adding a ton
of computational load, but with careful design, we can make the analytics system very low cost to operate. Also, note
that a Zulip application database has 2 huge tables: Message and UserMessage, and everything else is small and thus
not performance or space-sensitive, so it’s important to optimize how many expensive queries we do against those 2
tables.
There are a few important principles that we use to make the system efficient:
• Not repeating work to keep things up to date (via FillState)
• Storing data in the *Count tables to avoid our endpoints hitting the core Message/UserMessage tables is key,
because some queries could take minutes to calculate. This allows any expensive operations to run offline, and
then the endpoints to server data to users can be fast.
• Doing expensive operations inside the database, rather than fetching data to Python and then sending it back
to the database (which can be far slower if there’s a lot of data involved). The Django ORM currently doesn’t
support the “insert into .. select” type SQL query that’s needed for this, which is why we use raw database
queries (which we usually avoid in Zulip) rather than the ORM.
• Aggregating where possible to avoid unnecessary queries against the Message and UserMessage tables. E.g.
rather than querying the Message table both to generate sent message counts for each realm and again for each
user, we just query for each user, and then add up the numbers for the users to get the totals for the realm.
• Not storing rows when the value is 0. An hourly user stat would otherwise collect 24 * 365 * roughly .5MB
per db row = 4GB of data per user per year, most of whose values are 0. A related note is to be cautious about
adding queries that are typically non-0 instead of being typically 0.
Backend Testing
There are a few types of automated tests that are important for this sort of system:
• Most important: Tests for the code path that actually populates data into the analytics tables. These are most
important, because it can be very expensive to fix bugs in the logic that generates these tables (one basically
needs to regenerate all of history for those tables), and these bugs are hard to discover. It’s worth taking the time
to think about interesting corner cases and add them to the test suite.
• Tests for the backend views code logic for extracting data from the database and serving it to clients.
For manual backend testing, it sometimes can be valuable to use ./manage.py dbshell to inspect the tables
manually to check that things look right; but usually anything you feel the need to check manually, you should add
some sort of assertion for to the backend analytics tests, to make sure it stays that way as we refactor.
LoggingCountStats
The system discussed above is designed primarily around the technical problem of showing useful analytics about
things where the raw data is already stored in the database (e.g. Message, UserMessage). This is great because we can
always backfill that data to the beginning of time, but of course sometimes one wants to do analytics on things that
aren’t worth storing every data point for (e.g. activity data, request performance statistics, etc.). There is currently a
reference implementation of a “LoggingCountStat” that shows how to handle such a situation.
55.6. Backend Testing
281
Zulip Documentation, Release 1.4.0
Analytics UI development and testing
Setup and Testing
The main testing approach for the /stats page UI is manual testing. For UI testing, you want a comprehensive initial
data set; you can use manage.py populate_analytics_db to set up, login as the shylock user, and then go
to /stats.
Adding or editing /stats graphs
The relevant files are:
• analytics/views.py: All chart data requests from the /stats page call get_chart_data in this file. The bottom half
of this file (with all the raw sql queries) is for a different page (/activity), not related to /stats.
• static/js/stats/stats.js: The JavaScript and Plotly code.
• templates/analytics/stats.html
• static/styles/stats.css and static/styles/portico.css: We are in the process of re-styling this page to use in-app css
instead of portico css, but there is currently still a lot of portico influence.
• analytics/urls.py: Has the URL routes; it’s unlikely you will have to modify this, including for adding a new
graph.
Most of the code is self-explanatory, and for adding say a new graph, the answer to most questions is to copy what
the other graphs do. It is easy when writing this sort of code to have a lot of semi-repeated code blocks (especially in
stats.js); it’s good to do what you can to reduce this.
Tips and tricks:
• Use $.get to fetch data from the backend. You can grep through stats.js to find examples of this.
• The Plotly documentation is at https://plot.ly/javascript/ (check out the full reference, event reference, and function reference). The documentation pages seem to work better in Chrome than in Firefox, though this hasn’t
been extensively verified.
• Unless a graph has a ton of data, it is typically better to just redraw it when something changes (e.g. in the various
aggregation click handlers) rather than to use retrace or relayout or do other complicated things. Performance
on the /stats page is nice but not critical, and we’ve run into a lot of small bugs when trying to use Plotly’s
retrace/relayout.
• There is a way to access raw d3 functionality through Plotly, though it isn’t documented well.
• ‘paper’ as a Plotly option refers to the bounding box of the graph (or something related to that).
• You can’t right click and inspect the elements of a Plotly graph (e.g. the bars in a bar graph) in your browser,
since there is an interaction layer on top of it. But if you hunt around the document tree you should be able to
find it.
/activity page
• There’s a somewhat less developed /activity page, for server administrators, showing data on all the realms
on a server. To access it, you need to have the is_staff bit set on your UserProfile object. You can set it
using manage.py shell and editing the UserProfile object directly. A great future project is to clean up that
page’s data sources, and make this a documented interface.
282
Chapter 55. Analytics
CHAPTER
56
Translating Zulip
To make Zulip even better for users around the world, the Zulip UI is being translated into a number of major languages,
including Spanish, German, French, Chinese, Russian, and Japanese, with varying levels of progress. If you speak a
language other than English, your help with translating Zulip would be greatly appreciated!
If you’re interested in contributing translations to Zulip, please join #translation in the Zulip development community
server, and say hello. And please join the Zulip project on Transifex and ask to join any languages you’d like to
contribute to (or add new ones). Transifex’s notification system sometimes fails to notify the maintainers when you
ask to join a project, so please send a quick email to zulip-core@googlegroups.com when you request to join the
project or add a language so that we can be sure to accept your request to contribute.
Zulip has full support for Unicode, so you can already use your preferred language everywhere in Zulip.
Translation style guides
We are building a collection of translation style guides for Zulip, giving guidance on how Zulip should be translated
into specific languages (e.g. what word to translate words like “home” to):
• Chinese
• German
• Polish
• Russian
• Spanish
A great first step when getting started translating Zulip into a new language is to write a style guide, since it greatly
increases the ability of future translators to translate in a way that’s consistent with what your work.
Capitalization
We expect that all the English translatable strings in Zulip are properly capitalized in a way consistent with how Zulip
does capitalization in general. This means that:
283
Zulip Documentation, Release 1.4.0
• The first letter of a sentence or phrase should be capitalized.
– Correct: “Manage streams”
– Incorrect: “Manage Streams”
• All proper nouns should be capitalized.
– Correct: “This is Zulip”
– Incorrect: “This is zulip”
• All common words like URL, HTTP, etc. should be written in their standard forms.
– Correct: “URL”
– Incorrect: “Url”
We have a tool to check for the correct capitalization of the translatable strings; this tool will not allow the
Travis builds to pass in case of errors. You can use our capitalization checker to validate your code by running
./tool/check-capitalization. If you think that you have a case where our capitalization checker tool
wrongly categorizes a string as not capitalized, you can add an exception in the tools.lib.capitalization.
IGNORED_PHRASES list to make the tool pass.
Please, stick to these while translating, and feel free to point out any strings that should be improved or fixed.
Translation process
The end-to-end process to get the translations working is as follows.
Please note that you don’t need to do this if you’re translating; this is only to describe how the whole process is. If
you’re interested in translating, you should check out the translators’ workflow.
1. The strings are marked for translation (see sections for backend and frontend translations for details on this).
2. Translation resource files are created using the ./manage.py makemessages command. This command
will create, for each language, a resource file called translations.json for the frontend strings and
django.po for the backend strings.
The makemessages command is idempotent in that:
• It will only delete singular keys in the resource file when they are no longer used in Zulip code.
• It will only delete plural keys (see below for the documentation on plural translations) when the corresponding singular key is absent.
• It will not override the value of a singular key if that value contains a translated text.
3. Those resource files are uploaded to Transifex by a maintainer using the tx push -s -a command.
4. Translators translate the strings in Transifex.
5. The translations are downloaded back into the codebase by a maintainer, using tx pull -a -f
--mode=developer (since we don’t use the Transifex review flow).
Translators’ workflow
These are the steps you should follow if you want to help to translate Zulip:
1. Join us on Zulip and ask for access to the organization, as described at the beginning.
2. Make sure you have access to Zulip’s dashboard in Transifex.
284
Chapter 56. Translating Zulip
Zulip Documentation, Release 1.4.0
3. Ask a maintainer to update the strings.
4. Translate the strings for your language in Transifex.
Some useful tips for your translating journey:
• Follow your language’s translation guide. Keeping it open in a tab while translating is very handy. If one doesn’t
exist one, write one as you go; they’re easiest to write as you go along and will help any future translators a lot.
• Don’t translate variables or code (usually preceded by a %, or inside HTML tags <...>).
• When in doubt, ask for context in #translation in the Zulip development community server.
• If there are multiple possible translations for a term, search for it in the Concordance tool (the button with a
magnet in the top right corner).
It will show if anyone translated that term before, so we can achieve good consistency with all the translations,
no matter who makes them.
• Pay attention to capital letters and punctuation. Details make the difference!
• Take advantage of the hotkeys the Transifex Web Editor provides, such as Tab for saving and going to the next
string.
Testing translations
This section assumes you have a Zulip development environment setup.
First of all, download the updated resource files from Transifex using the tx pull -a --mode=developer
command (it will require some initial setup). This command will download the resource files from Transifex and
replace your local resource files with them.
Then, make sure that you have compiled the translation strings using ./manage.py compilemessages.
Django figures out the effective language by going through the following steps:
1. It looks for the language code in the url (e.g. /de/).
2. It looks for the LANGUAGE_SESSION_KEY key in the current user’s session.
3. It looks for the cookie named ‘django_language’.
LANGUAGE_COOKIE_NAME setting.
You can set a different name through the
4. It looks for the Accept-Language HTTP header in the HTTP request. Normally your browser will take care
of this.
The easiest way to test translations is through the i18n URLs, e.g., if you have German translations available, you can
access the German version of a page by going to /de/path_to_page in your browser.
To test translations using other methods you will need an HTTP client library like requests, cURL or urllib.
Here is some sample code to test Accept-Language header using Python and requests:
import requests
headers = {"Accept-Language": "de"}
response = requests.get("http://localhost:9991/login/", headers=headers)
print(response.content)
56.4. Testing translations
285
Zulip Documentation, Release 1.4.0
Setting the default language in Zulip
Zulip allows you to set the default language through the settings page, in the ‘Display settings’ section. The URL will
be /#settings/display-settings on your realm.
Organizations can set the default language for new users in their organization on the /#organization page.
Translation resource files
All the translation magic happens through resource files which hold the translated text. Backend resource files are
located at static/locale/<lang_code>/LC_MESSAGES/django.po, while frontend resource files are located at static/locale/<lang_code>/translations.json.
These files are uploaded to Transifex, where they can be translated.
Backend translations
All user-facing text in the Zulip UI should be generated by an HTML template so that it can be translated.
Zulip uses two types of templates: backend templates (powered by the Jinja2 template engine, though the original
Django template engine is still supported) and frontend templates (powered by Handlebars).
To mark a string for translation in the Jinja2 and Django template engines, you can use the _() function in the
templates like this:
{{ _("English text") }}
If a string contains both a literal string component and variables, you can use a block translation, which makes use
of placeholders to help translators to translate an entire sentence. To translate a block, Jinja2 uses the trans tag while
Django uses the blocktrans tag. So rather than writing something ugly and confusing for translators like this:
# Don't do this!
{{ _("This string will have") }} {{ value }} {{ _("inside") }}
You can instead use:
# Jinja2 style
{% trans %}This string will have {{ value }} inside.{% endtrans %}
# Django style
{% blocktrans %}This string will have {{ value }} inside.{% endblocktrans %}
Zulip expects all the error messages to be translatable as well. To ensure this, the error message passed to
json_error and JsonableError should always be a literal string enclosed by _() function, e.g.:
json_error(_('English Text'))
JsonableError(_('English Text'))
To ensure we always internationalize our JSON errors messages, the Zulip linter (tools/lint) checks for correct
usage.
286
Chapter 56. Translating Zulip
Zulip Documentation, Release 1.4.0
Frontend translations
Zulip uses the i18next library for frontend translations. There are two types of files in Zulip frontend which can hold
translatable strings: JavaScript code files and Handlebar templates. To mark a string translatable in JavaScript files,
pass it to the i18n.t function.
i18n.t('English Text', context);
i18n.t('English text with a __variable__', {'variable': 'Variable value'});
Note: In the second example above, instead of enclosing the variable with handlebars, {{ }}, we enclose it with
__ because we need to differentiate the variable from the Handlebar tags. The symbol which is used to enclose the
variables can be changed in /static/js/src/main.js.
i18next also supports plural translations. To support plurals make sure your resource file contains the related keys:
{
"en": {
"translation": {
"key": "item",
"key_plural": "items",
"keyWithCount": "__count__ item",
"keyWithCount_plural": "__count__ items"
}
}
}
With this resource you can show plurals like this:
i18n.t('key', {count: 0}); // output: 'items'
i18n.t('key', {count: 1}); // output: 'item'
i18n.t('key', {count: 5}); // output: 'items'
i18n.t('key', {count: 100}); // output: 'items'
i18n.t('keyWithCount', {count: 0}); // output: '0 items'
i18n.t('keyWithCount', {count: 1}); // output: '1 item'
i18n.t('keyWithCount', {count: 5}); // output: '5 items'
i18n.t('keyWithCount', {count: 100}); // output: '100 items'
For further reading on plurals, read the official documentation.
To mark the strings as translatable in the Handlebar templates, Zulip registers two Handlebar helpers. The syntax for
simple strings is:
{{t 'English Text' }}
The syntax for block strings or strings containing variables is:
{{#tr context}}
Block of English text.
{{/tr}}
var context = {'variable': 'variable value'};
{{#tr context}}
Block of English text with a __variable__.
{{/tr}}
The rules for plurals are same as for JavaScript files. You just have to declare the appropriate keys in the resource file
and then include the count in the context.
56.8. Frontend translations
287
Zulip Documentation, Release 1.4.0
Transifex config
The config file that maps the resources from Zulip to Transifex is located at .tx/config.
Transifex CLI setup
In order to be able to run tx pull (and tx push as well, if you’re a maintainer), you have to specify your Transifex
credentials in a config file, located at ~/.transifexrc.
You can find details on how to set it up here, but it should look similar to this (with your credentials):
[https://www.transifex.com]
username = user
token =
password = p@ssw0rd
hostname = https://www.transifex.com
This basically identifies you as a Transifex user, so you can access your organizations from the command line.
288
Chapter 56. Translating Zulip
CHAPTER
57
Clients in Zulip
zerver.models.Client is Zulip’s analogue of the HTTP User-Agent header (and is populated from UserAgent). It exists for use in analytics and other places to provide human-readable summary data about “which Zulip
client” was used for an operation (e.g. was it the Android app, the desktop app, or a bot?).
In general, it shouldn’t be used for anything controlling the behavior of Zulip; it’s primarily intended to assist debugging.
Analytics
A Client is used to sort messages into client categories such as ZulipElectron on the /stats page. For more
information see, Analytics.
Integrations
Generally, integrations in Zulip should declare a unique User-Agent, so that it’s easy to figure out which integration
is involved when debugging an issue. For incoming webhook integrations, we do that convenentialy via the auth
decorators (as we will describe shortly); other integrations generally should set the first User-Agent element on their
HTTP requests to something of the form ZulipIntegrationName/1.2 so that they are categorized properly.
The api_key_only_webhook_view auth decorator, used for most incoming webhooks, accepts the name of the
integration as an argument and uses it to generate a client name that it adds to the request (Django HttpRequest)
object as request.client.
In most integrations, request.client is then passed to check_send_message, where it is used to keep track
of which client sent the message (which in turn is used by analytics). For more information, see the webhook walkthrough.
289
Zulip Documentation, Release 1.4.0
290
Chapter 57. Clients in Zulip
CHAPTER
58
Logging and Performance Debugging
It’s good to have the terminal running run-dev.py up as you work since error messages including tracebacks along
with every backend request will be printed there.
The messages will look similar to:
2016-05-20 14:50:22,056 INFO
127.0.0.1
GET
302 528ms (db: 1ms/1q)
˓→(+start: 123ms) / (unauth via ?)
[20/May/2016 14:50:22]"GET / HTTP/1.0" 302 0
2016-05-20 14:50:22,272 INFO
127.0.0.1
GET
200 124ms (db: 3ms/2q) /
˓→login/ (unauth via ?)
2016-05-20 14:50:26,333 INFO
127.0.0.1
POST
302 37ms (db: 6ms/7q) /
˓→accounts/login/local/ (unauth via ?)
[20/May/2016 14:50:26]"POST /accounts/login/local/ HTTP/1.0" 302 0
2016-05-20 14:50:26,538 INFO
127.0.0.1
GET
200 12ms (db: 1ms/2q)
˓→(+start: 53ms) /api/v1/events [1463769771:0/0] (cordelia@zulip.com via internal)
2016-05-20 14:50:26,657 INFO
127.0.0.1
GET
200 10ms (+start: 8ms) /api/
˓→v1/events [1463769771:0/0] (cordelia@zulip.com via internal)
2016-05-20 14:50:26,959 INFO
127.0.0.1
GET
200 588ms (db: 26ms/21q) /
˓→[1463769771:0] (cordelia@zulip.com via website)
The format of this output is: timestamp, loglevel, IP, HTTP Method, HTTP status code, time to process, (optional perf
data details, e.g. database time/queries, memcached time/queries, Django process startup time, markdown processing
time, etc.), URL, and “email via client” showing user account involved (if logged in) and the type of client they used
(“web”, “Android”, etc.).
291
Zulip Documentation, Release 1.4.0
292
Chapter 58. Logging and Performance Debugging
CHAPTER
59
Zulip server release checklist
This document has reminders of things one might forget to do when preparing a new release.
A week before the release
• Upgrade all Python dependencies in requirements to latest upstream versions so they can burn in (use pip
list --outdated).
• Update all the strings on Transifex and notify translators that they should translate the new strings to get them
in for the next release.
• Update changelog.md with major changes going into the release.
• Create a burndown list of bugs that need to be fixed before we can release, and make sure all of them are being
worked on.
Final release preparation
• Update changelog.md with any final changes since the last update.
• Draft the release notes; see previous zulip-announce emails for the tooling needed.
• Download updated translation strings from Transifex and commit them.
• Use build-release-tarball to generate a release tarball.
• Test the new tarball extensively, both new install and upgrade from last release, on both Trusty and Xenial.
• Repeat until release is ready.
293
Zulip Documentation, Release 1.4.0
Executing the release
• Do final updates to changelog.md.
• Update ZULIP_VERSION in version.py.
• Use build-release-tarball to generate a final release tarball.
• Post the release tarball on zulip.org and update zulip.org.
• Create a git tag and push the tag.
• Upload the release on GitHub so it doesn’t provide a broken release tarball.
• Email zulip-announce with the release notes
• For a major release, post on the blog, tweet, etc.
Post-release
• Update ZULIP_VERSION in version.py to e.g. 1.6.0+git.
294
Chapter 59. Zulip server release checklist
CHAPTER
60
Zulip PyPI package release checklist
This document describes the steps to be followed when preparing a new release of the PyPI package for our API
bindings.
While performing the steps outlined below, we should adhere to the guidelines presented in the Python Packaging
User Guide.
The steps below assume that you are familiar with the material presented here.
1. Reconfigure the package, if need be (upgrade version number, development status, and so on).
2. Create a source distribution.
3. Create a pure Python Wheel.
4. Upload the distribution file(s) to zulip-beta.
5. Post about the beta release in #general and test the zulip-beta package extensively.
6. Respond to the feedback received in Step 5.
7. Make final changes, upload the distribution file(s) to the main zulip package.
8. Post in #general about the new release.
Note: We may upload directly to the main zulip package without beta-testing on zulip-beta, if we feel that the changes
made in the new release are minor and not disruptive enough to warrant extensive pretesting.
295
Zulip Documentation, Release 1.4.0
296
Chapter 60. Zulip PyPI package release checklist
CHAPTER
61
Documentation
Zulip has three major documentation systems:
• Developer and sysadmin documentation: Documentation for people actually interacting with the Zulip codebase
(either by developing it or installing it), and written in Markdown.
• Core website documentation: Complete webpages for complex topics, written in HTML, JavaScript, and CSS
(using the Django templating system). These roughly correspond to the documentation someone might look at
when deciding whether to use Zulip. We don’t expect to ever have more than about 10 pages written using this
system.
• General user documentation: Our scalable system for documenting Zulip’s huge collection of specific features
without a lot of overhead or duplicated code/syntax, written in Markdown. We expect to eventually have around
100 pages written using this system. The target audience for this system is individual Zulip users.
These three systems are documented in detail.
Developer and sysadmin documentation
What you are reading right now is part of the collection of documentation targeted at developers and people running
their own Zulip servers. These docs are written in Commonmark Markdown with a small bit of rST. We’ve chosen
Markdown because it is easy to write. The source for Zulip’s developer documentation is at docs/ in the Zulip git
repository, and they are served in production at zulip.readthedocs.io.
If you want to build the developer documentation locally (e.g. to test your changes), the dependencies are automatically
installed as part of Zulip development environment provisioning, and you can build the documentation using:
cd docs/
make html
and then opening http://127.0.0.1:9991/docs/index.html in your browser. The raw files are available
at file:///path/to/zulip/docs/_build/html/index.html in your browser (so you can also use e.g.
firefox docs/_build/html/index.html from the root of your Zulip checkout).
297
Zulip Documentation, Release 1.4.0
If you are adding a new page to the table of contents, you will want to modify docs/index.rst and run make
clean before make html, so that other docs besides your new one also get the new entry in the table of contents.
You can also usually test your changes by pushing a branch to GitHub and looking at the content on the GitHub web
UI, since GitHub renders Markdown, though that won’t be as faithful as the make html approach.
When editing dependencies for the Zulip documentation, you should edit requirements/docs.txt (which is
used by ReadTheDocs to build the Zulip developer documentation, without installing all of Zulip’s dependencies).
Core website documentation
Zulip has around 10 HTML documentation pages under templates/zerver for specific major topics, like the features list, client apps, integrations, hotkeys, API bindings, etc. These documents often have somewhat complex HTML
and JavaScript, without a great deal of common pattern between them other than inheriting from the portico.html
template. We generally avoid adding new pages to this collection unless there’s a good reason, but we don’t intend
to migrate them, either, since this system gives us the flexibility to express these important elements of the product
clearly.
General user documentation
To learn more about Zulip’s general user documentation, visit our guide on writing user documentation here.
298
Chapter 61. Documentation
CHAPTER
62
General user guide documentation
Our goal is for Zulip to have complete, high-quality user-appealing documentation about use Zulip’s features and
perform certain tasks, such as setting up an organization. This system is designed to make writing and maintaining
such documentation highly efficient.
Editing and testing
The user documentation is available under /help/ on any Zulip server; (e.g. https://chat.zulip.org/help/ or http:/
/localhost:9991/help/ in the Zulip development environment). The user documentation is not hosted on
ReadTheDocs, since Zulip supports running a server completely disconnected from the Internet, and we’d like the
documentation to be available in that environment.
The source for this user documentation is the Markdown files under templates/zerver/help/ in the main
Zulip server repository. The file foo.md is automatically rendered by the render_markdown_path function
in zerver/templatetags/app_filters.py when the user accesses a URL of the form /help/foo; with
special cases for /help/ going to index.md and /help/unknown_article going to missing.md (with a
404 response). Images are usually linked from static/images/help/.
This means that you can contribute to the Zulip user documentation by just adding to or editing the collection of
markdown files under templates/zerver/help. If you have the Zulip development environment setup, you
simply need to reload your browser on http://localhost:9991/help/foo to see the latest version of foo.
md rendered.
Writing documentation
Writing documentation is a different form of writing than most people have experience with.
When you write user-appealing documentation, keep in mind that most users don’t care about how the underlying code
works; they simply want to know how a feature works or how they can do something.
299
Zulip Documentation, Release 1.4.0
Think of writing user documentation as writing a recipe; you’re trying to direct your readers how to cook a dish while
introducing them to the dish itself, yet you shouldn’t go into detail about the scientific processes that allow you to cook
your dish.
By keeping these thoughts in mind, you’ll be able to write better user guide documentation.
Title
The title of your documentation should be a brief summary of your documentation. Your title should be formatted as
a heading by prepending your title with a #.
The title of your documentation needs to be included under a fitting section in templates/zerver/help/
index.md so users can access it from the user documentation index.
Introduction
By writing a brief introduction of what you will be discussing in your documentation, users will gain a better understanding of your document. For example, if you are writing about a feature, you could describe what the feature allows
users to do. Your introduction should be no longer than two paragraphs.
Directions
If you are guiding a user on how to perform a certain task in Zulip, it is best to explain the process through a series of
numbered steps. Your documentation’s goal is to direct users on how to do something so that they achieve their goals.
Thus, steps should be as detailed yet concise as possible.
For example, if you tried to explain copying and pasting text, you could describe it in the following steps:
1. Highlight the text you want to copy.
2. Press CTRL+C to copy the highlighted text.
3. Move your cursor to the location where you want to paste the text.
4. Press CTRL+V to paste the copied text.
You can continue steps with additional notes or instructions by breaking the line and indenting the next line, which is
especially useful for including images within a single step.
1. Do something.
You can also do this.
A general tip for breaking up a process is whenever a user interacts with the Zulip UI (e.g.: click a button, refresh the
page), a new step should be created.
Sections
If you are documenting multiple processes in your documentation, such as how to perform a task by using several
different features, you should differentiate the processes by dividing them into sections using headings. Like the title,
your section headings should give a brief description of the processes you describe.
All subsequent subsections should be appended with an increasing number of #‘s to make the headings smaller.
300
Chapter 62. General user guide documentation
Zulip Documentation, Release 1.4.0
# Title
## Section 1
### Subsection 1
### Subsection 2
## Section 2
### Subsection 1
Always leave a blank line on either side of a section heading.
User interface
When you refer to the features in the Zulip UI, you should bold the feature’s name followed by the feature itself (e.g.
Settings page, Change password button, Email field). No quotation marks should be used.
Keep in mind that the UI may change — don’t describe it in more detail than is needed. Never identify or refer to a
button by its color.
Features
Zulip’s Markdown processor allows you to include several special features in your documentation to help improve its
readibility:
• Since raw HTML is supported in Markdown, you can include arbitrary HTML/CSS in your documentation as
needed.
• Code blocks allow you to highlight syntax, similar to Zulip’s own markdown.
• Anchor tags can be used to link to headers in other documents.
• Images of Zulip UI can be added to documentation.
• Inline icons used to refer to features in the Zulip UI.
• You can utilize macros to limit repeated content in the documentation.
• You can create special highlight warning blocks using tips and warnings.
Images
Images and screenshots should be included in user documentation only if it will help guide the user in how to do
something (e.g. if the image will make it much clearer which element on the page the user should interact with). For
instance, an image of an element should not be included if the element the user needs to interact with is the only thing
on the page, but images can be included to show the end result of an interaction with the UI.
Using too many screenshots creates maintainability problems (we have to update them every time the UI is changed)
and also can make the instructions for something simple look long and complicated.
When taking screenshots, the image should never include the whole Zulip browser window in a screenshot; instead, it
should only show relevant parts of the app. In addition, the screenshot should always come after the text that describes
it, never before.
Images are often formatted as continuations of steps and must be indented on a new line to be formatted this way.
62.3. Features
301
Zulip Documentation, Release 1.4.0
Icons
You can refer to features in the Zulip UI by referencing their names and their FontAwesome (version 4.7.0) text
icons within parentheses. The source for the text icons is located in static/third/thirdparty-fonts.css.
Note: It is strongly recommended to use the new base class fa instead of the older base class icon-vector when
specifying icons. In future we will be removing support for the icons with base class icon-vector.
• cog () icon — cog (<i class="fa fa-cog"></i>) icon
• down chevron () icon — down chevron (<i class="fa fa-chevron-down"></i>) icon
• eye () icon — eye (<i class="fa fa-eye"></i>) icon
• file () icon — file (<i class="fa fa-file-text-o"></i>) icon
• filled star () icon — filled star (<i class="fa fa-star"></i>) icon
• formatting () icon — formatting (<i class="fa fa-font"></i>) icon
• menu () icon — menu (<i class="fa fa-bars"></i>) icon
• overflow ( ) icon — overflow ( <i class="fa fa-ellipsis-v"></i> ) icon
• paperclip () icon — paperclip (<i class="fa fa-paperclip"></i>) icon
• pencil () icon — pencil (<i class="fa fa-pencil"></i>) icon
• pencil and paper () icon — pencil and paper (<i class="fa fa-pencil-square-o"></i>)
icon
• plus () icon — plus (<i class="fa fa-plus"></i>) icon
• smiley face () icon — smiley face (<i class="fa fa-smile-o"></i>) icon
• star () icon — star (<i class="fa fa-star-o"></i>) icon
• trash () icon — trash (<i class="fa fa-trash-o"></i>) icon
• x () icon — x (<i class="fa fa-times"></i>) icon
Macros
Macros are elements in the format of {!macro.md!} that insert common phrases and steps at the location of the
macros. Macros help eliminate repeated content in our documentation.
The source for macros is the Markdown files under templates/zerver/help/include in the main Zulip
server repository. If you find multiple instances of particular content in the documentation, you can always create a
new macro by adding a new file to that folder.
Organization settings {!admin.md!} macro
• About: Links to the Organization settings documentation. Usually preceded by the Go to the macro and a link
to a particular section on the Organization settings page.
• Contents:
tab of the [Organization settings](/help/edit-administrator-settings) page.
• Example usage and rendering:
{!go-to-the.md!} [Organization settings](/#organization/organization-settings)
{!admin.md!}
302
Chapter 62. General user guide documentation
Zulip Documentation, Release 1.4.0
1. Go to the [Organization settings](/#organization/organization-settings) tab of
˓→the
[Organization](/help/edit-administrator-settings) page.
Administrator only feature {!admin-only.md!} macro
• About: Creates a note the feature discussed in the documentation is only available to organization administrators. It should be placed immediately after the title.
• Contents:
!!! warn ""
**Note:** This feature can only be controlled by organization administrators.
• Example usage and rendering:
{!admin-only.md!}
{!follow-steps.md!} change who can join your stream by changing the stream's
accessibility.
!!! warn ""
**Note:** This feature can only be controlled by organization administrators.
Follow the following steps to change who can join your stream by
changing the stream's accessibility.
All streams {!all-streams.md!} macro
• About: Explains how to view all streams in the organization on the Streams page. Usually formatted as a tip
and preceded by the Streams macro and the Filter streams macro.
• Contents:
If you wish to see streams that you aren't subscribed to, click on the
**All streams** tab; the tab will turn gray upon doing so.
• Example usage and rendering:
{!subscriptions.md!}
{!filter-streams.md!}
!!! tip ""
{!all-streams.md!}
1. [Find the relevant stream](/help/browse-and-join-streams#browse-streams) on the
[Streams](/#streams) page. You can search for specific streams by entering the
name of the stream in the **Filter streams** input.
!!! tip ""
If you wish to see streams that you aren't subscribed to, click on the
**All streams** tab; the tab will turn gray upon doing so.
62.3. Features
303
Zulip Documentation, Release 1.4.0
Down chevron {!down-chevron.md!} macro
• About: Instructs readers to click on the down chevron () icon to reveal an actions dropdown; usually preceded
by an command, such as the Message actions macro.
• Contents:
down chevron (<i class="fa fa-chevron-down"></i>) icon to reveal an actions
˓→dropdown.
• Example usage and rendering:
{!message-actions.md!}
{!down-chevron.md!}
1. Hover over a message to replace the message's timestamp with its message
actions, represented by three icons. From the icons that appear, select the
down chevron (<i class="fa fa-chevron-down"></i>) icon to reveal an actions
˓→dropdown.
Go to the {!go-to-the.md} macro
• About: Usually precedes the Settings macro or the Organization settings macro. Transforms following content
into a step.
• Contents:
1. Go to the
• Example usage and rendering:
{!go-to-the.md!} [Notifications](/#settings/notifications)
{!settings.md!}
1. Go to the [Notifications](/#settings/notifications) tab on the
[Settings](/help/edit-settings) page.
Filter streams {!filter-streams.md!} macro
• About: Explains how to search for specific streams in the Streams page using the Filter streams input. Usually
preceded by the Streams macro.
• Contents:
You can search for specific streams by entering the name of the stream in
the **Filter streams** input.
• Example usage and rendering:
{!subscriptions.md!}
{!filter-streams.md!}
1. [Find the relevant stream](/help/browse-and-join-streams#browse-streams) on the
[Streams](/#streams) page. You can search for specific streams by entering the
name of the stream in the **Filter streams** input.
304
Chapter 62. General user guide documentation
Zulip Documentation, Release 1.4.0
Follow steps {!follow-steps.md!} macro
• About: Prepends phrases with instructions to follow the following steps.
• Contents:
Follow the following steps to
• Example usage and rendering:
{!follow-steps.md!}
change your mobile notification settings.
Follow the following steps to change your mobile notification
settings.
Message actions {!message-actions.md!} macro
• About: Explains how to view the actions of message. Usually followed by an instruction to click a specific
icon, such as the Down chevron macro.
• Contents:
1. Hover over a message to replace the message's timestamp with its message
actions, represented by three icons. From the icons that appear, select the
• Example usage and rendering:
{!message-actions.md!}
{!down-chevron.md!}
1. Hover over a message to replace the message's timestamp with its message
actions, represented by three icons. From the icons that appear, select the
down chevron (<i class="fa fa-chevron-down"></i>) icon to reveal an actions
˓→dropdown.
Save changes {!save-changes.md!} macro
• About: Used in documentation that requests users to save their changes by pressing the Save changes button.
Usually followed by a phrase describing the settings they modified.
• Contents:
1. Click on the **Save changes** button to save any changes you made to
your
• Example usage and rendering:
{!save-changes.md!} notification settings.
1. Click on the **Save changes** button to save any changes you made to
your notification settings.
62.3. Features
305
Zulip Documentation, Release 1.4.0
Settings {!settings.md!} macro
• About: Links to the Edit Settings documentation. Usually preceded by the Go to the macro and a link to a
particular section on the Settings page.
• Contents:
tab on the [Settings](/help/edit-settings) page.
• Example usage and rendering:
{!go-to-the.md!} [Notifications](/#settings/notifications)
{!settings.md!}
1. Go to the [Notifications](/#settings/notifications) tab on the
[Settings](/help/edit-settings) page.
Stream actions {!stream-actions.md!} macro
• About: Explains how to view the actions of stream. Usually followed by the an instruction and the Down
chevron macro.
• Contents:
1. On the left sidebar in the **Streams** section, hover over a stream to reveal
a down chevron (<i class="fa fa-chevron-down"></i>) icon to the right of
the stream name.
• Example usage and rendering:
{!stream-actions.md!}
1. Click on the {!down-chevron.md!}
1. On the left sidebar in the **Streams** section, hover over a stream to reveal
a down chevron (<i class="fa fa-chevron-down"></i>) icon to the right of
the stream name.
2. Click on the down chevron (<i class="fa fa-chevron-down"></i>)
icon to reveal an actions dropdown.
Stream settings {!stream-settings.md!} macro
• About: Notifies readers about the changes in the Streams page when a stream is selected; usually followed by
an instruction.
• Contents:
the right side of the [Streams](/#streams) page, labeled
**Stream settings**, will now display the selected stream's settings.
• Example usage and rendering:
1. Click on the stream you want to edit; {!stream-settings.md!}
306
Chapter 62. General user guide documentation
Zulip Documentation, Release 1.4.0
1. Click on the stream you want to edit; the right side of the
[Streams](/#streams) page, labeled **Stream settings**, will
now display the selected stream's settings.
Stream settings scroll {!stream-settings.md!} macro
• About: Instructs readers to scroll down to a particular section on the Streams page after making sure their
cursors are hovering above the Streams Settings section.
• Contents:
1. After making sure that your cursor is hovering over the **Streams Settings**
section, scroll down to the
• Example usage and rendering:
{!stream-settings-scroll.md!} **Stream membership** section. This section
shows the usernames and emails of all users that are currently subscribed to the
selected stream.
1. After making sure that your cursor is hovering over the **Streams Settings**
section, scroll down to the **Stream membership** section. This section
shows the usernames and emails of all users that are currently subscribed to the
selected stream.
Streams {!subscriptions.md!} macro
• About: Used in documentation that direct users to the Streams page. Often followed by the Filter streams
macro.
• Contents:
1. [Find the relevant stream](/help/browse-and-join-streams#browse-streams) on
the [Streams](/#streams) page.
• Example usage and rendering:
{!subscriptions.md!}
{!filter-streams.md!}
1. [Find the relevant stream](/help/browse-and-join-streams#browse-streams) on the
[Streams](/#streams) page. You can search for specific streams by entering the
name of the stream in the **Filter streams** input.
Tips and warnings
A tip is any suggestion for the user that is not part of the main set of instructions. For instance, it may address a
common problem users may encounter while following the instructions, or point to an option for power users.
!!! tip ""
If you've forgotten your password, see the
[Change your password](/help/change-your-password) page for
instructions on how to reset it.
62.3. Features
307
Zulip Documentation, Release 1.4.0
A warning is a note on what happens when there is some kind of problem. Tips are more common than warnings.
!!! warn ""
**Note:** If you attempt to input a nonexistent stream name, an error
message will appear.
All tips/warnings should appear inside tip/warning blocks. There should be only one tip/warning inside each
block.They usually be formatted as a continuation of a numbered step.
Documentation template
Here is a template for writing user documentation that follows the guidelines listed in this document. It demonstrates
the proper formatting of several features discussed above.
You can always look at previously-written documentation in the templates/zerver/help folder as a reference
for writing your documentation.
# Title of documentation
This paragraph briefly summarizes the documentation. Please remember to
replace this paragraph with an appropriate description of the documentation.
## Writing documentation as a series of steps
1. This is the first step of the documentation.
![First step](/static/images/help/image1.png)
2. This is the second step of the documentation.
Here's an additional note about the second step. Click on the star (<i
class="fa fa-star-half-o"></i>) icon to learn more about it.
3. This is the third step of the documentation.
!!! tip ""
You can always add more steps to your documentation!
## Writing documentation to introduce a feature
If necessary, you can add another section to your documentation. Sections
can be used to differentiate different methods of performing a task or
describing a related task.
![Feature](/static/images/help/feature.png)
You can also conclude your documentation with some final notes.
!!! warn ""
**Note:** This is only a documentation template; you can always deviate
**from this template to suit your documentation needs.
308
Chapter 62. General user guide documentation
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertising