Merge branch 'staging' into reminders

CHANGELOG.md

@@ -2,52 +2,55 @@
 
 ## Goals
 
+Our main goal for this release was to complete the first steps of our ["Dialogue" project](https://nlnet.nl/project/CryptPadForms/), which will introduce surveys into CryptPad. We've also put considerable effort towards addressing some configuration issues, correcting some inconsistently translated UI, and writing some new documentation.
+
 ## Update notes
 
-* no default privacy policy
+This release removes the default privacy policy that has been included in CryptPad up until now. It included some assertions that were true of our own instance (CryptPad.fr) which we couldn't guarantee on third-party instances. We've updated our custom configuration to link to a privacy policy that was written in a rich text pad. You can do the same on your instance by editing `cryptpad/customize/application_config.js` to include the absolute URL of your instance, like so: `AppConfig.privacy = "https://cryptpad.your.website/privacy.html";`.
-* nginx update
+
-  * calendar
+We've clarified a point about telemetry in the notes of our 4.3.1 release. The text suggested that users on your instance would send telemetry to OUR webserver. It has been clarified to reflect that telemetry from your users is only ever sent to your instance.
-  * /api/broadcast
+
-* clarified TELEMETRY in the 4.3.1 release notes
+We've spent some time working on improving our (officially) unreleased integrations of OnlyOffice's presentation and document editors. We've advised against enabling these editors on your instance. This release includes changes that may not be fully backwards compatible. If your users rely on either editor we advise that you not update until they have had an opportunity to back up their documents. We still aren't officially supporting either editor and we may make further breaking changes in the future. Consider this a warning and not an advertizement of their readiness!
+
+This release also includes changes to the recommended NGINX configuration. Compare your instance's config against `cryptpad/docs/example.nginx.conf` and apply all the new changes before updating. In particular, you'll want to pay attention to the configuration for a newly exposed server API (`/api/broadcast`). This should work much the same as `/api/config`, so if you're using a non-standard configuration that uses more than one server you may want to proxy it in a similar fashion.
+
+To update from 4.3.1 to 4.4.0:
+
+1. Apply the documented NGINX configuration
+2. Stop your server
+3. Get the latest code with git
+4. Install the latest dependencies with `bower update` and `npm i`
+5. Restart your server
+
+This release requires updates to both clientside and serverside dependencies. **You will experience problems if you skip any of the above steps.**
 
 ## Features
 
-* prompt premium users to cancel their subscriptions before deleting their accounts
+* 4.4.0 includes a basic version of a calendar app. There are no links to it anywhere in the platform, its translations are hardcoded, and its title includes the text **BETA**. It's included in this release so that we can test and improve it for the next release, however, it should not be considered stable. Use it at your own risk! Our plan for this app is to offer the ability to set and review reminders for deadlines in CryptPad. We haven't secured funding for more advanced functionality, however, our team is available for sponsored development if you'd like to provide funding to include such improvements in our short-term roadmap.
-* check that headers for XLSX export are correctly set via the checkup app
+* The admin panel now includes several closely related features in its "broadcast" tab, which allows administrators to send a few types of notifications to all users:
-* remove HTML from most translations
+  1. _Maintenance notices_ inform users that the service may be unavailable during a specified time range.
-* localize links to the docs where a translation exists
+  2. _Survey notices_ inform users that the instance administrators have published a new survey and would like their feedback. We plan to use this on CryptPad.fr to perform some voluntary user studies on an ongoing basis.
-* implement admin-broadcast features
+  3. _Broadcast messages_ allow admins to send all users a custom message with optional localization in their users' preferred language.
-* add "getting started" banner in the drive
+* The drive now includes a "Getting started" message and a link to our docs, like all our other apps. This replaces the creation of a personal "What is CryptPad" pad in the user's drive when they register.
-* calendars: BETA
+* The /checkup/ page which performs some automatic tests to validate the host instance's configuration now includes a few more tests that check that the sheet editor has the correct HTTP headers applied and that the _broadcast API_ is available.
-* clear document cache when visiting /logout/
+* We recently wrote some scripts to automatically review our translations. This exposed some inconsistencies and incorrectly applied attributes in translations that included HTML. Since it's not reasonable to expect translators to know HTML, we've taken some steps to remove all but the most basic markup from translatable messages. Instead, more advanced attributes are applied via JavaScript. This makes it easier than ever to translate CryptPad as well as providing a more consistent experience to those using translations written by contributors.
 
 ## Bug fixes
 
-* bad channel IDs stored in your drive or accessed via bad links (corrupted somehow)
+* Premium users are now prompted to cancel their subscriptions before deleting their accounts.
-  * don't try to join invalid channels
+* The /logout/ page will now clear users' local document cache. Admins can recommend that users try loading this page when users are mysteriously unable to load their drive (or that of a team). If you find that this solves a user's problem, please report their exact problem so we can investigate the underlying cause.
-  * don't try to get their metadata
+* The _support_ page guards against type errors that appear to have been caused by third-party extensions interfering with some browser APIs and rewriting URLs.
-* guard against some type errors in the support page
+* We found that anonymous users who had not created a drive were not able to use the "Make a copy" functionality on a pad that they were viewing. This has been fixed.
-* remove redundant link from OpenCollective popup
+* We noticed that under some unknown circumstances it was possible for users to store documents with invalid document IDs in their drive. We've added a few guards that detect these invalid channels and we're working on a solution to automatically repair them, if possible.
-* guard against a type error when copying a pad in nodrive mode
+* Links to anchors in read-only rich text documents now navigate to the correct section of the document rather than opening a new tab.
-* correctly navigate to anchors when clicking links to anchors in read-only rich-text pads
+* We've made a large number of improvements to our OnlyOffice integration. This will primarily affect the sheet app, but it also paves the way for us to introduce presentations and text documents in a future release.
-
+  * We now inform OnlyOffice of user-list changes, which should fix the incorrect display of users names when they lock a portion of a document.
-
+  * Text documents and presentations use a different data format than sheets for locking the document. We've adjusted our code to handle these formats.
-* OnlyOffice
+  * We've fixed some lock-related errors in sheets that could be triggered when receiving checkpoints from other users while editing in strict mode.
-  * inform OnlyOffice of userlist changes
+  * We've adjusted some CSS selectors intended to hide parts of OnlyOffice's UI that are invalid within CryptPad, since those elements' IDs have changed since the last version.
-  * rename doc and slide editors
+  * OnlyOffice's cursors now use your CryptPad account's preferred color.
-  * handle different lock formats for docs and slides
+  * We now handle some errors that occurred when documents were migrated by a user editing a sheet in embed mode.
-    * relative to sheets
+  * OnlyOffice modified some of the APIs used to lock a document, so we've adjusted our code to match.
-  * handle some cursor logic outside of sheets
-  * handle locks when integrating remote checkpoints in strict mode
-  * OnlyOffice renamed buttons in slides and docs and we need to hardcode CSS that hides them by their randomly generated IDs
-  * support CryptPad cursor colors in OnlyOffice by adding opacity value
-  * use the appropriate APIs to detect if the document is modified
-  * display users cursor colors in the toolbar next to their name
-  * handle errors when migrating in embed mode
-  * change the method we use to lock the whole sheet since OnlyOffice changed their internal API's behaviour
-  * **soft release of OnlyOffice presentations and docs**
-    * if you've been using them, tell your users to export them before they break
-    * we still don't recommend that you use either editor!
 
 # 4.3.1
 

customize.dist/src/less2/pages/page-checkup.less

@@ -58,6 +58,13 @@ html, body {
         border: 1px solid red;
         background-color: @cp_alerts-danger-bg;
         color: @cp_alerts-danger-text;
+        code {
+            word-break: keep-all;
+            font-style: italic;
+        }
+        a {
+            color: @cryptpad_color_link;
+        }
     }
 
     iframe {

server.js

@@ -43,6 +43,7 @@ var canonicalizeOrigin = function (s) {
     }
 
     if (typeof(config.httpSafeOrigin) !== 'string') {
+        Env.NO_SANDBOX = true;
         if (typeof(config.httpSafePort) !== 'number') {
             config.httpSafePort = config.httpPort + 1;
         }
@@ -110,9 +111,22 @@ var setHeaders = (function () {
                 "Cross-Origin-Embedder-Policy": 'require-corp',
             });
 
+            if (Env.NO_SANDBOX) {
+                applyHeaderMap(res, {
+                    "Cross-Origin-Resource-Policy": 'cross-origin',
+                });
+            }
+
             // Don't set CSP headers on /api/config because they aren't necessary and they cause problems
             // when duplicated by NGINX in production environments
-            if (/^\/api\/(broadcast|config)/.test(req.url)) { return; }
+            if (/^\/api\/(broadcast|config)/.test(req.url)) {
+                if (!Env.NO_SANDBOX) {
+                    applyHeaderMap(res, {
+                        "Cross-Origin-Resource-Policy": 'cross-origin',
+                    });
+                }
+                return;
+            }
             applyHeaderMap(res, {
                 "Cross-Origin-Resource-Policy": 'cross-origin',
             });

www/assert/assertions.js

@@ -3,8 +3,9 @@ define([], function () {
         var failMessages = [];
         var passed = 0;
         var ASSERTS = [];
-
+        var MESSAGES = [];
         var assert = function (test, msg) {
+            MESSAGES.push(msg || false);
             ASSERTS.push(function (cb, i) {
                 test(function (result) {
                     if (result === true) {
@@ -17,7 +18,7 @@ define([], function () {
                             output: result,
                         });
                     }
-                });
+                }, msg);
             });
         };
 

www/checkup/main.js

@@ -19,38 +19,100 @@ define([
 ], function ($, ApiConfig, Assertions, h, Messages, DomReady,
             nThen, SFCommonO, Login, Hash, Util, Pinpad,
             NetConfig) {
-    var assert = Assertions();
+    var Assert = Assertions();
-
     var trimSlashes = function (s) {
         if (typeof(s) !== 'string') { return s; }
         return s.replace(/\/+$/, '');
     };
 
-    var _alert = function (content) {
+    var assert = function (f, msg) {
-        return h('span.advisory-text', content);
+        Assert(f, msg || h('span.advisory-text'));
+    };
+
+    var CONFIG_PATH = function () {
+        return h('code', 'cryptpad/config/config.js');
+    };
+    var API_CONFIG_LINK = function () {
+        return h('a', {
+            href: '/api/config',
+            target: '_blank',
+        }, '/api/config');
+    };
+
+    var RESTART_WARNING = function () {
+        return h('span', [
+            'Changes to ',
+            CONFIG_PATH(),
+            ' will require a server restart in order for ',
+            API_CONFIG_LINK(),
+            ' to be updated.',
+        ]);
     };
 
     var trimmedSafe = trimSlashes(ApiConfig.httpSafeOrigin);
     var trimmedUnsafe = trimSlashes(ApiConfig.httpUnsafeOrigin);
 
-    assert(function (cb) {
+    assert(function (cb, msg) {
+        msg.appendChild(h('span', [
+            "CryptPad's sandbox requires that both ",
+            h('code', 'httpUnsafeOrigin'),
+            ' and ',
+            h('code', 'httpSafeOrigin'),
+            " be configured in ",
+            CONFIG_PATH(),
+            '. ',
+            RESTART_WARNING(),
+        ]));
+
         //console.error(trimmedSafe, trimmedUnsafe);
         cb(Boolean(trimmedSafe && trimmedUnsafe));
-    }, _alert("Sandbox configuration: ensure that both httpUnsafeOrigin and httpSafeOrigin are defined"));
+    });
+
+    assert(function (cb, msg) {
+        msg.appendChild(h('span', [
+            h('code', 'httpUnsafeOrigin'),
+            ' and ',
+            h('code', 'httpSafeOrigin'),
+            ' are equivalent. ',
+            "In order for CryptPad's security features to be as effective as intended they must be different. ",
+            "See ",
+            CONFIG_PATH(),
+            '. ',
+            RESTART_WARNING(),
+        ]));
 
-    assert(function (cb) {
         return void cb(trimmedSafe !== trimmedUnsafe);
-    }, _alert('Sandbox configuration: httpUnsafeOrigin !== httpSafeOrigin'));
+    });
+
+    assert(function (cb, msg) {
+        msg.appendChild(h('span', [
+            h('code', 'httpUnsafeOrigin'),
+            ' and ',
+            h('code', 'httpSafeOrigin'),
+            ' must not contain trailing slashes. This can be configured in ',
+            CONFIG_PATH(),
+            '. ',
+            RESTART_WARNING(),
+        ]));
+        cb(trimmedSafe === ApiConfig.httpSafeOrigin && trimmedUnsafe === ApiConfig.httpUnsafeOrigin);
+    });
 
-    assert(function (cb) {
+    assert(function (cb, msg) {
-        cb(trimmedSafe === ApiConfig.httpSafeOrigin);
+        msg.appendChild(h("span", [
-    }, "httpSafeOrigin must not have a trailing slash");
+            "It appears that you are trying to load this page via an origin other than its main domain (",
+            h('code', ApiConfig.httpUnsafeOrigin),
 
-    assert(function (cb) {
+            "). See the ",
+            h('code', 'httpUnsafeOrigin'),
+            " option in ",
+            CONFIG_PATH(),
+            " which is exposed via ",
+            API_CONFIG_LINK(),
+            '.',
+        ]));
         var origin = window.location.origin;
         return void cb(ApiConfig.httpUnsafeOrigin === origin);
-    }, _alert('Sandbox configuration: loading via httpUnsafeOrigin'));
+    });
-
 
     var checkAvailability = function (url, cb) {
         $.ajax({
@@ -62,12 +124,38 @@ define([
         });
     };
 
-    assert(function (cb) {
+    assert(function (cb, msg) {
+        msg.appendChild(h('span', [
+            "The main domain (configured via ",
+            h('code', 'httpUnsafeOrigin'),
+            ' as ',
+            ApiConfig.httpUnsafeOrigin,
+            ' in ',
+            CONFIG_PATH(),
+            ' and exposed via ',
+            API_CONFIG_LINK(),
+            ') could not be reached.',
+        ]));
+
         checkAvailability(trimmedUnsafe, cb);
-    }, _alert("Main domain is not available"));
+    });
 
     // Try loading an iframe on the safe domain
-    assert(function (cb) {
+    assert(function (cb, msg) {
+        msg.appendChild(h('span', [
+            "Your browser was not able to load an iframe using the origin specified as ",
+            h('code', "httpSafeOrigin"),
+            " (",
+            ApiConfig.httpSafeOrigin,
+            ") in ",
+            CONFIG_PATH(),
+            ". This can be caused by an invalid ",
+            h('code', 'httpUnsafeDomain'),
+            ', invalid CSP configuration in your reverse proxy, invalid SSL certificates, and many other factors. ',
+            'More information about your particular error may be found in your browser console. ',
+            RESTART_WARNING(),
+        ]));
+
         var to;
         nThen(function (waitFor) {
             DomReady.onReady(waitFor());
@@ -82,16 +170,22 @@ define([
             clearTimeout(to);
             cb(true);
         });
-    }, _alert("Sandbox domain is not available"));
+    });
 
     // Test Websocket
     var evWSError = Util.mkEvent(true);
-    assert(function (cb) {
+    assert(function (_cb, msg) {
+        var cb = Util.once(Util.both(_cb, function (err) {
+            if (typeof(err) === 'string') {
+                msg.innerText = err;
+            }
+        }));
+
         var ws = new WebSocket(NetConfig.getWebsocketURL());
         var to = setTimeout(function () {
             console.error('Websocket TIMEOUT');
             evWSError.fire();
-            cb('TIMEOUT (5 seconds)');
+            cb('Could not connect to the websocket server within 5 seconds.');
         }, 5000);
         ws.onopen = function () {
             clearTimeout(to);
@@ -99,14 +193,25 @@ define([
         };
         ws.onerror = function (err) {
             clearTimeout(to);
-            console.error('Websocket error', err);
+            console.error('[Websocket error]', err);
             evWSError.fire();
-            cb('WebSocket error: check your console');
+            cb('Unable to connect to the websocket server. More information may be available in your browser console ([Websocket error]).');
         };
-    }, _alert("Websocket is not available"));
+    });
 
     // Test login block
-    assert(function (cb) {
+    assert(function (cb, msg) {
+        msg.appendChild(h('span', [
+            "Unable to create, retrieve, or remove encrypted credentials from the server. ",
+            "This is most commonly caused by a mismatch between the value of the  ",
+            h('code', 'blockPath'),
+            ' value configured in ',
+            CONFIG_PATH(),
+            " and the corresponding settings in your reverse proxy's configuration file,",
+            " but it can also be explained by a websocket error. ",
+            RESTART_WARNING(),
+        ]));
+
         var bytes = new Uint8Array(Login.requiredBytes);
 
         var opt = Login.allocateBytes(bytes);
@@ -132,7 +237,7 @@ define([
             // If WebSockets aren't working, don't wait forever here
             evWSError.reg(function () {
                 waitFor.abort();
-                cb("No WebSocket (test number 6)");
+                cb("No WebSocket available");
             });
             // Create proxy
             Login.loadUserObject(opt, waitFor(function (err, rt) {
@@ -200,28 +305,78 @@ define([
             cb(true);
         });
 
-    }, _alert("Login block is not working (write/read/remove)"));
+    });
+
+    var sheetURL = '/common/onlyoffice/v4/web-apps/apps/spreadsheeteditor/main/index.html';
 
-    assert(function (cb) {
+    assert(function (cb, msg) {
-        var url = '/common/onlyoffice/v4/web-apps/apps/spreadsheeteditor/main/index.html';
+        msg.innerText = "Missing HTTP headers required for .xlsx export from sheets. ";
+        var url = sheetURL;
         var expect = {
             'cross-origin-resource-policy': 'cross-origin',
             'cross-origin-embedder-policy': 'require-corp',
+            //'cross-origin-opener-policy': 'same-origin', // FIXME this is in our nginx config but not server.js
         };
 
         $.ajax(url, {
             complete: function (xhr) {
                 cb(!Object.keys(expect).some(function (k) {
                     var response = xhr.getResponseHeader(k);
-                    console.log(k, response);
+                    if (response !== expect[k]) {
-                    return response !== expect[k];
+                        msg.appendChild(h('span', [
+                            'A value of ',
+                            h('code', expect[k]),
+                            ' was expected for the ',
+                            h('code', k),
+                            ' HTTP header, but instead a value of "',
+                            h('code', response),
+                            '" was received.',
+                        ]));
+                        return true; // returning true indicates that a value is incorrect
+                    }
                 }));
             },
         });
-    }, _alert("Missing HTTP headers required for XLSX export"));
+    });
 
-    assert(function (cb) {
+    assert(function (cb, msg) {
+        msg.appendChild(h('span', [
+            "The spreadsheet editor's code was not served with the required Content-Security Policy headers. ",
+            "This is most often caused by incorrectly configured sandbox parameters (",
+            h('code', 'httpUnsafeOrigin'),
+            ' and ',
+            h('code', 'httpSafeOrigin'),
+            ' in ',
+            CONFIG_PATH,
+            "), or settings in your reverse proxy's configuration which don't match your application server's config. ",
+            RESTART_WARNING(),
+        ]));
+
+        $.ajax(sheetURL, {
+            complete: function (xhr) {
+                var csp = xhr.getResponseHeader('Content-Security-Policy');
+                if (!/unsafe\-eval/.test(csp)) {
+                    // OnlyOffice requires unsafe-eval
+                    return cb(false);
+                }
+                if (!/unsafe\-inline/.test(csp)) {
+                    // OnlyOffice also requires unsafe-inline
+                    return cb(false);
+                }
+                //console.error('CSP', csp);
                 cb(true);
+            },
+        });
+    });
+
+    assert(function (cb, msg) {
+        msg.appendChild(h('span', [
+            h('code', '/api/broadcast'),
+            " could not be loaded. This can be caused by an outdated application server or an incorrectly configured reverse proxy. ",
+            "Even if the most recent code has been downloaded it's possible the application server has not been restarted. ",
+            "Your browser console may provide more details as to why this resource could not be loaded. ",
+        ]));
+
         $.ajax('/api/broadcast', {
             dataType: 'text',
             complete: function (xhr) {
@@ -229,7 +384,7 @@ define([
                 cb(xhr.status === 200);
             },
         });
-    }, _alert("/api/broadcast is not available"));
+    });
 
     var row = function (cells) {
         return h('tr', cells.map(function (cell) {
@@ -249,7 +404,7 @@ define([
 
     var completed = 0;
     var $progress = $('#cp-progress');
-    assert.run(function (state) {
+    Assert.run(function (state) {
         var errors = state.errors;
         var failed = errors.length;