infra
/
cryptpad
2
1
Fork 0
Code Issues Pull Requests Projects Releases 1 Activity

Merge branch 'staging' into team

Browse Source
pull/1/head
yflory 5 years ago
parent fb22a4a792 5fe1627488
commit a7c274e6ae
13 changed files with 1353 additions and 692 deletions
Show Stats Download Patch File Download Diff File

71
CHANGELOG.md
Unescape Escape View File

@ -1,3 +1,74 @@
# Baiji release (v3.1.0)
## Goals
For CryptPad 3.1.0 we prioritized our work on team-centric features. In particular we wanted to finish some improvements to make our notifications system more private and start making use of our prior work on editable pad metadata.
## Update notes
* `config/config.example.js` has included the `inactiveTime` value for a while. It's used by our archival script (`scripts/evict-inactive.js`) to determine if a pad should be removed. This value is now shared with clients via the `/api/config` endpoint. Unregistered clients now use this value to inform users that unpinned pad will expire after that number of days of inactivity.
* previously the value was hardcoded to "3 months"
* Changes to channel metadata logs and users' pin logs now include the time of the modification.
* this is mostly to help with debugging, though we might use this value in the future
* newly created metadata will also include a `created` field with a timestamp indicating when it was first created on the server
* We've removed two files from our `scripts` directory:
* `delete-inactive.js`: because it ignored the configured values for archival
* `pinned-data.js`: because it was only used by `delete-inactive.js` and we will soon have better ways to accomplish the same goal
* We've made some updates to the server-side components of our caching logic
* CryptPad used to use the `version` value from `package.json` as a cache-busting string so that all assets would be reloaded and cached when you upgraded to a new version
* in practice, lots of administrators had problems with this where they made configuration changes and restarted the server, but their client was stuck with old values cached
* the new default is to generate a cache string at the server's launch time and use this value for the lifetime of the server
* server administrators can still change the cache string through the instance's admin panel
* this behaviour was previously available by launching the server with `FRESH=1 node server.js`
* the old behaviour is still available by launching the server with `PACKAGE=1 node server.js`
* We've refactored some small functions implemented in `historyKeeper.js` which halved our server's memory usage in the previous release and reused those functions in our RPC module.
* we hope this leads to even better performance under heavy load when doing things like
* reading metadata
* checking disk usage (global and for particular users)
* loading a user's pin log
Baiji depends on updates to clientside and serverside dependencies.
To update:
1. Take down your server
2. Pull the latest code
2. `npm install`
3. `bower update`
4. Launch your server
## Features
* Messages sent to a user's encrypted mailbox are now anonymized by the server.
* This means that clients other than the intended recipient of a message no longer have any information indicating the identity of the sender
* It is now possible to modify ownership of pads
* use the "properties modal", available by right-clicking on the pad in your drive or from the properties entry in the "toolbar drawer" in pads
* navigate to the "Availability tab" and click "manage owners" where you can:
* offer ownership to friends, who will receive a notification and will be able to accept or refuse ownership
* remove ownership from confirmed owners
* rescind pending offers
* Amendments to the "owners" field in pad metadata will now also change the "mailbox" field, allowing users with read-only access rights to request editing rights from any of the owners
* the current behaviour is to ask only the first owner in the list, but we'll be able to make use of the additional mailboxes in future releases
* We now consider changes to metadata to be "activity" for a channel for the purposes of deciding whether an unpinned channel should be archived.
* this means that if you offer other users ownership of a pad and remove yourself as owner, even if nobody is pinning the document it will not be removed until the configured period of inactivity from the time when you removed yourself as owner
* The "What is CryptPad" pad which is created in a user's CryptDrive when they first register is now created as an "owned pad" which they can remove from the server
* We've begun work on a basic command-line client which we're mostly using for automated testing of our history-related APIs and our serverside RPCs (Remote Procedure Calls).
* a stable command-line client API won't necessarily be available for the foreseeable future, but these tests should lead to fewer serverside regressions which will be better for the browser client as well
* as we write tests we're converting more and more of our browser-only modules to work in more environments, so native and mobile apps will be easier to implement in the future
* Finally, we've begun to detect and users that try to register with their email address as their username
* we don't prevent them from doing so, but we do warn them that their email address is not actually sent to the server, and we won't be able to use it to recover their account if they forget it or their password
## Bug fixes
* In our previous release we discovered that `config/config.example.js` did not include the configuration point which enabled the server to schedule tasks for the expiration of files.
* even though the pads were created with the expiration time in their metadata, and the server would not serve such files to clients that requested them, they would still remain in the database
* if these expired pads are ever requested and they should have expired over a day before, the server will now archive or delete the file immediately
* We've investigated and fixed a number of errors that were visible in the browser console even if they didn't have harmful effects on the client's behaviour
* when reconnecting
* "channel ready without callback"
* network "EJOINED" error
* Changes to the metadata logs for pads are now queued so that they are always written in the same order as they were received
# Aurochs release (v3.0.0)
The move to 3.0 is mostly because we ran out of letters in the alphabet for our 2.0 release cycle.

1
lib/common-util.js
Unescape Escape View File

@ -0,0 +1 @@
module.exports = require("../www/common/common-util");

4
lib/log.js
Unescape Escape View File

@ -13,9 +13,11 @@ var messageTemplate = function (type, time, tag, info) {
return JSON.stringify([type.toUpperCase(), time, tag, info]);
};
var noop = function () {};
var write = function (ctx, content) {
if (!ctx.store) { return; }
ctx.store.log(ctx.channelName, content);
ctx.store.log(ctx.channelName, content, noop);
};
// various degrees of logging

1
lib/metadata.js
Unescape Escape View File

@ -205,6 +205,7 @@ Meta.createLineHandler = function (ref, errorHandler) {
line: JSON.stringify(line),
});
}
if (typeof(line) === 'undefined') { return; }
if (Array.isArray(line)) {
try {

692
rpc.js
Unescape Escape View File

@ -21,9 +21,15 @@ const Meta = require("./lib/metadata");
const WriteQueue = require("./lib/write-queue");
const BatchRead = require("./lib/batch-read");
const Util = require("./lib/common-util");
const escapeKeyCharacters = Util.escapeKeyCharacters;
const unescapeKeyCharacters = Util.unescapeKeyCharacters;
const mkEvent = Util.mkEvent;
var RPC = module.exports;
var Store = require("./storage/file");
var BlobStore = require("./storage/blob");
var DEFAULT_LIMIT = 50 * 1024 * 1024;
var SESSION_EXPIRATION_TIME = 60 * 1000;
@ -45,42 +51,6 @@ var isValidId = function (chan) {
[32, 48].indexOf(chan.length) > -1;
};
/*
var uint8ArrayToHex = function (a) {
// call slice so Uint8Arrays work as expected
return Array.prototype.slice.call(a).map(function (e) {
var n = Number(e & 0xff).toString(16);
if (n === 'NaN') {
throw new Error('invalid input resulted in NaN');
}
switch (n.length) {
case 0: return '00'; // just being careful, shouldn't happen
case 1: return '0' + n;
case 2: return n;
default: throw new Error('unexpected value');
}
}).join('');
};
*/
var testFileId = function (id) {
if (id.length !== 48 || /[^a-f0-9]/.test(id)) {
return false;
}
return true;
};
/*
var createFileId = function () {
var id = uint8ArrayToHex(Nacl.randomBytes(24));
if (!testFileId(id)) {
throw new Error('file ids must consist of 48 hex characters');
}
return id;
};
*/
var makeToken = function () {
return Number(Math.floor(Math.random() * Number.MAX_SAFE_INTEGER))
.toString(16);
@ -110,14 +80,6 @@ var parseCookie = function (cookie) {
return c;
};
var escapeKeyCharacters = function (key) {
return key && key.replace && key.replace(/\//g, '-');
};
var unescapeKeyCharacters = function (key) {
return key.replace(/\-/g, '/');
};
var getSession = function (Sessions, key) {
var safeKey = escapeKeyCharacters(key);
if (Sessions[safeKey]) {
@ -272,28 +234,6 @@ var getChannelList = function (Env, publicKey, cb) {
});
};
var makeFilePath = function (root, id) { // FIXME FILES
if (typeof(id) !== 'string' || id.length <= 2) { return null; }
return Path.join(root, id.slice(0, 2), id);
};
var getUploadSize = function (Env, channel, cb) { // FIXME FILES
var paths = Env.paths;
var path = makeFilePath(paths.blob, channel);
if (!path) {
return cb('INVALID_UPLOAD_ID');
}
Fs.stat(path, function (err, stats) {
if (err) {
// if a file was deleted, its size is 0 bytes
if (err.code === 'ENOENT') { return cb(void 0, 0); }
return void cb(err.code);
}
cb(void 0, stats.size);
});
};
var getFileSize = function (Env, channel, cb) {
if (!isValidId(channel)) { return void cb('INVALID_CHAN'); }
if (channel.length === 32) {
@ -311,7 +251,7 @@ var getFileSize = function (Env, channel, cb) {
}
// 'channel' refers to a file, so you need another API
getUploadSize(Env, channel, function (e, size) {
Env.blobStore.size(channel, function (e, size) {
if (typeof(size) === 'undefined') { return void cb(e); }
cb(void 0, size);
});
@ -481,15 +421,14 @@ var getTotalSize = function (Env, publicKey, cb) {
return void getChannelList(Env, publicKey, function (channels) {
if (!channels) { return done('INVALID_PIN_LIST'); } // unexpected
var count = channels.length;
if (!count) { return void done(void 0, 0); }
channels.forEach(function (channel) { // FIXME this might as well be nThen
getFileSize(Env, channel, function (e, size) {
count--;
if (!e) { bytes += size; }
if (count === 0) { return done(void 0, bytes); }
nThen(function (w) {
channels.forEach(function (channel) { // TODO semaphore?
getFileSize(Env, channel, w(function (e, size) {
if (!e) { bytes += size; }
}));
});
}).nThen(function () {
done(void 0, bytes);
});
});
});
@ -823,7 +762,12 @@ var resetUserPins = function (Env, publicKey, channelList, cb) {
pins[channel] = true;
});
var oldChannels = Object.keys(session.channels);
var oldChannels;
if (session.channels && typeof(session.channels) === 'object') {
oldChannels = Object.keys(session.channels);
} else {
oldChannels = [];
}
removePinned(Env, publicKey, oldChannels, () => {
addPinned(Env, publicKey, channelList, ()=>{});
});
@ -838,48 +782,6 @@ var resetUserPins = function (Env, publicKey, channelList, cb) {
});
};
var makeFileStream = function (root, id, cb) { // FIXME FILES
var stub = id.slice(0, 2);
var full = makeFilePath(root, id);
if (!full) {
WARN('makeFileStream', 'invalid id ' + id);
return void cb('BAD_ID');
}
Fse.mkdirp(Path.join(root, stub), function (e) {
if (e || !full) { // !full for pleasing flow, it's already checked
WARN('makeFileStream', e);
return void cb(e ? e.message : 'INTERNAL_ERROR');
}
try {
var stream = Fs.createWriteStream(full, {
flags: 'a',
encoding: 'binary',
highWaterMark: Math.pow(2, 16),
});
stream.on('open', function () {
cb(void 0, stream);
});
stream.on('error', function (e) {
WARN('stream error', e);
});
} catch (err) {
cb('BAD_STREAM');
}
});
};
var isFile = function (filePath, cb) { // FIXME FILES
/*:: if (typeof(filePath) !== 'string') { throw new Error('should never happen'); } */
Fs.stat(filePath, function (e, stats) {
if (e) {
if (e.code === 'ENOENT') { return void cb(void 0, false); }
return void cb(e.message);
}
return void cb(void 0, stats.isFile());
});
};
var clearOwnedChannel = function (Env, channelId, unsafeKey, cb) {
if (typeof(channelId) !== 'string' || channelId.length !== 32) {
return cb('INVALID_ARGUMENTS');
@ -899,74 +801,49 @@ var clearOwnedChannel = function (Env, channelId, unsafeKey, cb) {
});
};
var removeOwnedBlob = function (Env, blobId, unsafeKey, cb) { // FIXME FILES // FIXME METADATA
var safeKey = escapeKeyCharacters(unsafeKey);
var safeKeyPrefix = safeKey.slice(0,3);
var blobPrefix = blobId.slice(0,2);
var blobPath = makeFilePath(Env.paths.blob, blobId);
var ownPath = Path.join(Env.paths.blob, safeKeyPrefix, safeKey, blobPrefix, blobId);
nThen(function (w) {
// Check if the blob exists
isFile(blobPath, w(function (e, isFile) {
if (e) {
w.abort();
return void cb(e);
}
if (!isFile) {
WARN('removeOwnedBlob', 'The provided blob ID is not a file!');
w.abort();
return void cb('EINVAL_BLOBID');
}
}));
}).nThen(function (w) {
// Check if you're the owner
isFile(ownPath, w(function (e, isFile) {
if (e) {
w.abort();
return void cb(e);
}
if (!isFile) {
WARN('removeOwnedBlob', 'Incorrect owner');
w.abort();
return void cb('INSUFFICIENT_PERMISSIONS');
}
}));
}).nThen(function (w) {
// Delete the blob
/*:: if (typeof(blobPath) !== 'string') { throw new Error('should never happen'); } */
Fs.unlink(blobPath, w(function (e) { // TODO move to cold storage
Log.info('DELETION_OWNED_FILE_BY_OWNER_RPC', {
safeKey: safeKey,
blobPath: blobPath,
status: e? String(e): 'SUCCESS',
});
if (e) {
w.abort();
return void cb(e.code);
}
}));
}).nThen(function () {
// Delete the proof of ownership
Fs.unlink(ownPath, function (e) {
Log.info('DELETION_OWNED_FILE_PROOF_BY_OWNER_RPC', {
safeKey: safeKey,
proofPath: ownPath,
status: e? String(e): 'SUCCESS',
});
cb(e && e.code);
});
});
};
var removeOwnedChannel = function (Env, channelId, unsafeKey, cb) {
if (typeof(channelId) !== 'string' || !isValidId(channelId)) {
return cb('INVALID_ARGUMENTS');
}
if (testFileId(channelId)) {
return void removeOwnedBlob(Env, channelId, unsafeKey, cb);
if (Env.blobStore.isFileId(channelId)) {
var safeKey = escapeKeyCharacters(unsafeKey);
var blobId = channelId;
return void nThen(function (w) {
// check if you have permissions
Env.blobStore.isOwnedBy(safeKey, blobId, w(function (err, owned) {
if (err || !owned) {
w.abort();
return void cb("INSUFFICIENT_PERMISSIONS");
}
}));
}).nThen(function (w) {
// remove the blob
Env.blobStore.remove(blobId, w(function (err) {
Log.info('DELETION_OWNED_FILE_BY_OWNER_RPC', {
safeKey: safeKey,
blobId: blobId,
status: err? String(err): 'SUCCESS',
});
if (err) {
w.abort();
return void cb(err);
}
}));
}).nThen(function () {
// remove the proof
Env.blobStore.removeProof(safeKey, blobId, function (err) {
Log.info("DELETION_PROOF_REMOVAL_BY_OWNER_RPC", {
safeKey: safeKey,
blobId: blobId,
status: err? String(err): 'SUCCESS',
});
if (err) {
return void cb("E_PROOF_REMOVAL");
}
});
});
}
getMetadata(Env, channelId, function (err, metadata) {
@ -1015,362 +892,6 @@ var removePins = function (Env, safeKey, cb) {
});
};
var upload = function (Env, publicKey, content, cb) { // FIXME FILES
var paths = Env.paths;
var dec;
try { dec = Buffer.from(content, 'base64'); }
catch (e) { return void cb('DECODE_BUFFER'); }
var len = dec.length;
var session = getSession(Env.Sessions, publicKey);
if (typeof(session.currentUploadSize) !== 'number' ||
typeof(session.pendingUploadSize) !== 'number') {
// improperly initialized... maybe they didn't check before uploading?
// reject it, just in case
return cb('NOT_READY');
}
if (session.currentUploadSize > session.pendingUploadSize) {
return cb('E_OVER_LIMIT');
}
if (!session.blobstage) {
makeFileStream(paths.staging, publicKey, function (e, stream) {
if (!stream) { return void cb(e); }
var blobstage = session.blobstage = stream;
blobstage.write(dec);
session.currentUploadSize += len;
cb(void 0, dec.length);
});
} else {
session.blobstage.write(dec);
session.currentUploadSize += len;
cb(void 0, dec.length);
}
};
var upload_cancel = function (Env, publicKey, fileSize, cb) { // FIXME FILES
var paths = Env.paths;
var session = getSession(Env.Sessions, publicKey);
session.pendingUploadSize = fileSize;
session.currentUploadSize = 0;
if (session.blobstage) { session.blobstage.close(); }
var path = makeFilePath(paths.staging, publicKey);
if (!path) {
Log.error('UPLOAD_CANCEL_INVALID_PATH', {
staging: paths.staging,
key: publicKey,
path: path,
});
return void cb('NO_FILE');
}
Fs.unlink(path, function (e) {
if (e) { return void cb('E_UNLINK'); }
cb(void 0);
});
};
var upload_complete = function (Env, publicKey, id, cb) { // FIXME FILES
var paths = Env.paths;
var session = getSession(Env.Sessions, publicKey);
if (session.blobstage && session.blobstage.close) {
session.blobstage.close();
delete session.blobstage;
}
if (!testFileId(id)) {
WARN('uploadComplete', "id is invalid");
return void cb('EINVAL_ID');
}
var oldPath = makeFilePath(paths.staging, publicKey);
if (!oldPath) {
WARN('safeMkdir', "oldPath is null");
return void cb('RENAME_ERR');
}
var tryLocation = function (cb) {
var prefix = id.slice(0, 2);
var newPath = makeFilePath(paths.blob, id);
if (typeof(newPath) !== 'string') {
WARN('safeMkdir', "newPath is null");
return void cb('RENAME_ERR');
}
Fse.mkdirp(Path.join(paths.blob, prefix), function (e) {
if (e || !newPath) {
WARN('safeMkdir', e);
return void cb('RENAME_ERR');
}
isFile(newPath, function (e, yes) {
if (e) {
WARN('isFile', e);
return void cb(e);
}
if (yes) {
WARN('isFile', 'FILE EXISTS!');
return void cb('RENAME_ERR');
}
cb(void 0, newPath, id);
});
});
};
var handleMove = function (e, newPath, id) {
if (e || !oldPath || !newPath) {
return void cb(e || 'PATH_ERR');
}
Fse.move(oldPath, newPath, function (e) {
if (e) {
WARN('rename', e);
return void cb('RENAME_ERR');
}
cb(void 0, id);
});
};
tryLocation(handleMove);
};
/* FIXME FILES
var owned_upload_complete = function (Env, safeKey, cb) {
var session = getSession(Env.Sessions, safeKey);
// the file has already been uploaded to the staging area
// close the pending writestream
if (session.blobstage && session.blobstage.close) {
session.blobstage.close();
delete session.blobstage;
}
var oldPath = makeFilePath(Env.paths.staging, safeKey);
if (typeof(oldPath) !== 'string') {
return void cb('EINVAL_CONFIG');
}
// construct relevant paths
var root = Env.paths.staging;
//var safeKey = escapeKeyCharacters(safeKey);
var safeKeyPrefix = safeKey.slice(0, 2);
var blobId = createFileId();
var blobIdPrefix = blobId.slice(0, 2);
var plannedPath = Path.join(root, safeKeyPrefix, safeKey, blobIdPrefix);
var tries = 0;
var chooseSafeId = function (cb) {
if (tries >= 3) {
// you've already failed three times in a row
// give up and return an error
cb('E_REPEATED_FAILURE');
}
var path = Path.join(plannedPath, blobId);
Fs.access(path, Fs.constants.R_OK | Fs.constants.W_OK, function (e) {
if (!e) {
// generate a new id (with the same prefix) and recurse
blobId = blobIdPrefix + createFileId().slice(2);
return void chooseSafeId(cb);
} else if (e.code === 'ENOENT') {
// no entry, so it's safe for us to proceed
return void cb(void 0, path);
} else {
// it failed in an unexpected way. log it
// try again, but no more than a fixed number of times...
tries++;
chooseSafeId(cb);
}
});
};
// the user wants to move it into their own space
// /blob/safeKeyPrefix/safeKey/blobPrefix/blobID
var finalPath;
nThen(function (w) {
// make the requisite directory structure using Mkdirp
Mkdirp(plannedPath, w(function (e) {
if (e) { // does not throw error if the directory already existed
w.abort();
return void cb(e);
}
}));
}).nThen(function (w) {
// produce an id which confirmably does not collide with another
chooseSafeId(w(function (e, path) {
if (e) {
w.abort();
return void cb(e);
}
finalPath = path; // this is where you'll put the new file
}));
}).nThen(function (w) {
// move the existing file to its new path
// flow is dumb and I need to guard against this which will never happen
// / *:: if (typeof(oldPath) === 'object') { throw new Error('should never happen'); } * /
Fs.move(oldPath, finalPath, w(function (e) {
if (e) {
w.abort();
return void cb(e.code);
}
// otherwise it worked...
}));
}).nThen(function () {
// clean up their session when you're done
// call back with the blob id...
cb(void 0, blobId);
});
};
*/
var owned_upload_complete = function (Env, safeKey, id, cb) { // FIXME FILES
var session = getSession(Env.Sessions, safeKey);
// the file has already been uploaded to the staging area
// close the pending writestream
if (session.blobstage && session.blobstage.close) {
session.blobstage.close();
delete session.blobstage;
}
if (!testFileId(id)) {
WARN('ownedUploadComplete', "id is invalid");
return void cb('EINVAL_ID');
}
var oldPath = makeFilePath(Env.paths.staging, safeKey);
if (typeof(oldPath) !== 'string') {
return void cb('EINVAL_CONFIG');
}
// construct relevant paths
var root = Env.paths.blob;
//var safeKey = escapeKeyCharacters(safeKey);
var safeKeyPrefix = safeKey.slice(0, 3);
//var blobId = createFileId();
var blobIdPrefix = id.slice(0, 2);
var ownPath = Path.join(root, safeKeyPrefix, safeKey, blobIdPrefix);
var filePath = Path.join(root, blobIdPrefix);
var tryId = function (path, cb) {
Fs.access(path, Fs.constants.R_OK | Fs.constants.W_OK, function (e) {
if (!e) {
// generate a new id (with the same prefix) and recurse
WARN('ownedUploadComplete', 'id is already used '+ id);
return void cb('EEXISTS');
} else if (e.code === 'ENOENT') {
// no entry, so it's safe for us to proceed
return void cb();
} else {
// it failed in an unexpected way. log it
WARN('ownedUploadComplete', e);
return void cb(e.code);
}
});
};
// the user wants to move it into blob and create a empty file with the same id
// in their own space:
// /blob/safeKeyPrefix/safeKey/blobPrefix/blobID
var finalPath;
var finalOwnPath;
nThen(function (w) {
// make the requisite directory structure using Mkdirp
Fse.mkdirp(filePath, w(function (e /*, path */) {
if (e) { // does not throw error if the directory already existed
w.abort();
return void cb(e.code);
}
}));
Fse.mkdirp(ownPath, w(function (e /*, path */) {
if (e) { // does not throw error if the directory already existed
w.abort();
return void cb(e.code);
}
}));
}).nThen(function (w) {
// make sure the id does not collide with another
finalPath = Path.join(filePath, id);
finalOwnPath = Path.join(ownPath, id);
tryId(finalPath, w(function (e) {
if (e) {
w.abort();
return void cb(e);
}
}));
}).nThen(function (w) {
// Create the empty file proving ownership
Fs.writeFile(finalOwnPath, '', w(function (e) {
if (e) {
w.abort();
return void cb(e.code);
}
// otherwise it worked...
}));
}).nThen(function (w) {
// move the existing file to its new path
// flow is dumb and I need to guard against this which will never happen
/*:: if (typeof(oldPath) === 'object') { throw new Error('should never happen'); } */
Fse.move(oldPath, finalPath, w(function (e) {
if (e) {
// Remove the ownership file
Fs.unlink(finalOwnPath, function (e) {
WARN('E_UNLINK_OWN_FILE', e);
});
w.abort();
return void cb(e.code);
}
// otherwise it worked...
}));
}).nThen(function () {
// clean up their session when you're done
// call back with the blob id...
cb(void 0, id);
});
};
var upload_status = function (Env, publicKey, filesize, cb) { // FIXME FILES
var paths = Env.paths;
// validate that the provided size is actually a positive number
if (typeof(filesize) !== 'number' &&
filesize >= 0) { return void cb('E_INVALID_SIZE'); }
if (filesize >= Env.maxUploadSize) { return cb('TOO_LARGE'); }
// validate that the provided path is not junk
var filePath = makeFilePath(paths.staging, publicKey);
if (!filePath) { return void cb('E_INVALID_PATH'); }
getFreeSpace(Env, publicKey, function (e, free) {
if (e || !filePath) { return void cb(e); } // !filePath for pleasing flow
if (filesize >= free) { return cb('NOT_ENOUGH_SPACE'); }
isFile(filePath, function (e, yes) {
if (e) {
WARN('upload', e);
return cb('UNNOWN_ERROR');
}
cb(e, yes);
});
});
};
/*
We assume that the server is secured against MitM attacks
via HTTPS, and that malicious actors do not have code execution
@ -1745,25 +1266,34 @@ var isAuthenticatedCall = function (call) {
].indexOf(call) !== -1;
};
const mkEvent = function (once) {
var handlers = [];
var fired = false;
return {
reg: function (cb) {
if (once && fired) { return void setTimeout(cb); }
handlers.push(cb);
},
unreg: function (cb) {
if (handlers.indexOf(cb) === -1) { throw new Error("Not registered"); }
handlers.splice(handlers.indexOf(cb), 1);
},
fire: function () {
if (once && fired) { return; }
fired = true;
var args = Array.prototype.slice.call(arguments);
handlers.forEach(function (h) { h.apply(null, args); });
}
};
// upload_status
var upload_status = function (Env, safeKey, filesize, _cb) { // FIXME FILES
var cb = Util.once(Util.mkAsync(_cb));
// validate that the provided size is actually a positive number
if (typeof(filesize) !== 'number' &&
filesize >= 0) { return void cb('E_INVALID_SIZE'); }
if (filesize >= Env.maxUploadSize) { return cb('TOO_LARGE'); }
nThen(function (w) {
var abortAndCB = Util.both(w.abort, cb);
Env.blobStore.status(safeKey, w(function (err, inProgress) {
// if there's an error something is weird
if (err) { return void abortAndCB(err); }
// we cannot upload two things at once
if (inProgress) { return void abortAndCB(void 0, true); }
}));
}).nThen(function () {
// if yuo're here then there are no pending uploads
// check if you have space in your quota to upload something of this size
getFreeSpace(Env, safeKey, function (e, free) {
if (e) { return void cb(e); }
if (filesize >= free) { return cb('NOT_ENOUGH_SPACE'); }
cb(void 0, false);
});
});
};
/*::
@ -1823,8 +1353,6 @@ RPC.create = function (
var Sessions = Env.Sessions;
var paths = Env.paths;
var pinPath = paths.pin = keyOrDefaultString('pinPath', './pins');
var blobPath = paths.blob = keyOrDefaultString('blobPath', './blob');
var blobStagingPath = paths.staging = keyOrDefaultString('blobStagingPath', './blobstage');
paths.block = keyOrDefaultString('blockPath', './block');
paths.data = keyOrDefaultString('filePath', './datastore');
@ -2057,13 +1585,13 @@ RPC.create = function (
Respond(void 0, "OK");
});
case 'UPLOAD':
return void upload(Env, safeKey, msg[1], function (e, len) {
return void Env.blobStore.upload(safeKey, msg[1], function (e, len) {
WARN(e, len);
Respond(e, len);
});
case 'UPLOAD_STATUS':
var filesize = msg[1];
return void upload_status(Env, safeKey, msg[1], function (e, yes) {
return void upload_status(Env, safeKey, filesize, function (e, yes) {
if (!e && !yes) {
// no pending uploads, set the new size
var user = getSession(Sessions, safeKey);
@ -2073,12 +1601,12 @@ RPC.create = function (
Respond(e, yes);
});
case 'UPLOAD_COMPLETE':
return void upload_complete(Env, safeKey, msg[1], function (e, hash) {
return void Env.blobStore.complete(safeKey, msg[1], function (e, hash) {
WARN(e, hash);
Respond(e, hash);
});
case 'OWNED_UPLOAD_COMPLETE':
return void owned_upload_complete(Env, safeKey, msg[1], function (e, blobId) {
return void Env.blobStore.completeOwned(safeKey, msg[1], function (e, blobId) {
WARN(e, blobId);
Respond(e, blobId);
});
@ -2086,7 +1614,7 @@ RPC.create = function (
// msg[1] is fileSize
// if we pass it here, we can start an upload right away without calling
// UPLOAD_STATUS again
return void upload_cancel(Env, safeKey, msg[1], function (e) {
return void Env.blobStore.cancel(safeKey, msg[1], function (e) {
WARN(e, 'UPLOAD_CANCEL');
Respond(e);
});
@ -2155,21 +1683,27 @@ RPC.create = function (
loadChannelPins(Env);
Store.create({
filePath: pinPath,
}, function (s) {
Env.pinStore = s;
Fse.mkdirp(blobPath, function (e) {
if (e) { throw e; }
Fse.mkdirp(blobStagingPath, function (e) {
if (e) { throw e; }
cb(void 0, rpc);
// expire old sessions once per minute
setInterval(function () {
expireSessions(Sessions);
}, SESSION_EXPIRATION_TIME);
});
});
nThen(function (w) {
Store.create({
filePath: pinPath,
}, w(function (s) {
Env.pinStore = s;
}));
BlobStore.create({
blobPath: config.blobPath,
blobStagingPath: config.blobStagingPath,
getSession: function (safeKey) {
return getSession(Sessions, safeKey);
},
}, w(function (err, blob) {
if (err) { throw new Error(err); }
Env.blobStore = blob;
}));
}).nThen(function () {
cb(void 0, rpc);
// expire old sessions once per minute
setInterval(function () {
expireSessions(Sessions);
}, SESSION_EXPIRATION_TIME);
});
};

3
scripts/tests/roster.js
Unescape Escape View File

@ -0,0 +1,3 @@
module.exports = require("../../www/common/outer/roster.js");

191
scripts/tests/test-rpc.js
Unescape Escape View File

@ -1,12 +1,15 @@
/* globals process */
var Client = require("../../lib/client/");
var Mailbox = require("../../www/bower_components/chainpad-crypto").Mailbox;
var Crypto = require("../../www/bower_components/chainpad-crypto");
var Mailbox = Crypto.Mailbox;
var Nacl = require("tweetnacl");
var nThen = require("nthen");
var Pinpad = require("../../www/common/pinpad");
var Rpc = require("../../www/common/rpc");
var Hash = require("../../www/common/common-hash");
var CpNetflux = require("../../www/bower_components/chainpad-netflux");
var Roster = require("./roster");
var createMailbox = function (config, cb) {
var webchannel;
@ -55,6 +58,7 @@ var EMPTY_ARRAY_HASH = 'slspTLTetp6gCkw88xE5BIAbYBXllWvQGahXCx/h1gQOlE7zze4W0KRl
var createUser = function (config, cb) {
// config should contain keys for a team rpc (ed)
// teamEdKeys
// rosterHash
var user;
nThen(function (w) {
@ -85,7 +89,7 @@ var createUser = function (config, cb) {
user.anonRpc = rpc;
}));
Rpc.create(network, user.edKeys.edPrivate, user.edKeys.edPublic, w(function (err, rpc) {
Pinpad.create(network, user.edKeys, w(function (err, rpc) {
if (err) {
w.abort();
user.shutdown();
@ -95,7 +99,7 @@ var createUser = function (config, cb) {
user.rpc = rpc;
}));
Rpc.create(network, config.teamEdKeys.edPrivate, config.teamEdKeys.edPublic, w(function (err, rpc) {
Pinpad.create(network, config.teamEdKeys, w(function (err, rpc) {
if (err) {
w.abort();
user.shutdown();
@ -103,15 +107,25 @@ var createUser = function (config, cb) {
}
user.team_rpc = rpc;
}));
}).nThen(function (w) {
user.rpc.reset([], w(function (err, hash) {
if (err) {
w.abort();
user.shutdown();
return console.log("RESET_ERR");
}
if (!hash || hash !== EMPTY_ARRAY_HASH) {
throw new Error("EXPECTED EMPTY ARRAY HASH");
}
}));
}).nThen(function (w) {
// some basic sanity checks...
user.rpc.send('GET_HASH', user.edKeys.edPublic, w(function (err, hash) {
user.rpc.getServerHash(w(function (err, hash) {
if (err) {
w.abort();
return void cb(err);
}
if (!hash || hash[0] !== EMPTY_ARRAY_HASH) {
if (hash !== EMPTY_ARRAY_HASH) {
console.error("EXPECTED EMPTY ARRAY HASH");
process.exit(1);
}
@ -149,58 +163,56 @@ var createUser = function (config, cb) {
}));
}).nThen(function (w) {
// pin your mailbox
user.rpc.send('PIN', [user.mailboxChannel], w(function (err, data) {
user.rpc.pin([user.mailboxChannel], w(function (err, hash) {
if (err) {
w.abort();
return void cb(err);
}
try {
if (data[0] === EMPTY_ARRAY_HASH) { throw new Error("PIN_DIDNT_WORK"); }
user.latestPinHash = data[0];
} catch (err2) {
w.abort();
return void cb(err2);
}
console.log('PIN_RESPONSE', hash);
if (hash[0] === EMPTY_ARRAY_HASH) { throw new Error("PIN_DIDNT_WORK"); }
user.latestPinHash = hash;
}));
}).nThen(function (w) {
user.team_rpc.send('GET_HASH', config.teamEdKeys.edPublic, w(function (err, hash) {
}).nThen(function () {
/*
// XXX race condition because both users try to pin things...
user.team_rpc.getServerHash(w(function (err, hash) {
if (err) {
w.abort();
return void cb(err);
}
/*
if (!hash || hash[0] !== EMPTY_ARRAY_HASH) {
console.error("EXPECTED EMPTY ARRAY HASH");
process.exit(1);
}
}));
*/
}).nThen(function () {
// TODO check your quota usage
}).nThen(function (w) {
user.rpc.send('UNPIN', [user.mailboxChannel], w(function (err, data) {
user.rpc.unpin([user.mailboxChannel], w(function (err, hash) {
if (err) {
w.abort();
return void cb(err);
}
try {
if (data[0] !== EMPTY_ARRAY_HASH) { throw new Error("UNPIN_DIDNT_WORK"); }
user.latestPinHash = data[0];
} catch (err2) {
w.abort();
return void cb(err2);
if (hash[0] !== EMPTY_ARRAY_HASH) {
console.log('UNPIN_RESPONSE', hash);
throw new Error("UNPIN_DIDNT_WORK");
}
user.latestPinHash = hash[0];
}));
}).nThen(function (w) {
// clean up the pin list to avoid lots of accounts on the server
user.rpc.send("REMOVE_PINS", undefined, w(function (err, data) {
user.rpc.removePins(w(function (err) {
if (err) {
w.abort();
return void cb(err);
}
if (!data || data[0] !== 'OK') {
w.abort();
return void cb("REMOVE_PINS_DIDNT_WORK");
}
}));
}).nThen(function () {
user.cleanup = function (cb) {
@ -216,19 +228,102 @@ var createUser = function (config, cb) {
});
};
var alice, bob;
var alice, bob, oscar;
var sharedConfig = {
teamEdKeys: makeEdKeys(),
teamCurveKeys: makeCurveKeys(),
rosterChannel: Hash.createChannelId(),
//rosterHash: makeRosterHash(),
};
nThen(function (w) {
// oscar will be the owner of the team
createUser(sharedConfig, w(function (err, _oscar) {
if (err) {
w.abort();
return void console.log(err);
}
oscar = _oscar;
oscar.name = 'oscar';
}));
}).nThen(function (w) {
// TODO oscar creates the team roster
//Roster = Roster;
// user edPublic (for ownership)
// user curve keys (for encryption and authentication)
// roster curve keys (for encryption and decryption)
// roster signing/validate keys (ed)
nThen(function (w) {
var sharedConfig = {
teamEdKeys: makeEdKeys(),
// channel
// network
// owners:
var team = {
curve: sharedConfig.teamCurveKeys,
ed: sharedConfig.teamEdKeys,
};
Roster.create({
network: oscar.network,
channel: sharedConfig.rosterChannel,
owners: [
oscar.edKeys.edPublic
],
keys: {
myCurvePublic: oscar.curveKeys.curvePublic,
myCurvePrivate: oscar.curveKeys.curvePrivate,
teamCurvePublic: team.curve.curvePublic,
teamCurvePrivate: team.curve.curvePrivate,
teamEdPrivate: team.ed.edPrivate,
teamEdPublic: team.ed.edPublic,
},
anon_rpc: oscar.anonRpc,
lastKnownHash: void 0,
}, w(function (err, roster) {
if (err) {
w.abort();
return void console.trace(err);
}
oscar.roster = roster;
}));
}).nThen(function (w) {
var roster = oscar.roster;
roster.on('change', function () {
setTimeout(function () {
console.log("\nCHANGE");
console.log("roster.getState()", roster.getState());
console.log();
});
});
var state = roster.getState();
console.log("CURRENT ROSTER STATE:", state);
roster.init({
name: oscar.name,
//profile: '',
// mailbox: '',
//title: '',
}, w(function (err) {
if (err) { return void console.error(err); }
}));
}).nThen(function (w) {
console.log("ALICE && BOB");
createUser(sharedConfig, w(function (err, _alice) {
if (err) {
w.abort();
return void console.log(err);
}
alice = _alice;
alice.name = 'alice';
console.log("Initialized Alice");
}));
createUser(sharedConfig, w(function (err, _bob) {
if (err) {
@ -236,7 +331,39 @@ nThen(function (w) {
return void console.log(err);
}
bob = _bob;
bob.name = 'bob';
console.log("Initialized Bob");
}));
}).nThen(function () {
// TODO oscar adds alice and bob to the team as members
var roster = oscar.roster;
var data = {};
data[alice.curveKeys.curvePublic] = {
name: alice.name,
role: 'MEMBER',
};
data[bob.curveKeys.curvePublic] = {
name: bob.name,
role: 'MEMBER',
};
roster.add(data, function (err) {
if (err) {
return void console.error(err);
}
console.log("SENT ADD COMMAND");
});
}).nThen(function () {
// TODO alice and bob describe themselves...
}).nThen(function () {
// TODO Oscar promotes Alice to 'ADMIN'
}).nThen(function () {
}).nThen(function (w) {
var message = alice.mailbox.encrypt(JSON.stringify({
type: "CHEESE",

2
server.js
Unescape Escape View File

@ -62,6 +62,7 @@ if (process.env.PACKAGE) {
config.flushCache = function () {
FRESH_KEY = +new Date();
if (!(DEV_MODE || FRESH_MODE)) { FRESH_MODE = true; }
if (!config.log) { return; }
config.log.info("UPDATING_FRESH_KEY", FRESH_KEY);
};
@ -118,7 +119,6 @@ app.head(/^\/common\/feedback\.html/, function (req, res, next) {
app.use(function (req, res, next) {
if (req.method === 'OPTIONS' && /\/blob\//.test(req.url)) {
console.log(req.url);
res.setHeader('Access-Control-Allow-Origin', '*');
res.setHeader('Access-Control-Allow-Methods', 'GET, OPTIONS');
res.setHeader('Access-Control-Allow-Headers', 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range');

413
storage/blob.js
Unescape Escape View File

@ -0,0 +1,413 @@
/* globals Buffer */
var Fs = require("fs");
var Fse = require("fs-extra");
var Path = require("path");
var BlobStore = module.exports;
var nThen = require("nthen");
var Util = require("../lib/common-util");
var isValidSafeKey = function (safeKey) {
return typeof(safeKey) === 'string' && !/\//.test(safeKey) && safeKey.length === 44;
};
var isValidId = function (id) {
return typeof(id) === 'string' && id.length === 48 && !/[^a-f0-9]/.test(id);
};
// helpers
// /blob/<safeKeyPrefix>/<safeKey>/<blobPrefix>
var makePathToBlob = function (Env, blobId) {
return Path.join(Env.blobPath, blobId.slice(0, 2));
};
// /blob/<safeKeyPrefix>/<safeKey>/<blobPrefix>/<blobId>
var makeBlobPath = function (Env, blobId) {
return Path.join(makePathToBlob(Env, blobId), blobId);
};
// /blobstate/<safeKeyPrefix>
var makePathToStage = function (Env, safeKey) {
return Path.join(Env.blobStagingPath, safeKey.slice(0, 2));
};
// /blobstate/<safeKeyPrefix>/<safeKey>
var makeStagePath = function (Env, safeKey) {
return Path.join(makePathToStage(Env, safeKey), safeKey);
};
// /blob/<safeKeyPrefix>/<safeKey>/<blobPrefix>
var makePathToProof = function (Env, safeKey, blobId) {
return Path.join(Env.blobPath, safeKey.slice(0, 3), safeKey, blobId.slice(0, 2), blobId);
};
// /blob/<safeKeyPrefix>/<safeKey>/<blobPrefix>/<blobId>
var makeProofPath = function (Env, safeKey, blobId) {
return Path.join(makePathToProof(Env, safeKey, blobId), blobId);
};
// getUploadSize: used by
// getFileSize
var getUploadSize = function (Env, blobId, cb) {
var path = makeBlobPath(Env, blobId);
if (!path) { return cb('INVALID_UPLOAD_ID'); }
Fs.stat(path, function (err, stats) {
if (err) {
// if a file was deleted, its size is 0 bytes
if (err.code === 'ENOENT') { return cb(void 0, 0); }
return void cb(err.code);
}
cb(void 0, stats.size);
});
};
// isFile: used by
// removeOwnedBlob
// uploadComplete
// uploadStatus
var isFile = function (filePath, cb) {
Fs.stat(filePath, function (e, stats) {
if (e) {
if (e.code === 'ENOENT') { return void cb(void 0, false); }
return void cb(e.message);
}
return void cb(void 0, stats.isFile());
});
};
var makeFileStream = function (dir, full, cb) {
Fse.mkdirp(dir, function (e) {
if (e || !full) { // !full for pleasing flow, it's already checked
//WARN('makeFileStream', e);
return void cb(e ? e.message : 'INTERNAL_ERROR');
}
try {
var stream = Fs.createWriteStream(full, {
flags: 'a',
encoding: 'binary',
highWaterMark: Math.pow(2, 16),
});
stream.on('open', function () {
cb(void 0, stream);
});
stream.on('error', function (/* e */) {
//console.error("MAKE_FILE_STREAM", full);
// XXX ERROR
//WARN('stream error', e);
});
} catch (err) {
cb('BAD_STREAM');
}
});
};
/********** METHODS **************/
var upload = function (Env, safeKey, content, cb) {
var dec;
try { dec = Buffer.from(content, 'base64'); }
catch (e) { return void cb('DECODE_BUFFER'); }
var len = dec.length;
var session = Env.getSession(safeKey);
if (typeof(session.currentUploadSize) !== 'number' ||
typeof(session.pendingUploadSize) !== 'number') {
// improperly initialized... maybe they didn't check before uploading?
// reject it, just in case
return cb('NOT_READY');
}
if (session.currentUploadSize > session.pendingUploadSize) {
return cb('E_OVER_LIMIT');
}
if (!session.blobstage) {
makeFileStream(makePathToStage(Env, safeKey), makeStagePath(Env, safeKey), function (e, stream) {
if (!stream) { return void cb(e); }
var blobstage = session.blobstage = stream;
blobstage.write(dec);
session.currentUploadSize += len;
cb(void 0, dec.length);
});
} else {
session.blobstage.write(dec);
session.currentUploadSize += len;
cb(void 0, dec.length);
}
};
// upload_cancel
var upload_cancel = function (Env, safeKey, fileSize, cb) {
var session = Env.getSession(safeKey);
session.pendingUploadSize = fileSize;
session.currentUploadSize = 0;
if (session.blobstage) { session.blobstage.close(); }
var path = makeStagePath(Env, safeKey);
Fs.unlink(path, function (e) {
if (e) { return void cb('E_UNLINK'); }
cb(void 0);
});
};
// upload_complete
var upload_complete = function (Env, safeKey, id, cb) {
var session = Env.getSession(safeKey);
if (session.blobstage && session.blobstage.close) {
session.blobstage.close();
delete session.blobstage;
}
var oldPath = makeStagePath(Env, safeKey);
var newPath = makeBlobPath(Env, id);
nThen(function (w) {
// make sure the path to your final location exists
Fse.mkdirp(makePathToBlob(Env, id), function (e) {
if (e) {
w.abort();
return void cb('RENAME_ERR');
}
});
}).nThen(function (w) {
// make sure there's not already something in that exact location
isFile(newPath, function (e, yes) {
if (e) {
w.abort();
return void cb(e);
}
if (yes) {
w.abort();
return void cb('RENAME_ERR');
}
cb(void 0, newPath, id);
});
}).nThen(function () {
// finally, move the old file to the new path
// FIXME we could just move and handle the EEXISTS instead of the above block
Fse.move(oldPath, newPath, function (e) {
if (e) { return void cb('RENAME_ERR'); }
cb(void 0, id);
});
});
};
var tryId = function (path, cb) {
Fs.access(path, Fs.constants.R_OK | Fs.constants.W_OK, function (e) {
if (!e) {
// generate a new id (with the same prefix) and recurse
//WARN('ownedUploadComplete', 'id is already used '+ id);
return void cb('EEXISTS');
} else if (e.code === 'ENOENT') {
// no entry, so it's safe for us to proceed
return void cb();
} else {
// it failed in an unexpected way. log it
//WARN('ownedUploadComplete', e);
return void cb(e.code);
}
});
};
// owned_upload_complete
var owned_upload_complete = function (Env, safeKey, id, cb) {
var session = Env.getSession(safeKey);
// the file has already been uploaded to the staging area
// close the pending writestream
if (session.blobstage && session.blobstage.close) {
session.blobstage.close();
delete session.blobstage;
}
if (!isValidId(id)) {
//WARN('ownedUploadComplete', "id is invalid");
return void cb('EINVAL_ID');
}
var oldPath = makeStagePath(Env, safeKey);
if (typeof(oldPath) !== 'string') {
return void cb('EINVAL_CONFIG');
}
var ownPath = makePathToProof(Env, safeKey, id);
var filePath = makePathToBlob(Env, id);
var finalPath = makeBlobPath(Env, id);
var finalOwnPath = makeProofPath(Env, safeKey, id);
// the user wants to move it into blob and create a empty file with the same id
// in their own space:
// /blob/safeKeyPrefix/safeKey/blobPrefix/blobID
nThen(function (w) {
// make the requisite directory structure using Mkdirp
Fse.mkdirp(filePath, w(function (e /*, path */) {
if (e) { // does not throw error if the directory already existed
w.abort();
return void cb(e.code);
}
}));
Fse.mkdirp(ownPath, w(function (e /*, path */) {
if (e) { // does not throw error if the directory already existed
w.abort();
return void cb(e.code);
}
}));
}).nThen(function (w) {
// make sure the id does not collide with another
tryId(finalPath, w(function (e) {
if (e) {
w.abort();
return void cb(e);
}
}));
}).nThen(function (w) {
// Create the empty file proving ownership
Fs.writeFile(finalOwnPath, '', w(function (e) {
if (e) {
w.abort();
return void cb(e.code);
}
// otherwise it worked...
}));
}).nThen(function (w) {
// move the existing file to its new path
Fse.move(oldPath, finalPath, w(function (e) {
if (e) {
// if there's an error putting the file into its final location...
// ... you should remove the ownership file
Fs.unlink(finalOwnPath, function () {
// but if you can't, it's not catestrophic
// we can clean it up later
});
w.abort();
return void cb(e.code);
}
// otherwise it worked...
}));
}).nThen(function () {
// clean up their session when you're done
// call back with the blob id...
cb(void 0, id);
});
};
// removeBlob
var remove = function (Env, blobId, cb) {
var blobPath = makeBlobPath(Env, blobId);
Fs.unlink(blobPath, cb); // TODO COLDSTORAGE
};
// removeProof
var removeProof = function (Env, safeKey, blobId, cb) {
var proofPath = makeProofPath(Env, safeKey, blobId);
Fs.unlink(proofPath, cb);
};
// isOwnedBy(id, safeKey)
var isOwnedBy = function (Env, safeKey, blobId, cb) {
var proofPath = makeProofPath(Env, safeKey, blobId);
isFile(proofPath, cb);
};
var listFiles = function (Env, handler, cb) {
cb("NOT_IMPLEMENTED");
};
BlobStore.create = function (config, _cb) {
var cb = Util.once(Util.mkAsync(_cb));
if (typeof(config.getSession) !== 'function') {
return void cb("getSession method required");
}
var Env = {
blobPath: config.blobPath || './blob',
blobStagingPath: config.blobStagingPath || './blobstage',
getSession: config.getSession,
};
nThen(function (w) {
var CB = Util.both(w.abort, cb);
Fse.mkdirp(Env.blobPath, w(function (e) {
if (e) { CB(e); }
}));
Fse.mkdirp(Env.blobStagingPath, w(function (e) {
if (e) { CB(e); }
}));
}).nThen(function () {
cb(void 0, {
isFileId: isValidId,
status: function (safeKey, _cb) {
// TODO check if the final destination is a file
// because otherwise two people can try to upload to the same location
// and one will fail, invalidating their hard work
var cb = Util.once(Util.mkAsync(_cb));
if (!isValidSafeKey(safeKey)) { return void cb('INVALID_SAFEKEY'); }
isFile(makeStagePath(Env, safeKey), cb);
},
upload: function (safeKey, content, _cb) {
var cb = Util.once(Util.mkAsync(_cb));
if (!isValidSafeKey(safeKey)) { return void cb('INVALID_SAFEKEY'); }
upload(Env, safeKey, content, Util.once(Util.mkAsync(cb)));
},
cancel: function (safeKey, fileSize, _cb) {
var cb = Util.once(Util.mkAsync(_cb));
if (!isValidSafeKey(safeKey)) { return void cb('INVALID_SAFEKEY'); }
if (typeof(fileSize) !== 'number' || isNaN(fileSize) || fileSize <= 0) { return void cb("INVALID_FILESIZE"); }
upload_cancel(Env, safeKey, fileSize, cb);
},
isOwnedBy: function (safeKey, blobId, _cb) {
var cb = Util.once(Util.mkAsync(_cb));
if (!isValidSafeKey(safeKey)) { return void cb('INVALID_SAFEKEY'); }
isOwnedBy(Env, safeKey, blobId, cb);
},
remove: function (blobId, _cb) {
var cb = Util.once(Util.mkAsync(_cb));
if (!isValidId(blobId)) { return void cb("INVALID_ID"); }
remove(Env, blobId, cb);
},
removeProof: function (safeKey, blobId, _cb) {
var cb = Util.once(Util.mkAsync(_cb));
if (!isValidSafeKey(safeKey)) { return void cb('INVALID_SAFEKEY'); }
if (!isValidId(blobId)) { return void cb("INVALID_ID"); }
removeProof(Env, safeKey, blobId, cb);
},
complete: function (safeKey, id, _cb) {
var cb = Util.once(Util.mkAsync(_cb));
if (!isValidSafeKey(safeKey)) { return void cb('INVALID_SAFEKEY'); }
if (!isValidId(id)) { return void cb("INVALID_ID"); }
upload_complete(Env, safeKey, id, cb);
},
completeOwned: function (safeKey, id, _cb) {
var cb = Util.once(Util.mkAsync(_cb));
if (!isValidSafeKey(safeKey)) { return void cb('INVALID_SAFEKEY'); }
if (!isValidId(id)) { return void cb("INVALID_ID"); }
owned_upload_complete(Env, safeKey, id, cb);
},
size: function (id, _cb) {
var cb = Util.once(Util.mkAsync(_cb));
if (!isValidId(id)) { return void cb("INVALID_ID"); }
getUploadSize(Env, id, cb);
},
list: function (handler, _cb) {
var cb = Util.once(Util.mkAsync(_cb));
listFiles(Env, handler, cb);
},
});
});
};

108
storage/file.js
Unescape Escape View File

@ -6,7 +6,8 @@ var Fse = require("fs-extra");
var Path = require("path");
var nThen = require("nthen");
var Semaphore = require("saferphore");
var Once = require("../lib/once");
var Util = require("../lib/common-util");
const Readline = require("readline");
const ToPull = require('stream-to-pull-stream');
const Pull = require('pull-stream');
@ -52,37 +53,50 @@ var channelExists = function (filepath, cb) {
};
// reads classic metadata from a channel log and aborts
var getMetadataAtPath = function (Env, path, cb) {
var remainder = '';
var stream = Fs.createReadStream(path, { encoding: 'utf8' });
var complete = function (err, data) {
var _cb = cb;
cb = undefined;
if (_cb) { _cb(err, data); }
};
stream.on('data', function (chunk) {
if (!/\n/.test(chunk)) {
remainder += chunk;
return;
}
stream.close();
var metadata = chunk.split('\n')[0];
// returns undefined if the first message was not an object (not an array)
var getMetadataAtPath = function (Env, path, _cb) {
var stream;
var parsed = null;
// cb implicitly destroys the stream, if it exists
// and calls back asynchronously no more than once
var cb = Util.once(Util.both(function () {
try {
parsed = JSON.parse(metadata);
complete(undefined, parsed);
stream.destroy();
} catch (err) {
return err;
}
catch (e) {
console.log("getMetadataAtPath");
console.error(e);
complete('INVALID_METADATA', metadata);
}
});
stream.on('end', function () {
complete();
}, Util.mkAsync(_cb)));
// stream creation emit errors... probably ENOENT
stream = Fs.createReadStream(path, { encoding: 'utf8' }).on('error', cb);
// stream lines
const rl = Readline.createInterface({
input: stream,
});
stream.on('error', function (e) { complete(e); });
var i = 0;
rl
.on('line', function (line) {
// metadata should always be on the first line or not exist in the channel at all
if (i++ > 0) { return void cb(); }
var metadata;
try {
metadata = JSON.parse(line);
// if it parses, is a truthy object, and is not an array
// then it's what you were looking for
if (metadata && typeof(metadata) === 'object' && !Array.isArray(metadata)) {
return void cb(void 0, metadata);
} else { // it parsed, but isn't metadata
return void cb(); // call back without an error or metadata
}
} catch (err) {
// if you can't parse, that's bad
return void cb("INVALID_METADATA");
}
})
.on('end', cb)
.on('error', cb);
};
var closeChannel = function (env, channelName, cb) {
@ -98,18 +112,14 @@ var closeChannel = function (env, channelName, cb) {
};
// truncates a file to the end of its metadata line
var clearChannel = function (env, channelId, cb) {
// TODO write the metadata in a dedicated file
var clearChannel = function (env, channelId, _cb) {
var cb = Util.once(Util.mkAsync(_cb));
var path = mkPath(env, channelId);
getMetadataAtPath(env, path, function (e, metadata) {
if (e) { return cb(new Error(e)); }
if (!metadata) {
return void Fs.truncate(path, 0, function (err) {
if (err) {
return cb(err);
}
cb(void 0);
});
}
if (!metadata) { return void Fs.truncate(path, 0, cb); }
var len = JSON.stringify(metadata).length + 1;
@ -214,7 +224,7 @@ How to proceed
// 'INVALID_METADATA' if it can't parse
// stream errors if anything goes wrong at a lower level
// ENOENT (no channel here)
return void handler(err);
return void handler(err, data);
}
// disregard anything that isn't a map
if (!data || typeof(data) !== 'object' || Array.isArray(data)) { return; }
@ -347,7 +357,7 @@ var removeChannel = function (env, channelName, cb) {
var channelPath = mkPath(env, channelName);
var metadataPath = mkMetadataPath(env, channelName);
var CB = Once(cb);
var CB = Util.once(cb);
var errors = 0;
nThen(function (w) {
@ -387,7 +397,7 @@ var removeArchivedChannel = function (env, channelName, cb) {
var channelPath = mkArchivePath(env, channelName);
var metadataPath = mkArchiveMetadataPath(env, channelName);
var CB = Once(cb);
var CB = Util.once(cb);
nThen(function (w) {
Fs.unlink(channelPath, w(function (err) {
@ -602,7 +612,7 @@ var unarchiveChannel = function (env, channelName, cb) {
var metadataPath = mkMetadataPath(env, channelName);
// don't call the callback multiple times
var CB = Once(cb);
var CB = Util.once(cb);
// if a file exists in the unarchived path, you probably don't want to clobber its data
// so unlike 'archiveChannel' we won't overwrite.
@ -690,7 +700,7 @@ var channelBytes = function (env, chanName, cb) {
var channelPath = mkPath(env, chanName);
var dataPath = mkMetadataPath(env, chanName);
var CB = Once(cb);
var CB = Util.once(cb);
var channelSize = 0;
var dataSize = 0;
@ -806,23 +816,13 @@ var getChannel = function (
// write a message to the disk as raw bytes
const messageBin = (env, chanName, msgBin, cb) => {
var complete = Util.once(cb);
getChannel(env, chanName, function (err, chan) {
if (!chan) {
cb(err);
return;
}
let called = false;
var complete = function (err) {
if (called) { return; }
called = true;
cb(err);
};
if (!chan) { return void complete(err); }
chan.onError.push(complete);
chan.writeStream.write(msgBin, function () {
/*::if (!chan) { throw new Error("Flow unreachable"); }*/
chan.onError.splice(chan.onError.indexOf(complete), 1);
chan.atime = +new Date();
if (!cb) { return; }
complete();
});
});

30
www/common/common-util.js
Unescape Escape View File

@ -1,6 +1,26 @@
(function (window) {
var Util = {};
// polyfill for atob in case you're using this from node...
window.atob = window.atob || function (str) { return Buffer.from(str, 'base64').toString('binary'); }; // jshint ignore:line
window.btoa = window.btoa || function (str) { return new Buffer(str, 'binary').toString('base64'); }; // jshint ignore:line
Util.bake = function (f, args) {
if (typeof(args) === 'undefined') { args = []; }
if (!Array.isArray(args)) { args = [args]; }
return function () {
return f.apply(null, args);
};
};
Util.both = function (pre, post) {
if (typeof(post) !== 'function') { post = function (x) { return x; }; }
return function () {
pre.apply(null, arguments);
return post.apply(null, arguments);
};
};
Util.tryParse = function (s) {
try { return JSON.parse(s); } catch (e) { return;}
};
@ -70,7 +90,7 @@
Util.base64ToHex = function (b64String) {
var hexArray = [];
atob(b64String.replace(/-/g, '/')).split("").forEach(function(e){
window.atob(b64String.replace(/-/g, '/')).split("").forEach(function(e){
var h = e.charCodeAt(0).toString(16);
if (h.length === 1) { h = "0"+h; }
hexArray.push(h);
@ -110,6 +130,14 @@
return C;
};
Util.escapeKeyCharacters = function (key) {
return key && key.replace && key.replace(/\//g, '-');
};
Util.unescapeKeyCharacters = function (key) {
return key.replace(/\-/g, '/');
};
Util.deduplicateString = function (array) {
var a = array.slice();
for(var i=0; i<a.length; i++) {

488
www/common/outer/roster.js
Unescape Escape View File

@ -0,0 +1,488 @@
(function () {
var factory = function (Util, Hash, CPNetflux, Sortify, nThen, Crypto) {
var Roster = {};
/*
roster: {
state: {
user0CurveKey: {
role: "OWNER|ADMIN|MEMBER",
profile: "",
mailbox: "",
name: "",
title: ""
},
user1CurveKey: {
...
}
},
metadata: {
}
}
*/
var canCheckpoint = function (author, state) {
// if you're here then you've received a checkpoint message
// that you don't necessarily trust.
// find the author's role from your knoweldge of the state
var role = Util.find(state, ['author', 'role']);
// and check if it is 'OWNER' or 'ADMIN'
return ['OWNER', 'ADMIN'].indexOf(role) !== -1;
};
var isValidRole = function (role) {
return ['OWNER', 'ADMIN', 'MEMBER'].indexOf(role) !== -1;
};
var canAddRole = function (author, role, state) {
var authorRole = Util.find(state, [author, 'role']);
if (!authorRole) { return false; }
// nobody can add an invalid role
if (!isValidRole(role)) { return false; }
// owners can add any valid role they want
if (authorRole === 'OWNER') { return true; }
// admins can add other admins or members
if (authorRole === "ADMIN") { return ['ADMIN', 'MEMBER'].indexOf(role) !== -1; }
// (MEMBER, other) can't add anyone of any role
return false;
};
var isValidId = function (id) {
return typeof(id) === 'string' && id.length === 44;
};
var canDescribeTarget = function (author, curve, state) {
// you must be in the group to describe anyone
if (!state[curve]) { return false; }
// anyone can describe themself
if (author === curve && state[curve]) { return true; }
var authorRole = Util.find(state, [author, 'role']);
var targetRole = Util.find(state, [curve, 'role']);
// something is really wrong if there's no authorRole
if (!authorRole) { return false; }
// owners can do whatever they want
if (authorRole === 'OWNER') { return true; }
// admins can describe anyone escept owners
if (authorRole === 'ADMIN' && targetRole !== 'OWNER') { return true; }
// members can't describe others
return false;
};
var canRemoveRole = function (author, role, state) {
var authorRole = Util.find(state, [author, 'role']);
if (!authorRole) { return false; }
// owners can remove anyone they want
if (authorRole === 'OWNER') { return true; }
// admins can remove other admins or members
if (authorRole === "ADMIN") { return ["ADMIN", "MEMBER"].indexOf(role) !== -1; }
// MEMBERS and non-members cannot remove anyone of any role
return false;
};
var shouldCheckpoint = function (state) {
//
state = state;
};
shouldCheckpoint = shouldCheckpoint; // XXX lint
var commands = Roster.commands = {};
/* Commands are functions with the signature
(args_any, base46_author_string, roster_map, optional_base64_message_id) => boolean
they:
* throw if any of their arguments are invalid
* return true if their application to previous state results in a change
* mutate the local account of the current state
changes to the state can be simulated locally before being sent.
if the simulation throws or returns false, don't send.
*/
// the author is trying to add someone to the roster
// owners can add any role
commands.ADD = function (args, author, roster) {
if (!(args && typeof(args) === 'object' && !Array.isArray(args))) {
throw new Error("INVALID ARGS");
}
if (typeof(roster.state) === 'undefined') {
throw new Error("CANNOT_ADD_TO_UNITIALIZED_ROSTER");
}
var changed = false;
Object.keys(args).forEach(function (curve) {
// FIXME only allow valid curve keys, anything else is pollution
if (curve.length !== 44) {
console.log(curve, curve.length);
throw new Error("INVALID_CURVE_KEY");
}
var data = args[curve];
// ignore anything that isn't a proper object
if (!data || typeof(data) !== 'object' || Array.isArray(data)) {
return;
}
// ignore instructions to ADD someone who is already in the roster
if (roster.state[curve]) { return; }
if (!canAddRole(author, data.role, roster.state)) { return; }
// this will result in a change
changed = true;
roster.state[curve] = data;
});
return changed;
};
commands.RM = function (args, author, roster) {
if (!Array.isArray(args)) { throw new Error("INVALID_ARGS"); }
if (typeof(roster.state) === 'undefined') {
throw new Error("CANNOT_RM_FROM_UNITIALIZED_ROSTER");
}
var changed = false;
args.forEach(function (curve) {
if (isValidId(curve)) { throw new Error("INVALID_CURVE_KEY"); }
// don't try to remove something that isn't there
if (!roster.state[curve]) { return; }
var role = roster.state[curve].role;
if (!canRemoveRole(author, role, roster.state)) { return; }
changed = true;
delete roster.state[curve];
});
return changed;
};
commands.DESCRIBE = function (args, author, roster) {
if (!args || typeof(args) !== 'object' || Array.isArray(args)) {
throw new Error("INVALID_ARGUMENTS");
}
if (typeof(roster.state) === 'undefined') {
throw new Error("CANNOT_DESCRIBE_MEMBERS_OF_UNITIALIZED_ROSTER");
}
var changed = false;
Object.keys(args).forEach(function (curve) {
if (!isValidId(curve)) { return; }
if (!roster.state[curve]) { return; }
if (!canDescribeTarget(author, curve, roster.state)) { return; }
var data = args[curve];
if (!data || typeof(data) !== 'object' || Array.isArray(data)) { return; }
var current = roster.state[curve];
Object.keys(data).forEach(function (key) {
if (current[key] === data[key]) { return; }
changed = true;
current[key] = data[key];
});
});
return changed;
/*
args: {
userkey: {
field: newValue
},
}
*/
// owners can update information about any team member
// admins can update information about members
// members can update information about themselves
// non-members cannot update anything
//roster = roster;
};
// XXX what about concurrent checkpoints? Let's solve for race conditions...
commands.CHECKPOINT = function (args, author, roster) {
// args: complete state
// args should be a map
if (!(args && typeof(args) === 'object' && !Array.isArray(args))) { throw new Error("INVALID_CHECKPOINT_STATE"); }
if (typeof(roster.state) === 'undefined') {
// either you're connecting from the beginning of the log
// or from a trusted lastKnownHash.
// Either way, initialize the roster state
roster.state = args;
return true;
} else if (Sortify(args) !== Sortify(roster.state)) {
// a checkpoint must reinsert the previous state
throw new Error("CHECKPOINT_DOES_NOT_MATCH_PREVIOUS_STATE");
}
// otherwise, you're iterating over the log from a previous checkpoint
// so you should know everyone's role
// owners and admins can checkpoint. members and non-members cannot
if (!canCheckpoint(author, roster)) { return false; }
// set the state, and indicate that a change was made
roster.state = args;
return true;
};
var handleCommand = function (content, author, roster) {
if (!(Array.isArray(content) && typeof(author) === 'string')) {
throw new Error("INVALID ARGUMENTS");
}
var command = content[0];
if (typeof(commands[command]) !== 'function') { throw new Error('INVALID_COMMAND'); }
return commands[command](content[1], author, roster);
};
var clone = function (o) {
return JSON.parse(JSON.stringify(o));
};
var simulate = function (content, author, roster) {
return handleCommand(content, author, clone(roster));
};
Roster.create = function (config, _cb) {
if (typeof(_cb) !== 'function') { throw new Error("EXPECTED_CALLBACK"); }
var cb = Util.once(Util.mkAsync(_cb));
if (!config.network) { return void cb("EXPECTED_NETWORK"); }
if (!config.channel || typeof(config.channel) !== 'string' || config.channel.length !== 32) { return void cb("EXPECTED_CHANNEL"); }
if (!config.owners || !Array.isArray(config.owners)) { return void cb("EXPECTED_OWNERS"); }
if (!config.keys || typeof(config.keys) !== 'object') { return void cb("EXPECTED_CRYPTO_KEYS"); }
if (!config.anon_rpc) { return void cb("EXPECTED_ANON_RPC"); }
var anon_rpc = config.anon_rpc;
var keys = config.keys;
var me = keys.myCurvePublic;
var channel = config.channel;
var ref = {};
var roster = {};
var events = {
change: Util.mkEvent(),
};
roster.on = function (key, handler) {
if (typeof(events[key]) !== 'object') { throw new Error("unsupported event"); }
events[key].reg(handler);
};
roster.off = function (key, handler) {
if (typeof(events[key]) !== 'object') { throw new Error("unsupported event"); }
events[key].unreg(handler);
};
roster.getState = function () {
return ref.state;
};
var ready = false;
var onReady = function (/* info */) {
ready = true;
cb(void 0, roster);
};
// onError (deleted or expired)
// you won't be able to connect
// onMetadataUpdate
// update owners?
// deleted while you are open
// emit an event
var onChannelError = function (info) {
if (!ready) { return void cb(info); } // XXX make sure we don't reconnect
console.error("CHANNEL_ERROR", info);
};
var onConnect = function (/* wc, sendMessage */) {
console.log("ROSTER CONNECTED");
};
var onMessage = function (msg, user, vKey, isCp , hash, author) {
//console.log("onMessage");
//console.log(typeof(msg), msg);
var parsed = Util.tryParse(msg);
if (!parsed) { return void console.error("could not parse"); }
var changed;
try {
changed = handleCommand(parsed, author, ref);
} catch (err) {
console.error(err);
}
if (changed) { events.change.fire(); }
return void console.log(msg);
};
var isReady = function () {
return Boolean(ready && me);
};
var metadata, crypto;
var send = function (msg, cb) {
if (!isReady()) { return void cb("NOT_READY"); }
var changed = false;
try {
// simulate the command before you send it
changed = simulate(msg, keys.myCurvePublic, ref);
} catch (err) {
return void cb(err);
}
if (!changed) { return void cb("NO_CHANGE"); }
var ciphertext = crypto.encrypt(Sortify(msg));
anon_rpc.send('WRITE_PRIVATE_MESSAGE', [
channel,
ciphertext
], function (err) {
if (err) { return void cb(err); }
cb();
});
};
roster.init = function (_data, cb) {
var data = clone(_data);
data.role = 'OWNER';
var state = {};
state[me] = data;
send([ 'CHECKPOINT', state ], cb);
};
// commands
roster.checkpoint = function () {
send([ 'CHECKPOINT', ref.state], cb);
};
roster.add = function (data, cb) {
send([ 'ADD', data ], cb);
};
roster.remove = function (data, cb) {
send([ 'REMOVE', data ], cb);
};
roster.describe = function (data, cb) {
send(['DESCRIBE', data], cb);
};
nThen(function (w) {
// get metadata so we know the owners and validateKey
anon_rpc.send('GET_METADATA', channel, function (err, data) {
if (err) {
w.abort();
return void console.error(err);
}
metadata = ref.metadata = (data && data[0]) || undefined;
console.log("TEAM_METADATA", metadata);
});
}).nThen(function (w) {
if (!config.keys.teamEdPublic && metadata && metadata.validateKey) {
config.keys.teamEdPublic = metadata.validateKey;
}
try {
crypto = Crypto.Team.createEncryptor(config.keys);
} catch (err) {
w.abort();
return void cb(err);
}
}).nThen(function () {
CPNetflux.start({
// if you don't have a lastKnownHash you will need the full history
// passing -1 forces the server to send all messages, otherwise
// malicious users with the signing key could send cp| messages
// and fool new users into initializing their session incorrectly
lastKnownHash: config.lastKnownHash || -1,
network: config.network,
channel: config.channel,
crypto: crypto,
validateKey: config.keys.teamEdPublic,
owners: config.owners,
onChannelError: onChannelError,
onReady: onReady,
onConnect: onConnect,
onConnectionChange: function () {},
onMessage: onMessage,
noChainPad: true,
});
});
};
return Roster;
};
if (typeof(module) !== 'undefined' && module.exports) {
module.exports = factory(
require("../common-util"),
require("../common-hash"),
require("../../bower_components/chainpad-netflux/chainpad-netflux.js"),
require("../../bower_components/json.sortify"),
require("nthen"),
require("../../bower_components/chainpad-crypto/crypto")
);
} else if ((typeof(define) !== 'undefined' && define !== null) && (define.amd !== null)) {
define([
'/common/common-util.js',
'/common/common-hash.js',
'/bower_components/chainpad-netflux/chainpad-netflux.js',
'/bower_compoents/json.sortify/dist/JSON.sortify.js',
'/bower_components/nthen/index.js',
'/bower_components/chainpad-crypto/crypto.js'
//'/bower_components/tweetnacl/nacl-fast.min.js',
], function (Util, Hash, CPNF, Sortify, nThen, Crypto) {
return factory.apply(null, [
Util,
Hash,
CPNF,
Sortify,
nThen,
Crypto
]);
});
} else {
// I'm not gonna bother supporting any other kind of instanciation
}
}());

41
www/common/pinpad.js
Unescape Escape View File

@ -1,29 +1,16 @@
define([
'/common/rpc.js',
], function (Rpc) {
var create = function (network, proxy, cb) {
if (!network) {
setTimeout(function () {
cb('INVALID_NETWORK');
});
return;
}
if (!proxy) {
setTimeout(function () {
cb('INVALID_PROXY');
});
return;
}
(function () {
var factory = function (Util, Rpc) {
var create = function (network, proxy, _cb) {
if (typeof(_cb) !== 'function') { throw new Error("Expected callback"); }
var cb = Util.once(Util.mkAsync(_cb));
if (!network) { return void cb('INVALID_NETWORK'); }
if (!proxy) { return void cb('INVALID_PROXY'); }
var edPrivate = proxy.edPrivate;
var edPublic = proxy.edPublic;
if (!(edPrivate && edPublic)) {
setTimeout(function () {
cb('INVALID_KEYS');
});
return;
}
if (!(edPrivate && edPublic)) { return void cb('INVALID_KEYS'); }
Rpc.create(network, edPrivate, edPublic, function (e, rpc) {
if (e) { return void cb(e); }
@ -76,7 +63,7 @@ define([
if (!(hash && hash[0])) {
return void cb('NO_HASH_RETURNED');
}
cb(e, hash[0]);
cb(e, Array.isArray(hash) && hash[0] || undefined);
});
};
@ -275,4 +262,10 @@ define([
};
return { create: create };
});
};
if (typeof(module) !== 'undefined' && module.exports) {
module.exports = factory(require('./common-util'), require("./rpc"));
} else if ((typeof(define) !== 'undefined' && define !== null) && (define.amd !== null)) {
define([ '/common/common-util.js', '/common/rpc.js', ], function (Util, Rpc) { return factory(Util, Rpc); });
}
}());

Write Preview
Loading…
Cancel
Save