744be9929f
handle stream errors when uploading
5 years ago
8326b8b270
disable automatic blob expiration since we can't quite rely on atime as an inactivity metric
5 years ago
f2db93834b
implement blob archival and WIP blob iteration methods
5 years ago
d1e1a11e38
fix path issues in blobStore
5 years ago
cb6b4a8eb3
always supply a callback when writing messages
5 years ago
c7f70d7264
implement blob management methods in a dedicated file
5 years ago
7cc1832a1f
handle stream errors when checking for metadata
5 years ago
ab868237a0
replace ad hoc line stream with readline module
5 years ago
1d5534b593
provide more debugging info on metadata parse errors
5 years ago
b59f03b949
consider changes to metadata logs as channel activity
5 years ago
d6fc35cab4
mkdirp the path to metadata before writing
5 years ago
b82c0a420a
merge latest staging
5 years ago
84e1057153
address remaining notes in the storage api
5 years ago
a63bea1f59
remove invalid callback which would totally break channelBytes
5 years ago
3c1aadd6c1
archive metadata even if there is no corresponding channel log
5 years ago
a3a7bcbe35
add comments for lots of functions
5 years ago
d6d7f4f601
include the size of the metadata log in the computed size of a channel
5 years ago
2aebadcff3
use the correct path for channel data when restoring logs from the archive
5 years ago
1c89e8d6ac
A few updates for editable metadata
...
* add and update many comments
* correctly handle metadata for young channels and range requests
* add a simple implementation of 'writeMetadata'
5 years ago
206de52ee8
call back from writeMetadata
5 years ago
bf058d8042
prevent multiple callbacks when reading dedicated metadata
5 years ago
b36f50f2c9
update metadata queries to include edits
5 years ago
b128b7f02c
updated methods for working with editable metadata
5 years ago
fd43da2dbf
archive and restore metadata logs and channel logs together
5 years ago
a16d7171ad
add some more notes
5 years ago
0668abf68c
correct the regular expression for iterating over channels
5 years ago
fc9aaf0ecb
remove unnecessary argument from channelExists
...
also add some notes
5 years ago
3513e437b6
update atime when accessing files
5 years ago
46cd796db5
label code that will need to change for editable metadata
5 years ago
0cb3a13aca
lint compliance
6 years ago
b59969f48b
archive channels at expiration time if configured to prefer archiving
6 years ago
d19b704bb8
fix an inverted boolean
6 years ago
1e5248ff9d
implement methods for checking if a file exists in the database or the archive
6 years ago
58cee9e13d
remove an XXX because the code looks safe
6 years ago
6aa6b706a4
don't release until you think about this XXX a bit more
6 years ago
ddf4a6a784
pass the correct path to 'listChannels' for 'listArchivedChannels'
6 years ago
7e455e6fce
drop unnecessary code, restore instead of remove archived files in restoration script, change restore archived name
6 years ago
cb0c4ee944
don't continue iterating in listChannels until a the current task has called back
6 years ago
c0d908af4d
implement 'unarchiveChannel'
6 years ago
11976b262e
add a note for later
6 years ago
dd375a2a22
every type of storage should have its own nested directory within the archive
6 years ago
7647f7c68a
implement four new storage APIs
...
* listChannels
* listArchivedChannels
* archiveChannel
* removeArchivedChannel
6 years ago
aa9a27634a
actual lint compliance
6 years ago
ec64de2c67
additional work on the storage API
...
* optimize layout on the filesystem
* drop some unused functions
* don't recurse over folders that you know won't be relevant
* remove empty folders
* add a migration API
* fix some minor bugs with asynchrony
* lint compliance
6 years ago
7b55df5931
rewrite task execution as API instead of a script
6 years ago
253545636b
add a note to fix some weird behaviour in storage/file
6 years ago
e1e6d51038
new logging api
6 years ago
fc43c08baf
un-break all the imports in tasks.js
6 years ago
51fae45cce
add notes re: logging and cold storage
6 years ago
0a1971cfee
improved support for nested data directories
6 years ago
ansuz