0.7.2 (2017-06-27)
Merged pull requests:
- update dependencies build files #169 (mvertes)
- update dependencies #168 (mvertes)
- workers: reduce amount of traces during shuffle #167 (mvertes)
- worker: manually call garbage collector at end of task. Experimental #166 (mvertes)
- add interactive REPL shell, supporting async/await #165 (mvertes)
- Fix and improved tracing of time measurements #164 (mvertes)
- worker: allow to retry connections using -r <nbretry> option #163 (mvertes)
- dataset: fix parsing of shuffle in aggregateByKey, clean code #162 (mvertes)
- performance: block processing for pipelines at source level, part 2 #161 (mvertes)
- eslint: allow arrow functions and async/await #160 (mvertes)
- Fix previous task serialization change which impacted sc.range() #159 (mvertes)
- Clean code, improve traces for performances, set maximum partitions. #158 (mvertes)
- update dependencies #157 (mvertes)
- textFile: Fix handling of gzipped files in local filesystem #156 (mvertes)
0.7.1 (2017-05-17)
Merged pull requests:
- API doc: add a section on environment variables #155 (mvertes)
- doc: fix some typos #154 (mvertes)
- Fix a worker crash when using ds.sample() with replacement. #153 (mvertes)
- add sample docker files #152 (mvertes)
- doc: fix links #151 (mvertes)
- mention the Skale Hacker's Guide #150 (mvertes)
- add a Skale Hacker's Guide #149 (mvertes)
- Clean up dependency, more info at worker-controller connection #147 (mvertes)
- Added .npmignore #146 (mvertes)
- worker: retry initial network connection to server #145 (mvertes)
0.7.0 (2017-04-04)
Fixed bugs:
Closed issues:
Merged pull requests:
- update dependencies #143 (mvertes)
- textFile, save: document new protocols and formats #142 (mvertes)
- save: support direct streaming to S3 #141 (mvertes)
- textFile: support file globbing (i.e. src/**/*.js) in path argument #140 (mvertes)
- save: support direct streaming to azure. #139 (mvertes)
- azure: handle server errors with internal retry policy filter #138 (mvertes)
- AggregateByKey: faster processing of shuffle files #137 (mvertes)
- aggregateByKey, coGroup: performance improvements #136 (mvertes)
- Performance increase of aggregateByKey and coGroup #135 (mvertes)
- azure: handle errors from storage server: fetch before process and implement retries. #134 (mvertes)
- distributed mode: pre-fork new workers at end of master #133 (mvertes)
- textFile: fix compute of partitions for Azure #132 (mvertes)
- save: upload datasets to azure for urls in wasb:// #131 (mvertes)
- textFile: support file globbing, azure blobs only for now #130 (mvertes)
- textFile: support reading from azure storage blobs #129 (mvertes)
- Simplify top and take actions, and fix out of memory errors for large datasets. #128 (mvertes)
- Aggregate now guarantees partitions ordering. Improve traces. #127 (mvertes)
- textFile: read parquet from local and S3 directories #125 (mvertes)
- Add Parquet support #124 (mvertes)
- Readme: Fix appveyor badge #123 (mvertes)
- fix benchmark #122 (mvertes)
- Fix handling internal dependency on self. #121 (mvertes)
0.6.11 (2017-02-09)
Merged pull requests:
0.6.10 (2017-02-09)
Merged pull requests:
- Add node 6 target for travis, use system default memory settings #119 (mvertes)
- Allow to run on windows #118 (mvertes)
0.6.9 (2017-01-02)
Merged pull requests:
- textFile: add a maxFiles option, to limit the number of files to process #117 (mvertes)
- Increase number of streams over skale protocol #116 (mvertes)
- stream() action: add option to terminate context on stream end #115 (mvertes)
- doc: Fix TOC link #114 (mvertes)
0.6.8 (2016-12-14)
Fixed bugs:
- In distributed mode, temporary files are not deleted at end of program #110
Merged pull requests:
- Advertise benchmark in README #113 (mvertes)
- server, worker-controller: add current statistics #112 (mvertes)
- Fix bug #110 where temporary files were not deleted at end of task in… #111 (mvertes)
- use SKALE_WORKERS to set number of workers in distributed mode, as in… #109 (mvertes)
- update dependencies #108 (mvertes)
- Doc: add a section on core concepts, label shuffle transforms #107 (mvertes)
- Refactor internal Task API, to propagate env and dependencies. In pro… #106 (mvertes)
0.6.7 (2016-11-22)
Merged pull requests:
- Document standalone and distributed modes #105 (mvertes)
- Distributed mode: Implement peer-to-peer through HTTP for shuffle data transfer #104 (mvertes)
- Improve traces and file transfer. #103 (mvertes)
- Improve task scheduling #102 (mvertes)
- Fix mocha dependency. Skip yarn files. #101 (mvertes)
0.6.6 (2016-11-04)
Merged pull requests:
- Task serialization is now correct in all cases. #100 (mvertes)
- Optimize task data transfer by sending sparse datasets. #99 (mvertes)
- Preliminary work for optimization of task data transfer #98 (mvertes)
- improve getReadStream #97 (mvertes)
- distributed mode: fixes in compressed transfers #96 (mvertes)
- Improve distributed mode #95 (mvertes)
- distributed mode: protocol debug traces enabled with SKALE_DEBUG=3 #94 (mvertes)
- sc.textFile: fix handling of S3 directories #93 (mvertes)
- Fix log in distributed worker #92 (mvertes)
0.6.5 (2016-10-23)
Merged pull requests:
- bin/worker.js: fix typo in dependencies #91 (mvertes)
- Debug traces sent to stderr instead of stdout #90 (mvertes)
- save to S3: extend http timeout to 1h #89 (mvertes)
0.6.4 (2016-10-11)
Merged pull requests:
- dataset stream and save: preserve structure #88 (mvertes)
- textFile: add capability to handle single AWS S3 files, gzipped or not. #87 (mvertes)
0.6.3 (2016-10-08)
Merged pull requests:
0.6.2 (2016-09-14)
Closed issues:
- bin/server doesn't use nworker parameter #82
Merged pull requests:
- Fix save() action to export a dataset to filesystem or S3 #84 (mvertes)
- Remove unused nworker command line parameter #83 (mark-bradshaw)
- Minor grammar update. #80 (mark-bradshaw)
- Add streaming source from AWS S3 #78 (mvertes)
- local worker: handle master disconnect #77 (mvertes)
- Increase buffer size of zlib from 16kB to 64kB. Better performances #76 (mvertes)
- textFile now supports directory as dataset source #75 (mvertes)
- Coding style change. We now use the same as NodeJS core. #74 (mvertes)
0.6.1 (2016-07-05)
Merged pull requests:
- Important improvements in task serialization, scheduling and debugging #73 (mvertes)
- save: reset file at init #72 (mvertes)
- Add SKALE_MEMORY env variable to set worker max memory #71 (mvertes)
- spillToDisk: do not duplicate memory when writing to disk #70 (mvertes)
0.6.0 (2016-06-24)
Merged pull requests:
- stop testing node-0.10 #69 (mvertes)
- add new source gzipFile() to process gzipped text files #68 (mvertes)
- New action Dataset#save() to save dataset content in text file. #67 (mvertes)
- Pass user options and worker context to reduce(), aggregate() and forEach() callbacks. #66 (mvertes)
- Serverless version, direct IPCs between workers and master #65 (mvertes)
- code cleaning #64 (mvertes)
- improve tmp dir handling. Add env SKALE_TMP #63 (mvertes)
- remove spurious test file #62 (mvertes)
0.5.3 (2016-05-17)
Closed issues:
- skale-engine can not be used as a dependency in a Meteor project #56
- Get rid of direct eval #53
- Use mkdirp.sync instead of lib/mkdir.js #51
Merged pull requests:
- rename lib/sizeof.js in lib/rough-sizeof.js as discussed in #52 #60 (mvertes)
- Use indirect eval, strenghten regexp which match arrow function definition; fix #53 #59 (mvertes)
- remove lib/mkdir.js and use external mkdirp module. Fix #51 #58 (mvertes)
- Ensure compatibility with node back to 0.10, fix #56 #57 (mvertes)
- better use of console.log() #50 (mvertes)
- ds.filter(): rename internal filter member into _filter to avoid coll… #49 (mvertes)
0.5.2 (2016-05-04)
Merged pull requests:
0.5.1 (2016-05-04)
Merged pull requests:
- doc: fix doc on actions #47 (mvertes)
- fix ds.first() to return an element instead of an array #46 (mvertes)
- remove lib/ml.js, restructure examples #45 (mvertes)
- various fixes #44 (mvertes)
0.5.0 (2016-05-02)
Merged pull requests:
- benchmark: update to new action syntax #43 (mvertes)
- Remove toArray(). Fix examples #42 (mvertes)
- All actions take an optional callback and return promises instead of … #41 (mvertes)
- simplify JS benchmark code to make it almost identical to python version #40 (mvertes)
- add benchmark #38 (mvertes)
0.4.5 (2016-04-25)
Merged pull requests:
- Provide a faster sizeof, without external dependency #37 (mvertes)
- code cleaning #36 (mvertes)
- fix broken arrow function example #35 (mvertes)
- Command line option to set max memory per worker #34 (mvertes)
- master: set process title to ease monitoring #33 (mvertes)
0.4.4 (2016-04-20)
Merged pull requests:
0.4.3 (2016-04-20)
Merged pull requests:
- doc: various fixes #31 (mvertes)
- Dataset: provide a better RNG. Expose Random and Poisson classes. #30 (mvertes)
- On workers: evict a partition if going out of memory #29 (mvertes)
- Handle connection error to terminate a master if a worker fails #28 (mvertes)
0.4.2 (2016-04-17)
Merged pull requests:
- Preliminary work to isolate machine learning capabilities from core engine #27 (CedricArtigue)
- fix lint #26 (mvertes)
- run eslint in pre-test #25 (mvertes)
- fix lint problems. Remove dead code #24 (mvertes)
- new lint fixes #23 (mvertes)
- Simplify code, fix lint errors. More to come. #22 (mvertes)
- add eslint. Start code cleaning #21 (mvertes)
0.4.1 (2016-04-07)
Merged pull requests:
- document partitioners #20 (mvertes)
- doc: fix cross references #19 (mvertes)
- document ds.top() #18 (mvertes)
- document ds.take() #17 (mvertes)
- document ds.sortByKey() #16 (mvertes)
- document ds.sortBy() #15 (mvertes)
- document ds.persist() #14 (mvertes)
- Document ds.partitionBy() and fix example #13 (mvertes)
- document ds.first() #12 (mvertes)
- document ds.aggregateByKey() #11 (mvertes)
- aggregateByKey: change args order to match aggregate #10 (mvertes)
- document sc.range() #9 (mvertes)
- use sizeof from external npm dependency #8 (mvertes)
- rename RDD in dataset. No functional change. #7 (mvertes)
0.4.0 (2016-04-04)
Merged pull requests:
0.3.5 (2016-04-03)
Fixed bugs:
- cartesian incorrect output #4
Merged pull requests:
- lib/dataset: cartesian.iterate(): fix partition index computation. Fix #4 #5 (mvertes)
- .travis.yml: set a fixed number of workers (4) #3 (mvertes)
0.3.4 (2016-04-01)
Fixed bugs:
- examples/core/parallelize.js fails with 2 workers #2
0.3.3 (2016-03-25)
0.3.2 (2016-03-23)
Merged pull requests:
- Add a Gitter chat badge to README.md #1 (gitter-badger)
* This Change Log was automatically generated by github_changelog_generator