Skip to content

Commit

Permalink
Added async throttling section to readme.
Browse files Browse the repository at this point in the history
Updated changelog for v5.
  • Loading branch information
Bilge committed Dec 8, 2019
1 parent 39d8e32 commit 5270f51
Show file tree
Hide file tree
Showing 3 changed files with 33 additions and 4 deletions.
17 changes: 17 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,22 @@
# Porter change log

## 5.0.0 – Async

Porter v5 introduces asynchronous imports and complete strict type safety (excluding union types and generics).

### Breaking changes

* Removed support for PHP 5.5, 5.6 and 7.0.
* Every interface has been updated to include return types which means all consuming projects must also add the same return type.
* Replaced `Connector::fetch` string source parameter with new `DataSource` interface.
* Removed `ConnectionContext` from `Connector` interface.
* Added `SingleRecordResource` interface that resources must implement to be used with `Porter::importOne()`.
* Prevented single record resources being imported with multi-record import methods.
* Replaced `RecoverableConnectorException` with `RecoverableException` interface.
* Removed failed abstractions: `ConnectorOptions` and `EncapsulatedOptions`.
* Removed abstraction: `CacheKeyGenerator`.
* Moved `ForeignResourceException` to Porter's namespace.

## 4.0.0 – Rewrite

Porter v4 fixes all known design flaws (#31, #43) and critically re-evaluates every part of Porter's design. All base classes have been discarded (`AbstractProvider`, `AbstractResource`), moving their code within Porter, relying solely on interfaces instead. This frees up the inheritance chain for applications to use as they wish, making it much easier to integrate Porter into existing projects.
Expand Down
18 changes: 15 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,8 +189,18 @@ We must be inside the async event loop to begin programming asynchronously. Let'
});
```

We would not usually code directly inside the event loop in a real application, however we always need to create the event loop somewhere, even if it just calls a service method in our application which delegates to other objects. To pass asynchronous data through layers of abstraction, our application's methods must return `Promise`s that wrap the data they would normally return directly in a synchronous application. For example, a method returning `string` would instead return `Promise<string>`, that is, a promise that returns a string.

Programming asynchronously requires an understanding of Amp, the async framework. Further details can be found in the official [Amp documentation][].

### Throttling

The asynchronous import model is very powerful because it changes our application's performance from being I/O-bound to being CPU-bound. That is, in the traditional synchronous model, each import operation must wait for the previous to complete before the next begins, meaning the total import time depends how long it takes each import's network I/O to complete. In the async model, since we send many requests concurrently without waiting for the previous to complete, on average each import operation will only take as long as our CPU takes to process it, since we are busy processing another import during network latency.

High volume synchronous imports are, in a way, self-throttling and it is rare to trip protection measures in this mode, however the naïve approach to asynchronous imports is often fraught with perils. For example, when we import 10,000 HTTP resources at once, one of two things usually happens: either we run out of PHP memory and the process is killed or the HTTP server blocks us for sending too many requests in a short period. The solution is throttling.

We provide [Async Throttle][] to throttle asynchronous imports. The Async Throttle is a separate project that does not have any direct integration with Porter because that is not needed. The throttle operates on any Amp promises, such as those returned by Porter. The throttle works by preventing additional operations starting when too many are concurrently executing, based on user-defined limits.

Transformers
------------

Expand Down Expand Up @@ -468,9 +478,10 @@ Limitations

Current limitations that may affect some users and should be addressed in the near future.

- No end-to-end data steaming interface yet.
- Caching does not support asynchronous imports yet.
- [Sub-imports][] do not support async yet.
- No end-to-end data steaming interface.
- Caching does not support asynchronous imports.
- [Sub-imports][] do not support async.
- No import rate throttle for synchronous imports.

Testing
-------
Expand Down Expand Up @@ -533,3 +544,4 @@ Porter is published under the open source GNU Lesser General Public License v3.0
[ECB test]: https://github.com/Provider/European-Central-Bank/blob/master/test/DailyForexRatesTest.php
[Amp]: https://amphp.org
[Amp documentation]: https://amphp.org/amp/
[Async Throttle]: https://github.com/ScriptFUSION/Async-Throttle
2 changes: 1 addition & 1 deletion composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
},
"suggest" : {
"connectors/http": "Provides an HTTP connector for Porter providers.",
"transformers/mapping-transformer": "Transforms records using Mappings.",
"transformers/mapping-transformer": "Transforms records using Mappings and provides sub-imports.",
"async/throttle": "Limits throughput of asynchronous imports."
},
"autoload": {
Expand Down

0 comments on commit 5270f51

Please sign in to comment.