0.21.0

Pony 0.21.0 is a recommended update. It's a huge release for us as it contains many stability fixes as well as a variety of other improvements. It contains a breaking change related to as and match statements. Updating existing codebases should be relatively easy.

Generalised runtime backpressure

There's a ton of significant changes in 0.21.0. Our headliner is generalised runtime backpressure. The Pony scheduler has been updated to apply backpressure between actors. I'll be writing a more in-depth blog post on the topic.

Before the addition of runtime backpressure, it was possible to create Pony programs that would be able to cause runaway memory growth due to a producer/consumer imbalance in message sending. There are a variety of actor topologies that could cause the problem.

Because Pony actor queues are unbounded, runaway memory growth is possible. This commit contains a program that demonstrates this. examples/overload has a large number of actors sending to a single actor. Under the original scheduler algorithm, each of these actors would receive a fairly equivalent number of chances to process messages. For each time an actor is given access to the scheduler, it is allowed to process up to batch size number of messages. The default for batch size is 100. The overload example many many actors sending to a single actor where it can't keep up with the send.

With this feature, the Pony scheduler can now apply backpressure. The basic idea is:

• Pony message queues are unbounded
• Memory can grow without end if an actor isn't able to keep up with the incoming messages
• We need a way to detect if an actor is overloaded and if it is, apply backpressure

We apply backpressure according to the following rules:

• If an actor processes batch size application messages then it is overloaded. It wasn't able to drain its message queue during a scheduler run.
• Sending to an overloaded actor will result in the sender being "muted."
• Muting means that an actor won't be scheduled for a period of time allowing overloaded actors to catch up.

Particular details on this

• Sending to an overloaded or muted actor will result in the sender being muted unless the sender is overloaded.
• Muted actors will remain unscheduled until any actors that they sent to that were muted/overloaded are no longer muted/overloaded

The basics of backpressure are in place. Still to come:

Backpressure isn't currently applied to the cycle detector so its queue can still grow in an unbounded fashion. More work/thought needs to go into addressing that problem.

It's possible that due to implementation bugs that this commit results in deadlocks for some actor topologies. I found some implementation issues that had to be fixed after my first pass. The basic algorithm though should be fine.

There are some additional work items that could be added on to the basic scheme. Some might turn out to be actual improvements; some might turn out to not make sense.

• Allow for notification of senders when they send to a muted/overloaded actor. This would allow application level decisions on possible load shedding or other means to address the underlying imbalance.
• Allow an actor to know that it has become overloaded so it can take application level
• Allow actors to have different batch sizes that might result in better performance for some actor topologies

This work was performance tested at Wallaroo Labs and was found under heavy loads to have no noticeable impact on performance.

Runtime stability improvements

This release address many edge cases that could lead to a lack of Pony runtime stability. Some issues related to possible runaway memory growth have been closed as well as a variety of smaller issues.

Add DTrace probes for all message push and pop operations

Scott Fritchie, while helping diagnose several of the runtime stability issues that are fixed in this release, realized Pony's DTrace coverage was lacking. Some message sends between actors where being traced but other paths were not. Scott updated and refactored DTrace support for message sends so that all messages will now be accounted for. If you are using DTrace for observability of Pony programs, this is a huge moment for you. Things just got much easier. Thanks, Scott!

Experimental support for LLVM 4.0.1 and 5.0.0

LLVM 4 and 5 can now be used to build Pony. However, support is experimental. To work around a bug that caused Pony to segfault, we had to introduce a performance degradation in Pony when building with LLVM 4 and 5. Additionally, it's possible that your programs might crash. If you can build with LLVM 4 and 5, we encourage you to do so and give us feedback. We do not recommend LLVM 4, and 5 be used to build Pony programs that will be used in a production environment. To follow our progress with making LLVM 4 and 5 officially supported, please see issue #2371.

Error on unreachable cases in match expressions and illegal as expressions

This change is adding some sanity checks to usages of match and as expressions.
It does two things:

Match Expressions

It validates that there is no unreachable code left in your match expressions. This was a subtle source of bugs and left dead code to linger in your codebase. Unreachable cases or else clauses are triggering a compiler error now.

Example:

class Foo
  fun ex_match(a: (String| Bool)): String => 
    match a
    | let a: Stringable => a.string()
    | let b: Bool => if b then "true" else "false" end // unreachable already
    else
      "unreachable"
    end

The second branch and the else clause are both unreachable and thus can (and should) be safely removed. In some cases it might instead make sense to rewrite the match, reordering the cases from more specific checks to less specific ones.

As Expressions

It also validates the correct use of as, which should only be used to safely increase the specificity of an object's type, that is casting. Previously it was possible to cast to the type of the expression to be cast or to one of its subtypes, which can be achieved by simple assignment. Using as here introduces a false positive partiality. Those incorrect or unnecessary usages of as trigger a compiler error with this change.

Example:

class Foo
   fun subtype_cast(a: String): Stringable ? =>
     a as Stringable // useless as, is actually not partial

   fun eq_cast(a: Array[String]) =>
     let tmp = a as Array[String] // no as needed
     ...

This error can easily be fixed by removing the as as it is not necessary in both cases.

[0.21.0] - 2017-12-17

Fixed

• Forbid structs with embed fields with finalisers (PR #2420)
• Fix codegen ordering of implicit finalisers (PR #2419)
• Fix GC tracing of struct fields (PR #2418)
• Remove redundant error message for unhandled partial calls that are actually in a try block. (PR #2411)
• Fix allocation sizes in runtime benchmarks (PR #2383)
• Fail pony_start if ASIO backend wasn't successfully initialized (PR #2381)
• Make windows sleep consistent with non-windows sleep (PR #2382)
• Fix Iter.{skip,take,nth} to check '.has_next()' of their inner iterator (PR #2377)
• Restart ASIO if needed while runtime is attempting to terminate. (PR #2373)
• fix Range with negative or 0 step and allow backward Ranges (having min > max) (PR #2350)
• Improve work-stealing "scheduler is blocked" logic (PR #2355)
• Make take_while short-circuit (PR #2358)
• Fix compilation error with 'use=dtrace' for FreeBSD 11 (PR #2343)
• Fix Set.intersect (PR #2361)
• Fixed state handling of HTTP client connections (PR #2273)
• Fix incorrect detection of exhaustive matches for structural equality comparisons on some primitives. (PR #2342)
• Fix poor randomness properties of first call to Rand.next(). (PR #2321)
• Fully close unspecified family TCP connections on Windows. (PR #2325)
• Make ContentsLogger implement the Logger interface (PR #2330)
• Fix alloc bug in String/Array trimming functions (PR #2336)
• Fix small chunk finaliser premature re-use bug (PR #2335)
• Make Payload.respond() send given parameter, not this. (PR #2324)
• Garbage collect actors when --ponynoblock is in use (PR #2307)
• Fix incorrect kevent structure size (PR #2312)
• Fix possible repetition in Iter.flat_map (PR #2304)

Added

• Add DTrace probes for all message push and pop operations (PR #2295)
• Experimental support for LLVM 4.0.1 (#1592) and 5.0.0. ([PR #2303](https://github.com/...

0.20.0

Pony 0.20.0 is recommended update. It contains one small breaking change that is unlikely to effect most users.

Replace memory-leak-inciting Writer.reserve with more intuitive Writer.reserve_current method

Writer.reserve has been removed from the Writer package. Any calls to it should be replaced with a call to Writer.reserve_current. Note, that the semantics of the calls are somewhat different, so a direct swap isn't advised.
Writer.reserve would allocate X amount of additional memory. This non-intuitive API was leading some users to leak memory accidentally. The new reserve_current will make sure that the buffer has enough space to hold X bytes.

Turn off LTO by default on OSX

Link-time optimization, or LTO, is an optimization that can be applied to Pony programs to increase performance. It was only on by default for OSX. The could, however, lead to "spooky action at a distance" breakage of a Pony install. Upgrading XCode without reinstalling Pony will result in a broken compiler if LTO is on. We've decided to turn off LTO by default so that users have to knowingly opt-in to using it and won't potentially be greeted with confusing error message should they upgrade XCode.

[0.20.0] - 2017-10-17

Fixed

• Forbid single '_' in case expr and case methods (PR #2269)

Changed

• Turn off LTO by default on OSX (PR #2284)
• Replace memory-leak-inciting Writer.reserve with more intuitive Writer.reserve_current method. (PR #2260)

0.19.3

Pony 0.19.3 includes a number of bug fixes including a possible memory leak. Upgrading as soon as possible is recommended.

[0.19.3] - 2017-10-09

Fixed

• Don't verify partial calls for method bodies inherited from a trait. (PR #2261)
• Fix broken method and type headings in generated documentation (PR #2262)
• Fix array inference from ReadSeq interface with tuple element type. (PR #2259)
• Fix compiler crash related to inferred lambda argument in apply sugar. (PR #2258)
• Fix excess work stealing under low loads (PR #2254)
• Fix compiler crash on case methods with _ as a method parameter. (PR #2252)
• Fix implicit fallthrough in array inference. (PR #2251)

0.19.2

Pony 0.19.2 features a high priority fix for a bug in Pony serialization. Before 0.19.2, serialising an empty string would cause a segmentation fault.

Fix empty string serialisation

The Pony 0.15.0 release introduced a bug in Pony serialization. From Pony 0.15.0 to 0.19.1 the following code would cause a segfault:

use "serialise"

actor Main
  new create(env: Env) =>
    try
      let auth = env.root as AmbientAuth
      Serialised(SerialiseAuth(auth), "")?
    end

RFC #46 implemented

Pony 0.19.2 also includes the implementation of RFC #46: More random methods.
The following methods were added to the Random trait, similar to the existing u8, u16, u32, u64, and u128 methods:

  fun ref ulong(): ULong =>
    """
    A random integer in [0, ULong.max_value()]
    """

  fun ref usize(): USize =>
    """
    A random integer in [0, USize.max_value()]
    """

  fun ref i8(): I8 =>
    """
    A random integer in [-2^8, 2^8)
    """

  fun ref i16(): I16 =>
    """
    A random integer in [-2^16, 2^16)
    """

  fun ref i32(): I32 =>
    """
    A random integer in [-2^32, 2^32)
    """

  fun ref i64(): I64 =>
    """
    A random integer in [-2^64, 2^64)
    """

  fun ref i128(): I128 =>
    """
    A random integer in [-2^128, 2^128)
    """

  fun ref ilong(): ILong =>
    """
    A random integer in [ILong.min_value(), ILong.max_value()]
    """

  fun ref isize(): ISize =>
    """
    A random integer in [ISize.min_value(), ISize.max_value()]
    """

Additionally, the existing int method of the Random trait was updated to allow specifying the integer type as a type parameter, instead of only supporting the U64 type:

  fun ref int[N: (Unsigned val & Real[N] val) = U64](n: N): N =>
    """
    A random integer in [0, n)
    """

The array-oriented shuffle method was added to the Random trait.

  fun ref shuffle[A](array: Array[A]) =>
    """
    Shuffle the elements of the array into a random order, mutating the array.
    """

To support Random.shuffle, the swap_elements was added to the Array class (because otherwise, it is not possible to swap two elements in place without aliasing them):

  fun ref swap_elements(i: USize, j: USize) ? =>
    """
    Swap the element at index i with the element at index j.
    If either i or j are out of bounds, an error is raised.
    """

[0.19.2] - 2017-09-24

Fixed

• Fix codegen failure on field access (PR #2244)
• Make Windows link step use the command-line --linker value if present. (PR #2231)
• Fix empty string serialisation (PR #2247)

Added

• Added Array.swap_elements, Random.shuffle and extends Random with methods for generating all Integer types (RFC 46). (PR #2128)

0.19.1

Pony 0.19.1 has no breaking changes and fixes no high-priority bugs, so you can update at your leisure.

Lambda and Array Inference

This release includes significant improvements to type inference for array literals and lambda expressions when the type of the expression has an unambiguous antecedent (such as the "left side" of an assignment, the parameter signature of a method call, or the return type of a method body), including the following improvements:

• Empty array literals are now valid syntax: let a: Array[U8] = []
• Non-homogenous concrete elements in an array literal can be treated as a trait or interface instead of as a union: let a: Array[Stringable] = [true; None; "string"]
• Array literals can have an implied capability recovery to iso or val: let a: Array[String] val = ["foo"; "bar"; "baz"]
• Lambda expressions can have implied parameter and return types: let fn: {(U64, U64): U64} = {(x, y) => x + y }
• Unused lambda parameters can use the "don't care" symbol (_) instead of a parameter name and type: let fn: {(U64, U64): U64} = {(_, y) => y * 2 }
• The receiver and object capability of lambda expressions can be inferred from context, instead of being inferred from the statefulness of the lambda.

For more information, see RFC #45.

[0.19.1] - 2017-09-14

Fixed

• Fix broken "make" command (PR #2220)
• Fix inconsistencies in multi-line triple-quoted strings (PR #2221)
• Fix undersized string buffer for library link command in Windows. (PR #2223)
• Fix Iter.take to handle infinite iterator (PR #2212)
• Fix handling of empty and multi-byte character literals (PR #2214)

Added

• Inference of lambda type and array element type from an antecedent (RFC 45). (PR #2168)

0.19.0

Pony 0.19.0 contains breaking changes. If you don't use the Itertools package then upgrading should be painless. If you do use Itertools, the process should be relatively straightforward. MacOS, FreeBSD and Dragonfly BSD users are recommended to upgrade as soon as possible to fix a race condition on the kqueue event system.

Improve the Itertools API

The implementation of RFC #49 makes many changes to the itertools package. The first significant change is that all classes other than Iter have been removed. All of these classes have had equivalent functions in the Iter class, the exception being Repeat, which is now handled by the repeat_value constructor. So, for example, Repeat[String]("a") should be replaced by Iter[String].repeat_value("a").

Another significant change is that fold has become non-partial and has had its arguments switched so that the lambda is placed last. For example:

let sum =
  try  
    Iter[I64]([1; 2; 3].values())
      .fold[I64]({(sum: I64, n: I64): I64 => sum + n }, 0)?
  else
    I64(0)
  end

Should be replaced by

let sum =
  Iter[I64]([1; 2; 3].values())
    .fold[I64](0, {(sum: I64, n: I64): I64 => sum + n })

If the lambda argument to fold needs to be partial, then you should use the new fold_partial method.

Other methods have been added to the Iter class such as flat_map, filter_map, as well as methods that allow lambdas with ref receivers for stateful transformations. These methods have a "_stateful" suffix, such as map_stateful and filter_stateful.

Fix signals on Sierra

MacOS Sierra changed how signals are delivered to multithreaded processes. Pony's signal handling has been broken on Sierra since it was introduced. It's now fixed as Sylvan and Sean finally sat down to figure out what was going on.

[0.19.0] - 2017-09-02

Fixed

• Fix codegen failures on incompatible FFI declarations (PR #2205)
• Disallow named arguments for methods of a type union where parameter names differ (PR #2194)
• Fix compiler crash on illegal read from '_' (PR #2201)
• Fix signals on Sierra (PR #2195)
• Fix race condition in kqueue event system implementation (PR #2193)

Added

• pony_chain runtime function

Changed

• The pony_sendv and pony_sendv_single runtime functions now take a message chain instead of a single message
• Improve the itertools API (RFC 49) (PR #2190)
• Forbid struct finalisers (PR #2202)

0.18.1

Pony 0.18.1 contains a high priority bug fix in the ProcessMonitor package. We recommend upgrading as soon as possible. There are no breaking changes in this release.

Process monitor async write buffering

Before this release, it was easy to swamp an external process started by the ProcessMonitor with too many writes. Excessive writes would result in errors and lost data. 0.18.1 adds write buffering thereby avoiding lost data.

DragonFly BSD 4.8 support

Pony has been ported to support DragonFly BSD 4.8. Note, we do not have CI for DragonFly, so support is provided on a "best-effort" basis. That is, we won't intentionally break support, but we also aren't actively testing commits against DragonFly so some breakage can be expected.

[0.18.1] - 2017-08-25

Fixed

• Don't print capabilities for type params when generating docs (PR #2184)

Added

• DragonFly BSD 4.8 support (PR #2183)
• Process monitor async write buffering (PR #2186)

0.18.0

Pony 0.18.0 contains a high priority compiler bug fix. We recommend updating as soon as possible. Note, however, that there is a small breaking change in this release as well. It should be a quick upgrade if you are impacted.

Fix compiler crash on union-of-tuples to tuple conversions

Previously, when trying to compile a union-of-tuples, the compiler would segfault. Very annoying if you had a reasonable program like:

primitive Add
primitive Dec
type AddOrDec is (Add | Dec)
type CmRDTCounterOp is (AddOrDec, U128)

class CmRDTCounter
  var _value: U128

  new create() => _value = 0

  fun read(): U128 =>
    _value

  fun ref add(number: U128): CmRDTCounterOp =>
    let op: CmRDTCounterOp =
      if number >= 0 then
        (Add, number)
      else
        (Dec, number)
      end
    apply(op)
    op

  fun ref apply(op: CmRDTCounterOp) =>
    match op
      | (Add, let number: U128) => _value = _value + number
      | (Dec, let number: U128) => _value = _value - number
    end

actor Main
  new create(env: Env) =>
    var counter = CmRDTCounter.create()
    let op1 = counter.add(10)

If you are interested in learning more, we suggest checking out issue #1513 for a bit more background and listen to the August 16, 2017, Pony development sync call where we discussed it in depth.

Change String.join to take Iterable

0.18.0 includes the implementation of RFC #48. It's a nice improvement to the usability of String.join. It is also, sadly, a breaking change. Such is life. Sometimes you break things to improve them. Upgrading should be straightforward.

Where you previously called "_".join(["zomg"]), you would now call "_".join(["zomg"].values()). Any place you hit a compiler error related to String.join, add a .values() to the ReadSeq you are passing and everything will work.

[0.18.0] - 2017-08-19

Fixed

• Fix compiler crash on union-of-tuples to tuple conversions (PR #2176)
• Fix compiler error on lambda capture of '_' (PR #2171)
• Fix read past the end of a buffer in pool.c. (PR #2139)

Changed

• Make actor continuations a build time option (PR #2179)
• RFC #48 Change String.join to take Iterable (PR #2159)
• Change fallback linker to "gcc" from "gcc-6" on Linux. (PR #2166)
• Change the signature of pony_continuation and pony_triggergc to take a pony_ctx_t* instead of a pony_actor_t*.

0.17.0

Pony 0.17.0 is a recommended release if you are using GCC 7 on Linux. Before this release, GCC 7 was unable to build the Pony compiler. If you aren't using GCC 7, you can take your time upgrading. Please note, there is a breaking change in this release, but it's quite unlikely that anyone will be impacted by it.

Treat as type of array literals as the alias of the element type

Previously, an array literal, even when the as type is given explicitly, uses the alias of the as type as the type argument for the Array type. For example:

trait iso T
class iso A is T
class iso B is T
class iso C is T

actor Main
  new create(env: Env) =>
    let a: T = A
    let b: T = B
    let c: T = C
    let abc: Array[T] = [as T^: A; B; C] // works
    let abc': Array[T] = [as T: A; B; C] // fails: literal has type Array[T!]

We doubt that anyone currently has the `// works`` line anywhere in their code. However, if you are impacted, update it to:

let abc': Array[T] = [as T: A; B; C]

default_pic build option

Previously on platforms that required PIC, you would have to pass the --pic flag to ponyc each time you compiled a Pony program. With the addition of default_pic, you can build ponyc so that --pic is automatically handled for you. Use make default_pic=true when building ponyc and away you go!

[0.17.0] - 2017-08-05

Fixed

• Fix cursor location for displaying compiler errors and info. (PR #2136)
• Fix indent detection when lexing docstrings that contain whitespace-only lines. (PR #2131)
• Fix compiler crash on typecheck error in FFI call. (PR #2124)
• Fix compiler assert on match including structural equality on union type. (PR #2117)

Added

• Support GCC7 on Linux (PR #2134)
• Add regex match iterator (PR #2109)
• Add more promise methods (RFC 35) (PR #2084)
• Add ability to default enable PIC when compiling ponyc (PR #2113)

Changed

• Treat as type of array literals as the alias of the element type. (PR #2126)
• docgen: ignore test types and add cli flag to only document public types (PR #2112)

0.16.1

Pony 0.16.1 is a quick follow on release to 0.16.0. After releasing 0.16.0, we closed out a couple high priority bugs. Upgrading as soon as possible is recommended. If you haven't upgraded to 0.16.0, check out it's release notes before starting your upgrade.

[0.16.1] - 2017-07-30

Fixed

• Fix reachability analysis for intersection types (PR #2106)
• Fix compiler assertion failure at code generation (PR #2099)
• FreeBSD builds(PR #2107)