HN Gopher Feed (2017-10-26) - page 1 of 10 ___________________________________________________________________
Why we used Pony to write Wallaroo
131 points by spooneybarger
https://blog.wallaroolabs.com/2017/10/why-we-used-pony-to-write-...___________________________________________________________________
eljimmy - 6 hours ago
I do enjoy reading these "Why we used
to build " posts, as
they're usually insightful and provide a glimpse into languages or
frameworks I'm not familiar with.On the flip side, I can't help but
wonder how likely they would have been to use Pony had none of
their core team members been active in the Pony community.
spooneybarger - 6 hours ago
Hope you enjoy this one as well.We weren't active in the Pony
community until we decided to use it for Wallaroo. We felt it was
very important to invest in the improvement of Pony. And to do
that, we needed to be part of the community.Pony became our
runtime instead of writing one ourself. That doesn't mean we
don't have to work on our runtime. It means we are sharing the
burden with others.
jtfmumm - 5 hours ago
Interestingly enough, none of us had even written anything
significant in Pony when we made the initial tentative decision.
After we came to the conclusion that it might be a good fit, we
started a test project and a couple of us tried ramping up as
fast as we could. It was actually only after we decided it was
the right choice that we started to get actively involved in the
community.
noncoml - 4 hours ago
Never heard Pony before. Sounds like a faster, type-safe erlang?
Sign me up, looks awesome!
spooneybarger - 4 hours ago
That's the general idea. But, Erlang has a wee bit of a head
start.
polskibus - 5 hours ago
Does Pony have preemptive scheduling? Does Pony have an equivalent
of Distributed Erlang?
aturley - 5 hours ago
Pony does not do preemptive scheduling. Pony actors have
behaviors, which can be thought of as methods that handle
messages that are sent to the actor; these are the unit of
scheduling, and a behavior runs from beginning to end without
preemption. Multiple behaviors can run at the same time, but only
one behavior per actor can be running at any time.There is
currently no equivalent to Distributed Erlang, but that's
something that the creator of the language has been looking into.
polskibus - 4 hours ago
Preemptive scheduling is one of the advantages of Erlang over
other actor implementations like Akka. Please consider
implementing it!
switchbak - 2 hours ago
Admittedly I'm not familiar with Erlang's preemptive model,
but I'll ask anyway :)When using async based APIs with
something like Akka, passing around futures and such, and
leveraging multiple actors in thread pools, doesn't that get
you to approximately the same place as Erlang?Sure, the JVM
isn't doing the preempting, but the OS is. Is the main
downside per-thread resource consumption and context
scheduling overhead?
anko - 1 hours ago
I think there is a lot to be said for the simplicity of the
erlang model. It's not just about stopping a thread from
hogging the cpu. The OS doesn't have a good idea when to
wake threads up, so they are effectively polling
needlessly.I do scala/akka in my dayjob and the amount of
time we spend tweaking threadpools (execution contexts) and
retesting just to get okay performance is insane. You wear
a pretty big cost when you layer on abstractions like that.
spooneybarger - 4 hours ago
A bit more info. Sebastian Blessing did a thesis on Distributed
Pony. You can read it here:
https://www.ponylang.org/media/papers/a_string_of_ponies.pdfAt
this time, there is no "in the wild" implementation.
macintux - 4 hours ago
Thanks. As an Erlang fan, I'm curious how types and
networking are going to overlap.
munificent - 6 hours ago
> The standard JVM garbage collection strategy is ?stop the
world.? This is quite an over-simplification for what is a
fundamental property to think about when designing a performance-
oriented system.The default collector in HotSpot does, I think,
stop the world when collecting. But it also does multiple small
collections between larger major collections. It, by default,
optimizes for throughput over latency since most applications care
more about overall throughput than low latency.Even with the
default, you can tune the maximum latency if you want to sacrifice
throughput:https://docs.oracle.com/javase/8/docs/technotes/guides/v
m/gc...Not only that, but HotSpot also features a true concurrent
collector, which is a command-line flag away.
dljsjr - 5 hours ago
The JVM concurrent GC has failure conditions. It does as much as
possible concurrently but still occasionally has to stop the
world.
krylon - 4 hours ago
I have been looking over to this language every now and then for a
long while now, and this article finally motivated me to actually
learn it.But it seems the documentation is a bit out of date - I
cannot even get the "Hello, world"-program to build correctly,
neither on Debian Stretch nor on openSUSE Tumbleweed.It is quite
frustrating if you try to learn a new language, and when you make
your first step, you step on a nail.And it is even more
frustrating, because Pony looks very interesting, for the reasons
the linked article explains. ):UPDATE: On Debian, telling the pony
compiler to --pic does the trick.UPDATE2: On suse, installing
binutils-gold solved the problem.UPDATE3: Now we're talking! This
language is very, very interesting, indeed!
aturley - 4 hours ago
Sorry to hear that. As far as I know the documentation should be
up to date, but it's entirely possibly you've found an issue.If
you like you can hit up the mailing list
(https://groups.io/g/wallaroo) or the IRC channel (#ponylang on
freednode, https://webchat.oftc.net/?channels=wallaroo) to see if
anybody can help, or if you think you've found a bug you can file
an issue in github
(https://github.com/ponylang/ponyc/issues).UPDATE: I'm glad to
hear you got it working!
krylon - 3 hours ago
> I'm glad to hear you got it working!Me, too! This looks
pretty exciting!
spooneybarger - 3 hours ago
I'm often lurking on IRC if you need help.
anonacct37 - 4 hours ago
Pony looks really interesting and I look forward to trying it out.
I can't help but try to reason about how my work language would
handle these problems. Go.Highly concurrent: check.Predictable
latencies: I think that's a check, the go gc has had alot of
engineering put into it to provide good pause times. There are some
edge cases still.Data safety: I think go is weakest here in terms
of language design. Go does have go build -race which works pretty
well for me, but I can see how some wouldn't consider it
sufficient. It also wouldn't catch single threaded ownership
problems, which can happen.Easy way to interact with other
languages: via CGO and buildmodes you can trivially call go from
other languages. Some don't like that the runtime is still shipped
and started, but the fact remains that you can cffi go functions
from python very easily.I'll have to check out pony this weekend.
I've read up on it a bit but haven't compiled anything.
aturley - 4 hours ago
That's great! We're collecting stories about peoples' first
impressions of Pony (https://www.ponylang.org/categories/my-
first-pony), so if you're interested in contributing to that
please get in touch.I'm an engineer at Wallaroo Labs and I've
been learning Go so that I can add a Go API to Wallaroo. Your
analysis looks pretty spot-on based on my experience.The
buildmodes is go definitely make it easy to call Go from other
languages. The trickiest thing I've run into so far with calling
Go is that you aren't supposed to hold on to Go pointers outside
of Go code, so I'm having to jump through some hoops to hold on
to objects between calls.I have an RFC for Pony that I need to
finish up that will let you call Pony function from C. I'm
planning on getting around to that soon, as I think it will help
improve our FFI story.
doall - 5 hours ago
JVM GC is an interesting topic since recently there was a proposal
for the ZGC project in the OpenJDK mailing list.
http://mail.openjdk.java.net/pipermail/announce/2017-October..."ZGC
has been designed with the following goals in mind:- Handle multi-
terabyte heaps- GC pause times not exceeding 10ms- No more than 15%
application throughput reduction compared to using G1"
Mikeb85 - 6 hours ago
Really good post. Haven't heard much about Pony but seems like a
really interesting language and like something I could use for a
particular project.
spooneybarger - 6 hours ago
Thanks.If you ever want to talk Pony, you can find me lurking in
the Pony IRC.
tytytytytytytyt - 17 minutes ago
Are there any statistics that compare this to some similar existing
software?
jorgeleo - 3 hours ago
From Pony documentation:SimplicitySimplicity can be sacrificed for
performance. It is more important for the interface to be simple
than the implementation. The faster the programmer can get stuff
done, the better. It?s ok to make things a bit harder on the
programmer to improve performance, but it?s more important to make
things easier on the programmer than it is to make things easier on
the language/runtime.This is an excellent design decision. Yes,
performance sometimes introduces complexity, and I have been in
teams where the philosophy is: "It does not matter if it is slower,
but any high schooler that knows HTML has to be able to understand
it". The purpose of a program is to satisfy the users, not the
developers or the managers, and if the next developer needs to
study a thing or two before understanding the code that is
OK.Simplicity should be sacrificed for performance.
spooneybarger - 10 hours ago
Author here. Small bit of background. I'm VP of Engineering at
Wallaroo Labs and a member of the Pony core team (many folks at
Wallaroo Labs are now actively involved in the Pony
community).Happy to answer any questions here or if you prefer, via
email:sean@wallaroolabs.com
snappyTertle - 6 hours ago
Why did you choose Pony over something like Erlang or Elixir?
spooneybarger - 6 hours ago
I want to preface this with: I love erlang. We use Elixir for
the service that powers metrics display for Wallaroo. When I
learned Erlang several years ago, it made me a better
programmer.At the time we made the decision, we were worried
about Erlang being able to meet the latency and throughput
goals we had. I knew a number of people who worked on Riak at
Basho and had a few lengthy discussions about Erlang
performance. They felt that we could end up struggling to get
the performance we were looking for.I had never used Erlang for
any large scale project and deferred to their wisdom and
knowledge.As it turns out, the approach that we've used to
support languages like Python is in process, using C to bind
Wallaroo with the other language. This probably would have been
more difficult with Erlang as well, but that's hindsight.Erlang
is an awesome language. It has an amazing VM. That we didn't
think it was right for us should in no way discourage anyone
else from using it.
fasquoika - 5 hours ago
Is your application CPU bound? Because that's generally where
Erlang's performance is lacking. I'm pretty sure you wouldn't
run into performance issues in Erlang if you're doing an IO
bound task
spooneybarger - 5 hours ago
Wallaroo is a framework for end users writing applications.
It is not an application itself. Many of the use cases that
people come to us with are very CPU intensive.
fasquoika - 2 hours ago
Thanks, that makes sense
MrBuddyCasino - 2 hours ago
I?ve worked with Scala/Akka, and while I like the actors model,
some things turn me off about the stack, mostly the
language.After reading the Pony guiding principles, I was
delighted. This looks like an implementation where the ecosystem
complexities don?t get in my way of ?getting things done?!
Cyph0n - 36 minutes ago
May I ask what turned you off about Scala?
kevinconaway - 6 hours ago
Thanks for the writeup Sean. I've been interested in pony since
I read about here on HN a while back.I have a question on this
quote> The standard JVM garbage collection strategy is ?stop the
world.? That is, when the JVM needs to reclaim unused memory, it
needs to pause all other processing so it can safely garbage
collect. These pauses are sometimes measured in seconds. That is
going to destroy your tail latencies.Thats why applications
rarely use the default serial collector.Does your comparison
still hold against the CMS and G1 collectors, which do a much a
better job at eliminating "stop the world pauses"?
spooneybarger - 5 hours ago
Thanks.At my previous job, we used the G1 collector and still
had issues with "stop the world" type pauses. G1 does a best
effort, but I've had applications that still experience rather
long pauses (sometimes measured in seconds, usually hundreds of
microseconds).I haven't used either CMS or G1 heavily since I
joined Wallaroo Labs a couple years back so, I don't have first
hand experience with either recently.Azul's Zing JVM does a
really nice job of concurrent collection and if you can afford
it, is a great way to improve the performance of clustered JVM
applications.
ZenoArrow - 6 hours ago
Have been impressed with Pony the few times I've seen it on HN,
thanks for your work on Pony and Wallaroo.I'm guessing that
tooling for Pony still needs some refinement, but as things are
now what would you recommend to someone new to Pony (e.g. editor,
package manager, build tools, debugging tools)?Oh and just out of
interest, did you evaluate Rust as a language for Wallaroo? I do
think Pony has the potential to be stronger in certain areas, but
it seems like there's a certain amount of overlap in terms of use
cases.
jtfmumm - 6 hours ago
I'm an engineer at Wallaroo Labs and I've been writing Pony for
the last two years. We're using pony_stable (which is the
primary package manager at the moment) and make for managing
builds. I currently use Sublime 3 as my editor and lldb for
debugging. I know that others are using emacs and vim as
well.We did consider Rust as there are certainly overlaps in
their approaches to safety (though coming from different
angles). In the end we thought that Pony would be better for
managing concurrency for our particular use cases, which we
thought mapped well onto the actor model.
kodablah - 6 hours ago
> we thought mapped well onto the actor model.Out of
curiosity, were there a widely accepted actor lib/approach in
Rust (I know there are a few like RobotS and others), would
that have affected your decision?
spooneybarger - 5 hours ago
It might have. When we were looking at Rust, it was a topic
of conversation.
steveklabnik - 5 hours ago
In general, it's not a space where there's a hyper-mature
library, but there are some cool stuff happening today:
http://cityboundsim.com/devblog/my-full-rustfest-talk-
with-n...That said, I mean, if you want to go full
actors, Pony is a great choice.
Cyph0n - 4 hours ago
Oh wow, I used to follow Citybound development a while
ago when it was still written in JS. Anselm is an
amazingly talented developer and it's great to see him
working with Rust!
aturley - 5 hours ago
I'm also an engineer at Wallaroo Labs. A while back I wrote up
some information for people who were interested in Pony:https:/
/gist.github.com/aturley/49b60c98306d90ffc2f981515827...I've
been using Emacs as an editor. We use a combination of make and
pony-stable to build projects. For package management we use
pony-stable. LLDB is your best bet for debugging, and if you
follow the link above there's a link to the pony-lldb project,
which is an LLDB extension that makes it easier to work with a
few Pony datatypes.
jack9 - 3 hours ago
How do you ingest data with Wallaroo? I see a mention of "...say
TCP" and in the github word count there is mention of a
(data)source framed-message-protocol but nothing that describes
it in the
(https://github.com/WallarooLabs/wallaroo/blob/0.2.0/book/cor...)
link.Dropping Python into the mix for a high throughput
processing pipeline seems counterproductive. Why isn't there a
tutorial in a more strict language like Go or C++, since someone
pursuing this kind of high-throughput framework that can't
benefit from wider parallelism, will 100% want that performance
guarantee?
spooneybarger - 2 hours ago
Ingestion:Kafka and framed TCP are the two ingestion sources
that Wallaroo currently ships with.Python:We added a Python API
because there was a lot of interest in it. There's more to
Wallaroo than just performance and folks were interested in
having it available via Python. See https://vimeo.com/234753585
for some more information on the "scale independent" nature of
the Wallaroo APIs.Go:We're working on that right now actually.
Planning to release later this year.C++:We had a C++ API (still
do), we aren't currently supporting it. There was limited
interest at the time. If folks show interest we would start
supporting it again.
model_m_warrior - 6 hours ago
What benefits does Pony have when compared to Erlang?
spooneybarger - 6 hours ago
From the Pony side:Biggest strength would be performance.
Biggest weakness would be maturity. I think almost every
pro/con I can think of can fall into those 2 buckets right
now.I'm a big fan of type systems so Pony having one is a big
win for me.The maleability of Pony and its immaturity helped us
in some ways. We were able to treat it as a runtime for us to
help mold and fit to our needs. That wouldn't have been
possible with Erlang.
model_m_warrior - 5 hours ago
Are Ponys classes akin to Erlang modules or does it take a
more enterprise-y perspective on OOP?
aturley - 5 hours ago
I'm not sure which features of Erlang modules you're
interested in comparing to Pony classes, but I'll take a
stab at trying to answer.Pony classes are defined using a
"class" keyword. Classes have properties and functions,
where functions are like methods that you would find in a
language like Java. Pony supports structural subtyping via
interfaces and nominal subtyping via traits, and it
disallows multiple inheritance.I'm not sure how much that
helps, but if you have more specific questions I'd be happy
to try to answer them. I'm a little rusty on my Erlang, but
hopefully it will come back to me.
scottlf - 3 hours ago
Howdy. Erlang's modules are involved in code namespace for
the compiler & runtime (i.e., the 'M' of the 'MFA' triple
of Module + Function name + Arity that names a function) as
well being the unit/scope/bound for hot code loading (i.e.,
you must load or unload an entire module at a time).I've
worked in Erlang-land far longer than I've lived in any
OOP-land, so I'm not sure what you mean by enterprise'y
OOP. Coincidentally, Pony's rules for "Packages" are
something I just smacked my ignorant head against a few
hours ago. The subsections of
https://tutorial.ponylang.org/packages/ in the tutorial can
probably answer at least some of your question:
specifically "Package System" and "Use Statement".
pdimitar - 2 hours ago
I'll take the liberty to speculate and assume your parent
comment was asking if Pony classes have the same features
as Java classes -- namely class-level attributes,
instance-level attributes, different access levels to
attributes and methods. Those things.So, does it?
spooneybarger - 2 hours ago
Pony classes have methods. Pony classes have
fields.Fields and methods can be public or private.
Private is akin to Java's package private.There are no
instance variables.
59nadir - 4 hours ago
> Biggest weakness would be maturity.Has there been any
progress towards a preemptive scheduler? Otherwise, I'd think
that's the most obvious weakness.
spooneybarger - 3 hours ago
There's pros and cons to preemptive and cooperative
scheduling policies. I wouldn't be comfortable labelling
either as weakness. It really depends on context.I've heard
rumor of someone working on a preemptive scheduler. The
current cooperative one is about to have runtime
backpressure added to it which is going to be a really nice
scheduling win.https://github.com/ponylang/ponyc/pull/2264
59nadir - 2 hours ago
For the kinds of systems that are built with Erlang,
which is a niche that Pony seems to want to occupy, I
would argue it's inherently better to have preemptive
scheduling. A cooperative scheduler will always be an
open invitation for bugs relating to CPU hogging. Not
having to worry about these things is priceless.
jdp23 - 2 hours ago
Just wanted to say, thanks for the excellent article and the
answers to questions here!
spooneybarger - 2 hours ago
Thank you!
charlesetc - 3 hours ago
I used Pony and bindings to opengl to write the (very) start of a
graphics engine. http://www.charlesetc.com/stars-game-7.htmlIt was
surprisingly easy to learn - I found the capabilities system rather
intuitive compared to other methods for safely managing data races.
Also the syntax is really simple and easy to read!
[deleted]
scardine - 4 hours ago
For a moment I thought this was about the Pony ORM (Python) - looks
like there are quite a few popular projects named Pony, perhaps we
should use "Pony Language" instead of just "Pony" for the sake of
disambiguation.
[deleted]