Functional programming has been influencing mainstream languages for decades, making developers more efficient whilst helping them reduce maintenance costs. As we are faced with a programming model that needs to scale on multi-core architectures and distributed environments, concurrency becomes critical. In these concurrency models, immutability, a key feature of functional programming paradigm, will become even more evident. To quote Simon Peyton Jones, future concurrent languages will be functional; they might not be called functional, but the features will be. In this talk, we explain why!

Q&A

Question: Has Akka within the JVM realm got every essential Concurrency traits from Erlang. Have there been any mass migrations in enterprise world to Erlang in the recent past?

Answer: No, the enterprise world is historically very slow at responding to these challenges. At the same time, they are also solving a different problem which might not require scalability and reliability. The adoption of Scala and AKKA has shown there is a need for immutability. But with multi-core architectures just beginning to make their mark, they are trying to address it through containerization (which is a form of immutability in itself). Personally, I think Paravirtualization, where we remove any abstraction layers and run the VM as close to the hardware as possible is the future, with orchestrators starting and stopping them automatically.

And to quote Simon Peyton Jones, future concurrent languages will be functional. They might not be called functional, but they will display those features. This was at his Erlang Factory London keynote, also in 2009!

Phoenix 1.5 is here and it comes powered up with out-of-the-box instrumentation and visualization thanks to Telemetry and Live Dashboard.

Phoenix now integrates Erlang's Telemetry library to aggregate and report on standard Phoenix, Ecto and Elixir VM Telemetry events as well as any custom events you'd care to emit from your own application. The Live Dashboard library allows us to visualize the metrics, performance and behavior of our app, as described by these events in real-time. These two offerings together empower every Elixir developer to hit observability goals by writing and shipping fully instrumented code with ease.

In this talk, we'll take a tour through Live Dashboard's usage and features and we'll dive under the hood to understand how it leverages Erlang and Elixir's Telemetry libraries to capture and visualize events as metrics. There will be an obligatory picture of Frankenstein.

Q&A

Question: Can you aggregate metrics of multiple phoenix instances behind a load balancer in the dashboard?

Answer: LiveDashboard does recognize the node from which metrics/events are being collected. I'm not sure if it has out-of-the-box support for aggregating events across nodes though but since nodes are treated as first class citizens by LiveDashboard, I have a feeling you'll be able to find out more in the docs.

Question: If you're in production, what about the privacy aspects, in that some of the telemetry might expose sensitive user data. Could there be some general mechanism for "tainting" some data as sensitive, so that it's handled differently, maybe with suitable anonymization or such? It would at least (in some circumstances) take off from the developer the burden of worrying about user privacy.

Answer: First off, I would definitely recommend putting your LiveDashboard routes behind authentication. I don't think LD has support for anonymization/obfuscation at this time. Some discussion of auth in the docs here https://hexdocs.pm/phoenixlivedashboard/Phoenix.LiveDashboard.html#module-extra-add-dashboard-access-on-all-environments-including-production

Question: It's more than just authentication, though. Just a thought. I guess sometimes for monitoring you don't need to know all the (possibly sensitive) details. Another project... A well-conceived framework to filter and aggregate appropriately what's shown to developers.

Answer: yea I think this is a bigger question on the topic of observability. at GitHub for example we have a hand-rolled whitelisting system that ensures we don't send overly sensitive data to exception reporters like Sentry.

Question: From my first impression it looks focussed on real-time telemetry, ie what’s going on right now. But is it possible to go back in time with LiveDashboard to see data from previous days/weeks?

Answer: in fact, there is a VERY new addition to live dashboard which deals with metrics history! https://hexdocs.pm/phoenixlivedashboard/metricshistory.html#content

Answer: Im looking through the docs here <a href="https://slack-redir.net/link?url=https%3A%2F%2Fhexdocs.pm%2Fphoenixliveview%2Ftelemetry.html">https://hexdocs.pm/phoenixlive_view/telemetry.html. I feel like that would have to involve the JS side of things though.

Question: Anyone have a recommendation for a metrics reporting system that can show frontend, backend and even higher level business events?

The closest I've used was Azure AppInsights - but it wasn't exactly fun to use

Answer: this doesn't quite fit the bill but we use Lightstep for distributed tracing at GitHub https://lightstep.com/

Software systems form an intrinsic feedback loop, the more we use and rely on a system, the more we demand of that system. This loop drives a cost and complexity that if left unchecked eventually inhibits growth and improvements to the software or in the worst case brings it crashing down.

The crux of this complexity in many (most?) systems is the management of state and data over time, and importantly how different systems co-ordinate around that state. If we look to modern, and even not so modern, software we can see a recurring pattern to dealing with this through the use of immutable, persistent data. We see this pattern from databases and distributed systems through to user interface frameworks. Building complex systems that work correctly, reliably and efficiently often comes down to building systems that don't forget.

In this talk we will look at the design, benefits and challenges of building a system around the ideas of immutable and persistent data using a specific example of a system that I work on to support urban planning & climate policy modelling.

Q&A

Question: #[derive(CacheHash)] looks interesting. Is that something internal to your codebase or is there a cool crate I'm not aware of?

Answer: Yeh, it is internal, using custom derive. But there are a few similar style crates around I think.

https://doc.rust-lang.org/reference/attributes/derive.html

Question: Question to anyone that knows… I'm not extremely familiar with Zippers. Am I right in saying that it's a data structure with a focus, and a collection of things to its left, and a collection of things to its right?

How does rewriting a Tree into a Zipper flip the arrows?

Answer Thread:

David Overton

A zipper doesn’t flip all the arrows, just those pointing back to the root of the tree.

Alexey Kotlyarov

I kind of get this, but I'd still like to see a type definition in Haskell/Rust/whatever

particularly how would you use it to patch a structure

George Wilson

I did a talk on this once https://m.youtube.com/watch?v=woK7ntZRwXQ

I don't remember if this talk was good; YMMV

Ed Kmett

Alexey: think of a data type describing a 'path' down into the tree. if your tree is a binary tree, at each node you could maybe stop there, go left or go right, if you go left, store the sibling and maybe the value on the current node, if you go right store the other sibling and the value, if you stay you're at the leaf. So now you can represent a tree with a selected element as a path + the node at that point. if you have an octree, then you tell me which child you walk into and all 7 of the other children. the zipper for a data type will depend on the recursive structure of it.

David Overton

I get all that (also did a talk on it once :) ), but it didn’t look like what Mark was doing.

Ed Kmett

Well, we can come up with a different way to store the spine, e.g. you can store the entire path as a flattened array. http://ekmett.github.io/transients/src/Data-Transient-WordMap-Internal.html#TWordMap

That is a 'word map' which is like an intmap, but modified to have a finger at the last accessed element. (it uses a funny trick i posted at some point to get away with fewer operations compared to an intmap) the finger makes it so you get a nice fast access near where you last accessed it

Tony Morris

I start with an (very loose) initial intuition for "make a data structure, make it physically, such as a list, now go stand on an element, look around, what do you see?" Well, you'd see elements to your left, elements to your right, and the element you are standing on. Now, what about a tree? Make a physical tree, go stand on an element, look around, what do you see?

Sanjiv Sahayam

it’s also covered in LYAHFGG: http://learnyouahaskell.com/zippers

Alexey Kotlyarov

So let's say I have a Node (Leaf 1) (Node (Leaf 2) (Leaf 3)) , and I want to replace 2 with 22, what would would (zipper versions of) both trees look like? I'm assuming they'll somehow still be useful as trees afterwards.

Tony Morris

loosely, toZipper > moveRightSibling > setFocus(22) > unZipper

You can calculate the zipper for a data structure by taking the partial derivative, if that helps; I have docs on it somewhere.

e.g. d/da. List a comes to List a * List a (focus is missing)

Mark Hibberd

For slightly additional context on my slides, the structure in rust is far from an immutable/persistent structure, and hence the hand-wavy nature of the explanation which could of been better, was trying to just provide a rough analogue for what it looks like from the outside of the system. +1 for Tony's explanation above for zippers.

Alexey Kotlyarov

Ah, I think I misunderstood that part in the talk then. I thought using zippers somehow achieves even greater sharing than possible with the regular tree structure.

Mark Hibberd

Yeh, apologies, I kind of fumbled the explanation a bit. The crux of the technique that it uses is to store a densely packed array of the path to nodes, the nodes contain references to the array rather than other nodes, such that you can efficiently do copy-on-write of that array changing a node without having to rebuild all the nodes (as the pointers between nodes are indirect, kind of like names). The way it is designed, relies on some overlap in techniques/concepts, and I made the mistake of being a bit fluffy in the comparison. (edited)

Question: why Rust in this context? What made it a good fit?

Answer: Query performance, I didn't get time to go into it in much detail, but the reality is that I didn't want to (and couldn't) spend very much time on it. I spent 3 days, building the query engine 18months ago, and have not had to worry about it and been able to concentrate on more important things. In most other situations I would of had to spend a lot of time tuning to get where I was with very little effort (partially because of libraries - decent enough array/simd support etc..., partially because I could write what I needed fairly explicitly).

Something about, the biggest bottleneck to performance is freeing up your time to optimise. And it had a good balance for this particular problem (and a decent enough fit for the team skillset).

Question: Do you have some particular query mechanism like Datomic, a loosely similar system that claims to never forgets, uses an implementation of Datalog ?

Answer: Not really, it far coarser. Queries are effectively parameterised by the scenario identifier + the table identifier, that will always point at the same data. The data-set it points to can be large, and of various different shapes (graphs, tables, vectors), and isn't all indexed by time as facts, but is a value that is repeatable as a whole.

Elixir is often described as a language which offers great support for massive concurrency. First-hand reports cite the ease of handling millions of connected users, sub-millisecond response times, and superb fault-tolerance. These are all great benefits, but we’re left wondering whether Elixir is useful only for large scale systems, or can it bring some benefits in the simpler cases too?

This talk aims to demonstrate that Elixir is also a great choice for building smaller systems. Through a very simple but still a real-life example, I’ll explain how using Elixir can help simplify the system architecture, and lead to a more homogeneous solution. The talk targets backend developers who are new to Elixir. After the talk, the audience will have a clearer idea about what makes Elixir attractive, and why should they consider using it to build their next backend system.

Q&A

Some Links:

https://github.com/sasa1977/erlangelist/#running-the-site-locally

The library for Let's Encrypt certification in Elixir: https://github.com/sasa1977/site_encrypt

Question: Do you have a process for migrating a system like you described to something with high availability? Or in your minimalist approach where do you start when the system needs to be more robust to failure in the underlying infrastructure?

Answer: There are many options, and I prefer to start as simple and as lightweight as possible. For example, for some basic fault-tolerance WRT to machine failure, it's enough to have some failover mechanism, which can be as simple as using heroku, or otherwise something like k8s running a singleton version of the system.

Obviously this is not going to suffice in all scenarios, but it's also a simple enough start which is going to be enough in some scenarios. Further increase of availability will require various techniques depending on the particular case.

Question: Have you run elixir in a clustered setup in production? I think a lot of people (us included) run phoenix behind a regular load balancer and a shared nothing architecture, which means you sadly cannot really use the awesome process model very much since non-clustered processes cannot talk to each other which is a bit of a shame.

Answer: The distributed BEAM is easy to setup, though in my view dealing with a distributed state has many challenges, so I like to avoid it as much as I can. That said, the abstractions in beam ecosystem are progressing nicely, and I'm particularly excited about the ra library (https://github.com/rabbitmq/ra). I feel that this could be a good building block for obtaining more of k8s-like service directly in beam.

In addition, I've started some experiments on a native Elixir load balancer (basic TCP passthrough). In general, I'd like it to be easy to build a small-to-medium distributed stateful fault-tolerant system with SQL persistence using nothing but Elixir (or any other beam language).

Question: i’m curious to learn some of the challenges you’ve faced dealing with a distributed setup.

Answer: Consistently replicating state is IMO the most difficult challenge, and the ecosystem used to be somewhat lacking in that regard. At some point I fell back to using a database as the source of truth as much as possible, because it's simpler to reason about. That said, the story definitely seems to be improving, with abstractions such as Phoenix Tracker, and the aforementioned ra library.

Question: I know Akka and Microsoft Orleans have had considerable success running distributed actors in complex systems, maybe there’s a few things Elixir/Erlang can learn from them in return (wink)

Answer: Yeah, I believe there's definitely potential to take some ideas from these techs. There is erleans (https://github.com/erleans/erleans) which aims to bring some ideas from Orleans to beam.

Question: I personally really like your "Just in Time"/DIY approach! But many times on the job I had trouble defending a homemade solution against a traditional one. Sometimes I was taken as "Afraid of the technology" or something like that. Do you have this similar problem? How do you overcome the complexity addiction of our industry?

Answer: Yeah, I've had similar experiences, and sometimes I find it difficult to argue. I feel that as an industry we became too dependent on out-of-the-box available solutions. I found that the most effective way to battle this attitude is to build a small incomplete demo, and try to explain the gains we can get if we roll our own solution.

To be clear, I'm certainly not opposed to using 3rd party products or libs, but I like to be a bit more critical and ask myself is it solving a real difficult problem for me, or can I get the similar kind of properties if I write a bit of code, and can this hand-made code actually simplify things for me. Sometimes the answer is yes, and other times it's no.

Question: Have you ever worn any serious pain because you've gone with an erlang native solution and needing to back pedal to something mainstream and non-native? I mean, that's always going to happen now and then, but any significant war story?

Answer: To be honest, can't think of a single case. In general, if a simple solution takes me where I want to be, and performs fine for some amount of time (e.g. a few months or more), then I think it was worth it even if I have to fallback to something else. To me it's just incremental/evolving system architecture. Start simple, move to more complicated when there's real need.

Question: Simplicity paying off is something that I can buy for sure as long as the system is observable enough to understand even though it's unique and unfamiliar (compared to using COTS devops things). Being able to prototype and move fast especially at the start of a product makes or breaks things. This feels like it grows out very nicely if you ever outgrow the mvp. That graph of the supervision tree in your blog is way more understandable than any devops thing I've ever seen. So it seems like you win a lot at the observability.

Answer: Yeah, I find it very intuitive to reason about supervision tree, and most importantly, it's one thing you learn and apply it at all levels of granularity.

Question: Are you using boundary (https://github.com/sasa1977/boundary) in production / commercial team setting?

Answer: I've added it experimentally to a small project of my client. Had no problems so far, but the project is still quite small to provide a reliable assessment. Boundary is more about in-app deps, though it has some support for cross-app deps too, like e.g. you can prevent calling Ecto from web modules, or calling Phoenix from contexts. You can even allow Mix to be used only at compile-time. The point is that boundary will emit a compile-time warning, whereas with releases using mix at runtime will break at runtime (so only after you deploy it).

Question: Did you have any real case number about how much load you had to migrate to a 3rd party solution? Of course it depends a lot on the details of the project, but just to have an idea of when it is time to change. Which are the metrics that lead you to make the change? Which 3rd party solution was that?

Answer: Not really. To be clear, in the past 10 years of using beam languages, I never had a need to migrate away from a hand-made solution. Admittedly, I didn't work on systems the size of Netflix, Twitter & such, it was definitely smaller scale in comparison, but still I believe that this is a practical testament that we can go really far without using this heavy 3rd party machinery.

In this talk Ed will give live coding introduction to normalization by evaluation. He will then show how Graal and Truffle, on the JVM, can be (ab)used to JIT functional languages. He discussesd why this seems like a promising direction for evaluating dependently typed languages in particular.

Q&A

Question: How usable is this for writing a real program today?

Answer: It is certainly more usable than the naive evaluation strategy.

Question: What does this mean for the overall Coda project going forward? Can we get any sort of status report or hints where it might go next?

Answer: A lot of my work lately has been on probabilistic programming, and using it to make a nicer equality saturation engine

Question: What extra tricks could a JIT built for a functional language expose?

Answer: Well, the bits I didn't get to really share are the examples of how it has to trampoline.

What I wind up doing is each call in tail position starts a "potential" trampoline, which is to say that the evaluator puts a little exception handler in place that handles tail call exceptions that make it through, but then what i do is i start a bloom filter of what closure bodies i've seen so far on the call chain locally, turning the java hash function into a little Int64 sized bloom filter with ~5 hashes in it.

The idea is that if I go to enter a closure I throw if the bits for the current closure are set in the filter and it handles it by turning it into a loop if it actually is in the local stack, and hitting the fall back trampoline in the cases where i got a false positive.

So that is trampolining. For polymorphic inline caching, each 'app' node becomes a little polymorphic inline cache of what closure body is 'expected' for the function. if it hits you get to pass the arguments directly in registers, etc. by the time it gets through the assembler, and it builds PAPs like GHC when there aren't enough arguments.The PIC is a bit strange because its indexed by arity of the functions expected, and turns into expected chains of result closure bodies in the presence of overapplication. e.g. if you call foo x y z -- and foo always evaluates to a 2-ary function with a particular closure body, it'll basically inline the whole thing right there, and then figure out what it expects the 1-ary result to be and do the same there using a nested PIC. The main headache is that the trampoline story wound up being way messier than i'd hoped. I'd originally wanted to have it gradually install better trampolines. e.g. check for self-tail-calls, check for self-tail-calls with a different environment, check for the need for a full trampoline, and degrade gracefully. But that didn't work well with the "partial escape analysis" pass that is used by truffle/graal to avoid actually passing arguments so i'd always wind up building a big object[] array with all the arguments, putting everything into boxes and my memory usage pattern was awful and it was doing all sorts of extra work the current bloom filter craziness was a compromise in theory a better functional jit would be able to avoid that tax.

Question: Are you using Haskell for that or python/Julia etc

Answer: I've been bouncing back and forth between haskell and rust for a lot of this. In particular looking at rust for representing the database that is my crazy syntax tree for equality saturation and the rule engine, but all my probabilistic programming work has been in haskell so far.

Question: You mentioned not quite getting the performance you were hoping for, but can you say something about the performance you did see?

Answer: Well, It was a few x faster than a fairly naive evaluator, but also a few x slower than the direct 'lets just write the example in java' test. I was hoping to much more cleanly lean towards one side or the other. i was like if i get 10x its a win, if i get 2x it’s a loss, getting 5x it's a meh =P - what i would like to see is a similar framework for something more haskelly, but i'm not sure how it'd work.

Question: What parts of probabilistic programming are you looking to use/improve? I vaguely remember in a guanxi talk you briefly mentioned something about pumping random bits into an arithmetic decoder to do sampling without replacement in succinct space(?), but you didn't explain.

Answer: That thing didn't work out so well in the end, it really got in the way of variable reordering. Unpacking the probabilistic programming thing a bit: step 1 is using control/antithetic variates for HMC, which Piponi wrote a nice paper. This has been really useful. But in general, there is a pretty straightforward way to make a probabilistic programming monad that uses 'scores' and replace it by traced MCMC with one that doesn't. If you make those scores use AD you can use HMC on that monad rather than naive MCMC. But this can be done in multiple layers to enable you to compute the expectation for the control variates.

To explain what I mean: if f is a function of a random variable, and g is another function of that same random variable we can look at something like

f(X) + b (g(X) - E[g(X)])

for some constant b

E[g(X) - E[g(X)]] = 0,

so this has the same expected value as f(X), but if f and g are correlated this can have much lower variance. But I can replace E[g(X)] with an approximation of it produced by another layer of MCMC/HMC and you just compute the b that minimizes variance, which is a little exercise in algebra. Now this works for multiple g's in parallel or nested. Basically as a way of taking crappy models and using them to improve the variance in better models. But the trick is that bayesians in general ignore computation time. So you spend time proportional to the relative computational cost working on improving each estimated E[g(X)] vs. computing the raw f(X) function. When mixed with HMC, this is a pretty good 'universal evaluator'. Next step is how to apply that to equality saturation. But that centers on rethinking equality saturation as a series of mutations, MCMC like moves, that turn one syntax tree into another. You could think of each rewrite rule as a bidirectional mutation that you might apply.

Now if you do this naively, it's strictly worse than equality saturation. You basically are doing random walks through program space and then judging based on whether it improved some theoretical or actual measurement of performance taking all that make it faster, and enough that make it slower to maintain detailed balance. But what I want is something more like equality saturation than this sort of STOKE-like mutation procedure. With equality saturation you just keep adding new facts to the database, and proofs that things are equal, and hope the unifier puts an amount of pressure that balances the work of the rewrite rules expanding the terms. Then pick the 'best' program using a crappy metric.

Here I want something more like an ensemble, where i have a batch of equivalent programs, but NOT all of them, and then transform the rewrite rules into things that either add or REMOVE (by running them backwards) programs from this set. Then when i pick one out of the 'swarm' (using probabilistic programming, again!), it basically lets me weight the rules based on how much they help in practice. The nice thing is the longer you run, the faster the code gets and the more I learn about the weights to apply. At least in theory. The headache is making the right kind of database for storing this, computing detailed balance calculations because every edge has to have an inverse lest I lack proper support, etc.

That is the gist of it anyways not sure how coherent that was.

Question: Lots of food for thought at least. Sounds kind of like primary sample space Metropolis light transport.

Answer: MLT is a huge influence on me. In particular all the recent Bitterli work where he managed to show how to use reservoir sampling and the like in an unbiased way that I just never would have thought could be made unbiased. The other bits are to use something called product sampling and variations on multiple importance sampling as yet more ways to take crappy approximations of a space and get something better. Each of these, control/antithetic variates, product sampling, MIS, all sort of hit the same space. In particular for a cheap source of crappy models, there is a nice paper I keep playing with: http://ddg.math.uni-goettingen.de/pub/GeodesicsInHeat.pdf

Discretize the space, use that to reconstruct geodesics and use that to move around, rather than the discrete approximation, super cheap to evaluate, but not as bad as the naive discrete version.

I do love that the probability theoretic equality saturation engine doesn't require me to figure out how to make my rules terminate. I just run, and get better and better programs over time, the idea being that the expectation of any individual program in the 'swarm' being in the swarm and then being selected should match the detailed balance criterion, it's a weird form of 'slightly dependent' replica exchange. Making the math work there has been where all my headaches have kicked in… what I'd kind of like is a compiler that is well suited to doing little bits of AD on parts of it, setting up these little residuated probabilistic programming, and also handle things like the heavily data-layout dependent stuff like how to store the database and implement the not-quite Rete network of rules. Then to implement all of this in that… but "there is a hole in that bucket dear liza dear liza" as the song goes.

One thing i completely abjectly forgot to mention is that the approach to NBE there is almost entirely taken from David Thrane Christiansen, he has a paper that covers pretty much the same code I livecoded.

I'm usually over on irc.freenode.net in ##coda and #haskell so assuming i didn't manage to answer all questions for all time, feel free to bug me there.

I talk about the history of how Scala's implicits evolved, about some of the mistakes we could have avoided in hindsight, and about aspects that I believe we got right. I then present the new system of givens in Scala 3, which replaces implicits. This system fixes most of the existing issues, increases safety and predictability and offers some exciting new possibilities to structure and modularize code.

Q&A

Question: I have to import an instance. With given instances, do I still need to import the instance? In my scala coding, I have spent a lot of time looking for the right implicit to import.

Answer: You still have to import it, but the compiler will suggest while instance(s) to import if they are missing.

Question: “What problems would you specifically use Scala to solve?” what would be a useful answer? I’m just wondering what niche (if any) Scala occupies at the moment.

Answer: I don't think Scala 3 is competing directly with other languages. We are all trying to figure out how to make functional programming work as best as possible in practice, which means we also need some approach to do modules. There are different ways of doing it.

The question is more: what is Scala not used for?

Question: What’s the current suggested solution to the Functor , Monad, Traversable example? I’ve been hit with that quite a lot in the past. Is the scala 3 suggestion to summon a locally scoped given and have the nesting rules ensure that’s the one used?

Answer: Yes

Question: Are given blocks legal in all places that a def would be legal?

Answer: Yes, given blocks are legal everywhere defs are

Question: Do you see any chance in the future to implement an effect system (IO) in the standard library (similar to cats-effect, ZIO) ? and what about testing, is there a chance for a standard framework?

Answer: I believe effects are the next big frontier to tackle. There are lots of interesting developments. My prediction is they will look in the end quite different from what we have today.

Question: What kind of 2 -> 3 migration curve in industry are you expecting after release?

Answer: Migration would take several steps, typically. 1. Cross build some libraries. That's what's happening now. Hard part for this is macros. But we have now a way to write dual macro implementations; one for Scala 2 and one for Scala 3 that are picked up by the respective compilers. The way to define an invoke a macro are different, but underneath they can share code. After that, I believe we will see migration to Scala 3 features only using rewrite rules. By Scala 3.2 to so, we'll switch everything over to Tasty

Question: If you were to redesign scala all over again, with no encumbrances, cleanly, would scala 3 be it? if not what would take to be it?

Answer: I'd drop implicit conversions entirely (maybe we can get there, there's a thread on Scala contributors). If Java interop would not be an issue, I'd also drop some other things. F-bounded polymorphism could be dropped then. That's the one that causes the most grief for the compiler.

Question: But not given/using?

Answer: given/using is core of Scala. I am very happy with the design.

Question: Any plan to enforce some global coherence? I vaguely remember there were a discussion regarding coherence in Scala 3 a while ago.

Answer: I don't see a way to do global coherence that would not be a pain. So, no immediate plans.

Question: Any thinking around if/when experimental support for Valhalla project in Scala? (value types for the JVM)

Answer: Let's see when Valhalla is released and what is released. We take a look at it then and see whether we can work with it. Same for Loom.

Question: Did you have to leave out any controversial features in Scala3? If so will there be a Scala4 at some point?

Answer: The one thing I backed away from because it was late and people were up in arms about it was use := for assignment. But it might be too late to change it in any version.

Question: SWOT analysis for Scala?

Answer: For a SWOT analysis we'd first have to define what the goal is? Scala as a language is a design. It can have weaknesses and strengths. But instead of opportunities and threats I'd just talk about areas of applications. Scala as an ecosystem is a different matter. I am more on the design side, but would be interested in an analysis done by others.

Question: What are some other features you think you got right/wrong in Scala 2?

Answer: Look at reference/dropped features in https://dotty.epfl.ch/docs/

Question: Would Refinement Types make it to Scala3 compiler in the near future?

Answer: Refinement types will take more time. I have started to work with a PhD student on them.

Question: Assuming you haven’t yet, did you ever consider giving an open intro into the current Scala 3 internals?

Answer: Yes there's a talk: "Compilers Are Databases"

Scala 3 introduces new features which help manage complexity. In this talk, we’ll look at porting Scodec from Scala 2 to Scala 3, using new language features to simplify the library.

You’ll see the ease of migrating projects to Scala 3 and perhaps be inspired to port some of your own.

Q&A

Question: The casts in your codecs - do you think that scala 3 will improve to the point where they are no longer necessary?

Answer: I don’t think so — there are a few in the scodec code base that should go away as Scala 3 improves, but the ones I showed in the slides I think will be necessary, as the compiler isn’t aware of the type equalities we claim at type level and how it impacts term level. As I understand it, these types of issues are common in more formal systems (e.g. Coq/Agda) but don’t quote me on it! :)

Question: Not directly related to Scala 3, but how much of scodec is about the binary representation versus the recursive-isomorphism-oriented generic code? I'm wondering how much can it share with JSON, CSV, etc. codec libraries.

Answer: Mostly about binary structure, especially given Scala 3's meta-programming capabilities. Before Scala 3, we definitely needed things like Shapeless and Magnolia to help make meta-programming possible, and some of the type level operators in scodec could have been ported directly to Shapeless (as an example). Another interesting thing to study is the variance model -- e.g. Encoder is contravariant, Decoder is covariant, Codec is invariant. So what does it mean to try to map a Codec? Since it's invariant, that's really like imap/xmap from InvariantFunctor. Now ask a similar question for flatMap - this leads to a blog series I wrote up, which eventually led to the adoption of invariant functors in scalaz 7 and later the adoption of more invariant structures in Cats. https://mpilquist.github.io/blog/2015/06/18/invariant-shadows/

Question: Thank you! I've tried writing an isomorphism oriented (as in impossible to write an Encoder without a matching Decoder) JSON library but got stuck in Shapeless quite soon, now I might give it another try with Scala 3.

Answer: Oh that’s an interesting point. Iso could definitely be made a stand-alone microlibrary.

Question: Did you define your own Iso? Now I recall I brought Iso and Prism from monocle

Answer: Yep! https://github.com/scodec/scodec/blob/main/shared...

Question: Did you come across anything macro-related that you couldn’t replicate with Scala 3

Answer: Not in scodec, but I had a popular library called Simulacrum that's not possible to port to Scala 3 - https://github.com/typelevel/simulacrum

Question: Oh dear. So will you drop support for the library? I would have presumed (incorrectly) that the new Scala3 features would be very aligned with what Simulcrum supports?

Answer: No, you're right in your assumption. Simulacrum isn't needed in Scala 3 b/c equivalent functionality (better even) is basically built-in to the language now. Also, unrelated to that, there's a Scalafix for converting a Scala 2 code base that was using Simulacrum to a Scala 2/3 cross compiling code base, which basically just expands @typeclass

Starting dotty REPL...
scala> trait Semigroup[A] {
    |   def (x: A) |+| (y: A): A
    | }
// defined trait Semigroup
scala> given Semigroup[Int] {
    |   def (x: Int) |+| (y: Int) = x + y
    | }
// defined object givenSemigroupInt
scala> 1 |+| 2
val res0: Int = 3

Question: In your example, it seemed that the type-level code and term-level code were isomorphic. Are there situations (maybe more complicated situations) where they might differ?

Answer: One way I like to think of that symmetry you noticed is that the term level acts as a proof of the theorem stated by the type level.

Question: My question is less about the library per se but more around for you as a library author/maintainer how much you think Scala 3 has helped you in let’s say amount of work and ergonomics (for example reflected in lines of code/boilerplate reduction and enjoyment).

Answer: I don't expect to see the degree of effectiveness in general as I saw with scodec. The built-in support for arity-polymorphic tuples was a huge simplifier for scodec.

One bummer about Scala 3 as a library author is that we can't take advantage of Scala 3 only features for quite a while. On the flip side, I've found bugs in Scala 2 code just by compiling it with the Scala 3 compiler. And the error messages, especially about missing implicits, are sooo much better.

Also, I'm glad you mentioned enjoyment. Working in Scala 3 reminds me a lot of my early Scala days, trying out new features and new ideas, seeing what works and what doesn't, finding new ways to express designs, etc.

This workshop is an online workshop that will take place over two days:

• Monday 20 July: 1pm - 5pm
• Tuesday 21 July: 1pm - 5pm

Elixir is an extremely accessible functional programming language that is rapidly gaining popularity for good reason. With it's well curated, batteries-included tool chain, excellent documentation and its sheer simplicity, not to mention its incredible 30+ year Erlang heritage.

The goal of this workshop is to get you a basic familiarity with Elixir and the tools you'll need to be effective working in the language. It will be aimed at programmers who don't know Elixir, and don't necessarily know any functional programming.

The workshop is organised around a set of exercises that should take you through the basics of the language. Once you've got to grips with the language and tools you'll be ready to build a real time game server. In this workshop you'll learn everything you need to start building amazing, production ready Elixir applications.

I will talk about a much simplified version of functional programming in Scala, in building an abstraction for Feature Generation in Data Engineering space.
The program made using this abstraction will get interpreted to the popular data source languages of our choice - such as Spark or Flink. However, before it gets interpreted to any of these engines, we will explain how these programs could be optimised by introspecting its nodes, and help run these interpretations faster. The core idea is similar to that of Free Applicative, however, implementing it in Scala hasn't been straight forward. Here, we provide a similar capability but without mentioning much about FreeAp, and without the usual Scala boilerplates of implicits, macros and a proliferated usage of type classes.

The purpose of the talk is not just to demonstrate a set of code, but to showcase the fact that sticking on to fundamentals of Functional Programming, and finding the right abstraction enables writing solutions quickly and relatively easily.

It proves we don't need to learn a bulk of libraries to apply these concepts in real world applications. The learning curve and a massive set of libraries was often termed as the functional programming in Scala industry, resulting in lesser adoption and developers moving away from it. With this talk my intention is to motivate developers to come back and start writing FP even if they are in the world of JVM.

Q&A

Question: After that journey of recreating applicatives and semigroups as well as some pluggable interpreter pattern for the data pipelines and getting the team comfortable with it, is there any consideration of using cats/scalaz for that?

Answer: Indeed, cats or scalaz gives me Monoid. Adding a dependency is not going to scare developers, I hope. It’s a matter of how much we are able to teach them (passionate developers who are keen to use these techniques within their limited time at work)

We should see a discussion of Free Applicative here, and using it through Scala is a great idea to solve the pbm as well, if the team is comfortable with it.

Question: The problem you have solved seems to be quite generic (within the data engineering domain), and the solution abstracted and expressed generically. Is this a capability that could be of general benefit to the community?

If this is the case, have you open sourcing this, or at least thought of doing so?

Answer: I am solving a part of the big surface area of a general big-data engineering problem. Unless we spend time on the Expr we discussed, the OSS may not be quite useful for all usecases. What really worked here is, identifying what are the specific problems at the client place, and make sure the code is geared to solving them.

Question: To summarise, for my understanding, what you have basically done is abstracted away the data structure and operations of a spark dataframe or flink (i think also) dataframe with a custom algebra? and then used ZIO to run the jobs. a bit like a double abstraction of the effect, ZO to start the jobs and then the underlying flink/spark effects to perform the computation.

Answer: In short, yes.

Details: It's an abstraction that still has a bridge to the original data-source language's Expr implementation in flink, it is Expression I think, and in Spark, it is called Column . With this bridge we are solving the problem of "hey I can use only a part of the spark's capabilities here".

At the same time, we managed to optimise the program that gave tangible performance differences (in terms of hours). I wanted to talk about the numbers, but that is in general would become a discussion about Spark/Flink. As you mentioned, we then trigger this to the effect system. You can now imagine, you can run two spark actions in parallel from the same driver program.

Question: I see. Yes, incorporating the Expr implementation is great, there is nothing more annoying than providing an abstraction that you constantly have to extend because you weren't exposing all the underlying capabilities before

Answer: Absolutely! It's a disaster in a place where we don't have time. However, I am not super convinced the way Column or Expression are implemented, and the optimisers behind it. So indeed, a bigger opportunity, if we have time, is to build our own

But with the laws in place, we put a wall against the impure nature of spark/flink column/expression.

Question: It was a bit fast in the talk but I found it really cool that you could optimise the program to a pretty large extent automatically. going from individual computations to a single optimised computation with different views (basically instead of calculating 3 results you calculated 2 and combined them for the third right?) seems non-trivial.

Answer: That's right. Given the summary of feature generation you can give that as the input back into the program again, that will merge the partitions together. It's hard in 30 mins to smash into the details, but I am pretty sure we can figure that out if we give a bit of time trying to solve it ourselves.

Question: Do you have any datapoints on how your solution performs vs something like Spark SQL?

Answer: It was using Spark and spark SQL, and abstracting over how the dataframe queries were constructed.

It’s more of making sensible data frames which is the responsibility of the developer and not catalyst optimiser and then feed the sensible program to catalyst optimiser. The implementation of Dataservice is using data frame. But keep a note that they are all one liner implementation. The composing of operations happens at the abstract level and not using dataframes. Meaning ul see multiple joins in feature gen, but will see only one join call using dataframe in the entire application.

Question: Ah, I didn't realise, I thought you were using the regular dataframe api under the hood.

Answer: Yes. The implementation of Dataservice is using data frame. But keep a note that they are all one liner implementation. The composing of operations happens at the abstract level and not using dataframes. Meaning ul see multiple joins in feature gen, but will see only one join call using dataframe in the entire application.

Question: You said "this is sellable to the client, we didn't talk about FP there" - do you find there's a resistance issue to FP from clients?

Answer: Well the answer is “yes and no”. It’s more of a matter of familiarity. Things are smooth as soon as we are able to talk about benefits and able to teach them. But if we are just developing in silo, without talking things get a bit hairy for them and result in rewrites. The resistance exists in general, and the usual feedback is we can’t get developers who know these topics to maintain the code. Solution, I think is to write the code step by step and work along with them.

There are many examples that demonstrate how to create a strongly typed abstract syntax in Haskell for a language with a simple type system. But there are many fewer examples that allow the embedded language to be polymorphic. I will work through what it takes to do so, touching on variable binding representations, and exploring the limits of dependently-typed programming in GHC.

Q&A

Code for the talk is here: https://github.com/sweirich/challenge

Question: Is there an example of implementing something like this in a language with a surface syntax and parser? I’m guessing one would need to push quite hard on singletons to instantiate the proofs required to construct the AST from the surface syntax.

Answer: I have an example of a typechecker for this AST in the repo. The idea is that you would parse into the weakly-typed AST.

And then use the typechecker to produce the strongly-typed version. https://github.com/sweirich/challenge/blob/canon/debruijn/src/TypeCheck.hs

Question: How far "away" is dependently typed GHC?

Answer: There are a few issues with syntax that hold off combining types and terms together.And there are thorny questions about how dependent types combine with all of the OTHER features of GHC. Richard Eisenberg has been focussed on GHC proposals that give us features that work towards dependent types.

Question: Followup question - How closely are you following the experience reports on using Dependently Typed Haskell in Industry. eg the Galois paper from last year’s ICFP?

Answer: I've been looking at the type theory for combining dependent types with roles, and with linear types, etc. I did a sabbatical at Galois during 2018-2019 to get some experience with their codebase. In fact this talk was inspired by thinking about how to add polymorphism to their strongly-typed intermediate language.

Question: Are there many differences or hurdles when you replace evaluation by substitution with an abstract machine?

Answer: I'm not sure I understand your question. Are you comparing my eval function to a different kind of evaluator that doesn't need substitution?

Maybe one that keeps around an environment or heap instead?

Do you mean like a Krivine machine?

Question: Yeah, like a Krivine machine for STLC.

Answer: I included eval in the talk as a simple motivation for the substitution operation.

Ed Kmett: I'll be doing a boring STLC using a sort of machine-like model in about 5 minutes, but unlike Stephanie I won't be trying so hard to make it safe.

Stephanie: Everyone should go see Ed’s talk. And Ed I'd love to talk to you about benchmarking substitution sometime. https://github.com/sweirich/lennart-lambda

Ed: I loved that paper. It taught me so much back when I was learning Haskell. The cadenza approach is rather different, so it should serve as an interesting point in the design space.

Stephanie: I've been adding a lot of new versions to it to compare different representations of binding. Including both bound and unbound.

As one who joined the Erlang community before it really even was one, I've had reason to consider and debate what makes Erlang special. Because it _is_ special. In this talk, I will summarize some of my own thoughts on the matter, exemplified by some war stories. My thesis is that while the impetus for creating the language is important, the evolution of the community and the niches where Erlang got some traction also play a vital role. Learning what has been successful in Erlang, what hasn't, and why, could give important clues not just to future adopters, but perhaps also to other communities (Elixir?) still developing a personality.

Q&A

Question: What was the programming language PLEX like, is it documented anywhere? What kind of language was it?

Answer: The PLEX programming language was a form of event-based high-level assembly. There are some papers, I think, where you can see what it looks like. It evolved into more of a structured programming language, and HL-PLEX (High-Level PLEX) actually had a form of selective message reception.

“The Execution Model for APZ/PLEX” http://www.es.mdh.se/pdf_publications/663.pdf

Found this comment about PLEX:

Syntactically it was a cross between Fortran and a macro assembler. You couldn't pass parameters to functions, for example; you assigned them to variables instead, much like the limited GOSUB of an 8-bit BASIC. The advantage was that there was a clean one-to-one correspondence between the source code and the generated assembly code, a necessity when it came to writing patches for live systems where taking them down for maintenance was Very Bad Indeed.

https://prog21.dadgum.com/22.html

And here is a more detailed description of it. I think it reflects how PLEX programming was very much a structural approach in the sense that the language reflected the structure of the system (not in the sense of structured programming as we know it). The language itself was fairly low-level, but the whole system, including the APZ CPU, was custom-built for the task, and PLEX was never intended to run outside of that system.

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.117.4843&rep=rep1&type=pdf

BTW, regarding patches for live systems, this was often done by local experts in different local markets. The old world of fixed-line telephony was quite fragmented, and each market had its own set of local protocols. This was one important reason why the culture of the fixed-line division was decentralized: there was simply no way to centrally support all local varieties.

Question: Why was Unix/Solaris not considered a good OS for telephony systems?

Was Erlang/OTP running on other Operating Systems other than Unix?

Answer: There were some concerns about the memory integrity of Unix systems when running non-stop. Basically, our systems were intended to support better than "5-nines" availability, so ideally, they should just keep running. While there were reasons to believe that Unix could deliver (we thought so), the tradition had been to use proprietary OS:es, custom-made for the stringent requirements. Scheduling was also an area where there was reason to be suspicious, but since Erlang executed in a VM, it wasn't so much of an issue for us.

We (and CSLab) did believe that it would work nicely, and it did.

A few years later, I was in a meeting where an architect of a proprietary real-time OS said: "You can't build 5-nines systems on top of UNIX!". I commented that "We have been doing it for years", and he said, "yeah, if you use Erlang!", as if this was somehow cheating.

Initially, Erlang/OTP was made for Unix, but there was also a port to OSE/Delta, one to VxWorks, one to Windows, …

Question: why the opposition to type checking? do you think this might change in the near future? many other dynamic languages have recently warmed to type hinting...

Answer (Franceso Cesarini): No one knew how to write a type system, so it was never done. The problem became worse when, because of language design issues, it became very hard to bolt one on without changing the semantics of the language.

Work is happening to make a static type system possible in one of the big tech companies, results will be published soon.

From our end, we actually got used to dealing with it in the programming model (as you need to handle corrupt state and bugs anyhow), so it never bothered us.

(Ulf): One concern was that it would need to derive types from the already fairly large code base, and also report errors in a way that didn't require a PhD in type theory to understand.

(Aside: Phil Wadler and Simon Marlow published this paper on building a type system for Erlang. https://homepages.inf.ed.ac.uk/wadler/papers/erlang/erlang.pdf)

Another concern was that Erlang was deliberately designed to support live code updates and evolving systems over multi-year life spans (ideally never being stopped in the process). It was by no means clear how to do that with global type analysis.

Lastly, many type analysis approaches did reasonably well with function calls, but couldn't help with message passing. Actually, this was (I believe) a reason why Concurrent ML went for fully synchronous message passing: not to break the type system.

Now, Dialyzer was eventually accepted, since it struck a nice compromise, and people should use it as much as they can.

Personally, I think that getting typing right, you have to do it from the very beginning. Evolving Erlang into a statically strongly typed language will probably never work. OTOH, perhaps Cloud Haskell is what you should go with instead, then.

Question: You mentioned UX not being a focus for Erlang (with a remark "Web, GUI..."). What about the UX of using the language itself? That seems to be gaining visibility and popularity recently with seemingly minor issues (whitespace, syntactic sugar) having lots of discussion in e.g. TypeScript, Go (and the recent Scala syntax changes).

Answer: the UX of the language itself? Yeah, well, it's an acquired taste. :) I'm not really sure what to say about it.

Question: my question is about whether Erlang is still used in Ericsson systems. also, do you know what languages other telecoms providers (Nokia, Alcatel, etc) use by any chance?

Answer: Yes, actually, Erlang is now used in some of Ericsson's most important products. However, the company has learned a lesson regarding being too transparent about technology choices. Also, there seems to be a silent agreement to simply do it and not talk about it much.

Regarding what other telco companies use, I actually have no idea.

Question: You mentioned being selective in which areas to not pursue in Erlang e.g. GUIs. Are there any areas that Erlang should be pursuing that it currently isn’t?

Answer: I happen to think that Erlang would be amazing for complex embedded IoT devices. This was something we tried to pursue at Feuerlabs, but as I said, I think we were way ahead of the curve. With e.g. GRISP2, I think the threshold should be lower today, and there is much to be done there.

Question: What's your thoughts about the Elixir language/community? Did you think it has the same values of Erlang? How do you see the adoption of Erlang by the new generation of developers?

Answer: Re. Elixir, I think it's a great development.

One thing that I particularly like is that the Elixir community actually has a slightly different focus, and a much better story not least in the area of Web UX. This seems to be in line with my idea that community focus is very important. Without this perspective, one might argue that Elixir doesn't really add much to Erlang: it relies on the goodness of OTP, offers a different syntax, but mostly the same semantics. Yet, these things do matter, and most importantly, they have managed to attract a new group of developers and build a strong culture of their own.

Having lived through a period where there was basically no job market outside Ericsson for Erlang developers, and some political turmoil inside the company around Erlang, I've come to think that Erlang is worth learning for its conceptual strengths, even if you later end up using another language. Akka is an example of how Erlang concepts can be imported and successfully applied in other environments, and Cloud Haskell was explicitly a project to implement Erlang-style concurrency in Haskell.

Of course, learning Elixir should give you access to the same concepts, but again, the Erlang community itself is a strong reason to learn Erlang, so I very much hope that new developers will give it a chance.