Anders Hejlsberg on Modern Compiler Construction

From what I've seen (MS folks can correct me here), language services are part of a much larger effort at Microsoft led by Anders Heijsberg (Delphi, C#) called Roslyn [1], which decouples code analysis from IDEs and provides things like refactoring, autocomplete, find-all-references, Intellisense, etc. as services. This means the compiler is now an abstracted platform that editors can get services from (CompilerAndCodeAnalysis-as-a-service?). Anders talks about the motivation for doing that here [2]. It's a pretty exciting idea. This might mean I would some day be able to use Vim and get IDE-like functionality. [1] Roslyn [LINK] [2] [VIDEO]

To better understand why this project is trying to re-implement the compiler frontend, I really recommend this video from Anders Hejlsberg (creator of C# and TypeScript) explaining how modern compiler construction oriented towards IDE works: [VIDEO]

There is an interview with Anders Hejlsborg [VIDEO] where he talks about how the Roslyn and TypeScript compilers are designed to support IDEs, with a structure that allows the compiler’s model of the program to be incrementally updated as the code is modified. So there has been some fairly prominent work along the lines you want...

Microsoft's Roslyn [1] design (now called .NET Compiler Platform) is what I think any language should strive for, turning the classical black box of the compiler into an incremental pipeline where each stage is annotated/augmented/refined by different mechanisms such as type checkers. If each step is incremental, it follows that it's easier to also let each step "integrate" deltas; for example, adding a comment shouldn't need to recompile anything at all, merely modify the AST, because it doesn't lead to any actual code artifacts. The second benefit is that all the information needed to inspect the program — from syntax highlighting to type information — is already there. If you can keep the entire high-level compilation state in memory, then it's easy to query it for all sorts of IDE features. This same design was used for the TypeScript compiler. Anders Heljsberg has a great little lecture [2] about it. [1] [LINK] [2] [VIDEO]

Anders Hejlsberg explains explains how “the IDE” way compiler works. First ~10 minutes is traditional compiler background. Then about ~8 minutes before he starts talking about how an IDE focused compiler works: [VIDEO]

In a query-oriented compiler, you can give it a source location and ask for type infomation, or where is the definition of the symbol at that location, and the compiler will do just enough work to give you the answer, using laziness and memoization to make it reasonably efficient. As far as I know, LISP systems don't work that way: they parse and evaluate (and maybe compile) as soon as an expression is entered, without the kind of laziness you see in a query-oriented compiler. This recent link was pretty good, I think [LINK] and this discussion with Anders Hejlsberg [VIDEO]

Anders explains it better than me, "Anders Hejlsberg on Modern Compiler Construction" [VIDEO] Basically ability to see the language AST at all time, do time checking with broken code, incremental compilation, REPL and IDE editing experience is fused together. Here is another short examples, "7 minutes of Pharo Smalltalk for Rubyists" [LINK] "MountainWest RubyConf 2014 - But Really, You Should Learn Smalltalk" [LINK]

To your first point, I've got a couple of bookmarks: - Query-based compiler architecture: [LINK] - Lezer [a parsing system used by CodeMirror]: [LINK] - Anders Hejlsberg on Modern Compiler Construction: [VIDEO]

I saw this great interview recently with Anders Hejlsberg at MSFT on how modern compiler construction differs from what is taught in traditional University courses. Is this what you're alluding to? After watching that interview, it's strange to read a comment like yours. While the architecture is completely different, it doesn't frankly seem like that big a leap to go from "senior engineer with X years working on thing that's tangentially related to compilers" to being able to be productive in a reasonable amount of time working with the new architecture. What am I missing? [VIDEO]

A (now 6 year old) discussion that might be helpful: Anders Hejlsberg on Modern Compiler Construction: [VIDEO] Also [LINK]