Practical ASTCompilers to ToolingSibelius Seraphini
Sibelius Seraphini
Abstract Engineer

Overview

  • Compilers
  • Parsing
  • Transformation
  • Code Generation
  • Pratical AST

What is a Compiler

Compilers are programs that transform code in a programming language to another language (low level or machine code)

Compiler Pipeline

Parsing

Lexical Analysis

  • It takes raw code and transform it into tokens
  • It is called tokenizer or lexer
  • Tokens are tiny objects that describe syntax

Tokenizer

Syntatic Analysis

  • It takes the tokens
  • It reformat then to a representation that describes syntax and their relation one to another
  • This representation is called Abstract Syntax Tree or Intermediate Representation.

Syntatic Analysis

Abstract Syntax Tree

{
  type: 'Program',
  body: [{
    type: 'CallExpression',
    name: 'add',
    params: [{
      type: 'NumberLiteral',
      value: '2',
    }, {
      type: 'CallExpression',
      name: 'subtract',
      params: [{
        type: 'NumberLiteral',
        value: '4',
      }, {
        type: 'NumberLiteral',
        value: '2',
      }]
    }]
  }]
}

Transformation

  • we can add/remove/replace properties of AST nodes
  • we can add/remove/replace nodes
  • we can create a new whole AST

Traversal

Visitor Pattern

Code Generation

  • It iterate recursively on final AST to print nodes
  • It can print machine code
  • It can print another language (graphql2ts)
  • It can print the same language (prettier)

Pratical AST

Where AST is used?

  • Babel
  • Eslint
  • Prettier
  • GraphQL
  • Relay Modern
  • Codemods
  • IDE
  • Docz
  • And more

Babel

  • You use Visitor pattern to write babel plugins to transform new javascript to old javascript

Eslint

  • You use AST to create custom rules for you codebase

Eslint Selectors

  • You can "query" AST node using syntax similar to CSS selectors
CallExpression[callee.name='setTimeout'][arguments.length!=2]

Prettier

  • You use AST to pretty print your code

GraphQL

  • You use AST to handle GraphQL SDL and extend your GraphQL capability

GraphQL2TS

  • Transforms .graphql files into graphql-js typescript code

Relay Modern

  • Relay uses GraphQL AST to compile and optimize GraphQL fragments and queries
  • It also polyfill GraphQL to enable fragments to have arguments, this provides a better DX without losing performance

Relay Compiler Repl

Codemods

  • Codemod is an AST-to-AST transformation to evolve codebase
  • Much more powerful than regex
  • AST has much more information than raw code

Codemod Examples

  • rn24-rn25: transform imports from @reactnative v24 style to @reactnative v25 style
  • ast-i18n: transform your raw strings to i18n keys and extract all of them to a locale file
  • rn-deprecated-modules by @lbentosilva: transform from react-native import to new @ReactNativeComm packages

Codemod Examples

Thanks!
We are hiring!
Join Us