Untitled

Building the Tumblr Neue Post Format

We’ve been looking at improving the posting and reblogging experience in our mobile apps for a long time. As many of our power users and public API consumers are aware, posts on Tumblr are stored and served in a sanitized HTML format. This choice made the most sense when Tumblr was originally built, when using Tumblr meant visiting via a web browser on your computer on the information superhighway back in 2007.

Storing post content primarily as HTML has remained our standard for ten years; there are a significant number of assumptions in our codebase about posts being primarily HTML. To compound this, when we want to change something about how posts are made or stored, we have to think in terms of the 150 billion posts on Tumblr and the billion new posts made every month. We have to spend a lot of time thinking about that scale whenever we consider how to make posting on Tumblr a better experience.

Over a year ago, Tumblr Engineering came up with a very ambitious idea: ditch HTML entirely and move to a brand new format for how posts are created and stored. HTML is fine, but its scope is limited as it was intended for the browser, long before the concept of mobile apps existed. Conversely, the JSON standard has been heavily favored by APIs and mobile development for years, and feels much cleaner and more flexible than HTML. We can apply an extensible format and schema with JSON easier than we can with HTML.

With this in mind, we’ve chosen to write a brand new JSON-based specification for post content. We’re calling it the Tumblr Neue Post Format, or simply NPF. The NPF specification breaks apart post content into discreet content blocks, each defined by a type field. All of our existing post content easily fits into this kind of specification, affording backwards-compatibility with the billions of posts on Tumblr.

For example, right now when you add text to a post, we store and serve:

<p>Some text in a post!<p>

With NPF, the same thing is created and served this way:

{
  "type": "text",
  "text": "Some text in a post!"
}

Those two representations are fully interchangeable, but we begin to gain advantages with JSON for things HTML cannot do well, providing flexibility and extensibility for future integrations. The power of NPF really becomes critical when we want to build content blocks for Tumblr that cannot be easily represented with HTML, such as a physical location:

{
  "type": "location",
  "latitude": 40.739485,
  "longitude": -73.988402,
  "map_style": "quirky"
}

This new JSON specification also gives us the benefit of not having to worry as much about potential security risks in malicious HTML payloads in post content. Moving from HTML to JSON allows us to have safer, more injection-proof defaults, and prevents us from having to do heavy DOM parsing at runtime, which means improved performance of our backend and mobile apps. With NPF, posting and viewing posts on Tumblr should be considerably faster and safer.

Our work so far with the NPF specification has been to reach feature parity with the rich text editor available to Tumblr users on the web, as well as extend those basic options with new ones, such as fun new text styles:

{
  "type": "text",
  "text": "Oh, worm?",
  "subtype": "quirky"
}

Our initial release includes support for text blocks (with inline formatting), GIF search blocks, and image upload blocks. All of these options are available in our mobile apps via the Text, Quote, and Chat post forms, as well as when you reblog a post. Yes, you can now upload images in a reblog on mobile.

Future releases of the mobile apps will continue to close the gap with our other post options as we build NPF support for link blocks, video upload blocks, third-party video and audio blocks, and more. We also plan on allowing third-party API consumers to view and create posts using the new specification sometime in the future.

- me (@cyle) and @noisysocks (with love for @keithmcknight who started the original NPF spec)

Flow and TypeScript

javascript

One of the Core Web team’s goals at Tumblr is to reduce the number of runtime issues that we see in our React codebase. To help move some of those issues from runtime to compile time, I evaluated the two leading type systems, Flow and TypeScript, to see if they could give us more type safety. I did a bit of background reading about the differences between Flow and TypeScript to see what the community had to say about them.

Background Reading

TypeScript, Flow and the Importance of Toolchains over Tools by Ben Teese

This post claims that Flow and TypeScript are similar enough that you should choose whichever of them is easier to integrate with your other tools. For Angular development, it recommends using TypeScript; for React, Flow.

TypeScript vs. Flow by Marius Schulz

This post claims that both TypeScript and Flow are equally good.

Flow vs. Typescript by Jan Varwig

This post outlines the author’s experience with using Flow in a React codebase. It advocates switching from Flow to TypeScript because of Flow’s unhelpful error messages, bad tooling, and propensity to spread untyped code. It also claims that most of the type annotations are able to be shared between Flow and TypeScript with only minor changes.

Type Systems for JavaScript by Oliver Zeigermann

This slideshow shows many differences around the philosophies and goals of TypeScript and Flow, and it gives detailed explanations in the differences between the two type systems. It explains IDE support and how to get access to third-party type definitions.

Lack of Consensus

It seems like many people have differing opinions about which type system is better for a React codebase. Because there wasn’t a broad consensus across the community, I decided to get some first-hand experience with each of these tools to see which one would be most practical and helpful for use at Tumblr.

Project Setup

I worked with a sample application to vet Flow and TypeScript. The application I used was Microsoft’s TypeScript React Starter. It uses a custom fork of create-react-app to get TypeScript set up. When testing out Flow, I used the standard version of create-react-app and used the source code from this exercise.

For the most part, Flow and TypeScript are basically interchangeable. I was able to reuse most of the source code between both projects with only minor changes. Here are some examples of changes I needed to make to get my TypeScript code working with Flow:

Flow requires that types are imported using import type where TypeScript re-uses import.
Some generic type constraints are different in redux’s type declarations between Flow and TypeScript, so I dropped the generic constraint for Flow.
Types cannot have the same name as constants, so I had to rename a few small things (see below).

Testing

After I got the project prepared I set up the following situations to see which tool performed better. These are my assumptions of the most common situations in which a type checker will help when writing React code on a day-to-day basis.

Handling an Unnecessary Case in a Switch

TypeScript

TypeScript realizes that 'not_real' is not a possible case for the switch.

Flow

Flow does not detect any issue.

Declaring Variables with Same Name as Type

TypeScript

TypeScript allows types to have the same name as constants, and it allows Command-clicking on the types to see their declarations.

Flow

Flow requires types and constants to have different names. In this case, I needed to rename the type to INCREMENT_ENTHUSIASM_T to appease Flow’s type checker.

Returning Incorrect Type from Function

TypeScript

[ts]
    Type '{ enthusiasmLevel: string; languageName: string; }' is not assignable to type 'StoreState'.
      Types of property 'enthusiasmLevel' are incompatible.
        Type 'string' is not assignable to type 'number'.

Flow 0.52

[flow] object literal (This type is incompatible with the expected return type of object type Property `enthusiasmLevel` is incompatible:)

Flow 0.53

[flow] property `enthusiasmLevel` of StoreState (Property not found in number) [flow] property `languageName` of StoreState (Property not found in number)

Missing Required Props When Instantiating a Component

TypeScript

TypeScript shows the error at the site where the properties are missing with the error:

[ts] Type '{}' is not assignable to type 'IntrinsicAttributes & Props'. Type '{}' is not assignable to type 'Props'. Property 'name' is missing in type '{}'.

Flow

Flow shows the error within the component where the property will be used, with no way to discover which call site is missing a property. This can be very confusing in codebases that have lots of reusable components. Flow displays this error:

[flow] property `name` of Props (Property not found in props of React element `Hello`)

Code Safety

TypeScript

TypeScript allows enforcing full type coverage on .ts files with the noImplicitAny flag in the tsconfig.

Flow

Flow provides a code coverage plugin so that you can see which lines are implicitly not typed.

Other Considerations

Flow has the most React community support and tooling, so there is much more documentation about how to get Flow and React working together. TypeScript is more popular with Angular developers. Choosing TypeScript may be breaking from community standards, so we may have more issues that don’t have a simple answer on Google.

Conclusion

I concluded that we should use TypeScript because it seems easier to work with. My experience seems to line up with this blog post. It has better error messages to debug type issues and its integration with VSCode makes coding more pleasant and transparent. If this ends up being the wrong choice later on, our codebase will be portable to Flow with some minor changes.

Shortly after arriving at this conclusion, Flow 0.53 was released and a blog post on Medium published touting it’s “even better support for React”. However, after running through the test cases above, I only found one case where Flow had improved its error messaging. TypeScript still seems like the more reliable, easier to use solution.

See, that’s what the app is perfect for.

Building the Tumblr Neue Post Format

Flow and TypeScript

Background Reading

Project Setup

Testing

Handling an Unnecessary Case in a Switch

Declaring Variables with Same Name as Type

Returning Incorrect Type from Function

Missing Required Props When Instantiating a Component

Code Safety

Other Considerations

Conclusion

Further Reading