The Best Tool for the Job

I often see programmers saying that programming languages are tools, and that instead of arguing over what the best language is, we should just pick the right tool for the job. I think this is a sensible philosophy, but I've never seen it used in a sensible way. One time I was talking to someone who told me that the Software Engineering program at RIT exclusively uses the Java programming language. When I pointed out how weird this was, he said, "Languages are tools, and you should just use the right tool for the job"1. If Java is a hammer in this analogy, then why is this major not teaching you how to use a screwdriver?

There are other times when the philosphy is used in sort of the right way, but to promote a language that doesn't fit at all. I keep seeing Go being recommended for web development, for example. That is the intended purpose of the language, but I don't recommend it due to the strange way it handles errors2.

To be clear, I don't hate this way of thinking. I approve of it. In fact, I find the suggestion that we should just use the same language for everything even worse. But I wish more thought went into the idea. In this article, I want to propose the actual ideal language for each task. Surely, this will end all debates related to the choice of programming language. If you currently in the middle of such a debate, link this article to your co-workers3, and give yourselves the rest of the day off.

My Biases

Just like everybody else, I have my own preferences for language design, which might not align with yours. I still think my recommendations are correct, but I'll make some premises clear here, so that if you want to adapt my work for your preferences, you may do so.

Something else that I think is worth pointing out is that I will be focusing more on the language features than the ecosystem. In a more practical discussion, you should probably be thinking about this. But I think the language features are more interesting, and it's not hard to imagine a good ecosystem being developed for any of these languages. Sometimes, a language feature means it might be easier to develop an ecosystem designed for a specific task.

Systems or Embedded Programming

You'll need Assembly. There's no getting around that. The question is, what language do you use to call your Assembly utilities? Given that most programming languages need a runtime, and that the runtime does not exist until you write it, you have a limited set of options:

I recommend Rust. Most of the errors in many projects are due to memory safety issues. Android has solved this problem by using more memory safe languages. Windows and Linux are also starting to introduce Rust into their codebases for the same reason.

I also mentioned having to write a runtime for C and C++. That's not necessary for Rust. Rust has, in addition to the standard library, a core library. The core library does not require an operating system, and has some very convenient utilities, including UTF-16. And even without the standard library, you can use the built-in alloc crate to use utilities that require memory allocation. You'll just need to bring your own memory allocator.

I'm not done yet. Rust has async/await. Rust's implementation of async/await does not rely upon the existence of an operating system. You can create your own async runtime, and then use async/await inside of your kernel! If that does not sell you, nothing will.

Parsers and Compilers

Now you have an operating system. Maybe you already have a shell and some command-line utilities. But you now you need a way to compile programs on your new system. You don't necessarily want to write everything in Rust. But you'll need to write a compiler for that to work. What language do you do it in? I have a couple ideas for a good parser language. I'll make the case for both.

Rust

There's one particular reason why I like Rust here: enums. Rust enums are discriminant unions, which can make it much easier to create a token type, and an abstract syntax tree. Here's an example of some types I'd make when creating a basic Lisp interpreter: 

enum TokenType {
	LeftParenthesis,
	RightParenthesis,

	LiteralNumber(f64),
	LiteralString(Arc<str>),
	Symbol(Arc<str>),
}

And here's what the abstract syntax tree could look like:

enum ExpressionType {
	LiteralNumber(f64),
	LiteralString(Arc<str>),
	Symbol(Arc<str>),

	SExpression(Arc<[ExpressionType]>),
}

It gets more complicated than that when you're doing error reporting, but you can at least understand how this makes things easier. I also recommend you take a look at the Icon programming language. You won't be using it, but it is a very useful framework for tokenizing. I wrote a Rust implementation of some of Icon's standard library, in the form of a library called snob. I still use it, and I recommend giving it a try.

I don't think Rust is the perfect language for this task though. You may have noticed all of my Arcs in that code snippet. Constantly copying a Box or a Vec will get expensive very quickly. But this interface isn't the greatest. Constantly copying an Arc isn't great either. In fact, the D programming language found that typical generational garbage collection is faster than implementing garbage collection using Arc4. Another problem is that Rust uses more memory than a garbage collected language would. Rust monomorphizes generic functions for more speed. But this bloats the binary size and uses a lot of memory. I can't find it now, but I remember seeing a GitHub issue stating that rustc couldn't be compiled on a Raspberry Pi, because compiling it requires too much memory.

So I would recommend a garbage collected language, but the problem is that I can't find a language that has Rust-style enums and garbage collection5. C# could do this once they figure out discriminated unions. But I don't hold much hope for that right now.

Dart

Now I'll admit that I've never attempted to write a parser in Dart before. I'm only thinking about it now. Dart doesn't have Rust-style enums. But it tried. Dart has sealed classes, and if you run a switch statement on a sealed class, then it will require an exhaustive match. Here, I'll duplicate the Token data structure I showed in Rust: 

sealed class Token {}

class LeftParenthesis extends Token {}

class RightParenthesis extends Token {}

class LiteralNumber extends Token {
	final double value;

	const LiteralNumber(this.value);
}

class LiteralString extends Token {
	final String value;

	const LiteralString(this.value);
}

class Symbol extends Token {
	final String name;

	const Symbol(this.name);
}

Notice how that was far more verbose than the Rust implementation. And I didn't even create an abstract syntax tree. We're also using inheritance, so we'll have to be careful about how we modify the base class. But there is one benefit. Something I did not show in the Rust implementation is the information about where the data is stored in the original source code. Adding that to the Rust implementation can be done using another struct. 

struct Token {
	token_type: TokenType,
	start: usize,
	end: usize,
	file: Arc<Path>,
}

Then any token we create has to be wrapped in this new structure. But in Dart, we can do it by modifying the base class 

sealed class Token {
	final int start;
	final int end;
	final File file;

	const Token({required this.start, required this.end, required this.file});
}

But then we'll also have to modify the constructors on the child classes to have the same information. So you might still want to use composition here. But this would cut down on the of times you have to write something like token.token_type.

Dart seems promising. I'll try writing a parser in it sometime.

Scripts

So now you can create a parser. But a parser for what language? Say you need a scripting language for some simple tasks. What can you use?

Bash is ok, if the only thing you need to do is run some simple commands. But most scripts will end up being more complicated than that. If your bash script needs a variable, then you shouldn't use Bash.

Here we have a different set of priorities. This code shouldn't take too long to write. In many cases it will be thrown away immediately. We should still make it easy to read, just in case. But we might not need it for very long.

Python was designed to be a middle-ground between Bash and C. It's painfully slow, but our scripts shouldn't take too long to run anyway. The syntax is kinda annoying, but not annoying enough that I would consider writing a new language to replace it. There are some conveniences like default parameters that I honestly wish Rust had, but that one in particular was implemented with some stupid mutability rules. There's no interfaces, only inheritance. Exceptions are bad. After getting used to the borrow checker, I don't appreciate the with blocks. So much of the standard library is deprecated. Python is certainly not the perfect language.

But it does come with some interesting ideas. Gradual typing was probably the right decision for a language like this. And I can't complain, because the type system comes with null safety. The idea of having big integers by default is also really clever. And the large set of builtins is both a blessing and a curse. It's a good idea for scripting, but not when half of the builtins are deprecated. The package manager installs everything globally, which is not a great idea, but nothing else really makes sense for a scripting language. Most scripts don't need their own directory and node_modules.

That said, there's more that could be done. I don't like whitespace as syntax, but maybe we could agree on automatic-semicolon-insertion as a middle ground. A derive-like syntax seems especially important for a language trying to cut down on developer time. This could even be done with macros. Speaking of which, I'd like to see interfaces, and Rust's enums. Dart has some nice ideas too, like the this.x parameter in constructors, as well as named constructors and factories. Hell, use 1-based-indexing and use a numerical tower as the default number type, or at least use a decimal type. Python is used mostly by novices, so we might as well make it easier for them.

At this point, I've essentially proposed my own programming language. To be clear, I don't hate Python enough to actually make this. I'm just dreaming here.

Game Development

Now you want to have some fun, so why not make a video game? What language can we use for that?

There are several constraints that game engines have that other programs do not. For example, we really do not want any stuttering, so a garbage collector is out of the question, unless you decide to manually run the garbage collector on every single frame. However, performance is another important goal for a video game, so we probably want to avoid that too. That means Rust is our only option.

The problem there is that the ecosystem for Rust game development isn't very ripe. There is the work-in-progress Bevy game engine, but I honestly don't like the way you query for components in it. So you'll be making your own game engine. Given that this is Rust, we will still use an entity-component system. But the queries will be much simpler: 

let mut player_sprite = cx.sprites.get("player")?;
let player_collider = cx.colliders.get("player")?;
player_sprite.set_position(player_collider.position());

It would be really cool if the engine supported hot reloading scripts, but Rust doesn't have that. We could make it work by compiling the scripts to WebAssembly and reloading them when the file changes. But this would introduce undefined behavior if the old script's state doesn't match what the new script is expecting (i.e. a struct definition changed). To resolve this, we will require that scripts not store any state. Instead, the memory of the script will be cleared every frame. Any information we need to keep will be stored in the engine code. 

fn run(state: ScriptState) {
	let mut velocity: i32 = state.get("player_velocity").parse();
	velocity += 1;
	state.store("player_velocity", velocity);
}

Now, something magical has just happened. We are clearing all state every single frame. All heap-allocated memory, as far as this script is concerned, is static. This will be the new memory-allocation strategy. We can use a bump allocator as the allocator. And that means we won't need a garbage collector (because we're clearing memory every frame), or a borrow checker (because all heap memory is static). It wouldn't be too hard to adapt an existing garbage-collected language to use this strategy, so we could just pick one that can theoretically be compiled to WebAssembly, and make a compiler that doesn't insert a garbage collector. That could include C#, Dart, Kotlin, or Swift. But we could also make a custom language suited to this task. This could make it even faster than Rust, because bump allocators are so fast.

In fact, I don't think the perfect language for game development exists yet. I want something easy to embed, like Lua, but statically typed6. If we could block the use of the file system or network, then we could ensure that mods don't contain any viruses. But the language server would need some way to know what libraries are being imported, and what they contain.

Another useful idea is a property declaration. Here's an analogy:

// this is determined at compile-time, and not modifiable at runtime
const PLAYER_SPEED: usize = 5;

// this can be mutated (not in Rust, but in other languages), but an initial value must be given
static PLAYER_SPEED: usize = 5;

// this is a value inserted by the game engine
property PLAYER_SPEED: usize;

The engine would need to parse the script to determine what properties exist, give the developer a slider that they can use to change it, and then insert that value into the script. Technically, this would probably just be syntactic sugar for calling a function. But you wouldn't have to worry about typos in the property name.

I'm sure are other features that I think could be useful as well, but this proves the point that a language specific to game scripting could be useful.

Backend Web

Now you have a video game. Maybe this game has online features. In that case, you'll need a web server to handle communication between players. Once again, we need to pick a language to write it in. Let's take a look at the usual suspects.

Rust (or Swift)

Rust is great because of the exceptional error handling. If all errors are handled using the Result type, and errors are sent to the client using the Result type, then you guarantee no cryptic 500 errors, except in the case of a panic. Swift also has a Result type, so most of what I say here should apply to Swift as well (although I'm not familiar enough with Swift to tell you when this advice might not apply).

A point to Rust specifically is the performance and lack of a garbage collector. In most languages, when the garbage collector starts, the entire program stops. But Rust doesn't have this problem, so you can send responses at the speed of the network. Another point to Rust is the fearless concurrency. No data races ever, and bad race conditions should be rare as well. Multithreading is important to a server, so this is a good thing to have. The ecosystem is also able to provide things that no other language can. Serde makes JSON and XML serialization/deserialization a piece of cake. Take a look at SQLx. Not only can Rust get rid of memory and race errors, it also gets rid of SQL errors.

AWS supports Lambda functions written in Rust, and even gives you a discount if you do so, since it runs faster. If you don't like cold boots, then you can always use an actual server application. There are a few backend web frameworks written in Rust, such as axum, which everyone seems to love despite the fact that it isn't finished yet. I use actix-web. It's slightly faster than axum, and it's both stable and production-ready.

Dart

Let's start with what Dart doesn't have. It doesn't have anything as nice as serde or SQLx because it doesn't have macros. It also doesn't have a Result type. AWS Lambda support for Dart is also lacking. So all in all, I recommend Rust over Dart. But I wanted to mention Dart because of the interesting strategy it uses to avoid data races.

Dart doesn't allow you to share memory between threads. Each thread is its own Instance with its own heap. But the Instances can communicate with each other over channels. So there are no data races, just like Rust.

Now, Dart does have a garbage collector. It's always going to be slightly slower than Rust under load. But since all of the threads have a separate heap, the garbage collector does not need to stop your application from running. It only stops one thread. And the garbage collector can be programmed to run early if a thread has no tasks waiting for it. So in practice, you might never experience a GC pause.

There are a number of Dart backend frameworks. I haven't personally tried any of them, and many of them are deprecated, but I think it's worth checking out if you really want to avoid garbage collection. Some of them have built-in ORMs and I've seen at least one with built-in OAuth2.

WebAssembly

In case you didn't guess, we're going down the same path I followed in my game development ideas.

The framework itself should be written in Rust, but we can do better for the route code. I really like the idea of serverless, but cold starts are a problem for me. But take a look at Fermyon's spin. It uses WebAssembly to get the best of both worlds. Just like serverless functions, they don't need to be loaded up all of the time. But because WebAssembly is already heavily sandboxed, they don't need to create a virtual machine to contain your code. Which makes the cold starts fast. So fast that they've decided to just cold start every single time. With the help of WASI, Spin can support other languages, such as Go, Python, JavaScript, C#, and Ruby.

So you might see where this is going. Again, we can use a bump allocator to avoid garbage collection and borrow checking. I think spin uses a custom implementation of WASI in order to keep everything sandboxed, so we could do that too. But, again, custom compilers would be needed, so we could forego WASI and use a more stable interface that we invent.

I also think that a new language to support this system could be helpful. We'll need Rust macros, Rust enums, and a Result type. Although, the macros would probably be implemented in the embedding, in order to make it more powerful. We'll also want a multithreading system similar to the one in Dart. Of course this new language would need async/await. Ideally, it would also have a core library separate from the main one. Ideally, it would also be impossible to panic, but that seems impossible.

There's something else we can do now that we're running everything inside of a WebAssembly framework. Spin provides its own key/value store. The neat thing about this is that we don't necessarily need a network request to access this store. It's stored locally on the machine. So we can do fast DB access by creating our own database, and embedding it directly into the framework. I think Spin had this idea with the built-in SQLite database, but SQLite doesn't work well in some cases . The key/value store Spin provides is also limited to about 1000 tuples, if I understand correctly. So I would prefer something with a built-in wide column DB, that doesn't require a network connection to access. Maybe also a relational database, but those are harder to create from scratch (speaking from experience).

I'm not actually convinced that WebAssembly is the best possible thing here either. WebAssembly is a stack-based bytecode, which is typically slower than one based on registers. This is a good idea for WebAssembly, because it's on the web, and stack-based bytecode is much smaller than register-based. So you'll be able to download the code more quickly. But a server doesn't have this requirement. The bytecode will need to be uploaded to the server, but the size doesn't matter too much after that. So I could imagine a new bytecode being invented here that takes advantage of registers. It could even be AOT-compiled once it reaches the server. The same idea applies to games as well. The only benefit to a smaller bytecode would be a smaller download size, but performance is probably more important there.

Frontend Web

Any language you choose here is going to either compile into JavaScript or WebAssembly. WebAssembly can be a problem for many languages, since you need glue code to make the language's runtime work in a browser, which is often the same size as a JavaScript framework. If there were a language that had a runtime more directly based on JavaScript, designed specifically for frontend WebAssembly, that could work7. But until that happens, I'm going to recommend avoiding this approach.

That leaves your options to JavaScript, and languages that compile to JavaScript. This shouldn't be a surprise to anyone, but TypeScript is your best bet here. I don't think it's perfect. It takes some of the flaws of JavaScript for itself. But it does make many of them more bearable. It doesn't cause the JavaScript to get much larger. You get built in lowering to older versions of JavaScript. And you inherit the whole JavaScript ecosystem.

Alright, here's the part where I propose a better language. First of all, we shouldn't be inheriting JavaScript's syntax. If we have to compile to JavaScript from another language, why not compile from a good one? So things like === don't need to be there. We should also get rid of undefined.

Now, one problem I've noticed in most JavaScript frameworks is the difficulty in figuring out which variables have been updated. Svelte tried to solve this by using the equal sign as a marker that a variable needs to be updated. But what if I use array.push(5)? Now I need to enter a hacky array = array to make the change visible. So we'll want some way of detecting which methods mutate a value.

Rust, as you may know, has that. Methods that mutate the value take &mut self. How would we implement this system in Rust? 

struct Signal<T> {
	value: T,
}

struct SignalGuard<'a, T> {
	signal: &'a mut Signal<T>,
}

impl<'a, T> Drop for SignalGuard<'a, T> {
	fn drop(&mut self) {
		mark_as_updated(self.signal);
	}
}

We won't be able to replicate this in TypeScript because TypeScript doesn't have a borrow checker, and thus cannot detect when a value needs to be dropped. That's the problem that the garbage collector is meant to solve. So how about we add a borrow checker to TypeScript?

Yes, I'm actually proposing taking a language which does not have or need a borrow checker, and adding a borrow checker to it. The borrow checker just makes so many problems easier. For instance, we said that we needed a language that compiles to WebAssembly but is more closely related to JavaScript. If we have a borrow checker, and plan the language properly, this language could even compile to WebAssembly, making it faster than JavaScript.

I've been working on a web framework for a bit now, and I think this would be a useful feature. I think I'll try it someday.

Conclusion

I didn't mention desktop or mobile apps, but the answer there would be incredibly simple: Dart. Dart was designed with that specifically in mind, and went through extensive user testing to ensure that it could be useful for that task. I wouldn't even have any suggestions for what to add to it (other than Rust enums and macros). Other languages try to have more general purposes, but if a language is built with a specific goal in mind, it will be good for that task. Python was designed with scripts in mind, and I recommended it, even though I hate dynamically typed languages. Rust was designed as a competitor to C++, and it does indeed replace C++ in many use-cases. Icon is a language that barely exists, and yet I still gave it a shoutout in the parser section.

The idea of languages being tools is a good one. But I don't think enough thought goes into it. Make a language with a task in mind. It will be a great one.