+ +

The Best Tool for the Job

+ +

+ I often see programmers saying that programming languages are tools, and that instead of arguing over + what the best language is, we should just pick the right tool for the job. I think this is a sensible + philosophy, but I've never seen it used in a sensible way. One time I was talking to someone who told + me that the Software Engineering program at RIT exclusively uses the Java programming language. When + I pointed out how weird this was, he said, "Languages are tools, and you should just use the right + tool for the job"¹. If Java is a hammer in this + analogy, then why is this major not teaching you how to use a screwdriver? +

+ +

+ There are other times when the philosphy is used in sort of the right way, but to promote a language + that doesn't fit at all. I keep seeing Go being recommended for web development, for example. That is + the intended purpose of the language, but I don't recommend it due to the strange way it handles + errors². +

+ +

+ To be clear, I don't hate this way of thinking. I approve of it. In fact, I find the suggestion that + we should just use the same language for everything even worse. But I wish more thought went into the + idea. In this article, I want to propose the actual ideal language for each task. Surely, this will + end all debates related to the choice of programming language. If you currently in the middle of such + a debate, link this article to your co-workers³, and + give yourselves the rest of the day off. +

+ +

My Biases

+ +

+ Just like everybody else, I have my own preferences for language design, which might not align with + yours. I still think my recommendations are correct, but I'll make some premises clear here, so that + if you want to adapt my work for your preferences, you may do so. +

+ +

+ I hate dynamic typing. The time it takes me to ensure type safety is far less than the time it + takes me to debug type errors, find the documentation that tells me what types I can pass into + a function, figure out what methods I can call on a given type, or even what type a given value + is supposed to be. Ensuring type safety doesn't take that much time anyway, as long as your + language has type inference. Most of the languages I recommend here have type inference. +
+ On that note, the language should try to prevent me from making runtime errors. No language can + fully prevent logic errors, but you shoud note that most, if not all, of the languages I + recommend have some sort of null safety, albeit not always by default. +
+ Similarly to the last two points, borrow checking provides many benefits (including + deadlock prevention) which I believe are worth the + hassle. I don't recommend a borrow checker all of the time, but it is a pretty common occurrence. +
+ I don't like object-oriented programming. I don't have a problem with classes. But when a + language forces me to put all of my behavior inside of a class, I get annoyed. I also recommend + against using inheritance. I can put up with languages like C#, but it's not an ideal situation. +

+ +

+ Something else that I think is worth pointing out is that I will be focusing more on the language + features than the ecosystem. In a more practical discussion, you should probably be thinking about + this. But I think the language features are more interesting, and it's not hard to imagine a good + ecosystem being developed for any of these languages. Sometimes, a language feature means it might + be easier to develop an ecosystem designed for a specific task. +

+ +

Systems or Embedded Programming

+ +

+ You'll need Assembly. There's no getting around that. The question is, what language do you use to + call your Assembly utilities? Given that most programming languages need a runtime, and that the + runtime does not exist until you write it, you have a limited set of options: +

+ +

More Assembly: Yuck. Do I need to explain this one?
C: + An oldie. Not quite a goodie. I think we can do better than 30 years of tech debt. +
C++: + Not much better than C. Linus Torvalds would actually prefer C. There's a lot of language bloat, + and creating a custom standard library for C++ is much harder than for C. +
Rust: + Not exactly made for kernel development. The Linux kernel can use it now, but they modified some + of the standard library features in order to make it work. +
Zig: This language isn't even 1.0 yet.

+ +

+ I recommend Rust. Most of the errors in many projects are due to memory safety issues. + + Android has solved this problem by using more memory safe languages. + Windows and Linux are also starting to introduce Rust into their codebases for the same reason. +

+ +

+ I also mentioned having to write a runtime for C and C++. That's not necessary for Rust. Rust has, in + addition to the standard library, a core library. The core library does not require an operating + system, and has some very convenient utilities, + including UTF-16. + And even without the standard library, you can use the built-in alloc crate to use utilities that require + memory allocation. You'll just need to bring your own memory allocator. +

+ +

+ I'm not done yet. Rust has async/await. Rust's implementation of async/await does not rely upon the + existence of an operating system. You can create your own async runtime, and then use async/await + inside of your kernel! If that does not sell you, nothing will. +

+ +

Parsers and Compilers

+ +

+ Now you have an operating system. Maybe you already have a shell and some command-line utilities. But + you now you need a way to compile programs on your new system. You don't necessarily want to write + everything in Rust. But you'll need to write a compiler for that to work. What language do you do it + in? I have a couple ideas for a good parser language. I'll make the case for both. +

+ +

Rust

+ +

+ There's one particular reason why I like Rust here: enums. Rust enums are discriminant unions, which + can make it much easier to create a token type, and an abstract syntax tree. Here's an example of + some types I'd make when creating a basic Lisp interpreter: +

+ +

+enum TokenType {
+	LeftParenthesis,
+	RightParenthesis,
+
+	LiteralNumber(f64),
+	LiteralString(Arc<str>),
+	Symbol(Arc<str>),
+}
+

+ +

And here's what the abstract syntax tree could look like:

+ +

+enum ExpressionType {
+	LiteralNumber(f64),
+	LiteralString(Arc<str>),
+	Symbol(Arc<str>),
+
+	SExpression(Arc<[ExpressionType]>),
+}
+

+ +

+ It gets more complicated than that when you're doing error reporting, but you can at least understand + how this makes things easier. I also recommend you take a look at the + Icon programming language. + You won't be using it, but it is a very useful framework for tokenizing. I wrote a Rust + implementation of some of Icon's standard library, in the form of a library called + snob. I still use it, and I recommend giving it a try. +

+ +

+ I don't think Rust is the perfect language for this task though. You may have noticed all of my + Arcs in that code snippet. Constantly copying a Box or a Vec + will get expensive very quickly. But this interface isn't the greatest. Constantly copying an + Arc isn't great either. In fact, the D programming language found that typical + generational garbage collection is faster than implementing garbage collection using + Arc⁴. Another problem is that Rust uses + more memory than a garbage collected language would. Rust monomorphizes generic functions for more + speed. But this bloats the binary size and uses a lot of memory. I can't find it now, but I remember + seeing a GitHub issue stating that rustc couldn't be compiled on a Raspberry Pi, because compiling it + requires too much memory. +

+ +

+ So I would recommend a garbage collected language, but the problem is that I can't find a language + that has Rust-style enums and garbage collection⁵. C# + could do this once they figure out discriminated unions. + But I don't hold much hope for that right now. +

+ +

Dart

+ +

+ Now I'll admit that I've never attempted to write a parser in Dart before. I'm only thinking about it + now. Dart doesn't have Rust-style enums. But it tried. Dart has sealed classes, and if you run a + switch statement on a sealed class, then it will require an exhaustive match. Here, I'll duplicate + the Token data structure I showed in Rust: +

+ +

+sealed class Token {}
+
+class LeftParenthesis extends Token {}
+
+class RightParenthesis extends Token {}
+
+class LiteralNumber extends Token {
+	final double value;
+
+	const LiteralNumber(this.value);
+}
+
+class LiteralString extends Token {
+	final String value;
+
+	const LiteralString(this.value);
+}
+
+class Symbol extends Token {
+	final String name;
+
+	const Symbol(this.name);
+}
+

+ +

+ Notice how that was far more verbose than the Rust implementation. And I didn't even create an + abstract syntax tree. We're also using inheritance, so we'll have to be careful about how we modify + the base class. But there is one benefit. Something I did not show in the Rust implementation is the + information about where the data is stored in the original source code. Adding that to the Rust + implementation can be done using another struct. +

+ +

+struct Token {
+	token_type: TokenType,
+	start: usize,
+	end: usize,
+	file: Arc<Path>,
+}
+

+ +

+ Then any token we create has to be wrapped in this new structure. But in Dart, we can do it by + modifying the base class +

+ +

+sealed class Token {
+	final int start;
+	final int end;
+	final File file;
+
+	const Token({required this.start, required this.end, required this.file});
+}
+

+ +

+ But then we'll also have to modify the constructors on the child classes to have the same + information. So you might still want to use composition here. But this would cut down on the of times + you have to write something like token.token_type. +

+ +

Dart seems promising. I'll try writing a parser in it sometime.

+ +

Scripts

+ +

+ So now you can create a parser. But a parser for what language? Say you need a scripting language for + some simple tasks. What can you use? +

+ +

+ Bash is ok, if the only thing you need to do is run some simple commands. But most scripts will end + up being more complicated than that. If your bash script needs a variable, then you shouldn't use + Bash. +

+ +

+ Here we have a different set of priorities. This code shouldn't take too long to write. In many cases + it will be thrown away immediately. We should still make it easy to read, just in case. But we might + not need it for very long. +

+ +

+ Python was designed to be a middle-ground between Bash and C. It's painfully slow, + but our scripts shouldn't take too long to run anyway. The syntax is kinda annoying, but not annoying + enough that I would consider writing a new language to replace it. There are some conveniences + like default parameters that I honestly wish Rust had, but that one in particular was implemented + with some stupid mutability rules. There's no interfaces, only inheritance. + Exceptions are bad. After getting used + to the borrow checker, I don't appreciate the with blocks. So much of the standard library + is deprecated. Python is certainly not the perfect language. +

+ +

+ But it does come with some interesting ideas. Gradual typing was probably the right decision for a + language like this. And I can't complain, because the type system comes with null safety. The idea of + having big integers by default is also really clever. And the large set of builtins is both a + blessing and a curse. It's a good idea for scripting, but not when half of the builtins are + deprecated. The package manager installs everything globally, which is not a great idea, but nothing + else really makes sense for a scripting language. Most scripts don't need their own directory and + node_modules. +

+ +

+ That said, there's more that could be done. I don't like whitespace as syntax, but maybe we could + agree on automatic-semicolon-insertion as a middle ground. A derive-like syntax seems especially + important for a language trying to cut down on developer time. This could even be done with macros. + Speaking of which, I'd like to see interfaces, and Rust's enums. Dart has some nice ideas too, like + the this.x parameter in constructors, as well as named constructors and factories. Hell, + use 1-based-indexing and use a numerical tower as the default number type, or at least use a decimal + type. Python is used mostly by novices, so we might as well make it easier for them. +

+ +

+ At this point, I've essentially proposed my own programming language. To be clear, I don't hate + Python enough to actually make this. I'm just dreaming here. +

+ +

Game Development

+ +

+ Now you want to have some fun, so why not make a video game? What language can we use for that? +

+ +

+ There are several constraints that game engines have that other programs do not. For example, we + really do not want any stuttering, so a garbage collector is out of the question, unless you decide + to manually run the garbage collector on every single frame. However, performance is another + important goal for a video game, so we probably want to avoid that too. That means + Rust is our only option. +

+ +

+ The problem there is that the ecosystem for Rust game development + isn't very ripe. There is the work-in-progress + Bevy game engine, but I honestly don't like the way you query + for components in it. So you'll be making your own game engine. Given that this is Rust, we will + still use an entity-component system. But the queries will be much simpler: +

+ +

+let mut player_sprite = cx.sprites.get("player")?;
+let player_collider = cx.colliders.get("player")?;
+player_sprite.set_position(player_collider.position());
+

+ +

+ It would be really cool if the engine supported hot reloading scripts, but Rust doesn't have that. + We could make it work by compiling the scripts to WebAssembly and reloading them when the file + changes. But this would introduce undefined behavior if the old script's state doesn't match what the + new script is expecting (i.e. a struct definition changed). To resolve this, we will require that + scripts not store any state. Instead, the memory of the script will be cleared every frame. Any + information we need to keep will be stored in the engine code. +

+ +

+fn run(state: ScriptState) {
+	let mut velocity: i32 = state.get("player_velocity").parse();
+	velocity += 1;
+	state.store("player_velocity", velocity);
+}
+

+ +

+ Now, something magical has just happened. We are clearing all state every single frame. All + heap-allocated memory, as far as this script is concerned, is static. This will be the new + memory-allocation strategy. We can use a bump allocator as the allocator. And that means we won't + need a garbage collector (because we're clearing memory every frame), or a borrow checker + (because all heap memory is static). It wouldn't be too hard to adapt an existing garbage-collected + language to use this strategy, so we could just pick one that can theoretically be compiled to + WebAssembly, and make a compiler that doesn't insert a garbage collector. That could include C#, + Dart, Kotlin, or Swift. But we could also make a custom language suited to this task. This could + make it even faster than Rust, because bump allocators are so fast. +

+ +

+ In fact, I don't think the perfect language for game development exists yet. I want something easy to + embed, like Lua, but statically typed⁶. If we could + block the use of the file system or network, then we could ensure that mods don't contain any + viruses. But the language server would need some way to know what libraries are being imported, and + what they contain. +

+ +

Another useful idea is a property declaration. Here's an analogy:

+ +

+// this is determined at compile-time, and not modifiable at runtime
+const PLAYER_SPEED: usize = 5;
+
+// this can be mutated (not in Rust, but in other languages), but an initial value must be given
+static PLAYER_SPEED: usize = 5;
+
+// this is a value inserted by the game engine
+property PLAYER_SPEED: usize;
+

+ +

+ The engine would need to parse the script to determine what properties exist, give the developer a + slider that they can use to change it, and then insert that value into the script. Technically, this + would probably just be syntactic sugar for calling a function. But you wouldn't have to worry about + typos in the property name. +

+ +

+ I'm sure are other features that I think could be useful as well, but this proves the point that a + language specific to game scripting could be useful. +

+ +

Backend Web

+ +

+ Now you have a video game. Maybe this game has online features. In that case, you'll need a web + server to handle communication between players. Once again, we need to pick a language to write it + in. Let's take a look at the usual suspects. +

+ +

Rust (or Swift)

+ +

+ Rust is great because of the exceptional error handling. If all errors are handled using the + Result type, and errors are sent to the client using the Result type, + then you guarantee no cryptic 500 errors, except in the case of a panic. Swift also has a Result + type, so most of what I say here should apply to Swift as well (although I'm not familiar enough + with Swift to tell you when this advice might not apply). +

+ +

+ A point to Rust specifically is the performance and lack of a garbage collector. In most languages, + when the garbage collector starts, the entire program stops. But Rust doesn't have this problem, so + you can send responses at the speed of the network. Another point to Rust is the fearless + concurrency. No data races ever, and bad race conditions should be rare as well. Multithreading is + important to a server, so this is a good thing to have. The ecosystem is also able to provide things + that no other language can. Serde makes JSON and XML + serialization/deserialization a piece of cake. Take a look at SQLx. + Not only can Rust get rid of memory and race errors, it also gets rid of SQL errors. +

+ +

+ AWS supports Lambda functions written in Rust, and even gives you a discount if you do so, since it + runs faster. If you don't like cold boots, then you can always use an actual server application. + There are a few backend web frameworks written in Rust, such as axum, + which everyone seems to love despite the fact that it isn't finished yet. I use + actix-web. It's slightly faster than axum, and it's both + stable and production-ready. +

+ +

Dart

+ +

+ Let's start with what Dart doesn't have. It doesn't have anything as nice as serde or SQLx because it + doesn't have macros. It also doesn't have a Result type. AWS Lambda support for Dart is + also lacking. So all in all, I recommend Rust over Dart. But I wanted to mention + Dart because of the interesting strategy it uses to avoid data races. +

+ +

+ Dart doesn't allow you to share memory between threads. Each thread is its own Instance + with its own heap. But the Instances can communicate with each other over channels. So + there are no data races, just like Rust. +

+ +

+ Now, Dart does have a garbage collector. It's always going to be slightly slower than Rust under + load. But since all of the threads have a separate heap, the garbage collector does not need to stop + your application from running. It only stops one thread. And the garbage collector can be programmed + to run early if a thread has no tasks waiting for it. So in practice, you might never experience a + GC pause. +

+ +

+ There are a number of Dart backend frameworks. I haven't personally tried any of them, and many of + them are deprecated, but I think it's worth checking out if you really want to avoid garbage + collection. Some of them have built-in ORMs and I've seen at least one with built-in OAuth2. +

+ +

WebAssembly

+ +

+ In case you didn't guess, we're going down the same path I followed in my game development ideas. +

+ +

+ The framework itself should be written in Rust, but we can do better for the route code. I really + like the idea of serverless, but cold starts are a problem for me. But take a look at Fermyon's + spin. It uses WebAssembly to get the best of both worlds. + Just like serverless functions, they don't need to be loaded up all of the time. But because + WebAssembly is already heavily sandboxed, they don't need to create a virtual machine to contain your + code. Which makes the cold starts fast. So fast that they've decided to just cold start every single + time. With the help of WASI, Spin can support other languages, such as Go, Python, JavaScript, C#, + and Ruby. +

+ +

+ So you might see where this is going. Again, we can use a bump allocator to avoid garbage collection + and borrow checking. I think spin uses a custom implementation of WASI in order to keep everything + sandboxed, so we could do that too. But, again, custom compilers would be needed, so we could forego + WASI and use a more stable interface that we invent. +

+ +

+ I also think that a new language to support this system could be helpful. We'll need Rust macros, + Rust enums, and a Result type. Although, the macros would probably be implemented in the + embedding, in order to make it more powerful. We'll also want a multithreading system similar to the + one in Dart. Of course this new language would need async/await. Ideally, it would also have a core + library separate from the main one. Ideally, it would also be impossible to panic, but that seems + impossible. +

+ +

+ There's something else we can do now that we're running everything inside of a WebAssembly framework. + Spin provides its own key/value store. The neat thing about this is that we don't necessarily need a + network request to access this store. It's stored locally on the machine. So we can do fast DB access + by creating our own database, and embedding it directly into the framework. I think Spin had this + idea with the built-in SQLite database, but + + SQLite doesn't work well in some cases + . + The key/value store Spin provides is also limited to about 1000 tuples, if I understand correctly. + So I would prefer something with a built-in wide column DB, that doesn't require a network connection + to access. Maybe also a relational database, but those are harder to create from scratch (speaking + from experience). +

+ +

+ I'm not actually convinced that WebAssembly is the best possible thing here either. WebAssembly is a + stack-based bytecode, which is typically slower than one based on registers. This is a good idea for + WebAssembly, because it's on the web, and stack-based bytecode is much smaller than register-based. + So you'll be able to download the code more quickly. But a server doesn't have this requirement. The + bytecode will need to be uploaded to the server, but the size doesn't matter too much after that. So + I could imagine a new bytecode being invented here that takes advantage of registers. It could even + be AOT-compiled once it reaches the server. The same idea applies to games as well. The only benefit + to a smaller bytecode would be a smaller download size, but performance is probably more important + there. +

+ +

Frontend Web

+ +

+ Any language you choose here is going to either compile into JavaScript or WebAssembly. WebAssembly + can be a problem for many languages, since you need glue code to make the language's runtime work + in a browser, which is often the same size as a JavaScript framework. If there were a language that + had a runtime more directly based on JavaScript, designed specifically for frontend WebAssembly, that + could work⁷. But until that happens, I'm going to + recommend avoiding this approach. +

+ +

+ That leaves your options to JavaScript, and languages that compile to JavaScript. This shouldn't be + a surprise to anyone, but TypeScript is your best bet here. I don't think it's + perfect. It takes some of the flaws of JavaScript for itself. But it does make many of them more + bearable. It doesn't cause the JavaScript to get much larger. You get built in lowering to older + versions of JavaScript. And you inherit the whole JavaScript ecosystem. +

+ +

+ Alright, here's the part where I propose a better language. First of all, we shouldn't be inheriting + JavaScript's syntax. If we have to compile to JavaScript from another language, why not compile from + a good one? So things like === don't need to be there. We should also get rid of + undefined. +

+ +

+ Now, one problem I've noticed in most JavaScript frameworks is the difficulty in figuring out which + variables have been updated. Svelte tried to solve this by using the equal sign as a marker that a + variable needs to be updated. But what if I use array.push(5)? Now I need to enter a + hacky array = array to make the change visible. So we'll want some way of detecting + which methods mutate a value. +

+ +

+ Rust, as you may know, has that. Methods that mutate the value take &mut self. How would + we implement this system in Rust? +

+ +

+struct Signal<T> {
+	value: T,
+}
+
+struct SignalGuard<'a, T> {
+	signal: &'a mut Signal<T>,
+}
+
+impl<'a, T> Drop for SignalGuard<'a, T> {
+	fn drop(&mut self) {
+		mark_as_updated(self.signal);
+	}
+}
+

+ +

+ We won't be able to replicate this in TypeScript because TypeScript doesn't have a borrow checker, + and thus cannot detect when a value needs to be dropped. That's the problem that the garbage + collector is meant to solve. So how about we add a borrow checker to TypeScript? +

+ +

+ Yes, I'm actually proposing taking a language which does not have or need a borrow checker, and + adding a borrow checker to it. The borrow checker just makes so many problems easier. For instance, + we said that we needed a language that compiles to WebAssembly but is more closely related to + JavaScript. If we have a borrow checker, and plan the language properly, this language could even + compile to WebAssembly, making it faster than JavaScript. +

+ +

+ I've been working on a web framework for a bit now, and I think this would be a useful feature. I + think I'll try it someday. +

+ +

Conclusion

+ +

+ I didn't mention desktop or mobile apps, but the answer there would be incredibly simple: Dart. Dart + was designed with that specifically in mind, and went through extensive user testing to ensure that + it could be useful for that task. I wouldn't even have any suggestions for what to add to it + (other than Rust enums and macros). Other languages try to have more general purposes, but if a + language is built with a specific goal in mind, it will be good for that task. Python was designed + with scripts in mind, and I recommended it, even though I hate dynamically typed languages. Rust was + designed as a competitor to C++, and it does indeed replace C++ in many use-cases. Icon is a language + that barely exists, and yet I still gave it a shoutout in the parser section. +

+ +

+ The idea of languages being tools is a good one. But I don't think enough thought goes into it. Make + a language with a task in mind. It will be a great one. +

+ +