Making of the PETSCII typer

▼ I thought it might be interesting to talk a bit about how my PETSCII typer web tool came together.

It all started with the Drop + Matt3O MT3 retro keycap set. I ordered this set because I like the old style key shape that's replicated here. And then as a bonus you get the PETSCII characters we know and love from the Commodore 8 bit computers printed on the front of the keycaps.

Not sure how I ended up there, but I found the Style64 C64 TrueType fonts, which replicate Commodore's take on the ASCII character set, usually referred to as PETSCII because the Commodore PET computer from 1977 introduced it.

So now I can have PETSCII graphical characters on my keyboard and PETSCII on my screen. The one missing thing: pressing a key and having the PETSCII character on that key show up on the screen. That's what the PETSCII typer does.

PETSCII, fonts and Unicode

PETSCII is actually two (four?) sets of 128 printable characters. One set (mostly) has the 95 printable ASCII characters and then about 30 graphical characters. The other set turns the lower case letters into upper case, and replaces the upper case letters with additional graphical characters. The two sets are also included in inverted (reverse video) form.

Style64's C64 Pro Mono font will display regular text in the Commodore 64's font, including Unicode characters that match PETSCII graphical characters. So when you change fonts, the characters will look different, but you still see the same characters. In addition to that, the four sets of 128 characters are also mapped into the Unicode private use space. Text that uses these Unicode code points will only show properly with the C64 font. If you change fonts, you'll just see a lot of "unknown character" boxes.

When using the C64 Pro Mono font, I try to keep most ASCII characters as ASCII and use the private use area mapping for the graphical characters. I also built in the ability to switch to a different font, which then requires additional character remapping. There are still some bugs there, but I decided to publish the tool now and do some more bug hunting at a later date.

To make things even more interesting, fairly recently additional "symbols for legacy computing" were added to Unicode, which makes it possible to map almost all non-inverted PETSCII characters to Unicode. However, my systems don't implement this version of Unicode (13 or 14) yet, requiring additional filtering/mapping options.

Javascript: mapping keyboards and Unicode

Javascript lets you run some code when the user presses a key. Your code then sees which physical key was pressed, which letter that key produced, and whether any modifier keys such as shift, control or alt were also pressed at the time.

The PETSCII typer keyboard event function then checks if there is a mapping from the key being pressed along with shift/control/alt to a PETSCII character. If so, it puts the mapped PETSCII character in the text area, and then uses event.preventDefault() to prevent the browser from performing the normal action associated with that keypress. Easy peasy. (If the key doesn't map, I just let the browser handle it the way it sees fit.)

But... I'm using the alt key for typing the PETSCII characters, so it's possible to type regular letters without alt. However, on the Mac the alt key (often called option there) can also be used to invoke dead keys for typing accented letters. And for some reason, in that case event.preventDefault() doesn't do its job, and an extra character appears. Turns out there is absolutely no way to get around that. Javascript actually has several events that let you catch dead key character composition, but no help there.

Eventually I did two things: allow the user to use control rather than alt for typing PETSCII characters, and catching the compositionupdate event, then noticing the size of the text in the text area, and then when a keydown event happens, if the text is now longer, remove the extra characters. On Safari this solves the problem, but not on Chrome.

And remember, Javascript handles characters as 16-bit values. So Unicode characters that don't fit into 16 bits (such as the new legacy characters, but not just those) are actually two code points in Javascript. So I had to handle that myself in order to be able to map to/from those Unicode characters.

Pasting rich text

The idea behind the PETSCII typer was that you can use it to type PETSCII and then copy and paste the results into another application, such as a text editor or word processor. To make that work as seamlessly as possible, I set things up such that when you copy text from the PETSCII typer text area (but not cut text), I catch that event and then put the plain text on the clipboard and also an HTML version of the same text with the C64 Pro Mono font specified. So in most other applications, the PETSCII will paste looking the way it should. Although often, it's necessary to tinker with the line height a bit to make sure everything lines up nicely.

(By they way, designers of modern fonts have no idea how box drawing characters and other graphical characters are supposed to work. The amount of whitespace above and below is all over the place so PETSCII graphics converted to fonts like Courier look pretty bad, even if all the characters could be mapped to Unicode.)

Loading files

I thought it would be a nice addition to be able to load PETSCII text files, as I'm not aware of an obvious way to convert those otherwise. The Javascript FileReader event makes the browser ask the user to select a local file, which is then made available to the Javascript script. So even though it looks like you're uploading a file, it's not transmitted over the network, it stays within the browser.

Mapping the characters from a text file to PETSCII in the text area was of course simple enough. Then I thought: what about Commodore BASIC programs? Those use line numbers in binary and they map all the BASIC keywords to "tokens". Again some mapping, so that was fairly easy, too. Although my code doesn't handle control codes inside strings in BASIC.

C64 colors

I couldn't resist the temptation to copy the C64 color scheme along with the PETSCII font. Turns out that there's many takes on how the C64 colors should look. I had a look at Secret colours of the Commodore 64 and Commodore VIC-II Color Analysis, but both delivered very dark and drab colors. (But both interesting reads anyway.) Especially the red is barely red. Eventually I used an HDMI capture dongle to capture the output of my THEC64 but then still increased the saturation a bit for several colors.

Of course today it seems weird that we can't agree on the objective color output specs of a computer. But the C64 and computers like it output NTSC or PAL directly, with no RGB values anywhere. Also, build variations between chip and board revisions and between individual computers made for significant differences. And we all set the saturation on our monitors or TVs to whatever we found most pleasing. But for sure, a Commodore 64 doesn't produce some drab brownish output when you ask for "red".

You can change the text color using control 1 - 8 and shift control 1 - 8, and the border and background colors by entering a number from 0 to 15 the "poke" boxes. And you can enable reverse video (background color becomes text color, text color becomes background color) using control 9 and turn it back off with control 0.

But that adds a new complication: what if the text cursor is on some text in reverse video? So the text cursor ("caret" in Javascript speak) needs to contrast with both the foreground and background colors. So I needed to keep track of (roughly) the brightness of each color, and then either use a bright cursor when text and background are dim, a dark cursor when both are bright, or a middle gray when one is dark and the other is bright.

It's always the small things that take an unexpected amount of extra time. Another example: when you click the "choose file" button, and select the last file you loaded, then no Javascript events trigger. So additional complexity is needed to be able to load the same file again.

Permalink - posted 2022-03-26

PETSCII typer: type your favorite C64 characters

▼ A few months ago, I ordered the Drop + Matt3O MT3 retro keycap set:

These keycaps have the cool/retro "PETSCII" characters printed on the front. But how do you actually type those PETSCII characters?

The first part of the solution is the C64 Pro Mono font by style64.org. This font replicates all the characters from a Commodore 64.

My web-based PETSCII typer tool connects everything together by producing the correct characters as you type as if your keyboard is a C64 keyboard, or using the layout of the MT3 keycap set. You can even load PETSCII text files or C64 BASIC programs which will show up more or less correctly. Or convert to current fonts to the degree possible.

When using the C64 Pro Mono font, if you copy some PETSCII text and then paste that text, it will use the C64 Pro Mono font if that font is installed on your system. This way, the PETSCII looks as intended in a word processing document.

There are still a few minor issues to be figured out, but I wanted to get this out sooner rather than later. Send me an email or a Twitter message if you have any comments or questions.

Permalink - posted 2022-03-23

My first big Ars Technica story in 2007: "Everything you need to know about IPv6"

▼ Fifteen years ago today, Everything you need to know about IPv6 was the first big story I wrote for Ars Technica.

Ah, those innocent days of the past when we still had more than a billion fresh IPv4 addresses to burn through... Back in those days, it was common to hear that IPv6 was unnecessary if we just used NAT. I decided to address this argument proactively as follows:

This is usually when someone brings up NAT. Home routers (and a lot of enterprise equipment) use a technique called "network address translation" so that a single IP address can be shared by a larger number of hosts. The discussion usually goes like this:

"Use NAT, n00b. All 1337 of my Linux boxes share a single IP and it's safer, too!"

"NAT is not a firewall."

"NAT sucks."

"You suck."

Still makes me smile to this day. One thing I didn't remember is how I illustrated the vastness of the IPv6 address space:

40 or 48 bits would have given us more than a trillion or even 281 trillion addresses, respectively, and 64 bits would have been a nice round number. But as the axiom goes, once bitten, twice shy, so the IETF opted for 128 bits this time around. The total number of possible addresses that this gives us:

340,282,366,920,938,463,463,374,607,431,768,211,456

To put this into perspective: there are currently 130 million people born each year. If this number of births remains the same until the sun goes dark in 5 billion years, and all of these people live to be 72 years old, they can all have 53 times the address space of the IPv4 Internet for every second of their lives.

You may also like my last (so far?) IPv6 story on Ars Technica from 2016: IPv6 celebrates its 20th birthday by reaching 10 percent deployment.

As we're discussing IPv6 anyway, a short status update:

Right now, Google's IPv6 statistics show that during week days, 33% of Google users have IPv6, wile during the weekends it's 38%. So it looks like more people have IPv6 at home than at work. These numbers seem to be increasing at a few percentage points per year.

Looking at the map, the dark green countries are all around 45 - 55 % or so IPv6 adoption, with India being in the lead at 61%.

I don't remember when Belgium reached 50% IPv6 adoption, but it was several years ago; they were already at 30% in 2014. They're now at 56.55%. So it seems that reaching 50% IPv6 adoption is fairly doable, but growing (much) beyond that seems to be more difficult.

Permalink - posted 2022-03-08

Making multilingual typing easier with just one new key

▼ Although in the 1980s and 1990s computer were capable of displaying text in many languages, they were limited to one set of characters at a time. So either Western Europe, Eastern Europe, Greek or Cyrillic, but not several of those at the same time. Unicode solved that, and modern computers can (in principle) display all characters found in all languages.

However, when it comes to typing those characters, we're still in 1990, with different keyboard layouts for different languages. Now obviously it would't be workable to make a keyboard that lets you type all Unicode characters. But it should be doable to come up with a system that provides access to all characters and diacritics that are used in latin script languages.

All it takes is one new key. I call it the globe key: 🌐

We'll have to repurpose one existing key to make room for the globe key. On US keyboards, I think the ` key (in the top left corner, left of 1) is a good candidate, as the ` character doesn't occur in normal text, and even in programming it's not used very often. (But don't worry, you can still get ` by typing 🌐 twice.)

The idea is that the globe key gives you access to a new keyboard "level", similar to how the shift or alt / AltGr keys provide access to additional characters, or different forms (uppercase) of the regular characters. When you type 🌐, the next key that you press will give you the following:

Just like the globe key itself, the keys for diacritics (accents) are "dead" keys. So if you type one of those, nothing happens just yet, until you type the letter you want the accent to appear on (or under). So if you want to type "café", that would go as follows:

c a f 🌐 ◌́ e

Now obviously, if you regularly type French, where the acute accent is very common, you wouldn't want to type it like this every time. You'd keep your existing AZERTY layout (or other keyboard that you're using right now) which has a dedicated é key. But what if you need to type an Eastern European name, such as Michał, Enikő or Věra? The globe key comes to the rescue:

🌐 ◌̷ l = ł
🌐 ◌̋ o = ő
🌐 ◌̌ e = ě

How this is different from other approaches

When computers started to become common, various countries were already using different keyboard layouts on typewriters. And that makes sense, as in addition to regular letters and digits, there's only a dozen keys available for punctuation and accented or special characters. So each of those really needs to earn its keep to deserve to get a key of its own, and languages vary wildly in which characters and accents are common.

Since the 1990s, ISO/IEC 9995 has been working on standards in this area. But you're forgiven for not noticing. It looks like so far, only the multilingual Canadian CSA layout based on ISO/IEC 9995 has gained any traction. EurKEY is a more informal effort, but seems to have stalled.

And of course very many people have made their own keyboard layouts that work for them. Others switch between different layouts to be able to type characters from different languages. And of course simply ignoring the issue and letting a spell checker take up the slack is also a widely used strategy.

Unless I'm mistaken, the US keyboard layout is the only one that doesn't use alt (AltGr / right alt on Windows) for typing certain characters. Both ISO/IEC 9995-3 and EurKEY specify a specific alt / shift+alt "level" for typing additional characters. However, the issue with that is that users need to learn the new positions of these characters if they adopt those layouts.

And although spending a bit of time learning something new that is more efficient is usually time well spent, people simply don't like change. And in the real world, often the new and the old exist side-by-side, so that means having to work with different keyboard layouts on different computers. Not great.

Now, I'm all for replacing the more or less random alt / shift+alt levels that are part of existing keyboard layouts with the new one shown above. If you want to do that, make your alt keys or AltGR key the globe key. In that case, you'll have to press the alt / AltGr = globe key along with the next key.

But if you're used to a layout that lets you type certain special characters or accents easily with alt / AltGr, you may not want to be forced to learn a new way to do that. So in those cases, it makes more sense to just sacrifice one existing key to get the globe key so you can type characters not supported by your existing layout, but otherwise keep typing the way you're used to. If you put the globe key somewhere else than under alt / AltGr, you need to press and release it before you press the next key.

Why this layout makes sense

The globe key layout is designed to work well for characters and diacritics that you don't type very often. So the ease of typing takes second place relative to the ease of remembering. As such, the layout is as straightforward as possible.

People used to the US and similar layouts will find ` ~ ! @ # $ % ^ & * ( ) - _ = + [ { ] } | ; : ' " < > / ? where they expect them to be after pressing 🌐. The cedilla, grave, hook and ring accents and (inverted) question mark are found under the keys matching their first letter. Acute, diaeresis/umlaut, tilde and (arguably) vertical stroke are found under the letters that most frequently bear that accent: é, ü, ñ, ł. Circumflex, comma below, dot below are under the keys that look the same. Accents below are all on the bottom row. Similar looking accents are next to each other or above/below each other.

The above applies when using a QWERTY layout. So æ is between caps lock and S, as that is obvious on a QWERTY layout where that key is A. On an AZERTY layout, that same key is Q, but after pressing 🌐, that same key still produces æ. This makes learning the globe layout a bit harder to learn for AZERTY typists, but the advantage is that once learned, it applies the same everywhere.

Whenever possible, the 🌐 key should be blue to make it easy to find.

Permalink - posted 2022-03-06