- Imperial College simulation code for COVID-19 | Clive Best
- In which someone runs the code, and it seems to work reasonably well.
(tags: covid19 simulation model mathematics programming)
- The Imperial College code | …and Then There’s Physics
- Someone else ran it too.
(tags: covid19 simulation model)
- Jared Yates Sexton on Twitter: “PLEASE. Tell people about this. I’m going to provide some history of Neo-Confederate, white-identity, apocalyptic evangelicalism, what I call the Cult of the Shining City. This is who Donald Trump was messaging yesterday wi
- some history of Neo-Confederate, white-identity, apocalyptic evangelicalism, what I call the Cult of the Shining City.
(tags: christianity politics usa evangelicalism)
Occasionally I write about debugging, for the edification of others and to try to explain to muggles what I do all day. I ran into a fun one the other day.
Joel Spolsky’s explanation of Unicode is excellent, but long. In brief: on a computer, we represent letters (“a”, “b” and so on) as numbers. Computers work with binary digits (or bits), usually in groups of 8 bits called bytes. Back in the mists of time, someone came up with ASCII, a way to represent decent American letters by giving each letter a number. All those numbers fitted a single byte (a byte can represent 256 different numbers), so one byte was one letter, and all was well… unless you weren’t American and wanted to represent funny foreign letters like “£”, or some non-Latin alphabet, or a frowning pile of poo.
The modern way of handling those foreign letters and poos is Unicode. Each different letter still has a number assigned to it, but there are a lot them, so the numbers can be bigger than you can fit in a byte. Computers still like to work in bytes, so you need to represent a letter using a sequence of one or more bytes. A way of doing this is called an encoding. One popular encoding, UTF-8, has the handy feature that all those decent American letters have the same single byte representation as they did in ASCII, but other letters get longer sequences of bytes.
The series of tubes we call the Internet is a way of carrying bytes around. As a programmer, you often end up writing code to connect to other computers and read data. Suppose we just want to sit there forever doing something with a continuous stream of bytes the other computer is sending us1:
connection = connect_to_the_thing() while True: # loop forever bytes = connection.recv(1024) # receive up to 1024 bytes from the other computer do_something_with(bytes)
The data that comes back from the other computer is a series of bytes. What if you know it’s UTF-8 encoded text, and you want to turn those bytes into that text?
connection = connect_to_the_thing() while True: # loop forever bytes = connection.recv(1024) # receive up to 1024 bytes from the other computer text = bytes.decode("utf-8") # turn it into text do_something_with(text)
This seems to work fine, but very occasionally crashes on line 5 with a mysterious error message: “UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xe2 in position 1023: unexpected end of data”. Whaaat?
Some frantic Googling of “UnicodeDecodeError” turns up a bunch of people getting that error because they weren’t actually reading UTF-8 encoded text at all, but something else2. So, you check what the other side is sending, and in this case, you’re pretty sure it is sending UTF-8. Whaaat?
Squint at the error message a bit more, and you find it’s complaining about the last byte it’s read. You have to give the
recv() a maximum number of bytes to read, so you picked 1024 (a handy power of 2, as is traditional). “Position 1023” is the 1024th byte received (since we start counting from 0, as is tradidional). That “0xe2” thing is hexadecimal E2, equivalent to 11100010 in binary. Read the UTF-8 stuff a bit more, and you find that 11100010 means “this letter is made up of this byte and the two more bytes following this one”. It stopped in the middle of the sequence of bytes which represent a single letter, hence the “unexpected end of data” in the error message.
At this point, if you have control over the other computer, you might be thinking up cunning schemes to ensure that what it passes to each
send() is always less than 1024 bytes at a time, without breaking up a multi-byte letter. After all, the data goes out in packets, so what you get when you invoke
recv() must line up with the other side’s
send()s, right? Wrong.
The series of tubes is narrower in some places than others, and your data may be broken up to fit. A single carrier pigeon can only carry so much weight, you see, and the RSPB is pretty strict about that sort of thing. All that’s guaranteed is that you get the bytes out in the order they went in, not how many you get out at a time.
Fortunately, Guido thought of this and blessed us with
IncrementalDecoder, which knows how to remember that it was part way through a letter when it left off, so that the next time around the loop, it’ll hopefully get the rest of the bytes and give you the letter you were hoping for:
connection = connect_to_the_thing() decoder_class = codecs.getincrementaldecoder("utf-8") decoder = decoder_class() # Make a new instance of the decoder_class while True: # loop forever bytes = connection.recv(1024) # receive up to 1024 bytes from the other computer text = decoder.decode(bytes) do_something_with(text)
We’ll not worry about the other side closing the connection or the wifi packing up, for now. ↩
I do wonder whether questions on Stack Overflow about errors from Python’s Unicode handling have more views in the aggregate than the “How do I exit Vim?” question (which is at 2.1 million views as I write this). ↩
- post modern C tooling – draft 5
- ‘My ties to England have loosened’: John le Carré on Britain, Boris and Brexit | Books | The Guardian
- “At 87, le Carré is publishing his 25th novel. He talks to John Banville about our ‘dismal statesmanship’ and what he learned from his time as a spy”
(tags: spies intelligence MI5 MI6 le-carre politics)
- The New Zealand Shootings: The Untold Stories | GQ
- A moving account of the shootings and their aftermath. Via Metafilter.
(tags: shooting terrorism racism new-zealand)
- How Derren Brown Remade Mind Reading for Skeptics | The New Yorker
- Introducing Derren Brown to the Americans. Via Mefi.
(tags: magic derren-brown mentalism)
- WSJ, WaPo, NYT Spread False Internet Law Claims | Cato @ Liberty
- Rebutting nonsense about the supposed publisher/platform distinction in Section 230 of the US’s Communications Decency Act. From the Cato Institute, so can’t be dismissed as leftist propaganda.
(tags: law censorship internet)
- Towards an understanding of technical debt | Kellan Elliott-McCrea
- Different categories of things which all get lumped together as technical debt.
(tags: programming technical-debt software)
- Type punning isn’t funny: Using pointers to recast in C is bad.
- A common C programming technique (casting between pointers to structures) leads to problems when strict aliasing is turned on (as it is if you set -O2 -O3 in gcc).
(tags: C programming casting punning)
- Type Punning, Strict Aliasing, and Optimization – Embedded in Academia
- More on the type punning/aliasing business.
(tags: C punning aliasing programming)
- Ask HN: Best way to learn modern C++? | Hacker News
- Thread with book and video recommendations
(tags: c++ programming)
- Epistemic extremism – UseOfReason
- Contra Internet (“shoe”) atheism: I don’t need to be able to prove a thing to you before I can rationally believe it.
(tags: philosophy belief epistemology Atheism proof)
- Compressing and enhancing hand-written notes
- How Office Lens might do it, but open source. Introduces various colour spaces.
(tags: images python colour RGB HSV)