While trying out illumos, I ran into a problem that boiled down to “sh sometimes doesn’t update PWD when it starts”.

#!/usr/bin/python3

from subprocess import run

command = ["sh", "-c", "echo $PWD"]

# The parent script is run from /export/home/user/foo
run(command)                           #=> /export/home/user/foo (ok)
run(command, cwd="/export/home/user")  #=> /export/home/user     (ok)
run(command, cwd="/")                  #=> /                     (ok)
run(command, cwd="/export")            #=> /export/home/user/foo (wrong)
run(command, cwd="/tmp")               #=> /export/home/user/foo (wrong)

The pwd command works as expected, and Bash doesn’t seem to be affected by the issue.

run(["sh" "-c", "pwd"],         cwd="/tmp")  #=> /tmp (ok)
run(["bash" "-c", "echo $PWD"], cwd="/tmp")  #=> /tmp (ok)

§Is this a bug?

The crux of the problem is that, in all of the failing cases, the current working directory is changed in a way that does not change PWD. The relevant part of POSIX says:

If a value for PWD is passed to the shell in the environment when it is executed, the value is an absolute pathname of the current working directory that is no longer than {PATH_MAX} bytes including the terminating null byte, and the value does not contain any components that are dot or dot-dot, then the shell shall set PWD to the value from the environment. Otherwise, if a value for PWD is passed to the shell in the environment when it is executed, the value is an absolute pathname of the current working directory, and the value does not contain any components that are dot or dot-dot, then it is unspecified whether the shell sets PWD to the value from the environment or sets PWD to the pathname that would be output by pwd -P. Otherwise, the sh utility sets PWD to the pathname that would be output by pwd -P.

That’s… yeah, I’ll try to unpack that. There are three sentences here, in the form “If (A) then {AA}. Else if (B) then {BB}. Else {CC}.” Both A and B are false because “the value is an absolute pathname of the current working directory”, which occurs on both clauses, is false in our case. This leaves us with {CC}:

the sh utility sets PWD to the pathname that would be output by pwd -P.

In simpler terms, when sh launches, if PWD is not equivalent to the current working directory, it must be set to the current working directory with symlinks resolved.

illumos sh fails to do this unless the current directory is / or the user’s home directory, so I believe this is indeed a bug. Interestingly enough, PWD does get updated after a single pwd call.

§Minimal reproducer

The code at the beginning of this post shows one way to demonstrate the problem: changing the working directory without changing PWD. However, the same issue can be reproduced by doing it the other way around: changing PWD without actually changing the working directory.

cd /tmp
PWD=/dev sh -c 'echo $PWD'

• Expected result: /tmp.
• Actual result: /dev.

This is a lot simpler and is what I use in my subsequent tests.

§Which shells and OSes are affected?

In illumos, sh is a symlink to ksh93 (KornShell 93). ksh93 has a weird history and there are now multiple variants of it, but KSH_VERSION shows version 93u+, which greatly narrows down the options.

AT&T ksh93u+ is the last released version of the original KornShell line published by AT&T. A bit of searching led me to the illumos fork, vendored into the main illumos repository at contrib/ast.

To make sure that this is not an illumos-only problem, I also tried reproducing it on Linux. Attempting to build ksh93u+ on my Linux machine led to compile errors that didn’t look trivial to fix. Eventually I had to run Debian 11 (Bullseye), the last Debian version that packaged ksh93u+, and—yes—the problem is reproducible there.

I also tested some ksh93 forks (ksh2020 and ksh93u+m), as well as other KornShell implementations (mksh and oksh). None of these exhibit the problem.

Out of curiosity, I tried the same on a lot of non-KornShell Unix shells and found one where the problem is reproducible: OSH (from the oils-for-unix project).

§What now?

AT&T ksh93 is not maintained anymore so I don’t think reporting a bug there will be useful. I've filed a bug report on illumos but I don’t expect them to have the interest or manpower to fix this niche problem.

As for OSH, I found an open bug report that described a symptom of the same problem, so I added a comment explaining the likely underlying bug. As it turned out, it had been fixed in the development version not long before my comment.

(This post was mostly written in May 2025, when I did all this investigation and bug reporting; I only cleaned up and finished it nine months later.)

Python provides a simple key-value database API in its dbm namespace, and since version 3.13 this includes an SQLite backend. As SQLite is neither simple nor merely a key-value database, there are a few quirks to this dbm.sqlite3 library that I’m documenting in this post.

§Schema and data types

You can see the SQL schema of a dbm.sqlite3 database at the top of CPython’s Lib/dbm/sqlite3.py. In summary:

• Dict table
  • key column (unique, non-null)
  • value column (non-null)

The Dict table is created with default settings. In SQLite that means the table has rowid and does not have STRICT typing.

The key and value columns are declared with BLOB affinity, and because the table is non-STRICT, that means in theory they can contain any SQLite data type (null excluded due to the NON NULL declaration). However, dbm.sqlite3 queries always coerce keys and values to BLOBs first; as long as you always edit the database using dbm.sqlite3, all keys and values will only contain BLOBs.

In practice, you should probably serialise keys and values yourself into Python bytes objects (which round-trip to SQLite BLOBs), because otherwise the database will do it for you in a way that you may not expect. For example:

>>> db = dbm.sqlite3.open("test.sqlite", "c")
>>> num = 2.000000000000002

>>> db[b"direct"] = num
>>> db[b"direct"]
b'2.0'

>>> db[b"via str and bytes"] = str(num).encode()
>>> db[b"via str and bytes"]
b'2.000000000000002'

Handling the serialisation yourself also makes you less likely to forget to deserialise the values you get back. If you use Python’s shelve module as a higher-level abstraction to the dbm database, value (de)serialisation is done automatically for you using pickle (and is thus affected by the code execution security issue mentioned in the documentation).

§Using the database from multiple threads

Python’s sqlite3 module (which dbm.sqlite3 is built on top of) has a connection-level check_same_thread option that, by default, prevents the connection from being used from multiple threads. This option is not exposed by dbm.sqlite3. In other words, dbm.sqlite3 connections can only be used from the thread it is created on.

If your use case needs check_same_thread disabled, the only solution available at the moment is to copy the dbm.sqlite3 source code to your project and modify it there.

Depending on the value of the sqlite3.threadsafety constant, and especially when check_same_thread is disabled, you may need to regulate access to the database using a mutex. Refer to the sqlite3.threadsafety documentation to see the level of locking you need to perform. In practice, if you’re on a mainstream Linux distribution, SQLite is likely compiled with Serialized threading mode enabled, making it safe to use from multiple threads (with the exception of transactions).

The simplest way to handle the threading issue is to check that sqlite3.threadsafety is 3 (Serialized)—raising an exception otherwise—and hope that none of your users encounter this limitation. But that’s up to you; feel free to handle it according to your needs.

This is an account of my experience using the mdoc format to write a manpage, also touching upon the ecosystem around writing manpages in general.

§man or mdoc

When writing a manpage, the first thing you must do is choose between two formats: man or mdoc. The code for the two formats look somewhat similar because they are both extensions to the roff macro language, but they use completely different sets of macros.

In the Linux world, the man format is more commonly seen, and the manpage renderer of choice is usually GNU roff (groff). The newer mdoc format is more often used by BSD operating systems, usually rendered with mandoc. Both groff and mandoc support man and mdoc; there are minor differences—more on those later—but for the most part both formats work on both renderers, as well as on others such as Heirloom doctools.

The way the two formats are usually described is that man is more presentation-oriented while mdoc is more semantic-oriented.

Due to its presentational nature, there are countless ways people have written man pages for even very simple programs. Here is just one example:

.TH MYPROG 1 1970-01-01 "MyProg 2.0"
.
.SH NAME
myprog \- does something
.
.SH SYNOPSIS
.B myprog
.RB [ \-o
.IR output ]

And here is the equivalent in mdoc:

.Dd January 1, 1970
.Dt MYPROG 1
.Os MyProg 2.0
.
.Sh NAME
.Nm myprog
.Nd does something
.
.Sh SYNOPSIS
.Nm myprog
.Op Fl o Ar output

Both may be rendered into:

Volume 1: User commands — MYPROG

---

NAME

myprog — does something

SYNOPSIS

myprog [-o output]

§On being semantic-oriented

mdoc makes you think less about things that don’t matter and lets you just write the stupid manpage and get it over with.

It may not be as flexible as man, but as a result you don’t ever have to think about whether a command-line flag should be written in bold or italics. The renderer will pick one formatting and you just have to trust that it will look acceptable.

mdoc does not always succeed in being semantic-oriented, though; the biggest offender being…

§The SYNOPSIS problem

The traditional “synopsis” manpage section is treated in a special way in mdoc. For this special treatment to work, the title has to be written exactly SYNOPSIS, in English, in all-caps.

These two sections will be rendered very differently despite the bodies being identical in the source:

.Sh SYNOPSIS
.Nm pkg Cm install Ar package
.Nm pkg Cm uninstall Oo Fl \-purge Oc Ar package
.
.Sh Not synopsis
.Nm pkg Cm install Ar package
.Nm pkg Cm uninstall Oo Fl \-purge Oc Ar package

Result:

SYNOPSIS

pkg install package
pkg uninstall [--purge] package

Not synopsis

pkg install package pkg uninstall [--purge] package

Luckily, groff at least accepts Synopsis, and the mandoc author has said that they want to support that as well in the future. I’m not sure if either project supports or plans to support non-English section titles. Frankly I find it extremely questionable for macros to behave differently based on the title of the section they are in.

§roff-isms

Despite being not horrible in terms of readability, mdoc is still just a roff extension and you still sometimes need to deal with oddities stemming from the roff syntax.

The \& dummy escape sequence is one such oddity that you may find in several instances:

• To prevent a period (.) at the end of a word from being considered the end of a sentence. For example, to write “fruits, e.g. apples” you have to write fruits, e.g.\& apples; otherwise the renderer will put two space characters after “e.g.” when rendering to the terminal.

• To prevent a period (.) or a single quote (') at the start of a line from being considered the start of a roff command. For example, to write “...” at the start of a paragraph you have to write \&....

• Not actually a roff-ism, as macro nesting is an mdoc feature, but \& is also used to prevent a string in a macro line from being considered a nested macro invocation. For example, .Em At midnight prints out “AT&T UNIX midnight”, not “At midnight” with emphasis. To avoid the “At” being interpreted as a macro call, you have to write .Em \&At midnight.

Other escape sequences often encountered:

• \e, \(rs, or \[rs] for \, which is the (default) escape character.

• \(aq or \[aq] for '. The plain ' character itself may be rendered as ’ to cater for English contractions such as “can’t”.

• \(dq or \[dq] for ". The plain " character is used on roff macro lines to quote string arguments.

• \- for ASCII -, as opposed to some other dash or minus character that the renderer may decide to turn plain - into. In practice I don’t think you need this on groff and mandoc.

§Editor support

Emacs, Vim, and Neovim support roff syntax highlighting but do not understand mdoc macros. And that’s before even talking about code completion, error checking, or any form of in-editor documentation; none exist as far as I can tell. There are a lot of mdoc macros, all with very terse 2- or 3-character names, some with very particular syntaxes, and without proper editor support I found myself constantly looking things up in the documentation.

To help make things more convenient for myself, and perhaps eventually others, I started writing a (Web-based) mdoc editor with some of these features. But because that ended up being a huge digression from what I was originally trying to do—which was simply to write a manpage for my program—I’ve shelved this editor project for now. It’s in a usable state, though, and it was quite helpful for me when writing my manpage, so maybe I’ll clean it up and publish it one day.

§This all looks so annoying; can’t I just use Markdown?

That was my thought at various points in this journey, and I kept going back and forth between “I should write this in mdoc”, “I should write this in man”, and “I should write this in 〈some intermediate format〉”.

There are several projects that let you write manpages in Markdown or similar ‘human-readable’ markup languages and compile them into man or mdoc format. The Git project, for example, uses AsciiDoc, which is not Markdown but has a similar goal.

There are also a few XML-based solutions that you can try if that’s more your style. DocBook would be the most well-known option, although my experience using it to produce a manpage has been very poor.

I’ve looked into most of these manpage transpilers, but for the particular project I was working on I decided to stick with plain mdoc for the time being. Even so, I encourage you to try them and see if there is one that fits your project. The simpler you make the documentation process, the more you and your peers will want to write documentation.

It has always bothered me that the Node.js standard library doesn’t provide an easy way to create ZIP files. What the standard library does have is an implementation of the Deflate algorithm, which is the compression scheme normally used in ZIP files. This means that the most complicated and performance-sensitive part of the job has already been done for us, and all we need to do is add the wrapper. So I decided to write it for fun: a ZIP implementation for Node.js that uses the standard library’s Deflate implementation.

This post contains some (very unorganised) notes that document the trickier parts of this process. Even though my implementation only covers the compression side (zip, not unzip), some of these notes are applicable to the decompression side as well.

Note: In this post, Explorer refers to Windows Explorer on Windows 10, 7-Zip refers to the official 7-Zip program (not p7zip), and Info-ZIP is the project supplying the zip and unzip programs included on most Linux distributions and on MSYS2/Cygwin.

§What we need

First we need the ZIP specification. I was pleasantly surprised to learn that this specification is so easily available and freely implementable by anyone. PKWARE, the creators of the ZIP format, have continually published it in a single text file named APPNOTE.TXT. What follows are a number of decisions you need to make depending on how complete you want your implementation to be.

There are two types of ZIP files: (normal) ZIP and ZIP64. The biggest difference is that a normal ZIP file uses 32-bit values for file sizes and offsets, while ZIP64 uses 64-bit values. The details of ZIP64 can be a bit weird due to it being shoehorned into an existing format that was not designed to accommodate such a change. You can opt to ignore ZIP64, but that will mean that you won’t be able to handle any individual file greater than 4 GiB—this includes any uncompressed file size, any compressed file size, or (roughly) the size of the ZIP file itself.

We don’t actually need compression! ZIP files can store uncompressed files, though obviously that would greatly limit the applicable use cases. Technically the ZIP specification supports many different compression formats, including modern ones like xz and Zstandard, but in practice you should probably stick to Deflate and maybe Deflate64 because those are the ones with widespread support (and by that I mean they are the only ones supported by Explorer). As mentioned previously, Node.js includes a Deflate implementation; this is available in the zlib module.

ZIP files use CRC-32 for error detection. Node.js again has our back here, as the zlib module also contains a CRC-32 implementation. If you want to write your own implementation, the code listing in RFC 1952 is very easy to follow.

File sizes (compressed and uncompressed) can be indicated either before or after each file payload. Putting them before the payload results in slightly simpler ZIP file structure. However, if your use case involves live streaming of the ZIP file, putting file sizes after each payload may be your only option.

A proper implementation might include concurrency/multithreading capability. The simplest way probably is to run the compression procedure on multiple input files at once, outputting to temporary files, and then combining them into the actual ZIP file.

§High-level view

The ZIP specification is fairly good in terms of giving a general overview of the format. Refer to APPNOTE.TXT section 4.3 “General Format of a .ZIP file”.

Ignoring some of the fancier features like encryption, a ZIP file consists of a bunch of file sections (each including the contents of a file and some metadata), followed by a bunch of “central directory records”, and finally followed by an “end of central directory record” section. Each central directory record partially duplicates some of the metadata specified in a file section. The purpose of the central directory is to locate the file sections within the archive (which, back when people regularly had to use split/multi-volume archives, could be in a different disk).

§File metadata times

By default, each file in a ZIP archive has limited metadata: name, size, modification time, and OS-specific attribute bits. The modification time is in the horrific MS-DOS date/time format, which can only represent years from 1980 to 2107 (inclusive), has a granularity of two seconds, and doesn’t specify a timezone (most implementations will use the local timezone, meaning that the output is not portable). Use extended metadata if you want to be more accurate than that, or ignore the problem and live like it’s 1989, when ZIP was first released.

§The choice of file attribute format

The external file attributes section of a file entry header allows specifying file attributes based on the host operating system. For example, Windows file attributes include the Archive flag while Unix file attributes include permission bits. If you want to only ever write in one format, though, the Unix format may be the best choice because that’s the platform where users often have to deal with file permissions.

Info-ZIP, being normally a Unix-only program, uses the Unix format even on Windows. The way it determines the mode of a file comes directly from the Cygwin emulation layer, which sets the executable (x) permission bits on a file if any of these applies:

• The file name ends with .exe (executable) or .lnk (shortcut). Since 2022 .com (DOS executable) is not in this list anymore.
• The file content starts with MZ (Portable Executable (PE)), #! (Unix script), or : followed by carriage return. I have no idea what that last one is for.

However, you probably shouldn’t copy this behaviour, because the executable bit is irrelevant when extracting on Windows. What you should do instead is think about what could be considered executable on Unix. Obviously you want to keep the shebang detection. You may want to add ELF file detection (0x7f followed by ELF) because, for example, the user could be cross-compiling from Windows to Linux and wants to zip up the output. The .exe and/or MZ heuristics are still valuable due to things like Wine, Mono, and .NET Core combined with binfmt_misc.

Of course, you can also ignore all this and store every file with no executable permissions. Or you can use the Windows file attributes format when writing on Windows; this is not something I explored so I can’t comment much on it.

§How various zip and unzip programs deal with symbolic links on Windows

Windows distinguishes between a file symlink and a directory symlink—you can make a file symlink pointing to a directory or vice versa if you wanted to for some reason, and when creating a symlink to a nonexistent location you must decide which type you expect the target to be. This distinction does not exist on Unix and cannot be made when the file metadata is recorded in Unix format.

zip:

• Explorer and 7-Zip (on Windows at least) do not support recording symlinks in ZIP files. Both will only store file copies.
• Info-ZIP will normally also store file copies unless given the --symlinks flag, in which case it will store symlinks as actual symlinks. Since Info-ZIP always uses the Unix metadata format, the distinction between file symlink and directory symlink on Windows is lost.

unzip:

• Explorer will straight up ignore the Unix symlink flag and extract the symlink as a text file containing the target path.
• On Windows, 7-Zip will always extract a Unix symlink as file symlink.
• On Windows, Info-ZIP will extract a Unix symlink according to whether the target path points to a file or a directory; if the target is nonexistent, it reports an error and skips the symlink.

§Preventing security issues from very large extraction sizes

This is absolutely irrelevant to creating ZIP files (only to extracting ZIP files), but I’ve included it here because it’s a fascinating topic.

A ZIP file that is designed to cause resource (CPU/disk/memory/bandwidth/stack/filedesc) exhaustion is called a ZIP bomb. They employ one or several forms of nasty tricks; this is a summary of what you need to look out for, but there may be others I’m not aware of.

• It’s actually very easy to achieve compression ratio of millions (yes, 1000000-to-1 or higher) naturally without any special tricks if you have highly-compressible data, such as a long string of identical characters, leading to surprisingly large decompressed files. When handling untrusted data, enforce upper limits on the amount of memory and disk space you can use for extraction. If you are developing a ZIP extractor, give users the ability to set these limits.

• The uncompressed size declared in a file header may not match the actual decompressed size of its payload. While decompressing a file, check that it hasn’t exceeded the stated uncompressed size. Also note that the uncompressed size for a file is listed in two places (local file header and central directory), neither of which can be trusted.

• If you are developing software that wants to analyse ZIP files recursively (inside one another)—for example an antivirus program—be aware that a tiny ZIP file at the top level can lead to absolutely gargantuan files a few levels deeper, and that a ZIP file can be crafted to nest infinitely (!). Do not analyse or extract ZIP files recursively in an unbounded manner.

• A file entry can be referred to multiple times from the central directory; similarly, it’s also possible to nest or overlap file entries. This results in the same payload potentially being extracted many times. A ZIP extractor needs to scan or keep track of file entry start and end points and make sure that none of them duplicate or overlap.

§The result

This is really nothing to write home about, but here is the code of my Node.js-based ZIP archiver if you are interested. As the project name suggests, it’s a basic implementation that takes shortcuts to produce the simplest (but still compressed) ZIP files. The code is licensed under AGPL-3.0-or-later.

While comparing a Rust executable for Windows targetting x86_64-pc-windows-gnu and one targetting x86_64-pc-windows-msvc, I noticed that the -gnu one included an embedded application manifest resource. This particular manifest does two things: setting requestedExecutionLevel to asInvoker, and setting supportedOSes from Windows Vista to Windows 10.

As far as I can tell, the first part attempts to disable the Windows installer detection heuristics. However, the documentation appears to indicate that these heuristics are only used on 32-bit binaries, and the fact that the -msvc executable doesn’t have the manifest reinforces the idea that it’s not needed.

The second part of the manifest is only useful if you want to indicate that you don’t support Windows versions prior to Vista. I think for most people that would be the default assumption these days.

These things considered, it looks to me that removing the manifest shouldn’t cause any issues. The problem is that there doesn’t seem to be any built-in way to do this provided by either the OS or the compiler toolchain. You may have to rely on a third-party tool to do this.

If you don’t mind deleting all embedded resources in the executable—by default there will just be the application manifest—you can use this simple C code (replace file.exe with your executable path):

#include <windows.h>
int main(void) {
	return 1 - EndUpdateResourceW(BeginUpdateResourceW(L"file.exe", 1), 0);
}

Less-barebones Rust alternative:

use std::{
	ffi::{c_int, c_void},
	os::windows::ffi::OsStrExt as _,
};

unsafe extern "system" {
	fn BeginUpdateResourceW(pFileName: *const u16, bDeleteExistingResources: c_int) -> *mut c_void;
	fn EndUpdateResourceW(hUpdate: *const c_void, fDiscard: c_int) -> c_int;
}

fn main() {
	for path in std::env::args_os().skip(1) {
		let mut encoded = Vec::from_iter(path.encode_wide());
		encoded.push(0);
		let handle = unsafe { BeginUpdateResourceW(encoded.as_ptr(), 1) };
		if handle.is_null() || unsafe { EndUpdateResourceW(handle, 0) } == 0 {
			eprintln!(
				"Failed removing resources from \"{}\"",
				path.to_string_lossy(),
			);
			std::process::exit(1);
		}
	}
}

Or in Python:

import ctypes
from ctypes import wintypes
import sys

kernel32 = ctypes.windll.kernel32

def check_trueish(result, *_):
	if result:
		return result
	raise ctypes.WinError()

BeginUpdateResource = kernel32.BeginUpdateResourceW
BeginUpdateResource.restype = wintypes.HANDLE
BeginUpdateResource.argtypes = (ctypes.c_wchar_p, wintypes.BOOL)
BeginUpdateResource.errcheck = check_trueish

EndUpdateResource = kernel32.EndUpdateResourceW
EndUpdateResource.restype = wintypes.BOOL
EndUpdateResource.argtypes = (wintypes.HANDLE, wintypes.BOOL)
EndUpdateResource.errcheck = check_trueish

for arg in sys.argv[1:]:
	file_name = ctypes.create_unicode_buffer(arg)
	handle = BeginUpdateResource(file_name, True)
	EndUpdateResource(handle, False)

The slightly bad news here is that, in my testing, removing this manifest only reduces the executable size by exactly 1024 bytes. Considering the x86_64-pc-windows-gnu target generally produces executables in the hundreds of kilobytes at least, this is a fairly inconsequential saving which I probably won’t bother with.