bsnes :: Legacy Formats2011-02-27
bsnes is a personal project. I do not release it publicly for the sake of popularity. Rather, it is released as a research platform, in the hopes of advancing SNES emulation as a whole.
As such, there is no incentive for me to target popular legacy formats that are in wide use. Further, I am strongly opposed to the idea of sticking with inferior file formats simply because they came first. I prefer instead to focus on technical merits and choose the most logical options possible. It is not a matter of the complexity involved in supporting these old formats, it is that I feel a philosophical hypocrisy in eschewing these formats while implicitly endorsing their continued existence by supporting them.
On this page, I will explain my rationale for not supporting various legacy formats. I do not ask you to agree with me on any of these, merely to respect my choices. As bsnes is an open source emulator, you are of course free to make any modifications that you like, and even to distribute those modification.
I truly believe that many people tend to be very head-strong about these sorts of issues, so I ask that you please read this with an open mind, and let logic win out over passion and tradition.
File extensions
bsnes will only load SNES cartridges ending in the ".sfc" extension, BS-X flash cartridges ending in the ".bs" extension, Sufami Turbo cartridges ending in the ".st" extension, and Game Boy (Color) cartridges ending in either the ".gb" or ".gbc" extension.
The most popular SNES extension is ".smc", and this tends to be used for not only SNES cartridges, but also for BS-X flash cartridges and Sufami Turbo cartridges as well.
The reason ".smc" is the dominant extension is due to the origins of SNES software backups. The first mainstream devices to allow this were Chinese-based copiers, devices that would plug between the SNES and the cartridge, and copy games onto floppy disks. One of the first such devices was the Super Magicom. ".smc" is an abbreviation for this device.
Rather ironically, the Super Magicom was a rather niche copier, quickly supplanted by the Super Wildcard, Game Doctor and Super UFO copiers, which were much more popular. But following the initial images, other dumpers and even copiers chose to support this first extension.
But it is by no means the only extension. Here is a list of some of the most common extensions, as well as what they stand for ... this list is by no means exhaustive:
- SFC - Super Famicom
- SMC - Super Magicom
- SWC - Super Wildcard DX
- FIG - Pro Fighter
- UFO - Super UFO
- SF2 - Professor SF II
- GD3 - Game Doctor 3
- GD7 - Game Doctor 7
- DX2 - Super Wildcard DX2
- MGD - Multi-Game Doctor
- MGH - Multi-Game Hunter
- 048, 058, 068, 078 - Split images for Game Doctor
- USA, EUR, JAP, AUS, etc - region-specific images
- BSX - alternative extension for BS-X flash cart images
- BIN - short for binary image
Believe it or not, all of these extensions are still in public use. To me, this is ridiculous. An extension should indicate what it represents. bsnes is not a Super Magicom emulator, it is a Super Famicom (SNES outside of Japan) emulator.
There is of course the pragmatic approach of simply supporting all of these extensions. However, I feel this introduces unfortunate downsides. First, consider how operating systems such as Windows handle file-assignment. It is based on extension. So if you want to set up your SNES images to open in your favorite SNES emulator, you must assign each of the above extensions to point to your emulator of choice. Now what happens if you decide to change emulators? You have to do it all over again.
This further complicates my emulator in particular: I have special support to load folders that end in a specific extension as though it were a complete cartridge with all associated files (ROM, RAM, memory mapping data, etc.) inisde of it. What would be a simple wildcard search becomes a rather complex list-based search instead.
And then there are the false positives. SNES emulators do not exist in a vacuum, and it is not inconceivable that other programs share these extensions. The more extensions we attempt to claim, the more likely we are to match false positives. Most notorious for this is the ".bin" extension, which is used almost universally.
Lastly, there is my choice to use ".bs" for BS-X flash cartridges, and ".st" for Sufami Turbo cartridges. By using separate extensions for separate files, my custom cartridge-type loading dialogs make more sense. When selecting a base SNES cartridge, you would not want to see your BS-X flash cartridges; and when selecting a BS-X flash cartridge to connect to a base cartridge, you would not want to see your actual SNES cartridges.
Copier headers
The old copier devices I mentioned earlier had a tendency to be rather lazy. Rather than perform two seek+read commands to determine the memory mapping mode from the internal cartridge information stored in every SNES image, copiers instead chose to store a 512-byte header at the top of every image to specify this information instead. That specific size was chosen at it is the exact size of one floppy disk data sector.
These copier headers are of no use to emulators. In fact, they only get in the way. They cause checksums to fail, they require custom loading code to strip the data from them, they complicate detection of the legitimate internal cartridge headers, and they cause patching issues.
Perhaps even more ironically, they aren't even of use to copiers themselves. You see, copiers are extremely obscure. These days, flash cartridges that can store headerless images on CF/SD cards are far more accessible and common. There are no manufacturers still making SNES copiers, and very few people still even have them. Copiers are slow, bulky, and cheap. They break very easily, designed with shoddy parts like EEPROM chips for BIOSes that don't even have tape over the UV windows.
Further consider all of the different types of copiers I have mentioned above. Their headers are, for the most part, incompatible with one another. And even the file extensions do not help anymore. As most everyone renames SNES image extensions freely, especially to ".smc", one cannot even tell which, if any, type of header is on any given SNES image.
And as if all of that wasn't enough, nearly half of the games in the SNES library are too big to fit on a single floppy disk anyway, meaning that the images need to first be split, and headered for this operation, before they can be used.
So to cater to at best 0.01% of potential SNES image users, with a perhaps one in a hundred chance of getting an image with a compatible header in the first place, we continue to keep copier headers around to this day.
Compressed archives
I remember my first hard drive. It had 200MB of storage, and it was very expensive. The early '90s were something. But we are in the 21st century now, and we are talking about a 20-40% compression savings on files that are at best 4MB in size, on $99 hard drives with 2TB worth of storage space.
And there are only ~2,000 unique SNES games. Doing the math, we see that it costs, at best, ten cents to store a complete SNES image set. So why do we still use compression? Is it bandwidth? The days of 28K dial-up are long past, but even then, it is not so unreasonable to distribute a collection of games in an archive, and require extraction first. As you would for literally any other file type in the world (images, movies, songs, etc.)
But why not support compression anyway? Well, many reasons. First, writing decompression code is hard. Very hard. We pretty much have to rely on third-party libraries, and these libraries are often larger than our entire emulators. zlib is something like 200-300KB of source code, which is about four times the size of my entire Game Boy emulator. And that's just for ZIP alone. People also want 7-zip, RAR, JMA, Gzip, etc support as well.
And then there are the licenses. Each of these decompression libraries impose restrictions on us emulator authors in order to use them. If I were to add RAR support, no Linux distribution would touch my emulator, because of their ridiculous licensing policies. Using JMA without a license exception would require an emulator author to release all of their source code, and allow others to sell their work for money. Not so great for non-commercial projects.
All of this code adds up, significantly slowing down compilation times. It may not seem like much, an extra minute or two every compile. But consider that I have compiled bsnes at least 20,000+ times now, and that really starts to add up after a while. Time is valuable.
All of that code also bloats executables. Programs take longer to start up, programs are larger as a result of all of these DLLs, and so on. And then there are the shared libraries. Every Linux distro expects you to use their shared libraries or they will throw a fit. But Windows doesn't have shared decompression libraries, making everything that much more complicated.
And then trying to load these images. Having to decompress a game first adds to the time it takes to start playing. A good second or so for each game you load, wasted, on file decompression. Get into 7-zip solid archives and things can get really crazy. The popular "Super Mario World.7z" bundle has nearly one thousand images inside of it. Loading a single game from this archive takes a good 5-15 seconds, depending on CPU power, and a monstrous 1GB of RAM to do so. Solid archives require you to decompress all data before it, so you pretty much have to extract and cache 1,000 games in memory, just to play one. Ridiculous.
I also mentioned the issue of false-positive matches based on generic file extensions, and trying to associate images directly to SNES emulators. This becomes two orders of magnitude worse when using popular archive extensions.
Perhaps worst of all is just how meaningless this all is. Even if, for some reason, you have a legitimate need for compression ... modern operating systems have compression support built-in! Take Windows and NTFS for instance. Simply right-click on a folder, choose properties, choose advanced, and tick "compress folder and subfolders to save space." and hit okay. The operating system will compress all of the data for you, and decompress it completely transparently to individual programs and users.
Doesn't that sound more rational? Have the OS integrate a common operation, rather than having every last software program support every last popular compression format of the month.
IPS patches
IPS was a format used to distribute patches for Amiga applications. It is a very old, inferior format, with many limitations. IPS patches cannot be applied to images larger than 16MB, they lack checksum information to verify that you are applying to the correct file, or to verify that the patch was applied successfully to a given file.
When combined with the issue of copier headers, this has become a very annoying issue for SNES ROM hackers and fan translators. What happens when you present people with two choices? Some choose one, some choose another. And since even the SNES hackers cannot agree on whether to make their IPS patches against headered or unheadered images; let alone end users and ROM verification tool authors; IPS patches have, at best, a 50/50 chance of actually applying successfully to a given game. Throw in various game revisions, quite popular on the SNES, and your odds of success go down even more. And since you can't verify successful application, the only way to tell is to see if the game starts, or if you get a black screen upon load.
Once this is determined, the user is now tasked with figuring out what is wrong. It is not exactly common knowledge how to add or delete 512 bytes of data from the top of a binary file, so most users give up in frustration.