ashg: compressed sprites and textures

Source

docs/formats/ashg.md

SHA-256

c6df9f8c…697d4c0b

An ashg is a 16-byte header and an LZ77-style stream that unpacks to one resource: a btx texture (portraits, class faces, shop pictures) or a sprite pack, a pakd of a sprite sheet and two btx layers. The sheet is an image entry with no magic, which Gibbed reads as ImageFile. Sprite packs are also stored uncompressed, often in the same files. This doc covers the ashg wrapper, sprite packs and image entries. The textures are in btx.md, and the animations that draw the sheets are in sas3.md. Gibbed calls the compression RLE.

Where they are

confirmed (C# scan, all 10,016 .dat files): an ashg starts a pakd entry or a whole payload, never anything else. Every one unpacks to a btx (2,026) or a sprite pack (2,259). Entry paths are as in containers.md.

• confirmed: a unit sprite file is always a pakd of two entries: an uncompressed sprite pack, then an ashg holding another sprite pack. The two sheets differ in size, 512 × 384 (1,960 sheets) against 1,024 × 256 (1,965) in most files. Source: C# scan.
• likely: the unit's draw code picks one of the two by a flag of the unit (+0x53, passed to 0x1402e5180). Entry #0 is used in place. Entry #1 is unpacked into a slot. Source: 0x14033e6c0 (Taco::Application::UNIT::vfunction10), 0x1402e5180, 0x1402e5220.

Header

16 bytes. Offsets are from the start of the ashg.

• confirmed: as a pakd entry an ashg is stored with 1–16 zero bytes after it, to align16(packed_size + 1): the usual entry padding (containers.md). As a whole payload (menu/face/, menu/shopbg/) the payload is exactly packed_size bytes. Source: C# scan, all 4,285.

Stream

Method 2, from byte 16. Tokens follow one another until unpacked_size bytes are written. Every stored stream fits this description (C# scan, all 4,285 decode to exactly unpacked_size bytes using exactly packed_size − 16 bytes). The decoder is 0x1403852b0, so the token layout is also likely from the code.

So with window_bits = w, a match reaches back at most 2^w bytes and copies 3 to (0x7FFF >> w) + 3 bytes (258 at w = 7). The files use runs of up to 127 bytes, matches of up to 258 bytes and offsets of up to 2^w.

• likely: method 1 (unused): a control byte c gives a count (c & 0x7F) + 1. With bit 7 set the next byte is repeated that many times; otherwise that many literal bytes follow. Source: 0x140385610.
• likely: method 0 (unused): unpacked_size bytes are copied from byte 16. Source: 0x140384ff0.
• likely: the decompressor checks nothing. There is no bound on the output, and a match can reach before the start. It reads input past the length it was given from its second input pointer, which every caller passes as null. So a stream that runs past packed_size, or past the entry, would crash the game. Source: 0x140384ff0, 0x1403852b0; all five callers pass 0 as the second buffer.

Sprite packs

A pakd (containers.md) with an ids table whose first ID has 0x1FF1 in its high 16 bits. confirmed (C# scan, all 6,196):

• confirmed: the mask is always the sheet's size. The aura is 8 × 8 in most packs, and 96 × 144 or 112 × 144 in others. Source: C# scan; the btx width and height (btx.md).
• confirmed: the header's unk_header_end word and padding are zeros, and each entry is padded as usual. The pack's own size (pack_size) is a multiple of 16. Source: C# scan.
• likely: the game reads only the first ID. With 0x1FF10000 it registers entry #0 alone. With 0x1FF10001 it registers #0 as the sheet, then #1 as <name>_aura and #2 as <name>_mask, at the offsets in the header. With any other first ID it registers nothing. It never reads count or the other IDs. Source: 0x1402702a0.

Image entries

The sheet: a 16-byte file header, a 32-byte image header, then a texture, cluts and a clut table. confirmed (C# scan, all 6,180): in the file, the image header is followed by the texture, then the cluts back to back, then the clut table, then zeros to a 16-byte boundary. Every byte up to the entry's total_size is one of these. As a pakd entry it is then padded to align16(total_size + 1).

File header

Image header

At header_size. Its offsets are from its own start.

Texture

At the image header + texture_offset. Offsets are from the texture's start. confirmed (C# scan, all 6,178 textures): every value given as fixed below holds in every one.

Cluts

Each at clut_base + its table offset. Offsets are from the clut's start. confirmed (C# scan, all 2,037 cluts): every one has 256 colours, so every one is 0x430 bytes.

Command words

Eight u32 PSP display-list words, the same in every texture and in every clut (C# scan): confirmed for the values.

On the PSP these are texture mode, clut mode and clut load commands (Gibbed Texture.ValidateGECommandsReborn, Palette.ValidateGECommands). The PC game reads only bit 0 of the texture's first word (0: not swizzled), so likely none of the rest matters. Source: 0x14026fb20.

• likely: a texture is created without its clut, as a texture of 8-bit indices (0x14026fb20 passes no clut to 0x14019e500). The clut is given to the draw separately, by index into the image's clut list (0x14026f250). Unit and weapon sheets have no clut of their own. Their colours come from the 65 cluts of battle_data_release_2 #2/10 and the 16 cluts of weapon_graphics #1/0, chosen by the draw code. The weapon draw 0x1402e5f20 registers #1/0 and picks a clut with 0x14026f250.

How the game reads it

Every fact here is static analysis of the unpacked executable (Ghidra 12.1.4, the decompiled code in local/exe/decomp/), so at most likely. The functions are in the names list.

• likely: there is one decompressor, 0x140384ff0. It reads the 16-byte header, then dispatches on method. It reads its input as two pieces, a buffer of a given length and a second buffer for the rest; every caller passes null as the second. It never checks the magic, crc or unk_0x0F, nor whether the output fits. 0x140385900 returns unpacked_size. Source: 0x140384ff0, 0x1403852b0, 0x140385610; its five callers are in the table below (exe.calls). A search of the decompiled code for the match-code arithmetic finds no other decoder.
• likely: every loader finds its ashg by position: an entry index, a pakd ID or a whole file. It unpacks into a fixed slot, except the shop, which allocates unpacked_size bytes. A bigger resource would overflow its slot without any check.

• likely: unit sprite files are cached whole in 105 slots of 0x11E000 (1,171,456) bytes. A bigger file takes several slots in a row, so there's no limit per file, only 105 × 0x11E000 in all. Source: 0x1402e4bf0.
• likely: a unit's two object/ files (the IDs in entry_unit_1) are read as follows. The bundle (group 30) is read whole. Each entry goes by its pakd ID: kind 0 (high 16 bits) is a unit sprite file, cached under the low 16 bits; kind 1 is a face, cached the same way. The plain file (group 20) is read only when no bundle entry already gave that sprite ID. Source: 0x1402efac0, 0x14031eeb0.
• likely: an event scene finds its entries by pakd ID through 0x140350380, which subtracts a base per kind: 3 → 0x11000 (faces), 4 → 0x12000 (sprite and animation), 5 → 0x50000 (unit sprite files); kind 8 (0x13000) is the scene's sound banks (sabf.md). Source: 0x140350380 and its callers 0x1403593b0, 0x140323530, 0x1403267c0.
• likely: everything unpacked or found ends up in one register routine, 0x14026f4b0, which picks by the first bytes. btx\0 as the u32 0x787462 goes to 0x14026fe10 and on to the btx loader (btx.md). p, a, k goes to the sprite pack reader 0x1402702a0. Anything else is read as an image entry by 0x1402701a0. The image reader creates the texture through 0x14026fb20 → 0x14019e500 and keeps a pointer to each clut.

So the game reads these fields and nothing else:

• ashg: packed_size, unpacked_size, method, window_bits, the stream, and (weapon sprites only) the first three bytes.
• Sprite pack: offsets[0]–offsets[2] and ids[0].
• Image entry: header_size; texture_offset, clut_base, clut_table_offset, clut_count, frame_width, frame_height; the texture's data_size, data_offset, command_offset (and bit 0 of the word there), width, height, bpp, texture_key and pixels; each clut's data_size, data_offset and colours.

Replacing a sprite or texture

What static analysis says a replacement must keep. All of it is likely; none of it has been tried in game.

• The ashg header must be valid, but needn't be byte-identical. unpacked_size must be exact, and method 0, 1 or 2 with a stream that ends inside the entry. crc and unk_0x0F aren't checked, but writing the CRC byte and 0 keeps the file like the originals. Method 0 (stored) is the simplest valid choice. Its stream is just the data, so packed_size = 16 + unpacked_size.
• Stay inside the slot sizes in the table above: unpacked unit variant-1 packs up to 0x1403F8 bytes, weapons 0x50000, faces 0x48000 unpacked and 0x28000 stored. The game doesn't check, so a bigger one would overwrite memory.
• Keep a sprite pack's shape. Keep the first ID 0x1FF10001, or the aura and mask aren't loaded. Keep the sheet, aura and mask in that order, and keep the mask the size of the sheet, as in every file.
• Keep the image's fixed values. Keep header_size 0x10, texture_offset 0x20, data_offset 0x30 and bpp 8 (or 4 or 32). Keep pixels linear, with bit 0 of the first command word clear. data_size must cover width × height.
• Change texture_key when the pixels change. The game reuses a texture it already has under the same key in the same sprite object, so an edited sheet that keeps its old key may show the old picture. The key is shared by every copy of the same sheet. For example, the Dra_Dark variant-1 sheet has one key in 12 object/ files and an event scene. Any value no other sheet uses should do.
• Keep where the quads are. The animations draw rectangles of the sheet by pixel position (sas3.md): units by the 25 shared sets in battle_data_release.dat #0 (unit_animation_sets), event actors by the SAS3 next to their pack. A redrawn sheet must keep each quad's rectangle, or come with an edited SAS3.
• Clut-less sheets take their colours from elsewhere. Units' indices go through the shared cluts in battle_data_release_2 #2/10, and weapons' through weapon_graphics #1/0. 1,956 sheets carry one 256-colour clut of their own, as many as there are scene actor packs (scene ID 0x12000 + n), so probably those.

Rebuilding

• confirmed: the original encoder can be reproduced. It works greedily: at each byte it takes the longest match of at least 3 bytes, and among matches of that length the one farthest back; bytes with no match collect into literal runs of at most 127. It encodes with each window_bits from 7 to 10 and keeps the shortest stream; no two window sizes ever tie. With the header filled in as above (packed_size from the stream, method 2, the CRC byte, 0), this gives back every ashg in the game byte for byte. Source: C# scan, encode2.cs: 4,285 of 4,285 identical.
• likely: to rebuild a file, unpack each ashg, rebuild what's inside, then re-encode it this way. An unchanged entry comes back byte-identical, and a changed one gets a valid stream. Sprite packs follow the pakd rules (containers.md). Image entries keep their fixed values and are padded to 16 bytes.

Leads

• Confirm in game (R7) that a sheet repacked with method 0, or with a new texture_key, loads; and what happens to an edited sheet that keeps its key.
• How the quads of the 25 shared unit animation sets map onto the unit sheets. Their rectangles reach 1,024 × 384 (#0/0) and 1,024 × 512 (#0/7–9), while the sheets are 512 × 384 (variant 0) and 1,024 × 256 (variant 1). The draw divides a quad's pixel position by the sheet texture's width and height (0x1402e3340). Which set and which variant a unit uses has to be traced from the unit and class code.
• What decides variant 0 or 1 of a unit sprite (the unit's +0x53, 0x1402e5180).
• How the draw picks a unit's clut among the 65 in battle_data_release_2 #2/10, and what the aura and mask layers do (0x14026f2f0, 0x14026f360, 0x14026f3d0, 0x14026f440 bind them).
• How scene unit sprite files (ID 0x50000 + n) reach the unit sprite code: 0x140323530 stores the pointer, and the rest is not traced.
• The clut-only images: who reads battle_data_release_2 #2/10 and how the 65 cluts are numbered.
• The bg/ files start with the same 16-byte file header with unk_0x0C 0 (394 entries). That format belongs with bg/.
• "Image entry" comes from Gibbed's ImageFile; the only game support is that the unit graphics code is Unit\UnitImage.cpp. A name of the game's for these entries would replace it.