Repository navigation
Conversation
724df7f to
8a25d23
Compare
be380d9 to
55b2df1
Compare
55b2df1 to
d223dbf
Compare
dustinspace217
left a comment
There was a problem hiding this comment.
Ran this on desktop (no change, no cache files) and forced the transcode path on Mesa by hiding S3TC/BPTC: textures look right, and one SM1-01 load wrote 5,840 cache files (243 MB). A few things:
- On Vulkan this needs #7699: master's
vulkan_preloadhas no ETC2 case, so a transcoded texture goes up as 32-bit BGRA and reads well past its buffer. Worth saying so in the description and keeping it draft until #7699 lands. - The cache key only checks the first 64 KB. A DDS file's length is set by its size, format and mips, so a repainted texture keeps the same length and reuses the old transcode (across mods too, since the folder is shared).
cf_chksum_longover the whole file, plus the etcpak revision in the tag, would cover it. - With
-portable_modethere's no user root, so the startup prune'scf_get_file_listpasses an empty path intocf_get_list_of_files, which doespath.back()on it. That runs on every portable start, even on desktop. AddingCF_LOCATION_ROOT_GAMElike the shader cache does fixes it. - The cap is only enforced at startup, so it gets passed after about a dozen mission loads. Pruning after writes, or touching the mtime on a hit, would help.
- Smaller:
- bound
stored_sizeto the payload size in the reader; - the mip halving also halves the reported size of 2D art, so maybe tie it to the existing texture-detail setting;
- reuse the existing Vulkan format probe;
- drop the unused
ProcessDxtcfiles; - add
CAPABILITY_ETC2togr_capabilities[].
- bound
|
-ProcessDxtc is keep because it will be used in ta future PR to add the inverse path of ETC2->BCn in ktxutils. There is no ETC2 support in desktop gpus, so someone making a mobile-first mod including ktx/etc2 textures will need something similar to this in ktxutils for vulkan. This is why the cache system is intended to be used for both. ill work on that one after this and #7699 are merged. -Pruning while writting may cause additional problems, on top of deleting cache files that might be used in the current load. What i would want and It might be a good idea is to run it at the end of mission load, as the last step, but i dont know how to do it. -"add CAPABILITY_ETC2 to gr_capabilities[]." where? i belive i already added to all relevant places. -You are 100% sure the change in in the 2nd point is needed to avoid a cache colision? its easy to change, i priorised speed, i was worried that doing checksum over the entire file would slow down mission load even more. -On the first point, yes, this is intended to go after #7699, i was not worried too much as the only real use is android devices and i expected both PRs to be in in the same stable release. And 7699 to be merged earlier. |
|
Thanks.
|
Allright, of the requested changes i did: -Added CF_LOCATION_ROOT_GAME to cache location flags. What i did not do: Please check. Also remember i need to set the localization ids before merging on this one. |
dustinspace217
left a comment
There was a problem hiding this comment.
I rebuilt at 6cc2c1f on Linux and played SM1-01 with MediaVPs 5.0.2 against master on OpenGL and Vulkan: no visible change and no cache files on desktop. Then I forced the transcode path on Mesa by hiding S3TC and BPTC. One load wrote 5,843 cache files (240 MB), a second launch hit the cache for every texture, -no_transcode_cache wrote nothing, and with the cache set to 0.1 GB the prune at the end of loading cut it to 80 MB. The portable-mode fix, cache key, size check, capability row and Vulkan probe all look good.
One thing still needs a change: the extra mip drop. With transcoding on, every DDS with mipmaps loads at half size, so 2D art drawn at its own size shrinks. The MediaVPs squad logos drew at 64x64 instead of 128x128, while master's fallback on the same setup keeps them at 128. The software path needs no change; it only shrinks textures above 1024. Taking the extra level only above 1024 in conversion_resize keeps 2D art intact and still halves the big ship maps.
Smaller: a cache hit never updates the file time, so once the cap is reached the prune can delete textures the mission just used.
|
i could change it to only drop a level if the texture is bigger than 512... or i might just keep it at max of 4096. i need to test the potential conversion time implications of both options. BTW 2D art is compressed? i trought it was uncompressed because you could not use compressed images in the 2D renderer. Example: WCS saga eff mainhall animations are uncompressed for this reason. Medals in mvp im petty sure are uncompressed. |
Allright. -Now cache hits modifies the file written time |
dustinspace217
left a comment
There was a problem hiding this comment.
Rebuilt at 2707a9a and reran the forced-transcode test on Mesa (S3TC and BPTC hidden, MediaVPs 5.0.2, SM1-01). The squad logos now load at 128x128 and the radar icons at full size, while a 512x512 effect still drops to 256. For a warm load I filled the cache with every file dated two days back: the mission hit the cache for all of them, wrote nothing new, and every file's time was current afterward. The new DDS and cache tests pass locally along with the rest of the suite, and CI is green.
To your question: yes, MediaVPs ships its squad logos and a couple of radar icons as DXT5 DDS with mipmaps, which is why they were shrinking. I also scanned the interface, HUD, squad, medal and main hall folders in MediaVPs 5.0.2, SG 2025.1.0, BtA and Blue Planet, and none of the mipped DDS there is larger than 256 on both sides, so the 256 cutoff keeps all of that art at full size.
Not tested: a real GLES/Android device, Vulkan on the transcode path (NVIDIA's Vulkan driver has ETC2 off here), and portable mode.
dustinspace217
left a comment
There was a problem hiding this comment.
All points now look green to me.
dustinspace217
left a comment
There was a problem hiding this comment.
Checked the two master merges and the renumbered string IDs (1942-1944 don't collide with anything on master). All points now look green to me.
dustinspace217
left a comment
There was a problem hiding this comment.
Built at b43218e merged onto master, I played SM1-01 with MediaVPs 5.0.2 on OpenGL and Vulkan against master: both reached gameplay with no visible difference from master, but this run couldn't force the transcode path or show a cockpit display. Reading the code turned up one regression on the transcode path that my earlier approvals missed, so I'm asking for a change.
EFF streams: bm_lock_dds ignores the BMP_TEX_NONCOMP the stream asks for at generic.cpp:378 (bmpman.cpp:2302), so on a GPU with ETC2 but no BC support, DDS frames that master decoded to BGRA now arrive as ETC2 blocks and go to gr_update_texture as 32-bit pixels. Vulkan skips that update (VulkanTexture.cpp:1770), so the animation freezes, and on OpenGL/GLES the 8-bit HUD-color path, which head animations use unless full-color heads are on, reads width x height x 4 bytes from the ETC2-sized frame (gropengltexture.cpp:1636-1658), past the end of the buffer. Desktop with native S3TC already streams these blank (#7887), so one fix covers both: have bm_lock_dds decode to BGRA when the caller's flags exclude compressed data, using the existing dds_decompress_top_mip_bgra.
Smaller:
- tex_cache.cpp:113-116: a file that fails the header check is closed but not deleted, and on Windows the cf_rename at :221 can't replace an existing file, so that texture re-transcodes on every load. Deleting it there and checking the cfclose and rename results fixes it.
- tex_cache.cpp:265 and :279: max_bytes and total are size_t, so on a 32-bit build a cap above about 4.3 GB is an out-of-range float conversion and a large cache wraps the total; uint64_t avoids both.
- tex_cache.cpp:249-308: the prune deletes files without logging anything, and a cap of 0 empties the cache on every load; one mprintf with files and bytes removed would make that visible.
- ddsutils.cpp:542: the cache key (file lookup, 64 KB checksum, MD5) is built whenever -no_transcode_cache is absent, even with the cache option at 0 where load and store return at once; one shared enabled check would skip that.
- Building on the 11-07 squad-logo point (gr_bitmap, 2d.cpp:2542): mipped DDS art between 257 and 1024 px still draws at half size through gr_bitmap and the Lua size calls, since only cockpit displays read bm_get_original_size. Having the 2D helpers take their extent from it would cover third-party mods.
- test_ddsutils.cpp:105: the size check covers only DXT1 at 128 with no resize and zero data; a DXT5 case and a resized case with non-zero data would pin the 16-byte block size and the mip skip.
- cfilesystem.cpp:1948: the third push site never sets the new file_list_info.size; tinfo.size = f->size there keeps a later caller from reading garbage.
- bmpman.cpp:3082: bm_reload keeps the previous file's original size; nothing reads it on that path yet, but refreshing it there keeps the field right.
- cfile.h:292: the new cf_rename parameter is int, while cf_delete takes uint32_t location flags.
- #7998 also uses XSTR 1942-1944, so whichever lands second needs to renumber.
- Description item 6 still says textures are always downsized by 50%, but the code only drops a level when both sides exceed 256.
|
Allright got all the changes, i tested as best as i can, but some parts i just cant, this is really getting out of hand now, it all started with a simple, "reconvert using the same strategy as software decoding and reuse the ETC2 upload flow"... |
There was a problem hiding this comment.
Re-reviewed at 5e613fc. Everything from this morning's review is addressed, and this time I could test the streamed-animation fix.
Tested (Release build of this head against master 6b4bb81, retail data, RTX 5080, Linux):
- A test mod that replaces the main hall crane animation with a 20-frame EFF whose frames are DXT5. On master, OpenGL draws garbage in its place and Vulkan stays on the first frame. With this PR both renderers play the animation.
- SM1-01 with MediaVPs 5.0.3, OpenGL and Vulkan, this PR only: reaches gameplay and the picture looks right. A Release build doesn't log warnings, so I can't speak to those.
From reading: the cache fixes (bad-header file deleted, close and rename checked, 64-bit sizes, prune logged, shared enabled check), the original size kept on reload, the third file-list site, the new DXT5 and resized tests, and the merged cf_rename are all in.
One small thing, not blocking, on the transcode path only: the radar icon scale and centering now come from the original size (radar.cpp:214), but the draw goes through gr_aabitmap, which still sizes from the loaded bitmap. A radar icon that lost a mip level would draw at half size and off center. On desktop the two sizes match, so nothing changes there.
Not tested: the transcode path itself (no ETC2-only hardware here), Windows, and Android.
Edited: the SM1-01 line first said the log had no errors or warnings.
dustinspace217
left a comment
There was a problem hiding this comment.
Re-checked the two new commits at 9290bdd. The radar icon point from my last review is fixed: radar.cpp:277 now draws with the original size, so the size and the centering match. The other gr_aabitmap() callers keep the loaded size, since the new argument defaults to false.

Mobile GPUs might not have S3TC or BPTC, while it has become more common on Adreno 800s and Mali V5 Valhall you can never be sure, and support on older gen gpus is rare.
The objective of this PR is to do transcoding when the bitmap data is requested, if ETC2 support is present. Impact on loading times, specially for missions with a lot of ships and weapons are noticeable, but usable. Additionally a transcoding cache is added so you will only transcode a texture once, unless it is evicted from cache due to size limitation.
This process is about the same used by the bcn_layer that pc games emulators for android uses. And has been optimised for transcode speed and smallest vram usage over quality.
Current Features: