Create more compressed QR codes #167
Conversation
Rationale:
Some cons (such as Glasgow2024) use UUIDs which are repeatative and
large and somewhat redundant. As a result their QR codes can be oversized,
so it is worth trying to match the entropy to QR codes better.
By compressing the ids and base32 formatting them we can put all
the entropy in 5bit tokens which are representin in 5.5 bits in the
QR code.
Todo:
- Refactor ItembyIdList and GzItembyIdList to share common code.
- Change qr library to one which uses the 5.5 bit ALPHANUM encoding
scheme
Return ids for non compressed links to lower case
|
I love the idea of this. Making the URL more compact would be a great benefit, particularly for conventions using Planorama, which uses GUIDs for the IDs. The all upper-case URL isn't ideal. I would prefer not to do that if we could avoid it. It will particularly affect sites placing ConClar in a subdirectory, as that directory would have to be upper case. Have you looked at react-native-rqcode-svg? It seems to allow different encodings for different parts. Example from it's documentation: |
|
Another thing to think about. Would it be possible to detect they types of the IDs and encode them more efficiently. GUIDs are pretty inefficient as text strings. Would converting them to 128-bit binary values save space? Or perhaps take a leaf out of git's book, and shorten the GUIDs to enough characters to uniquely identify them, and expand them when the URL is decoded? |
lostcarpark
left a comment
lostcarpark
left a comment
There was a problem hiding this comment.
Could you have a look at these suggestions, please?
This will also need rebasing for the latest changes.
| // const deflator= new Deflate(); | ||
| // deflator.push(linkItems,true) | ||
| const param=base32.encode(deflate(linkItems)).replaceAll("=","-"); | ||
| return configData.BASE_PATH.toUpperCase() + "GZIDS/" + param; |
There was a problem hiding this comment.
Current version has removed BASE_PATH from config and uses window.publicUrl instead. Can you update to use this?
|
|
||
| function makeLink(linkItems, compress) { | ||
| if (!compress) { | ||
| return "ids/" + linkItems; |
There was a problem hiding this comment.
This should also include the base path, using window.publicUrl.
| "DESCRIPTION": "Copy and paste this link to share your selections with other devices. Or point your camera at the QR code below.", | ||
| "LINK_LABEL": "Shareable link", | ||
| "MAX_LENGTH": 2500, | ||
| "COMPRESS": false, |
There was a problem hiding this comment.
Can you add the new setting to the README.md, please?
|
I think this is definitely an improvement, though I share @lostcarpark concerns about forcing the path to be uppercase. An issue with just gzip compressing the ids is that the main culprit of long ids is UUIDs, typically UUIDv4, which are inherently poorly compressible. Rather than maxing out at around 67 items in a single QR code, we can pack around 77. An improvement, but not a lot. We could use something like a truncated hash-set. This is a probabilistic data structure, so there is a small chance of false positives where an item shows up in the link that isn't meant to. But we can make that small enough not to matter. Because we have all the ids when we're decoding, we don't need to communicate the whole id, we only need to communicate enough of it to pick the correct id when decoding. Generate a fingerprint of the id, something like selecting the top 32 bits of the SHA-256 hash. SHA-256 are uniformly distributed, so it allows us to truncate it to generate the fingerprint. (If we just took a prefix, then ids like "mylovelycon-11", "mylovelycon-12" etc. would be creating collisions if we truncate too early). There is a small chance (~0.01%) two truncated hashes would generate identical fingerprints. To work around that, mix in a salt. It can just be an integer. If there is a collision after generating the fingerprints for every item, increment the salt. Repeat until there are no collisions. This salting means we never create a false positive for the case where the schedule doesn't change between encoding and decoding. If items are added to the schedule in-between (think: someone posted a QR code of the items they're on to social media), then if ~500 items are added, there would be a 0.01% chance a random item would also show up as being in the link. That probability can be tweaked by changing how many bits we take from the hash. This allows us to pack 390 items into a single QR code. But we can pack even more in by doing gap encoding. Sort the fingerprints, and then take the gaps between them and encode that. Because the gaps are small, we can pack them fairly efficiently. A common encoding here is varint. That would let us pack around 408 items into a single QR code. A more efficient scheme is Golomb-Rice encoding, which takes us to 506 items in a single QR code. However, I can't find any good libraries to do Golomb-Rice encoding and, while it's only a few hundred lines to implement, it's probably more than we want to bother maintaining. The catch is that these schemes tend to perform worse for integer ids. We could inspect the structure of the IDs and swap approach. Though I tend to think
[1] https://lunduke.locals.com/post/5488507/myth-bill-gates-said-640k-ought-to-be-enough-for-anybody |
|
The inherent uncompressibility of GUIDs is difficult to get around, and I wanted to stay with the neutral stance against backend ids. It's not actually required to stay with an uppercase prefix; although most of it is case-insensitive (scheme and hostname), forcing it to that is an easy way to ensure the QR library creates a good bitstream of the string. QR codes can change 'alphabet' in the string so the prefix could use the full alphabet, and then we drop back to the 5.5-bit alphabet afterwards. The problem is that most QR code libraries I've seen don't expose this level of control. I'll have a look at those changes and think about the hash proposal... it might work quite well with dynamic length, rather than using a salt. But I ll have to refresh my memory about the innards here. |
|
I did an experiment with a previous Worldcon programme, trimming the program IDs and people IDs to the minimum number of characters to uniquely identify the item. The vast majority could be trimmed to the first 3 or 4 characters, with a handful needing 5. Of course there's no guarantee that any two IDs won't need a longer string to uniquely identify them. I think an encode routine that took every ID, and finds the shortest string that uniquely identifies it. The decode just needs to find the first entry starting with that string. The danger is that new program items will get added that could clash with the stored ones. It's impossible to guarantee that this won't happen, but the majority of program changes are time changes or cancellations. It's relatively rare for whole new items to be added during the convention. The risk of a clash could be mitigated (but not completely eliminated) by adding an extra character over the minimum to identify the item. |
3 participants
This set of patches tries to produce a more optimally reduced URL/ QR Code representation for the MySchedule links.
Principle of Operation:
We first use deflate to reduce bitstream to a minimum. Unfortunately that gets us a binary bitstream which can't be encoded into a URL, so we need to use an expanding encoding to fit the data into the URL-safe characters.
We chose to use a Base32 scheme for this which will fit it into the QR codes 'Alphanumeric' encoding alphabet, as a result of this, 5 bits in the deflate stream should take 5.5 bits on the QR code bitstream.
The downside of this is that I couldn't find a react qr component which supports using different encoding schemes for different segments of the QR code data (allowed by the QR standard). Additionally the Qr component currently used by conclar only supports the full eight-bit encoding scheme, so I have swapped the whole component for react-qr-pretty
at the cost of a different size of presentation of the QR code on the page.
As a result, the whole URL has to be rendered as upper case, it wouldn't be impossible to add the multi-segment feature to a react QR component but I felt it was out of scope for this PR.
The URL generated also uses a different subpath from the non-compressed one so that even if compression on codes is switched off both forms of URLs can be accepted, allowing conventions using thissoftware to switch modes without invalidating 'released' QR codes.
Coding Choices.
I've tried to keep the changes to conclar as small as possible to implement this feature. Instead of making intrusive changes to ItemByIdList.js, I've copied it and added the changes to GzItemByList.js It should be possible to combine them if that was considered a good idea.