You don't seem to be able to give a KeyPairGenerator custom prime numbers, only a key size.

Furthermore, I don't know how to limit the pool size. I am not experienced with how Java threads work.

Why do you need custom prime numbers? I don't see any use case for it.

Limiting pool size is pretty easy - just use fixedThreadPool or singleThreadExecutor instead of cachedThreadPool. To assign priority of thread you may just create your own ThreadFactory that will create low-priority threads.

Prime numbers are mainly there so you can generate a non-random key set in case you need that for some reason.

Thanks for the thread pool suggestions. I will look into making a custom ThreadFactory.

But these keys are completely useless - they cannot be used for their purpose because Lua numbers are so small that resulting modulus can be factorized even on PC

You can still encrypt and decrypt small numbers. It's mainly a gimmick...

An issue I see is that when multiple keys are supposed to be generated at the same time, it will take a very long time for the queue to empty, meaning that the last key to get generated could take 10-13 seconds to generate, maybe even longer. With the current system, they all take roughly the same amount of time.

The main issue is that non-priveleged users can take down entire server by producing very high CPU load. By limiting pool to 1 thread you will limit load to 1 core, by limiting key size to 2048 you will make key generation process last 1 or 2 seconds for every key.

I will limit it to 2048 for sure. I'm just not sure about the thread pool limit... Maybe I can limit it to 2 threads, then it could be much better, and with 2048 as the max key size it'd really be rarely noticable (you'd have to generate a LOT of keys at once to notice).

Maybe. Better create config option so every server owner will be able to decide, and default this option to 2 threads. You may also put generating machine to sleep at N seconds (another config parameter) thus limiting key rate.

I cannot make the computer or the cipher block pause unfortunately due to me not knowing in advance how long it will take. The config option for threads is a good idea, I think.

You not need to sleep exactly while key is generating. Just put to sleep for 0.5 seconds - even this timeout will reduce possibility to spam worker thread.

This is probably a good idea. I will look into it.

Also, I just noticed, that you are generating keys 2x large than argument. Bit length of key is bit length of modulus, not of primes.

You are right, that is a mistake on my part, I guess switching to KeyPairGenerator would fix that already.

Hmmm... Do you know how to extract the modulus and the exponents from a generated key pair? It appears I'm unable to figure that part out.

Just cast keys to RSAPublicKey and RSAPrivateKey, and use appropriate getters. If you cast private key to RSAPrivateCrtKey you may also obtain additional coefficients that will speed up decryption, but you will have to write code that implements Chinese Remainder Theorem.

Changes look great, now malicious user cannot abuse key generation.

I may just leave it with the restrictions for now. I have tested with casting and so far it wasn't working well. Until I find out more, this will have to do. The restrictions are in place, so abusal has much less of an impact now. Thank you a lot for all your help.

https://gist.github.com/makkarpov/2ea196848547f47a84c5

Hmm... Casting works well for me, please try the gist code. Also, keys have their own serialization and deserialization methods that produces/accepts data in standard formats, so you can use these. But it will break a lot of existing things, of course.

Keys have already been broken because I changed the encoding for the upcoming update, so it's fine.

Updated gist to show serialization and deserialization example with final length comparasion.

Okay, serializing and deserializing works just fine now, thanks for the example. Encryption and decryption (the old way) doesn't work though. Do you have any suggestion for that?

Yeah. Added encryption example to gist

Okay, encryption and decryption works, but the max length for a string to encrypt would now apparently be 245 bytes (at a key length of 2048). Is that enough? It sounds quite short to me. Anything larger than that is throwing an IllegalBlockSizeException.

This is not limitation of implementation, plain RSA also have this "feature". RSA is rarely used directly due to it's low speed. Instead, some random AES key is generated, encrypted with RSA and data encrypted with this key. So AES keys and hashes (for signatures) fits in this limit excellently.

Ah, okay. So it will be a combination of your new Data Card and this block that will make excellent encryption. I see.

Worst case you could still just split a string into 200-byte pieces and encrypt each part separately, I guess.

Yes, chunk split can be done in lua. Padding overhead is 11 bytes, so to determine maximum chunk length we need to subtract it from key length.

Also, you can split data to chunk at java side and instead of plain byte string return an array of encrypted chunks.

Ah, that explains the 245 bytes.

You can remove this overhead by replacing PKCS1Padding with NoPadding, but in this mode cipher will return decrypted strings in format like 0000000......000000<payload> (where zeros are binary, not string ones), so you will need to remove these zeroes manually. In PKCS1Padding mode cipher is binary-safe, so you can pass any data to it (even zero bytes) and this data will not get truncated.

Hmm, I guess the padding is fine then. Thank you a lot for your help!

The KeyPairGenerator, KeyFactory and Cipher instances I could keep in static fields, right? I only need to get them once.

You should keep them in ThreadLocal because I do not know about thread-safety of JCA and actual providers. But you do not need to create them every time, you are right.

I still need to do in-game testing, but I think it's pretty much done now. It would be nice if you checked my latest commit for any mistakes I might have made.

Looks good. I do not understand why you do not remove generation routine for custom p and q primes (or maybe adapt it to BigInteger so it can support really large primes), but it is your choice. Also I think that if key format is broken anyway, it's time to wrap these keys into userdata because these Maps<> looks ugly. Maybe something like createKey(n, e): userdata and userdata with functions process(msg: string): string, getModulus and getExponent

They are mainly Lua tables so people can easily turn them into a single string for networking (for instance using table.concat(pubKey, "\n"). I know I can make userdata index-able like a table, too, but I'm not really fond of that idea.

I won't have process on the key itself because I want the encryption/decryption to require a connected Advanced Cipher Block.

p and q are still there so there's a way of generating small non-random keys; I agree it's utterly useless, it is mainly a gimmick.

Ok then, everything else looks good.

Thanks for all your help. Once I tested this a bunch in game I'll probably release a new Computronics version. In case you would like to test as well, here is a new test build.

Just got to testing this and it appears to work very well now. Thanks again for your help!