I trying to load public ssh rsa key (id_rsa.pub) from a file:

X509EncodedKeySpec spec = new X509EncodedKeySpec(readBytes(keyFile));
return (RSAPublicKey) KeyFactory.getInstance("RSA").generatePublic(spec);

but I get

java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: invalid key format

How to load this file to RSAPublicKey?

The x509 encoded data should be in DER format. You can produce such keys using OpenSSL: (Note, credit to http://codeartisan.blogspot.com/2009/05/public-key-cryptography-in-java.html for the OpenSSL refresher.)

To generate a key, use:

$ openssl genrsa -out private_key.pem 2048

Or, with a passphrase:

$ openssl genrsa -aes256 -out private_key.pem 2048

Or, using ssh-keygen (following your example):

$ ssh-keygen -t rsa -C "myEmail" -I X.509

I'll assume you saved the key as 'private_key.pem' Generate a public key in DER format:

$ openssl rsa -in private_key.pem -pubout -outform DER -out tst_public.der

A byte array containing the contents of the file will now be accepted by X509EncodedKeySpec.

If you want to load the private key, use OpenSSL to save an unencrypted copy of your private key (don't do this in an insecure environment):

$ openssl pkcs8 -topk8 -inform PEM -outform DER -in private_key.pem -out private_key.der -nocrypt

You can then pass this file as a byte array to `PKCS8EncodedKeySpec`.

You can also generate your key pairs in Java:

private static int rsabits = 2048; // or higher, if you've got the cpu*time
public static SecureRandom sr = new SecureRandom(); // reseed periodically 

public static KeyPair newKeyPair() throws NoSuchAlgorithmException {
    KeyPairGenerator generator = KeyPairGenerator.getInstance("RSA");
    generator.initialize(rsabits, sr);
    return generator.generateKeyPair();
}

Then you can use KeyPair.getPublic() and KeyPair.getPrivate() to access your keys. You can save or load them as byte arrays with something like:

public static byte[] pubKeyToBytes(PublicKey key){
    return key.getEncoded(); // X509 for a public key
}
public static byte[] privKeyToBytes(PrivateKey key){
    return key.getEncoded(); // PKCS8 for a private key
}
public static PublicKey bytesToPubKey(byte[] bytes) throws InvalidKeySpecException, NoSuchAlgorithmException{
    return KeyFactory.getInstance("RSA").generatePublic(new X509EncodedKeySpec(bytes));
}
public static PrivateKey bytesToPrivKey(byte[] bytes) throws InvalidKeySpecException, NoSuchAlgorithmException{
    return KeyFactory.getInstance("RSA").generatePrivate(new PKCS8EncodedKeySpec(bytes));
}

more