Hybrid Unimodular Hill Cipher (UHC) and Rivest-Shamir-Adleman (RSA) Encryption with Chaotic Key Generation for Secure Binary Files

Abstract

This study introduces an intelligent adaptive hybrid cryptographic framework that integrates the Unimodular Hill Cipher (UHC), RSA-OAEP, and Logistic Chaotic Map-based key generation to provide comprehensive security for heterogeneous binary files. Unlike conventional cryptographic approaches that generally apply a predetermined encryption mechanism regardless of the characteristics of the input data, the proposed framework incorporates an AI-based adaptive decision mechanism to identify an appropriate encryption configuration based on the statistical properties of the target file. The decision mechanism evaluates multiple file-level characteristics, including Shannon entropy, byte-frequency distribution, file size, standard deviation, and the number of unique byte values. Based on these characteristics, the framework assigns most of the data to the computationally lightweight UHC mechanism, whereas selected sensitive segments are protected using RSA-OAEP to introduce an additional layer of asymmetric cryptographic security. The generation of dynamic unimodular matrices is supported by Logistic Chaotic Maps, whose sensitivity to initial conditions contributes to enhanced key unpredictability and randomness. The proposed framework was experimentally assessed using a collection of binary PDF files with different matrix dimensions and password configurations. The experimental evaluation produced a ciphertext entropy of 7.999891 and a plaintext–ciphertext correlation coefficient of −0.000152, indicating an almost negligible statistical relationship between the original and encrypted data. In addition, the Chi-Square statistic decreased substantially from 8159.84 for the plaintext to 269.85 after encryption, demonstrating improved statistical uniformity and diffusion. Data integrity and lossless recovery were further confirmed by obtaining identical SHA-256 hash values for the original and decrypted files. These results indicate that the proposed adaptive hybrid cryptographic framework can simultaneously provide strong statistical security, computational efficiency, adaptive encryption behavior, and compatibility with heterogeneous binary data, making it a promising approach for secure digital file protection across diverse computing and transmission environments.