Differences in Performance and Accuracy between Concrete Numpy and FHE in Concrete Library

Hello everyone,

I’ve been experimenting with the Concrete library for a project that involves secure computation. Initially, I opted for Concrete Numpy (import concrete.numpy as cnp) due to its compatibility with my requirements, especially for executing basic logical operations (AND, XOR, etc.) on binary arrays. However, when I switched to using the core FHE functionalities of the Concrete library (from concrete import fhe), simply substituting cnp operations with fhe, I noticed significant differences in both execution time and accuracy of the results.

Execution Time: With Concrete Numpy, the computation time for each array ranged between 30 to 40 seconds. Surprisingly, when performing the same logic using the core FHE functionalities, the time reduced dramatically to approximately 1.5 seconds per array.
Accuracy of Results: While the results obtained from Concrete Numpy were accurate, I encountered inaccuracies in some of the outcomes derived using the FHE operations.
Could someone help clarify if there’s a misunderstanding on my part regarding the use of these two components of the Concrete library? Is there a specific reason behind these discrepancies in performance and accuracy between Concrete Numpy and the core FHE operations?

Thank you for your insights!

Hi @mahzad_mahdavisharif,

Could you share your function so we can investigate more?

Thanks!

Thanks, @umutsahin, for your prompt response. Here is the code snippet where I’ve been using Concrete Numpy:

import concrete.numpy as cnp
#from concrete import fhe
import numpy as np
import csv
import time

def _PSI_impl(g_key, given_key):
intersect = np.bitwise_xor(g_key, given_key)
print(“Encrypted Intersection is :”, intersect)
if_match = (np.sum(np.bitwise_xor(g_key, given_key))==0)
is_sub_match = (np.sum(np.bitwise_and(g_key, given_key)))
if_sub_match = np.logical_xor(is_sub_match, 0)
if_no_match = (np.sum(np.bitwise_xor(g_key, given_key))==32)
match_status = if_no_match * 2 + (1 + if_sub_match) - if_match

return match_status

class HomomorphicPSIS:
_PSI_circuit: cnp.Circuit

      def __init__(self):
        configuration = cnp.Configuration(
            enable_unsafe_features=True,
            use_insecure_key_cache=True,
            insecure_key_cache_location=".keys",
        )

        PSI_compiler = cnp.Compiler(
            _PSI_impl,
            {"g_key": "encrypted", "given_key": "encrypted"}
        )

        inputset_PSI = [
            (
                np.ones(32, dtype=np.int32),  
                np.ones(32, dtype=np.int32),  
            )
        ]

        print("Compiling PSI circuit...")
        start = time.time()
        self._PSI_circuit = PSI_compiler.compile(inputset_PSI, configuration)  
        end = time.time()
        print(f"(took {end - start:.3f} seconds)")

      def load_data_from_csv(self, csv_file_path):
        mask_array = []
        with open(csv_file_path, 'r') as csv_file:
            csv_reader = csv.reader(csv_file)
            next(csv_reader)  
            for row in csv_reader:
                mask = int(row[2])  
                mask_array.append(mask)
        return np.array(mask_array, dtype=np.int32)
    
      def PSI(self, g_key, given_key):

          print("\nProcessing keys:", g_key, given_key)
          start = time.time()
          encrypted_keys = self._PSI_circuit.encrypt(g_key, given_key)
          result_keys = self._PSI_circuit.run(encrypted_keys)
          end = time.time()
          print(f"Encryption (took {end - start:.3f} seconds)")
          start = time.time()
          decrypted_result = self._PSI_circuit.decrypt(result_keys)
          end = time.time()
          print(f"Decryption (took {end - start:.3f} seconds)")
          print("Decrypted results for intersecting keys:", decrypted_result)
          return decrypted_result

if name == “main”:
server = HomomorphicPSIS()
decrypted_results_array =

for i in range(5):
        start = time.time()
        g_key = server.load_data_from_csv(f'g_keys_dataset_with_third_column_{i}.csv')
        end = time.time()
        print(f"Load g_key from documet{i} (took {end - start:.3f} seconds)") 

        start = time.time()
        given_key = server.load_data_from_csv("snomed_code_mask_with_conditions.csv")
        end = time.time()
        print(f"Load given key from document (took {end - start:.3f} seconds)") 

        print("    g_key is:", g_key)
        print("given_key is:", given_key)



        start = time.time()
        print("encrypt and serialize data")
        decrypted_result = server.PSI(g_key, given_key) 
        decrypted_results_array.append(decrypted_result) 
        end = time.time()
        print(f"Pair encryption and serialization (took {end - start:.3f} seconds)")

        
# Iterate over intersection_array with document index
documents_with_full_match = []
documents_with_sub_match = []
documents_without_match = []
for i, intersection_result in enumerate(decrypted_results_array):
    if intersection_result == 1:
        documents_with_full_match.append(i)
    elif intersection_result == 2:
        documents_with_sub_match.append(i)
    elif intersection_result == 3:
        documents_without_match.append(i)

# Print the lists to see the categorized documents
print("Documents with full match:", documents_with_full_match)
print("Documents with sub match:", documents_with_sub_match)
print("Documents without match:", documents_without_match)

This function is designed to compute the encrypted intersection of two sets represented by g_key and given_key , and determine the match status based on logical operations. Additionally, this script has been executed as described previously, utilizing both Concrete Numpy(cnp) and Concrete FHE(fhe), with noted differences in performance and accuracy outcomes.

Thank you in advance for your help and insights on this matter.

1 Like

Okay, so here is the issue:

  • Your inputset only contains ones.
  • Which makes np.sum(np.bitwise_xor(g_key, given_key)) always 0.
  • So it’s assigned uint1.
  • But in the runtime, if you give ones for g_key and zeros for given_key, it results in a value which wouldn’t fit to uint1.
  • So it overflows.

What you can do is to improve your inputset, and use hinting!

The idea is:

sum_xor = np.sum(g_key ^ given_key)
fhe.hint(sum_xor, can_store=g_key.size)
return sum_xor == 0

or

return fhe.hint(np.sum(g_key ^ given_key), can_store=g_key.size) == 0

You need to hint all intermediate values appropriately, or create an inputset which would capture the extreme values of all intermediate values.

I’d recommend hinting as it’s easy to do for your use case.

Hope this helps!

1 Like

Thank you, @umutsahin, for your suggested solution, which resolved the issue I was facing with incorrect results for some inputs. However, I have two follow-up questions for a better understanding:

  1. Why didn’t the overflow issue occur when the same code was executed with Concrete Numpy and I always got accurate results?

  2. Also, why is Concrete (fhe modules) noticeably faster than Concrete Numpy? My limited understanding, shaped by discussions within the community, hints that it might be due to how Numpy uses the fhe compiler.
    Your answer to these questions will help me understand which of them is more compatible with me. Initially, I thought Concrete Numpy was better based on my requirements, but now I’m a bit confused about the abilities and the reasons we should select one over the other.

Thank you for your support and help in advance.

  1. Why didn’t the overflow issue occur when the same code was executed with Concrete Numpy and I always got accurate results?

I need to see the graphs produced by cnp and fhe to give an accurate answer, but it could be due to the fact that bitwise operations in cnp not being multi precision, or cnp itself being single precision.

Also, why is Concrete (fhe modules) noticeably faster than Concrete Numpy?
Lots of improvements in the compiler and code generation from the python side in the past year :slightly_smiling_face:

Concrete Numpy is deprecated and replaced with Concrete. All new features and improvements happen in Concrete so you should use it instead of Concrete Numpy.

Glad to help :wink: