JChem Base/Cartridge Performance Information

 

Scalability

How scalable is JChem Cartridge / JChem Base?

Lutz Weber's presentation about very large databases (200 million structures) and 2D- and 3D-Similarity Searching.

What is the maximum number of structures for JChem Base / JChem Cartridge?

In a single table the maximum number of rows is 231-1 (2,147,483,647).
We may increase this in the future if needed.
There is no practical limit for the number of tables.


Hardware requirements

How can I estimate hardware requirements for a JChem database?

  1. Memory:
    JChem may use significant amount of memory for building a structure cache in memory.
    Please see this topic on how to estimate the size of the structure cache.

    The total Java memory consumption (heap size) consists of:

    • The size of the structure cache
    • Processing memory for JChem Base: we recommend to multiply the estimated cache size by a factor of 0.75 (up to 8 processor cores). In case of more than 8 cores you may have higher memory need since it scales up with the number of the threads started.
    • Processing memory for your Java application (built on JChem Base)

    Cache and Java memory usage measured for different table size on a 8 core 64-bit Linux (version 3.2.0-48-generic) environment:

    Target count Cache size measured (MB) Cache size estimated (MB) Java memory allocation (MB)
    986,581 148.85 152.45 250
    5,032,252 701 720 1,250
    10,002,807 1,610 1,637 3,000
    19,987,542 3,174 3,206 6,500
    38,221,498 6,002 6,144 12,000
    The Java memory usage could be further improved (compared to the table values) with fine-tuning the JVM parameters.

    To determine the total amount of RAM requirement you should add:

    • Java memory consumption (detailed above)
    • Other programs running on that computer
    • Memory requirement for the operating system

    64 bit systems: A single Java process cannot allocate more than around 2 GB on 32 bit systems. If your Java memory needs will exceed this limit, a 64 bit system is recommended (including hardware, operating system and Java).

    Please also visit this link on how to enable the Java Virtual Machine to allocate the necessary amount of memory

  2. Disk space:

    Apart from the number of structures it also depends on the format of the input file, and type of RDBMS engine used by JChem Base.

    A benchmark result with 1 million structures (the NCI dataset was multiplied for the test), the RDBMS was Oracle:

    Format Size of JChem table Original file size
    SDF 1.2 GB 3.7 GB
    SMILES 270 MB 50 MB

    Note: JChem compresses SDF in the cd_structure field by default (in the case of the NCI dataset to roughly about 4.4 times smaller). This can be disabled (e.g. if they need to be displayed directly by non-ChemAxon tools), but the storage size increases in this case.

  3. Processing power:

    Comparing the performance of different hardware architectures is a complex topic and not the subject of this FAQ.

    Some quick facts:

    • JChem automatically uses all processors during a search
    • The search speed scales well with the number of processors
    • The search time is directly proportional to the size of the table (assuming there is no constraint on the number of hits - see below)

    The following benchmarks can be used as a starting point:

    For the same database the search time greatly depends on the type of query structure. For a very general query (e.g. benzene) there will be a lot of hits, meaning longer execution time, while more specific queries run very fast on the same large database.

    The search time consists of
    • Cash load time : the cash is built up during the very first search
    • Screening time : this is a quick pre-filtering using fingerprints
    • Graph search: slower, but only performed for the screened compounds

    The following can be stated:

    • The screening time is directly proportional to the size of the database table.
    • The graph search time is directly proportional to the number of screened compounds.
    • The number of hits is roughly proportional to the number of of screened compounds.
    • The graph search time is roughly proportional to the number of hits.

    Tip: it is rarely useful to return a huge number of hits (especially for human consumption).
    If the number of hits is limited, only the rapid screening time will increase with table size, which means the total search time will remain almost constant regardless of the table size.

How to estimate the memory need for the Structure Cache?

Typically 1 million drug-like structures consume around 160 MB memory in the structure cache of JChem.
Note: although JChem can drop the least recently used table from the structure cache if low on memory, it is recommended that all structure tables should fit in the cache (as cache loading can take a considerable amount of time). When estimating the memory need simply sum the number of rows in the tables.
The following table shows typical memory needs for standard structures:

Test specifications:
Number of molecules: 10,000,000
Fingerprint size: 16*32=512 bits
Average SMILES length per molecule: 60.4
Memory consumption: 1,604.1 MB
Caching time: 900 seconds

Memory need increases with the number of molecules, the size of fingerprints, and the average SMILES string size. The following approximation can be used when number_of_molecules > 25000:

memory_need[bytes] =
    number_of_molecules * (0.5 * average_smiles_length[characters] + fingerprint_size[bits] / 8 + 74)

The structure table fingerprint statistics generation function can be used to report the average smiles length and fingerprint size of a JChem table or JChem index. See more information in the JChem Manager command line usage (s command), at the Cartridge index statistics function and the Statistics tab at Instant JChem Schema editor.

Benchmarks

How fast is importing in JChem Base?

The following table shows the duration of import in some cases. The configuration was the same as in the cartridge benchmark.

Number of structures Elapsed Time (ms)
Duplicates allowed Duplicates not allowed
1000039118.040987.0
100000380272.0414200.0
200000808910.0879162.0

Notes:

  • If duplicates are allowed, time increases linearly with the number of molecules imported.
  • If duplicates are not allowed, JChem performs a search for every molecule to check if it is already in the structure table or not.
  • The table into which the structures were imported always contained an initial 10 structures and were stats-collected

How fast is substructure searching in JChem Base?

The following tests demonstrate the speed of substructure search in JChem.
The test configuration was the same as in the cartridge benchmark, and the same query structures were used.

  • A chemical table containing 38,165,924 molecules was used;
  • Fingerprints and SMILES were cached by JChem;
  • JChem version: 6.2.0;
  • Java version: Oracle Corporation 1.7.0_17;
  • OS: amd64 Linux 2.6.18-164.0.0.0.1.el5xen.

Query Structure Search Options Number of Hits Screened Count Search Time (ms)
t:s006944.0
t:s004343.0
t:s337388.0
t:s771156649.0
t:s931934121.0
t:s1171174003.0
t:s1371375077.0
t:s6967208016.0
t:s112713376572.0
t:s3841397310336.0
t:s10047104218944.0
t:s23347223385215355.0
t:s5937200599078691155.0
t:s83394369189891148637.0

The column names have the following meaning:
  • Query Structure: The query tested.
  • Search Options: The search options applied.
  • Number of Hits: The number of structures returned by the query.
  • Screened Count: The number of structure left over from the fingerprint screening as possible candidates meeting the search criteria.
  • Search Time: The total time spent executing the query.

How fast is inserting in JChem Cartridge?

The following table shows the duration of inserting in some cases. The configuration was exactly the same as in the cartridge benchmark.

Number of structures Elapsed Time (ms)
Duplicates allowed Duplicates not allowed
10000178493.0212149.0
1000001838819.03282147.0
2000003728915.010428034.0

How fast is indexing in JChem Cartridge?

The following table shows the duration of indexing in some cases. The configuration was exactly the same as in the cartridge benchmark.

Number of structures Elapsed Time (ms)
Duplicates allowed Duplicates not allowed
1000012184.015544.0
10000079475.0898017.0
200000163747.03700983.0

How fast is searching in JChem Cartridge?

The following table shows the duration of search in JChem Cartridge using the following configuration:
  • JChem version: 6.2.0;
  • Oracle version: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit;
  • Hardware: Amazon EC2, m2.4xlarge instance, EBS storage;
  • Other environmental data;
  • Target molecule set: PubChem;
  • Number of structures: 38,165,924.

Session Date: 2013-10-05
Query Structure Search Options Number of Hits Screened Count Search Time (ms)
t:s earlyResults:2000005770.0
t:s earlyResults:2000005255.0
t:s earlyResults:2000335107.0
t:s earlyResults:2000771154644.0
t:s earlyResults:2000931934650.0
t:s earlyResults:20001171174619.0
t:s earlyResults:20001371375290.0
t:s earlyResults:20006967205095.0
t:s earlyResults:2000112713375826.0
t:s earlyResults:2000384139735881.0
t:s earlyResults:200010047104216660.0
t:s earlyResults:200023347223385212518.0
t:s earlyResults:200059372005990786103787.0
t:s earlyResults:200083394369189891162485.0
t:t simThreshold:0.9003559.0
t:t simThreshold:0.9003051.0
t:t simThreshold:0.9002830.0
sep=! t:s!ctFilter:(PSA() <= 200) && (rotatableBondCount() <= 10) && (mass() <= 500) && (aromaticRingCount() <= 4)1301374875.0
sep=! t:s!ctFilter:(mass() <= 500) && (logP() <= 5) && (donorCount() <= 5) && (acceptorCount() <= 10)151374839.0

The column names have the following meaning:
  • Query Structure: The query tested.
  • Search Options: The name of the operator tested.
  • Number of Hits: The number of structures returned by the query.
  • Screened Count: The number of structure left over from the fingerprint screening as possible candidates meeting the search criteria.
  • Search Time: The total time spent executing the SQL statement.

Tuning

What can/should I do to make JChem Cartride searches faster?

You can find a few performance tuning hints here.

Do you have a question? Would you like to learn more?

Please browse among the related topics on our support forum or search the website. If you want to suggest modifications or improvements to our documentation email our support directly!