Cyc Administrator Handbook/Performance Tuning

Prev: Support Components Home:Cyc Administrator Handbook Next: Regression Tests

This page is based on the original document Enterprise Cyc Administrator Handbook at https://www.cyc.com/documentation/enterprise-cyc-administrator-handbook/ There is no intent to infringe on Cycorp's Copyright.
It is Copyright (c) Cycorp 2019
Cycorp's address is 7718 Wood Hollow Drive Suite 250 Austin, TX 78731 USA
mailto:info@cyc.com   Main Phone: 512.342.4000

Benchmarking Infrastructure

Cyc ships with a standard benchmark test for inference, appropriately called CycLOPS.
In addition, there is a benchmark test for Cyc browser performance that is not part of the standard release, but can be obtained from Cycorp, called cb-ops.

The CycLOPS Inference Benchmark Background

Cyc ships with a standard benchmark test for inference called CycLOPS.
Historically, CycLOPS has been shown to be representative of a wide mix of Cyc server operations.
The test constructs a (scalable) hierarchy of family relationships between a newly created set of terms, queries the relationships between them, and tears the setup down again.
Thus, CycLOPS tests all of the standard operations–term creation, fact assertion, query, fact and term removal.
Because CycLOPS extends the KB, the knowledge base in which it is run influences the overall performance: On identical hardware, smaller KBs with fewer rules will yield higher CycLOPS than bigger KBs with more rules.#  
Consequently, CycLOPS comparisons are only meaningful across commensurate KB sizes.
On UN*X systems, where bogomips information is readily available via /proc/cpuinfo, CycLOPS automatically computes and reports the relationship between CycLOPS and bogomips.

Running Single-Worker CycLOPS

The basic invocation pattern for CycLOPS is

(benchmark-cyclops-compensating-for-paging)

This will cause Cyc to run several iterations of the CycLOPS benchmark, discarding its results to compensate for any effects of paging in KB contents from the disk. The default is 30 iterations, and can be adjusted by passing an appropriate throw-away value as the first optional argument:

 ;; do 40 warm-up runs

(benchmark-cyclops-compensating-for-paging 40)

After that “warm up” period, the benchmarking runs the number of actual samples.# The default is 7, and can be adjusted by passing an appropriate sampling value as the second optional argument.

do 40 warm-up runs, then sample 10 times

(benchmark-cyclops-compensating-for-paging 40 10)

The individual tracking runs are measured and reported as they run.

CycLOPs Benchmark Results
========================================
System 10.140088 KB 5520
Bogomips : 4986.67
Scaling factor : 63
Elapsed time (seconds) : 3.219
Efficiency (seconds/op) : 0.0511
CycLOPs : 19.57
Bogomips/CycLOPS :# 254.8

Finally, the overall results are summarized and analyzed for statistical significance.

CycLOPs Benchmark Statistical Summary
========================================
System 10.140088 KB 5520
Raw Data : (18.82 19.81 19.87 19.9 19.56 20.38 19.57)
Sorted Data : (18.82 19.56 19.57 19.81 19.87 19.9 20.38)
N : 7
Min : 18.82
Max : 20.38
Median : 19.81
Mean : 19.7
Standard Deviation : 0.4394

Sizing the CycLOPS Test Ontology

The CycLOPS benchmark is sensitive to the depth of the test ontology that is constructed. This ontology can be scaled by increasing its power, which refers to the depth of its subsumption and transitivity hierarchies. By default, the power level is set to 6, but the value can be adjusted as the third optional argument.

 ;; do 40 warm-up runs, then sample 10 times at power 7

(benchmark-cyclops-compensating-for-paging 40 10 7)

Running Multi-Worker CycLOPS

It is possible to run multiple CycLOPS concurrently. This requires additional synchronization to be introduced.

(with-cyclops-lock

(benchmark-parallel-cyclops-compensating-for-paging)

)

By default, this runs the benchmark with a two-fold parallelism. In order to make it easy to increase the parallelism, the optional arguments have all been shifted in the signature, so that the n-way of the parallelism is now the first argument, the throw-away count the second, the sample size the third, and the CycLOPS power the fourth.

(with-cyclops-lock

 ;; run 4 threads in parallel, throw 40 runs away,

 ;; then take 10 samples at CycLOPS power 6

(benchmark-parallel-cyclops-compensating-for-paging 4 40 10 6)

)

Mitigating Bad CycLOPS Performance

The main ways to keep Cyc performing well are to

• Equip the Cyc server machine with appropriately fast CPUs.
• Ensure the KB units files are kept on an appropriately fast disk, preferably SSD or RAM if possible.
• Make adequate RAM available to the Cyc server, both physically and in software via tuning the -Xms and -Xmx parameters in the cyc-runner script.
• Notice that the CycLOPS benchmark is also sensitive to the quality of the KB’s forward rules, esp those on #$isa. It is good practice to periodically rerun the CycLOPS benchmark during project development and to follow-up on reducing CycLOPS (on identical hardware and configuration) with detailed inference metrics investigation.