Search Cluster
Our turn-key solution for your identity records. Your ultimate power Enterprise PII Platform.
Built for Enterprise Scale
Search Cluster is a turn-key solution designed to store, search, and manage person records at enterprise scale. Built on proven cluster technology with redundancy, encryption, and semantic matching capabilities.
System Overview
Encrypted End-to-End
TLS transport encryption and AES-256 storage encryption
Distributed Cluster
Automatic failover and load balancing across nodes
Continuous Backup
Off-site backups with point-in-time recovery
One cluster, many workflows
A single index can simultaneously support multiple use cases — automated pipelines and human-facing search side by side.
End-to-end automated processing
No human in the loop — API calls handled by defined rules. Typical patterns:
- •Searches by unique identifier (SSN, email, phone) returning matching records.
- •Upsert requests that update existing records or insert new ones based on matches.
- •Copy search requests comparing complete records against indexed data using predefined scoring rules.
Human-assisted processing
Autocomplete and full-text search designed for human interaction — users see and assess the results, then decide what to do next. Same index, different consumption pattern.
Match people, not strings
A two-phase approach designed for person records, CRM systems, and customer databases — well beyond traditional database search.
Recall
SearchCluster establishes indices that accommodate phonetic variations and simplifications. The query is transformed internally, identifying candidate matches — records sharing enough commonalities to warrant deeper examination.
Match
Candidate records are retrieved, decrypted, then compared entity-by-entity and attribute-by-attribute using the Matcher component — producing detailed match results delivered back to the requesting system.
Your data, your structure
Flat, relational, or deeply nested — SearchCluster ingests data in the shape you already have.
Tabular
Flat structures with columns and rows — typically from SQL databases or CSV feeds.
Relational
Multiple sets of tabular data linked through keys and IDs.
Document
Nested objects and arrays representing naturally structured information.
Index schema
SearchCluster builds an internal schema organising entities with semantic data types:
Typed elements — GIVENNAME, SURNAME, RECORDID.
Collections of fields — PERSON, ADDRESS.
Fields and structures support multiple values via CSV or JSON arrays.
Protect the people behind the data — as well as your company — by preventing data theft.
Every Optimaize SearchCluster stores the data for each record not in plain text, but in encrypted form.
Compare a traditional database on the left, such as a relational database table, with a SearchCluster on the right — drag the slider:
Built for billions of records, real-time response
Storage layer
OpenSearch (formerly Elasticsearch) provides built-in redundancy and unlimited scalability. Each data item is replicated across separate hardware, leveraging Apache Lucene underneath. Capacity scales far beyond global population figures.
Matching layer
Stateless matcher nodes scale horizontally across the cluster. They perform detailed record comparisons on candidates identified by OpenSearch queries. Since the layer is stateless, scaling is effectively infinite.
Typical deployment: 100M records, sub-100ms search response, 100 concurrent requests, writes visible within one second. Real-world numbers vary with data shape, query patterns, and infrastructure.
No single point of failure
Redundancy at every layer — data store, app nodes, hardware, data centers, and even API endpoints. Plus continuous off-site backup as a disaster-recovery safety net.
Data store
Records stored multiple times across separate hardware. Nodes can go offline without service interruption. Auto-sync, auto-scale. Over 10 years of operation without downtime using this technology.
App nodes
Stateless matcher nodes scale dynamically with demand. Nodes can be added or removed without affecting service availability.
Data centers & endpoints
Indices replicated across multiple data centers — surviving full DC outages from network failures, power loss, DDoS, or physical disasters. REST endpoints accessible through fully redundant URLs with different domains, IPs, and SSL certificates.
Continuous backup
Changes write through to an off-site incremental backup — safety net for human error, unauthorized access, software bugs, or large-scale hardware failure. Cross-cluster replication available for geo-distributed read access.
Keep history when it matters
Customer records often need to preserve previous versions — name changes, address updates, identity transitions. SearchCluster gives you three patterns to choose from.
Latest version only
Versioning disabled. Only the most recent write is indexed and returned. Applications can maintain history independently in a relational database or audit log.
Full record versions
New version stored on every update — incremental number + timestamp. Searches can target current, historical, or both. Retention strategies:
- • Last N versions (e.g., 10)
- • Time-based (e.g., past 3 years)
- • Always keep the initial version
- • Any combination of the above
Self-managed
Assemble records in JSON. Instead of storing a single email, maintain a list with timestamps: [{"email":..., "last-seen":...}].
Maintenance that doesn't hurt you
Predefined routines that traverse records without downtime, service interruption, or performance degradation.
Retention policy enforcement
Privacy regulations require deletion after defined inactivity periods. Bake compliance into the deployment — automated processes with continuous backup.
Encryption key rotation
The data encryption system uses background processing to rotate keys safely across the live record set.
Data model changes
When schemas expand with new fields, re-indexing may become necessary or beneficial — performed in the background.
Technical evolution
Search and match logic upgrades require internal re-indexing, scheduled in coordination with customers.
Record evolution
Each record carries an evolution version. On changes — key rotation, model updates — records are processed individually through separate transactions with optimistic locking. Processing happens distributed across cluster nodes. The procedure runs until all records reflect the new evolution number, potentially taking minutes to hours depending on dataset size. Crucially, the application works with both evolution versions during the transition — ensuring continuous operation throughout.
Regulatory Requirements
Encryption
Record-level AES-256 encryption. No plain text data stored. Semantic search over encrypted data.
Audit + Traceability
Complete audit trail. Record versioning. Full traceability of all data access and modifications.
Continuity + Recovery
Continuous backups. Point-in-time recovery. Data lifecycle management and automated expiration.
European Union
United States
Why Search Cluster?
Hard to Scale
Traditional databases struggle with billions of encrypted person records. Performance degrades with volume.
Hard to Prove Compliant
Meeting GDPR, DORA, NIS2 requires extensive auditing, encryption, and data lifecycle controls.
Hard to Guarantee Availability
Mission-critical systems need 99.99% uptime with automatic failover and disaster recovery.
Semantic Search & Matching
Flexible Schema
Semantic Matching
Advanced AI-powered name matching finds duplicates even with typos, nicknames, and cultural variations.
Search Over Encrypted Data
Query encrypted records without decryption. Proprietary indexing enables fast semantic search.
Enterprise Volumes
Technical deep dive
Open any topic to see how Search Cluster is engineered for scale, availability and operational resilience.
Performance & ScalingSub-50 ms search across hundreds of millions of records
Sub-50 ms search across hundreds of millions of records
Search latency. Complex search results against hundreds of millions of person records, with detailed match information, return in under 50 milliseconds. Typical 100M-record deployments respond within 100 ms while serving 100 concurrent requests.
Update propagation. Create, update and delete commands take effect throughout the distributed cluster within one second.
Scale. Production deployments handle 10 billion person records, 100 billion transaction records and 10k updates and searches per second. Storage runs on OpenSearch (Apache Lucene); the matching layer is a stateless application that scales horizontally across cluster nodes.
Exact numbers depend on data shape, search scenario and hardware setup.
Reliability & Disaster RecoveryRedundancy, off-site backup, cross-cluster replication
Redundancy, off-site backup, cross-cluster replication
No single points of failure. Data is stored across multiple hardware nodes; some can go offline without service disruption. Stateless matcher nodes scale dynamically and can be added or removed without affecting availability. Optimaize has operated this infrastructure for over 10 years without downtime.
Multi-DC replication. Search Cluster indices are replicated across multiple data centres to maintain availability during full-facility outages — network failures, power loss, or other infrastructure issues. REST endpoints are reachable through separate, fully redundant URLs with different top- and second-level domains, IP addresses and SSL certificates.
Continuous backup. Changes to the data store write through to an off-site continuous incremental backup — protecting against accidental modification, unauthorised access, software bugs and large-scale hardware failure.
Cross-cluster replication. Maintain a read-only replica of your primary cluster for failover and geographically distributed query performance. Note: replication is not a substitute for backup — unwanted changes to the main cluster propagate to the replica.
OperationsBackground processing & record versioning
Background processing & record versioning
Background maintenance. Long-running routines traverse records without service interruption: retention-policy enforcement (deleting customer data after a defined inactivity period to satisfy privacy law), encryption-key rotation, data-model migrations, and re-indexing for search/match logic upgrades.
Evolution versions. Each record carries an evolution-version field. During migrations, records are traversed and updated individually with optimistic locking; processing distributes across cluster machines and runs from minutes to hours depending on dataset size. Applications can operate against both versions during the transition.
Record versioning options.
- Latest version only (default). The last write wins; previous versions are discarded.
- Fully managed full record versioning. Each new write is stored with an incremental version number and timestamp. Search across current, historical or combined versions. Retention strategies: keep N recent versions, time-based windows (e.g. last 3 years), preserve initial version, or combinations.
- Self-managed versioning. Embed your own version logic into the JSON record — e.g. an e-mail field stored as objects with last-seen timestamps.
Data SchemaMapping your data into the cluster
Mapping your data into the cluster
1. Original data sources. Search Cluster accepts three shapes:
- Tabular — flat rows and columns from SQL databases or CSV feeds.
- Relational — multiple linked tables connected through keys and IDs.
- Document — nested objects and arrays representing natural data structures.
2. Record feed. Records can be sent as simple CSV (spreadsheet-like rows), grouped CSV (using prefixes for entity namespaces), flat JSON (equivalent to CSV) or nested JSON (data from multiple joined tables).
3. Index schema. Internally the cluster organises data into entities with specific types (PERSON, ADDRESS) and fields with semantic data types (GIVENNAME, SURNAME, POSTALCODE).
Multiple values. Fields accept comma-separated strings or JSON arrays; structures support multiple instances via arrays. Combined with record versioning, the schema captures your data as it really is.
API-First Platform
REST API
Comprehensive RESTful API with JSON payloads. Full documentation and SDKs available.
Deployment Options
Cloud-hosted or on-premise. Your choice of infrastructure and data residency.
Example Integration
Everything you need, in one platform
Click any capability for details — or scan the grid for an overview.
Semantic record matching
Record level encryption
Data maintenance
Your schema
Your use case
Customize behavior
Unlimited scaling
Blazing fast
Powerful APIs
Fail-safe and redundant
Continuous backup
Keeping history
Integration Options
Ready when you are
The Time to Act is Today
Regulatory pressure is increasing. Data breaches are costly. Your customers expect protection. Search Cluster provides the infrastructure you need to stay compliant, secure, and scalable.