Data Engineering at Josys

At Josys, we work with data across multiple domains, including employees, devices, assets, accounts, licenses, and security insights. By ingesting and integrating this data at scale, we enable intelligent, data-driven features that align with our business objectives and long-term product roadmap.

The section below outlines the core data domains ingested by Josys and the feature value derived from each -

‍

As the company grew, so did the diversity of our data practices. Newly formed teams built data engineering solutions tailored to their immediate needs. Some relied on traditional analytical workflows using MySQL, while others experimented with NoSQL systems such as MongoDB.

Around mid-2024, we introduced a common aggregator microservice built with Node.js to standardize analytical data generation across systems. This service reads data from MySQL replicas, performs compute-heavy aggregations, and writes the final reports to MongoDB. The pipeline remains in active production today, powering multiple customer dashboards and reporting features.

However, we were not fully leveraging the benefits of distributed compute. As data volumes increased, we began to see higher latency during both aggregation processing and downstream writes to MongoDB. To put this into context, let’s take a closer look at our data load.

‍

These trends made our direction clear. To sustain—and ideally improve—ingestion and analytical performance at any scale, we needed a more robust foundation. This realization marked the start of our journey toward building an industry-standard, distributed data engineering framework: IDAC (Ingestion, Data, Analytics, and Compute).

‍

The Shift in Engineering with IDAC at Centre

‍

Medallion Architecture

• The Bronze layer serves as the raw data zone, where ingested data lands in its original, unprocessed form.
• The Silver layer is where data cleansing, deduplication, and standard transformations take place, ensuring consistency and quality.
• The Gold layer contains curated, business-ready datasets - this is where complex computations, aggregations, and customer - facing analytical reports are generated.

‍

‍

IDAC Tech Stack

Below is a tech-stack diagram illustrating our Spark-based data engineering architecture. The diagram highlights the specific technologies used at each layer of the data stack, from storage and processing to orchestration and infrastructure.

‍

‍

Apache Spark - the distributed compute engine

As our data volumes grew, we needed a system capable of processing large workloads efficiently-without being constrained by a single server’s capacity. While our earlier Node.js and MongoDB–based aggregation services worked well initially, they were fundamentally single-node systems: suitable for lightweight analytics, but not for large-scale, distributed computation.

Apache Spark provided a clear path forward. Its ability to distribute compute across multiple machines, process data in parallel, and automatically optimize workloads enabled us to handle growing datasets with consistent performance. By adopting Spark as the backbone of our data engineering framework, we ensured our pipelines remain scalable, reliable, and future-ready-capable of supporting both today’s requirements and tomorrow’s growth.

‍

How does data come to IDAC

At Josys, every feature is designed with data at the core. To support this, our engineering team streamlined how data flows into our unified data platform, IDAC. IDAC is built for flexibility, enabling data ingestion through multiple approaches-including Change Data Capture (CDC), real-time streaming, and custom ingestion functions. This adaptability ensures that, regardless of source system or format, data can seamlessly enter IDAC, move through the Silver and Gold layers, and ultimately power high-quality analytics and reporting across our platform.

‍

IDAC - Centralized Analytics with Built-In Security

As organizations scale, data becomes their most strategic asset, but only when it is consolidated, governed, and universally accessible. IDAC is designed to serve as a single source of truth for all analytical needs across the enterprise. By unifying batch, real-time, and search-driven insights into one cohesive platform, IDAC eliminates fragmented data silos and enables a unified, high-trust analytics ecosystem.

With IDAC at the center, our engineering team could quickly unlock:

• Consistent insights across business units
• Cross-domain visibility for advanced analytics
• Reliable operational reporting with standardized schemas
• A shared analytical language that ensures everyone works from the same truth

By bringing together all analytical workloads into one secure, scalable platform, IDAC empowers teams to move faster without compromising customer trust. It becomes the single place to query, explore, model, and operationalize every form of data - structured, semi-structured, streaming, or search-indexed IDAC’s vision is simple yet powerful:

One platform for all analytics. One contract for all data trust. Built once, used everywhere - always secure

‍