Internal Architecture

Overview

Q01 Core APIs implement a CQRS (Command Query Responsibility Segregation) pattern with three microservices:

CoreService (Go/Buffalo) - API Gateway
CoreQuery (PHP/Symfony) - Read Operations
CoreWrite (PHP/Symfony) - Write Operations

From the user perspective:

All requests go to CoreService endpoints only
HTTP method determines operation type
Internal routing is completely transparent

Benefits:

✅ Read/write optimization (separate databases possible)
✅ Independent scaling (scale reads and writes separately)
✅ Clear separation of concerns
✅ Improved maintainability
✅ Security isolation

Architecture Diagram

┌─────────────────────────────────────────────────────┐
│                   Applications                      │
│  (Web Apps, Mobile Apps, Backend Services, etc.)   │
└──────────────────────┬──────────────────────────────┘
                       │
                       │ HTTP/HTTPS
                       │
┌──────────────────────▼──────────────────────────────┐
│              CoreService (API Gateway)              │
│              Go/Buffalo - Port 8080                 │
│                                                      │
│  • CORS handling                                    │
│  • SSL/TLS termination                              │
│  • Request logging                                  │
│  • Route-based routing logic                        │
│  • Error normalization                              │
└──────────────┬────────────────────┬─────────────────┘
               │                    │
               │ GET requests       │ POST/PUT/PATCH/DELETE
               │                    │
               ▼                    ▼
┌──────────────────────┐   ┌──────────────────────┐
│    CoreQuery         │   │    CoreWrite         │
│  PHP/Symfony         │   │  PHP/Symfony         │
│  Port 8081           │   │  Port 8082           │
│                      │   │                      │
│  • TB_DIM metadata   │   │  • TB_DIM metadata   │
│  • TB_VAR, TB_COST   │   │  • CRUD operations   │
│  • TB_MENU grants    │   │  • TREC lifecycle    │
│  • Context filtering │   │  • Cascade ops       │
│  • Field visibility  │   │  • OUTBOX events     │
│  • Read-only SQL     │   │  • Pre-insert funcs  │
└──────────┬───────────┘   └──────────┬───────────┘
           │                          │
           │                          │
           ▼                          ▼
┌──────────────────────────────────────────────────┐
│              Database Layer                       │
│                                                   │
│  • TB_META (metadata database)                   │
│  • TB_DATA (data database)                       │
│  • TB_AUTH (authentication database)             │
│                                                   │
│  MySQL / Oracle / PostgreSQL                     │
└──────────────────────────────────────────────────┘
           │
           │ RabbitMQ Events
           ▼
┌──────────────────────────────────────────────────┐
│         Event Subscribers                        │
│                                                   │
│  • search-indexer (Elasticsearch)                │
│  • cache-invalidator (Redis)                     │
│  • analytics-writer (Data warehouse)             │
│  • notification-sender (Email/SMS)               │
│  • webhook-dispatcher (External APIs)            │
└──────────────────────────────────────────────────┘

CoreService (API Gateway)

Technology Stack

Language: Go 1.19+
Framework: Buffalo v1.1.0
Port: 8080 (default)
Repository: q01sdk/mscoreservice/dkcoreservice

Responsibilities

Single Entry Point
- All applications call CoreService endpoints
- Public-facing API surface
Request Routing
- Route-based routing (named routes, not path matching)
- Intelligent routing to QUERY_API or WRITE_API
Cross-Cutting Concerns
- CORS handling
- SSL/TLS termination
- Request/response logging
- Error normalization
No Business Logic
- Pure gateway - no data validation
- No metadata interpretation
- No database access

Routing Logic

Route-based routing (not path-based):

func RoutingDefault(c buffalo.Context) error {
    // Get route name from context
    currRouteStruct := buffalo.RouteInfo{}
    currRoute := c.Data()["current_route"]
    jsonData, _ := json.Marshal(currRoute)
    json.Unmarshal(jsonData, &currRouteStruct)
    routeName := currRouteStruct.PathName

    resource := c.Request().URL.Path

    // Route based on named route
    switch routeName {
    // READ OPERATIONS → CoreQuery
    case "coreIndexPath",        // GET /api/v4/core/{dim}
         "coreShowPath",         // GET /api/v4/core/{dim}/{id}
         "coreFieldsPath",       // GET /api/v4/core/{dim}/fields
         "coreMenuPath":         // GET /api/v4/core/menu
        code, contentType, data = routingGet(c, resource, "QUERY_API")

    // WRITE OPERATIONS → CoreWrite
    case "coreCreatePath":       // POST /api/v4/core/{dim}
        code, contentType, data = routingPost(c, resource, "WRITE_API")

    case "coreUpdatePath":       // PUT /api/v4/core/{dim}/{id}
        code, contentType, data = routingPut(c, resource, "WRITE_API")

    case "corePatchPath":        // PATCH /api/v4/core/{dim}/{id}
        code, contentType, data = routingPatch(c, resource, "WRITE_API")

    case "coreDestroyByIDPath":  // DELETE /api/v4/core/{dim}/{id}
        code, contentType, data = routingDelete(c, resource, "WRITE_API")
    }

    return c.Render(code, r.JSON(data))
}

Key characteristics:

Routing based on route name (e.g., "coreIndexPath")
Route names end with "Path" suffix
Same URL path can route to different backends based on route name
No path pattern matching - purely based on named routes

Backend Destination Resolution

func getDestination(destination string) (string, string) {
    var host string
    var port string

    if destination == "QUERY_API" {
        host = os.Getenv("R_API_HOST")   // CoreQuery host
        port = os.Getenv("R_API_PORT")   // CoreQuery port
    } else {
        host = os.Getenv("W_API_HOST")   // CoreWrite host
        port = os.Getenv("W_API_PORT")   // CoreWrite port
    }

    return host, port
}

func getUrl(destination string, resource string) string {
    host, port := getDestination(destination)

    apiUrl := host
    if port != "" {
        apiUrl = host + ":" + port
    }

    return apiUrl + path.Clean(resource)
}

HTTP Method Handlers

GET requests:

func routingGet(c buffalo.Context, resource string, destination string) (int, string, *interface{}) {
    apiurl := getUrl(destination, resource)

    // Create HTTP GET request
    api_req, _ := http.NewRequest("GET", apiurl, nil)

    // Clone headers from original request
    api_req.Header = c.Request().Header

    // Clone query parameters
    api_req.URL.RawQuery = c.Request().URL.Query().Encode()

    // Forward request
    core_api_client := &http.Client{}
    api_resp, _ := core_api_client.Do(api_req)

    // Read response
    data, _ := io.ReadAll(api_resp.Body)

    return api_resp.StatusCode, api_resp.Header.Get("Content-Type"), &data
}

POST/PUT/PATCH requests:

func routingPost(c buffalo.Context, resource string, destination string) (int, string, *interface{}) {
    apiurl := getUrl(destination, resource)

    // Read request body
    body, _ := io.ReadAll(c.Request().Body)

    // Create HTTP POST request
    api_req, _ := http.NewRequest("POST", apiurl, bytes.NewBuffer(body))

    // Clone headers
    api_req.Header = c.Request().Header

    // Clone query parameters
    api_req.URL.RawQuery = c.Request().URL.Query().Encode()

    // Forward request
    core_api_client := &http.Client{}
    api_resp, _ := core_api_client.Do(api_req)

    // Parse response
    var result interface{}
    json.NewDecoder(api_resp.Body).Decode(&result)

    return api_resp.StatusCode, "application/json", &result
}

Environment Variables

CoreQuery (Read API):

R_API_HOST - CoreQuery hostname (e.g., corequery)
R_API_PORT - CoreQuery port (e.g., 8081)
R_API_VERSION - Read API version (e.g., v4)

CoreWrite (Write API):

W_API_HOST - CoreWrite hostname (e.g., corewrite)
W_API_PORT - CoreWrite port (e.g., 8082)
W_API_VERSION - Write API version (e.g., v4)

CoreService:

GO_ENV - Environment (development/production)
ADDR - Bind address (default: 0.0.0.0)
PORT - Server port (default: 8080)

Binary/Media Handling

Special handler for binary responses:

case "coreMediaPath",
     "coreMediaPublicPath":
    code, contentType, data = routingGetImage(c, resource, "QUERY_API")

func routingGetImage(c buffalo.Context, resource string, destination string) (int, string, []byte) {
    apiurl := getUrl(destination, resource)

    api_req, _ := http.NewRequest("GET", apiurl, nil)
    api_req.Header = c.Request().Header
    api_req.URL.RawQuery = c.Request().URL.Query().Encode()

    core_api_client := &http.Client{}
    api_resp, _ := core_api_client.Do(api_req)

    // Read binary data
    data, _ := io.ReadAll(api_resp.Body)

    return api_resp.StatusCode, api_resp.Header.Get("Content-Type"), data
}

CoreQuery (Read Service)

Technology Stack

Language: PHP 8.0+
Framework: Symfony 6.x
Port: 8081 (default)
Repository: q01sdk/mscorequery/dkcorequery

Responsibilities

Metadata-Driven Queries
- TB_DIM, TB_VAR, TB_COST, TB_MENU interpretation
- Runtime schema resolution
Read-Only Operations
- GET endpoints only
- SELECT queries only
- No database modifications
Multi-Level Permissions
- peso (user level) filtering
- COD_ON_OFF field visibility
- TB_MENU grant checking
- Context filtering (source, ambiente, centro_dett)
Field Visibility
- COD_ON_OFF bitmask evaluation
- L (list), D (detail), R (search) flags
- Dynamic field inclusion/exclusion
Query Optimization
- Index-aware query generation
- Efficient JOIN construction
- Pagination support

Core Components

Main Query Engine:

src/Core/Modules.php - 4200+ lines of query logic
Metadata-driven SQL generation
Field visibility enforcement
Context filtering application

Request Validation:

src/EventDispatcher/Subscriber/RequestValidatorSubscriber.php
Token validation
Grant checking
Required parameter validation

CoreWrite (Write Service)

Technology Stack

Language: PHP 8.0+
Framework: Symfony 6.x
Port: 8082 (default)
Repository: q01sdk/mscorewrite/dkcorewrite

Responsibilities

CRUD Operations
- INSERT (POST)
- UPDATE (PUT)
- PARTIAL UPDATE (PATCH)
- DELETE (DELETE)
Entity Lifecycle
- TREC state management (N→M→C)
- Cascade operations (INSERT_CASCADE, UPDATE_CASCADE, DELETE_CASCADE)
- Soft delete support
Pre-Insert Functions
- uuid, md5, counter, slugify, sequence
- Metadata-driven field initialization
Event Sourcing
- OUTBOX pattern implementation
- RabbitMQ event publishing
- At-least-once delivery guarantee
Transaction Management
- ACID compliance
- Row-level locking
- Rollback on failure

Core Components

CRUD Engine:

src/Core/DbInterface/DataStore.php - 3354 lines
INSERT, UPDATE, PATCH, DELETE operations
Cascade logic
OUTBOX event generation

Request Validation:

src/EventDispatcher/Subscriber/RequestValidatorSubscriber.php
Token validation
Grant checking (write permissions)
Required field validation
COD_ON_OFF flag checking

Request Flow

Read Request Flow

1. Client → CoreService
   GET https://api.q01.io/api/v4/core/products?limit=10

2. CoreService → CoreQuery
   • Route name: "coreIndexPath"
   • Destination: QUERY_API
   • Forwards to: http://corequery:8081/api/v4/core/products?limit=10

3. CoreQuery Processing
   • Validate token (JWT)
   • Check TB_MENU grants (read permission)
   • Load TB_DIM metadata (dimension: PRD)
   • Load TB_VAR metadata (field definitions)
   • Apply context filters (source, peso, ambiente, centro_dett)
   • Evaluate COD_ON_OFF flags (L=list visible)
   • Generate SQL: SELECT PRD_ID, XPRD01, XPRD02... FROM TB_ANAG_PRD00 WHERE source=? AND TREC='N' LIMIT 10
   • Execute query
   • Build JSON response

4. CoreQuery → CoreService
   HTTP 200 OK
   {"data": [...], "total": 150}

5. CoreService → Client
   HTTP 200 OK
   {"data": [...], "total": 150}

Write Request Flow

1. Client → CoreService
   POST https://api.q01.io/api/v4/core/products
   {"XPRD01": "Widget", "XPRD02": 19.99}

2. CoreService → CoreWrite
   • Route name: "coreCreatePath"
   • Destination: WRITE_API
   • Forwards to: http://corewrite:8082/api/v4/core/products

3. CoreWrite Processing
   • Validate token (JWT)
   • Check TB_MENU grants (create permission)
   • Load TB_DIM metadata (dimension: PRD)
   • Load TB_VAR metadata (field definitions)
   • Validate required fields
   • Apply context (source, peso, ambiente, centro_dett)
   • Execute pre-insert functions (uuid, counter, etc.)
   • Begin transaction
   • INSERT INTO TB_ANAG_PRD00 (...)
   • INSERT INTO OUTBOX (event_type='ProductCreated', ...)
   • Commit transaction
   • Build JSON response

4. CoreWrite → CoreService
   HTTP 201 Created
   {"PRD_ID": 123, "XPRD01": "Widget", ...}

5. CoreService → Client
   HTTP 201 Created
   {"PRD_ID": 123, "XPRD01": "Widget", ...}

6. Background: OUTBOX Publisher → RabbitMQ
   • Poll OUTBOX table
   • Publish event to fanout exchange
   • Subscribers: search-indexer, cache-invalidator, etc.

CQRS Benefits

Read Optimization

CoreQuery can:

Use read replicas
Implement aggressive caching
Optimize for query performance
Scale independently

Example:

┌──────────────┐       ┌──────────────┐
│  CoreQuery   │──────▶│ Read Replica │
│   (3 pods)   │       │   (MySQL)    │
└──────────────┘       └──────────────┘

Write Optimization

CoreWrite can:

Use write master
Implement write-ahead logging (OUTBOX)
Optimize for transaction throughput
Scale independently

Example:

┌──────────────┐       ┌──────────────┐
│  CoreWrite   │──────▶│ Write Master │
│   (2 pods)   │       │   (MySQL)    │
└──────────────┘       └──────────────┘
        │
        │ Events
        ▼
┌──────────────┐
│   RabbitMQ   │
│   Fanout     │
└──────────────┘

Deployment Architecture

Docker Compose Example

version: '3.8'

services:
  coreservice:
    image: q01/coreservice:latest
    ports:
      - "8080:8080"
    environment:
      R_API_HOST: http://corequery
      R_API_PORT: 8081
      R_API_VERSION: v4
      W_API_HOST: http://corewrite
      W_API_PORT: 8082
      W_API_VERSION: v4
      GO_ENV: production
    depends_on:
      - corequery
      - corewrite

  corequery:
    image: q01/corequery:latest
    ports:
      - "8081:8081"
    environment:
      DATABASE_HOST: mysql-read
      DATABASE_PORT: 3306
      DATABASE_NAME: q01_data
      DATABASE_USER: q01_reader
      DATABASE_PASSWORD: ${DB_READ_PASSWORD}
    depends_on:
      - mysql-read

  corewrite:
    image: q01/corewrite:latest
    ports:
      - "8082:8082"
    environment:
      DATABASE_HOST: mysql-write
      DATABASE_PORT: 3306
      DATABASE_NAME: q01_data
      DATABASE_USER: q01_writer
      DATABASE_PASSWORD: ${DB_WRITE_PASSWORD}
      RABBITMQ_HOST: rabbitmq
      RABBITMQ_PORT: 5672
    depends_on:
      - mysql-write
      - rabbitmq

  mysql-read:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
      MYSQL_DATABASE: q01_data

  mysql-write:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
      MYSQL_DATABASE: q01_data

  rabbitmq:
    image: rabbitmq:3-management
    ports:
      - "5672:5672"
      - "15672:15672"

Kubernetes Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: coreservice
spec:
  replicas: 3
  selector:
    matchLabels:
      app: coreservice
  template:
    metadata:
      labels:
        app: coreservice
    spec:
      containers:
      - name: coreservice
        image: q01/coreservice:latest
        ports:
        - containerPort: 8080
        env:
        - name: R_API_HOST
          value: "http://corequery-service"
        - name: R_API_PORT
          value: "8081"
        - name: W_API_HOST
          value: "http://corewrite-service"
        - name: W_API_PORT
          value: "8082"
---
apiVersion: v1
kind: Service
metadata:
  name: coreservice
spec:
  type: LoadBalancer
  selector:
    app: coreservice
  ports:
  - port: 80
    targetPort: 8080

Performance Considerations

Latency

Additional hop:

Client → CoreService: ~5ms
CoreService → CoreQuery/CoreWrite: ~2ms
Total overhead: ~7ms

Mitigation:

Deploy all services in same region/VPC
Use connection pooling
Enable HTTP keep-alive

Throughput

CoreService bottleneck:

Go is highly concurrent (goroutines)
Can handle 10,000+ req/sec per pod
Scales horizontally

Monitoring:

# Check CoreService throughput
curl http://coreservice:8080/metrics | grep http_requests_total

Best Practices

✅ DO:

Use environment variables for configuration:

# ✅ Good - external configuration
export R_API_HOST=http://corequery-prod
export W_API_HOST=http://corewrite-prod

Scale services independently:

# ✅ Good - scale reads more than writes
kubectl scale deployment corequery --replicas=5
kubectl scale deployment corewrite --replicas=2

Monitor all three services:

# ✅ Good - comprehensive monitoring
curl http://coreservice:8080/health
curl http://corequery:8081/health
curl http://corewrite:8082/health

❌ DON'T:

Don't call CoreQuery/CoreWrite directly:

# ❌ Bad - bypass CoreService
curl http://corequery:8081/api/v4/core/products

# ✅ Good - through CoreService
curl http://coreservice:8080/api/v4/core/products

Don't hardcode backend URLs:

// ❌ Bad - hardcoded
apiUrl := "http://corequery:8081" + resource

// ✅ Good - environment variable
apiUrl := os.Getenv("R_API_HOST") + resource

Summary

✅ CoreService is the single public-facing API gateway
✅ Route-based routing (named routes, not path matching)
✅ CoreQuery handles all read operations (GET)
✅ CoreWrite handles all write operations (POST/PUT/PATCH/DELETE)
✅ CQRS enables independent scaling and optimization
✅ Environment variables control routing
✅ All business logic in CoreQuery/CoreWrite, not CoreService
✅ Clean separation of concerns

Key Takeaways:

Users only call CoreService endpoints
Internal routing is completely transparent
Named routes determine destination (not HTTP method alone)
CQRS enables read/write optimization
Scale CoreQuery and CoreWrite independently
Monitor all three services
Deploy in same region for low latency
Never bypass CoreService in production

CQRS Pattern - Martin Fowler
API Gateway Pattern
Query Patterns - Read operations
Write Patterns - Write operations
Advanced Topics - Overview

Internal Architecture

Overview​

Architecture Diagram​

CoreService (API Gateway)​

Technology Stack​

Responsibilities​

Routing Logic​

Backend Destination Resolution​

HTTP Method Handlers​

Environment Variables​

Binary/Media Handling​

CoreQuery (Read Service)​

Technology Stack​

Responsibilities​

Core Components​

CoreWrite (Write Service)​

Technology Stack​

Responsibilities​

Core Components​

Request Flow​

Read Request Flow​

Write Request Flow​

CQRS Benefits​

Read Optimization​

Write Optimization​

Deployment Architecture​

Docker Compose Example​

Kubernetes Example​

Performance Considerations​

Latency​

Throughput​

Best Practices​

✅ DO:​

❌ DON'T:​

Summary​

Related Concepts​

Overview

Architecture Diagram

CoreService (API Gateway)

Technology Stack

Responsibilities

Routing Logic

Backend Destination Resolution

HTTP Method Handlers

Environment Variables

Binary/Media Handling

CoreQuery (Read Service)

Technology Stack

Responsibilities

Core Components

CoreWrite (Write Service)

Technology Stack

Responsibilities

Core Components

Request Flow

Read Request Flow

Write Request Flow

CQRS Benefits

Read Optimization

Write Optimization

Deployment Architecture

Docker Compose Example

Kubernetes Example

Performance Considerations

Latency

Throughput

Best Practices

✅ DO:

❌ DON'T:

Summary

Related Concepts