Blog | Tuan Le's Blog

Researching CDC: How to sync data between two mongo clusters using debezium, rabbitmq and golang consumer. Part 1

Sat, 25 Oct 2025 00:00:00 +0000

Introduction

Synchronizing data between two MongoDB clusters can be deceptively complex — especially when we need both consistency and low latency. Traditional approaches either prioritize speed (but risk losing data) or safety (but introduce delays).
This article explores a more balanced solution using Change Data Capture (CDC) with Debezium, RabbitMQ, and a Golang consumer.

The goal: achieve reliable, real-time synchronization without sacrificing consistency.

Table of Contents

Why This Research Exists?

When we follow a microservices architecture, where each service owns its own database and domain logic.
This independence is great for scalability and deployment, but it creates a major challenge when customers need aggregated views across domains.

A frequent request from our users was a centralized page to search, filter, and view all products in one place.
However, since each microservice stores its product data separately, there’s no single data source to support that unified search efficiently.
To handle this, we built a new aggregation service — essentially a search index that consolidates product data from multiple services into one dedicated database.

The critical part was keeping this data continuously synchronized with upstream product services in real time.

Solutions Applied

1. Direct Event Push

In this model, the service publishes an event to RabbitMQ immediately after writing to MongoDB.

flowchart TD U["User"] S["Product Service"] DBM["MongoDB (Product)"] MQ["RabbitMQ"] CEH["Consumer"] DBF["MongoDB (fusion)
(Centralized Product Index)"] %% User action and domain write U --> S -->|Write product change| DBM %% Event publish path S -->|Publish ProductChanged| MQ --> CEH -->|Upsert/Project product| DBF %% (Optional) failure note %% If publish fails: data in Product committed but event not delivered → inconsistency risk %% Styling (optional) style U fill:#f5f5f5,stroke:#aaa style S fill:#b4d4f7,stroke:#333 style DBM fill:#fef6d8,stroke:#999 style MQ fill:#fdddb1,stroke:#999 style CEH fill:#c8f2c2,stroke:#999 style DBF fill:#f6c2f0,stroke:#999

This approach is simple and works well in happy cases. However, when failures occur (e.g., RabbitMQ is down), data consistency cannot be guaranteed since the database update and event publishing are not atomic.

Pros:

Low latency — target systems get updates almost instantly.
Simple to implement — just hook after the database write.

Cons:

Ricky: data might be updated in MongoDB, but the message broker could fail to receive the event.
Service must care about infra concerns (data synchronization).

2. Outbox Pattern with Polling publisher

Here, both the data and the event are stored within the same MongoDB transaction. A separate background process (cron/worker) later publishes the event to RabbitMQ.

flowchart TD U["User"] S["Product Service"] subgraph Product["MongoDB (Product)"] D1["Product Collection"] O1["Outbox Collection"] end POLL["Polling Publisher (cron/worker)"] MQ["RabbitMQ"] CEH["Consumer"] DBF["MongoDB (fusion)
(Centralized Product Index)"] %% User action and atomic DB changes U --> S -->|TX: write product| D1 S -->|TX: insert outbox event| O1 %% Background publisher flow O1 -.->|poll & fetch events| POLL -->|publish| MQ --> CEH -->|Upsert/Project product| DBF POLL -->|mark processed| O1 %% Styling (optional) style U fill:#f5f5f5,stroke:#aaa style S fill:#b4d4f7,stroke:#333 style Product fill:#fef6d8,stroke:#999 style POLL fill:#ffe7a1,stroke:#999 style MQ fill:#fdddb1,stroke:#999 style CEH fill:#c8f2c2,stroke:#999 style DBF fill:#f6c2f0,stroke:#999

This approach resolves consistency issues but introduces operational complexity and latency from the polling process.

Pros:

Strong consistency — events are recorded only after data is safely stored.
Reliable delivery — supports retry until success (at-least-once guarantee).

Cons:

Introduces delay — events are published on a schedule.
Increased operational complexity (polling, cron, cleanup).
Service must care about infra concerns (data synchronization).

Trade-offs Summary

Approach	Consistency	Latency	Complexity	Risk	Infra Concern
Direct Push	Weak	Low	Low	High	Yes
Outbox Pattern	Strong	Moderate	Medium	Low	Yes

Towards a Better Approach

What if we could:

Send events right after the data is safely committed
Without needing to modify application logic
Without adding cron jobs or extra tables?

The answer lies in Change Data Capture (CDC) — tailing the database transaction log.

Transaction log tailing including transaction log

Every MongoDB cluster maintains an oplog (operations log) that records all changes applied to the database.
By listening to this stream, we can detect inserts, updates, and deletes in real-time — even after the application finishes its writes.

flowchart TD S["product Service"] subgraph DBM["MongoDB (Product)"] COLL["Product Collection"] OPLOG["Oplog / Transaction Log"] end PROD["Change Stream Producer"] MQ["RabbitMQ"] CEH["Sink Consumer"] DBF["MongoDB (central)"] %% Flow connections S -->|"Write product changes
[insert/update/delete]"| COLL COLL -->|Record operation| OPLOG OPLOG -->|Stream changes| PROD PROD -->|Publish event| MQ MQ --> CEH CEH -->|Upsert / Project product| DBF %% Styling style S fill:#b4d4f7,stroke:#333 style DBM fill:#fef6d8,stroke:#999 style COLL fill:#fffbea,stroke:#aaa style OPLOG fill:#fff3e0,stroke:#aaa style PROD fill:#cce0ff,stroke:#333 style MQ fill:#fdddb1,stroke:#999 style CEH fill:#c8f2c2,stroke:#999 style DBF fill:#f6c2f0,stroke:#999

MongoDB provides a built-in change stream API that allows us to listen to the oplog instead of directly tailing it. It compromise that are optimized, providing more efficient resource utilization and faster execution from MongoDB version 5.1.
However, this approach requires additional implementation effort. This led me to explore Debezium, which I’ll discuss in the next section.

Let’s review what we’ve achieved with this architecture:

Strong consistency: since changes are captured post-commit.
Low latency: since events are streamed continuously.
Non-intrusiveness: no changes needed in the application code.

Introducing Debezium

1. What is Debezium?

Debezium is an open-source Change Data Capture (CDC) platform built on top of Kafka Connect.
It monitors databases for real-time changes — including MongoDB, MySQL, PostgreSQL, and others — and publishes structured change events (in JSON or Avro) to your chosen message broker such as Kafka, RabbitMQ, or other supported systems.

When used with MongoDB, Debezium reads from the oplog to detect inserts, updates, and deletes, and then streams those events downstream for consumers to process.

2. why not just use MongoDB’s built-in features?

First, MongoDB already provides a Change Stream API, which allows applications to listen for changes directly without manually tailing the oplog. However, this requires additional custom implementation, error handling, and message delivery logic — all of which add complexity and operational overhead.

Second, MongoDB officially provides Kafka Connector tools for both source and sink connectors ( ).
Unfortunately, our infrastructure uses RabbitMQ instead of Kafka. This means the native connectors do not fit into our existing message pipeline.

That’s where Debezium comes in — it offers a reliable and flexible CDC layer that integrates smoothly with RabbitMQ through the Debezium Server.
By connecting to MongoDB’s oplog, Debezium can publish changes as events directly into RabbitMQ — eliminating the need to write a custom listener or message dispatcher.

3. The Limitation

While Debezium simplifies the source side of CDC, it has one notable limitation: The Debezium MongoDB Sink Connector supports only relational database (RDBMS) sources.

This means it cannot process change events generated by another Debezium connector — including the Debezium MongoDB Source Connector. In our case, since both the source and sink are MongoDB clusters, the official sink connector cannot be used directly.

So, we still need to implement our own sink logic — a custom consumer that reads change events from RabbitMQ and writes them to the target MongoDB (fusion) cluster.

Architecture Overview

Here’s how the entire system fits together:

Product MongoDB Cluster → produces oplog entries.
Debezium Server (Change Stream Producer) → tails the oplog, transforms to events.
RabbitMQ Exchange/Queue → distributes events.
Go App (Sink Consumer) → consumes, filters, transforms, and writes to Target MongoDB Cluster.
Fusion MongoDB Cluster -> centralized products.

flowchart TD S["Product Console Service"] subgraph DBM["MongoDB (Product)"] COLL["Product Collection"] OPLOG["Oplog / Transaction Log"] end PROD["Debezium Server"] MQ["RabbitMQ"] CEH["Consumer(Golang)"] DBF["MongoDB (fusion)"] %% Flow connections S -->|"Write product changes
[insert/update/delete]"| COLL COLL -->|Record operation| OPLOG OPLOG -->|Stream changes| PROD PROD -->|Publish event| MQ MQ --> CEH CEH -->|Upsert / Project product| DBF %% Styling style S fill:#b4d4f7,stroke:#333 style DBM fill:#fef6d8,stroke:#999 style COLL fill:#fffbea,stroke:#aaa style OPLOG fill:#fff3e0,stroke:#aaa style PROD fill:#cce0ff,stroke:#333 style MQ fill:#fdddb1,stroke:#999 style CEH fill:#c8f2c2,stroke:#999 style DBF fill:#f6c2f0,stroke:#999

Benefits Achieved

✅ Consistency — events are emitted post-commit.
✅ Low Latency — oplog tailing means near-real-time sync.
✅ Resilience — RabbitMQ buffering ensures durability.
✅ Scalability — multiple consumers can handle different domains or tables.

Conclusion

In this first part, we explored the real-world challenge of keeping data synchronized across multiple MongoDB clusters — and how Change Data Capture (CDC) provides a modern, reliable way to solve it. We walked through the evolution from traditional approaches to a CDC-based architecture, and saw how tools like Debezium can help bridge consistency, latency, and scalability in distributed systems.

In Part 2, we’ll get hands-on with the implementation details, including:

Setting up Debezium with MongoDB
Configuring RabbitMQ for CDC event routing
Building a Golang consumer to apply data changes
Sharing code samples, deployment notes, and best practices

If this topic interests you — or you’d like to see the setup and code walkthrough in the next part — 💬 leave a comment below to let me know! Your feedback gives me the motivation to keep sharing deeper, practical guides.

Understanding n8n: Architecture and Core Concepts

Sat, 10 May 2025 00:00:00 +0000

Welcome to this guide on n8n! 👋

Table of Contents

What is n8n?

n8n is a powerful, free, and open-source workflow automation tool. It enables you to connect different applications and services to automate repetitive tasks without extensive coding. You can design complex workflows visually, using a wide array of pre-built nodes for popular apps and services, or even create your own custom nodes. This flexibility makes n8n a versatile solution for personal projects, business process automation, and more.

Core n8n Architecture

Understanding n8n’s architecture helps in leveraging its full potential, especially when it comes to scaling, customizing, or embedding n8n. While deep technical details can be extensive, here’s a high-level overview of its key components:

Key Components

The n8n Editor (Frontend/UI):
- This is the visual interface where you design and manage your workflows.
- It’s a web application, typically built with modern JavaScript frameworks.
- When you create a workflow, the editor essentially generates a JSON representation of that workflow.
Workflow Execution Engine (Backend/Worker):
- This is the core engine responsible for actually running your workflows.
- It takes the JSON definition of a workflow and processes it step-by-step.
- It handles node execution, data transformation between nodes, error handling, and logging.
- For scalability, n8n can run in different modes, including a “main” process and separate “worker” processes that can handle concurrent workflow executions. This is often referred to as “Queue Mode” for more robust, production-grade deployments.
Nodes:
- Trigger Nodes: These are special nodes that start a workflow. They can be activated by webhooks, schedules (cron), manual triggers, or events from third-party services.
- Regular Nodes: These perform specific actions like reading data from a database, sending an email, manipulating data, making API calls, or implementing custom logic.
- Each node is essentially a self-contained unit of code (often JavaScript/TypeScript) that performs a specific task.
Database:
- n8n uses a database (SQLite by default, but PostgreSQL and MySQL/MariaDB are supported for production) to store:
  - Workflow definitions
  - Credentials for connecting to various services
  - Execution logs and history
  - User data (if user management is enabled)
REST API:
- n8n exposes a REST API that allows for programmatic interaction. You can use this API to manage workflows, trigger executions, retrieve execution data, and more, enabling integration with other systems.

How It Works Together (Simplified Flow)

Design: You design a workflow in the n8n Editor. Your workflow is saved as a JSON object in the database.
Trigger: A trigger node initiates the workflow (e.g., an incoming webhook request, a scheduled time).
Execution:
- The Workflow Execution Engine loads the workflow’s JSON definition from the database.
- It starts with the trigger node and executes subsequent nodes in the defined order.
- Data output from one node becomes the input for the next, allowing for complex data flows and transformations.
- The engine communicates with external services as defined by the nodes (e.g., fetching data from an API, sending a message to Slack).
Logging: Execution details, including successes, failures, and data at each step, are logged in the database.

Scalability Considerations

Execution Modes: n8n can run in a single main process mode (default, good for smaller setups) or a queue mode with dedicated worker processes for better performance and scalability, especially when handling many concurrent workflows.
Database Choice: For production environments, using PostgreSQL or MySQL/MariaDB instead of SQLite is recommended for better performance and reliability under load.
Resource Allocation: Sufficient CPU, memory, and network bandwidth are crucial, especially for worker instances handling complex or high-volume workflows.

Planning Your Custom Node

Before writing any code, it’s important to plan your custom n8n node carefully. This ensures your node is maintainable, user-friendly, and fits well within your workflow automation goals.

1. Choosing the Node Type

n8n nodes generally fall into two categories:

Trigger Nodes: Start workflows when an event occurs (e.g., webhook received, new email).
Regular (Action) Nodes: Perform actions or process data within a workflow (e.g., HTTP request, data transformation).

Questions to consider:

Should your node initiate workflows (Trigger) or process data mid-flow (Action)?
Does your node need to poll an external service, or will it wait for incoming events?

2. Selecting a Node Building Style

n8n supports two main approaches for building nodes:

Declarative Style: Uses JSON to define node properties, inputs, outputs, and operations. This approach is simpler, more maintainable, and recommended for straightforward integrations.
- Routing-Based Development: Declarative style excels at creating HTTP-based nodes through a routing system.
- Simplified API Integration: Easily map API endpoints to operations without complex code.
- Operation-Centric: Define multiple operations (GET, POST, PUT, etc.) in a structured way.
Programmatic Style: Uses TypeScript code to define node behavior, offering more flexibility and control for complex logic or custom functionality.

Declarative Style Example with HTTP Routing:

{
 "name": "MyApiNode",
 "httpRequestDefaults": {
 "baseURL": "https://api.example.com/v1",
 "headers": {
 "Accept": "application/json"
 }
 },
 "properties": [
 {
 "displayName": "Resource",
 "name": "resource",
 "type": "options",
 "options": [
 {
 "name": "User",
 "value": "user"
 },
 {
 "name": "Product",
 "value": "product"
 }
 ]
 },
 {
 "displayName": "Operation",
 "name": "operation",
 "type": "options",
 "options": [
 {
 "name": "Get",
 "value": "get",
 "routing": {
 "request": {
 "method": "GET",
 "url": "=/{{$parameter.resource}}/{{$parameter.id}}"
 }
 }
 },
 {
 "name": "Create",
 "value": "create",
 "routing": {
 "request": {
 "method": "POST",
 "url": "=/{{$parameter.resource}}",
 "body": "={{$parameter.data}}"
 }
 }
 }
 ]
 }
 ]
}

Programmatic Style Example:

export class MyCustomNode implements INodeType {
 description = {
 displayName: 'My Custom Node',
 // Other properties
 };

 async execute() {
 // Custom logic implementation
 // More flexibility and control
 }
}

You can also use a hybrid approach, combining declarative definitions with programmatic execution when needed.

Repository Layout for n8n-node-starter:

When developing custom nodes using the n8n-node-starter template, your project structure will typically look like this:

n8n-nodes-my-custom/
├── credentials/ # Credential definitions for your nodes
│ └── MyServiceApi.credentials.ts
├── nodes/ # Your custom node implementations
│ ├── MyService/
│ │ ├── MyService.node.ts # Main node implementation
│ │ ├── MyService.node.json # Declarative definition (if used)
│ │ └── descriptions/ # Localized descriptions
│ │ └── MyServiceDescription.ts
│ └── actions/ # Reusable actions
│ └── common.ts
├── package.json # Project configuration and dependencies
├── tsconfig.json # TypeScript configuration
├── .eslintrc.js # Linting rules
├── README.md # Documentation
└── LICENSE.md # License information

Test Node: You can test your node as you build it by running it in a local n8n instance.

Install n8n using npm:

npm install n8n -g

When you are ready to test your node, publish it locally:

# In your node directory
npm run build
npm link

Install the node into your local n8n instance:

# In the nodes directory within your n8n installation
# node-package-name is the name from the package.json
npm link <node-package-name>

3. Designing the Node UI

The user interface for your node in the n8n Editor is defined by its description object:

Display Name & Description: Choose clear, concise names and helpful descriptions.
Inputs & Outputs: Decide how data will flow into and out of your node.
Parameters: Plan which fields, dropdowns, credentials, and options users will interact with.
Defaults & Appearance: Set default values and color for easy identification.

Tips for UI Design:

Group related parameters for clarity.
Use descriptive labels and tooltips.
Minimize required fields to streamline the user experience.
Consider advanced options for power users, but keep the main UI simple.

🎉 Easily create your own simple yet highly customizable blog

Fri, 27 Oct 2023 00:00:00 +0000

Welcome 👋

Table of Contents

Overview

The Hugo Blox website builder for Hugo, along with its starter templates, is designed for professional creators, educators, and teams/organizations - although it can be used to create any kind of site
The template can be modified and customised to suit your needs. It’s a good platform for anyone looking to take control of their data and online identity whilst having the convenience to start off with a no-code solution (write in Markdown and customize with YAML parameters) and having flexibility to later add even deeper personalization with HTML and CSS
You can work with all your favourite tools and apps with hundreds of plugins and integrations to speed up your workflows, interact with your readers, and much more

Get Started

Quick Start Guide

New to Hugo Blox? Follow these steps to get your site up and running in minutes!

👉
📚
💬 or
🐦 Twitter: #MadeWithHugoBlox
💡
⬆️ Updating Hugo Blox? View the and

Important

Remember to backup your site before making major updates!

Crowd-funded open-source software

To help us develop this template and software sustainably under the MIT license, we ask all individuals and businesses that use it to help support its ongoing maintenance and development via sponsorship.

As a token of appreciation for sponsoring, you can unlock awesome rewards and extra features 🦄✨

Ecosystem

: Automatically import publications from BibTeX

Inspiration

are building with this template.

Features

Enhanced Markdown Support

Hugo Blox now supports GitHub and Obsidian-style callouts! Use standard Markdown alert syntax like > [!NOTE] for better portability.

Page builder - Create anything with no-code and
Edit any type of content - Blog posts, publications, talks, slides, projects, and more!
Create content in , , or
Plugin System - Fully customizable
Display Code and Math - Code syntax highlighting and LaTeX math supported
Integrations - , , Maps, Contact Forms, and more!
Beautiful Site - Simple and refreshing one-page design
Industry-Leading SEO - Help get your website found on search engines and social media
Media Galleries - Display your images and videos with captions in a customizable gallery
Mobile Friendly - Look amazing on every screen with a mobile friendly version of your site
Multi-language - 35+ language packs including English, 中文, and Português
Multi-user - Each author gets their own profile page
Privacy Pack - Assists with GDPR
Stand Out - Bring your site to life with animation, parallax backgrounds, and scroll effects
One-Click Deployment - No servers. No databases. Only files.

Version Requirements

The new Markdown alert syntax requires Hugo v0.132.0 or later. Make sure you’re using a compatible version!

Themes

Hugo Blox and its templates come with automatic day (light) and night (dark) mode built-in. Visitors can choose their preferred mode by clicking the sun/moon icon in the header.

for your site. Themes are fully customizable.

License

Released under the license.

🧠 Sharpen your thinking with a second brain

Thu, 26 Oct 2023 00:00:00 +0000

Create a personal knowledge base and share your knowledge with your peers.

Hugo Blox web framework empowers you with one of the most flexible note-taking capabilities out there.

Create a powerful knowledge base that works on top of a local folder of plain text Markdown files.

Use it as your second brain, either publicly sharing your knowledge with your peers via your website, or via a private GitHub repository and password-protected site just for yourself.

Mindmaps

Hugo Blox supports a Markdown extension for mindmaps.

With this open format, can even edit your mindmaps in other popular tools such as Obsidian.

Simply insert a Markdown code block labelled as markmap and optionally set the height of the mindmap as shown in the example below.

Mindmaps can be created by simply writing the items as a Markdown list within the markmap code block, indenting each item to create as many sub-levels as you need:


```markmap {height="200px"}
- Hugo Modules
- Hugo Blox
- blox-plugins-netlify
- blox-plugins-netlify-cms
- blox-plugins-reveal
```

renders as

- Hugo Modules
- Hugo Blox
- blox-plugins-netlify
- blox-plugins-netlify-cms
- blox-plugins-reveal

Anh here’s a more advanced mindmap with formatting, code blocks, and math:


```markmap
- Mindmaps
- Links
- [Hugo Blox Docs](https://docs.hugoblox.com/)
- [Discord Community](https://discord.gg/z8wNYzb)
- [GitHub](https://github.com/HugoBlox/hugo-blox-builder)
- Features
- Markdown formatting
- **inline** ~~text~~ *styles*
- multiline
text
- `inline code`
- Math: $x = {-b \pm \sqrt{b^2-4ac} \over 2a}$
```

renders as

- Mindmaps
- Links
- [Hugo Blox Docs](https://docs.hugoblox.com/)
- [Discord Community](https://discord.gg/z8wNYzb)
- [GitHub](https://github.com/HugoBlox/hugo-blox-builder)
- Features
- Markdown formatting
- **inline** ~~text~~ *styles*
- multiline
text
- `inline code`
- Math: $x = {-b \pm \sqrt{b^2-4ac} \over 2a}$

Highlighting

Highlight important text with mark:

<mark>Highlighted text</mark>

Callouts

Use (aka asides, hints, or alerts) to draw attention to notes, tips, and warnings.

Use the > [!NOTE] syntax to create a callout.

> [!NOTE]
> A Markdown aside is useful for displaying notices, hints, or definitions to your readers.

renders as

Note

A Markdown aside is useful for displaying notices, hints, or definitions to your readers.

Or use the warning callout type so your readers don’t miss critical details:

Warning

A Markdown aside is useful for displaying notices, hints, or definitions to your readers.

📈 Communicate your results effectively with the best data visualizations

Wed, 25 Oct 2023 00:00:00 +0000

Hugo Blox is designed to give technical content creators a seamless experience. You can focus on the content and Hugo Blox handles the rest.

Use popular tools such as Plotly, Mermaid, and data frames.

Embed Rich Content

HuggingFace Model

google/embeddinggemma-300m

sentence-similarity

2.070728M

1.824k

HuggingFace Dataset

fka/prompts.chat

dataset

a.k.a. Awesome ChatGPT Prompts This is a Dataset Repository mirror of prompts.chat — a social platform for AI prompts. 📢 Notice This Hugging Face dataset is a mirror. For the …

34.054k

9.778k

GitHub Repository

kit

by HugoBlox • HTML

🧱 Describe your site, AI builds it, you own it as Markdown. Snap together Tailwind blocks like Lego — landing pages, blogs, portfolios, docs & more. No AI slop. Free to deploy anywhere 👇

2.948k

9.62k

academic blog documentation-tool +17

Custom embed with manual data

My Custom Resource

example.com

A great resource for learning

Custom Images

Embed beautiful images from any source with Hugo image processing (Unsplash, custom URLs, etc.):

Data Visualization Guide

example.com

Beautiful data visualization workspace

You can also add images to any platform embed:

plotly.py

by plotly • Python

The interactive graphing library for Python ✨

2.832k

774

18.721k

d3 dashboard declarative +11

Charts

Hugo Blox supports the popular format for interactive data visualizations. With Plotly, you can design almost any kind of visualization you can imagine!

Save your Plotly JSON in your page folder, for example line-chart.json, and then add the {{< chart data="line-chart" >}} shortcode where you would like the chart to appear.

Demo:

You might also find the useful.

Diagrams

Hugo Blox supports the Mermaid Markdown extension for diagrams.

An example flowchart:

```mermaid
graph TD
A[Hard] -->|Text| B(Round)
B --> C{Decision}
C -->|One| D[Result 1]
C -->|Two| E[Result 2]
```

renders as

graph TD A[Hard] -->|Text| B(Round) B --> C{Decision} C -->|One| D[Result 1] C -->|Two| E[Result 2]

An example sequence diagram:

```mermaid
sequenceDiagram
Alice->>John: Hello John, how are you?
loop Healthcheck
John->>John: Fight against hypochondria
end
Note right of John: Rational thoughts!
John-->>Alice: Great!
John->>Bob: How about you?
Bob-->>John: Jolly good!
```

renders as

sequenceDiagram Alice->>John: Hello John, how are you? loop Healthcheck John->>John: Fight against hypochondria end Note right of John: Rational thoughts! John-->>Alice: Great! John->>Bob: How about you? Bob-->>John: Jolly good!

An example class diagram:

```mermaid
classDiagram
Class01 <|-- AveryLongClass : Cool
Class03 *-- Class04
Class05 o-- Class06
Class07 .. Class08
Class09 --> C2 : Where am i?
Class09 --* C3
Class09 --|> Class07
Class07 : equals()
Class07 : Object[] elementData
Class01 : size()
Class01 : int chimp
Class01 : int gorilla
Class08 <--> C2: Cool label
```

renders as

classDiagram Class01 <|-- AveryLongClass : Cool Class03 *-- Class04 Class05 o-- Class06 Class07 .. Class08 Class09 --> C2 : Where am i? Class09 --* C3 Class09 --|> Class07 Class07 : equals() Class07 : Object[] elementData Class01 : size() Class01 : int chimp Class01 : int gorilla Class08 <--> C2: Cool label

An example state diagram:

```mermaid
stateDiagram
[*] --> Still
Still --> [*]
Still --> Moving
Moving --> Still
Moving --> Crash
Crash --> [*]
```

renders as

stateDiagram [*] --> Still Still --> [*] Still --> Moving Moving --> Still Moving --> Crash Crash --> [*]

Data Frames

Save your spreadsheet as a CSV file in your page’s folder and then render it by adding the Table shortcode to your page:

{{< table path="results.csv" header="true" caption="Table 1: My results" >}}

renders as

Table 1: My results
customer_id	score
1	0
2	0.5
3	1

Interactive Buttons

Add engaging call-to-action buttons to your data visualization posts:

Basic Buttons

Learn Plotly

{{< button url="/" >}}Contact Us{{< /button >}}

{{< button url="https://plotly.com/python/" new_tab="true" style="secondary" >}}Learn Plotly{{< /button >}}

Styled Buttons for Data Actions

View Dashboard

Download Data

View Source Code

{{< button url="#" style="primary" size="lg" align="center" icon="chart-bar" >}}View Dashboard{{< /button >}}

{{< button url="/data/results.csv" style="outline" icon="document-arrow-down" >}}Download Data{{< /button >}}

{{< button url="https://github.com/HugoBlox" new_tab="true" style="ghost" icon="arrow-top-right-on-square" icon_position="right" >}}View Source Code{{< /button >}}

Multiple Aligned Buttons

Try Jupyter

Open in Colab

{{< button url="https://jupyter.org/" new_tab="true" style="secondary" rounded="full" align="center" >}}Try Jupyter{{< /button >}}

{{< button url="https://colab.research.google.com/" new_tab="true" style="primary" rounded="full" align="center" icon="rocket-launch" >}}Open in Colab{{< /button >}}

👩🏼‍🏫 Teach academic courses

Tue, 24 Oct 2023 00:00:00 +0000

is designed to give technical content creators a seamless experience. You can focus on the content and the Hugo Blox Builder which this template is built upon handles the rest.

Embed videos, podcasts, code, LaTeX math, and even test students!

On this page, you’ll find some examples of the types of technical content that can be rendered with Hugo Blox.

Citation

Here’s an example of citing a publication using the cite shortcode:

Le Anh Tuan (2019). An example preprint / working paper.

Preprint Code Slides Dataset Poster Source Document Video Custom Link Preprint

You can also use the default view by omitting the view parameter:

Le Anh Tuan

Equal contribution

, Robert Ford

Equal contribution

(2013). An example conference paper. In ICW.

PDF Code Dataset Slides Source Document Video PDF DOI

Video

Teach your course by sharing videos with your students. Choose from one of the following approaches:

Youtube:

{{< youtube D2vj0WcvH5c >}}

Bilibili:

{{< bilibili BV1WV4y1r7DF >}}

Video file

Videos may be added to a page by either placing them in your assets/media/ media library or in your , and then embedding them with the video shortcode:

{{< video src="my_video.mp4" controls="yes" >}}

Podcast

You can add a podcast or music to a page by placing the MP3 file in the page’s folder or the media library folder and then embedding the audio on your page with the audio shortcode:

{{< audio src="ambient-piano.mp3" >}}

Try it out:

Test students

Provide a simple yet fun self-assessment by revealing the solutions to challenges with the spoiler shortcode:

{{< spoiler text="👉 Click to view the solution" >}}
You found me!
{{< /spoiler >}}

renders as

👉 Click to view the solution

You found me 🎉

Math

Hugo Blox Builder supports a Markdown extension for $\LaTeX$ math. Enable math by setting the math: true option in your page’s front matter, or enable math for your entire site by toggling math in your config/_default/params.yaml file:

features:
 math:
 enable: true

To render inline or block math, wrap your LaTeX math with $...$ or $$...$$, respectively.

Example math block:

$$
\gamma_{n} = \frac{ \left | \left (\mathbf x_{n} - \mathbf x_{n-1} \right )^T \left [\nabla F (\mathbf x_{n}) - \nabla F (\mathbf x_{n-1}) \right ] \right |}{\left \|\nabla F(\mathbf{x}_{n}) - \nabla F(\mathbf{x}_{n-1}) \right \|^2}
$$

renders as

$$\gamma_{n} = \frac{ \left | \left (\mathbf x_{n} - \mathbf x_{n-1} \right )^T \left [\nabla F (\mathbf x_{n}) - \nabla F (\mathbf x_{n-1}) \right ] \right |}{\left \|\nabla F(\mathbf{x}_{n}) - \nabla F(\mathbf{x}_{n-1}) \right \|^2}$$

Example inline math $\nabla F(\mathbf{x}_{n})$ renders as $\nabla F(\mathbf{x}_{n})$.

Example multi-line math using the math linebreak (\\):

$$f(k;p_{0}^{*}) = \begin{cases}p_{0}^{*} & \text{if }k=1, \\
1-p_{0}^{*} & \text{if }k=0.\end{cases}$$

renders as

$$ f(k;p_{0}^{*}) = \begin{cases}p_{0}^{*} & \text{if }k=1, \\ 1-p_{0}^{*} & \text{if }k=0.\end{cases} $$

Code

Hugo Blox Builder utilises Hugo’s Markdown extension for highlighting code syntax. The code theme can be selected in the config/_default/params.yaml file.

```python
import pandas as pd
data = pd.read_csv("data.csv")
data.head()
```

renders as

import pandas as pd
data = pd.read_csv("data.csv")
data.head()

Inline Images

{{< icon name="python" >}} Python

renders as

Python

✅ Manage your projects

Mon, 23 Oct 2023 00:00:00 +0000

Easily manage your projects - create ideation mind maps, Gantt charts, todo lists, and more!

Ideation

Hugo Blox supports a Markdown extension for mindmaps.

Simply insert a Markdown code block labelled as markmap and optionally set the height of the mindmap as shown in the example below.

Mindmaps can be created by simply writing the items as a Markdown list within the markmap code block, indenting each item to create as many sub-levels as you need:


```markmap {height="200px"}
- Hugo Modules
- Hugo Blox
- blox-plugins-netlify
- blox-plugins-netlify-cms
- blox-plugins-reveal
```

renders as

- Hugo Modules
- Hugo Blox
- blox-plugins-netlify
- blox-plugins-netlify-cms
- blox-plugins-reveal

Diagrams

Hugo Blox supports the Mermaid Markdown extension for diagrams.

An example Gantt diagram:

```mermaid
gantt
section Section
Completed :done, des1, 2014-01-06,2014-01-08
Active :active, des2, 2014-01-07, 3d
Parallel 1 : des3, after des1, 1d
Parallel 2 : des4, after des1, 1d
Parallel 3 : des5, after des3, 1d
Parallel 4 : des6, after des4, 1d
```

renders as

gantt section Section Completed :done, des1, 2014-01-06,2014-01-08 Active :active, des2, 2014-01-07, 3d Parallel 1 : des3, after des1, 1d Parallel 2 : des4, after des1, 1d Parallel 3 : des5, after des3, 1d Parallel 4 : des6, after des4, 1d

Todo lists

You can even write your todo lists in Markdown too:

- [x] Write math example
 - [x] Write diagram example
- [ ] Do something else

renders as

Write math example
- Write diagram example
Do something else

Blog | Tuan Le's Blog

Researching CDC: How to sync data between two mongo clusters using debezium, rabbitmq and golang consumer. Part 1

Introduction

Why This Research Exists?

Solutions Applied

1. Direct Event Push

2. Outbox Pattern with Polling publisher

Trade-offs Summary

Towards a Better Approach

Transaction log tailing including transaction log

Introducing Debezium

Architecture Overview

Conclusion

Understanding n8n: Architecture and Core Concepts

What is n8n?

Core n8n Architecture

Key Components

How It Works Together (Simplified Flow)

Scalability Considerations

Planning Your Custom Node

1. Choosing the Node Type

2. Selecting a Node Building Style

3. Designing the Node UI

🎉 Easily create your own simple yet highly customizable blog

Overview

Get Started

Crowd-funded open-source software

Ecosystem

Inspiration

Features

Themes

License

🧠 Sharpen your thinking with a second brain

Mindmaps

Highlighting

Callouts

Did you find this page helpful? Consider sharing it 🙌

📈 Communicate your results effectively with the best data visualizations

Embed Rich Content

google/embeddinggemma-300m

fka/prompts.chat

kit

My Custom Resource

Custom Images

Data Visualization Guide

plotly.py

Charts

Diagrams

Data Frames

Interactive Buttons

Basic Buttons

Styled Buttons for Data Actions

Multiple Aligned Buttons

Did you find this page helpful? Consider sharing it 🙌

👩🏼‍🏫 Teach academic courses

Citation

Video

Podcast

Test students

Math

Code

Inline Images

Did you find this page helpful? Consider sharing it 🙌

✅ Manage your projects

Ideation

Diagrams

Todo lists

Did you find this page helpful? Consider sharing it 🙌