📋

Description

This error indicates that a MongoDB change stream has encountered an unrecoverable issue and has stopped processing real-time data changes. It typically occurs when the underlying connection or database state changes unexpectedly, preventing the stream from continuing.

💬

Error Message

Change Stream Fatal Error

🔍

Known Causes

4 known causes

⚠️

Network Connectivity Issues

Transient network disruptions, timeouts, or complete loss of connection between the client and the MongoDB replica set can terminate an active change stream.

⚠️

Replica Set State Changes

Events like primary elections, replica set reconfigurations, or member failures can disrupt change stream operations, leading to a fatal error.

⚠️

Insufficient User Permissions

The user account attempting to open or maintain the change stream may lack the necessary `changeStream` or other required privileges on the database or collection.

⚠️

Oplog Truncation

If the replica set's oplog (operation log) is truncated and the change stream's resume token is no longer available, the stream cannot resume and will encounter a fatal error.

🛠️

Solutions

4 solutions available

1. Restart the MongoDB Server easy

A simple restart can resolve transient issues causing change stream termination.

1

Identify your MongoDB server's service name. This can vary based on your operating system and installation method.

For Linux (systemd):
systemctl status mongod

For Linux (SysVinit):
service mongod status

For Windows (Services):
Open 'Services' application and look for 'MongoDB Server'.

2

Restart the MongoDB service.

For Linux (systemd):
sudo systemctl restart mongod

For Linux (SysVinit):
sudo service mongod restart

For Windows:
Right-click on 'MongoDB Server' in the Services application and select 'Restart'.

3

Verify that the MongoDB server has started successfully and check the logs for any recurring errors.

For Linux:
tail -f /var/log/mongodb/mongod.log

For Windows:
Check the MongoDB log file specified in your configuration.

2. Check and Recreate the Change Stream medium

The change stream cursor might have become invalid; recreating it can resolve the issue.

1

In your application code, locate the section where the change stream is opened. This typically involves a `db.collection.watch()` call.

Example (Node.js):
const changeStream = db.collection('mycollection').watch();

2

Implement error handling around the change stream to catch the termination error.

Example (Node.js):
const changeStream = db.collection('mycollection').watch();

changeStream.on('error', (error) => {
  console.error('Change stream error:', error);
  // Handle the error, e.g., by attempting to re-open the stream
  if (error.code === 280) {
    console.warn('Change stream terminated fatally. Attempting to reconnect...');
    // Implement reconnection logic here
  }
});

changeStream.on('change', (change) => {
  console.log('Change detected:', change);
});

3

When the error occurs, close the existing change stream cursor and open a new one. You might need to store the resume token from the last processed event to resume from the correct point.

Example (Node.js - simplified reconnection):
let resumeToken = null;

function openChangeStream() {
  const options = resumeToken ? { resumeAfter: resumeToken } : {};
  const changeStream = db.collection('mycollection').watch(options);

  changeStream.on('error', (error) => {
    console.error('Change stream error:', error);
    if (error.code === 280) {
      console.warn('Change stream terminated fatally. Reconnecting...');
      changeStream.close();
      setTimeout(openChangeStream, 5000); // Wait before retrying
    }
  });

  changeStream.on('change', (change) => {
    console.log('Change detected:', change);
    resumeToken = change._id; // Store the resume token
  });

  console.log('Change stream opened.');
}

openChangeStream();

3. Investigate Underlying System or Network Issues advanced

External factors can lead to change stream instability.

1

Review MongoDB server logs for any other critical errors or warnings that coincide with the change stream termination. Pay attention to messages related to disk I/O, memory usage, or network connectivity.

For Linux:
tail -f /var/log/mongodb/mongod.log

For Windows:
Check the MongoDB log file specified in your configuration.

2

Check the health of the underlying storage system. Disk full, slow disk performance, or hardware issues can affect MongoDB's ability to maintain change streams.

Linux:
df -h
iostat

Windows:
Check Disk Management and Event Viewer for storage-related errors.

3

Ensure stable network connectivity between the MongoDB server and the application consuming the change stream. Network interruptions or high latency can cause connections to drop.

Use tools like ping, traceroute, or MTR to test network stability.

4

Monitor system resources (CPU, memory, disk I/O) on the MongoDB server. Resource exhaustion can lead to unexpected behavior and process termination.

Linux:
top
virtop

Windows:
Task Manager, Resource Monitor

4. Ensure Sufficient WiredTiger Cache Size medium

An undersized cache can lead to excessive disk I/O and instability, impacting change streams.

1

Locate your MongoDB configuration file (e.g., `mongod.conf`).

Common locations:
- Linux: `/etc/mongod.conf`
- Windows: `C:\Program Files\MongoDB\Server\<version>\bin\mongod.cfg`

2

Within the configuration file, find or add the `storage.wiredTiger.engineConfig.cacheSizeGB` parameter. Ensure it is set to a reasonable value, typically 50% of your system's RAM, or a value that balances performance and other system needs.

Example (`mongod.conf`):
storage:
  wiredTiger:
    engineConfig:
      cacheSizeGB: 4  # Example: allocate 4GB for the cache

3

Restart the MongoDB server after modifying the configuration file.

Use the appropriate restart command for your operating system as described in Solution 1.

4

Monitor MongoDB's performance metrics, particularly cache hit rates and disk I/O, to assess the effectiveness of the cache size adjustment.

Use `mongostat` or MongoDB Atlas monitoring tools.

🔗

Related Errors

5 related errors

MongoDB Error 280: Change Stream Termination