MongoDB Saver

The MongoDB Saver is an essential component of DatErica that allows you to store your processed data directly into a MongoDB database. Here’s how you can configure it to meet your data persistence requirements.

Configuration Fields

Mongo URI

Enter the MongoDB URI which should include the database name. If the database name is not specified, MongoDB will default to creating a database named "test".

The MongoDB URI is a connection string that specifies the client, server, and authentication details required to connect to a MongoDB database. Below is a breakdown of the common MongoDB URI syntax elements and their meanings:

Standard URI Format:

mongodb+srv://[username]:[password]@[host]/[database]?retryWrites=true&w=majority

Components Explained:

• mongodb+srv:// - The protocol used; srv indicates that this is a connection string for MongoDB with DNS Seedlist Connection Format.

• [username] - The username for database authentication.

• [password] - The corresponding password for the username.

• [host] - The hostname or IP address of the MongoDB server.

• [database] - The specific database name to connect to within the MongoDB server. If the database doesn't exist, MongoDB creates it upon the first data insertion.

Optional Parameters:

• retryWrites=true - An option for automatic retry of write operations if they fail initially.

• w=majority - A write concern setting that waits for the write operation to be acknowledged by the majority of hosts in a replica set.

Example URI:

mongodb+srv://johndoe:secret123@cluster0.example.mongodb.net/myDatabase?retryWrites=true&w=majority

New Collection Name

Specify the name for the new collection where your data will be saved.

Use Timestamps

Toggle the ‘Use Timestamps’ switch to include created_at and updated_at timestamps in your saved documents. When this feature is enabled, each document saved will automatically include these timestamps, reflecting when each document was created and last updated.

Data Inserting Mode

Choose between two modes of data insertion:

• Insert: Adds new documents to the collection.
• Upsert: Updates existing documents based on a primary key, or inserts new documents if they don't already exist.

Build Primary Key (Applicable for Upsert Mode)

If the 'Upsert' mode is selected, you’ll need to define a primary key. You can select one or more fields from your data to constitute the primary key using the multi-value dropdown.

Important Considerations

• When you define a primary key and execute the pipeline, DatErica will automatically create the necessary index in the corresponding collection.
• Please note that DatErica does not currently support editing or deleting indexes through the GUI. Should you need to modify an index associated with a primary key, you would have to do so manually in the database or opt to create a new collection altogether.

Remember to test your connection after configuration to ensure everything is set up correctly. Once satisfied with the settings, hit 'Save Config' to preserve your MongoDB Saver configuration.