Mehdinfo

DevOps

Containerizing Jenkins and Nexus

by Mehdi El-Filahi

For people interested to be able to work with the Jenkins and Nexus into containers, i have created an infra configuration with docker that you can find here:

https://github.com/djmhd/DockerJenkinsNexus

The readme will help you run the containers and access the services.

Middleware

Get rid of Zookeeper and use Kraft

by Mehdi El-Filahi

It this post, i will explain how to be able to use Kraft and why use it.

Kraf is a concensus protocol allowing Kafka to work independently without Zookeeper.

This simplifies the fact there is nore more need of two different services (Kafka and Zookeeper) to share metadata. It also enables kafka fail over control to be almost instant. Thus the effect of kafka start and stop is faster.

The feature began with the version Kafka 2.8 and is enhanced with every new version BUT for the moment (June 2022), this solution is not yet production ready.

Of course this changes a little the infrastructure and the connection method.

	With Zookeeper	Without Zookeeper
Client and service configuration	zookeeper.connect=zookeeper:2181	bootstrap.servers=broker:9092
Schema registry configuration	kafkastore.connection.url=zookeeper:2181	kafkastore.bootstrap.servers=broker:9092
Kafka admin tool	kafka-topics –zookeeper zookeeper:2181	kafka-topics –bootstrap-server broker:9092 … –command-config properties to connect to brokers
REST Proxy API	v1	v2 and v3
Get Cluster ID	zookeeper-shell zookeeper:2181 get/cluster/id	`kafka-metadata-quorum` or view `metadata.properties` or `confluent cluster describe --url http://broker:8090 --output json`

How to configuration and start Kafka with Kraft

Generate a UID:

./bin/kafka-storage.sh random-uuid
xtzWWN4bTjitpL3kfd9s5g

Format the data directory to be compatible with Kraft (to run on each Kafka broker), do not forget to set the cluster ID.

./bin/kafka-storage.sh format -t <uuid> -c ./config/kraft/server.properties
Formatting /tmp/kraft-combined-logs --cluster-id XXXXXXXXXX

In the file server.properties, do not forget to set all ther brokers hosts in the line “controller.quorum.voters”

process.roles=controller
node.id=1
listeners=CONTROLLER://controller1.example.com:9093
controller.quorum.voters=1@controller1.example.com:9093,2@controller2.example.com:9093,3@controller3.example.com:9093

Start Kafka with pointing to kraft config file

./bin/kafka-server-start.sh ./config/kraft/server.properties

You are now ready to work with a broker running with no zookeeper dependencies to be healty.

Big Data

MongoDB and Python

by Mehdi El-Filahi

In this post, i will provide the minimal requirement to work with Python and apply CRUD operations.

Fist of all you will need Python V.3 and the module pymongo.

After installing python 3. Please read this article to see how to install pymongo ==> https://www.mongodb.com/docs/drivers/pymongo/

You will also see the compatibility matrix of pymongo driver and mongodb versions on the link above.

Now that you installed pymongo, we are ready to start.

I suggest you to install Visual Studio Code ==> https://code.visualstudio.com/download

My git repository is available here ==> https://github.com/djmhd/PythonMongoDB

Clone the repo and update the property file “config.properties” if needed to put the correct mongodb:

git clone https://github.com/djmhd/PythonMongoDB.git

Hostname port
Credentials: user password
Query parameter connection string
Database name

Run the file testCRUD.py

python3 testCRUD.py

This will, create a collection sampleCollection.

It will add two documents.

{
_id:6298b571f5643949130bda03
firstName: "John"
lastName: "Doe"
Address: "First addresss"
}
and
{
_id:6298b571f5643949130bda55
firstName: "Brandon"
lastName: "Don"
Address: "Secondaddresss"
}

The unit test file will wait for any key on the prompt command line.

You can go check that the two document are well inserted.

After it will delete the second document.

The unit test file will wait for any key on the prompt command line.

You can go check that the second document is well removed.

Then it will update the first document to update the field from the value to …..

{
_id:6298b571f5643949130bda03
firstName: "Marc"
lastName: "Doe"
Address: "First addresss"
}

Big Data

RDBMS vs NoSQL

by Mehdi El-Filahi

In this topic i will show the difference between RDBMS and NoSQL.

Topics	RDBMS	NoSQL
Formalization	Formalizazion must be respected to get consitent data	It does not need to respect formalization
Integrity constraint	Relation data consists in primary foreign keys	It support integrity but is not mandatory.
Data Structure	Data is composed with Table, Row and relations between data.	Data consists in key-value pairs or json data.
Schema and models	Data is less flexible and must live with fixes columns and data types.	Data can be unstructured or can have dynamic schema.
Scaling	Vertical scaling is quite easy but horizontal scale demands more effrots	Vertical and horizontal scale are more flexible.

The next table will help you understand the transition from RDBMS to NoSQL.

RDBMS	NoSQL
Database	Database
Table	Collection
Row	Document
Index	Index
Foreign Key	Reference

Archives