It takes good amount of time to find a file in Windows machines. Here is simple system where we can index all the desktop files in solr search engine, a famous open source search engine and can be searched in seconds.

Solr is an open source search engine from apache solr https://lucene.apache.org/solr/.

Step 1: Download the solr and run it in cloud mode. Here is the guide to get start on solr. https://lucene.apache.org/solr/guide/7_7/solr-tutorial.html#solr-tutorial

once you download and extract the solr into a file, just start the solr with following command

>./bin/solr start -c -p 8983 -s ../example/cloud/node1/solr

This will start the first node.  This may take approximately 30 sec to start it. And it will also start zookeeper ensemble to manage the nodes. Once first node starts, here is the command to start second node.

>./bin/solr start -c -p 7574 -s ../example/cloud/node2/solr -z localhost:9983

this will start the solr engine in two node cluster and can be accessed at http://localhost:8983/solr

Step 2: Create a collection for indexing files

>solr create -c latest -s 2 -rf 2

Step 3: Write a program to extract and index the files names in solr search Engines.

package com.kvn.web.solrClients;

import java.io.File;
import java.io.IOException;
import java.sql.Timestamp;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Optional;
import java.util.function.Predicate;
import java.util.stream.Collectors;

import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.impl.HttpSolrClient;

public class ListFilesOfType {

	static String baseFoldersToIndex[] = { "C:/docs", "C:/Java Majestic", "C:/JavaMajestic",
			"C:/Users/prabhukvn/Documents", "c:/Users/prabhukvn/Downloads", "C:/POCS", "C:/softwares" };

	Predicate<File> isFile = (f1) -> f1.isFile();
	String baseUrl = "http://localhost:8983/solr/latest";
	HttpSolrClient client = null;

	static List<String> fileTypes = new ArrayList<>();
	static List<String> filePaths = new ArrayList<>();

	static {
		fileTypes.add(".pdf");
		fileTypes.add(".doc");
		fileTypes.add(".docx");

	}
	static {
		filePaths.addAll(Arrays.asList(baseFoldersToIndex));

	}

	public static void main(String[] args) {
		ListFilesOfType obj = new ListFilesOfType();

		long startTime = System.currentTimeMillis();
		filePaths.stream().map(path -> new File(path)).forEach(obj::processFile);
		System.out.println("Total Time:"+(System.currentTimeMillis()-startTime));

	}

	public void processFile(File file) {
		//System.out.println("----" + file.getAbsolutePath() + "--------");
		File listOfFiles[] = file.listFiles();

		if (listOfFiles != null) {
			List<FileDoc> fileDocs = Arrays.stream(listOfFiles).parallel().filter(isFile.and(this::filterFiles))
					.map(this::createFileDoc).collect(Collectors.toList());
			if (fileDocs != null && fileDocs.size() > 0) {
				this.sendToSolar(fileDocs);
			}
			Arrays.stream(listOfFiles).filter(f2 -> f2.isDirectory()).forEach(this::processFile);
		}	

	}

	private void sendToSolar(List<FileDoc> fileDocs) {
		try {
			// System.out.println("--------------------------------------");
			fileDocs.parallelStream().forEach(f3 -> System.out.println(f3.getName()));
			HttpSolrClient solrClient = this.connect();
			solrClient.addBeans(fileDocs);
			solrClient.commit();
		} catch (SolrServerException e) {
			e.printStackTrace();
		} catch (IOException e) {
			e.printStackTrace();
		}

	}

	FileDoc createFileDoc(File file) {
		FileDoc fileDoc = new FileDoc(file.getName(), file.getAbsolutePath(), new Timestamp(file.lastModified()));
		return fileDoc;
	}

	public boolean filterFiles(File f) {
		
		Optional<String> f4 = fileTypes.parallelStream().filter(fileType -> f.getName().endsWith(fileType)).findFirst();
		return f4.isPresent();
	}

	/**
	 * Connect to solr using solr client.
	 * 
	 * @return
	 */
	private HttpSolrClient connect() {
		if (client == null) {
			client = new HttpSolrClient.Builder(baseUrl).withConnectionTimeout(10000).withSocketTimeout(60000).build();
		}
		return client;

	}

}

And Maven Dependency
		<dependency>
			<groupId>org.apache.solr</groupId>
			<artifactId>solr-solrj</artifactId>
			<version>7.7.0</version>
		</dependency>

And use http://localhost:8983/solr/#/pdf-files/query to search the data.

Or http://localhost:8983/solr to open the dashboard

The following program can handle almost 65000 (65K) request with a single instance.

/**
 * This server socket program has been created to test the programe with -client and -server JVM parameters.
 */
package sockets;

import java.io.IOException;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;
import java.net.SocketException;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.LinkedBlockingQueue;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
 * @author prabhu kvn
 *
 */
public class ServerSocketDemoWithBlockingQueue {

	Logger logger = LoggerFactory.getLogger(this.getClass().getName());

	/**
	 * A blocking queue to handle connections. this can hold up to one lakh sockets
	 */
	/**
	 * Default values
	 */
	private int queueLength = 10000;
	int serverThreadCount = 1;
	BlockingQueue<Socket> queue = new LinkedBlockingQueue<>(queueLength);

	/**
	 * Default Constructor
	 */
	public ServerSocketDemoWithBlockingQueue() {

	}

	/**
	 * Default Constructor
	 */
	public ServerSocketDemoWithBlockingQueue(int serverThreadCount, int queueLength) {

		this.serverThreadCount = serverThreadCount;
		this.queueLength = queueLength;
		this.queue = new LinkedBlockingQueue<>(queueLength);

	}

	/**
	 * Entry point for server start up
	 * 
	 * @param args
	 */
	public static void main(String[] args) {

		ServerSocketDemoWithBlockingQueue demo = new ServerSocketDemoWithBlockingQueue();
		demo.startServer();
	}

	/**
	 * Starting the server to handle asynch connections
	 */
	private void startServer() {

		logger.info("Starting the server with {} threads and queue size {}", serverThreadCount, queueLength);
		ExecutorService service = Executors.newFixedThreadPool(serverThreadCount);
		for (int i = 0; i < serverThreadCount; i++) {
			service.submit(new ServerProcessing(queue));
		}
		ServerSocket serverSocket = null;
		try {

			// create the server socket
			serverSocket = new ServerSocket(8181, 100000);

			while (true) {
				Socket socket = serverSocket.accept();
				try {
					queue.put(socket);
				} catch (InterruptedException e) {
					logger.error("Problem in accepting new connections : {}", e);
					e.printStackTrace();
				}

			}

		} catch (IOException e) {
			logger.error("Problem in creating server socket {}", e);
			e.printStackTrace();
		} finally {
			if (null != serverSocket) {
				try {
					serverSocket.close();
					logger.info("Server Socket closed Successfully.");
				} catch (IOException e) {
					logger.error("Problem in closing server socket.");
					e.printStackTrace();
				}
			}
		}

	}

}

/**
 * Processing class
 * 
 * @author prabhukvn
 *
 */
class ServerProcessing extends Thread {
	BlockingQueue<Socket> queue;
	volatile int counter = 0;

	public ServerProcessing(BlockingQueue<Socket> queue) {
		this.queue = queue;
	}

	@Override
	public void run() {
		while (true) {
			try {

				process(queue.take());
			} catch (InterruptedException e) {
				// TODO Auto-generated catch block
				e.printStackTrace();
			} catch (SocketException e) {
				// TODO Auto-generated catch block
				e.printStackTrace();
			} catch (IOException e) {
				// TODO Auto-generated catch block
				e.printStackTrace();
			}
		}
	}

	/**
	 * Execute the actual work.F
	 * @param socket
	 * @return
	 * @throws IOException
	 * @throws SocketException
	 * @throws InterruptedException
	 */
	private int process(Socket socket) throws IOException, SocketException, InterruptedException {
		System.out.println("Server Socket:" + socket);
		OutputStream out = socket.getOutputStream();
		out.write(("Request received" + counter).getBytes());
		counter++;
		socket.setKeepAlive(true);
		// Thread.sleep(1000);
		// allow client to close the connection
		socket.close();
		return counter;
	}
}

Selection sort is a kind of sorting mechanism where every element is compared with the every other element to sort the elements.
The following programs performs a selection sort on integers and also also calculates number of iteration required for sorting entire array. And also it simulates how the array is actually sorted on each iteration.

 

package sorting;

import java.util.Arrays;

public class SelectionSort {

public static void main(String[] rawInput) {

int numberOfIterations = 0;
int length = rawInput.length;
int input[] = new int[length];
for (int k = 0; k < length; k++) {
input[k] = Integer.parseInt(rawInput[k]);
}

System.out.println("Raw input" + Arrays.toString(input));

if (length == 0) {
System.out.println("Invalid input ");
} else {
for (int i = 0; i < length; i++) {
numberOfIterations++;
// iterate the rest of the array
for (int j = i; j input[j]) {
int temp = input[i];
input[i] = input[j];
input[j] = temp;
System.out.println(Arrays.toString(input));
}

}

}
}

System.out.println("after sorting" + Arrays.toString(input));
System.out.println("Total Number of iterations:" + numberOfIterations);

}

}

If we run the above program with following input,
Raw input[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
[8, 9, 7, 6, 5, 4, 3, 2, 1, 0]
[7, 9, 8, 6, 5, 4, 3, 2, 1, 0]
[6, 9, 8, 7, 5, 4, 3, 2, 1, 0]
[5, 9, 8, 7, 6, 4, 3, 2, 1, 0]
[4, 9, 8, 7, 6, 5, 3, 2, 1, 0]
[3, 9, 8, 7, 6, 5, 4, 2, 1, 0]
[2, 9, 8, 7, 6, 5, 4, 3, 1, 0]
[1, 9, 8, 7, 6, 5, 4, 3, 2, 0]
[0, 9, 8, 7, 6, 5, 4, 3, 2, 1]
[0, 8, 9, 7, 6, 5, 4, 3, 2, 1]
[0, 7, 9, 8, 6, 5, 4, 3, 2, 1]
[0, 6, 9, 8, 7, 5, 4, 3, 2, 1]
[0, 5, 9, 8, 7, 6, 4, 3, 2, 1]
[0, 4, 9, 8, 7, 6, 5, 3, 2, 1]
[0, 3, 9, 8, 7, 6, 5, 4, 2, 1]
[0, 2, 9, 8, 7, 6, 5, 4, 3, 1]
[0, 1, 9, 8, 7, 6, 5, 4, 3, 2]
[0, 1, 8, 9, 7, 6, 5, 4, 3, 2]
[0, 1, 7, 9, 8, 6, 5, 4, 3, 2]
[0, 1, 6, 9, 8, 7, 5, 4, 3, 2]
[0, 1, 5, 9, 8, 7, 6, 4, 3, 2]
[0, 1, 4, 9, 8, 7, 6, 5, 3, 2]
[0, 1, 3, 9, 8, 7, 6, 5, 4, 2]
[0, 1, 2, 9, 8, 7, 6, 5, 4, 3]
[0, 1, 2, 8, 9, 7, 6, 5, 4, 3]
[0, 1, 2, 7, 9, 8, 6, 5, 4, 3]
[0, 1, 2, 6, 9, 8, 7, 5, 4, 3]
[0, 1, 2, 5, 9, 8, 7, 6, 4, 3]
[0, 1, 2, 4, 9, 8, 7, 6, 5, 3]
[0, 1, 2, 3, 9, 8, 7, 6, 5, 4]
[0, 1, 2, 3, 8, 9, 7, 6, 5, 4]
[0, 1, 2, 3, 7, 9, 8, 6, 5, 4]
[0, 1, 2, 3, 6, 9, 8, 7, 5, 4]
[0, 1, 2, 3, 5, 9, 8, 7, 6, 4]
[0, 1, 2, 3, 4, 9, 8, 7, 6, 5]
[0, 1, 2, 3, 4, 8, 9, 7, 6, 5]
[0, 1, 2, 3, 4, 7, 9, 8, 6, 5]
[0, 1, 2, 3, 4, 6, 9, 8, 7, 5]
[0, 1, 2, 3, 4, 5, 9, 8, 7, 6]
[0, 1, 2, 3, 4, 5, 8, 9, 7, 6]
[0, 1, 2, 3, 4, 5, 7, 9, 8, 6]
[0, 1, 2, 3, 4, 5, 6, 9, 8, 7]
[0, 1, 2, 3, 4, 5, 6, 8, 9, 7]
[0, 1, 2, 3, 4, 5, 6, 7, 9, 8]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
after sorting[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Total Number of iterations:65

The queue which works with First In First Out (FIFO) mechanism. This is the queue program in java with insertion complexity of O(1) in java.

/**
*
*/
package java8pract.datastructures;

import java.util.ArrayList;
import java.util.List;

/**
* @author prabhukvn
*
*/
public class CustomQueue<T extends Comparable> {
Node firstNode;
Node lastNode;
Node tempNode;
int stackSize = 0;

public void push(T element) {
Node node = null;
if (null == firstNode) {
node = new Node(null, element, null);
firstNode = node;
stackSize++;

} else {

Node middleNode = tempNode;
node = new Node(middleNode, element, null);
middleNode.next = node;
lastNode = node;
stackSize++;
}
tempNode = node;

}

public int getSize() {
return this.stackSize;
}

public T pop() {

T element = firstNode.element;
firstNode = firstNode.next;
firstNode.previous=null;
stackSize--;
return element;

}

/**
* Get all the elements in linked list
*
* @return
*/
public List<Node> getAll() {
List<Node> list = new ArrayList();
Node temp = firstNode;
while (temp != null) {
list.add(temp);
System.out.println(temp.element);
temp = temp.next;

}
return list;
}

}

This is a stack program where the algorithm works on Last in First Out (LIFO) concept. This is a typical stack program with insertion complexity as O(1) i.e constant.


/**
*
*/
package java8pract.datastructures;

import java.util.ArrayList;
import java.util.List;

/**
* @author prabhukvn
*
*/
public class CustomStack<T extends Comparable> {
Node firstNode;
Node lastNode;
Node tempNode;
int stackSize = 0;

public void push(T element) {
Node node = null;
if (null == firstNode) {
node = new Node(null, element, null);
firstNode = node;
stackSize++;

} else {

Node middleNode = tempNode;
node = new Node(middleNode, element, null);
middleNode.next = node;
lastNode = node;
stackSize++;
}
tempNode = node;

}

public int getSize() {
return this.stackSize;
}

public T pop() {

T element = lastNode.element;
lastNode = lastNode.previous;
lastNode.next=null;
stackSize--;
return element;

}

/**
* Get all the elements in linked list
*
* @return
*/
public List<Node> getAll() {
List<Node> list = new ArrayList();
Node temp = firstNode;
while (temp != null) {
list.add(temp);
System.out.println(temp.element);
temp = temp.next;

}
return list;
}

}

A simple linked list program where insertion complexity is O(1) order of one i.e independent of data size.

Here is the code for the same

Here is the Node/Element/Data object of linked list

Note: This program has been written with minimum code so that users can copy and experiment with it. This is not a full fledged program, hence not recommended for production use.


public class Node {

Node previous;
T element;
Node next;

public Node(Node previous, T element, Node next) {
this.previous=previous;
this.element=element;
this.next=next;

}

@Override
public String toString() {
return "Node [element=" + element + "]";
}

}

And here is the linked list implementation,

package java8pract.linkedlist_custom;
/**
* An implementation of linked list. Entire linked list is dependent on first element in the list.
*/
import java.util.ArrayList;
import java.util.List;

public class CustomLinkedList {

Node firstNode;
Node tempNode;
Node lastNode;
/**
* Add an element to the linked list
* +/----------------------------------------
* + 1. if first node is null, create a node and assign it to first node.
* + 2. and assign the newly created node to a temp Node;
* + 3. for consecutive additions, get the temp node into a middle methods level node
* + 4. Create new node and assign previous node as middle node.
* + 5.update the last node.
* + 6. middle node.next to newly created node
* + 7. and update temp node with newly created node.
* @param element
*/

public void add(T element) {
Node node = null;
if (null == firstNode) {
node = new Node(null, element, null);
firstNode = node;

} else {
Node middleNode = tempNode;
node = new Node(middleNode, element, null);
lastNode = node;
middleNode.next = node;

}
tempNode = node;
}

/**
* Get all the elements in linked list
*
* @return
*/
public List getAll() {
List list = new ArrayList();
Node temp = firstNode;
while (temp != null) {
list.add(temp);
System.out.println(temp.element);
temp = temp.next;

}
return list;
}

/**
* Get all the elements in the linked list in reverse order.
*
* @return
*/
public List getAllReverse() {
List list = new ArrayList();
Node temp = lastNode;
while (temp != null) {
list.add(temp);
System.out.println(temp.element);
temp = temp.previous;

}
return list;

}

public Node get(T t) {

Node temp = firstNode;
while (temp != null) {

if (temp.element !=null && temp.element.compareTo(t)==0) {
break;
} else {
temp = temp.next;
}
}

return temp;

}
/**
* Get the first node (HEAD) in the linked list
* @return
*/
public Node getFirstNode(){
return firstNode;
}
/**
* Get the last node in the linked list.
* @return
*/
public Node getLastNode(){
return lastNode;
}

}


An easiest way to create mongodb cluster in MacOs environment.

These steps can be executed one after the other in Mac Terminal.
Note: Make sure that the folders paths are correct before running these commands.

Create Initial directories required for mongo db cluter

cd /Users/prabhu/Documents/softwares/mongodb-osx-x86_64-4.0.6/bin/data

mkdir replica1 replica2 replica3 replica4 replica5 replica6 replica7 replica8 replica9 config1 config2 config3

Sudo chmod –R 777 repl*

Sudo chmod –R 777 conf*

shard1

./mongod –dbpath=/Users/prabhu/Documents/softwares/mongodb-osx-x86_64-4.0.6/bin/data/replica1 –port=27011 –bind_ip_all –replSet=replica1 –shardsvr

./mongod –dbpath=/Users/prabhu/Documents/softwares/mongodb-osx-x86_64-4.0.6/bin/data/replica2 –port=27012 –bind_ip_all –replSet=replica1 –shardsvr

./mongod –dbpath=/Users/prabhu/Documents/softwares/mongodb-osx-x86_64-4.0.6/bin/data/replica3 –port=27013 –bind_ip_all –replSet=replica1 –shardsvr

rs.initiate({_id:”replica1″,members:[{“_id”:1,”host”:”localhost:27011″}]})

rs.add(“localhost:27012”)

rs.add(“localhost:27013”)

shard2

./mongod –dbpath=/Users/prabhu/Documents/softwares/mongodb-osx-x86_64-4.0.6/bin/data/replica4 –port=27021 –bind_ip_all –replSet=replica2 –shardsvr

./mongod –dbpath=/Users/prabhu/Documents/softwares/mongodb-osx-x86_64-4.0.6/bin/data/replica5 –port=27022 –bind_ip_all –replSet=replica2 –shardsvr

./mongod –dbpath=/Users/prabhu/Documents/softwares/mongodb-osx-x86_64-4.0.6/bin/data/replica6 –port=27023 –bind_ip_all –replSet=replica2 –shardsvr

rs.initiate({_id:”replica2″,members:[{“_id”:1,”host”:”localhost:27021″}]})

rs.add(“localhost:27022”)

rs.add(“localhost:27023”)

Shard 3

./mongod –dbpath=/Users/prabhu/Documents/softwares/mongodb-osx-x86_64-4.0.6/bin/data/replica7 –port=27031 –bind_ip_all –replSet=replica3 –shardsvr

./mongod –dbpath=/Users/prabhu/Documents/softwares/mongodb-osx-x86_64-4.0.6/bin/data/replica8 –port=27032 –bind_ip_all –replSet=replica3 –shardsvr

./mongod –dbpath=/Users/prabhu/Documents/softwares/mongodb-osx-x86_64-4.0.6/bin/data/replica9 –port=27033 –bind_ip_all –replSet=replica3 –shardsvr

rs.initiate({_id:”replica3″,members:[{“_id”:1,”host”:”localhost:27031″}]})

rs.add(“localhost:27032”)

rs.add(“localhost:27033”)

config

./mongod –dbpath=/Users/prabhu/Documents/softwares/mongodb-osx-x86_64-4.0.6/bin/data/config1 –port=27018 –bind_ip_all –replSet=config1 –configsvr

./mongod –dbpath=/Users/prabhu/Documents/softwares/mongodb-osx-x86_64-4.0.6/bin/data/config2 –port=27019 –bind_ip_all –replSet=config1 –configsvr

./mongod –dbpath=/Users/prabhu/Documents/softwares/mongodb-osx-x86_64-4.0.6/bin/data/config3 –port=27020 –bind_ip_all –replSet=config1 –configsvr

rs.initiate({_id:”config1″,members:[{“_id”:1,”host”:”localhost:27018″}]})

rs.add(“localhost:27019”)

rs.add(“localhost:27020”)

Start Mongos with config cluster
./mongos –configdb=”config1/localhost:27018,localhost:27019,localhost:27020″ –port=27017 –bind_ip_all

Connect to Mongos Server
./mongod –port 27017

sh.addShard(“replica1/localhost27011”)

sh.addShard(“replica2/localhost27021”)

sh.addShard(“replica3/localhost27031”)

sharding Collection

sh.enableSharding(“shoppingdb”); // enable sharding for db

sh.shardCollection(“shoppingdb.Orders”,{_id:1},true); // shard the collection

There are lot of articles which talks about domain driven design, Event Driven and CQRS architectural practices. However,In this blog post, an effort to put domain driven design (DDD), Event Driven and Command Query Responsibility Segregation in a simple application which will explain all of above concepts. This application has been developed using spring boot, axon ,axon db server, java programming language, maven for build and eclipse etc.
In this application we are just trying to create an order with some order items (Command) and retrieve the same (Query) using order id. This is a very common scenario in any e commerce application and also a good example to explain above concepts.
Here is the out line of flow,

And here is the git hub url for code

https://github.com/prabhukvn/axon-springboot-poc.git

Here is en example where iterating over an array is much faster than iterating over a list directly. This we can apply if we have to iterate a list over multiple time in a single thread.

private static void checkArrayList() {
ArrayList listOne = new ArrayList();

long startTime = System.nanoTime();
for (int i = 0; i <span id="mce_SELREST_start" style="overflow:hidden;line-height:0;">&#65279;</span>&lt; 1000; i++) {
listOne.add(i);
}
// Total time taken to construct an array list with 1000 elements
System.out.println(&quot;Total Time 1:&quot; + (System.nanoTime() - startTime));

// Iterate entire list
startTime = System.nanoTime();
for (int k : listOne) {

}
// total time to iterate over 1000 elements
System.out.println(&quot;Total Time 2:&quot; + (System.nanoTime() - startTime));

Integer a[] = new Integer[1000];
// Convert list into array
startTime = System.nanoTime();
listOne.toArray(a);
// time to convert list into array
System.out.println(&quot;Total Time 3:&quot; + (System.nanoTime() - startTime));
// Iterate entire array
startTime = System.nanoTime();
for (int k : a) {

}
// total time taken to iterate over array
System.out.println(&quot;Total Time 4:&quot; + (System.nanoTime() - startTime));
}