Scala installation on AWS

Scala Version: scala-2.10.5

Installation Steps:

[hadoop@ip-xx-y-zz-12 ~]$ cd /mnt1/analytics/arvind/softwares/
[hadoop@ip-xx-y-zz-12 softwares]$ wget http://downloads.typesafe.com/scala/2.10.5/scala-2.10.5.tgz
[hadoop@ip-xx-y-zz-12 softwares]$ tar -xzvf scala-2.10.5.tgz
[hadoop@ip-xx-y-zz-12 softwares]$ vim ~/.bashrc
    export SCALA_HOME=/mnt1/analytics/arvind/scala-2.10.5
    export PATH=$PATH:/mnt1/analytics/arvind/scala-2.10.5/bin
   
    CLASSPATH=/mnt1/analytics/arvind/scala-2.10.5/lib/*

[hadoop@ip-xx-y-zz-12 softwares]$ source ~/.bashrc
[hadoop@ip-xx-y-zz-12 softwares]$ scala -version
    Scala code runner version 2.10.5 -- Copyright 2002-2013, LAMP/EPFL

Reference:

https://gist.github.com/diegopacheco/ee621e8106d0736f21f2d0a6c46261f2

---

Redis Standalone Installation on AWS

1. Login to AWS using PuTTy

    $ ec2-user
    $ sudo su -
    $ cd /mnt1/analytics/arvind

2. Download, extract and make Redis

    $ wget http://download.redis.io/redis-stable.tar.gz
    $ tar xvzf redis-stable.tar.gz
    $ cd redis-stable
    $ make
    $ sudo make install  // Copies redis-server & redis-cli to /usr/local/bin/

3. Test the Connections:

+-------------------------------------------------------------------+
| $ redis-server                    |    $ redis-cli ping            |
| [...] ... The server is now         |      PONG                        |
|        ... ready to accept         |    $ redis-cli                 |
|        ... connections on            |    redis 127.0.0.1:6379> ping    |
|        ... port 6379                |    PONG                        |
|                                    |    redis 127.0.0.1:6379> exit    |
|                                    |    $ redis-cli shutdown        |
+-------------------------------------------------------------------+

4. Create directories to store Redis config files and data

    $ sudo mkdir /etc/redis
    $ sudo mkdir /var/redis
   

5. Copy the init script from utils directory into /etc/init.d

    $ sudo cp utils/redis_init_script /etc/init.d/redis_6379
   

6. Copy the template configuration file from root directory into /etc/redis/

    $ sudo cp redis.conf /etc/redis/6379.conf

7. Edit the configuration file:

    A. Set daemonize to yes
    B. Set the pidfile to /var/run/redis_6379.pid
    C. Change the port accordingly. Default port is 6379.
    D. Set your preferred loglevel.
    E. Set the logfile to /var/log/redis_6379.log
    F. Set the dir to /var/redis/6379 (very important step!)
    G. Comment the bind directive
    H. Set the protected mode option to 'no'

7. Create a directory inside /var/redis for data and work

    $ sudo mkdir /var/redis/6379
   

8. Add the new Redis init script to all the default runlevels

    $ sudo update-rc.d redis_6379 defaults
   

9. Run the instance

    $ sudo /etc/init.d/redis_6379 start
   

10. Do a test save

    $ redis-cli save  // dump file will be stored into /var/redis/6379/dump.rdb
   

11. Check the log file

    $ tail -f /var/log/redis_6379.log
   

12. Connect with Server / Instance IP

    $ redis-cli -h 10.3.50.12 -p 6379
    redis 10.3.50.12:6379> ping
    PONG
   
Reference: https://redis.io/topics/quickstart

---

Kafka Installation & Configuration on Windows 10

1. Download & install Java

2. Create a Junction Link.  (Needed as the Java Path contains spaces)

    C:\Windows\system32>mklink /J "C:\Program_Files" "C:\Program Files"
    Junction created for C:\Program_Files <<===>> C:\Program Files

3. Set Path and Classpath for Java

4. Download and extract Apache Kafka

5. Set Path for Kafka bin/windows directory (in Environment Variables)

6. Download and extract Apache Zookeeper

7. Set Path for Zookeeper bin directory

8. Edit Zookeeper config:

    dataDir=C:\\zookeeper\\zookeeper-3.4.9\\data

9. Edit Kafka windows server.properties:

    log.dirs=C:\\kafka\\kafka_2.11-0.10.1.0\\kafka-logs

10. Run Zookeeper

    C:\> zkServer
   

11. Run Kafka Server

    C:\kafka\kafka_2.11-0.10.1.0> .\bin\windows\kafka-server-start.bat .\config\server.properties
    - OR -
    C:\> kafka-server-start.bat %KAFKA_HOME%\config\server.properties

12. Create Topic

    C:\> kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

13. List Topics

    C:\> kafka-topics.bat --list --zookeeper localhost:2181

14. Describe Topic

    C:\> kafka-topics.bat --describe --zookeeper localhost:2181 --topic test

15. Create Producer

    C:\> kafka-console-producer.bat --broker-list localhost:9092 --topic test

16. Create Consumer

    C:\> kafka-console-consumer.bat --zookeeper localhost:2181 --topic test

17. Create Consumer (Read messages from beginning)

    C:\> kafka-console-consumer.bat --zookeeper localhost:2181 --topic test --from-beginning

18. Delete Topic

    C:\> kafka-run-class.bat kafka.admin.TopicCommand --delete --topic test --zookeeper localhost:2181

19. Stop Kafka Server

    C:\kafka\kafka_2.11-0.10.1.0> .\bin\windows\kafka-server-stop .\config\server.properties
    - OR -
    C:\> kafka-server-stop.bat %KAFKA_HOME%\config\server.properties
   

20. Terminate Zookeeper

    Press Ctrl + C
    Terminate batch job (Y/N)? Y   
---