Friday, 9 April 2021

What is Apache Spark ?

 
APACHE SPARK
General engine which can combine difference type of computations (SQL queries, text processing & ML)
Main factor : Speed

Spark :
  • Integrates closely with other Bigdata tools (can run in Hadoop clusters & access any Hadoop data source + Cassandra)
  • offers simple APIs in Python, Java, Scala, R & SQL and built-in libs
  • allows querying data via SQL and HQL (Hive QL)
  • supports many data sources : Hive tables, Parquet & formats : JSON, CSV, txt etc.
Spark Architecture
  • Executor : separate JVMs running on working node, runs tasks on worker nodes
  • Driver node : executes the 

Hadoop vs. Spark
  • Spark : streaming (incoming in real time), Hadoop : batch mode

Monday, 29 March 2021

What is GitOps ?

GitOps

  • implementation of CD (Continuous Delivery) on cloud infra
  • focused on developers using :
    • Git
    • CD tool(s) & automated pipeline
    • IaC (Infra as Code) & configuration

  • Implementation
    • Manage 2 repos
      1. Source code & manifests
      2. Deployment configs
    • Use one of the 2 deployment strategy
      • Push based
        • Code pushed > trigger pipeline > build & push container image > update env config repo > trigger deploy pipeline
      • Pull based (preferred, more secure)
        • Code pushed > trigger pipeline > build & push container image > update env config repo
        • Operator observes the changes in env repo > update infra
        • Operator observes the changes in image registry > deploy new version

      • Deploy on different environments when specific branches are updated

    • Prerequisites
      • Git
      • Infra manageable by IaC
      • Kubernetes (Operators for pull-based deployment)

    • Tools for GitOps
      • ArgoCD : K8S operator with an UI
      • JenkinsX : CD on K8S
      • Flux : K8S operator
      • GitKube : build/deploy docker images on K8S
      • Helm operator : K8S operator with Helm

    Monday, 22 March 2021

    How to quick start with Spring boot ?

    EXAMPLE : Create a simple API

    PRE-REQUISITES TO QUICK START WITH SPRING BOOT :

     - JDK
     - IDE
       STS (Spring Tool Suite) - wrapper over eclipse providing all the Spring boot support.
        OR
        Eclipse with Spring boot plugin
     - Maven (Dependency management tool + Build tool) : declares all the dependencies (in pom.xml) instead of downloading & putting Jars in classpath 


    1. Open IDE > Create new project > Maven project

    2. Provide project config :
        Check Create simple project
        Provide group id = com.shaan , artifact id = userapi , packaging = Jar
        Click Finish

        creates : 
        - project skeleton (src/main/java, src/main/resource, src/test/java, src/test/resource)
        - pom.xml with info provided in project config
        - JRE libs

    3. Set Java version :
        <property>
           <java.version>1.8</
    java.version>
        </property>

    4. Add spring dependency as parent (inherits all spring boot dependencies + config in the application) :
        <parent>
           <groupId>org.springFramework.boot<groupId>
           <artifactId>spring-boot-starter-parent</artifactId>
           <version>1.4.2.RELEASE</version>
         </parent>

    5. Add Web application dependency (As we need to create a REST API)

        <depedencies>
           <groupId>org.springFramework.boot<groupId>
           <artifactId>spring-boot-starter-web</artifactId>
         </
    depedencies>

    After saving pom.xml, it will add all the required Jars under : 
       - Maven dependencies

    6. Update project config with pom.xml  (There may be errors in markers tab of IDE)
      - Right click on project name > Maven > Update project

    7. Create a main Java class : MainApi
       
    @SpringBootApplication
      public clas MainApi {
           p.s.v.p.(String[] args) {
             
    // static method taking class having main()                          SpringApplication.run(UserApi.clas, args);
           }
      }

    If you run MainApi class, with the help of Spring boot it will :
      - Setup config
      - Start Spring application context 
      - Check class paths (registers annotations)
      - Run container (embedded tomcat server)

    8. Create REST controller (API which can handle HTTP requests)
    @RestController
    public class UserController {
       
    // Method mapping with URL
        @RequestMapping("/hello")
        public String show() {
            return "Hello";
        }
    }

    9. Run MainApi class
    It will enable registration of all the annotations and able to expose the API using controller. Once the sever is started you can test it.

    10. Test your API 
     - Open browser and call API through declared URL : localhost:8080/hello


    Why Spring boot ?

    Spring boot ease the creation of Spring application.
    Eliminates needs of :
      - Lot of configuration
      - Import lot of Jar
    gives more time to focus on business logic


    Spring boot relies on Spring framework.

    Sprint is whole framework for enterprise Java applications handles common concerns :
     - Dependency injection
     - handling HTTP requests
     - Transactions
     - DB connection through templates (for SQL & NoSQL DBs)

    so that you focus on business problem & logic and handle the common concerns through annotations.

    WHY SPRING BOOT ?
     Spring is :
     - a big framework
     - need Complex setup
     - need lot of configuration

    Spring boot :
     - addresses above issues
     - Stand-alone : implicit server (Tomcat) which enables application ready to run, so eliminates external runtime server needs


    NEEDS TO QUICK START WITH SPRING BOOT :
     - JDK
     - 
    Spring boot provides STS (Spring Tool Suite) - wrapper over eclipse providing all the Spring boot support.
     - Maven (Dependency management tool + Build tool) : declares all the dependencies (in pom.xml) instead of downloading & putting Jars in classpath 

    Saturday, 20 March 2021

    How to transfer files from iphone to computer ?

    • Connect iphone with USB cable
    • Open itunes and search for option "Authorize this computer"
    • Login with apple ID to authorize
    • Disconnect cable and connect again 

    Monday, 3 June 2019

    Types of Cloud


    Public : AWS, GCE, MS Azure, Alibaba cloud, IBM cloud, Flexible engine by Orange
    Private : in-premises cloud using vmware, Openstack
    Hybrid : Public + Private clouds
    Multicloud : Combination of different cloud offers

    Wednesday, 15 May 2019

    What are 2 types of tokens ?


    TOKENS are :

    • generated by Authorization (Auth) server
    • issued when client requests Auth server


    2 types of Tokens
    1. ACCESS TOKEN

    • sent by client as request param / header to Resource server
    • have Limited lifetime / Expiry time (defined by Auth server)


    2. REFRESH TOKEN

    • issued with Access token but not sent in each request from client to Resource server
    • sent to Auth server to renew Access token when it expires