Showing posts with label Hive. Show all posts
Showing posts with label Hive. Show all posts

Tuesday, 17 May 2016

How to use rank() function ?


Scenario. Rank the companies by country based on their turnover in Hive ?
companyName, country, turnover

Solution
Hive>
SELECT companyName, country, turnover,
       rank() over (PARTITION BY country ORDER BY turnover DESC) as rank
FROM sales;

SORT BY vs. ORDER BY


  • ORDER BY performs a total ordering of the query result set
    • All the data is passed through a single reducer
    • takes long time to execute for larger data

  • SORT BY sorts the data within each Reducer - Local ordering
    • Each each reducer’s output will be sorted
    • doesn't achieve a total ordering on the dataset