Sorting the strings in 20GB file with one string in each line.

Problem

Imagine you have a 20 GB file with one string per line. Explain how you would sort strings in the file.

This is quite similar to "K way merge problem" except for the fact that we first need to bring 20 GB file to the K partially sorted files to perform the K way merge.

Here is how we do it.

1. Split 20 GB files into K files 

Since we have limited memory, we first need to split 20 GB files into K sub-arrays, each of which contains M integers that can fit into the memory available.

2. Sort K files individually, $O(K \times M \log M)$

Now, we can sort each chunk with size of M either through quick sort or merge sort. In this case, in place Quick Sort will be more efficient. Running time of this step will be $O(K \times M \log M)$, were K is the number of sub-arrays and M is the number of integers in each sub-array.

3. Perform the K way merge, $O(N \log K)$

Now, we have K partially sorted array of integers, which can be merged through K way merge.
The details algorithm of K way merge can be found in "K way merge problem". 
Running time of K way merge will be $O(N \log K)$, where N is the number of integers in 20 GB file and K is the number of partially sorted array.

Overall time complexity of this sorting algorithm will be $$O(K \times M \log M)$$.


Comments

Post a Comment

Popular posts from this blog

Planting flowers with no adjacent flower plots

Image
Problem Suppose you have a long flowerbed in which some of the plots are planted and some are not. However, flowers cannot be planted in adjacent plots - they would compete for water and both would die. Given a flowerbed (represented as an array containing booleans), return if a given number of new flowers can be planted in it without violating the no-adjacent-flowers rule  Sample inputs  Input: 1,0,0,0,0,0,1,0,0  3 => true  4 => false  Input: 1,0,0,1,0,0,1,0,0  1 => true  2 => false  input: 0  1 => true  2 => false  public boolean canPlaceFlowers(List flowerbed, int numberToPlace) {  // Implementation here  }
Read more
  Given an array of  intervals  where  intervals[i] = [start i , end i ] , merge all overlapping intervals, and return  an array of the non-overlapping intervals that cover all the intervals in the input . Example 1: Input: intervals = [[1,3],[2,6],[8,10],[15,18]] Output: [[1,6],[8,10],[15,18]] Explanation: Since intervals [1,3] and [2,6] overlap, merge them into [1,6]. Example 2: Input: intervals = [[1,4],[4,5]] Output: [[1,5]] Explanation: Intervals [1,4] and [4,5] are considered overlapping.   Constraints: 1 <= intervals.length <= 10 4 intervals[i].length == 2 0 <= start i <= end i <= 10 4
Read more

Find the maximum number of bomb that can be detonated

  Problem You are given a list of bombs. The range of a bomb is defined as the area where its effect can be felt. This area is in the shape of a circle with the center as the location of the bomb. The bombs are represented by a 0-indexed 2D integer array bombs where bombs[i] = [xi, yi, ri]. xi and yi denote the X-coordinate and Y-coordinate of the location of the ith bomb, whereas ri denotes the radius of its range. You may choose to detonate a single bomb. When a bomb is detonated, it will detonate all bombs that lie in its range.  These bombs will further detonate the bombs that lie in their ranges. Given the list of bombs, return the maximum number of bombs that can be detonated if you are allowed to detonate only one bomb. Input: bombs = [[2,1,3],[6,1,4]] Output: 2 Explanation: The above figure shows the positions and ranges of the 2 bombs. If we detonate the left bomb, the right bomb will not be affected. But if we detonate the right bomb, both bombs will be detonated. So...
Read more