IVE ALREADY CODED MOST OF THIS PROGRAM BUT I CANNOT DO THIS LAST QUESTION. I\'VE

Question

IVE ALREADY CODED MOST OF THIS PROGRAM BUT I CANNOT DO THIS LAST QUESTION. I'VE PROVIDED THE CODE FOR ALL FOUR CLASSES AT THE BOTTOM:

• Add a new class Drill. The class will implement all methods needed for studying (e.g., pickCard, getChallenge, getResponse, checkAnswer, hasCard). Under this method, the flashcards of Box 1 are copied into a separate pool. The application randomly picks one flashcard from the pool. If the user knows the answer, the card is removed from the pool. If the user does not know the answer the card remains in the pool. Drill ends when the pool is empty. When users end the drill session, the pool can be discarded. During a drill session the Leitner boxes remain unchanged.

------code-----

Lietner.java:

package mngrace_a5;

import java.util.ArrayList;
import java.util.Iterator;
import java.util.Random;

/**
* A class implementing the Leitner system for flashcard boxes.
*
* @invariant boxes != null
*/
public class Leitner
{
/**
* Constructs a new Leitner object.
*
* @precondition theBoxes != null
*/
public Leitner(ArrayList theBoxes)
{
assert theBoxes != null;

boxes = theBoxes;
}

/**
* Internal method that computes the weight of a box.
*

* The weight of a Leitner box depends on the box id the number of cards
* it stores. The current implementation assigns a card in a lower box double
* the weight as a card in the next higher box.
*
* @param box the box for which this weight is computed.
* @param numBoxes total number of boxes. This is needed to compute the
* distance from box to the box with the highest Leitner id.
* @return the weight of this box.
*
* @precondition box != null && numBoxes >= box.id()
*/
private int weight(Box box, int numBoxes)
{
assert box != null && numBoxes >= box.id();

// Each card in box number N has a weight of 2 ^ (|boxes| - N).
// Note: 1 << X left-shifts the number 1 by X position ( = 2^X )
return (1 << (numBoxes - box.id())) * box.size();
}

/** Computes the total weight of all boxes */
private int totalWeight()
{
int num = 0;

for (Box box : boxes)
{
num += weight(box, boxes.size());
}

return num;
}

/**
* Picks a random card from all boxes.
*
* Cards in lower boxes receive higher weight (priority).
*
* @precondition there must be at least one box with one card.
*/
public void pickCard()
{
int maxRand = totalWeight();
assert maxRand > 0; // there must be at least one box with one card.

// rndVal identifies the box where the card is located
// | weight of box 1 | w. of box 2 | weight of Box 3 ... |
// ^=0 [min rndVal] ^first value for box 2 ^max(rndVal) [last value for highest box]
// ^ rndVal [falls in Box 2]
int rndVal = rand.nextInt(maxRand);

// Identify the box from where the card will be picked.
Iterator boxIter = boxes.iterator();
Box currBox = boxIter.next();

// Subtract the box's weight from rndVal until rndVal < weight(box)
// => we have identified the box.
while (rndVal >= weight(currBox, boxes.size()))
{
rndVal -= weight(currBox, boxes.size());

assert boxIter.hasNext();
currBox = boxIter.next();
}

// Set the box from where the next card will be picked.
box = currBox;

// All cards in a box are equally likely to be picked,
// thus we generate another pseudo random number.
card = box.get(rand.nextInt(box.size()));

// Choose side of card.
displayFront = (rand.nextInt(2) == 0);
}

/**
* Returns the question for the last picked card.
*
* @return the question
*
* @precondtion pickCard() has been called at least once
*/
public String getQuestion()
{
assert card != null;

return displayFront ? card.getChallenge() : card.getResponse();
}

/**
* Returns the answer for the last picked card.
*
* @return the answer
*
* @precondtion pickCard() has been called at least once
*/
public String getAnswer()
{
assert card != null;

return displayFront ? card.getResponse() : card.getChallenge();
}

/**
* Validates the response against the last picked card.
*
* @param s the response.
* @return true, iff s was correct. s is assumed to be correct, if it is empty
* or the string returned by getAnswer equals s.
*
* @precondition s != null
*/
public boolean checkAnswer(String s)
{
assert s != null;

int newBoxId = 1; // set target box id in case the response is incorrect
boolean isCorrect = "".equals(s) || getAnswer().equals(s);

if (isCorrect)
{
// response is correct -> update target box id
newBoxId = Math.min(box.id() + 1, boxes.size());
}

box.remove(card); // remove card from current box
boxes.get(newBoxId - 1).add(card); // store card in target box

return isCorrect;
}

private final Random rand = new Random(); /// random number generator.
private Box box = null; /// last picked box.
private FlashCard card = null; /// last picked card.
private boolean displayFront = true; /// indicates side to display.
private final ArrayList boxes; /// the list of boxes.
}

FlashCardApp.java

package mngrace_a5;

import java.util.ArrayList;
import java.util.Iterator;

/**
* Class implementing a flashcard application
*
* @invariant boxes != null && boxes.size() > 0
*/
public class FlashCardApp
{
private final int MAX_BOXES = 5; /// Number of boxes

/** Constructs a new flashcard app object and sets the number of boxes */
public FlashCardApp()
{
for (int i = 1; i <= MAX_BOXES; ++i)
{
boxes.add(new Box(i));
}
}

/** Returns an object according to the Leitner study method. */
public Leitner leitner() { return new Leitner(boxes); }

/** Returns an iterator that lists all flash cards in the system. */
public Iterator listAll()
{
ArrayList allCards = new ArrayList();

for (Box box : boxes)
{
allCards.addAll(box.getCards());
}

return allCards.iterator();
}

/**
* Returns an iterator that lists all flash cards containing a given pattern.
*
* @param pattern search pattern for texts on flashcards.
* @return Iterator where all elements contain pattern in either
* front or back of the card.
*
* @precondition pattern != null
*/
public Iterator list(String pattern)
{
assert pattern != null;

ArrayList foundCards = new ArrayList();

for (Box box : boxes)
{
Iterator cardIter = box.iterator();

while (cardIter.hasNext())
{
FlashCard currCard = cardIter.next();
boolean inclCard = ( currCard.getChallenge().indexOf(pattern) >= 0
|| currCard.getResponse().indexOf(pattern) >= 0
);

if (inclCard) foundCards.add(currCard);
}
}

return foundCards.iterator();
}

/**
* Returns an iterator that lists all flash cards in a given box.
*
* @param boxid Leitner box id.
* @return Iterator where all elements contain pattern in either
* front or back of the card.
*
* @precondition 0 < boxid <= number of boxes in the app
*/
public Iterator list(int boxid)
{
assert boxid > 0 && boxid <= boxes.size();

return boxes.get(boxid - 1).iterator();
}

/**
* Creates a new flashcard and adds it to the first box.
*
* @precondition challenge != null && response != null
*/
public void create(String challenge, String response)
{
assert challenge != null && response != null;

boxes.get(0).add(new FlashCard(challenge, response));
}

private final ArrayList boxes = new ArrayList(); /// List of boxes
}

FlashCard.java

package mngrace_a5;

/**
* Represents a text based flashcard.
*
* This class is immutable.
* @invariant front != null && back != null
*/
public class FlashCard
{
/**
* Constructor setting up a flashcard object.
*
* @param challenge front of the card
* @param response back of the card
* @precondition challenge != null && response != null
*/
FlashCard(String challenge, String response)
{
assert challenge != null && response != null;

front = challenge;
back = response;
}

/** Returns the front side. */
public String getChallenge() { return front; }

/** Returns the back side. */
public String getResponse() { return back; }

/**
* Produces a textual representation of this flashcard.
*
* @return a string containing two lines, one for front and one for back.
*/
public String toString()
{
return front + ' ' + back + ' ';
}

private final String front;
private final String back;
}

Box.java

package mngrace_a5;
import java.util.List;
import java.util.ArrayList;
import java.util.Iterator;

/**
* Implements a Leitner box holding flashcards.
*
* @invariant boxnum > 0 && cards != null
*/
public class Box
{
/**
* Constructs a new Box.
*
* @param num Leitner box number
* @precondition num > 0
*/
public Box(int num)
{
assert num > 0;

boxnum = num;
cards = new ArrayList();
}

/** returns current number of flashcards in the box. */
public int size() { return cards.size(); }

/**
* returns specific flashcard from the box.
*
* @param idx flashcard index
* @return the queried Flashcard
* @precondition 0 <= idx < size()
*/
public FlashCard get(int idx) { return cards.get(idx); }

/**
* Adds a new flashcard to this box.
*
* @param card the new flashcard
* @precondition card is not yet in the box
*/
public void add(FlashCard card) { cards.add(card); }

/**
* Removes a flashcard from the box.
*
* @param card the card to be removed
* @precondition card is in this box
*/
public void remove(FlashCard card) { cards.remove(card); }

/**
* Removes a flashcard from the box.
*
* @param idx of the card to be removed
* @precondition 0 <= idx < size()
*/
public void remove(int idx) { cards.remove(idx); }

/**
* Returns an iterator to go through all flashcards in this box.
*
* @return an Iterator
*/
public Iterator iterator() { return cards.iterator(); }

/** Returns the Leitner-ID of this box. */
public int id() { return boxnum; }

/**
* "Convenience function" that allows to add all of these cards to another box.
* @return a reference to the internal box storage.
* @todo This breaks encapsulation and exposes more class internals than needed.
*/
List getCards() { return cards; }

private final int boxnum; /// Leitner id
private final ArrayList cards; /// Flashcard storage
}

Explanation / Answer

Apache Drill

allows users to explore any type of data using ANSI SQL. This is great, but Drill goes even further than that and allows you to create custom functions to extend the query engine. These custom functions have all the performance of any of the Drill primitive operations, but allowing that performance makes writing these functions a little trickier than you might expect.

In this article, I'll explain step by step how to create and deploy a new function using a very basic example. Note that you can find lot of information about Drill Custom Functions in the documentation.

Let's create a new function that allows you to mask some characters in a string, and let's make it very simple. The new function will allow user to hide x number of characters from the start and replace then by any characters of their choice. This will look like:
MASK( 'PASSWORD' , '#' , 4 ) => ####WORD

You can find the full project in the following Github Repository.

As mentioned before, we could imagine many advanced features to this, but my goal is to focus on the steps to write a custom function, not so much on what the function does.

Prerequisites

For this you will need:

·         Java Developer Kit 7 or later

·         Apache Drill 1.1 or later

·         Maven 3.0 or later

Dependencies

The following Drill dependency should be added to your maven project

<dependency>

      <groupId>org.apache.drill.exec</groupId>

      <artifactId>drill-java-exec</artifactId>

      <version>1.1.0</version>

</dependency>

Source

The Mask function is an implementation of the DrillSimpleFunc.

Developers can create 2 types of custom functions:

·         Simple Functions: these functions have a single row as input and produce a single value as output

·         Aggregation Functions: that will accept multiple rows as input and produce one value as output

Simple functions are often referred to as UDF's which stands for user defined function. Aggregation functions are referred to as UDAF which stands for user defined aggregation function.

In this example, we just need to transform the value of a column on each row, so a simple function is enough.

CREATE THE FUNCTION

The first step is to implement the DrillSimpleFunc interface.

package org.apache.drill.contrib.function;

import org.apache.drill.exec.expr.DrillSimpleFunc;

import org.apache.drill.exec.expr.annotations.FunctionTemplate;

@FunctionTemplate(

        name="mask",

        scope= FunctionTemplate.FunctionScope.SIMPLE,

        nulls = FunctionTemplate.NullHandling.NULL_IF_NULL

)

public class SimpleMaskFunc implements DrillSimpleFunc{

    public void setup() {

    }

    public void eval() {

    }

}

The behavior of the function is driven by annotations (line 6-10) * Name of the function * Scope of the function, in our case Simple * What to do when the value is NULL, in this case Reverse will just returns NULL

Now we need to implement the logic of the function using setup() and eval()methods.

·         setup is self-explanatory, and in our case we do not need to setup anything.

·         eval that is the core of the function. As you can see this method does not have any parameter, and return void. So how does it work?

In fact the function will be generated dynamically (see DrillSimpleFuncHolder), and the input parameters and output holders are defined using holders by annotations. Let's look into this.

import io.netty.buffer.DrillBuf;

import org.apache.drill.exec.expr.DrillSimpleFunc;

import org.apache.drill.exec.expr.annotations.FunctionTemplate;

import org.apache.drill.exec.expr.annotations.Output;

import org.apache.drill.exec.expr.annotations.Param;

import org.apache.drill.exec.expr.holders.IntHolder;

import org.apache.drill.exec.expr.holders.NullableVarCharHolder;

import org.apache.drill.exec.expr.holders.VarCharHolder;

import javax.inject.Inject;

@FunctionTemplate(

        name = "mask",

        scope = FunctionTemplate.FunctionScope.SIMPLE,

        nulls = FunctionTemplate.NullHandling.NULL_IF_NULL

)

public class SimpleMaskFunc implements DrillSimpleFunc {

    @Param

    NullableVarCharHolder input;

    @Param(constant = true)

    VarCharHolder mask;

    @Param(constant = true)

    IntHolder toReplace;

    @Output

    VarCharHolder out;

    @Inject

    DrillBuf buffer;

    public void setup() {

    }

    public void eval() {

    }

}

We need to define the parameters of the function. In this case we have 3 parameters, each defined using the @Param annotation. In addition, we also have to define the returned value using the@Output annotation.

The parameters of our mask function are:

·         A nullable string

·         The mask char or string

·         The number of characters to replace starting from the first

The function returns :

·         A string

For each of these parameters you have to use an holder class. For the String, this is managed by aVarCharHolder or NullableVarCharHolder -lines 21, 24,30- that provides a buffer to manage larger objects in a efficient way. Since we are manipulating a VarChar you also have to inject another buffer that will be used for the output -line 33-. Note that Drill doesn't actually use the Java heap for data being processed in a query but instead keeps this data off the heap and manages the life-cycle for us without using the Java garbage collector.

We are almost done since we have the proper class, the input/output object, we just need to implement the eval() method itself, and use these objects.

public void eval() {

    // get the value and replace with

    String maskValue = org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.getStringFromVarCharHolder(mask);

    String stringValue = org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(input.start, input.end, input.buffer);

    int numberOfCharToReplace = Math.min(toReplace.value, stringValue.length());

    // build the mask substring

    String maskSubString = com.google.common.base.Strings.repeat(maskValue, numberOfCharToReplace);

    String outputValue = (new StringBuilder(maskSubString)).append(stringValue.substring(numberOfCharToReplace)).toString();

    // put the output value in the out buffer

    out.buffer = buffer;

    out.start = 0;

    out.end = outputValue.getBytes().length;

    buffer.setBytes(0, outputValue.getBytes());

}

The code is quite simple:

·         Get the mask itself - line 4

·         Get the value - line 5

·         Get the number of character to replace - line 7

·         Generate a new string with masked values - lines 10/11

·         Create and populate the output buffer - lines 14 to 17

This code does, however, look a bit strange to somebody used to reading Java code. This strangeness arises because the final code that is executed in a query will actually be generated on the fly. This allows Drill to leverage Java's just-in-time (JIT) compiler for maximum speed. To make this work, you have to respect some basic rules:

·         Do not use imports, but instead use the fully qualified class name, this is what is done on line 10 with the Strings class. (coming from the Google Guava API packaged in Apache Drill)

·         The ValueHolders classes, in our case VarCharHolder and IntHolder should be manipulated like structs, so you must call helper methods, for example getStringFromVarCharHolder andtoStringFromUTF8. Calling methods like toString will result in very bad problems.

Starting in Apache Drill 1.3.x, it is mandatory to specify the package name of your function in the./resources/drill-module.conf file as follow:

drill {

classpath.scanning {

    packages : ${?drill.classpath.scanning.packages} [

      org.apache.drill.contrib.function

    ]

}

}

We are now ready to deploy and test this new function.

Package

Once again since, Drill will generate source, you must prepare your package in a way that classes and sources of the function are present in the classpath. This is different from the way that Java code is normally packaged but is necessary for Drill to be able to do the necessary code generation. Drill uses the compiled code to access the annotations and uses the source code to do code generation.

An easy way to do that is to use maven to build your project, and, in particular, use the maven-source-plugin like this in your pom.xml file:

<plugin>

    <groupId>org.apache.maven.plugins</groupId>

    <artifactId>maven-source-plugin</artifactId>

    <version>2.4</version>

    <executions>

        <execution>

            <id>attach-sources</id>

            <phase>package</phase>

            <goals>

                <goal>jar-no-fork</goal>

            </goals>

        </execution>

    </executions>

</plugin>

Now, when you build using mvn package, Maven will generate 2 jars:

·         The default jar with the classes and resources (drill-simple-mask-1.0.jar)

·         A second jar with the sources (drill-simple-mask-1.0-sources.jar)

Finally you must add a drill-module.conf file in the resources folder of your project, to tell Drill that your jar contains a custom function. If you have no specific configuration to set for your function you can keep this file empty.

We are all set, you can now package and deploy the new function, just package and copy the Jars into the Drill 3rd party folder; $DRILLHOME/jars/3rdparty , where $DRILLHOME being your Drill installation folder.

mvn clean package

cp target/*.jar $DRILL_HOME/jars/3rdparty

Restart drill.

Run !

You should now be able to use your function in your queries:

SELECT MASK(first_name, '*' , 3) FIRST , MASK(last_name, '#', 7) LAST FROM cp.`employee.json` LIMIT 5;

Conclusion

In this simple project you have learned how to write, deploy and use a custom Apache Drill Function. You can now extend this to create your own function.

One important thing to remember when extending Apache Drill (using a custom function, storage plugin or format), is that Drill runtime is generating dynamically lot of code. This means you may have to use a very specific pattern when writing and deploying your extensions. With our basic function this meant we had to:

·         deploy classes AND sources

·         use fully Qualified Class Names

·         use value holder classes and helper methods to manipulate parameters *

Related Questions

Navigate

• Browse (All)
• Browse I
• Subjects

---

---

Integrity-first tutoring: explanations and feedback only — we do not complete graded work. Learn more.